..most real world ancient oltp databases i worked in past 30 years, typically wrongly evolve data model (e.g. instead of subclassing a concept on a new table, often they just add 1 new column-flag, and store both type of entities on same table, meaning need to replicate everywhere the IF logic on that column table all over the place.
Once i create a business semantic layer, via 12 new tables (core business tables), aggregating data from 4 different databases with same entities but different modelling, additionally added business confirmed business rules boolean columns on couple of those new tables (e.g. is provider active, etc), that solution offered not only clean vision of core tables, and despite was used to fix CRM performance and data quality, those tables could simply be sent to an EDWH, removing any or most of any logic any/aoo services needed replicated to calculate business concepts (e.g. continuous tech debt).
Today everyone still following a data waterfall, meaning the poor guys of EDWH have to fix model, data, and ideally calculate those extra business column concepts. I'm very adept of Data Vault pattern, and this real world of core business tables wirh 30, 40 years or more, could/can be fixed, but DV supposed to waterfall data for EDWH marts, consumers, and not to server any OLTP or sideways systems.
And with the current AI hype, i still think what i did (semantic core usable both upstream oltp and downstream olap) should be implemented by each source system data team, a curated set ready to be sent to EDWH or some speedy answering OLTP requests as well.
But seems we continue following a waterfall data scheme (even with or without hybrid data-mesh or hub-spoke) where things fixed on last mile.. before was the poor guys maintaining the EDWH, now the fixing is about buulding AgenticAI semantic views etc, but.. that sadly continues to be a last mile fix.
KISS. That’s all IT will event-driven **TDD** say ;o) #lean #agile @Peter Andrew Nolan
Hello Jason, I can see i will have to do response posts to posts like this.
Nice post 😊
..most real world ancient oltp databases i worked in past 30 years, typically wrongly evolve data model (e.g. instead of subclassing a concept on a new table, often they just add 1 new column-flag, and store both type of entities on same table, meaning need to replicate everywhere the IF logic on that column table all over the place.
Once i create a business semantic layer, via 12 new tables (core business tables), aggregating data from 4 different databases with same entities but different modelling, additionally added business confirmed business rules boolean columns on couple of those new tables (e.g. is provider active, etc), that solution offered not only clean vision of core tables, and despite was used to fix CRM performance and data quality, those tables could simply be sent to an EDWH, removing any or most of any logic any/aoo services needed replicated to calculate business concepts (e.g. continuous tech debt).
Today everyone still following a data waterfall, meaning the poor guys of EDWH have to fix model, data, and ideally calculate those extra business column concepts. I'm very adept of Data Vault pattern, and this real world of core business tables wirh 30, 40 years or more, could/can be fixed, but DV supposed to waterfall data for EDWH marts, consumers, and not to server any OLTP or sideways systems.
And with the current AI hype, i still think what i did (semantic core usable both upstream oltp and downstream olap) should be implemented by each source system data team, a curated set ready to be sent to EDWH or some speedy answering OLTP requests as well.
But seems we continue following a waterfall data scheme (even with or without hybrid data-mesh or hub-spoke) where things fixed on last mile.. before was the poor guys maintaining the EDWH, now the fixing is about buulding AgenticAI semantic views etc, but.. that sadly continues to be a last mile fix.
🙏✨😊
Emanuel O.