Comments on: Some rare common sense http://andyonenterprisesoftware.com/2006/07/some-rare-common-sense/ Andy Hayler, founder of Kalido and The Information Difference, gives his views on the enterprise software market. Issues covered include data warehousing, master data management, business intelligence and data quality. Tue, 14 Oct 2008 10:54:38 +0000 http://wordpress.org/?v=2.0.3 by: Andy Hayler http://andyonenterprisesoftware.com/2006/07/some-rare-common-sense/#comment-110 Thu, 20 Jul 2006 10:32:04 +0000 http://andyonenterprisesoftware.com/2006/07/some-rare-common-sense/#comment-110 You raise an important distinction. You should NEVER lose track of valid data that has simply become out of date e.g. when a reorganisation happens you want to expire the old relationship and add a new one. Ideally all relationships should be time-stamped in thsi way, allowing you to recreate past history (this is the way that Kalido does it, for example). There is no reason why this cannot be achieved in custom warehouses also. What I meant was data that was not yet validated as being correct i.e. has yet to be loaded into the warehouse. In the Kalido philosophy at least, data should be put into a staging area prior to loading, and then validated against the business rules of the warehouse. Genuinely bad data should never be entered into the warehouse or it will destroy its integrity and credibility with the business users. If they see a report where the numbers add up wrongly then they will be reluctant to trust the warehouse again. This "incomplete" or "unvalidated" data does have a place in a master data repository though, as improving it to "golden copy" quality is part of the master data management process. Only when master data is validated should it be fed into the warehouse. You raise an important distinction. You should NEVER lose track of valid data that has simply become out of date e.g. when a reorganisation happens you want to expire the old relationship and add a new one. Ideally all relationships should be time-stamped in thsi way, allowing you to recreate past history (this is the way that Kalido does it, for example). There is no reason why this cannot be achieved in custom warehouses also.

What I meant was data that was not yet validated as being correct i.e. has yet to be loaded into the warehouse. In the Kalido philosophy at least, data should be put into a staging area prior to loading, and then validated against the business rules of the warehouse. Genuinely bad data should never be entered into the warehouse or it will destroy its integrity and credibility with the business users. If they see a report where the numbers add up wrongly then they will be reluctant to trust the warehouse again. This “incomplete” or “unvalidated” data does have a place in a master data repository though, as improving it to “golden copy” quality is part of the master data management process. Only when master data is validated should it be fed into the warehouse.

]]> by: Vincent McBurney http://andyonenterprisesoftware.com/2006/07/some-rare-common-sense/#comment-109 Thu, 20 Jul 2006 08:21:17 +0000 http://andyonenterprisesoftware.com/2006/07/some-rare-common-sense/#comment-109 Andy, when you talk about the DW only having validating data, how do you get a complete view of data in the DW if you drop off unclean data? For example what happens when a key item such as a customer is unclean and keeping that out of the DW loses all history, transactions and revenue from that customer. I have been on some DW projects where the data had to go through. Incomplete records had default or dummy values set. Missing referential integrity meant augmenting key tables with dummy records. The users wanted all the measures to be present in the reports and that meant loading unclean data to support those measures. Andy, when you talk about the DW only having validating data, how do you get a complete view of data in the DW if you drop off unclean data? For example what happens when a key item such as a customer is unclean and keeping that out of the DW loses all history, transactions and revenue from that customer.

I have been on some DW projects where the data had to go through. Incomplete records had default or dummy values set. Missing referential integrity meant augmenting key tables with dummy records. The users wanted all the measures to be present in the reports and that meant loading unclean data to support those measures.

]]>