I must have nodded off in the armchair after a glass or two of seasonal cheer when I was confronted by an apparition - a ghost of data warehouses past. He was not a pretty sight - dating back to the mid 1990s and with a grumpy attitude. In those days all you had was a database and a compiler, and a customer who wanted to somehow bring together data from multiple, incompatible systems. There were no ETL tools, no data warehouse design books and little in the way of viewing the data once you had managed to wrestle it into the new data warehouse, just reporting tools that just about saved you coding SQL but confronted users with the names of the tables and columns, usually restricted to eight characters. Those were the days, a salutary reminder of how primitive things were.

Following this was a ghost of data warehouses present. This was a much chirpier looking fellow, who could gain access to data from even devious and recalcitrant source systems via ETL tools like Ascential, and had at least some idea how to design the warehouse. These designs had suitably festive names: “snowflake schema”, “star schema”. If we had a reindeer schema then it would have completed the festive scene. This wraith had a sackful of reporting tools to access the data, count it, mine it and graph it. Yet not all was well - the pale figure seemed troubled. Every time the source systems changed, the warehouse design was impacted, and teams of DBA elves had to scurry around building tables, sometimes even dropping them. Moreover the pretty reporting tools were completely unable to deal with history: when reports were needed from past seasons there was no recollection of the master data at the time, and it did not reflect the current structures. The only ones really happy here were the DBA elves, who busied themselves constructing ever more tables and indices - they love activity, and they were content.

Last came a ghost of data warehouse future. In this idyllic world, changes in the source systems have no effect on the warehouse schema at all. The warehouse just absorbs the change like a sponge, and can produce reports to reflect any view of structure, present, past or even future. There are less DBA elves around, but they have discovered new things to do with their free time - with SAP up to 32,000 tables these days, there is no unemployment in DBA elf land. Best of all the customers are happy, as they can finally make sense of the data as quickly as their business changes. New master data can be propagated throughout their enterprise like so much fairy dust. I have seen the future of data warehousing, and it is adaptive.

Well, enough of that. Time to talk turkey, or at least eat it. While doing so, it may be relaxing to look back at some of the turkeys that our industry has managed to produce over the years. Human nature being what it is, there is more than one web site devoted to the follies of the technology industry, and even to the worst software bugs. Have fun reading these. I recall being told by a venture capitalist that the dot-com boom of the late 1990s proved that “in a strong enough wind, even turkeys can fly”. These are some that did not.

I would like to wish readers of this blog and very happy Christmas.

In an article today Peggy Bocks of EDS asks whether MDM is all hype. I think it is fascinating how some terms in technology catch on, while others wither away. In 2002 MDM was essentially an unknown term. I recall discussing with analysts calling our new Kalido product offering “Kalido MDM” and being greeted with polite derision, since this was “not a market”. Customers seemed to recognize the problem though, and I discovered that we were not alone when SAP launched their own SAP MDM offering soon afterwards. Though that product is now retired, when a vendor the size of SAP anoints a term then you can be sure that the industry will take it more seriously than when a start-up uses it. Three years on and there is a much higher level of noise around MDM, with around 60 vendors now claiming to have some sort of MDM offering, at least in Powerpoint form. Moreover further validation has come from Oracle (with its Data Hub), Hyperion (who bought Razza) and IBM, who have bought several technologies related to MDM. IDC reckon the market for MDM will be worth USD 9.7 billion within fove years.

However, I do have some sympathy with Ms Bock’s point - a lot of MDM at present is froth and discussion rather than concrete projects. Certainly Kalido has some very real MDM deployments, Razza has some, Oracle presumably does, and SAP managed about 20 deployments before giving up and starting again, buying A2i and retiring its existing offering. Still, outside the tighter niches of product information management and customer data integration (arguably a subset of the broad MDM market) this hardly constitutes a landslide of software deployments.

A skeptic would argue that the industry has got these things wrong before, getting all excited over technology that fizzles out. Remember how “e-procurement” was going to take over the world? Ask the shareholders of defunct Commerce One about that trend. I recall IDC projecting some vast market for object databases in a report they published in 1992, and at the time I wrote a paper at Shell arguing that the ODBMS market would likely never get even close to a billion dollars, and indeed it never did. However object databases always struck me as a solution in search of a problem, whereas the issues around managing master data are very real, and very expensive, for large companies, and they are not well addressed today. There are major costs associated with inconsistent master data e.g. deliveries being delivered wrongly, duplicate stock being held etc. Shell Lubricants though they had 20,000 unique pack-product combinations when in fact they had around 5,000, meaning major savings to be had through eliminated duplication in marketing, packaging and manufacturing, for example.

Because it addresses a real business problem, with the potential for significant hard business savings, I believe that the MDM market will in fact catch light and grow, but there will inevitably be a confusing period while analysts get their heads around the new market and start to segment it, and customers begin to understand the various stages they need to go through in order to run an effective master data project.