As usual, Philip Howard has some thoughtful comments on the subject of enterprise data warehousing. The recent plethora of data warehouse appliances, pioneered by Netezza but now popping up from companies ranging from start-ups to HP, certainly has the potential to change the data warehouse landscape. However as Philip points out, it is less clear that data warehouse appliances need be connected with” ubiquitous BI”. I have written previously st some length on my view that there is really no obvious reason for the “democratisation of data” i.e. with anyone in the company having unfettered access to corporate data using whizzy reporting tools. Quite apart from whether the tools are really cuddly enough (doubtful) the question rarely asked is why would this vision be necessary or even appropriate? There are certainly people in a company whose job it is to analyse data: they would be, er, analysts. Everyone else pretty much needs a limited set of data to get on with their jobs, and certainly I would be nervous if every factory worker and truck driver in a company decided to spend an hour or two a day investigating corporate data warehouses. A salesman needs a limited of set of numbers in a year: “here is your quota” while I struggle to see why people outside finance or marketing (and only a subset of those) really need to be spending their time wrestling with data at all.

To be sure one class of people benefits from a “BI tool on every desktop”: vendors, both BI vendors and those selling associated databases and hardware. I have yet to read any articles in Harvard Business Review from CEOs complaining that their profits would be higher if only every employee in the company had a BI tool. BI ubiquity seems to me a solution in search of a problem.

The CDI/MDM Institute conference has just wrapped up in Kensington, and I felt it was very successful. Organisers IRM ran an efficient conference, and attendance on day 2 held up pretty well. There was a fine keynote speech given by a certain dashing ex-Kalido founder, and the rest of the day was in a series of tracks with customer case studies and some vendor and consultant presentations. The second day felt a bit light on customer case studies compared to day one, but overall this conference did well at getting a reasonable number of real customers talking about projects, rather than just being a string of subtle or not-so-subtle sales pitches from vendors.

Aaron Zornes gave a reasonably balanced overview of the main thirteen MDM vendors (D&B, Dataflux, Data Foundations, Hyperion, I2, IBM, Initiate, Kalido, Oracle Purisma, SAP, Siperian, Teradata, Visionware) with a recurring theme that few have really got to grips with providing support for the data governance/collaboration element of MDM. Certainly the workflow around how master data is a key part to most MDM projects, but it looks as if most of the projects out there at present today are doing this via home-grown efforts, e.g. one UK company I spoke to had done this, using Biztalk as a framework. As more and more MDM projects go beyond pilots into production then the aspect of maintenance of master data will become more pressing, so I would expect to see customers increasingly demanding this. Indeed they should really be insisting that workflow support for MDM processes are one of the key evaluation criteria for software selection.

Overall, it seems that MDM is moving into the mainstream in the UK.

In the first of a series of articles on MDM, Richard Skriletz of RCG starts by trying to define master data. RCG has an excellent reputation as a consulting company, but after saying some sensible things the article seems to me to get tangled up. To me master data is anything that is shared between multiple systems. This is captured well in the Wikipedia definition which Richard references, before immediately going on to try and “improve” on this very clear definition. He wants to distinguish between master data and “reference data”, in his case by splitting the world into physical entities like product and abstract entities like organisation. I have written about this before. Not only is there no need for this distinction, it can be misleading, and cause one to start treating data in different ways when there is no need to. I’m not sure whether IT people just love to classify things (all that being brought up on “int”, “char”, “varchar” etc) but this desire to split master data up is just confusing. Data that is shared between systems needs to be managed, and it needs to be managed whether it is physical or abstract. By imposing artificial distinctions between types of master data we introduce complication where none is necessary. William of Ockham had the right idea on this, and he did so in the 14th Century.

Market sizing is a slippery thing. Just how big is the MDM market, for example? Well, it all depends on what you include and what you exclude, which is why answers like “it is $x” are not that useful in themselves. Does the figure include software only, or also services like associated consulting? Since for any IT project services can be several times the prices of software, this matters. Moreover, within MDM have they chosen the pure-play MDM vendors only, or thrown in PIM and CDI solution providers? This is the kind of issue that the new press release for Arc Advisory Group’s estimate of MDM market size omits, rendering the quoted figure of USD 680M in 2006 somewhat meaningless on its own. Whatever the real figure is, they reckon it grew 30% in 2006 over 2005 and will essentially double by 2011. Remember that the term MDM barely existed before 2004.

As a point of reference, IDC put the MDM market at USD 5 billion in 2005, expecting it to growing at 14% annually. There is a big difference between USD 5 billion and USD 610 million, which just shows how careful you need to be when taking these analyst figures blindly. The true size will remain a complete mystery until these kinds of press releases spell out what is included and not, and what methodology was used, which at least allows informed debate.

My personal take is that the pure-play MDM software market is actually pretty small. Even throwing in CDI solutions like Siperian and DWL (now part of IBM) as well as pure-play MDM solutions like Kalido, it is hard to get a really big revenue figure if you aggregate the revenues of these vendors e.g. DWL’s revenues were less than USD 20M when bought by IBM. Throw in some sales for Oracle and SAP, but not that much, since early in 2006 Gartner reckoned Oracle had maybe 10 customers only for its CDI solution. SAP similarly has a small number of customers for its troubled MDM offering. Hence it is hard to see where even the Arc figure comes from. IDC are usually pretty thorough when it comes to their numbers, but they must have included a lot of related things to get to their figure. Even chucking in the data quality vendors still won’t get the figure that high, since even the biggest data quality vendors have revenues of about USD 50M.

My perception is that the interest level in MDM is very high but the deployed dollars in software solutions for true MDM (i.e. ignoring data quality and assorted wannabees in this field) is as yet very small, maybe of the order of USD 100M in 2006. Anyone with any insights out there feel free to chip in.

I have wondered for some time why business intelligence has been so slow to come up with software as a service solutions.  Celequest has done so, and this week sees the launch of another, called LucidEra.  This company aims to offer a ciomplete BI suite including ETL, data quality, database schema, OLAP server and reporting.  Given that enterprises are prepared to trust their customer data to third parties e.g. salesforce.com, there is no reason I can see why they would not do the same with business intelligence. 

The advantages of a service offering is seem to me twofold,  First is the easier and more reliable deployment.  Many problems in software stem from environmental incompatibilities e.g. some weird combination of releases of Oracle and Tomcat and something else that cause obscure bugs which the vendor could never have tested for, and which are hard to reproduce.  This problem goes away with hosted solutions, where the web browser is just about the only software the client can screw around with.  Secondly, though this is a commercial rather than technical issue, the leasing that software as a service typically uses means an easier point of entry.  One mid-ranking customer can sign off on a few months of leasing in a way that they could not for a multi-hundred thousand dollar software purchase, which would end up in steering committees and a formal procurement process.  

Salesforce has shown what can be done with this approach if well executed.  It will be interesting to track the progress of LucidEra, Celequest and others that emerge into this space.

In IDC’s latest annual report, it gives the size of the “data analysis” market, which includes data warehousing, business intelligence and generally anything analytic, as being worth a chunky USD 16.5 billion in software (systems integration related to this would be greater than this), up 11% from last year.  They also reckon that this market will grow at a healthy clip of 10% a year for the next five years, based on the fact that “analysis” is now one of the top two spending items for IT executives.

Recently I pointed out that the specialist players in what is more commonly called the business intelligence market grew revenues at 23% in calendar 2005 over 2004, though the IDC figures include the BI offerings of the industry giants Oracle, SAP, IBM and Microsoft, as well as a broad set of other companies and categories e.g. data mining offerings. 

A large and growing market not only causes the behemoths to want to gobble up smaller players with good technology, as Oracle recently did with Sunopsis, but in principle should interest venture capital firms to back innovative start-ups in the area.  However VCs seem too starry-eyed at the moment over social networking web sites to want to return to anything as tedious as enterprise software, with all its long sales cycles, costly software development and grumpy and conservative enterprise buyers.  Still, fashions change, and who in 2002 (when venture firms turned firmly against the internet in the wake of the crash) would have been betting that a web site company set up in 2005 for consumers, with no obvious mechanism for making money, would be snapped up just over a year later for USD 1.65 billion?  Perhaps it is time to get ahead of the curve and look ahead to when enterprise software will be fashionable again.  Any VCs feeling brave?

On the 13th and 14th of September there was a business intelligence forum in London run by a fairly new organisation called “Obis Omni” (no, I don’t know what it means either). I was a speaker at the event, which was quite well attended given that it is a rather new conference.  The customer attendees seemed to be what one was billed i.e. people really involved in BI projects (some events seem to struggle to keep focus), and the conference seemed generally very successful based on the conversations that I had with assorted attendees.  

What was rather endearing and a little scary was the sheer efficiency of the conference administration. The event ran very tightly to time, and there seemed to be armies of helpers to guide you around, all rather disturbingly fitted out with communication devices in their ears just like those of Agent Smith in the Matrix. Indeed the only criticism would be that they were a little over-enthusiastic at times.  After my talk I was speaking to a delegate in the corridor, when one of the Agent Smith types came up, interrupted oiur conversation and said “you are due to attend session X now Ms Jones, please come along”.  This wasn’t the speaker who was late you understand, just a delegate.  God forbid that an unauthorised corridor conversation should take place during session time.  The delegate looked as stunned as I was and was led meekly away to her session without putting up a fight. 

I can never see this kind of thing catching on in Italy or Spain.  At the ETRE conference the tragically mistitled “organisers” struggle to keep sessions within half a day of schedule, and generally mooch around in a resigned if amiable state of chaos.  Here an eerie calm was the order of the day (come to think of it, I never did see that delegate again…)

The pre-conference administration and exhibit set up was as spookily efficient as everything else, with briefings just after dawn for exhibtors and, it has to be said, nicely set out booths with careful traffic flow.  I even had a new experience of having my slides lightly censored.  My crime was using a (fully credited) Bloor slide to show an overview of the market, and a chart which listed several relational databases in one of the bullets.  Apparently this violated rule 438 subparagraph (c) in the conference rulebook about vendor promotion.  Well, I have nothing to do with Oracle, IBM etc, and just put them in a list as examples, so I am a little hazy as to how exactly this was ”promotion” (DB2 is a database, shock horror), but the offending slides were duly excised from the presentation, and probably ceremonially burnt as well.  I actually think it is quite admirable that they would go through the slides and try to ensure there were no blatant vendor sales pitches, but this did seem just a tad over-zealous.

Anyway, enough teasing.  I would much rather that an event ran with military efficiency than collapsed in a shambolic heap, so congratulations to the organisers for arranging such a slick conference.  If only they could be persuaded to take over the British railway system….

A new report from Butler Group finds that “less than 8% of the IT budget is actually spent on initiatives that bring value to the enterprise”. This is not an encouraging number for those who believe that CIOs have their finger on the pulse of the business, but merely confirms other studies e.g. one from A.T. Kearney, which found that business executives perceive their IT departments to be out of touch with the business. Yet the same study found that “70% of respondents see IT innovation as important or critical to their company’s success”. I’m afraid that my own experience would side with the cynics. There are certainly some talented and hard-working people in corporate IT departments, but in general the management of IT is woefully out of synch with the needs and desires of the operational business executives. CIOs continually act as gatekeepers rather than enablers, often seeing their role as one of cost-cutting and standard-bearers in the fight to reduce the number of vendors, preferably to a set that can be represented on a single Powerpoint slide.

Unfortunately this is generally not what business executives need. Since exciting new business applications are unlikely to come from the big vendors that wine and dine the CIOs, the business will look to IT to bring them new ideas that technology can enable geneuine value, and help them sift out the genuinely interesting from the slideware and the charlatans. I had a conversation with a business executive in a large company this weekend, who explained that his IT department had recently made a software product from a small vendor “non strategic” in favor of a less functional one from SAP, despite some objections from him. A year later this same IT team has had to admit that his new global retail management information system cannot be delivered by the SAP technology. Just what sort of business value is this IT department delivering to its customers? People working in IT at that company can hardly be surprised when one-third of its IT jobs were recently shifted to India. If IT departments do not bring perceived value, then they are just a commodity cost as far as executives are concerned, in which case they can expect to be treated as such.

The management of IT badly needs to speak the language of business and to become more aligned to the priorities of its customers, rather than its own agendas and technology religious wars, most of which leave business people cold and are perceived as irrelevant. The status quo will see a lot more IT jobs headed towards Bangalore, and not just the low-level programming ones.