There is a useful overview of the growing appliance market in Computer Business Review:

http://www.cbronline.com/article_cbr.asp?guid=9104551D-56C1-4EE7-BDF9-BD219E8685BF

The appliance market is nothing if not growing, with no fewer than ten appliance vendors now identified by analyst Madan Sheina (who by the way, is one of the smarter analysts out there). Of course apart from Teradata many of these are small or very new. Teradata accounts for about USD 1 billion in total revenue (the accounts will become much clearer once they separate from NCR) though this includes services and support, not just licences. The next largest vendor is Netezza, who does not publish their revenue (though I would estimate over USD 50M). Kognitio used to be around USD 5M in revenue, though they seemed perky when I last spoke to them so may be a little bigger now. DataAllegro will certainly be smaller than Netezza, as will be the other new players. It is too early to say how well HP’s Neoview appliance will do, though clearly HP has plenty of brand and channel clout, especially now that it has acquired Knightsbridge.

Still, so many entrants to a market certainly tell you that plenty of people feel that money can be made. So far Teradata and Netezza have had the field pretty much to themselves, but the entrance of HP and the various newer vendors will create greater competition, which ultimately can only be of benefit to customers.

I spent the last two days at the eWorld Purchasing and Supply conference on London, where I had a couple of speaking slots (you can’t escape your past, and a long time ago I worked in Shell’s technology planning area, which involved software.procurement). I was pleasantly surprised by the scale of the conference. These days most IT conferences are struggling to get decent attendance, as more and more people seek out information on-line. Yet this specialist conference managed to get over 300 attendees on both days, and from the conversations I had these were mostly “real” delegates i.e. people with actual projects and problems to solve, rather than the tyre-kickers who sales people dread and often seem to constitute much of the attendance of some IT conferences.

For another perspective on the conference see the following blog:

http://www.esourcingforum.com/?p=388

Credit where credit is due to the organisers, Revolution Events, who did an excellent job with administration and organisation (only let down a bit by the caterers on day 1 who were of the “oh, we didn’t think this many people would be coming” variety). The exhibits area had decent flow of traffic and the speaking slots stayed well on time, a particular bugbear of mine. Perhaps these more specialist conferences, concentrating on a particular vertical or in this case functional niche, are the way to go.

What we call things is important. The recent rise of data warehouse “appliances”, pioneered by Netezza (and arguably Teradata before that) is an interesting case in point. For years the relational database vendors spent their energy in making sure that transaction systems ran quickly and reliably. Business intelligence applications were not a major focus, and this led to a number of approaches to dealing with very large data warehouse applications. Certain types of index scheme would work very well for read-only BI queries, for example, and Red Brick was an early example of a database optimised as such. Later Teradata did a superb job of carving out a high end niche by using parallel processing hardware and specialist database software to take advantage of this properly. They did such a good job that after a while Teradata almost became synonymous with large data warehouses, of the types typically encountered in retail banks, supermarket chains, telcos etc. Oracle and othe others made some half-hearted attempts to fight back with features like star joins, but by then it was too late: the specialist data warehouse device, in the form of Teradata, had become established. Of course such projects were still large and complex. Most data warehouse project costs are associated with people, not hardware or software, and this does not change whether you are using SQL Server or Teradata as your database.

However, marketing can at times (not often, but sometimes) be a clever and subtle thing. When Netezza brought out essentially a device like Teradata, but quicker and cheaper, the label “appliance” was used, and a very clever one it is. In normal English usage an appliance is something that we just plug in, like a toaster or a coffee maker. Without making any such overt claims, the “appliance” label has a comforting implication that your data warehouse project will have that toaster installation-like quality previously lacking with pesky traditional databases. Given that a DW appliance is just some clever hardware and an optimised database, your project issues are in fact identical to those of any other DW project. Analysis, user requirements, data quality, sourcing, design and reporting all have to be done, although the appliance may certainly be able to handle large volumes of data at a much better price point than a traditional hardware/database combination. Since the hardware and software on a project may typically account for less than 20% of the project costs, this is an undeniably useful thing, but hardly takes us into toaster territory.

Yet the label matters. In a rather breathless blog yesterday:

http://www.itbusinessedge.com/blogs/mia/index.php/2006/09/05/flaming-web-20/

Mike Stevens, who I don’t know personally but appears to have a background in PR rather than hands-on data warehouse project implementation, claims that appliances spell “trouble for traditional data warehouse vendors” since an appliance may cost just USD 150k whereas “conventional solutions cost millions”. He falls into the language trap of the appliance. Your data warehouse still has to to deal with all those people-intensive things (data sourcing reporting, testing) whether you use a conventional SQL database and a regular server, or a specialist DW appliance. The issues are all identical, except with an appliance you have some additional cost since less familiar skills will need to be brought to bear (there are more Oracle skills out there than Netezza ones). The savings on hardware by using an appliance may be very significant and comfortably justified on a large data warehouse, but such a project is not going to cost USD 150k and a quick plug in the wall socket.

If this kind of misconception is so easily repeated by journalists (or at least bloggers) then I wonder how widespread this view is amongst IT managers, and how much this has helped data warehouse “appliances” catch on? Would Netezza have done quite so well if they had been labelled something less reassuring, like a “data warehouse turbo toolkit”? It was said that HP was so bad at marketing it would, if it sold sushi, describe it as “cold dead fish”. The “appliance” vendors shows that smart marketing can still be done within hi-tech.

FAST is the latest enterprise search company to dip its toe in the water of business intelligence, following Autonomy’s recent announcements. In the case of FAST, which is arguably the leader in enterprise search (and must surely be the leading Norwegian software company), they have done so through acquisition. They bought Corporate Radar, a small BI vendor who had some quite clever reporting technology (based around the Microsoft platform) that was quite flexible, and browser-based. On one project that I encountered at Kalido in the US, it was particularly good at building specialist financial reports e.g. gross margin “waterfall” analysis.

What is less clear to me is how “FAST Radar” as it is now known, really integrates with the FAST search engine. Superficially it is appealing to have “search” applied to BI, after all, if Google can scan the whole internet in seconds, why can I never find my monthly sales figures? However the problem in dealing with structured data is the ambiguity of metadata within corporate organisations (”which sales figures do you mean, exactly”), a problem that search technologies, clever though they are, barely scratch the surface of. Putting a very efficient index on a keyword is great for text searching, but it is less obvious to me how useful this would be in resolving ambiguous or inconsistent metadata. Hence I wonder whether the “integration” of these technologies goes much more than skin deep.

If anyone out there has practical experience of a project that uses one of these search engines combined with a BI tool or data warehouse, then please make a comment on this blog, as I am sure that your experiences will be of considerable interest to others.

For some time now I have been impressed with the analysis of Philip Howard of Bloor, and today he came out with another well-written article that is long overdue:

http://www.it-director.com/technology/applications/content.php?cid=9087

As regular readers of this blog know, I have been arguing for some time that the MDM industry has been digging a pit for itself by the way it has ended up with silos of CDI and PIM technology in addition to more generic MDM offerings like Kalido. Phil eloquently points out the shortcomings of such an approach. By implementing solutions only designed for one particular type of master data (say, customer) companies are recreating the silos that they have today, but in a different form. At present the issue is that master data is locked up in mutliple monolithic ERP systems and assorted others, and the point of MDM is to reconcile these multiple master data versions in a master data repository which can keep track of them and manage the process of updating and creating new master data i.e. improving long term master data quality. By setting up a CDI hub, and then a PIM product, and then another repository for the next type of master data in vogue, we will move from having multiple ERP systems holding duplcated and inconsistent master data to multiple, incompatible MDM repositories doing the same. Indeed in some ways the situation will be worse, since the ERP systems are not going away and will still hold (and probably create) their own master data in addition. Master data is not just customer and product: you have to be concerned about assets, people, production facilities, brands etc. In one application at BP Kalido manages 350 separate master data types.

It is key that a master data repository is designed from the ground up to handle multiple datatypes. Buying solutions from a large vendor will not help, as Phil points out in his article: Oracle have multiple solutions, SAP’s MDM product is really an acquired catalogue management product that anyhow does not work properly right now, while with IBM you get the delight of buying several acquired technologies with different approaches and hiring IBM consultants to figure out the mess. My fear is that the menagerie of tools that the industry is delivering will stunt the growth of MDM in general, as customers realise that they have leapt from the frying pan into the fire. Few commentators seem to acknowledge this issue, so well done to Phil for weighing in.

I’d also like to wish all my readers a very happy Christmas! Have a good break.

Market sizing is a slippery thing. Just how big is the MDM market, for example? Well, it all depends on what you include and what you exclude, which is why answers like “it is $x” are not that useful in themselves. Does the figure include software only, or also services like associated consulting? Since for any IT project services can be several times the prices of software, this matters. Moreover, within MDM have they chosen the pure-play MDM vendors only, or thrown in PIM and CDI solution providers? This is the kind of issue that the new press release for Arc Advisory Group’s estimate of MDM market size omits, rendering the quoted figure of USD 680M in 2006 somewhat meaningless on its own. Whatever the real figure is, they reckon it grew 30% in 2006 over 2005 and will essentially double by 2011. Remember that the term MDM barely existed before 2004.

As a point of reference, IDC put the MDM market at USD 5 billion in 2005, expecting it to growing at 14% annually. There is a big difference between USD 5 billion and USD 610 million, which just shows how careful you need to be when taking these analyst figures blindly. The true size will remain a complete mystery until these kinds of press releases spell out what is included and not, and what methodology was used, which at least allows informed debate.

My personal take is that the pure-play MDM software market is actually pretty small. Even throwing in CDI solutions like Siperian and DWL (now part of IBM) as well as pure-play MDM solutions like Kalido, it is hard to get a really big revenue figure if you aggregate the revenues of these vendors e.g. DWL’s revenues were less than USD 20M when bought by IBM. Throw in some sales for Oracle and SAP, but not that much, since early in 2006 Gartner reckoned Oracle had maybe 10 customers only for its CDI solution. SAP similarly has a small number of customers for its troubled MDM offering. Hence it is hard to see where even the Arc figure comes from. IDC are usually pretty thorough when it comes to their numbers, but they must have included a lot of related things to get to their figure. Even chucking in the data quality vendors still won’t get the figure that high, since even the biggest data quality vendors have revenues of about USD 50M.

My perception is that the interest level in MDM is very high but the deployed dollars in software solutions for true MDM (i.e. ignoring data quality and assorted wannabees in this field) is as yet very small, maybe of the order of USD 100M in 2006. Anyone with any insights out there feel free to chip in.

Business intelligence applications typically deal with data that is already stored, often in a depressing number of places i.e. a number greater than 1. Much BI data is of its nature not real-time e.g. looking at monthly averages or trends. However at the other end of the spectrum there are some applications that are truly real-time, and not just in the sense that a marketer puts the term in a brochure.

An interesting start-up in this area is StreamBase, which specialise in genuinely real-time applications, such as trading systems but also inventory monitoring and anti-fraud applications. StreamBase provides StreamSQL, which essentially extends SQL to a real-time environment, a run-time engine as well as a graphical developer environment that allows transformation logic to be written. For example you might have a need to compare the current stock price of an equity to a competitor, and take some action e.g.”buy” if the price hits some threshold related to the competitor. Such applications would occur in memory, but StreamSQL also provides in-memory hash tables and also an embedded database in case you want to persist data (temporarily or permanently, respectively). To continue the trading example, you might want to take an action based on the stock price relative to its average over the last month, for which you would need to store some data temporarily in order to carry out the calculation.

Set up in 2003 by database luminary Mike Stonebraker (who founded Ingres and Illustra) the company has now done two venture rounds, including a series B round led by premier league VC Accel. They have over 60 employees and 50 customers, though this includes pilot customers i.e. not all these are fully paying yet. Public customers include Goldman Sachs and Bridgewater, a leading hedge fund. The company is cagey about revenues but assures me that they are growing.

StreamBase plans to provide in its product roadmap easier integration of real-time and historical data i.e. more StreamSQL enhancements, continued performance enhancements and improved ease of programmability. Two OEM agreements are already in place.

Competition is mostly in-house coding, though there are some vertical point solutions (Progress software in trading) and there is potentially some overlap with EAI tools at some point. For example, StreamBase partner with Tibco Rendezvous but there is an offering of Tibco for event processing that could potentially compete. From a marketing viewpoint the relationship to EAI and middleware tools will need to be carefully stressed as these tools themselves develop. However the company has certainly picked an attractive niche to operate in, and its high quality VC backers and experienced management will make it a credible player.

In a surprise move HP has snapped up Knightsbridge in a move to bolster its technology services business.  Knightsbridge had carved out a strong reputation for handling large data warehouse and BI projects for US corporations, and had grown to over USD 100M in revenue.  It was up with IBM as one of the two leading data warehouse consulting organisations.  This in itself makes it clear why it was attractive to HP, who do not have anything like such a strong reputation in this area.  Knightsbridge was growing strongly in 2006, and the financial terms of the deal are not public, but one would assume HP paid a good price for such a good business.  This will no doubt provide a happy retirement for the Knightsbridge founders, but it is less clear as to how well the Knightsbridge culture, which was quite fiercely vendor-independent, will sit within a behemoth like HP, which has its own technology offerings.  It was revealing that Knightsbridge CEO Rod Walker had dismissed service company acquisitions in an interview just a year ago, and for reasons which sounded pretty sensible.   No doubt this will present an interesting spin challenge for the Knightsbridge PR staff, but perhaps they will have other things on their minds, such as dusting off resumes.

“If the cultures of the two companies are not a near-perfect match, people will leave, and services is a people business.”  I couldn’t have put it better myself Rod.

One thing that continues to surprise me is how little developed the business case for data quality and master data management is.  When I look at data quality vendors speaking at conferences I can sit through whole sessions which do not mention the amount of actual dollars their clients saved by using their technology. In the case of MDM there is some excuse for this, since MDM as a term only recently became mainsteam, and so few vendors have real projects that are in production with clients.  Indeed just 4% of companies have completed an MDM project, according to a recent survey by Ventana (though 37% claim to have initiated a project).  However in the highly related field of data quality there are no such excuses: tools have been around for years, and yet trying to find examples of well justified projects with a hard dollar payback is like pulling teeth.

While data quality has remained something of a backwater (the largest data quality vendor does around USD 50M in revenue) it is surely one of the things that should be relatively easy to produce a cost benefit case for.  After all the tools will enable you to detect the proporton of bad data in a given application or enterprise, and it should not be beyond the wit of man to be able to assign a cost of poor data quality.  Even ignoring tricky things like customer satisfaction, poor data causes very real things: deliveries going to wrong places, misplaced inventory, incorrect payments, problems in manufacturing.  In certain industries it can be worse: drilling an oil well in the wrong place is an expensive affair, for example.  An 2003 AT Kearney study showed that USD 4 was saved for every dollar spent on data cleansing activity. 

By going back and looking at completed projects and carrying out cost/benefit analysis the data quality (and MDM) vendors will be doing themselves a favour, since by quantifying the savings these projects bring they can not only make it easier to justify new projects, but they beging to justify the price of their products: indeed they may be able to gain improved pricing if they can demonstrate that their products bring sufficient value to customers. It is a mystery to me as to why vendors have made such a poor show of doing so.

An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement.