One thing that continues to surprise me is how little developed the business case for data quality and master data management is.  When I look at data quality vendors speaking at conferences I can sit through whole sessions which do not mention the amount of actual dollars their clients saved by using their technology. In the case of MDM there is some excuse for this, since MDM as a term only recently became mainsteam, and so few vendors have real projects that are in production with clients.  Indeed just 4% of companies have completed an MDM project, according to a recent survey by Ventana (though 37% claim to have initiated a project).  However in the highly related field of data quality there are no such excuses: tools have been around for years, and yet trying to find examples of well justified projects with a hard dollar payback is like pulling teeth.

While data quality has remained something of a backwater (the largest data quality vendor does around USD 50M in revenue) it is surely one of the things that should be relatively easy to produce a cost benefit case for.  After all the tools will enable you to detect the proporton of bad data in a given application or enterprise, and it should not be beyond the wit of man to be able to assign a cost of poor data quality.  Even ignoring tricky things like customer satisfaction, poor data causes very real things: deliveries going to wrong places, misplaced inventory, incorrect payments, problems in manufacturing.  In certain industries it can be worse: drilling an oil well in the wrong place is an expensive affair, for example.  An 2003 AT Kearney study showed that USD 4 was saved for every dollar spent on data cleansing activity. 

By going back and looking at completed projects and carrying out cost/benefit analysis the data quality (and MDM) vendors will be doing themselves a favour, since by quantifying the savings these projects bring they can not only make it easier to justify new projects, but they beging to justify the price of their products: indeed they may be able to gain improved pricing if they can demonstrate that their products bring sufficient value to customers. It is a mystery to me as to why vendors have made such a poor show of doing so.

An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement. 

In IDC’s latest annual report, it gives the size of the “data analysis” market, which includes data warehousing, business intelligence and generally anything analytic, as being worth a chunky USD 16.5 billion in software (systems integration related to this would be greater than this), up 11% from last year.  They also reckon that this market will grow at a healthy clip of 10% a year for the next five years, based on the fact that “analysis” is now one of the top two spending items for IT executives.

Recently I pointed out that the specialist players in what is more commonly called the business intelligence market grew revenues at 23% in calendar 2005 over 2004, though the IDC figures include the BI offerings of the industry giants Oracle, SAP, IBM and Microsoft, as well as a broad set of other companies and categories e.g. data mining offerings. 

A large and growing market not only causes the behemoths to want to gobble up smaller players with good technology, as Oracle recently did with Sunopsis, but in principle should interest venture capital firms to back innovative start-ups in the area.  However VCs seem too starry-eyed at the moment over social networking web sites to want to return to anything as tedious as enterprise software, with all its long sales cycles, costly software development and grumpy and conservative enterprise buyers.  Still, fashions change, and who in 2002 (when venture firms turned firmly against the internet in the wake of the crash) would have been betting that a web site company set up in 2005 for consumers, with no obvious mechanism for making money, would be snapped up just over a year later for USD 1.65 billion?  Perhaps it is time to get ahead of the curve and look ahead to when enterprise software will be fashionable again.  Any VCs feeling brave?

The Software 500 is a very useful annual listing from Software Magazine.  It is the one place where you can find a ranking in revenue terms of software companies, many of which are privately held and are can therefore be awkward to find out information about.  The list does have its flaws: it looks back at 2005 calendar revenues, and so is somewhat out of date.  It also bizarrely includes companies that are clearly not software companies but systems integrators, such as Accenture and Logica.  However, it is what it is, and is reportedly used by some CIOs as a “checklist” when evaluating software companies, so it is somewhere that even shy privately held companies want to be listed. 

In Business Intelligence we have the following picture: 

From a macro perspective, average growth of the BI vendors was a healthy 23% in 2005 over 2004.  Companies that did particularly well were Global Software Inc with 77% growth, Kalido with 68% growth and Insight Software with 42% growth.  Even the giants did well in terms of growth, though some of this was from acquisitions as well as organic growth.  This is a useful piece of data when looking at the generally quoted industry growth rates, which are based on analyst estimates of spend.  Here you have the actual revenue growth of all the significant companies that operate in the space, which may be a more real measure. This would suggest that the business intelllgence space had a more healthy year than was generally credited by industry analysts.

After a poor Q2 Microstrategy followed Business Objects in achieving a good Q3 set of results.  The critical measure of software revenue was up 8% this quarter though 1% down year over year, while overall revenue at just under USD 78M was up 18% year over year.  The company still has USD 52 million in cash, and its operations generated free cash flow of about USD 18M. 

Investors have responded well, as can be seen from the share price chart below.

This is further evidence of a quite good market sentiment particularly in the US.  However it must be remembered that software sales are actually down year over year, and the company has added 30% headcount since a year ago.  For a software company, you really want to see increases in software licence revenue, not just services. This seems to be increasingly a struggle for BI vendors, and not just Microstrategy.

I spent the early part of this week at ETRE, an unusual IT conference that has been running since 1990.  Organised by ex-journalist Alex Vieux, the conference is for technology CEOs and founders, general partners at VCs and the usual hangers on, rather than for customers.  It moves around to a different European city each year, and this year attracted about 500 people.  The conference is notable for two consistent things: the very high quality of people attending (Bill Gates used to be a regular) and the utter inability of the “organisers” to keep to schedule. This year, it has to be said, the overruns were of more manageable proportions than usual, and indeed the opening session only started 14 minutes late.  John Thompson of Symantec and Niklaas Zenmstrom of Skype were the star names this year.  Skype now constitutes 7% of all long distance calls, which does rather make one wonder at what point the phone companies who generously provide the infrastructure will send the boys round to collect some money from eBay.

The “future of enterprise software” session was definitely the odd one out, since the future was clearly all about social networks, at least in the eyes of investors.  They have a sort of Dragon’s Den session called ”meet the money” where early stage companies pitch to a panel of VCs, and this year the company funkysexycool.com (I couldn’t make that up), a sort of myspace wannabee for mobile phones whose business cards feature a voluptuous fantasy woman, had the VCs lining up to throw money at it.  By contrast a shipping ecommerce company, who had been around a few years, had several million in revenues and was profitable, could not have caught a cold, never mind funding.  Perhaps a social networking site for melancholy enterprise software executives?  No takers?  Oh well.

What interest there was around enterprise software was confined to “software as a service” companies.  Rightnow has now reached USD 100 million in revenue, joining salesforce.com in that rarefied air, and certainly there seem to be a few other early stage companies branching out into software as a service for things like HR and ERP.  Given that as much as 80% of technical problems with software are to do with the client environment (often some odd combination of software versions that the vendor had not, or could not, test) then the model certainly makes a lot of sense.  The drawback is that the rental model that usually goes with this means relatively slow growth, though the recurring revenue generated certainly means less sleepless nights near the end of a quarter for software executives.  

One of the few enterprise areas prospering was security, where there was a general consensus that the hackers and spammers were comfortably winning the war.  I was impressed with a company called BitDefender, who are a Rumanian security software firm that has grown since launch in 2001to USD 60 million in revenues.  This has all been done without a dollar of venture capital (there are not too many VC conferences in Bucharest).

The lack of organisational skills of the conference remain legendary.  They denied all knowledge of my booking until I produced the bank transfer details, though they at least seemed embarrassed when I pointed out that they had done exactly the same thing last year.  The conference check-in was a procession of people with lost reservations and people who had booked airport transfers that never arrived.  To be fair, a very helpful gentleman called Farley Duvall did a bang-up job of sorting out my misbehaving video presentation, and the Red Herring people always seem to cope with problems with willingness and good humour. Perhaps they just need some German organisers. 

With its baffling inability to stick to a schedule, ETRE remains something of an enigma.  Attendees wonder aloud whether it is worth the high cost, and yet each year they come back as there is nowhere else quite like this for networking.  If your company is not there, what does this say about you?  With other conferences very much in decline (even Esther Dyson’s US conference just bit the dust) you certainly have to give a lot of credit to Alex Vieux and his team for managing to attract a healthy turnout of people back every year.  Not many tech conferences can claim a 17 year unbroken heritage.

The open source movement is creeping into BI in various guises, as pointed out by Jacqueline Ernigh.  However, while Linux is undoubtedly taking a lot of market share from proprietary UNIX variants, it is less clear of progress higher up the stack. The article mentions a number of organisation that provide some form of open source reporting tools e.g. Pentaho, Greenplum and Jaspersoft, and indeed there are others still. However it is by no means clear what penetration these are really getting.  It was noticeable that one of the two customer examples reported merely had a database running on Linux, but had yet to deploy open source reporting tools. 

The article unfortunately loses credibility when it cites an example of the savings to be made: ”At pricing of $1,000 per user seat, for example, a company with 16,000 employees would need to pay $160,000 for a full-fledged BI deployment, Everett hypothesized.”  Hmm.  It is some time since I did my mathematics degree but I am thinking that 16,000 * 1,000 = 16,000,000 i.e. 16 million dollars, not $160,000.  Even if you are kind and assume that a major discount could be obtained for such a large deployment, even an unlikely 90% discount to list would still get you USD 1.6 million.  I doubt that Dan Everett at the entirely respectable research company Ventana would really have made such a dramatic numerical blunder, so perhaps it was a journalistic error.  Such carelessness does make one wonder about the accuracy of rest of the article, which is a pity since it is discussing an interesting trend.  

I still have yet to really come across significant deployments of open source reporting tools in production applications, but presumably they will catch on to a certain extend, just as MySQL is making steady inroads into the database market.  Perhaps the most significant point at this stage is not made by the article though.  The very existence of open source reporting tools puts pricing pressure on the established BI vendors.  Procurement people doing large deals with BI vendors will treat the open source BI movement as manna from heaven, since they have a stick to beat down the price of reporting tools from the major vendors.  Anyone about to engage in a major BI deployment or negotiation would be well advised to look carefully at these tools, if only as weapons in the armoury against pushy software salesmen.  This is further bad news for the BI vendors, who have enough to worry about with the push of Microsoft into their space and the general saturation of the market. In this case even a handful of customer deployments will suffice to send a shiver down the spine of the major vendors.

My prize for the most creative marketing jargon of the week goes to IBM, who announced that they now consider their offerings to be a “third generation” of business intelligence.  Come again?  In this view of the world, first generation BI was mainframe batch reporting, while the second generation was data warehousing and associated BI tools like Cognos, Business Objects etc.  So, as you wait with bated breath for the other shoe to drop, what is the “new generation”?  Well, it would seem that this should include three things:

(a) pre-packaged applications

(b) focus on the access and delivery of business information to end users, and support both information providers and information consumers

(c) support access to all sorts of information, not just that in a data warehouse.

Well (a) this is certainly a handy definition, since IBM just happens to provide a series of pre-built data models (e.g. their banking data model) and so (surprise) would satisfy the first of these criteria.  It is in fact by no means clear how useful such packages are outside of a few specific sectors that lend themselves to standardisation.  Once you take a pre-existing data model and modify it even a little (as you will need to) then you immediately create a major issue for how you support the next vendor upgrade.  This indeed is a major challenge that customers of the IBM banking model face.  Nothing in this paper talks about any new way of delivering these models e.g. any new semantic integration and versioning capability.

Criteria (b) is essentially meaningless since any self respecting BI tool could reasonably claim to focus on information consumers.  After all, the “universe” of Business Objects was a great example of putting user-defined terminology in front of the customer rather than just presenting tables and columns.  Almost any existing data warehouse with a decent reporting tool could claim to satisfy this criteria.

On (c) there is perhaps a kernel of relevance here, since there is no denying that some information needs are not always kept in a typical data warehouse e.g. unstructured data.  Yet IBM itself does not appear to have any new technology here, but merely is claiming that DB2 Data Joiner allows links to non-DB2 sources. All well and good, but this is not new. They haven’t even done something like OEM an unstructured query product like Autonomy, which would make sense.

Indeed all that this “3rd generation” appears to be is a flashy marketing label for IBM’s catalog of existing BI-related products.  They have Visual Warehouse, which is a glorified data dictionary (now rather oddly split into two separate physical stores) and scheduling tool, just as they always have.  They talk about ETI Extract as an ETL tool partner, which is rather odd given their acquisition of Ascential, which was after all one of the two pre-eminent ETL tools, and given ETI’s near-disappearance in the market over recent years.  They have DB2, which is a good database with support for datatypes other than numbers (just like other databases).  They also have some other assorted tools like Vality for data quality.

All well and good, but this is no more and no less than they had before. Moreover it could well be argued that this list of tools actually misses several important points that could be regarded as important from a “next generation” data warehouse architecture.  The paper is oddly silent on the connection between this and master data management, which is peculiar given IBM’s buying spree in this area and its direct relevance to data warehousing and data quality.  There is nothing about time-variance capabilities and versioning, which are increasingly important.  What about the ability to handle a federation of data warehouses and synchronise these?  What about truly business model-based data warehouse generation and maintenance?  How about the ability to be embedded into transactional systems via SOA?  What about “self discovery” data quality capabilities, which are starting to appear in some start ups.

Indeed IBM’s marketing group would do well to examine Bill Inmon’s DW 2.0 material, which while not perfect at least has a decent go at setting out some of the capabilities which one might expect from a next generation business intelligence system.

There is no denying that IBM has a lot of technology related to business intelligence and data warehousing (indeed, its buying spree has meant that it has a very broad range indeed).  Yet there is not a single thing in this whitepaper that constitutes a true step forward in technology or design.  It is simply a self-serving definition of a “3rd generation” that has nothing to do with limitations in current technology or new features that might actually be useful.  Instead it just sets out a definition which conveniently fits the menagerie of tools that IBM has developed and acquired in this area. To put together a whitepaper that articulates how a series of acquired technologies fits together is valid, and in truth this is what this paper is.  To claim that it represents some sort of generational breakthrough in an industry is just hubris, and destroys credibility in the eyes of any objective observer.  This is by no means unique in the software industry, but is precisely why software marketing has a bad name amongst customers, who are constantly promised the moon but delivered something a lot more down to earth.

I suppose when presented with the choice of developing new capabilities and product features that people might find useful, or just relabelling what you have lying around already as “next generation”, the latter is a great deal easier.  It is not, however, of any use to anyone outside a software sales and marketing team.

For quite some time now I have been banging on (bloggin on?) about the gradual but steady progress than Microsoft has been making with its business intelligence offerings, and how these pose a long term threat to the pure-play BI vendors that seems to me underestimated by most.  Of course opinions are one thing but hard data is another, so it is nice to be able to put some numbers behind this thesis (”smug mode”, as they would say on “Red Dwarf”).  In the last year Microsoft’s BI offerings grew in revenue by a very healthy 36%.  Compare and contrast this progress with that of the pure play vendors.  Cognos just announced its quarterly results, which showed license revenue 1% down on a year on year quarterly basis, and down 7% overall (modest growth overall came from its services business). Cognos is still generating healthy profits at 12.8% operating margin, which although down from 15.5% last year is certainly very respectable. But the underlying difficulty in selling more licenses in what is supposed to be a fast growing BI market shows that something is not right.  Business Objects has managed slightly better sales growth but at the cost of halving its profitability over the last year, while Hyperion has managed 8.9% growth in revenue with just a small slip in profitability.  Of these, Hyperion with its financial consolidation speciality is the least threatened by the fairly low-end Microsoft BI tools.

Microsoft may not be particularly agile, but it is chipping away steadily at the business of the pure-play BI players, and I can only see this trend continuing.                                                                     

The data quality market has been a strange beast.  The problem is huge, with just about every company in the world having issues regarding data quality.  Duplicate customer information is the area that is most familiar, but the problem can be a lot more serious than a few bits of duplicated junk mail.  More than a decade ago a Shell exploration drill neatly drilled into part of an existing drilling operation because the co-ordinates of the existing pipe were incorrect in a database.  Fortunately there was no oil flowing through the pipe at the time or a major spillage would have occurred, but even so the rig was off-line for some time, at a cost of about half a million dollars a day even then. I can certainly testify that a large chunk of every data warehouse project is spent dealing with data quality issues, even when the data comes from supposedly authoritative sources like ERP systems. 

Yet despite a real problem and real dollars attached to this problem, the data quality software market is something of a minnow.  Forrester estimates it at USD 1 billion, and that includes consulting. The software part of the market is maybe USD 200 million.  Over recent years pure play vendors have been bought up and incorporated into broader offerings e.g. Vality by Ascential (now IBM), FirstLogic by Business Objects, Similarity Systems by Informatica, Trillium by Harte Hanks.  The price of FirstLogic, USD 69 million for a business with USD 50 million in revenue, was hardly something to make investors salivate (to be fair, Similarity’s price tag of USD 49 million was at a much better multiple, and there were some peculiarities around Firstlogic).  Why should this be?  Part of the problem is that data quality issues remain a resolutely human problem, typically meaning that when validation rules are defined in a data quality tool, only part of the problem surfaces.  Business processes around who actually owns the data and who can fix problems are as big a problem as actually finding errors in the data itself.  It is encouraging to see a couple of start-ups take a new approach to this old chestnut.  Though differing somewhat in the technologies used, both take the approach of trying to discover relationships between existing data by analysing existing datasets rather than relying on building up a top-down rules profile. Exeros just secured a USD 12 million series B funding round, and has high quality venture backing from Globespan Capital Partners and Bay Partners. A rival company with a similar approach is Zoomix.  Based on technology developed in the Israeli military, Zoomix uses data mining techniques to seek out the relationships amongst existing data, presenting its first attempt to a business analyst and then learning from the responses so as to improve the algorithms in future iterations. They also have an interesting new product which can apply these rules in real-time to an existing transaction system, called up as an SOA service.  This effectively acts like a firewall, but for data quality. Zoomix has customers in Israel and Europe, and has set up a European HQ in London.  These newcomers present fresh competition to the largest vendor (Trillium) as well as to other start-ups such as Datanomic (based in Cambridge - the original one rather than the Massachusets one) and more specialist quality tools such as Silver Creek, who have taken a tightly targeted approach, in their case dealing with complex product data.

Investors in these companies have clearly seen that, while data quality may be more of a component in a broader solution (BI, MDM, whatever) there is enough in the way of a real problem here to allow room for innovative approaches.  The exits for these companies may well be acquisition by larger companies, just as with Similarity Systems, but it is good to see a fresh approach being taken to this very real, if not exactly sexy, business problem.