There is an interesting web forum which seeks to bring an open source approach to the world of data management. Of interest are topics involving the creation of open source de-duplication, profiling, matching and cleansing tools (hat tip to CW for pointing this out).

No doubt the tools here are at an early stage and won’t directly compare in broad functionality with a major data quality vendor. However, for many people with less sophisticated requirements that may not matter. The rise of products like MySQL has shown how influential an open source product can become given the right circumstances.

I would be very interested as to whether any readers of the blog have any experience with the tools here, or any views on the merits or otherwise of an open approach to data quality and data integration.

I have recently noticed a pronounced division in the mentality of software vendors to disclosing information. On the credit side of the column are vendors like Kognitio, whose CEO happily discussed the company strategy, their revenues, profitability and customer deployments. At the other end of the spectrum was a data quality vendor who would not even tell me how many employees they had (actually, at the far, far end of the spectrum is Ab Initio, who won’t demo their software to a customer without a non-disclosure agreement). What I am curious about is what these so-shy vendors think they are achieving by hiding information. To either prospects or analysts, if a vendor looks shiftily to one side and says “ah, as a policy we don’t disclose xxx” (substitute: employees, revenues, customers, profitability, whether they have a working product, etc) then do they think the prospect is going to be (a) reassured (b) more nervous than before?

If a vendor is a small start-up then we all know that it is likely to be loss-making, have just a few customers so far and be fairly small. That is what software start-ups are. The reason we are talking to them is that (hopefully) have something interesting to offer that the big brand vendors do not. It is OK if there are only a handful of customers if the product is fairly new, and it is OK to not be profitable if the company strategy is rapid growth. As a customer, it is often much easier to deal with smaller vendors who actually care about you, rather than some vast marketing machine where raising a software bug is as useful as dropping a message in a bottle in the ocean.

However anyone contemplating purchasing software is at some point going to want to get a sense of whether they are customer number #32, or customer #2, and how this element of risk stacks up against what the company has to offer. Incidentally, as I have written previously, it is far from clear that it is always safe to buy from large vendors. I have personally been stung a couple of times as an enterprise buyer when giant vendors decided that their product was not doing well enough, so just dropped it. If small company has just the one product you can be pretty sure they will care about it. Presumably vendors who are coy are thinking “let’s not scare them off now, we’ll demo the software, get it installed and then they will be so impressed that they won’t notice we are small/unprofitable/early stage/etc”. Well, I have news for you, at some point they will care, and if this moment comes after you have invested a lot of time with them then it is a lot more painful than if this came out right at the start, when at least you can redirect your attentions elsewhere.

So in my view, openness is the best policy. If a customer is terrified about the idea of dealing with an early stage company, better to find out at he beginning of the conversation than after you have invested scarce pre-sales time in doing demos and perhaps even a software trial. IBM or Oracle can afford a few wasted sales calls, but for a small start-up every conversation and pre-sales demo is precious. If a company is serious about a purchase and they have concerns about the vendor, they will find about your dark secret during their due diligence (when they will run one of those pesky Dun & Bradstreet reports), and they will not thank you if you have glossed over an important issue due to your “policy” of not talking openly about your company.

A Computerworld article highlights the risks that enterprise buyers run in an age of vendor consolidation. In this case the article talks about Peoplesoft and Oracle, but the point is a general one. Just how anxious should software buyers be about their vendor being acquired?

I would argue that the vendor risk issue is frequently overplayed. You may “never get fired by buying IBM” but I recall when IBM dropped its “strategic” 4GL ADF for CSP in the late 1980s, leaving plenty of seriously large customers in the lurch (I worked for Exxon at the time, which had standardised on ADF). There is a risk in any software purchase, not only about whether the vendor will go bust at some point, but as to whether the vendor will continue to maintain and enhance the particular product you are buying. People often agonise about buying software from small vendors, but in the case of a company with one product in their portfolio, you can at least be sure that they will care a lot about that product. An industry giant may have ultra-solid finances, but can decide to drop a product line if it does not do well commercially, or for other internal reasons, as in the IBM example I mentioned. There are numerous other cases e.g. SAP MDM was dumped in favour of a new product based on acquired technology from A2i just a couple of years ago, while Oracle has plenty of “prior” in abandoning acquired product lines that did not meet its view of the world.

I believe that buyers should look at a few things in terms of risk. Look beyond the finances of the vendor to the installed base of the particular product they are buying. A product with hundreds or thousands of enterprise customers is likely to live a lot longer than one with a few. Moreover what is the growth trajectory of the customer base? A fast growing customer base will very likely receive continued investment, either internally in the case of an industry behemoth, or externally from venture capital firms in the case of smaller companies. The situation to be wary of, whatever the vendor size, is where there is a small customer base that is not growing. This situation should send warning bells off, whatever the vendor size. Of course vendors may be very coy about revealing figures, but you can for example try and talk to the chairman of a product user group to get a sense of how well the customer base is growing; a user group with shrinking numbers of attendees would be a worrying sign.

Above all, customers need to ensure that their investment has a clear and rapid payback. If you spend a million dollars in licences, with 20% annual support and 4 million in services putting it in, you should be able to stack up on the other side of the balance sheet the benefits that you are expecting to see. If the benefit case has a payback period of (say) a year, then it is less of an issue to worry about the vendor will be around in ten years time. If you have a choice between a mediocre product from a “safe” vendor and a much more productive product from a smaller riskier, vendor, then you should be able to quantify what the difference in productivity is worth to you. If the better, riskier, technology saves you millions of dollars a year and pays back in eight months v the alternative, then what sense does it make to accept an inferior technology that will actually cost you many millions in poor productivity, however “safe” it may be.

As discussed earlier, very few product lines are completely safe anyway, given the tendency of vendors to cull non-performing product lines and encourage “migration” to newer (read “profitable”) newer products. If you have a fast enough payback then you can be philosophical about a migration a few years down the road. It all comes down to rigorous cost benefit analysis of the software life-cycle, sadly something all too few customers pay proper attention to.

Happy New Year. There was a useful post in SearchStorage.com regarding 2007 technology IPOs. Some of these are outside the scope of this column, but what I found interesting was that it seems that enterprise software companies were able to tap the capital markets at levels of revenue and profitability unseen over the last few years. A couple of years back the message from investment bankers was that you needed quarterly revenues of not less than USD20 million (and preferably more) and several quarters of profitability before even considering an IPO. Yet Netezza’s IPO got away OK despite a lack of profitability (though strong growth), while Sourcefire had quarterly revenue of under USD 15M and was still not profitable, yet also managed an IPO. It looks as if the markets have taken a slightly harder view since then judging by the early performance of these shares, but these IPOs would simply not have happened in 2004 or 2005.

What is less clear is whether 2008 will show the same softening of view, or whether the financial debt crisis afflicting banks will have collateral damage in the IPO market. Software companies and their backers will be hoping for a continued thaw rather than a return to the wintry outlook the capital markets have seen in the past few years.

Back in the late 1990s companies filed for IPOs even though they had modest revenues and were losing money. Due to the tulip mentality of the time investors suspended disbelief and bought in anyway, giving way to the crash of 2001. A couple of years after that bankers were telling me that in order to have an IPO you would need “at least a couple of years of solid trading profits”, quarterly revenues of at least $25 million and preferably more, as well as strong growth. Those heady days of the late 1990s were a freak occurrence, like the South Sea Bubble. Certainly technology IPOs dried up almost entirely.

With the recent gloom on Wall Street I was therefore surprised to see Initiate Systems filing for an IPO. They are growing quite rapidly but not only have never made a cent of profit, but their losses appear to be, if anything, widening slightly at about a third of their revenues. Throw in an admitted financial misstatement and does this start to feel to you like the late 1990s again? No doubt Initiate is expertly and expensively advised, but this will certainly be one to watch, as if the IPO goes ahead and well then it will change perceptions of exit strategies for high tech companies.

Purisma has been acquired by Dun & Bradstreet, the business information company that provide, amongst other things, assessment of credit risk of companies and company statistics. On the face of it this is a somewhat peculiar acquisition, since D&B is not a pure provider of enterprise software solutions in the way that, say Oracle, is. However D&B did have its own data quality offering (clearly data quality is a big issue for an information supplier) and Pursima’s customer hub technology is certainly complementary to this data quality offering. It seems possible that D&B has bought Purisma primarily for its own internal purposes, and at this point it is unclear whether Purisma will even continue to be sold as a product in its current form. Rather ironically, Purisma had a product offering allowing integration of D&B into its CDI application. I guess that will come in handy now.

Purisma does not publish public financial data, so it is tricky to tell whether how good or bad the price paid of USD 48 million for the company was. I believe that Purisma had less than 50 employees and I would speculate that its revenues were in the USD 15-20M range. In general it is known that stand-alone CDI and PIM players have been struggling somewhat in the market. This is part due to a gradual dawning on customers that master data management is a broader topic than just “customer” or “product”, a long term theme of this blog. When customers ask “ah, but what about other kinds of master data” (asset, location, employee etc) then specialist CDI and PIM vendors do not have good answers, however good their offerings in their particular domains are. Even IBM has done an about turn on this topic recently, laying out a roadmap for a single MDM Server that will eventually bring together its menagerie of acquired technologies into a platform that will handle multiple master data domains consistently. For this reason I suspect that D&B did not pay over the odds for Purisma.

D&B has had phases in the past of buying software companies, and then moving away from this business e.g. those with long memories will recall the 4GL Nomad, which it sold off after some years. The press release that is tucked away on the Purisma web site today is not giving anything away. If press releases played poker, this one would be a tough player. Purisma customers need to seek guidance from D&B about its future intentions, and consider their alternatives.

Business Objects announced a witches brew tie-up with Big Blue, bundling DB2 and its data warehouse capabilities with Business Objects together as a business intelligence “offering”. Given that Business Objects already works happily enough with DB2, it is rather unclear as to whether this is a any more a ghostly smoke and mirrors marketing tie-up rather than anything deeper, but it certainly makes some sense for both companies. It does, however, hint at Business Objects moving away from pure database platform independence, which takes on a new significance given the takeover (sorry: “merger” - much background cackling) of Business Objects by SAP. Is this really a subtle move to try and irritate Oracle, the other main DBMS vendor, give the highly competitive situation between SAP and Oracle, who are locked in a nightmare struggle for world domination? In this case, is SAP just manipulating Business Objects like a possessed puppet, pulling the strings behind the scenes, or was this just a hangover from the pre-takeover days, with the Business Objects marketing machine rolling on like a zombie that stumbles on yet does not realise it already has no independent life, clinging to some deep-held memory of those days in its old life. SAP has a more tense relationship with IBM itself these days. IBM sells cauldrons of consulting work around SAP implementations, but found a knife in its back when SAP started promoting the Netweaver middleware in direct competition with IBM’s Web Sphere.

Announcements from Business Objects from now on all need to be looked at through the distorting mirror of the relationship with its new parent, as there may be meaning lurking that would not have existed a month ago. Everything needs to be parsed for implications about the titanic Oracle v SAP struggle, as Business Objects should strive as far as possible to appear utterly neutral to applications and databases in order to not spook its customers. Arch rivals Cognos, Microstrategy and SAS will take advantage of any hint that the giant behind Business Objects is just pulling its strings.

Happy halloween everyone!

Defining a software segment’s market size is a tricky thing, partly because is all about what you include and what you exclude. Take MDM as an example. A much quoted IDC figure reckoned the MDM market would be USD 10 billion in 2009, implying a USD 5 billion market size in 2005 given compound growth of 14%. Such figures are regularly bandied about by the computer press, but mean little unless you qualify such statements by explaining what is included or excluded. For example this figure includes an estimate for services business associated with MDM. This is itself hard to pin down, but in my experience an MDM project where the software costs X will spend about 3X on services to implement it. Hence that USD 5 billion market size actually only has about USD 1.6 billion of software sales. Then MDM itself is a broad church, including CDI and PIM as well as a generalist MDM solutions such as those from Orchestra Networks and Kalido. I was still puzzled as to why even this USD 1.6 billion figure number was so large, but by deduction I think that the IDC figure was including data quality within the picture also. Fair enough, but it needs to be explicitly stated to make sense of the market, and as we will see still does not explain the gap.

Let’s come at this another way. A Gartner figure just released reckoned the CDI market was worth USD 310 million in 2006. This appears to be an estimate for software rather than services. Getting a figure for the product information management market is murkier, but I believe it will be broadly at a similar level. The generalist MDM vendors are these days mostly from smaller companies (products like Razza having been swallowed and digested by Oracle, and Stratature by Microsoft for example) and I doubt would add USD 100 million in software sales to this picture. Hence, adding PIM + CDI + specialist MDM (but excluding data quality) you get a software market of maybe USD 700M (probably a bit less), which is a far cry from the apparent IDC figure of USD 5 billion, or even the likely USD 1.6 billion of software revenues only. I still struggle to bridge the gap here, as the data quality market is not that large. Again you have to be careful about what is in and what is out, but other than leader Trillium data quality vendors are mostly very small (e.g. Exeros, Datanomic, etc) or are now buried within larger companies through acquisition (e.g. Informatica, Business Objects). However though I have seen estimates like USD 500M for the data quality market, again I wonder how much of this is services; personally I am unconvinced that the software sales of the data quality market would be much beyond USD 100M or so (companies like FirstLogic were not that large prior to their acquisition). So if we take the USD 700M figure and throw in USD 150M for data quality software sales (let’s be generous) this is still a far cry from the USD 1.6 billion estimate we arrived at earlier. Of all the analyst firms I respect the market size figures from IDC best, as they do actually check with the vendors what their revenues really are (they used to do this every year when I was running Kalido) but as you can see their MDM market size figure is still a mystery to me. If someone from IDC is reading this and can shed some light on it I would be interested to hear from them.

MDM is certainly growing quickly: each analyst firm agrees on this, and is clear enough from the number of companies entering the market or (more commonly) re-labelling existing products as MDM. However it can be seen that you can take a figure like the IDC 5 billion number, and also produce a valid market estimate of under USD 850 million, just based on what you include or exclude, for seemingly the same market. Quite a range. I guess it is hoping too much to expect the IT press to actually mention pesky caveats like what a number includes, since it is more headline inducing to say “MDM market worth $5 billion”, but if you are to actually use these figures to help with a decision then you would be well advised to dig deeper, below the headline numbers.

I enjoyed an article by Madan Sheina on the failure of BI projects. 87% of BI projects fail to meet expectations, according to a survey by the UK National Computing Centre. I wish I could say this was a surprise, but it is not. Any IT project involves people and processes as well as technology, yet many project focus almost entirely on the technology: tool choices, database performance etc. Yet in practice the issues which confound a BI project are rarely the technology itself. Projects fail to address actual customer needs, and frequently don’t acknowledge that there are several user constituencies out there with quite different requirements. Frequently a new technology is stuffed down the customer’s throat, and projects often neglect data quality to their peril.

From my experience, here are a few things that cause projects to go wrong.

1. Not addressing the true customer need. How much time does the project team spend with the people who are actually going to use the results of the BI project? Usually there are a subset of users who want flexible analytical tools, and others who just need a basic set of numbers once a month. A failure to realise this can alienate both main classes of user. Taking an iterative project to project development rather than a waterfall appraoch is vital to a BI project.

2. Data is only useful if it is trusted, making data quality a key issue. Most data is in a shocking state in large companies, and the problems often come to light only when data is brought together and summarised. The BI project cannot just gloss over this, as the customers will quickly avoiding using the new shiny system if they find they cannot trust the data within it. For this reason the project teams needs to encourage the setting up of data governance processes to ensure that data quality improves Such initiatives are often outside the project scope, are unfunded and require business buy-in, which is hard. The business people themselves often regard poor data quality as an IT problem when in fact it is an ownership and business process problem.

3. “Just one more new user interface” is not what the customer wants to hear. “Most are familiar with Excel and are not willing to change their business experience” was one quote from a customer in the article. Spot on! Why should a customer whose main job is, after all, not IT but something in the business, have to learn a different tool just to get access to data that he or she needs? Some tool vendors have done a good job of integrating with Excel, and yet are often in denial about this since they view their proprietary interface as a key competitive weapon against other vendors. Customers don’t care about this; they just want to get at the data they need to do their job on an easy and timely way. Hence a BI project should, if at all possible, look at ways of allowing users to getting data into their familiar Excel rather than foisting new interfaces on them. A few analyst types will be prepared to learn a new tool, but this is only a small subset of the audience for a BI project, likely 10% or less.

4. Data being out of date, and the underlying warehouse being unable to respond to business change, is a regular problem. Traditional data warehouse designs are poor at dealing with change caused by reorganisations, acquisitions etc, and delays in responding to business change cause user distrust. Unchecked, this causes users to hire a contractor to get some “quick and dirty” answers into a spreadsheet and bypass the new system, causing the proliferation of data sources to continue. Using packaged data warehouse that are good at dealing with business change is a good way of minimising this issue, yet even today most data warehouse are hand-built.

5. Training on a new application is frequently neglected in IT projects. Spend the time to sit down with busy users and explain to them how they are to access the data in the new system, and make sure that they fully understood how to use the system. It is worth going to some trouble to sit down with users and train them one to one if you have to, since it only takes a few grumbling voices to sow the seeds of discontent about a new system. Training the end users is never seen as a priority for a project budget, yet this can make the world of difference to the likelihood of a project succeeding.

6. Running smaller projects sounds crass but can really help. Project management theory shows that the size of a project is the single biggest predictor of success: basically, if projects fail, small ones do better, and yet you still see USD 100 million “big bang” BI projects. Split the thing into phases, roll out by department and country, do just about anything to bring the project down to a manageable size. If your BI project has 50 people or more on it then you are already in trouble.

7. Developing a proper business case for a project and then going back later and doing a post implementation review happens surprisingly rarely, yet can help shield the project from political ill winds.

You will notice that not one of the above issues involves a choice of technology, technical performance or the mention of the word “appliance”. Yes, it is certainly important to pick the right tool for the job, to choose a sufficiently powerful database and server and to ensure adequate systems performance (which these days appliances can help with in the case of very large data volumes). The problem is that BI projects tend to gloss over the “soft” issues above and concentrate on the “hard” technical issues that we people who work in IT feel comfortable with. Unfortunately there is no point in having a shiny new server and data warehouse if no one is using it.

The annual IDC business intelligence report shows a reassuring 11% rise in overall BI tool revenues in 2006. Regular readers of my blog know that a couple of my long-term viewpoints are that: (a) Microsoft is the vendor with the long-term best position due to its ownership of Excel, which is still the BI front-end that end-users actually want and (b) that specialist BI tools will never, contrary to many BI vendor projections, have a place on every desktop, due to the rather dull reason that most people do not need one.

Is there any evidence for these hypotheses? Well, Microsoft’s BI revenues grew 28%, while the major specialist BI vendors grew by 7% (Business Objects) and 9.8% (Cognos). As IDC analyst Dan Vesset notes, “IDC does not yet see a substantial impact on the market from the strategy and marketing messages of most BI vendors seeking to reach a broader use base”. Nor will it ever, in my opinion.

Also of note is the formidable performance of Qliktech, which grew by a little matter of 97% to USD 43.6 million revenue. Its offering to the mid-market offering based on in-memory search technology continues to get considerable customer traction. I have a soft spot for this company, having been asked to look at it for a venture capital firm as an investment when it was still a tiny company; I am very relieved that I recommended that they invest - otherwise I imagine that they would be hunting me down like a dog right now.