An Accenture study:

http://www.informationweek.com/research/showArticle.jhtml?articleID=196800921

quantifies how much time middle managers in enterprises waste seeking out information, and comes out at two hours a day i.e. a quarter of an average working day. When they find it, half of the information turns out to be of no use. This sounds about right to me, and ilustrates just how far BI really has yet to go in being genuinely useful, and also shows just how bad the true state of information is in large companies.

The issue is not only that technologies are insufficiently intuitive. In my experience there are a number of factors that come into play:

- no culture of sharing information
- inconsistent data definitions
- poor data quality
- inability to locate appropriate data sources
- insufficient understanding of how to use BI tools effectively.

If you set out to produce a useful new report in some area and succeed in doing so, what incentive is there for you to make this easily shared around the company, and to help others find it? In most companies this would be pure altruism, and so people just keep the information on their hard disk, and indeed may gain kudos from the “information is power” syndrome. Overcoming such cultural barriers is hard, and few companies succeed. I should say that Accenture themselves do as good a job as anyone I have seen, where their consultants are actively tasked with documenting project lessons and storing these, with appropriate keywords, in an internal knowledge management system. However I have not seen this in other consultancies to anything like the same extent.

The other problems are all too familiar to people working in BI. Inconsistent data definitions and poor data quality are the heart of what MDM is all about, and we know how immature that is. Yet without fixing this then accurate and easy to obtain information is still elusive. A further problem which some technologies are starting to address is the sheer job of finding an existing report. Ironically there is an excellent chance that if yoiu want some partioular report, then someone else did too and has already built it. The troiuble is that may be in an Excel spreadsheet on a hard drive, or sitting on a shared server but you simply have no easy way of finding that it is there. It is ironic that Google allows us to search the whole internet in moments, yet finding a report within our own company is a much tougher proposition. Enterprise search vendors like Fast and Apptus, as well as Google itself, are beginning to apply smart technology to the problem, but here it is still early days.

Finally, most end users either don’t have access to create a new report easily, or are not trained in making best use of BI tools, or simply don’t have time to learn. This is why Excel is so popular; it is familiar and ubiquitous, and so people would rather get data into Excel and play with it there than learn a new BI tool.

I believe that these are mostly quite intractable problems, only some of which lend themselves to new and better technology. So anyone with a magic bullet e.g. “the answer is SOA” is talking nonsense. It is only by addressing the organisational, cultural and data ownership issues in combination iwth enterprise search and better tool training that a company can improve that two hours a day per person. It will be long, hard slog, and buying the latest trendy tool is not enough, whatever the salesman tells you.

In a surprise move HP has snapped up Knightsbridge in a move to bolster its technology services business.  Knightsbridge had carved out a strong reputation for handling large data warehouse and BI projects for US corporations, and had grown to over USD 100M in revenue.  It was up with IBM as one of the two leading data warehouse consulting organisations.  This in itself makes it clear why it was attractive to HP, who do not have anything like such a strong reputation in this area.  Knightsbridge was growing strongly in 2006, and the financial terms of the deal are not public, but one would assume HP paid a good price for such a good business.  This will no doubt provide a happy retirement for the Knightsbridge founders, but it is less clear as to how well the Knightsbridge culture, which was quite fiercely vendor-independent, will sit within a behemoth like HP, which has its own technology offerings.  It was revealing that Knightsbridge CEO Rod Walker had dismissed service company acquisitions in an interview just a year ago, and for reasons which sounded pretty sensible.   No doubt this will present an interesting spin challenge for the Knightsbridge PR staff, but perhaps they will have other things on their minds, such as dusting off resumes.

“If the cultures of the two companies are not a near-perfect match, people will leave, and services is a people business.”  I couldn’t have put it better myself Rod.

As regular readers of this blog are aware, I believe that one of the potential trends in business intelligence will be towards “on demand” offerings.  This model avoids the hassle of installation at your own site (with all the complexities of combinations of operating system, database and middleware software versions that implies).  Since it is usually is accompanied by a rental pricing model, this makes it easier to try out from the customer’s viewpoint, whilst from the vendor’s viewpoint this often avoids triggering the dreaded procurement review that bogs down so many sales cycles.  As well as new vendors entering the space, Business Objects has just made a tentative step in this direction by acquiring NSite, a small vendor essentiially providing complementary offerings to the salesforce.com offering. 

I  think that this purchase is less for the application that Nsite has than for the experience that it has in offering software as a service i.e. Business Objects is mostly buying a team of software engineers with relevant experience.  This is probably a good idea, since for all its sales and marketing clout, R&D has been a consistent weak spot for Business Objects over the years.  Judging from the press release it looks like Steve Lucas is behind the acquisition; Steve is a smart guy and understands the need for Business Objects to improve its offerings.

Informatica continues to broaden its offerings, in this case by acquisition.  Itemfield is a company selling software aimed at translating unstructured (or indeed structured) data into XML.  Its technical strength was smart algorithms that could deduce structure from samples of emails, Word documents etc.  It is fully built around SOA principles, allowing it to be embedded into other offerings fairly easily. 

Founded in 2000, the company had 50 or so employees, half of which were in Israel, where the R&D was based.  The company had achieved penetration into some good accounts including GE and American Airlines.  Its revenues are not public, but I believe that they were hoping for USD 10M sales in 2007.  Interestingly, their typical deal size of about USD 225k was higher than that of Informatica itself.  Informatica, which already had a partnership with Itemfield, has paid a quite full price for the company, USD 55M of cash meaning a price/sales ratio of 5.5 on projected future 2007 earnings.  A healthy result for the founders, and further demonstration that companies will pay a premium price for sufficiently differentiated technology.

I have wondered for some time why business intelligence has been so slow to come up with software as a service solutions.  Celequest has done so, and this week sees the launch of another, called LucidEra.  This company aims to offer a ciomplete BI suite including ETL, data quality, database schema, OLAP server and reporting.  Given that enterprises are prepared to trust their customer data to third parties e.g. salesforce.com, there is no reason I can see why they would not do the same with business intelligence. 

The advantages of a service offering is seem to me twofold,  First is the easier and more reliable deployment.  Many problems in software stem from environmental incompatibilities e.g. some weird combination of releases of Oracle and Tomcat and something else that cause obscure bugs which the vendor could never have tested for, and which are hard to reproduce.  This problem goes away with hosted solutions, where the web browser is just about the only software the client can screw around with.  Secondly, though this is a commercial rather than technical issue, the leasing that software as a service typically uses means an easier point of entry.  One mid-ranking customer can sign off on a few months of leasing in a way that they could not for a multi-hundred thousand dollar software purchase, which would end up in steering committees and a formal procurement process.  

Salesforce has shown what can be done with this approach if well executed.  It will be interesting to track the progress of LucidEra, Celequest and others that emerge into this space.

An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement. 

In IDC’s latest annual report, it gives the size of the “data analysis” market, which includes data warehousing, business intelligence and generally anything analytic, as being worth a chunky USD 16.5 billion in software (systems integration related to this would be greater than this), up 11% from last year.  They also reckon that this market will grow at a healthy clip of 10% a year for the next five years, based on the fact that “analysis” is now one of the top two spending items for IT executives.

Recently I pointed out that the specialist players in what is more commonly called the business intelligence market grew revenues at 23% in calendar 2005 over 2004, though the IDC figures include the BI offerings of the industry giants Oracle, SAP, IBM and Microsoft, as well as a broad set of other companies and categories e.g. data mining offerings. 

A large and growing market not only causes the behemoths to want to gobble up smaller players with good technology, as Oracle recently did with Sunopsis, but in principle should interest venture capital firms to back innovative start-ups in the area.  However VCs seem too starry-eyed at the moment over social networking web sites to want to return to anything as tedious as enterprise software, with all its long sales cycles, costly software development and grumpy and conservative enterprise buyers.  Still, fashions change, and who in 2002 (when venture firms turned firmly against the internet in the wake of the crash) would have been betting that a web site company set up in 2005 for consumers, with no obvious mechanism for making money, would be snapped up just over a year later for USD 1.65 billion?  Perhaps it is time to get ahead of the curve and look ahead to when enterprise software will be fashionable again.  Any VCs feeling brave?

The Software 500 is a very useful annual listing from Software Magazine.  It is the one place where you can find a ranking in revenue terms of software companies, many of which are privately held and are can therefore be awkward to find out information about.  The list does have its flaws: it looks back at 2005 calendar revenues, and so is somewhat out of date.  It also bizarrely includes companies that are clearly not software companies but systems integrators, such as Accenture and Logica.  However, it is what it is, and is reportedly used by some CIOs as a “checklist” when evaluating software companies, so it is somewhere that even shy privately held companies want to be listed. 

In Business Intelligence we have the following picture: 

From a macro perspective, average growth of the BI vendors was a healthy 23% in 2005 over 2004.  Companies that did particularly well were Global Software Inc with 77% growth, Kalido with 68% growth and Insight Software with 42% growth.  Even the giants did well in terms of growth, though some of this was from acquisitions as well as organic growth.  This is a useful piece of data when looking at the generally quoted industry growth rates, which are based on analyst estimates of spend.  Here you have the actual revenue growth of all the significant companies that operate in the space, which may be a more real measure. This would suggest that the business intelllgence space had a more healthy year than was generally credited by industry analysts.

After a poor Q2 Microstrategy followed Business Objects in achieving a good Q3 set of results.  The critical measure of software revenue was up 8% this quarter though 1% down year over year, while overall revenue at just under USD 78M was up 18% year over year.  The company still has USD 52 million in cash, and its operations generated free cash flow of about USD 18M. 

Investors have responded well, as can be seen from the share price chart below.

This is further evidence of a quite good market sentiment particularly in the US.  However it must be remembered that software sales are actually down year over year, and the company has added 30% headcount since a year ago.  For a software company, you really want to see increases in software licence revenue, not just services. This seems to be increasingly a struggle for BI vendors, and not just Microstrategy.

Business Objects posted a strong quarter, rebounding from a weak Q2.  The key metric of software license revenue of USD 132M was up 10%, and 9% year over year.  There were nine deals over USD 1 million.  Operating margin is a healthy 17%.  Overall revenue was USD 311 million. 

About the only party pooper to be found in the figures is if you dig in to see that the licence revenue growth is mainly in enterprise performance management and enterprise information management, which are heavily influenced by acquisitions. The core business intelligence licences (the bulk of the business) actually shrank by 1% year over year. 

The Americas was the star area, up 27%, with Europe up 9% (just 3% if you strip out currency effects) and Asia Pacific 19%.  Someone I spoke to at Business Objects in the US this week said that the mood on the ground was very positive, with lots of hiring going on.

Business Objects has also been known for its strong sales and marketing, and this engine seems to be purring along well at preent.