An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement. 

An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement.