A worthy competition was recently held by the Business Intelligence Network to pick the best data visualisation solution to a series of problems presented. The first article in a series of five to discuss this (by Stephen Few) nicely illustrates how to meaningfully display data in an example, and perhaps even more usefully some instructive examples of how not to.  It is perhaps appropriate that the winning solution was from Spotfire, a company that has carved out a strong niche in visualisation, initially in the pharmaceutical industry and recently on a broader basis.

I have written previously that visualisation software is not a mass market, but it can have a major impact in areas where smart visual displays add value e.g. in trading, life sciences or oil exploration.  Stephen’s article also shows up something very illuminating: having a flashy tool is less improtant than thinking carefully about design.  When Powerpoint came out presentations everywhere started to sprout absurd clipart and pschedelic colours, but eventually it was realised that there was still a place for graphic designers.  Similarly in visual analysis it is rarely the case that a more sophisticated tool is needed - even Excel has quite decent charting abilities.  However application designers put too little thought into thoughtful design that is intuitive to use. Before rushing off to buy the latest tool, try reading the following book: “The Visual Display of Quantitative Information” by Edward Tufte, which elegantly illustrates best practice in making sense of complicated data sets in ways that are meaningful and easy to understand.  It should be mandatory reading for anyone designing a system involving the visual display of information.

Next time you see an incomprehensible or misleading graph presented by your IT department, smile sagely and buy them a copy of this book.

Teradata has taken on I2’s MDM solution as its own, moving forward from a previous partner relationship to one where they licence the code.  Indeed Teradata has hired I2 staff responsible for the I2 MDM product, so this is more a technology acquisition than a regular partnership. For i2 this will effectively remove it from the MDM space, but in actual fact Teradata is a more logical home for this technology.  I2 has struggled recently, with its share price at $14.30 today barely more than half that last September.  Since I2 is very much a supply chain vendor, the general purpose MDM technology it has was a somewhat ungainly fit. For them this move is about concentrating on their core supply chain competence. 

Teradata’s strong data warehouse offering has a much more natural fit to MDM, as indeed does any data warehouse vendor.  It makes sense for a data warehouse to sit alongside a separate master data repository, as I have written about previously.  Since I2’s MDM technology had a decent reputation (though rather limited customer deployments) Teradata’s strong channel and more natural technology fit should gain better traction for it, as well as plugging a gap in Teradata’s offerings.

If you go into Google you can find most things remarkably quickly amongst the vastness of the internet, so why can’t you find your sales data inside a large company?  This perfectly reasonable question has prompted some BI vendors to team up with Google in order to put a Google search front end onto enterprise data.  Sound too good to be true?  Sadly I fear that it is.  The search capabilities of Google are superb at searching for keywords on websites, and enable you to quickly zero in on what you are looking for provided you can make your search keywords precise enough.  Unfortunately the same technique does not translate well to the semantic nuances of enterprise data, where finding a database with “price” data in it unfortunately does not give you sufficient context (which for which product, under which commission scheme, on what date, within which sales area, etc?).  Moreover a search engine does not yet generate the SQL to get the wretched data out of the corporate databases where the answers lie.  Hence to put a Google front end on to a BI tool you are probably going to have to run a bunch of reports, give them some tags and publish them as web pages - Google will certainly be able to deal with that, but then is this so much better than just picking the report you want out from a list anyway?

Andrew Binstock writes a useful article about this in Infoworld but perhaps glosses over the magnitude of the problem in terms of finding answers to data on an ad hoc basis.  Indeed early implementations essentially throw the problem back to a BI tool, which generates results in a form that a search front end like Google can use.  Usually this is not the biggest problem anyway, as it easy enough to put menus together with the top 20 or whatever regular reports for users to choose from.  I can see a real use for this when the sheer number of canned reports gets out of hand though.  If you have thousands of reports to trawl through then having a search front end could be genuinely useful.  But the lack of semantic understanding needed of enterprise definitions will make it just as hard for a search tool to make any sense of a mass of numbers as a BI tool, which relies on either some from of front end semantic layer (as Business Objects uses) or assumes the existence of a data warehouse where the semantic complexity has been pre-resolved into a single consistent form.  As the article correctly points out, the only way to fix this is through better metadata.  Indeed, greatly improved master data definitions could find a further use as tags to help search engine front-ends make more sense of large numbers of pre-built corporate reports. Unfortunately the nirvana of ad hoc access to corporate data via an intuitive search front-end seems to me no closer than before. 

What is certainly true is that the BI vendors can use these Google front-ends to make pretty demos to try and sell more software.  However they do run a hidden danger in doing so.  Given that at present the Bi vendors compete partly through the ease of use of their graphical interfaces, by handing over the user interface to Google they may be in danger of commoditising part of their competitive advantage.  If you have a simple search front-end, who knows whether the report originally came from Business Objects, Cognos, or Information Builders?  I wonder whether the BI vendors have really thought through the danger to their own businesses that this seemingly innocent search front end could become.  By jumping on the Google bandwagon they could be unleashing something that removes their direct contact from the end user, a key element in differentiation.

William McKnight writes an interesting article in DM Review this month looking at business intelligence trends in retail.  As he says, retailers ideally need to track promotions and customer buying trends, yet are often poorly served by management information systems.  One reason for this is the sheer volume of data: if you consider every item on a till receipt as a transaction, then each store will be putting through many thousands of transactions per day. A convenience store might have 20,000 stock keeping units (SKUs) but a department store might have over 200,000.  In addition a retailer will want to keep track of customers who have loyalty cards, will be concerned about space optimisation and stock control.  The more astute retailers vary the stock on their shelves in accordance with buying patterns they have observed: clearly the customer profile in mid morning is different to just after schools finish, for example.  One Japanese chain reckons to change the stock profile on its shelves seven times per day.

To get this kind of insight you need a high quality, robust data warehouse that is able to handle large data volumes and keep up with rapid change. However one thing that I learnt when working on a Shell retail project a few years ago was that you can make life easier for yourself by quickly archiving the transaction detail.  A category manager may want to do basket analysis for a few days, but beyond that is interested in trends, which can be satisfied by aggregate information.  This allows you to rapidly archive the high volume transaction data and so keep the data in the warehouse to manageable levels. 

Good BI on retail can make a major effect on profits, as I have written about before.  The dawn of RFID, though still in its infancy, will further extend the possibilities for more elaborate analysis, though my suspicion is that most retailers are barely scratching the surface of what can be done today using the current technology.

The early success of Netezza has not only prompted other “me too” database appliance players like DataAllegro but also now an open source variant.  Greenplum has combined its open source database with Sun hardware to come up with an appliance of its own.  This is an interesting move that makes sense for Greenplum, since Sun obviously has a far larger sales channel than itself and so is a potentially powerful partner.  The article in Information Week is rather too breathless in its description of a DBMS + some hardware as an “instant data warehouse” however.  This is nonsense, and is no more so than a copy of SQL Server and a PC is “an instant data warehouse” (and at least SQL Server has some functionality directly useful to data warehousing in it). If all you had to do was supply a DMBS to have a data warehouse then there would be a lot of unemployed systems integrators and consultants.

However, leaving aside the inability of the journalist to see beyond the words of the company press release, this is an astute move by Greenplum, which can use Sun’s credibility to make it reassure nervous enterprise customers who may otherwise be twitchy about entrusting their data warehouse data to an open source platform.  However any prospect should be careful to check the performance characteristics of their application to see how well the Bizgres DBMS stacks up, since just throwing hardware at a problem can be an expensive thing to do.  But from a customer perspective it is good to have another choice to compare with Teradata and Netezza if you have an ultra-high transaction volume problem.