Supposedly “timing is everything”, yet analysis across time is a surprisingly neglected topic in many data warehouse implementations. If you are a marketer, it is clear that time is a critical issue: you want to be able to compare seasonal sales patterns, for example. A retailer may even be interested in the pattern of buying at different times of the day, and change stock layout in response to this. Yet in many data warehouse designs, time is an afterthought. For example in SAP BW you can only analyze a field for date/time reporting if you specify this up-front at the time of implementation, and this carries a performance penalty. Even this is an improvement on many custom-build warehouses, where data is not routinely date-stamped and so even basic reporting using time is impractical.

Advanced data warehouse technology should enable you to not only do simple time-based analysis like “last summer’s sales v this summer’s sales” but also be able to keep track of past business hierarchies. For example you may want to see the sales profitability before and after a reorganization, and so want to look at a whole year’s data as if the reorg never happened, or as if it had always happened. One major UK retailer has a whole team of staff who take historic data and manually edit a copy of that data in order to be able to make such like-for-like comparisons, and yet this type of analysis should be something that their data warehouse can automatically provide. An example of doing it right is Labatt where the marketing team now had access to a full-range of time-based analysis, enabling to take more data-driven decisions.

Another sophisticated user of time-based analysis is Intelsat, who used sophisticated time-based analysis to improve their understanding of future satellite capacity. Satellite time is sold in blocks, usually in recurring contracts to news agencies such as CNN or the BBC e.g. “two hours every Friday at 16:00 GMT”. Each of these contracts has a probability of being renewed, and of course there are also prospective contracts that salesmen are trying to land but may or may not be inked. Hence working out the amount of satellite inventory actually available next Tuesday is a non-trivial task, involving analysis that was previously so awkward that it was only done on occasion. After implemented a data warehouse that inherently understands time-variance, Intelsat were able to identify no less than USD 150 million of additional capacity, and immediately sell USD 3 million of this, a handsome return on investment on a project that was live in just three months and cost less in total than even the immediate savings.

If your data warehouse can’t automatically give you sophisticated time-based analysis then you should look at best-practice cases like this. Make time to do it.

How should customers go about “due diligence” prior to buying software? Certainly enterprise software is a major, multi-year commitment, and the overall costs of it will be many times the actual purchase price, so it is worth looking carefully before you leap. However many companies seem to look in the wrong place.

Firstly there is an assumption that if you buy software from an industry behemoth, then this is “safe”, whereas buying from a smaller vendor is inherently more dangerous. This is not necessarily the case. While the actual finances of an industry giant are rarely in doubt, the question to ask is not the the size of the balance sheet, but how committed are they to this particular product? When I was working at Exxon in the 1980s we discovered that the “strategic” 4GL called ADF that IBM sold was to be dropped in favor of another tool they had built called CSP. The fact that we were a big oil company and they were ultra-safe IBM did not help us one bit. Migration? We could hire their consultants to help us rewrite all the applicatons: thanks a lot. Or consider all the technologies that Oracle has acquired over the years and quietly dropped when theyfailed to perform. When looking at products from large vendors I believe the key to risk assessment is to see how far the vendor is straying from its core competence. For example, Oracle is hardly likely to abandon its core database product, which still accounts for a huge share of its profits, but just how committed will it be to something a long way from this core area of its expertise? SAP has come to dominate the ERP space, but its execution on products away from its core competence has been shaky, to say the least. The most recent example was its dropping its MDM offering after two years, now promising a new product based around an acquisition. Cold comfort to those loyal customers who pioneered SAP MDM thinking that it was the “safe” choice. Vendors tend to misfire the further they stray from their core area of business, and customers should factor this into their risk assessment.

Assuming the software you are interested in is not from an industry giant, then how do you assess the risks then? Small software vendors always dread the following sentence: “We like your software but we will need to bring in our financial due diligence team before we go any further”. This is partly because large corporations frequently lack experience in understanding how software companies are financed, and end up asking the wrong questions. Financial analysts used to dealing with large, stable public companies are often surprised at how small, and how apparently shaky, the balance sheets of privately held software companies are. This is partly because most are venture funded, and venture capital firms are careful to dole out their capital as their portfolio companies need it, rather than investing cash just to bolster company balance sheets.

Before looking at the right questions to ask, here is a true story to illustrate the wrong way. When working at Shell I was asked to look at a small company called Dodge Software (the name in itself did not inspire confidence), a general ledger vendor with some innovative technology that a subsidiary of Shell had already purchased. Before making a deeper commitment it was decided that due diligence should be done, so I was teamed up with a banking type with a very posh accent to look at the company. The company was very reluctant to share its accounts, but as it wanted the business it had little choice. There was a clear problem in terms of cash, with the company having less than six months of cash left at the rate it was burning. The finance analyst called the companies VCs, who hardly surprisingly sang its praises and its rosy future - well, what else were they going to do? The banker then met with the company CFO, who assured him that everything was fine and that further funds could be raised as needed. This actually comforted the banker, but not me, because I could see that, while the company had built up 16 customers with good names, there was no momentum: there were few recent customer names. This meant that the company in fact was stalling, and so would very likely struggle to raise another round of capital. Even if it did, why would the market situation improve for this company? It was pig-headedly selling a best-of-breed general ledger package at a time when broad integrated finance packages were all the rage, and it seemed unlikely to change this mind set. Consequently I wrote a negative assessment and the banker a guarded but positive one. The company duly folded about six months later, unable to raise new money.

The key here is not to focus entirely on the cash position of the vendor. If the company is growing fast and acquiring prestigious customers at a steady clip then it will very likely be able to raise more cash when it needs it. However there is a saying in venture capital: “raise money when you don’t need it, because it is hard when you do need it”. When things are going well then VCs flock around, but when they are problems then they stay away on droves.

The message is that there are certainly risks in buying software, but a risk assessment should be carried out even if buying from the largest vendors. For smaller vendors their market momentum is critical, and needs to be assessed just as much as their cash reserves.

It is interesting how success attracts competition. Teradata have built up a billion dollar business from selling high end hardware and proprietary database technology to handle extremely large transaction-based data warehouses, such as those occurring in retail, Telcos and retail banking. Netezza has done an excellent job of raising its profile as a start-up in clear competition to Teradata, while now there are even newer startups like DATAllegro, offering a data warehouse appliance in competition to both (with a new offering out today) and Calpont. This is a healthy sign in an industry that is undeniably very large (business intelligence is variously estimated to be USD 25-35 billion in size, though the vast bulk of this is consulting services) yet has remained extremely fragmented in terms of vendors. Software vendors, other than the DBMS vendors, are few and far between in the data warehouse space, since the industry is mostly locked into a custom build mentality, with Kimball v Inmon design religion wars being the order of the day. SAP have, after some false starts, brought their BW product to a wide (SAP) audience but, other than Kalido, there are few data warehouse software companies. Of course there are ETL vendors such as Informatica and Ascential (now bought by IBM) and the reporting tools of Business Objects, Cognos and Hyperion, but the data warehouse itself has lacked much in the way of software automation

Teradata have succeeded despite an apparently major obstacle: the highly proprietary nature of their offering. Large companies CIO departments generally loathe proprietary infrastructure, especially when they have just spent years trying to (just about) standardize on a particular database or hardware platform, so it is an uphill struggle for the appliance vendors. Red Brick briefly did well selling a database tuned for data warehouse applications, but eventually it could not shake off the idea that Oracle or IBM could just add a “star join” feature to their products and make it redundant. Hence it is to Teradata’s credit that they have maintained clear blue water between themselves and Oracle/IBM/Microsoft at the high end of large data warehouses. This in turn has created a market large enough to attract new entrants such as Netezza and DATAllegro, who can offer an easy to understand “like Teradata, but cheaper” message to customers who have giant transaction datasets to analyze but balk at Teradata’s high price tag and opaque pricing when it comes to maintenance payments. It will be very interesting to see whether IT departments will pass a blind eye over the proprietary nature of these offerings (after all, this objection was essentially what killed off object databases) in the way they have with Teradata, though rumor has it that Netezza at least is making good early progress.

Of course only a small subset of data warehouses have the kind of volumes and processing requirements that require such technology. A TDWI survey showed Teradata at just 3% market penetration of deployed data warehouse databases, but of course this is a very attractive 3%, with typical deals in the million dollar range. Teradata has managed to overcome the proprietary stigma that bedeviled object databases in the 1990s and carved out an attractive high end niche that Oracle etc seem unable to really compete with. Its challenge now is growth, with competitors like Netezza nibbling into its margins and general purpose databases that get more powerful with each release. However the boom in raw data e.g. RFID seems likely to mean there is plenty of demand yet for raw power.

Gartner have recently been using the term “enterprise information management” as a blanket term to describe the technology and processes around a company’s efforts to control and best use its information assets. The term extends beyond structured data into text, and even to digital content such as movies or music. As they identify, a key to making progress in such a potentially monumental task is to resolve semantic inconsistencies across the technical boundaries. The problem will be familiar to anyone who has worked in a large company. A product code used in the ERP system has a different code in the CRM system, and a different one again in the manufacturing system. There are godo reaons why such differences have emerged. If we take a physical product, then a manufacturing group will care about the materials that go into that product, its manufacturing process, and perhaps health and safety information associated with it. From a distribution perspective, its dimensions are important e.g. how many will fit in a container. From a marketing viewpoint it is important to understand the branding used, the packaging (perhaps the product is marketed in different ways in different countries) and the pricing. Each business unit cares about certain aspects of the product, but has limited direct interest in other aspects, so it is hardly surprisng that the ways in which the product is classified are different depending on whether you have a manufacturing, distribution or marketing viewpoint. For example, even something familiar like a Big Mac actually has quite different recipes in different countries; in some cases it is no longer even made from beef (e.g. in India, where the cow is sacred). A branded automative lubricant will have quite different technical specifications if you buy a can of it in a hot country like Vietnam than if you buy apparently the same product in, say, Iceland.

These different perspectives cause a complex web of differing classifications and semantics to occur for products in a large enterprise, and it is similar story for other terms like “customer”, where again the key points of interest varies dramatically if you are a salesman from if you are trying to deliver a consignment to that customer, to whether you are trying to collect a payment from them. This is not just of academic interest: according to a Reuters survey, 30% of operational errors (e.g. incorrect deliveries) can be traced back to poor quality data. Those who have ever had the joy of trying to change your address on your bank or savings account will be familiar with the issue that your details do not just occur in one system only in many banks.

Trying to manage the various types of data in a large company is a mammoth task, and one which is on an uncertain footing since brands and customer details do not stay constant forever. This underlying patern of change means that initiative which seek to standardize (say) product codes “once and for all” are doomed to failure, because the things themselves are changing. However progress is possible. The first stage is to gain an understanding of the data across the company, then to describe the processes used to update this master data, and finally to bring automation to these processes. There is no single technology silver bullet since business processes are just as important as integration technology, but there a number of technologies do help matters e.g. data quality tools to help identify issues, emerging master data management products to assist with process automation, data warehouse technology to help understand and classify reference data, and EAI technology to actually link up and automate processes once they are under control. “Think big, start small” is the mantra, starting with a manageable scope and going through this process: identify data -> capture the processes -> automate the processes. Modern technologies that are better able to deal with change, along with universal access to the internet and so to applications that can automate workflow, are starting to make it possible to begin this enterprise information management journey. Confuscious said: “A journey of a thousand miles begins with a single step”, and companies can set about this journey with more confidence than ever before.