The software industry is not good at learning from previous lessons and mistakes. We seem to re-invent the wheel at fairly regular intervals, perhaps because a lot of people working in the technology industry are quite young and perhaps because we assume that anything done ten or more years ago is inherently outdated. One area I regularly observe this collective blind spot is in estimating and project management. Software projects have a poor track record of coming in on time and budget, and this has a number of causes. One is unrealistic expectations. A wise aeronautical engineer once said: “better, cheaper, faster: pick any two, but never all three” yet we all still encounter projects where the end date seems to be set by some surreal remit (”the CEO wants it in time for Christmas” syndrome) without regard to the feasibility or effect on the project deliverable. Moreover, when a project does hit problems, as all too many do, there still seems to be the impression that throwing more resources at it will claw back the time lost. Sadly this is rarely the case.

There is some useful theory on this subject. A number of writers in the 1980s published algorithms to help estimate project duration and team size, based on the observation of many software projects. You can read more about this subject in books such as “Controlling Software Projects“by Tom deMarco, Software Engineering Economics by Barry Boehm and Measures for Excellence by Putnam and Myers. However these sources agree that the evidence is that in order to bring a project end-date forward you need to deploy exponentially more resources to do that. The theory actually shows an equation relating the elapsed time to effort as:

effort = constant/time to the power four

which for the less mathematically inclined means that the end date (project time) has a BIG effect on the effort needed. For example a project that was estimated at 18 months elapsed (a nice round number selected by IT management) could, if it was extended to 19 months, be done with 20% less effort. That’s right: by extending your elapsed time by 5% you need 20% less effort. When I first saw this it seemed almost absurd, until I got involved briefly in project estimating when I worked at Shell and observed two projects in the upstream. They were that rare thing: the same project, but being done in different Shell subsidiaries. They were independent, and were the same size and scope. One project was estimated to 13 months, the other was to take the same, but in one project a decision was taken to bring forward the elapsed time to 12 months in order to fit in with another project. Money was not a major factor on this particular project, and more resource was piled in to bring the date forward. Remarkably, the compressed project took 50% more effort to bring in than the one which ran its “natural” course, something that caused general bewilderment at the time but one which actually fits in tolerably well with the software equation above.

Why is the effect so dramatic? By adding new people to a project, people that were working on the project have to stop what they were doing an on-board the new people. Now there is a bigger team, the communication becomes trickier, as more people need to be involved and the project specification is open to interpretation by more people. As you add more and more people the problem worsens: now people don’t fit into one room any more, so they have to have a meeting in order to solve a problem rather than just turning to their neighbor etc.

The message is that if you have a project that is having problems with its schedule, it is much easier to reduce the scope of the project slightly, and deliver the rest of the functionality in a later phase, than it is to try and pile on more resources to cram it into the original schedule. If you can’t reduce the scope of the project then you can only make the people on project more productive (good luck with that) or add a LOT of resources. At least there is a formula you can look up to tell you many more resources you need, even if management won’t like the answer.

Now that master data management is gaining attention as an issue, it is interesting to observe the stances of the industry giants. As perhaps might be expected, each claims to have an all-encompassing solution to the issue (though they curiously had no product offering at all in this area two years ago, so presumably must count as quick learners) - all you have to do is adopt their middleware stack. Oracle have their DataHub, SAP have MDME or whatever it is called this week, IBM have an offering crafted by their acquisitions of DWL, Trigo and Ascential, Microsoft is well, Microsoft. All of them seem to be missing a key point. Intent on expanding their own infrastructure footprint at the expense of their rivals, they do not seem to grasp that large enterprise customers simply aren’t in a position to move wholesale to one middleware stack. Large global companies have SAP and IBM Websphere, and Oracle, and Microsoft and have a huge base of deployed applications using these, so any solutions which says: “just scrap the others and move to ours” is really not going to fly for the vast majority of customers.

By contrast what customers need is software that can, in a “stack neutral” way, deal with semantic inconsistency of their business definitions, whatever technology these definitions reside in. Surely by now it is clear after the billions of dollars spent on ERP that “just standardize” is simply a doomed approach for companies of any scale. Large companies can just about manage a common chart of accounts at the highest level, but soon as you drill down into the lower level details these definitions diverge, even in finance. In marketing (which by definition has to respond to local markets) manufacturing and others there is even less chance of coming up with more than a small subset of common high-level business definitions. Just as the Eskimos are said to have fifty-two words for snow, you;d be surprised at how many different ways large companies can describe a product, or even something apparently unambiguous like “gross margin” (26 different ways in one company I worked for). Hence you need technology that can help resolve the semantic differences here, and support the workflow required to allow maintenance. For example, DWL was strong at customer data integration at the detailed level, but many types of MDM problems require complex human interaction. For example a new international product hierarchy does not just get issued; there are versions of it, people need to review, modify and finally publish a golden copy. Most MDM tools today simply ignore this human interaction and workflow issue.

I think IBM have the best chance of figuring this out of the big four, simply because unlike SAP and Oracle they don’t have an applications business to defend, while Microsoft has never really figured out how to deal with the complex end of large enterprises. IBM’s acquisitions in this area may have been multiple but they have been shrewd. Ascential was the strongest technology of the ETL vendors, while DWL and especially Trigo were well respected. Ironically, IBM may need yet another leg to their strategy, since they too have yet to really address the “semantic integration” problem that is at the heart of MDM.

A big thank you to those of you who nominated this blog for the blog awards. I am pleased to say that it has been short-listed as one of the ten finalists for the independent tech blog of the year. I am so glad that you are finidng the blog interesting. Please register your vote for the final round.

An October 2005 survey by IT Toolbox shows that, even amongst large companies, the size of the corporate data warehouse is, er, not that big. Out of 156 responses (40% US), only 12% had enterprise data warehouses larger than 4TB, while 18% had ones between this and 1 TB, while the rest had data warehouses less than 1TB. Indeed 25% had data warehouses less than half a terabyte. Admittedly only 20% of customer had just one data warehouse, with 26% having over five warehouses, but these figures may seem odd when you hear about gigantic data warehouses in the trade press. Winter Group publish a carefully checked list of the 10 largest data warehouses in the world, and their 2005 survey shows the winner at Yahoo weighing in at 100TB. The tenth largest, however (at Nielsen), is 17TB, which shows that such mammoths are still a rarity.

Why are IT folks obsessed about this? I can recall speaking at a data warehouse conference a few years ago and speaker after speaker eagerly quoted the size of his data warehouse as some sort of badge of courage: “Well, you should see how big mine is…”. Of course companies that sell hardware and disk storage love such things, but why is there such a big discrepancy between the behemoths in the Winter Group survey and the less than a terabyte brigade? The answer is quite simple: business to business companies don’t have large transaction volumes. If you are a large retailer or a high-street bank, then you may have thousands of branches, each one contributing thousands of individual transactions a day. These add up, and constitute the vast majority of the volume in a data warehouse (perhaps 99% of the volume). The rest of the data is the pesky master data (or reference data, or dimension data - choose your jargon) such as “product”, “customer”, “location”, “brand”, “person”, “time” etc that provides the context of these business transactions. You may have millions of transactions a day as a retailer, but how many different products do you stock? 80,000 for a convenience store chain? 300,000 for a department store? Certainly not tens of millions. Similarly McDonalds has 27,000 retail outlets, not millions. The same for organizational units, employees etc. One exception that can be very large is “customer” but again this is true only for business to consumer enterprises e.g. retailers or Telcos. Companies like Unilever are very large indeed, but primarily sell to other businesses, so the number of direct customers they deal with is measured in the many thousands, but not millions.

So B2B enterprises usually have quite small data warehouses in volume, even though they may have extremely sophisticated and complex master data e.g. elaborate customer segmentation or product or asset classification. One way to measure such complexity is by adding up the types of business entity in the data model e.g. each level of a product hierarchy might count as one “class of business entity” (CBE), “customer” as another. Some very large data warehouses in volume terms often have very simple business models to support, perhaps with 50 CBEs. On the other hand a marketing system for a company like BP may have 400 or more CBEs. This dimension of complexity is actually just as important as raw transaction size when looking at likely data warehouse performance. A data warehouse with 1TB of data but 50 CBEs may be a lot less demanding than one with 200GB of data but 350 CBEs (just think of all those database joins). Oddly, this complexity measure never seems to feature in league tables of data warehouse size, perhaps because it doesn’t sell much disk storage. I feel a new league table coming on. Anyone out there got a model with more than 500 CBEs?

A 2005 research note from Gartner says that more than 50% of data warehouse projects through 2007 will be either outright failures or have limited acceptance. This does not surprise me in the least. There are several aspects in which data warehouse projects are under unusual strain, as well as the normal problems that can beset any significant project. Data warehouses take in data from several separate data sources (ERP, supply chain, CRM etc) and consolidate it. Consequently they are dependent upon both the quality of the data and the stability of those source systems: if any of the underlying source systems has a major structural change (e.g. a new general ledger structure or customer segmentation) then it will affect the warehouse. You might think that data quality was a minor problem these days with all those shiny new ERP and CRM systems, but you’d be wrong. In Kalido projects in the field we constantly encounter major data quality issues, including with data captured in the ERP systems. Why is this?

An inherent problem is that systems typically capture information that is directly needed by the person entering the data, and other things as well, which are useful to someone, but not directly to that person. I remember doing some work in Malaysia and seeing a row of staff entering invoice data into a J.D. Edwards system. I was puzzled to see them carefully typing in a few fields of data, and then just crashing their fingers at random into the keyboard. After a while, they would resume normal typing. After seeing this a few times my curiosity got the better of me and I asked one of then what was going on. The person explained that there were about 40 fields that they were expected to enter, and many of the fields were unnecessary they could not move to the next screen without tabbing through each field in turn unless they entered some gibberish in one of the main fields, at which point the system conveniently took them to the last field. So by typing nonsense data into a field that turned out to be quite relevant (but not to them) they could save lots of keystrokes.

Of course this is an extreme case, but have you ever filled out an on-line survey, got bored or frustrated because they are asking something you don’t have the data for and started just answering any old thing just to get to the end? The point is that people care about data quality when they are going to get something back. You can be sure they enter their address correctly on the prize draw form. But in many IT systems people are asked to enter information that doesn’t affect them, and human nature says that they will be less accurate with this than something which does mean something directly to them. Some data quality issues can be dramatic. In the North Sea one oil company drilled through an existing pipe because it was not there according to the system that recorded the co-ordinates of the undersea pipes: this merely cost a few million dollars to fix, as fortunately the pipe was not in active use that day or the consequences would have been much worse. Another company discovered that it was making no profit margin on a major brand in one market due to a pricing slip in its ERP system that had gone undetected for two years.

The reason data warehouses suffer so much from data quality issues is that they not only encounter the data problems in each of the source systems they deal with, but because they also bring all the information together the data problems often only becomes clear at this point e.g. the problem with the pricing became apparent because the data warehouse showed that they were making zero gross margin, which was not apparent inside the ERP system since the margin calculation was made up of data from several systems combined. It is the data warehouse that shines light on such issues, but often is wrongly blamed when the project is delayed as a result.

Data quality problems are one major issue where there is no magic solution. Data quality tools can help, but this is a people and process issue rather than a technology issue. However another reason data warehouse projects are perceived to fail is that they take a long time, and cost a lot to maintain. Since it takes 16 months to build an average data warehouse (TDWI) survey) it is not surprising that some changes to the business occur in this time. The only way to really address this is to use a packaged data warehouse solution, which takes less time to implement (typically less than 6 months for Kalido). Maintenance costs are another major problem, and here again there are modern design techniques that can be applied to improve this situation. See my blog the data warehouse carousel.

It is only by making use of the most modern design approaches, iterative implementation approaches that show customers early results, and the most productive technologies that data warehouse project success rates will improve. There will always be projects that run into trouble due to poor project management, political issues and lack of customer commitment, but data warehouse projects at least need to stop making life harder for themselves than they need be.

I made a discovery the other day. Not one of those “eureka” moments beloved of bathing Greeks, but something that prompted me to wonder about the accuracy of some of the figures we take for granted. We are so used to Excel on every desktop that we trust it implictly, and so when I needed to work out the standard deviation of some figures, I naturally turned to Excel. For those of you whose maths is rusty, standard deviation is how spread out a sample of numbers are. For example: the average of 1,3,5,7,9 is 5, and so is 3,4,5,6,7 but it can be seen that the latter sequence has its numbers more closely bunched. Standard deviation is just a mathematical measure of how close or otherwise that bunching is (in the examples above the standard deviation of the first set is 2.83 and the second set 1.41 i.e. the second set is closer bunched than the first set).

In Excel to use a function you just type into a cell something like “=average(1,3,5,7,9)” and magically you get the answer (5 in this case). So, what could be easier to do than type in:
“=stdev(1,3,5,7,9)” and see the answer appear? The trouble is that it doesn’t. The answer pops up as 3.16, not 2.83 as I was expecting. Just in case you doubt my ability to calculate a standard deviation, feel free to do it the old-fashioned way by hand, and you will see that you get 2.83, not 3.16, so what is going on? After digging around the Excel help and chatting to a mathematician friend to check I was not going completely mad, I discovered that there are actually two different standard deviation functions in Excel, one designed for where you want the whole sample set measured, and one where you want to estimate a large population from a sample, which has a slightly different formula. Now I may be getting a bit slow these days, but I did do a maths degree and yet this distinction had eluded me all these years, so I doubt I’m the only person out there unaware of this difference. If you were the person at Microsoft naming Excel functions, which do you think that people would think was the “normal” version, “STDEV” or “STDEVP”, which is what they actually named the function that calculates the standard deviation of a whole population. I am guessing that not too many of us go “aha, I’ll try “=STDEVP I expect that will be it”.

Now this may seem like a lot of fuss about an esoteric mathematical function, but be aware that standard deviation is one of the most commonly used statistical functions, used to look at samples of population, mechanical failure rates, delivery errors, temperatures, patient response rates, you name it. People take serious decisions based on statistics: which drug to put forward for clinical trials, traffic planning, machine maintenance and endless others; standard deviation is the most commonly used tool in “statistical process control”, widespread in the manufacturing industry. Given that the most of the modern world uses Excel, I find it pretty surprising that a sizeable proportion of the world has been using the wrong standard deviation function for the last twenty years all because some idiot in Seattle chose a “precise” name rather than the obvious name that most of us would have chosen.

I suppose this is only what I should have expected from a product who thinks that 1900 is a leap year. Try the formula “=DATE(1900,2,29)” and watch it happily display the 29th February 1900. As we should all be aware after the fuss over Y2k, 1900 is NOT a leap year (leap years are every four years, except centuries which are not, expect every fourth century, which is, so 1600 and 2000 are leap years, but not 1800 or 1800 or 1900). The moral of this little story: don’t take everything on trust!

Tony Lock writes an excellent article in this week’s IT-Director.com newsletter, highlighting the communication gap between IT departments and their customers. A new survey by Coleman-Parkes found that amongst 214 FTSE 750 organizations, only 18 percent held weekly meetings between business managers and the IT teams. The research also indicated that 31 percent of those surveyed claim that they never or hardly ever have such meetings. In large corporate IT departments there can be a culture of avoiding contact with “users”, who always seem to have strange and unreasonable demands that don’t fit into the perceptions of the IT department. The atmosphere can become quite hostile if IT departments set themselves up as “consultancy” organizations that charge themselves out to their internal customers. The internal customers resent being forced to use an internal service that they often perceive as unresponsive, and can be outraged to find themselves being charged at similar rates to external service providers. Some of this is not reasonable - those same customers are forced to use internal legal counsel and are charged through the nose, whether or not they like it. However there is a peculiar frustration with many business users over their IT departments that can boil over when discussing charge-back mechanisms and service level agreements.

Over-elaborate internal billing systems can cause unnecessary cost and frustration. I recall when I was at Exxon seeing an instructive project to review internal data centre charges. The existing system was extremely elaborate and charged based on mainframe usage, disk storage, communications costs and a whole raft of items. Most users didn’t understand half the items on their bills, or played games to try and avoid hitting arbitrary pricing thresholds. None of this added one iota of revenue to Exxon. The project manager, a very gifted gentleman called Graham Nichols (on his way rapidly up the organization), successfully recommended replacing the entire system with a single charge once per year. This saved a few million pounds in administration and endless arguments, and people’s tempers were much improved all round.

Perhaps some of the problem is when an organization grows very large, it is difficult to keep perspective. Shell employed around 10,000 IT staff in the 1990s, directly or indirectly, so it perhaps not surprising that the IT staff concentrated on their own internal targets and objectives, rather than troubling themselves too much to align themselves with the objectives of the core energy business. At a time when the oil industry was struggling with oil prices heading down towards 10 dollars, and so with serious cost-cutting going on all round, the internal IT group, living through the internet boom, was hiring to keep up with demand e.g. dealing with the Y2K problem. Seeing redundancies going on in engineering and marketing at the same time as a hiring boom in internal IT, tempers became frayed, to put it mildly.

Clearly senior internal IT staff do need to spend more time with their business customers, and find out how they can help them achieve their objectives. Moreover they need to communicate this throughout their organizations. How many internal IT staff know the top three business objectives of their company this year? Without even a vague idea of the goals that the business is pursuing, it is hardly surprising that business leaders become frustrated with internal IT groups. Those 31% of internal IT groups who never or hardly ever meet with their customers need to change this attitude or get used to living on a Bangalore salary in the future.

IDC today announced their annual survey results of the size of the data warehousing market. IDC sizes the overall market in 2004 at 8.8 billion. The “access” part of market e.g. Business Objects, Cognos, was USD 3.3 billion,”data warehouse management tools” (which includes databases like Teradata, and data warehouse appliances) was USD 4.5 billion Data warehouse generation software (which includes data quality) was sized at USD 1 billion. This was 12% growth over 2003, the fastest for years, and IDC expect to see compound annual growth of 9% for the next five years.

One feature of this analysis is how small the “data warehouse generation” part of the market is relative to databases and data access tools. It is in some ways curious how much emphasis has been on displaying data in pretty ways (the access market) and the storage mechanism (data warehouse management market) rather than how to actual construct the source of the data that feeds these tools. This is because today that central piece is still in the cottage industry stage of custom-build. Indeed with an overall market size of USD 35 billion (Ovum) it can be seen that the bulk of spending in this large market is still with systems integrators. Only a few products live in the “data warehouse generation” space e.g. SAP BW and Kalido (data quality tools should really be considered a separate sub-market). Hence the bulk of the industry is still locked in a “build” mentality, worrying about religious design wars (Inmon v Kimball) when one would have expected them to move into a “buy” mentality. This inevitably will happen, as it did to financial applications. Twenty or so years ago it was entirely normal to design and build a general ledger system, and who would do that today? As large markets mature, applications will gradually replace custom-build, but it is a slow process, as can be seen from these figures.

The average data warehouse costs USD 3 million to build (according to Gartner) and only a small fraction of this is the cost of software and hardware, the majority being people costs. It also takes 16 months to deliver (a TDWI survey) which is an awful long time for projects which are supposedly delivering critical management information. To take the example of Kalido, the same size project takes less than 6 months instead of 16 months, so for that reason alone people will eventually come around to buying rather than building warehouses. Custom data warehouses also have very high maintenance costs, which is another reason for considering buy rather than build.

The rapid growth in the market should not be surprising. As companies have bedded down their ERP, supply chain and CRM investments it was surely inevitable that they started to pay attention to exploiting the data captured within those core transaction systems. The diversity of those systems means that most large companies today still have great difficulty answering even simple questions (”who is my most profitable customer”, “what is the gross margin on product X in France v Canada”) which causes senior management frustration. Indeed a conversation I had at the CEFI conference this week with a gentleman from McKinsey was revealing. In recent conversation with CEOs he explained that McKinsey were struck by how intensely frustrated CEOs were at the speed of response of their IT departments to business needs, above all in the area of management reporting. 16 month projects will not do any longer, but IT departments have are still stuck in old delivery models that are not satisfying their business customers - the ones who actually pay their salaries.

Douglas Adams once remarked that mathematics is universal, with its rules being consistent across the universe with the exception of the numbers on an Italian waiter’s billpad. To this he might have added the supposed attendee numbers at IT industry trade shows. Ever been to a show recently where the organizers claim 500 paying attendees, yet you can only count half that? Some trade show exhibitor sections in the last few years have become depressing affairs, with a thin trickle of conference attendees registering for prize draws, or in some cases just handing over their resumes at the booths. Vendors have been increasingly desperate to drum up attendance. At one trade show recently I was handed a brochure about a database appliance by a pretty girl. I asked her about the technology and she said “actually, I don’t know anything about this - I’m a model”. Is this what things have come to?

I had assumed that the malaise would have improved with the general modest recovery in the IT industry’s fortunes, but this barely seems to have happened. I am unaware of any proper data on the subject, but anecdotally the punters seems to be staying away in droves. Where have they gone?

For one thing they are attending webinars. Enabled by modern technology such as Webex, webinars are a more targeted way of reaching people interested in your message. They are attractive to vendors because they are relatively cheap (a trade show exhibit can cost USD 10-30k in fees, plus travel and people costs) and you get people attending who are genuinely interested. Instead of a trickle of bored looking geeks in search of free giveaways, with the odd five minute conversation thrown in, at a webinar people log in and listen to you for perhaps an hour. The number of contacts can compare well also. At a regular trade show you might get (say) 40-50 contacts, but perhaps only 5-10 of these will be of any real level of interest. By contrast, at a webinar you have people who have bothered to take an hour of their time to listen to you. At Kalido we have run webinars with over 300 attendees, so it can be seen that this compares very favorably to trade shows.

The other forum that people still attend are user groups e.g. there were 2,000 attendees at last week’s Business Objects user group. Customers still want to hear about product directions, meet other customers and get a bit of free education. While trade shows are not yet an endangered species, I wonder whether the rise of the webinar will gradually cast them in the role of the slide rule against the pocket calculator.

A recent Business Week article gave a good example of a small software company providing an important solution to retailer Circuit City in the face of competition from industry giants. The dot-com madness made CIOs understandably wary of small software companies bearing gifts, yet it is important for these same CIOs to realize that they do their shareholders no favours by taking an ultra-conservative “buy only from giants policy”. For a start, this option is by no means always safe. It also a flawed strategy.

Industry giants rarely produce innovative software. For example the founding executives of both Siebel and Salesforce.com were both Oracle executives, but were unable to create what they knew the market wanted at Oracle itself. Large companies become inevitably less fast-moving as they grow, and frequently become more inwardly focused and cease listening to their customers, the very people that made them successful when they were small. In my years as a strategic technology planner I learned that the key to success in software portfolio planning was a twin approach: standardize on commodity infrastructure, yet encourage innovation above this layer. For example, it is pretty clear by now that the major relational databases (Oracle, DB2, SQL Server) are all functionally rich and basically work. Nobody uses more than a fraction of the features they have, and which one you choose is largely a matter of taste. However it is best if you can standardize on one of them, since you then get easy interoperability, and your IT staff build up skills in that technology that transfer when they switch departments. This is an example of a layer of infrastructure that has matured to the extent that the benefits of standardizing outweigh a few features at the edges.

Yet at the application layer this is by no means the case, except perhaps in the area of finance, where no one is really likely to produce a deeply innovative general ledger system any more. Yet this is clearly not the case in marketing, sales and many other business areas, where exciting applications are still popping up, and companies like Salesforce can radically change an existing application area. Here it makes no sense to try and second guess the market, where evolution is still working its magic to see what technologies work best. Trying to standardize too quickly in an area that is evolving will not only most likely make you look foolish when you get it wrong, but also misses real opportunities for companies to take advantage of new and exciting offerings.

Giant software behemoths are not the place where innovation flourishes, and the further they get from their core area of competence, the less likely they are to succeed. As a strategic technology planner, your job is to solidify the core infrastructure but to enable your customers to take advantage of innovation in fast-moving or evolving areas. This state of affairs is not going to change in software, where fairly low barriers to entry enable innovation to be created without massive capital investment. Best of breed software does indeed live!