I just read a provocative blog on SOA which raises an interesting point. Articles on SOA tend to focus on the technical issues e.g. performance, standards etc. While I don’t agree with everything in the article, Robin Harris is correct in pointing out that how a new piece of infrastructure is perceived depends in part on the pricing mechanisms that end users see. Different IT departments charge in different ways. Some levy a “tax” on the business lines, perhaps based on a simply charge-back mechanism “retail is 20% of the business, so they pay 20% of the IT department’s costs”. Others charge out for services in a much more granular way e.g. a charge for each desktop, a charge for each GB of server storage, a charge for DBA services etc. The latter has the big advantage of being related to usage, meaning that heavy users of IT pay more, presumably in some relation to the costs that they cause the IT department to incur. The disadvantage is that the pricing can easily become over-complex, with user departments receiving vast itemised bills each month for storage, telecoms, data, networking, applications support etc in minute detail. This can cause some users to try and “game” the system by taking advantage of any flaws in the pricing model, which may make logical sense to the individual department but may actually cause the true costs to the enterprise to rise. For example if the IT department prices storage in bands then a department may resort to all kinds of tricks to avoid moving into the next pricing band, and yet the effort involved in fiddling around may exceed the true costs to the company of just buying some more storage.

At one time I worked in Esso UK, and a study was done of the pricing mechanism, which was of the complex/sophisticated type. The recommendation, from a bright young manager called Graham Nichols, was simply to scrap the charge-back systems altogether and just levy a simplistic charge by department. This actually saved three million pounds in costs, which is what it took for th charge-back systems to be administered. No doubt years later things have changed, but this was an example of how the internal systems added no value at all to the company, and by simplifying them could remove a layer of administration. The drawback to simplified systems like this is that there is no incentive for increased efficiency, since the departments know what they are going to get charged so perceive no extra cost in heavier usage of the systems. This may eventually cause heavier systems costs which will be charged back eventually to departments; it is a question of balancing the costs of the internal processes v the potential higher costs that may occur.

SOA is an example of core infrastructure which pricing mechanism have always struggled with i.e. how do you justify an investment in infrastructure which has no benefit at first, but will incrementally benefit future applications? However the investment in justified and charged back, a key point is that the investment should be justified, like any other. IT departments should view a new piece of infrastructure like other departments consider capital expenses e.g. a new fleet of trucks or a new factory. What is the payback compared to the investment? What is the IRR, NPV and time to break-even? I have not seen much if at all written about this aspect of SOA, and yet we all need to understand what productivity gains are actually going to occur before we head down this path. There may be significant productivity improvements, or none at all (indeed it could be worse than today) and yet commentators seem to take SOA as a given. If a whole industry moves in a certain direction then eventually this can be hard for end-user companies to avoid e.g. if you decided a decade or two ago that client/server was just an expensive way of distributing data from one safe, secure place (mainframe) to lots of unsafe and insecure places (PCs) then you could have tried to hang on to your mainframe, but eventually fewer and fewer applications would run on your old mainframe, and you would be obliged to switch whether you liked it or not. It is not yet clear that SOA has that kind of momentum. However I am sure that understanding its economic impact would be valuable for all sorts of reasons. I look forward to seeing someone addressing this issue seriously (I do not count breathless marketing material from vendors selling SOA services claiming 10:1 improvements in everything from productivity to quality, without any actual pesky real examples), but I am not holding my breath.

I fully agree with an article by Steve Miller:

http://www.dmreview.com/article_sub.cfm?articleId=1076651

about how the Harvard Business Review is a surprisingly useful resource for people working in business intelligence. One of the recurring themes I have noticed over the years with projects going wrong is that the root cause of problems is more often people communications than technology. Of course as technologists we are inevitably drawn to the technical issues around the latest technology - performance, how buggy the software is etc, but few pieces of commercial software are so poor that they will cause a project to fail directly due to the software (I exempt Commerce One from this generalisation; it was that bad). The useful thing about Harvard Business review is that it gives some insight into the kind of issues that are confronting senior management, or at least about the kind of issues they are reading about.

However the HBR is rather hard work. There are rarely articles about technology directly (an exception was the November 2006 “Mastering the Three Worlds of Information Technology”) but technology often crops up within other articles, as Steve Miller points out. What I would add is that HBR can be a rather ponderous read. Their articles tend to be long and in-depth rather than bright and breezy, and there is a politically correct element about HR issues which can seem quite sanctimonious. But for every painfully worded article about the joys of diversity training there are several useful ones about current management trends and hot topics.

Speaking the same language as senior management is a stepping stone on the road to better understanding and communication, and that in turn will help improve the propsects of success for a BI project.

HP continues with its plans for the business intelligence space with an announcement of in-house data warehouse technology:

http://www.computerworld.com:80/action/article.do?command=viewArticleBasic&articleId=9008218&intsrc=news_ts_head

with a new business unit. The offering with be based around HP’s attempt at a “data warehouse appliance”, called Neoview. This is a competitor to Teradata and Netezza, but at this stage it is hard to tell how functional this is, since it is unclear that there are any deployed customers other than HP itself.

The timing of this announcement is curious given HP’s acquisition of data warehouse consultancy Knightsbridge. Certainly data warehousing is a big market and Teradata is a tempting target - after all, most of the really big data warehouse deployments in retail, telco and retail banking use Teradata. There are lots and lots of juicy services to be provided in implementing an “appliance”, which in fact is no such thing. An appliance implies something that you just plug in, whereas data warehouse appliances are just a fast piece of hardware and a proprietary database, still requiring all the usual integration efforts, but with the added twist of non-standard database technology. Certainly plenty of business for consultants there.

However HP’s home-grown offering will not sit well with its newly acquired Knightsbridge consulting services, who made their reputation through a quite fiercely vendor-independent culture which always prided itself in choosing the best solution for the customer. People trust independent consultants to give them objective advice, since they are not (or at least they hope they are not) tied to particular vendor offerings. Presumably HP’s consultants will be pushing HP’s data warehouse solution in preference to alternatives, and so can hardly be trusted as impartial observers of the market. An analogy would be with IBM consultants, who while they may work with non-IBM software are clearly going to push IBM’s offerings given half a chance.

If you were a truly independent consultant how would you react to a brand new data warehouse appliance with a track record only of one deployment, and that in the vendor itself? Would you immediately be pushing that as your preferred solution, or would you be counseling caution, urging customers to wait and see how the new tool settles down in the market and how early customers get on with it? If you are a Knightsbridge consultant now working for HP, what would your advice be? Would it be any different to the advice you’d have offered in December 2006 before you became part of HP?

This kind of conflict of interest is what makes thing difficult for customers when choosing consultants. It is hard to find ones who are truly independent. Of course consultants always have their own agenda, but usually this is about maximising billable hours. If they are tied to a particular solution then that is fine if you are already committed to that solution, but you will need to look elsewhere for objective advice about it.

One thing that continues to surprise me is how little developed the business case for data quality and master data management is.  When I look at data quality vendors speaking at conferences I can sit through whole sessions which do not mention the amount of actual dollars their clients saved by using their technology. In the case of MDM there is some excuse for this, since MDM as a term only recently became mainsteam, and so few vendors have real projects that are in production with clients.  Indeed just 4% of companies have completed an MDM project, according to a recent survey by Ventana (though 37% claim to have initiated a project).  However in the highly related field of data quality there are no such excuses: tools have been around for years, and yet trying to find examples of well justified projects with a hard dollar payback is like pulling teeth.

While data quality has remained something of a backwater (the largest data quality vendor does around USD 50M in revenue) it is surely one of the things that should be relatively easy to produce a cost benefit case for.  After all the tools will enable you to detect the proporton of bad data in a given application or enterprise, and it should not be beyond the wit of man to be able to assign a cost of poor data quality.  Even ignoring tricky things like customer satisfaction, poor data causes very real things: deliveries going to wrong places, misplaced inventory, incorrect payments, problems in manufacturing.  In certain industries it can be worse: drilling an oil well in the wrong place is an expensive affair, for example.  An 2003 AT Kearney study showed that USD 4 was saved for every dollar spent on data cleansing activity. 

By going back and looking at completed projects and carrying out cost/benefit analysis the data quality (and MDM) vendors will be doing themselves a favour, since by quantifying the savings these projects bring they can not only make it easier to justify new projects, but they beging to justify the price of their products: indeed they may be able to gain improved pricing if they can demonstrate that their products bring sufficient value to customers. It is a mystery to me as to why vendors have made such a poor show of doing so.

The open source movement is creeping into BI in various guises, as pointed out by Jacqueline Ernigh.  However, while Linux is undoubtedly taking a lot of market share from proprietary UNIX variants, it is less clear of progress higher up the stack. The article mentions a number of organisation that provide some form of open source reporting tools e.g. Pentaho, Greenplum and Jaspersoft, and indeed there are others still. However it is by no means clear what penetration these are really getting.  It was noticeable that one of the two customer examples reported merely had a database running on Linux, but had yet to deploy open source reporting tools. 

The article unfortunately loses credibility when it cites an example of the savings to be made: ”At pricing of $1,000 per user seat, for example, a company with 16,000 employees would need to pay $160,000 for a full-fledged BI deployment, Everett hypothesized.”  Hmm.  It is some time since I did my mathematics degree but I am thinking that 16,000 * 1,000 = 16,000,000 i.e. 16 million dollars, not $160,000.  Even if you are kind and assume that a major discount could be obtained for such a large deployment, even an unlikely 90% discount to list would still get you USD 1.6 million.  I doubt that Dan Everett at the entirely respectable research company Ventana would really have made such a dramatic numerical blunder, so perhaps it was a journalistic error.  Such carelessness does make one wonder about the accuracy of rest of the article, which is a pity since it is discussing an interesting trend.  

I still have yet to really come across significant deployments of open source reporting tools in production applications, but presumably they will catch on to a certain extend, just as MySQL is making steady inroads into the database market.  Perhaps the most significant point at this stage is not made by the article though.  The very existence of open source reporting tools puts pricing pressure on the established BI vendors.  Procurement people doing large deals with BI vendors will treat the open source BI movement as manna from heaven, since they have a stick to beat down the price of reporting tools from the major vendors.  Anyone about to engage in a major BI deployment or negotiation would be well advised to look carefully at these tools, if only as weapons in the armoury against pushy software salesmen.  This is further bad news for the BI vendors, who have enough to worry about with the push of Microsoft into their space and the general saturation of the market. In this case even a handful of customer deployments will suffice to send a shiver down the spine of the major vendors.

I came across a really inspired blog the other day which I would highly recommend that you read.  The Cranky Product Manager is written by an anonymous American product manager at a software company.  There are several fine aspects to the blog, not least of which is that it is well written (let’s face it, too many blogs out there look like they were written by a dyslexic 12 year old).  However the best aspect is that her anonymity enables her to be delightfully rude about many aspects of the merry go round that is the software industry.  Her blog “Streetwalkers in Disguise” is a delightful example of this.

Those who have worked for some time in the software industry will have many a wry smile at the trials and tribulations of the Cranky PM, whose writing clearly reflects the very realities of product management, rather than some ultra-spun anodyne story that is so often fed to eager journalists as an “insider” story but is just clever PR.  

As someone who wades through more blogs than I care to admit to, I wish more blogs were like this one.

Sreedhar Srikant writes about the importance of the logical data model in a data warehouse project in DM Review. This well-written article describes the process of building a model, highlights five pitfalls and suggests some ways of avoiding them.  It is in this last area that I feel the article could be enhanced.  In my experience there are two serious dangers that a data warehouse project faces that go beyond issues of project problems like trouble agreeing on a model.  These are:

(a)   The project gets insufficient business buy-in through lack of a well-articulated and robust business case

(b)   The project takes too long to deliver, making it vulnerable to budget cutbacks since it has not shown tangible benefit early enough.

I am constantly surprised how often IT projects in major corporations seem to get off the ground without a strong business case.  IT projects compete for capital in a company with many other project proposals, and so it can be a Darwinian process when times get tough: projects with the strongest business case and sponsorship will survive.  As a minimum, the project needs to set out the expected returns that it will make, set against the project costs, over a three year (sometimes five year) period.  A simple example is shown below:

Costs:  $3M one-off, $2.16M annual

Benefits: $5M from year 2 onwards, with $2M only in year 1.

In this instance the project costs $3M to deliver and just over $2M to support each year (a Data Warehouse Institute survey showed that the average data warehouse costs 72% of its build costs to support every year).  Against this are some benefits as shown.  In this instance the project is tolerably attractive, since it has a positive net present value (USD $536k using a typical 18% discount rate) and a decent 27% IRR, though its payback period is a little slow.  However, while not stellar, it is respectable as cases go, and it is at least written in the language of business. 

What might the project benefits be?   These will vary from project to project and from industry to industry, but examples might include either profit-enhancing benefits, such as reduced customer churn or improved pricing ability, or cost reductions such as fewer misplaced deliveries due to improved data quality, or better procurement margins to due to improved understanding of supplier spend.  In order to articulate these you need to find a business sponsor, preferably one who has a problem related to poor information.  Trust me; you should not have to look too far in a big company for one of these. 

Having a business case that is properly set out will act as a safety net when project reviews happen, and reduce the chances of a project being cancelled when the knives come out. 

The second thing that can help your project is to deliver something tangible early.  Traditional waterfall methodologies, often used by large systems integrators, are not always well suited to data warehouse projects, where requirements are often rather loose.  The average data warehouse project takes 16 months to deliver according to TDWI, and that is a long time in this turbulent world when management has its budgets adjusted and people are looking for projects to cut.  If your project can deliver something meaningful quickly i.e. a piece of the overall problem, then your project sponsor has a lot better chance of defending the project.  If all the review committee can see is costs then things will be harder.  Many real projects I have been involved with have been killed in this way.

One way to improve your odds of delivering something quickly is to use a data warehouse package, where at least some of the functionality is already pre-built for you.  Packages may or may not be cheaper than custom-build, but they should be quicker.  If you can pick off a chunk of the project and deliver reports back to the sponsor that add value early on then the project is much more likely to survive than one that is still delivering a grand enterprise logical data model. These days there are several packaged or semi-packaged alternatives to custom build.  A good overview of the packaged data warehouse market was done this year by Bloor and can be downloaded for free here.  

By developing a robust business case and by delivering benefits iteratively your project greatly improves its chances of survival.  When the budget sharks come circling, it is nice to have a life raft.

In a beye network article Gabriel Fuchs raises the issue of culture when it comes to business intelligence projects. I think the issue is valid, but it goes far beyond the rather crude generalisations about nationalities that he makes in the article.  In my experience corporate culture is every bit as important as the general culture of the country where you are doing the implementation.  Take two contrasting examples from the oil industry (I choose this because I have worked for both these companies): Exxon has a highly centralised culture driven from central office in the US.  Shell, by contrast, is decentralised in nature and is much more consensus oriented.  This is highly relevant when implementing a BI project, because in a company with a decentralised culture you need to take into account the needs of the local subsidiaries far more than in a centralised culture. In Shell, if something was decided centrally then this was a bit like traffic signals in Manila: they are a starting point for negotiation. Someone in central office could define some new reporting need or technical standard, but the subsidiaries had a high degree of leeway to ignore or subvert the recommendation (Shell is less decentralised now than it was, but even so it is still highly decentralised compared to Exxon).  In such situations it was important to get buy-in from all the potential influences in the project; for example it was wise to produce reports or BI capabilities that the subsidiaries find useful rather than just the notional project sponsors in central office.  By contrast in Exxon, while it is sensible practice to get buy-in, if you were in a hurry or things were intractable then central office would be able to ram decisions down the throats of the subsidiaries without much resistance.  Incidentally, both cultures have advantages and disadvantages, and both companies are highly successful, so it would be a mistake to think that one culture is inherently better than the other.

In BI projects such issues come up a lot when discussing things like data models, and agreeing on international coding structures v regional or local ones.  Sometimes projects that did not go well were ones where these inherent cultures were ignored.  For example Shell spent a lot of time and money trying to ram together five European SAP implementations, and ultimately failed to do so, ending up with five slightly different implementations.  There was no technical reason why this could not be done, or in truth any real business reason, but it went against the culture of the company, so encountered resistance at every level. 

In my view such company cultural issues are very important to consider when carrying out enterprise BI projects, and are often ignored by external consultants or systems integrators who blindly follow what is in the project methodology handbook. 

I really wonder about corporate espionage, the subject of a new book which claims that network technology has meant that industrial espionage is on the rise and easier than ever.  I observe that people who write books or articles about this subject, while no doubt very knowledgeable, also tend to be security consultants selling their services.  I certainly would not want to criticise such fine people, especially as they could no doubt easily find out where I lived.  But, let’s face it, a consultant is hardly about to publish an article “corporate espionage is no big deal really; no need to invest much here”.  My scepticism is prompted by a couple of personal experiences.

In a really big company there is actually very little data that is truly “secret”, and then usually only for a certain period of time e.g. quarterly resuts just prior to making them public.  Or plans for an acquisition perhaps prior to a bid, or bidding information for large contracts, maybe certain aspects of R&D.  In most cases company executives have enough trouble making sense of their own corporate information.  Let’s face it, if you can’t figure out who your most profitable customers are (a significant problem for most companies, whether they admit it or not), how are your competitors going to work it out by accessing your information systems?  However, as I say, there are some very specific pieces of data with commercial value.  When I used to work in Esso Expro we were contacted by an employee of another oil company (let’s call it Toxico) offering to sell Esso information on their bid for the next round of North Sea acreage.  Now this information was of real value, and appeared to come from one of the bid team, so was genuine.  What did Esso do?  They rang up the Metropolitan police, followed by the security department of Toxico and told them the full story.  There was no debate about this, no hesitation; it was a decision taken in a heartbeat.  Esso has well-grounded ethical principles and was having none of this.

A second personal experience was of a friend who is one of the three smartest people I ever met.  She had a meteoric rise through management in a large corporate that I suppose I should keep nameless, and was promoted to be in charge of their competitive analysis unit.  This unit did spend its time (legally) analysing its competitors and trying to pick up any snippets of competitive information that it could.  After six months my friend recommended that they close her department down.  Why?  Because she could not find a single of example of her quite well-funded department’s findings ever actually being acted upon.  In other words management liked to know what was going on, but basically did whatever they were going to do anyway. The company didn’t have the courage to actually follow through on this, and my friend duly moved on to another job (she is now in a very senior position as a major investment bank).

So, at least in these two cases, the work of a major corporate competitive analysis unit was assessed by its own boss to have no tangible value at all, while when someone did actually ring up and offer to sell valuable information, Esso declined and turned the would-be informant over to the police.

While internet hackers can undoubtedly cause a great deal of trouble, I honestly wonder just how realistic are the stories of doom and gloom on corporate espionage, in particular the fears about someone hacking into secret information systems. In most companies, the information really isn’t that exciting.  Only a tiny fraction of information is genuinely valuable/sensitive, and even if you got hold of it most ethical companies would do the decent thing as Esso did and turn in the informant.  I am sure there are some exceptions, no doubt involving tales of derring-do, but how many of these are there?

However, stories like mine do not sell security consultancy.  If anyone ever does write a book debunking corporate espionage then I promise to buy it.

Christopher Koch highlights some recent research by Wharton School of Business regarding “customer focus”.  The work is certainly right to dig deeper into what the costs are of “customer focus” and whether it is really a good thing.  It is absolutely correct to try and measure the lifetime value of a customer.  I was surprised when someone I knew working at a world famous investment bank admitted that they had little idea of the total amount of business done with a particular corporate client, let alone to what extent the aggregate business was profitable.  The problem in this case was that different parts of the bank had responsibilities for different aspects of the relationship in different countries, with different IT systems supporting those aspects.  Hence taking an aggregate view was surprisingly difficult.  

Understanding this will be much more difficult (and will make more sense) in some industries than others  A manufacturer like Unilever does not deal with the end consumer, so their customers are really the retailers.  However even in consumer facing industries, small improvements in customer “churn” can have a major effect on profits, since it costs much more to acquire a new customer than to retain and sell new things to an existing customer.  Moreover some customers actually cost a lot do deal with and yet generate little revenue; some may actually end up costing more to deal with than they bring in revenue, in which case it may be better to ease them out towards your competitors. Understanding which customers are actually worth dealing with is consequently important, yet few companies really have a good grasp of this.  Indeed, how many companies even survey their customers regularly and track how happy they are?

So far so good.  However the article makes some rather questionable assertions.  In particular it concludes that an enterprise IT strategy is “left out” when companies announce plans to be closer to the customer.  I don’t think this is really the issue.  It is not that the strategy is left out but rather than it turns out to be very difficult to execute.  After all, weren’t ERP systems supposed to create a “single business model” for the enterprise?  You didn’t find that?  Yet I could have sworn that was what those nice people in smart suits at PWC et al told people was the reason for spending all that money in the 1990s.  In that case, surely CRM was the answer?  I mean all that money on Siebel implementation surely must mean that companies now have a single view of the customer?  No?

The problem is only partially an IT one.  Sales people frequently resist “sales force automation” or, if they are forced into it, will do so under sufferance and so create entries that are, how shall we say this politely, of variable data quality.  This happens whether you are using Siebel, Salesforce or whatever system.  If the sales people don’t feel that they get a direct benefit then the system is just an overhead getting in the way of them making more sales calls.  Any data quality initiative would do well to start by examining the sales force automation system if you quickly want to find fertile ground for improvement.

Even if you overcome this issue, either by excellent discipline or somehow incenting the sales people to care about data quality (good luck with that), the next problem is that the sales force system does not control all the customer touch points.  Customers may well have lots of contact with the helpdesk. Yet how often is this seamlessly linked in with the salesforce system?  Similarly direct marketing campaigns to upsell have a real cost, yet how easy is it to assign these costs back to actual customers?  The costs of marketing, of sales and or support typically reside in entirely different systems in different departments, and few companies can consequently add all their costs up, even assuming that all these systems identify each customer in a consistent and unique way. 

How bad is the reality?  You only have to examine you own junk mail to get a fair idea.  I get four separate copies of every mailshot from Dell, a well run company, suggesting there are at least four places where my information is held, and that is just in their marketing systems.  Ever tried shifting your address when you move house and telling your various utilities and banks?  How quickly and smoothly did everything get redirected?

The article is right in pointing out that organisational issues are at the heart of what makes things difficult here, yet the survey of company respondents that suggested that “within three years they would be organised by customer” (rather than product, function or geography) seems to me entirely wishful thinking.  What does your organisation chart look like?  How realistic would it be organise a whole company by something as transient as “customer”?  If you think you get reorganised too often now, just imagine what this picture would be like.

In my view the difficulty of dealing efficiently with customers by assessing their lifetime value is an issue primarily of organisation and company power struggles, not one of technology.  For this reason I don’t expect it to be fixed any time soon.  If Dell stop sending me those multiple identical mailshots then I’ll let you know, but I am not holding my breath.