Cost Benefit Newsletter No. 69, ´ROI That Never Arrives: The Devil is in the Assumptions´

ROI estimates in business fail primarily because managers give too much attention to the "pay out" odds, and too little attention to measuring and managing "probability" odds. A good risk and sensitivity analysis of the assumptions behind the predictions allows you to do both.

Somewhere between 70 and 80% of all major IT project proposals in the industrialized world now require a return on investment (ROI) analysis before funding. The figure varies, depending on which analyst you read, but there's no question that the "ROI Required" rate is high and going higher (See Newsletter 28, "All Eyes on ROI").

The same analysts then turn right around and tell us that more than half of such projects never deliver the expected ROI. Several years ago, for example, Nucleus Research published a now-famous study after interviewing SAP reference customers. Nucleus found that 57% of them did not receive a positive ROI. Yet SAP had for years made return on investment (or, "return on information") a centerpiece of its marketing and sales strategy.

"Why then," you might ask, "do decision makers continue to focus on ROI when so many ROI figures don't predict what actually happens? If ROI can't do any better than that, doesn't ROI cause more bad decisions than it prevents?

Two Kinds of Odds

The term "odds" means two different things to gamblers. Serious gamblers know that one kind of odds has to be weighed against the other kind, in order to maximize the chances of coming out ahead.

At the racetrack, "odds" can mean (1) the ratio of the winning pay out to the cost of the bet (just like simple ROI !). A winning $2 bet on a 20-to-1 long shot puts $40 in your pocket. The skill in betting lies in knowing how the pay out odds compare to the other kind of odds, (2) the actual probability of winning.

One glance at the tote board shows you the current pay out odds. (One glance at the bottom line of your ROI spreadsheet shows the predicted returns on your business investment, if everything goes as planned).

However, if you do go to the races, just watch the serious gamblers working, furiously, down to the last minute before the betting windows close, trying to estimate the probability odds. They visit the paddock area to have a look at the horse, they pour over racing forms and tip sheets, and they work and re-work their racing calculators. Otherwise, they will tell you, it would all be "luck" and nothing more.

If the gambler estimates that a horse has a one-in-five chance of winning (20% probability of winning), but the posted pay out odds are 20-to-1 for a win, that is a "good bet." (in statistical terms). If only business people went through the same thought process.

I believe that ROI estimates in business are often unlucky because managers give too much attention to the pay out odds and too little attention to the probability odds.

Very few know how to how to maximize ROI by measuring and managing uncertainty.

Assumptions, Assumptions, Assumptions

Every forward-looking ROI estimate and business case stands on a foundation of assumptions. It can be no other way, after all, because the case predicts the future. Everything about the future is an assumption. A detailed case for a major business investment may require dozens or hundreds of assumptions about such things as future:

Business volume
Competitor's actions
Market share
Fuel prices
Government regulation
Software development time
Salary increases
Productivity improvements
Implementation costs
"Ramp up" time
Labor requirements
Raw materials prices

Some assumptions come with high certainty, others come with high uncertainty. But every assumption adds some uncertainty to the bottom line cash flow or ROI estimate. It's tempting to lump all these factors together into a single risk figure.

The key to managing your investment so as to minimize risk and maximize returns, is not to treat risk as a single factor, but instead carefully weigh individual risk factors to determine:

Which assumptions carry the most weight in driving results?
What happens to results if we don't manage critical success factors to target levels?
Sensitivity analysis addresses questions like these.
What is the likelihood of other results besides the most likely predicted outcome?
Which risk factors have to be watched carefully, as indicators that predictions need revising?
Risk analysis answers questions like these.

One of our clients, for instance, had a proposal to begin a major ERP system implementation. This bottom line prediction was an incremental cash flow gain of $19.5 million over five years. That was a good looking "R" on an "I" of about $3 million.

Here, however, is part of what turned up in a sensitivity analysis of individual assumptions:

Target Forecast
The table shows the correlation between each assumption and the overall projected results, based on Monte-Carlo simulation with the same financial model that predicted a $19.5 million gain.

There were some messages here that made management sit up and take notice.

The ROI model assumed the ERP system would help lower new product development time by 30%. The high correlation between this assumption and financial results shows how the attractive ROI depends on reaching that target. In this case, a reduction of only 15% in development time (instead of 30%) brought the predicted gains down to $5 million.
The ROI model also assumed the ERP system would be fully implemented in 2 years (50% implementation per year). Clearly, missing the 2 year target would adversely impact the overall return on this investment. Stretching the implementation out to 4 years would raise costs and lower benefits to bring a zero net gain.

It's one thing to know generally that factors like implementation time and reduced development time are important. It's another thing entirely to bring home the importance of those factors in concrete terms.

If you know ahead of time where the risks are, you can manage them (or at least watch them) and avoid unpleasant surprises down the road.

Marty Schmidt

To learn and practice the ROI analysis steps above, we invite you to consider attending a "Building the Business Case" seminar in your own area.

The figure above is one of several Monte Carlo output screens from Crystal Ball, a risk, sensitivity and forecasting tool from Decisioneering (www.crystalball.com). Crystal Ball ® is a registered trademark of Decisioneering, Inc.