The analogue trap in forecasting

Gary Johnson

Inpharmation

One of the most popular approaches to forecasting is to search for a product that looks very similar to your product and then assume that your product’s performance will be similar to this so-called ‘analogue’.

Analogues are very popular because they make great stories: “Our product is a lot like product X. This is how product X performed. So, we expect similar uptake and sales for our product.”

Unfortunately, using analogues often produces terrible forecasts. In this article, I will explain the two traps that lure us into the use of narrow analogues for forecasting and explain some of the reasons why they can produce such bad forecasts.

"Analogues are very popular because they make great stories…"

Trap 1: Obvious similarities blind us to not-so-obvious, but more important differences

Investors in the construction of the Panama Canal lost a fortune because they used an obvious analogue: the Suez Canal. The similarities were clear: both were shipping canals across narrow isthmuses of land that would shorten long and hazardous sea journeys. Emboldened by their analogue forecast, they went ahead – and lost a fortune. The problem was that blinded by the obvious similarities, the investors failed to identify the even more important differences. In particular, the Suez Canal cut across a flat desert whilst the Panama Canal would have to cut across mountainous, mosquito-infested jungle.

Exactly the same thing happens in pharmaceutical forecasting: managers think of “obvious” analogues and fail to see fundamental differences that render the analogues inappropriate for forecasting.

For example, I was recently working on the forecast for fixed-dose combination product. My client sent me a list of products that she thought might be used as “analogues”. The list was a role-call of all the most successful fixed-dose combination products ever. Why? Because it is the highly successful products that we tend to know about.

"Investors in the construction of the Panama Canal lost a fortune because they used an obvious analogue: the Suez Canal."

But, when we are forecasting, we have to be far more systematic than this. If we look across a large selection of fixed dose combinations, we see that sometimes they are very successful and sometimes they are incredibly unsuccessful, selling almost nothing at all. The key forecasting question is: have we got the sort of fixed dose combination that will succeed or have we got the sort that will sink without trace? To answer this, we have to look at both the great and the small products and work out what it is that the great combination products do have that the small combination products do not have. (It turns out to be a number of factors – such as are the products used to treat the same disease?)

But even if we could find close analogues without hidden important differences, analogues would still not be the best way to forecast. The reason for this lies in the second analogue trap…

Trap 2: Rejecting the information that non-analogous products provide

Imagine that you are to be visited by your 9-year-old nephew. You have never seen your nephew because he lives in a far away hot country. Because it is cold where you live, you decide you want to buy him a coat. You don’t know how tall he is and so you need a forecast of how high a 9-year-old child is likely to be. There are a number of children living in your street and so you ask them all for their age and their height. You plot the children’s ages and heights on a chart. Knowing a little about statistics, you fit a “best-fit line” through the data points. You end up with a chart that looks like this…

In effect you have now built a forecasting “model”. You have found a pattern in the relationship between age and height and you can use this pattern to predict the height of your nephew.

So, you now have two options for forecasting the height of your nephew: you can use the “model” that you have built and read the expected height of your nephew off the trend line, or you can use the analogue approach and select the height of the child that most closely matches the one thing you know about your nephew – that he is nine.

If you used the trend-line, you would estimate that your nephew will be 1.3 metres tall. If you used the analogue, you would estimate that he would be a little below 1.2 metres tall.

Which approach do you think is most likely to produce the best forecast?

Common sense tells you that the trend line is more likely to provide an accurate forecast than the height of the one nine year old in your “sample”. And your common sense would be correct. The one nine-year old is obviously an “outlier”. If you had been less industrious and had just jumped to the obvious analogue, you would not know that this child was an outlier.

Notice that, counter-intuitively, inclusion of non-analogues increases forecast accuracy over using analogues alone.

"Counter-intuitively, inclusion of non-analogues increases forecast accuracy over using analogues alone."

And what if your nephew was aged 11? An analogue-forecaster would conclude that they could not forecast their nephew’s height because there was no 11-year-old living in the street. A model-forecaster would have no problems.

People fall into this analogue-trap all the time. We were recently working on a forecast for a new class of agent in Alzheimer’s. We needed to predict the price at which this new class of agent would gain reimbursement around the world. To do this, we use a huge database of actual payer decisions across a vast number of therapy areas to find the important predictable patterns in payers’ behaviour. For example, drug classes that treat larger numbers of patients tend to get lower prices than drug classes that treat smaller numbers of patients.

The client had enormous difficulty understanding that using the price experience of the one existing class of Alzheimer drugs (the cholinesterase inhibitors) could not be the basis of a price forecast. This would be akin to trying to predict a 9-year-old’s height based on one example of a 9-year-old.

So, in summary, everyone loves a good story and analogues make great stories. However, analogues often make terrible forecasts.

About the author:

Gary Johnson is author of Sales Forecasting for Pharmaceuticals: An Evidence Based Approach. His company provides modelling software and consultancy to all the world’s leading pharma companies. Gary has been short listed for the MCA Business Book of the Year Awards and has won numerous best paper and speaking awards.

Email: gary@inpharmation.co.uk web : www.inpharmation.co.uk

Are your forecasts resting on dangerous analogues?