The Best Fit Forecasting Conundrum
With snowballing attention on Automation and Artificial Intelligence in the tech sector, clients today are bound to expect their supply chain forecasting system to be highly intelligent and generate decidedly accurate forecasts, with little or no human intervention. All tools in the market, from SAP’s own APO to the latest IBP, have some form of auto-pilot mode which will do all the work for you- from selecting the ideal statistical model to coming up with the best set of input parameters.
However, one needs to be very careful in letting their forecasting system run autonomously. A forecasting system is like a formula one car: it needs a lot of constant fine-tuning and still requires a skilled driver to steer the sophisticated piece of machinery. For a purely quantitative analysis, one may tend to rely on a forecast error measure like Mean Absolute Percentage Error(MAPE) to choose best-fit forecast method without realizing that MAPEs with ex-post forecast can be really skewed at times.
One of the prime examples of the need for caution with best fit forecasting is the use of Croston’s method. Croston’s method is supposedly beneficial for demand streams that are sporadic. Now in the absence of an ability to recognize such demand patterns in IBP, if we rely on a ‘choose best Forecast’ based on best MAPE, the following surprising results come up.
To start with, if we do an analysis of what the best Alpha is for use with Croston’s method, a quantitative investigation with 0.1, 0.3, 0.6, 0.9 gives unrealistically good results in favor of 0.9, with <5% MAPE. This can be attributed to the calculation formula of Croston’s Method as per the SAP HANA Predictive Analytics Library:
Where, V(t) is Actuals Qty, Z(t) is estimate of the Demand Value, X(t) is estimate of the intervals between demand.
With this formula and Alpha = 0.9, the Ex-Post Forecast matches demand very very closely leading to a minuscule MAPE of 2.45%.
Even SAP’s illustrative example on help.sap.com for Croston’s shows this behavior of Ex-Post forecast very closely matching Actuals Qty:
So is this the best forecast possible for the product in question? Let’s keep that discussion for another post on Croston’d method. However, what is interesting to note is how Croston’s performs with a product with stable demand- if you run the following demand pattern with both Croston’s (alpha = 0.9) and Single Exponential Smoothing (alpha 0.9), we see that they come up with the exact same forecast:
The most stimulating observation here is that, although the forecast is the same, the ex-post differs greatly: Croston’s (based on the formula above) gives a closely matched ex-post forecast with MAPE of 6%, while Single Exponential Smoothing gives a MAPE of 61%.
This clearly suggests that a layman cannot use best-fit forecasting and purely rely on quantitative analysis. It further necessitates the need of better demand pattern recognition in IBP. (SAP has something like that in SPP- Service Parts Planning). Fortunately, SAP does have a Time Series Analysis app in the IBP product roadmap so soon we would be able to control forecast outputs and better align to demand patterns. Until then, we just have to be careful in turning on the auto-pilot in IBP statistical forecasting. The role of a good ‘Consultant’ is not going away anytime soon!
Shivaditya is a Senior Manager in the Supply Chain Planning and Analytics practice at IBM India Pvt Ltd.
SAP Predictive Analytics Library:
SAP Help documentation on Croston’s Method: