When I wrote the blog Time Series Forecasting in SAP Analytics Cloud Smart Predict in Detail, I mentioned that the accuracy of predictive forecasts is calculated by an indicator named Horizon Wide Mean Absolute Percentage Error or in short the HW-MAPE. The goal of this blog is to lift the veil on the following aspects of this indicator.

1. Why we have used this one?
2. Understand its definition
3. How is HW-MAPE calculated?

Understand the definition of HW-MAPE

When a user wants to estimate the revenue of his products, or how the stock will evolve, generally, it is not to have one prediction. It is to have estimations for each month of the next year, or the daily evolution for the next month. This is the reason an indicator is needed that considers the number of predictive forecasts requested in the future. HW-MAPE with its horizon dimension (the number of forecasts requested in the future), fills this condition.

Now, look at the definition of the HW-MAPE. The formula to obtain it is:

Where:

• signal is an actual value
• H is the horizon, or the number of forecasts requested by the user in the settings of the predictive scenario. It refers to the period in the future you want to predict.

What are the important parts of this formula? It is a sum of elements divided by the number of these elements. In fact, this formula is an average of a quality indicator calculated for the entire range of the horizon.

This quality indicator is the MAPE. It shows how much the forecast differs from the actual value. The formula to calculate the MAPE is:

Again, inspect this formula. It is a sum of terms divided by the number of these terms. Again, it is an average of distances of all predictive forecasts from the actual values.

How to interpret the HW-MAPE?

In this average of average, a kind of distance is calculated. The smaller this distance is, the nearer the predictive forecasts will be from the actual values. Thus, we can say that, the smaller the HW-MAPE is, the better the accuracy of the predictive forecasts.

How is HW-MAPE calculated?

When the user requests forecasts, let’s say for the next twelve months, he needs an indicator that tells him how correct the predictive forecasts are. To do that, we need to compare the predictions with the real values. The problem is that for the predicted events in the future, real values are not still known, and the comparison is not possible. Consequently, it is not possible to get a real HW-MAPE. The following table illustrates this situation.

Table 1: Actuals and forecasts at one level. Comparison is possible only for the past

To remedy to this, the idea is to calculate an expected MAPE. The goal is to optimally capture the performance of the model, and to give an estimate of the expected performance on new data. To do that, we generate the same number of predictions requested by the user also for historical values. If the behavior of data will not change between the past and the future, we will have all the information to compute our quality indicator. The table 1 above becomes like table 2 below with our example of a horizon of twelve months.

Table 2: Actuals and forecasts on all the range of the horizon

HW-MAPE has these goals:

• Capture the performance of the predictive model with multi-horizon.
• Multiply the measure of the performance on several points of validation. This is to avoid a bias (tendency for the predictive forecasts to be consistently higher or lower than the actual value), due to an arbitrary choice of cut between past and future.

When Smart Predict is creating prediction models, it also creates an internal table containing the predictions for all the range of the horizon and for all actual values of the validation dataset. It is from this internal table that all the steps of the calculation of the HW_MAPE are done. Now let’s see what the steps are to compute this indicator.

Open the hood

Smart Predict uses the validation dataset to estimate the HW-MAPE. The historical dataset is considered in the chronological order by the prediction engine and is split into two parts:

1. The first 75% of the data is used to train the predictive model.
2. The remaining 25% is used to validate the predictive model. Note that once the predictive model is selected, it is refitted on the whole period, including validation. The refit process does not rebuild the predictive model, but adjust its parameters.

Fig 1: Validation dataset versus training dataset

Three steps are necessary to compute the HW-MAPE:

1. Compute the Absolute Percentage Errors (APEs) by comparing predictive forecasts and actual.
2. Compute the Mean Absolute Percentage Error (MAPE) for different horizons.
   MAPE (for a given horizon) = mean (APEs for a given horizon)
3. Compute the Horizon-Wide Mean Absolute Percentage Error (HW-MAPE).
   HW-MAPE = mean (MAPEs for all horizons)

Now, let see these steps in detail. For this, I use an example of 60 events of historical data. These events are the monthly sales of cars from January 2015 to December 2019. The goal is to get predictions of sales for the next twelve months.

Thus, the training dataset contains the first 45 rows while the validation dataset contains last 15 rows.

Compute the Absolute Percentage Errors (APEs)

It is necessary during the calculation of the APEs to go up to a maximum number of rows equal to the horizon. This is needed to compute all the APEs. This will become clearer with the example below.

The APE (Absolute Percentage Error) is the absolute value of the difference between the predicted value for a given horizon and the actual value divided by the actual value.

APE = ABSOLUTE ((Forecast – Actual)/Actual)

Let’s see that from the internally computed table: The first column is the date of each event. The second column SALES is the actual value of the target to predict. The columns Forecast_h_i (for i = 1 to 12) are the predictions of the target for the future.

For example, here is how to read the first row of data. On 01/07/2017, the actual value of SALES is 65783 while the predictive model predicts for this month Forecast_h_1 = 68293.16426. As the unit of prediction is the month, for next month, the prediction is Forecast_h_2 = 69030.07359, and so on until Forecast_h_12 as I request predictions with a horizon of 12 months.

Fig 2: Actuals and forecasts on all the range of the horizon

From this table, I can compare predictions with actuals and get the error of the predictions. This is the Absolute Percentage Error. The table below gives all the APEs for the entire range of the horizon (at the end of the document, there is the Excel file with the formula).

Fig 3: Absolute Percentage Errors for all the horizon

Let’s see how the APEs of each column are computed.

For column APE_h_1, APEs are calculated by taking the cells shown below (for one calculation, consider rectangles of the same color).

For example,

APE(01/10/2018, h=1) 
      = ABSOLUTE((90860.36692 – 79628) / 79628) = 0.1410605

Fig 4: Which values to compute APE for the first horizon

For column APE_h_2, APEs are calculated by taking the cells shown below (for one calculus, consider rectangles of the same color).

For example,

APE(01/10/2018, h=2)
      = ABSOLUTE((90860.36692 – 79628) / 79628) = 0.141060518

Fig 5: Which values to compute APE for the second horizon

Let’s finish with an example related to column APE_h_3.

For example,

APE(01/10/2018, h=3) 
      = ABSOLUTE((90860.36692 – 79628) / 79628) = 0.141060518

Fig 6: Which values to compute APE for the third horizon

This process goes on until column APE_h_12.

Compute the Mean Absolute Percentage Error (MAPEs)

There is not only one MAPE, but one per range of the horizon. The MAPE for a given horizon is the mean of all the APEs.

MAPE (horizon) = AVERAGE(APE_i) for i=1 to horizon

For example, in the table below,

MAPE(h=1) = AVERAGE(APE of column APE_h_1) = 0.1042796

Fig 7: MAPEs for the 12 horizons.

Compute the Horizon-Wide Mean Absolute Percentage Error (HW-MAPE).

HW_MAPE is simply the mean of all the previously calculated MAPEs. In our example, it is the sum of the MAPEs divided by the horizon.

HW-MAPE (horizon) = SUM(MAPEi)/horizon for i = 1 to horizon

In our example

HW-MAPE(12) = ( 0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 +
                0.104279594 ) / 12 = 0.1042796

In percentage this gives: 10.43%

Note: all MAPEs are here the same because in the predictive model there is no auto-regressive component.

If we create a time series model in SAP Analytics Cloud, we get the same HW-MAPE expressed in percentage: 10.43%

Fig 8: HW-MAPE calculated by Smart Predict

The Excel files available in GitHub are the training dataset if you want to test it yourself on Smart Predict and a sheet that contains all the calculations shown above.

Conclusion

At the end of this blog, I hope that the veil is lifted on why we have this precision indicator as well as on the way it is calculated. The product focuses on what is important for the user and automates more of the process to get insights, to get forecasts inside your planning models and to justify them.

I hope also these explanations increase your confidence in the product. If you appreciated reading this blog, I would be grateful if you’d like it. You might want to comment it as well. Thank you.

Resources

• Time Series Forecasting in SAP Analytics Cloud Smart Predict in Detail
• SAC Smart Predict – What goes on under the hood
• Mean absolute percentage error
• The accuracy of extrapolation (time series) methods: Results of a forecasting competition

Files that will be available for download

In GitHub (https://github.com/antoinechabert/predictive), you will find two Excel files:

1. The dataset I use in this blog: France-data.xlsx
2. A worksheet with all the steps of the computation of the HW-MAPE (with the Excel formula included): France-MAPE.xlsx