How to Forecast using Custom External Algorithms
I am a Product Manager for SAP Integrated Business Planning for Supply Chain at SAP and in this blog post I would like to share some insights on how to use your own custom algorithms within the forecasting framework of SAP IBP for demand.
What does forecasting with custom external algorithms mean?
It means that you can boost the already existing forecasting capabilities of SAP IBP for demand by using custom-built forecasting algorithms in a tightly integrated process within IBP.
SAP IBP takes care of the pre processing (including reading and aggregation of relevant key figure data, applying Product Lifecycle settings in case of new product introduction) and post processing (including evaluation in a best fit model, applying Product Lifecycle settings and disaggregation), while the custom algorithm creates the forecast.
This is done in a synchronous process, meaning that the external algorithm is called only if and when you trigger a forecast in SAP IBP, either through interactive forecasting in Excel or via a forecast job.
The data exchange is done through an OData Service, and the external algorithm can be written in R, Python or any other suitable tools.
Here is a simplified idea of how the process works.
Why should you use custom external forecasting?
- You have complete freedom in the choice of the forecasting algorithm: Use cases include additional algorithms not available in the standard solution, customized implementation of already existing algorithms and use of additional data not present in the SAP IBP system.
- The custom algorithm is tightly integrated within the forecasting process of SAP IBP: You do not need to take care of data (dis)aggregation or of applying product lifecycle settings, SAP IBP will do it for you!
- Same look and feel of SAP IBP native algorithms: You can add your algorithm to a forecast model simply by selecting it in the list (see step 4 below). The model can then be used in interactive forecasting as well as within forecasting jobs.
- Choose the best algorithm: You can use your custom algorithm together with existing SAP IBP algorithms in a “Best Fit” model and let the system evaluate which one performs best.
What is the difference of this approach compared to already existing solutions?
Before this tightly integrated process was available, you already had the opportunity to use external forecasting algorithms to compute your forecast. This meant downloading the data from the SAP IBP System, computing the forecast wherever and however you wanted, and uploading back the results to the system.
This approach however has a few drawbacks:
- No Product Lifecycle settings: When the data are downloaded to be used as an input for the forecasting algorithm, no Product Lifecycle settings are applied. For example, if a product you want to forecast has a reference product, you would have to manually copy the history of the reference product in order to get a reliable forecast.
- No “Best fit”: If you wanted to know how the external algorithm performed in comparison to other algorithms, you would have to compute the forecast error for all of these and then compare them in an additional step to find out which is the best algorithm for the combination you are interested in.
- No disaggregation: After you have computed the forecast with the external algorithm, you can upload the results to the system, but no automatic disaggregation to lower levels will be done during the upload.
- No interactive forecasting: if you are working in the SAP IBP add-in for Microsoft Excel and want to see right then and there how the forecast of a product looks like with your external algorithm… Well, you can’t. You would have to wait for the asynchronous process of downloading-computing-uploading to be done in order to see the results.
In case these drawbacks are not critical for your use case, you may not use this new tightly integrated process described in this document but use the side-by-side approach instead.
How to setup a custom external algorithm – step by step
Step 1 – Create your custom external algorithm
This can be anything you want. You can use freely available algorithms (for example Prophet) or just implement your own algorithm.
In order to help with the first setup we have created some sample code that you can use to jump-start. You can find it in SAP Note 3170544.
The sample code also includes the code needed for the OData Service to work (see step 2).
Step 2 – Understand what the OData Service API looks like
In order for your external environment (which can be Private Cloud, Public Cloud or On Premise) to be able to communicate with the SAP IBP system, you need to set up an HTTP server in the external environment to get notifications from IBP and call the OData service IBP_DEMAND_EXT_FCS_SRV to exchange data with IBP.
Essentially, this is what happens when the service is used:
- Using the SAP_COM_0830_0001_REST outbound OData service, SAP IBP sends a notification to an HTTP server or service in the external system to notify it that an external forecast calculation request needs to be processed. This notification contains a request ID.
- Using the IBP_DEMAND_EXT_FCS_SRV inbound OData service, the external system extracts the required input data from SAP IBP. This input data consists of the following:
- Parameters that were specified for the external algorithm in the forecast model
- IDs that uniquely identify planning objects within the request
- Attribute values that describe the planning objects on the forecast aggregation level
- Time-series values for the key figures and time horizons defined in the forecast model
- The external forecasting system reads the input data, calculates the forecast, and sends it to SAP IBP using the same OData service.
- SAP IBP reads the incoming results and leverages them the same way as the forecasts calculated by internal algorithms.
For more information see the official documentation on the OData API for External Forecasting and the documentation in the SAP API Hub.
Step 3 – Create a communication scenario
To access the SAP IBP system from outside the SAP IBP cloud and use OData services, you need to create and inbound and an outbound communication user in the Maintain Communication Users app and associate them with the SAP_COM_0830 communication scenario.
The authentication methods applied by SAP IBP and the external system can be certificate-based or basic authentication, depending on the methods you implemented in the external tool.
If you want to set up certificate-based authentication, make sure that the client certificate you use is signed by a certification authority accepted for inbound integration. For more information, see SAP Note 2871840 .
To find out how to set up certificate-based authentication go to the official documentation on Integrating External Forecasting Algorithms.
You can set up integration with the external system as follows:
- Navigate to the Communication Arrangements app and choose New.
- Select the communication scenario SAP_COM_0830 and enter a name for your communication arrangement.
- Choose New next to the Communication System field. Enter an ID and name of your choice and choose Create.The Communication Systems app opens, showing the screen for creating new communication systems with the system ID and name you entered in the Communication Arrangements app.
- For the host name, enter the IP address or DNS name of the external system.
- Under Users for Inbound Communication, add the communication user you have previously created for inbound communication, and select the authentication method you specified for it in the external system.
- Under Users for Outbound Communication, add the communication user you have previously created for outbound communication, and choose the authentication method you specified for it in the external system.
- Choose Save. You are now navigated back to the Communication Arrangement app.
The URL shown under Inbound Services can be used to call up the OData service that allows SAP IBP to read the request for input data send the input data to the external system, and read the results calculated by the external algorithm. Make sure this URL is included in the code you implement in the external system.
The URL shown under Outbound Services is the one to which SAP IBP sends the notification request that a forecast needs to be calculated based on specific parameters. This URL includes the host name you defined for the communication system.
Step 4 – Create a forecast model with your custom external algorithm
Go to the Manage Forecast Models app and click on Create.
Choose whatever pre-processing steps are needed, then go to the Forecasting Steps tab and in the Algorithms section click on +.
Choose External Algorithm from the list of available algorithms.
You can specify the following settings for each external algorithm:
- Algorithm Name: Choose a name that helps you identify the algorithm in the external system.
- Algorithm Parameters (optional): Specify the parameters that the external algorithm should consider during forecasting. For example, you can choose the error measure that the external algorithm should use for parameter optimization. For each parameter, you need to specify a value.
- Independent Variables (optional): Specify the independent variables that you expect to have a significant effect on the forecast calculated by the external algorithm. For each variable, you need to choose a time horizon. For example, if you know that the future values of an independent variable are not required for a training-only call, choose the past only. This will result in better performance.
- Target Key Figure for Forecast (optional): The key figure in which you want to save the forecast calculated by the external algorithm. Defining a target key figure is not mandatory since the final forecast results are saved in the main target key figure of the forecast model.
- Target Key Figure for Ex-Post Forecast (optional): The key figure in which you want to save the ex-post forecast calculated by the external algorithm. The ex-post forecast is calculated for past periods for which actual demand history is also available.
Step 5 – Use the custom external algorithm to compute your forecast
As already mentioned, you can now use the newly created forecast model exactly as you would use any other forecast model: You can call it through interactive forecasting or run the forecast in a scheduled job in SAP Fiori.
If you want to know more, have a look at our documentation:
And, last but not least
Have a look at our highlight video
I hope this helps you in setting up your own custom external algorithm.
If so, feel free to share this with anyone that might be interested as well.
In case you have any comments or questions do not hesitate to leave them here below!
Very detailed explanation!! Thanks Laura
Hello Laura Tozzo,
Nice Article and Write up!! Thanks for the detailed info.
Is it possible to connect to SAP BTP system for external forecasting ?
yes it is! We are working on some Best Practices documentation to explain step-by-step how to set it up, currently planned for 2208.
Thanks Laura Tozzo!!
Hello Dr Laura,
Thanks for the Blog and nice documentation in the Discovery Center on the same topic. I am kind of implementing this functionality in our demo environment. What external system can be used to connect with SAP IBP Demand planning- external algortithm. Is this only SAP Data Intelligent tool or external system can be non-SAP system as well.
the external system can be any non-SAP system as well, as explained in the blog. The documentation in the Discovery Center concentrates on the use case with SAP Data Intelligence, but it is not the only option.