Concur - PO: Integration centric process for Standard Accounting Extract (SAE) posting

[engswee]

Update 22 April 2015: Correct statement to indicate that OAuth 2.0 was not available during the project timeline but is now available.

Introduction

Everyone knows that cloud applications are the hot topic these days. In the area of business travel, Concur is the market leader for cloud based travel & expense solutions. It is no wonder that SAP acquired Concur recently in SAP's march towards cloud era.

In the area of integration, Concur provides a rich and mature set of RESTful APIs. The online API documentation is very extensive, and it also provides a web-based UI to test those APIs without any external tool or creating custom programs.

One of the key areas of integration for Concur is the generation of the Standard Accounting Extract (SAE). Once the business expenses are approved on Concur, an extract of the approved expenses can be generated to be fed into a payment system. In SAP's case, this would typically be integrated into the Finance module on ECC as GL/AP postings. At this point in time, there is no prepackaged PI/PO content for integration with Concur. Hopefully with the acquisition, SAP will deliver prepackaged contents similar to those available for SuccessFactors and Ariba.

In this blog, I will share my experience for the integration centric process developed on PO's BPM to extract the SAE file from Concur. Consumption of Concur's RESTful API is achieved via Advancto's REST adapter as it requires OAuth 2.0 authentication method, which is not yet available on SAP's own REST adapter which was not yet available on SAP's own REST adapter during the project timeline (it has recently been announced in PO 7.4 SP10 release). The focus of the blog will be on the high level overview of the solution and will not go into too much details of each step of the implementation.

Standard Accounting Extract Process Flow

In order to extract the SAE file from Concur, 4 different APIs need to be executed in sequence. As such, it cannot be achieved by a single stateless integration scenario in PI. It requires a stateful process which can theoretically be achieved by an ABAP program, however it is well suited to be modeled in BPM as there are no business logic involved.

Following are the sequence of steps required to extract the SAE file via Concur's Extract Web Service:

1. Get Extract Definition
2. Post Extract Job Initiation Request
3. Get Extract Job Status - Repeat until Status = Completed
4. Get Extract File

Note: Step 1 can be skipped if the Extract Definition is always the same, therefore the same Definition ID is always used for Step 2.

Challenges

Whilst the sequence of steps is relatively straightforward to implement in BPM (sequentially go through each step), there are various challenges in order to achieve a robust solution.

1. Retry mechanism

Step 3 has to be repeated indefinitely until the status returned by the synchronous call is Completed. Depending on the load and resource availability of the Concur system, this could be as fast as 1 minute or as long as 3-4 hours! As such, a multi-tiered retry interval was implemented so that the step is not repeated every minute and therefore unnecessarily consuming resources on the PO system.

2. BPM Process Starting Trigger

In addition to the RESTful APIs listed above, I added an additional call to the ReportDigests API as the first step that triggers the start of the BPM. This API checks if there are any approved expenses in Concur. If there are no approved expenses, the BPM is terminated normally without further execution of the Extract APIs.

This first step is triggered by a sender REST communication channel in polling mode with a polling frequency of every hour.

3. Exception Handling for Step 2 (Post Extract Job Initiation)

Once an extract job for a particular definition has already been submitted to the Concur system, it will be queued in Concur while waiting for available resources. Concur does not allow more than one job for the same definition to be queued at the same time. Therefore if there is a long running job already submitted, a subsequent call of Step 2 will return an application error. A boundary event is defined for the fault message associated to this step to catch the application error. If there is an error, the BPM will be routed to the exception path to trigger an email notification and terminate normally.

4. Optimize BPM process context by reducing payload size

After completion of Step 4 to retrieve the generated extract file, the response of the web service call will be returned to the BPM process. It is possible that the response contains a large payload and thus increases the memory usage of the BPM process context. One approach to avoid loading the large response payload into the BPM process is by using the Claim Check Integration Pattern as described in the blog below. However, my system is on a lower SP so it was not available.

Claim Check enhancements in SAP Process Orchestration

An alternative approach was used whereby the response payload was zipped and encoded into a Base64 string. The Base64 content is stored in the following field in an XML payload. It is this XML payload that is returned back to the BPM process. This conversion is performed via a Java Mapping on the response flow of Step 4.

With this, the size of the payload loaded into the BPM was about 10% of the original payload size - a 90% reduction!

5. Multiple files from Extraction job

Depending on the extract definition configured in Concur, it is possible to have multiple files as the output of the extraction job. If there are multiple files, the response payload of Step 4 will be a Zip file containing multiple pipe delimited files.

In order to handle multiple files in the extraction, a custom Java mapping was developed to unzip the file and create additional attachments from each file in the Zip archive. The following document covers that solution.

Java Mapping: Unzip file and create additional attachments

References

As this was my first development on NW BPM compared to previous ccBPM developments on PI dual stack system, the following SCN articles came in very handy to shorten the learning curve for developing in NW BPM as well as implementing advanced error handling.

NetWeaver BPM for System-to-System Message Orchestration (Including Migration of ccBPM to NW BPM)

Integration Centric Process - Advanced Exception Handling in BPM

Appendix

Below is the full diagram of the BPM process flow.