Concur – PO: Integration centric process for Standard Accounting Extract (SAE) posting
Update 22 April 2015: Correct statement to indicate that OAuth 2.0 was not available during the project timeline but is now available.
Introduction
Everyone knows that cloud applications are the hot topic these days. In the area of business travel, Concur is the market leader for cloud based travel & expense solutions. It is no wonder that SAP acquired Concur recently in SAP’s march towards cloud era.
In the area of integration, Concur provides a rich and mature set of RESTful APIs. The online API documentation is very extensive, and it also provides a web-based UI to test those APIs without any external tool or creating custom programs.
One of the key areas of integration for Concur is the generation of the Standard Accounting Extract (SAE). Once the business expenses are approved on Concur, an extract of the approved expenses can be generated to be fed into a payment system. In SAP’s case, this would typically be integrated into the Finance module on ECC as GL/AP postings. At this point in time, there is no prepackaged PI/PO content for integration with Concur. Hopefully with the acquisition, SAP will deliver prepackaged contents similar to those available for SuccessFactors and Ariba.
In this blog, I will share my experience for the integration centric process developed on PO’s BPM to extract the SAE file from Concur. Consumption of Concur’s RESTful API is achieved via Advancto’s REST adapter as it requires OAuth 2.0 authentication method, which is not yet available on SAP’s own REST adapter which was not yet available on SAP’s own REST adapter during the project timeline (it has recently been announced in PO 7.4 SP10 release). The focus of the blog will be on the high level overview of the solution and will not go into too much details of each step of the implementation.
Standard Accounting Extract Process Flow
In order to extract the SAE file from Concur, 4 different APIs need to be executed in sequence. As such, it cannot be achieved by a single stateless integration scenario in PI. It requires a stateful process which can theoretically be achieved by an ABAP program, however it is well suited to be modeled in BPM as there are no business logic involved.
Following are the sequence of steps required to extract the SAE file via Concur’s Extract Web Service:
- Get Extract Definition
- Post Extract Job Initiation Request
- Get Extract Job Status – Repeat until Status = Completed
- Get Extract File
Note: Step 1 can be skipped if the Extract Definition is always the same, therefore the same Definition ID is always used for Step 2.
Challenges
Whilst the sequence of steps is relatively straightforward to implement in BPM (sequentially go through each step), there are various challenges in order to achieve a robust solution.
1. Retry mechanism
Step 3 has to be repeated indefinitely until the status returned by the synchronous call is Completed. Depending on the load and resource availability of the Concur system, this could be as fast as 1 minute or as long as 3-4 hours! As such, a multi-tiered retry interval was implemented so that the step is not repeated every minute and therefore unnecessarily consuming resources on the PO system.
2. BPM Process Starting Trigger
In addition to the RESTful APIs listed above, I added an additional call to the ReportDigests API as the first step that triggers the start of the BPM. This API checks if there are any approved expenses in Concur. If there are no approved expenses, the BPM is terminated normally without further execution of the Extract APIs.
This first step is triggered by a sender REST communication channel in polling mode with a polling frequency of every hour.
3. Exception Handling for Step 2 (Post Extract Job Initiation)
Once an extract job for a particular definition has already been submitted to the Concur system, it will be queued in Concur while waiting for available resources. Concur does not allow more than one job for the same definition to be queued at the same time. Therefore if there is a long running job already submitted, a subsequent call of Step 2 will return an application error. A boundary event is defined for the fault message associated to this step to catch the application error. If there is an error, the BPM will be routed to the exception path to trigger an email notification and terminate normally.
4. Optimize BPM process context by reducing payload size
After completion of Step 4 to retrieve the generated extract file, the response of the web service call will be returned to the BPM process. It is possible that the response contains a large payload and thus increases the memory usage of the BPM process context. One approach to avoid loading the large response payload into the BPM process is by using the Claim Check Integration Pattern as described in the blog below. However, my system is on a lower SP so it was not available.
Claim Check enhancements in SAP Process Orchestration
An alternative approach was used whereby the response payload was zipped and encoded into a Base64 string. The Base64 content is stored in the following field in an XML payload. It is this XML payload that is returned back to the BPM process. This conversion is performed via a Java Mapping on the response flow of Step 4.
With this, the size of the payload loaded into the BPM was about 10% of the original payload size – a 90% reduction!
5. Multiple files from Extraction job
Depending on the extract definition configured in Concur, it is possible to have multiple files as the output of the extraction job. If there are multiple files, the response payload of Step 4 will be a Zip file containing multiple pipe delimited files.
In order to handle multiple files in the extraction, a custom Java mapping was developed to unzip the file and create additional attachments from each file in the Zip archive. The following document covers that solution.
Java Mapping: Unzip file and create additional attachments
References
As this was my first development on NW BPM compared to previous ccBPM developments on PI dual stack system, the following SCN articles came in very handy to shorten the learning curve for developing in NW BPM as well as implementing advanced error handling.
NetWeaver BPM for System-to-System Message Orchestration (Including Migration of ccBPM to NW BPM)
Integration Centric Process – Advanced Exception Handling in BPM
Appendix
Below is the full diagram of the BPM process flow.
Hi Eng,
Thanks for the blog.
We are in the process of building the SAE process flow.
We stuck at the last step 4 where the SAE file which we get as a response from the concur extract file API, that our requirement is to write to file server.
We used java mapping in response mapping but we are not getting the response from REST adapter to the mapping program and output of step 4 is coming as null.
But we could see the response coming from concur in the channel log.
Could you please provide more details on how you handled and the inbound structures used in teh response mapping.
Please share the logic been used in the increment step and the initialization of Retry Counter.
Our current SAP PO version is SAP PI 7.31 SP08.
Request you to share more details if possible through mail.
My Mail Id -: narayana.bommareddy@gmail.com
Thanks & Regards
Narayanareddy B
Thank you for the blog. I may tend to disagree on the following point.
Consumption of Concur's RESTful API is achieved via Advancto's REST adapter as it requires OAuth 2.0 authentication method, which is not yet available on SAP's own REST adapter.
In fact ,OAuth authentication method is available in SAP's REST adapter.
Hi Sridhar
Thanks for your comment. If you check the date of this blog, it was written before Alexander Bundschuh announced on 13 April in the blog below that OAuth 2.0 is available in the latest SP.
What’s new in SAP Process Orchestration 7.31 SP15 / 7.4 SP10
Even the release notes for 7.4 SP10 in SAP Help Portal do not indicate that OAuth 2.0 is available.
SAP Process Integration - What's New in SAP NetWeaver 7.4 (Release Notes) - SAP Library
Anyway, I have updated the statement to reflect that OAuth 2.0 was not available during the project timeline, which led us to choose Advantco REST adapter - this was also the advice provided by our SAP account executives.
Rgds
Eng Swee
Hi Narayanareddy
You can initialize the retry counter by just setting it to 0 at the output mapping of the starting step.
The increment step is an optional feature, it's just an internal counter just for logging purposes. I'd suggest that you skip implementing that feature.
Regarding the mapping issue, I think the comments section of this blog is not the best place to try and resolve that. Best that you open a new thread and provide as much details as possible there - channel log, java mapping code, REST channel config, ESR definitions, etc.
Rgds
Eng Swee
Hi Eng,
Are we getting encrypted file in get extract step,if so where can we apply decryption.
Hi Eng,
Thanks for your reply.
Could you please share some more information on how you are handling the text/csv response coming from the extract file api in the rest receiver channel
Thanks
Narayanareddy B
Hi Eng,
Could you please let me know in your case u always get zip payload in the response or test/csv response as a payload.Could yoy please provide some more information on how we can handle if we get both the test/csv or zip format as response payload from extract file API.
Thanks
Narayanareddy B
Narayanareddy,
In my case, it is zip as detailed in the blog.
I'm more than happy to help as best I can but as already mentioned previously:
Rgds
Eng Swee
Hi Eng,
Thanks for your reply.
I have created a new discussion.
One last check , are you also getting the same response like the below one where i see at my end.
PKâE�F:ws_extract_attendee_detail_p0600908soav_20150424083413.txts� rt©1204Õ50Ñ52©1¨áåPKzà@ÆPKâE�F5ws_extract_CES_SAE_v3_p0600908soav_20150424084014.txtíTß��@~oÒÿ�÷Fº»ìV}ëòCÏz Qljß8ÜöL�`N�ùão8�Ò«�´M�Ü@�awvçûfgÇùâϤå#Ttï0Ø�B�²·olÇ�£[4Ï](L�]«�ÑbÀôÁÒc�0ÀÏdAªÙТ¨Ìû�Êv¡�b¤eJA%��=Þ7Í>PÚ=í2��7ò[�¼¡=q/°¶�*ø)
K|<`BgÉÔºÜWs[§J�&Ñ:Ìá:´�ç«�RÐv�ÚÅAD'\�ÁtR²*DP3gî:B@JëhU�Ò'¬�ùQ\�È\D{·O×±JS\ï�-ò�¸�Þ²êó�{Å¡2®â,MmÉù XAzßüBn&®Sl-§�l¶A�×ú½³ÙFI®0¿©Ú¯¤�oT�iV²RÀJ��¼«`õ»�í Ûéwa#�àpY�««óq)U°JaøÁA�ûì>Ù³üHåÒe¾�7��Ð/§£u°Nzã÷ç4×·èãþ�¾}õ0ÙµYÄ�+J��eX\�E±ïsR%®yÜÕðsªáÚ$qÚÎÇ�Û$^%1¸AF*ý¶VÑêxÏZÏ¢U½T~Ñ®ØkW5ç×®õ¿w5¬zò:oN»�ô���íø÷úÛÙ=¬å�[ôÚ�þmczPKhÏ��i PKâE�Fzà@Æ:ws_extract_attendee_detail_p0600908soav_20150424083413.txtPKâE�FhÏ��i 5�ws_extract_CES_SAE_v3_p0600908soav_20150424084014.txtPKËö
Thanks & Regards
Narayanareddy B
Hi Eng,
thanks for the blog. Did you use also the REST APIs to update concur with master data from SAP (cost centers, employee information, organizational structure, etc.)
Thanks,
Jorge
Hi Jorge
Yes, we used the ListItem API to create/change/delete master data from SAP for cost centers, org hierarchy.
We did not use the Employee/User API because it did not support some fields which we needed, so for that we went for SFTP file transfer instead.
Rgds
Eng Swee
Hello All,
Does anybody has the useful links or iFlow zip files to get the invoice extract from Concur's API using sap CPI?
Thanks in advance.