SAP Customer Data Cloud Integration via SAP CPI(OAuth based)
In this blog I would like to share how we can implement integration between SAP Customer data cloud(CDC) to SAP Marketing & Service Cloud using SAP Cloud platform integration(CPI) as Middleware.
Authentication we wanted to be OAuth based rather basic authentication as SAP CDC Identity sync comes out of the box.
We are using IdentitySync to push data out from SAP CDC to SAP CPI and we will focus on dataflow configuration in this blog.
Second part of this series will cover SAP CPI Iflow designs as well.
Design Approach –
SAP CDC & Marketing cloud has out of the box point to point integration. Where it directly calls SAP Marketing Cloud Odata services in IdentitySync dataflow.
You can refer to details of this integration in SAP CDC developers Wiki.
But there is no out of the box integration available between SAP CDC & Service Cloud and no standard way to have OAuth based authentication as well.
We used SAP CPI as middleware and built our custom Iflows to direct the profiles data to SAP Service & Marketing endpoints.
We can achieve this through “External Generic” writer steps in the IdentitySync data flow.
To support OAuth based authentication we used “Record.evaluate” step in Dataflow and formatted it to pass it on to final writer steps.
Let’s start building the Dataflow for our integration.
- Whitelisting of SAP CPI end point urls – This is a mandatory step as we will be using “External generic” writer step in our integration. All the endpoints urls which we are going to use, should be whitelisted for your partner key(tenant).
- You need to raise a support ticket to SAP CDC product team to get end point urls whitelisted. Else you will get a standard error as below.
Dataflow Configuration –
Create your data flow by selecting the SAP Marketing outbound dataflow template. As we are going to use mostly similar steps “Rename” & ”Field.evaluate”, “Remove” to get the fields synced with other systems.
- Write a select query to pick the fields you want to send to SAP Marketing or Service Cloud.
- Select Email Account in field “from”
- Enter DeltaField = LastUpdatedTimestamp
Note – Email accounts are selected when we want to send Consents & Preferences along with Profile information. By default it has “Accounts” selected and wont send Consents with profile information.
You can modify as per your field requirements Rename, Fields.Evaluate, Remove steps in the dataflow.
Step #3 – Remove last Writer step of SAP Marketing cloud and replace with an “External generic” steps and modify it as shown below to receive OAuth token from CPI end point.
Step #3a –
- Edit the step and enter SAP CPI end point url to get OAuth token as below.
- Enter Method as “POST”.
- Don’t forget to select checkbox “AddResponse”. By this you can add token response to next step.
- Maintain your header Authorization – header name & value(taken from SAP CPI)
Step #4 – Now we need to send this OAuth token received in last step to the next writer step. But we have an issue here as OAuth Token will not be constant or fixed length and will be updated in every response from SAP CPI.
To handle this, we need to first format the token and then pass it to final writer steps of SAP Marketing & Service.
We will write our own script to format the token and for that SAP CDC provides “Record.evaluate” step. In this step we can write our own custom script.
Note – You can also test your script by pressing button Test script and validate the output.
Step #5 – Now in the final step you need to call your SAP CPI endpoints for SAP Marketing & Service Iflows.
You can call both in the same iflow as below and pass the bearer token formatted in the last step.
Add 2 more “external.generic” steps and point them to SAP Marketing & Service cloud iflow urls.
Step #5a –
- In the url enter your SAP CPI Marketing cloud iflow end point url.
- Maintain Method as Post for this call.
- Enter Authorization and pass the value of token received in the last step.
Note – Don’t forget to maintain “maxConnections” to handle parallel processing of the records. It will help you execute your records without running into any memory issues. Like below common error.
Step #6 – Final step is to schedule a job as per your defined regular interval to push profiles data from SAP CDC.
Step #7 – Define the parameter to run in defined frequency and maintain email address to get email on success and failure of the job.
You can find more information on 2nd part of my next blog related to Iflow design we have built for SAP Marketing & Service cloud in SAP CPI.
Monitoring Job status & Error logs –
You can select the dataflow from the list and from actions you can navigate to “status” to monitor the status of the job scheduled and check error logs.
Click on information to check the details of it.
Trace – It will display all the steps executed sequentially in your dataflow. Log level, step name and message output from it.
Error section – This displays the detailed error and steps on which error occurred and count as well.
Hope this blog helps In integration of SAP Customer data cloud with SAP Marketing & Service Cloud via SAP Cloud Platform Integration Suite.
Thanks to my colleagues – Carlos Raimundo , Garvit Agrawal, Manu Gupta
Gaurav Singh Jat
Nicely articulated and too much useful as there is no standard way to connect C4C via CDC.
Great Blog.. Thanks for sharing..
Very well explained, Great Blog!
Just wanted to understand what is the reason you choose to use Oauth based authentication here. Because I am trying the same in my project as well, I used the external writer component and used the ilfow endpoint URL to trigger the iflow sending data from CDC to C4C. Therefore need and opinion is if it is the correct approach or should OAuth be used for such scenarios.
Hi ritanshi bansal,
Oauth or certificate based authentication are better secured connection than basic authentication.
We already implement the dataflow follow the obove guide, nonetheless we got serious performance problems
In marketing cloud we have a hard restriction related to API_MKT_CONTACT_SRV version 4 we must use this version so in the middle we need use CPI to adapt the CDC contact request before to send to marketing, this is why in CDC in our Custom Data Flow we use a “datasource.write.external.generic” to send the contacts. We notice that this component doesn´t have a batch parameter to group contacts before send them to CPI this could be avoid high volume calls to CPI and maybe increase the performance. At the same time we realize that the standard stept “datasource.write.hybrismarketing” already have this parameter
Could you confirm us if its a recommendable practice group contacts before send them in order to grant and acceptable performance and could you give us additional support to add this a batch parameter to our custom data flow step “datasource.write.external.generic”
you can send customer profiles in batch payload also from IDX with the help of "record.evaluate" and write your script to generate that. SAP CPI & Marketing cloud are able to handle that, you just need to test and adjust the batch size which CPI and Marketing can handle to achieve right performance. IDX will generate payload in array format and send defined number of customer profile in batch mode.
Hope this will help resolve your issue.
Hi Gaurav Singh Jat,
Great blog! 🙂
Is the second part of this series, covering the CPI iflows, already published?
Thanks and kind regards,
Nice blog 🙂