Extending and Adapting Integration Flows for File-Based Upload into SAP Marketing Cloud
** Update for release 1811 (November 2018) **
With release 1811, the integration flows for contacts, accounts and interactions have been renovated. The call of CUAN_IMPORT_SRV has been replaced with API_MKT_CONTACT_SRV for contacts, API_MKT_CORPORATE_ACCOUNT_SRV for accounts and API_MKT_INTERACTION_SRV for interactions. Also the graphical message mapping of source XML to target XML has been replaced with an XSLT mapping. The steps for extensions below have been enhanced to fit to the 1811 release.
With the integration package SAP Marketing Cloud – File-Based Data Load you can use CSV files to load contacts, accounts and interactions into SAP Marketing Cloud systems. The files can come from different origins, like legacy systems or service providers.
The standard integration flows in the package provide many fields, which not all might be needed for your scenario. It is also probable that other fields are needed such as custom fields or different standard fields. In this blog the necessary steps are described to adapt the integration flow to individual needs.
With the latest release of the SAP Cloud Platform Integration Web IDE, the changes can be achieved with few steps directly in the Web IDE, without the need to use Eclipse IDE with Cloud Integration plugin. They are described below.
You have access to your SAP Cloud Platform Integration tenant and have downloaded or updated the integration package into your tenant.
You have also access to your SAP Marketing Cloud tenant. In case you want to upload data for custom fields, you ideally have already performed the necessary extensions in the SAP Marketing Cloud tenant according to the instructions in the Extensibility Guide.
Ideally, you are also to a certain extend familiar with the SAP Cloud Platform Integration Web application (Web IDE), for example by loading the integration package to your tenant and configuring the integration flow you want to use. You can also do some first tests with the example CSV file available in the documentation section of the package content.
The integration flows in the package for file upload consist of several steps:
- Read File from SFTP server or get file content via HTTPS
- Convert CSV to XML
- Split the resulting XML into packages
- Map the source XML to the target XML for sending via OData
- Send the message to the SAP Marketing Cloud
The steps relevant for extending and adapting are step 2 and step 4.
Copy the Integration Flow
As there is currently no modification-free extension concept in place for these integration flows, it is recommended to perform the steps on a copy of the standard integration flow.
The easiest way to obtain a copy is just to import the integration package again. You add a suitable suffix (for example, Adapted) to your package and the integration flows have this suffix then too.
Alternatively, you can copy an individual integration flow by downloading it, changing the Bundle-SymbolicName in the manifest file in folder META-INF and importing it again to your integration tenant.
All changes can now be made in this copy.
Adapt Step “Convert CSV to XML”
In this step the imported CSV file is transformed to an XML message. The XML Schema used for transforming the file can be found under tab “Processing” of the step. Depending on the version of the integration flow you use, for the contacts integration flow it is named as follows:
/xsd/ImportHeadersEntityPOST1.xsd(up to version 1.1.3)
/xsd/CSV_Contacts.xsd(for version 1.2)
/xsd/CSV_Contacts_Validation.xsd(from version 1811 on)
This schema definition file can now be adapted in the Web IDE directly by following steps to be executed exemplarily in the “File-Based Load – Contacts” integration flow:
- Click Edit.
- Select the “Resources” tab in the editor (click on the white space area to see the integration flow properties).
- Under “Schemas” you find the different XSD files. Choose the one relevant for your integration flow version. From 1811 on for the contact integration flow, this is file
xsd/CSV_Contacts_Validation.xsd. Note that you can see the access path of the file via the tooltip (mouse over).
- Click on the hyperlink of the filename to get to the editor.
You can now modify the segment named “Contact”. All attributes listed below are used for transformation. Note that the changes you do (adding and removing fields) need to be reflected in your source CSV file. In this example the CSV file for contacts has the following columns:
Exactly these fields have to be available in the same order under the segment “Contact”. This leads to the structure displayed in figure 1. Note that the type information provided for the fields in the XSD file is not considered during the CSV to XML transformation.
Having executed these steps, the integration flow is not yet operational. The next step is to adapt the mapping.
Figure 1 – source XSD after adaptions (click to enlarge)
Adapt Step “Mapping”
The mapping step maps the now adapted source XML to the target XML, which is used as payload for the OData service. In my example, two additional custom extension fields are added to the mapping: HasCat and NameOfCat. The OData service used for the inbound call must already be extended for that. The example fields have benn added as Custom Fields to the Marketing Contact via the app “Custom Fields and Logic” as described in the SAP Marketing Cloud Extensibility Guide.
To ensure that the OData service has exposed these fields – in this case in the Contact entity – you can access the metadata document of the OData service in a browser of your choice by accessing following URL:
For versions up to 1811, you check the CUAN_IMPORT_SRV:
From 1811 on, the link to the metadata document is:
When entering the URL, you must to provide the credentials of the Communication User maintained for the Communication Arrangement necessary for accessing the OData service. You can then see the metadata document of the service and check the attributes of the Contact entity type. In the example, the custom properties are:
<Property Name="YY1_HasCat_MPS" Type="Edm.Boolean" sap:label="Has Cat" sap:filterable="false" sap:is-extension-field="true"/> <Property Name="YY1_NameofCat_MPS" Type="Edm.String" MaxLength="99" sap:label="Name of Cat" sap:filterable="false" sap:is-extension-field="true"/>
Graphical Message Mapping up to 1811 Release
Up to release 1811, the mapping of source XML to target XML has been done using the graphical message mapping of the CPI. The mapping itself can be accessed directly under Resources -> Mappings. For “File-Based Load – Contacts”, click on the link named MM_Contacts to get directly to the mapping editor. Alternatively, the mapping can be accessed by clicking the “Mapping” artifact and selecting /MM_Contacts.mmap in the lower part of the editor under Message Mapping -> tab Processing. Make sure you are in the Edit mode of the integration flow.
Remove the unwanted mappings:
In the left area, you will now see the adapted source structure, but as I removed many attributes from standard source structure, there are many “orphaned” mappings still available. They are marked with a red error icon, see figure 2. You need to click each of them individually, so the mapping line gets “highlighted”. Then remove the unnecessary mappings by a right mouse click and choosing “Delete”. After having removed all obsolete mappings, click “OK” to save. You get back to the integration flow editor.
Add the new mappings:
The additional fields are added to the Contact Entity Type of the OData service CUAN_IMPORT_SRV, as this service is used in the integration flow version lower than 1811. The quickest way is now to directly add these fields to the XSD of the target mapping:
- Check that you are in the “Edit” mode of the integration flow.
- In the mapping, check the used target structure XSD by clicking “Edit” and checking the tool-tip of the target structure.
- Up to integration flow release 1.1.3 the target structure is wsdl/ImportHeadersEntityPost1.xsd,
- For release 1.2 it is xsd/ImportHeadersEntityPOSTorig.xsd
- Go back to the integration flow editor, click the white space area to get to the integration parameters and select the “Resources” Tab.
- Click the hyperlink of the respective target structure XSD file to get to the “Edit” mode. It can be helpful to copy the message and format it in an XML Editor (e.g. Notepad++ with XML plugin) to get a better overview of the attributes.
- Add the custom fields to the respective segment, in the example here under Contact segment.
Now these added attributes can be used for mapping the source structure attributes. For that, navigate again into the mapping editor, and connect the custom attributes from source to target structure (see example in figure 3). Don’t forget to click “OK” to save the mapping and to save the adapted integration flow.
Adapt XSLT Mapping as of Release 1811
As of release 1811, the mapping of source to target XML happens via XSLT. The transformation can be found under Resources -> Mappings -> MM_Contacts.xsl of type XSLT Mapping. Click the hyperlink to get to the editor, make sure you are in the “Edit” mode of the integration flow.
You can now remove all the mappings that are not needed from the transformation. Next, you can add the two extension fields of this example to the transformation as shown figure 4. Don’t forget to click “OK” to save the mapping and to save the adapted integration flow.
For more complex changes to the XSLT mapping document, you can also use the Eclipse XSL Developer tools. To obtain a source XML for developing and testing the transformation, you can trigger a run of the iflow message in “Trace” mode. With this source XML you can adapt and test your XML transformation directly in the Eclipse tool or in any other XSLT editor.
Final Steps and Summary
Now, the adapted integration flow must to be configured if not already done. Enter the necessary parameters for source (SFTP) and target (SAP Marketing) access and deploy the integration flow. You can then load your individually adapted CSV data files into the SAP Marketing Cloud system.
The steps to achieve an adapted integration flow from a pre-delivered standard one can all be executed in the SAP Cloud Platform Integration tenant Web IDE.
When performing the steps for writing this blog, I found the following features of the SAP Cloud Platform Web IDE useful:
- In the Mapping Editor, the “Simulate” function is useful to see the result of the graphical mapping. For that, the source message need to be provided. You can easily obtain one by using the Log “Trace” functionality of the integration flow. Then, check the resulting target structure. For the mapping via XSLT transformation, I used the Eclipse XSL Developer tools for developing and testing.
- Logging feature using “Trace” functionality: Enable log level “trace” for the integration flow and send the file. Then, click on “Trace”, search for the step (ID) you want to see the message. Make sure you select the first entry for the step (Started At) and click on “view step details”. Then, you can find the message in Message Content, where header and payload are displayed.
See more details in the blog Enable message tracing
Please feel free to comment on this blog in case you have questions or feedback.
Further useful links: