With the integration package SAP Marketing Cloud – file based integration you can use CSV files to load contacts, accounts and interactions into SAP Marketing Cloud systems. The files can come from different origins, like legacy systems or service providers.
The standard integration flows in the package provide many fields, of which not all might be needed for your scenario. It is also probable that other fields are needed such as custom fields or different standard fields. In this blog the necessary steps are described to adapt the integration flow to individual needs.
With the latest release of the SAP Cloud Platform Integration Web IDE, the changes can be achieved with some few steps directly in the Web IDE, without the need to use Eclipse IDE with Cloud Integration plugin. They are described below.
You have access to your SAP Cloud Platform Integration tenant and have downloaded or updated the integration package into your tenant.
You have also access to your SAP Marketing tenant. In case you want to upload data for custom fields, ideally you have already performed the necessary extensions in the SAP Marketing tenant according to the instructions in the Extensibility Guide.
Ideally you are also somewhat familiar with the SAP Cloud Platform Integration Web application (Web IDE), e.g. through loading the integration package to your tenant and configuring an integration flow you want to use. You can also do some first tests with the example CSV file available in the documentation section of the package content.
The integration flows in the package for file upload consist of several steps
- Read File from SFTP server
- Convert CSV to XML
- Split the resulting XML into packages
- Map the source XML to the target XML for sending via OData
- Send the message to the SAP Marketing Cloud
The steps relevant for extending and adapting are step 2 and step 4.
Copy the integration flow
As there is no modification-free extension concept currently in place for these integration flows, it is recommended to perform the steps on a copy of the standard integration flow.
The easiest way to obtain a copy is just to import the integration package again. You add a suitable suffix (e.g. Adapted) to your package and the integration flows have this suffix then too.
Alternatively, you can copy an individual integration flow by downloading it, changing the Bundle-SymbolicName in the manifest file in folder META-INF and importing it again to your integration tenant.
All changes can now be made in this copy.
Adapt step”Convert CSV to XML”
In this step the imported CSV file is transformed to an XML message. The XML Schema used for transforming the file can be found under tab “Processing” of the step. Depending on the version of the integration flow you use, it is named
/xsd/ImportHeadersEntityPOST1.xsd (up to version 1.1.3) respective
/xsd/CSV_Contacts.xsd (from version 1.2 on) for the contacts integration flow.
This schema definition file can now be adapted in the Web IDE directly by following steps executed exemplary in the “File based load – Contacts” integration flow:
- Click Edit
- Select the “Resources” tab in the editor (click on the white space area to see the integration flow properties)
- Under “Schemas” you can find the different XSD files. Choose the one relevant for your integration flow version. From 1.2 on for the contact integration flow this is file xsd/CSV_Contacts.xsd. Note you can see the access path of the file via the tooltip (mouse over).
- Click on the hyperlink of the filename to get to the editor
You can now modify the segment named “Contact”. All attributes listed below are used for transformation. Note, that the changes you do (adding and removing fields) need to be reflected in your source CSV file. In this example the CSV file for contacts has the following columns:
Exactly these fields have then to be available in the same order under the segment “Contact”. This leads to the structure displayed in figure 1
Note that the type information provided for the fields in the XSD file is not considered during the CSV to XML transformation
Having done these steps, the integration flow is not yet operational. The next step is now to adapt the mapping.
Adapt step “Mapping”
The mapping step maps the now adapted source XML to the target XML, which is used as payload for the OData service. The mapping itself can be accessed directly under Resources -> Mappings. For “File based load – Contacts”, click on the link named MM_Contacts to get directly to the mapping editor. Alternatively, the mapping can be accessed through clicking on the “Mapping” artifact and selecting in the lower part of the editor under Message Mapping -> tab Processing -> Resource: /MM_Contacts.mmap. Make sure you are in the Edit mode of the integration flow.
Remove the unwanted mappings:
You will now see in the left area the adapted source structure, but as I removed many attributes from standard source structure, there are many “orphaned” mappings still available. They are marked with a red error icon, see figure 2. You need to click on them each individually, so the mapping line gets “highlighted”. Then remove the unnecessary mappings by right mouse click and choosing “Delete”. After having removed all obsolete mappings, click on OK to save. You get back to the integration flow editor.
Add the new mappings:
In my example 2 additional custom extension fields are added to the mapping: HasCat and NameOfCat. Of course, the OData service used for the inbound call needs already to be extended for that. Follow the extensibility instructions in the SAP Marketing Cloud Extensibility Guide:
The example fields were added as Custom Fields to the Marketing Contact via the app “Custom Fields and Logic” following the Extensibility Guide. The additional fields are added to the Contact Entity Type of the OData service CUAN_IMPORT_SRV, as this service is currently used in the integration flow. To access the names and types of the extension fields, open the metadata document of the CUAN_IMPORT_SRV e.g. in a browser of your choice by accessing following URL:
When entering the URL, you need to provide the credentials of the Communication User maintained for the Communication Arrangement necessary for accessing the OData service CUAN_IMPORT_SRV (Communication Scenarios SAP_COM_003 and SAP_COM_004). You can then see the metadata document of the service and check the attributes of the Contact entity type. In the example, the custom properties are:
<Property Name="YY1_HasCat_MPS" Type="Edm.Boolean" sap:label="Has Cat" sap:filterable="false" sap:is-extension-field="true"/> <Property Name="YY1_NameofCat_MPS" Type="Edm.String" MaxLength="99" sap:label="Name of Cat" sap:filterable="false" sap:is-extension-field="true"/>
The quickest way is now to directly add these fields to the XSD of the target mapping:
- Check that you are in edit mode of the integration flow.
- In the mapping, check the used target structure XSD by clicking Edit and checking the tool-tip of the target structure.
- Up to integration flow release 1.1.3 the target structure is wsdl/ImportHeadersEntityPost1.xsd,
- for higher releases it is xsd/ImportHeadersEntityPOSTorig.xsd
- Go back to the integration flow editor, click on the white space area to get to the integration parameters and select the Resources Tab.
- Click on the hyperlink of the respective target structure XSD file to get to the edit mode. It can be helpful to copy the message and format it in an XML Editor (e.g. Notepad++ with XML plugin) to get a better overview of the attributes.
- Add the custom fields to the respective segment, in the example here under Contact segment.
Now these added attributes can be used for mapping the source structure attributes. For that, navigate again into the mapping editor, and connect the custom attributes from source to target structure (see example in figure 3). Don’t forget to click OK to save mapping and to save the adapted integration flow.
Final steps and summary
The adapted integration flow now needs to be configured if not already done. Enter the necessary parameters for source (SFTP) and target (SAP Marketing) access and deploy the integration flow. You can then load your individually adapted CSV data files into the SAP Marketing system.
The steps to achieve an adapted integration flow from the standard delivered one can all be made in the SAP Cloud Platform Integration tenant Web IDE.
When performing the steps for writing this blog I found following features of the SAP Cloud Platform Web IDE useful:
- In the Mapping Editor, the “Simulate” function is useful to see the result of the mapping. For that, the source message need to be provided. You can easily obtain one by using the Log “Trace” functionality of the integration flow. Check then the resulting target structure.
- Logging feature using “Trace” functionality: Enable log level “trace” for the integration flow and send the file. Click then on “Trace”, search for the step (ID) you want to see the message. Make sure you select the first entry for the step (Started At) and click on “view step details”. You can find the message then in Message Content, where header and payload are displayed.
See more details in the blog Enable message tracing
Please feel free to comment on this blog in case you have questions or feedback.
Further useful links: