Skip to Content

You can perform batch operations in the OData V2 Adapter for SAP Cloud Platform Integration. Batch, as its name suggests, implies that multiple requests are clubbed into one HTTP call. What this means is that if you are updating 100 records, instead of making 100 PUT or update request, the adapter would just make one call to the server with all the essential data. This would greatly improve time taken to achieve such requirements, also improving the performance.

The current OData V2 Adapter in SAP Cloud Platform Integration supports POST, PUT, MERGE and DELETE HTTP operations in batch. Using the POST operation, you can insert ‘n’ number of new entities in your OData backend, using PUT/MERGE operation you can update entities and DELETE operation to delete entities. We will visit POST and PUT examples in the following sections.

The OData V2 Adapter as of version 2.27.0 does not support QUERY or READ HTTP operation in batch.

To know more on how the OData Service supports batch you can go through OData documentation, http://www.odata.org/documentation/odata-version-2-0/batch-processing/ which will give you a brief overview on structure and format of a batch request.

 

Request Body for Batch Operation

The request body of batch contains an ordered sequence of operations to be performed. These modifying operations are called ‘ChangeSet’. ChangeSets are atomic units that contain the entity on which create, update or delete operations are to be performed. The ChangeSets can also contain an unordered sequence of operations. All changes in a ChangeSet must be all processed successfully or none of them. Implementation of ChangeSet varies with different services.

Here’s an example of the request body in SAP Cloud Platform Integration for ‘Batch Request’ which is denoted by a <batchParts> tag. This contains 2 ‘ChangeSets’ denoted by <batchChangeSet> tags. It is mandatory to have at least one ChangeSet in a batch request payload.

The <method> tag denotes which operation is to be performed. Iin this case, the operation is POST and the entity set is Categories. These tags are mandatory to be sent in all batch operations. There are a few optional tags like <headers> to send custom headers to ChangeSets. The <uri> tag is used to mention the exact entity on which you wish to perform the operation. It is mandatory in case of DELETE operation. Remember to add the key and its value when using <uri> tag for DELETE, MERGE or UPDATE operations.

Configuring Batch in the OData Adapter

We are using the OData Demo V2 service, http://services.odata.org/(S(btmy2xygehkm1gfrnpezr4xx))/V2/OData/OData.svc in this example.

  1. Select OData as the Receiver Adapter; screenshot of the adapter specific details is as seen below.
  2. Modelling Resource Path; connect to the service by adding a new system
  3. Enable ‘Batch Processing’ by switching to ON, after selecting the entity and operation of your choice. Here, entity is ‘Categories’ and Operation is ‘POST’.
  4. Click OK. Here’s a screenshot of how the adapter settings should look after modelling. Notice that the ‘Enable Batch Processing’ checkbox is enabled by default.

 

Note: Soon after modelling the operation, XSD file is generated and a message is displayed based on the fields selected by you in Step 3. This is to help you to map the request payload when using message mapping.

 

Understanding the Batch Response

Here’s an example of how the batch response looks like.

The first ChangeSet is the generic format and the second ChangeSet is an actual response received for the above request. Each individual ChangeSet response can be viewed under <batchChangeSetResponse> tag, which is followed by the <headers> tag. This <headers> tag denotes response headers from the OData Endpoint.

The <statusCode> and <statusInfo> gives us the HTTP status code and status line details respectively. <body> tag shows us the successfully inserted response in case of insert operations. It is empty for update operations. An error response is shown if either of the operations is unsuccessful. However, in the SAP Cloud Platform Integration’s message processing log (MPL), you will see that the status is COMPLETED. This is because, the batch requested has been accepted successfully by the OData server and it has provided a response.

 

Integration Flow with OData Batch

Scenario 1:With timer triggered event- Creating two new categories with body specified in the content modifier

The timer is configured to run once; the content modifier contains the request body. The response is captured in SFTP Server.

An example of batch request body is as seen below; we are inserting 2 new Categories with ID ‘25’ and ‘26’.

<batchParts>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>POST</method>
			<Categories>
				<Category>
					<Name>Example1</Name>
					<ID>25</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
	</batchChangeSet>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>POST</method>
			<Categories>
				<Category>
					<Name>Example2</Name>
					<ID>26</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
	</batchChangeSet>
</batchParts>

The output as observed in SFTP server is

<batchPartResponse>
    <batchChangeSetResponse>
      <batchChangeSetPartResponse>
        <headers>
          <Accept-Language></Accept-Language>
          <DataServiceVersion>1.0;</DataServiceVersion>
          <Location>http://services.odata.org/(S(btmy2xygehkm1gfrnpezr4xx))/V2/OData/OData.svc/Categories(25)</Location>
          <Accept></Accept>
          <Content-Type>application/atom+xml;charset=utf-8</Content-Type>
          <Cache-Control>no-cache</Cache-Control>
        </headers>
        <statusCode>201</statusCode>
        <body>
          <Name>Example1</Name>
          <ID>25</ID>
        </body>
        <contentId/>
        <statusInfo>Created</statusInfo>
      </batchChangeSetPartResponse>
    </batchChangeSetResponse>
    <batchChangeSetResponse>
      <batchChangeSetPartResponse>
        <headers>
          <Accept-Language></Accept-Language>
          <DataServiceVersion>1.0;</DataServiceVersion>
          <Location>http://services.odata.org/(S(btmy2xygehkm1gfrnpezr4xx))/V2/OData/OData.svc/Categories(26)</Location>
          <Accept></Accept>
          <Content-Type>application/atom+xml;charset=utf-8</Content-Type>
          <Cache-Control>no-cache</Cache-Control>
        </headers>
        <statusCode>201</statusCode>
        <body>
          <Name>Example2</Name>
          <ID>26</ID>
        </body>
        <contentId/>
        <statusInfo>Created</statusInfo>
      </batchChangeSetPartResponse>
    </batchChangeSetResponse>
  </batchPartResponse>

 

Scenario 2: Using XSD generated to form the batch request payload

SOAP adapter triggers the call to the OData Endpoint. The response is sent to SFTP server via SFTP adapter. The target mapping used is the XSD generated from the OData Receiver.

Batch request body is shown below. The first entity should be successfully updated and the second one should fail because there is no Category(123) currently in the OData server.

<batchParts>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>PUT</method>
			<Categories>
				<Category>
					<Name>NewExample1</Name>
					<ID>25</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
	</batchChangeSet>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>PUT</method>
			<Categories>
				<Category>
					<Name> NewExample2</Name>
					<ID>123</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
	</batchChangeSet>
</batchParts>

The output as observed in SFTP server is

<batchPartResponse>
	<batchChangeSetResponse>
		<batchChangeSetPartResponse>
			<headers>
				<Accept-Language/>
				<DataServiceVersion>1.0;</DataServiceVersion>
				<Accept/>
				<Cache-Control>no-cache</Cache-Control>
			</headers>
			<statusCode>204</statusCode>
			<body/>
			<contentId/>
			<statusInfo>No Content</statusInfo>
		</batchChangeSetPartResponse>
	</batchChangeSetResponse>
	<batchChangeSetResponse>
		<batchChangeSetPartResponse>
			<headers>
				<Accept-Language/>
				<DataServiceVersion>1.0;</DataServiceVersion>
				<Accept/>
				<Content-Type>application/xml</Content-Type>
			</headers>
			<statusCode>400</statusCode>
			<body>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; standalone=&quot;yes&quot;?&gt;
&lt;error xmlns=&quot;http://schemas.microsoft.com/ado/2007/08/dataservices/metadata&quot;&gt;
  &lt;code&gt;&lt;/code&gt;
  &lt;message xml:lang=&quot;en-US&quot;&gt;Error processing request stream. The URI specified is not valid.&lt;/message&gt;
  &lt;innererror&gt;
    &lt;message&gt;Invalid Uri specified. The query &apos;DataServiceProvider.DSPLinqQuery`1[DataServiceProvider.DSPResource]&apos; must refer to a single resource&lt;/message&gt;
    &lt;type&gt;System.ArgumentException&lt;/type&gt;
    &lt;stacktrace&gt;   at DataServiceProvider.DSPUpdateProvider.GetResource(IQueryable query, String fullTypeName) in c:\TFS\OData\Main\ndp\fx\src\DataWeb\ODataSDK\services.odata.org\DataServiceProvider\DSPUpdateProvider.cs:line 276&amp;#xD;
   at System.Data.Services.UpdatableWrapper.GetResource(IQueryable query, String fullTypeName)&lt;/stacktrace&gt;
  &lt;/innererror&gt;
&lt;/error&gt;</body>
			<contentId/>
			<statusInfo>Bad Request</statusInfo>
		</batchChangeSetPartResponse>
	</batchChangeSetResponse>
</batchPartResponse>

 

Scenario 3: Batch Request with multiple entries in one ChangeSet

The request payload that we will be using for this scenario is as seen below.

<batchParts>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>POST</method>
			<Categories>
				<Category>
					<Name>Create Example 1</Name>
					<ID>16</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
		<batchChangeSetPart>
			<method>POST</method>
			<Categories>
				<Category>
					<Name>Create Example 2</Name>
					<ID>17</ID>
				</Category>
			</Categories>
		</batchChangeSetPart>
	</batchChangeSet>
	<batchChangeSet>
		<batchChangeSetPart>
			<method>PUT</method>
			<Categories>
				<Category>
					<Name>Updated Example 1</Name>
					<ID>16</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
		<batchChangeSetPart>
			<method>PUT</method>
			<Categories>
				<Category>
					<Name>Updated Example 2</Name>
					<ID>17</ID>
				</Category>
			</Categories>
		</batchChangeSetPart> 
	</batchChangeSet>
</batchParts>

Notice that, the request payload contains two ordered ChangeSets(batchChangeSet).

The first ChangeSet consists of two request to be performed(batchChangeSetPart), i.e creation of ID 16 and 17. If either of the ‘batchChangeSetPart’ fails neither will be completed. The rollback is performed by the OData Server if failure occurs.

These ‘batchChangeSetPart’ are unordered, meaning that any batchChangeSetPart can be performed first irrespective of the sequence in the payload.

The second ChangeSet consists of two update request of ID 16 and 17.

Successful response for the above request payload can be seen below. As of 2.27.0 version of Cloud Platform Integration, we allow only one response to be listed in the payload for each ChangeSet.

<batchPartResponse>
	<batchChangeSetResponse>
		<batchChangeSetPartResponse>
			<headers>
				<Accept-Language/>
				<DataServiceVersion>1.0;</DataServiceVersion>
				<Location>http://services.odata.org/(S(btmy2xygehkm1gfrnpezr4xx))/V2/OData/OData.svc/Categories(17)</Location>
				<Accept/>
				<Content-Type>application/atom+xml;charset=utf-8</Content-Type>
				<Cache-Control>no-cache</Cache-Control>
			</headers>
			<statusCode>201</statusCode>
			<body>
				<Name>Create Example 2</Name>
				<ID>17</ID>
			</body>
			<contentId/>
			<statusInfo>Created</statusInfo>
		</batchChangeSetPartResponse>
	</batchChangeSetResponse>
	<batchChangeSetResponse>
		<batchChangeSetPartResponse>
			<headers>
				<Accept-Language/>
				<DataServiceVersion>1.0;</DataServiceVersion>
				<Accept/>
				<Cache-Control>no-cache</Cache-Control>
			</headers>
			<statusCode>204</statusCode>
			<body/>
			<contentId/>
			<statusInfo>No Content</statusInfo>
		</batchChangeSetPartResponse>
	</batchChangeSetResponse>
</batchPartResponse>
To report this post you need to login first.

15 Comments

You must be Logged on to comment or reply to a post.

  1. Mark Bernabe

    Hi Sana,

    Thanks for the well explained blog! One thing we’ve noticed with the Batch Request is its performance. It is really slow compared to a normal OData request. For example, when creating Interactions and Interaction Contacts from external system, we use CUAN_IMPORT_SRV OData service. In the File based load – Interactions iFlow inside the SAP Hybris Marketing Cloud – file based data load package, Batch Request is used. However, it seems that we can also achieve the same results by just using the normal ImportHeader instead of Batch processing. In addition, it is a lot faster.

    So when is Batch Request recommended over the normal ImportHeaders and vice versa specifically for CUAN_IMPORT_SRV?

    (0) 
    1. Sana Faraz
      Post author

      Hi Mark,

      Thank you for your comment. From what I know, ImportHeader was added 3 years ago when batch was not available, it works by internally using deep insert. I also checked with Hybris colleagues and added a new example to this blog [Scenario 3: Batch Request with multiple entries in one ChangeSet] which would considerably improve the performance of batch calls.

       

      I am not an expert in SAP Hybris Marketing services but some differences that I could notice with service ‘CUAN_IMPORT_SRV’ are

      Batch ImportHeader
      Batch- POST call Deep Insert Operation- POST Call
      Can be used with all entities defined in the metadata Can only be used with entities ‘Contacts’ and ‘Interactions’ (navigations as defined in the metadata)
      Create,Merge,Update,Delete operations supported Only Create supported

      i would not be able to comment on the performance with ImportHeader but if you have huge chuck of data to be loaded, batch would be the suggested approach.

      For smaller data, as you mentioned Import Header might be better.

      (0) 
  2. Mark Bernabe

    Hi Sana,

    Thanks for taking the time to answer my question. It’s really helpful and greatly appreciated. With regards to the performance, I think it’s the other way around. For huge volume of data, ImportHeader is faster compared to using BatchRequest. From our tests, a batch of 30k Interactions and Contacts only took 4 mins to complete with ImportHeader as compared to using Batch that took more than 2 hours. Moreover, most of the time it fails because of timeout error.

    (0) 
  3. Marcus Englert

    Hi Sana,

    thank you for your post, it’s very helpful!

    I tried a similar scenario, but the call of the OData service fails with HTTP 400 – bad request

    com.sap.gateway.core.ip.component.odata.exception.OsciException:  : 400 : HTTP/1.1
    
    OData Method             :   BATCH
    Request URI              :   POST https://*****.hana.ondemand.com/service/odata.srv/$batch HTTP/1.1
    Request Headers          :   
      Authorization            : Basic ********
      Content-Type             : multipart/mixed;boundary=batch
      x-csrf-token             : ********
      Cookie                   : JSESSIONID               = COULD_NOT_BE_RESOLVED; Path=/service; Secure; HttpOnly;
                                 JTENANTSESSIONID_a3bb64753= ********;
                                  Domain                  = ********; Path=/; Secure; HttpOnly;
                                 BIGipServer*****.hana.ondemand.com= ********; path=/; httponly; secure;
    
    HTTP Status Line         :   HTTP/1.1 400 
    Response Headers         :   
      DataServiceVersion       : 1.0
      Date                     : Tue, 30 Jan 2018 12:13:31 GMT
      Content-Type             : application/xml
      Content-Length           : 213
      X-Cnection               : close
      Server                   : SAP
      Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
    , cause: com.sap.gateway.core.ip.component.odata.exception.OsciException:  : 400 : HTTP/1.1

    Now I found the problem, it’s because of the line feed in the batch request, the CPI creates the body of the batch request only with line feed (LF), but the OData server expects carriage return and line feed (CRLF). Do you know if there’s a possibility to change the request from LF to CRLF?

    Thank you for your help!

    (0) 
    1. Sana Faraz
      Post author

      Hi Marcus,

      As of now we do not have any such mechanism. Could you please raise a ticket on component, LOD-HCI-PI-GB so that we can analysis this further.

       

      (0) 
    2. Saranya Baskaran

      The batch request body has LF as line separator by default.

      To change the request with CRLF, the property “SAP_BatchLineSeparator” has to be set with value “CRLF“.

      (1) 
  4. Hari Sonnenahalli

    Sana-

     

    Hope all is well. I have a question regarding using OData operation to perform a delete operation before posting the data to HANA database. The scenario is I am making an API call and posting the data to HANA after exposing it as OData service. I am using a timer to do GET functions but I need to delete the records for the same entity list before GET function. How can I use ODATA adapter to perform this action? Any taught?

     

    Thanks for the support.

    Regards

     

    HS

    (0) 
    1. Sana Faraz
      Post author

      Hi Hari,

      You will have to perform a GET to receive all the current entities and then you can perform a batch delete with all the available keys. Then you can POST all the new data.

      (0) 
  5. Cesar Mejia

    Dear Sana

     

    Thanks for the explanation, its very nice.

     

    I have a issue related.

    My scenario is: Read a JON file from sFTP server and call a Odata service to Post Interactions.  (I know Convert JSON to XML format have troubles and we need a grovvy to ajust the JSON, but doesnt matter because I request the JSON should be as HCI needs)

     

     

    The JSON file example is:

    {
    “Interactions”: [{
    “Interaction”: [{
    “InteractionContactOrigin”: “SAP_HYBRIS_CONSUMER”,
    “InteractionContactId”: “92a2fc8549a4ad28”,
    “CommunicationMedium”: “ONLINE_SHOP”,
    “InteractionType”: “SHOP_ITEM_VIEW”,
    “InteractionTimeStampUTC”: “1970-01-18T17:14:48”,
    “InteractionAmount”: “0”,
    “InteractionCurrency”: “EUR”,
    “DeviceType”: “Smartphone”,
    “SourceSystemType”: “iOS 11.3”,
    “InteractionProducts”: [{
    “InteractionProduct”: [{
    “ProductOrigin”: “SAP_HYBRIS_PRODUCT”,
    “Product”: “1288120”
    }]
    }],
    “InteractionProductCategories”: [],
    “InteractionInterests”: []
    }]
    }]
    }

    I already tested it in Postman successfully

     

     

    But in CPI , it finish with the following error:

    message : OSCI issue
    cause-exception : com.sap.gateway.core.ip.processor.exception.ODataProcessingException
    cause-message : OSCI issue
    class : java.util.ArrayList
    required-type : java.util.ArrayList
    converter-type : com.sap.gateway.core.ip.processor.converter.ConverterXMLToList
    path : /batchParts/batchChangeSet/batchChangeSetPart/Interactions
    line number : 2
    version : null
    ——————————-, cause: org.apache.olingo.odata2.api.edm.EdmSimpleTypeException: The metadata do not allow a null value.
    LastErrorModelStepId= MessageFlow_4
    Node = vsa4479707
    ProcessId = 30fc0a027688f4fd2d78c40052ab2523b73c1612

     

     

    Do you have any idea?

     

    Best Regards

    Cesar

     

     

    (0) 
    1. Sana Faraz
      Post author

      Hi Cesar,

       

      The OData Adapter in CPI perform metadata validation with the request payload.

      Kindly check if something is nullable false in your Interactions metadata that is not mentioned in your request payload.

       

      (0) 

Leave a Reply