Skip to Content
Author's profile photo Karthik Bangera

Implementing dynamic Lookups in CPI using Hashmap and lightweight XML parsers

You would often have come across a requirement to perform lookups dynamically to an external cloud OData service entity and fetch the details and modify existing payloads in CPI. We had a similar requirement in one of our C4C Data Migration Implementations.

Requirement-

We were required to lookup a Customer Table exposed by OData, provide it the External ID from our main business payload coming from a Legacy CRM and fetch the corresponding C4C Internal ID and update the same in our original payload before feeding it finally to C4C.

Options-

Initially, we planned to use a content enricher to achieve this, however we wished to persist those values by storing them in data store for subsequent calls on the same day to improve the iflow performance and also there was a requirement in the same iflow to perform lookups with multiple primary keys for another C4C entity which wouldn’t have been possible with content enricher. Therefore, we decided to give Hashmap object a try in CPI along with XMLSlurper and XMLParser for parsing and modifying the XMLs since these are lightweight xml parsers and therefore highly recommended.

Test Scenario-

There are 2 parts to the below scenario-

In the first part, we will fill the Hashmap object with all the Key Value pairs which are present in XML format. In the real-life scenario this xml is fetched from C4C OData call. We would test this with around 0.3 million records to see how good its performance is.

Below is the iFlow screenshot-

Simple groovy script for parsing the lookup XML into the HashMap object-

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.xml.*;
import java.util.regex.*;
import java.io.*;

def Message processData(Message message) {
	
	def body = message.getBody(java.io.Reader);
	HashMap<Integer, String> hmap1 = new HashMap<Integer, String>();	
	map = message.getProperties();
	def Root = new XmlSlurper().parse(body);

    Root.Record.each{
        try{
        hmap1.put(Integer.parseInt(it.ExternalID.text().toString()), it.InternalID.text().toString());
        }
        catch(Exception ex){
            //do nothing, skip the record
        }
    }
	message.setProperty("hashmapOutput", hmap1);
	return message;
}

In the next 2 steps we are just logging the hashmap output.

The test payload with around 0.3 million key value pairs will be triggered to the iFlow.

Within 30 seconds we can get the output.

Message Processing in CPI-

The HashMap output with 0.3 million records will be stored in Properties in the below format-

Don’ts of XMLSlurper Parsing!!!

Often we commit the mistake of looping XMLSlurper object using our conventional and favourite ‘For’ statement. This drastically impacts the performance of the iFlow as shown below. This may not be an issue when your lookup size is within say 15000 records. The processing time increases exponentially for higher numbers and impacts iFlow performance.

Below is an example-

Code Snippet-

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.xml.*;
import java.util.regex.*;
import java.io.*;

def Message processData(Message message) {
	
	def body = message.getBody(java.io.Reader);  
	HashMap<Integer, String> hmap1 = new HashMap<Integer, String>();	
	map = message.getProperties();	
	def Root = new XmlSlurper().parse(body);

	for(int j=0; j<Root.Record.size(); j++){
        hmap1.put(Integer.parseInt(Root.Record.ExternalID[j].text()), Root.Record.InternalID[j].text());
	}
	message.setProperty("hashmapOutput", hmap1);
	return message;
}

As we can see below the performance of this interface was affected and it took almost 2 hours to process these records-

2nd Part of the scenario would be using the previously stored HashMap in Property to do a lookup to get the InternalID for the corresponding ExternalID in each of the records.

Below is the updated iFlow-

Advantages of XMLParser over XMLSlurper-

We would be using the XMLParser class instead of XMLSlurper since we wish to Read as well as Modify the XML at the same time. This isn’t possible in XMLSlurper since it has a Lazy way of Parsing xml files, i.e., if you update any field while in the loop, you must parse the whole document again, therefore this isn’t performance efficient. Using XMLParser, you can read and modify a XML at the same time with a single line of code.

Code Snippet-

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.xml.*;
import java.util.regex.*;
import java.io.*;

def Message processData(Message message) {
	
	def body = message.getBody(java.lang.String);
	def messageLog = messageLogFactory.getMessageLog(message);
	map = message.getProperties();
	def hmap1 = map.get("hashmapOutput");   
   	
   	def Root = new XmlParser().parseText(body);
	Root.Record.each{r->r.InternalID[0].value = 
        hmap1.get(Integer.parseInt(r.InternalID[0].text().toString()))}
       
	message.setBody(XmlUtil.serialize(Root));
        messageLog.addAttachmentAsString("FinalOutput:", body.toString(), "text/plain");   
   	return message;
}

Once ready, again the xml containing the 0.3 million key pair values are triggered to the iFlow-

 

Again the processing is seen to be quite efficient at close to 59 seconds. It should be noted that within this timeframe an xml of around 0.3 million records was read and parsed into a HashMap variable and then the HashMap was looked up against an xml with 30K business data records to update field (InternalID) of each record. This involved Read and Update operation on each of these records.

Logged Payloads-

Below is the sample of 2 of the records from the business payload which were modified (around 30K records in total).

 

Advantages as compared to Content Enricher-

  1. Multiple Key values can be concatenated and stored in the Key variable of a Hashmap.
  2. Can be used after a Looping process to add the collated values into a Hashmap to be referred anywhere in the iFlow
  3. Can be stored in Data stores to be accessible across iFlows

In the next blog I will cover the Json Parser method which can be used in cases where the response from your lookup service is in JSON format and you wish to avoid converting large payloads from XML to JSON format.

 

Regards,

Karthik Bangera

Assigned tags

      8 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Sriprasad Shivaram Bhat
      Sriprasad Shivaram Bhat

      Hello Karthik,

      Thanks for very informative blog !!

      Regards,

      Sriprasad Shivaram Bhat

      Author's profile photo Praveen Tirumareddy
      Praveen Tirumareddy

       

      Hi Karthik,

      thanks for the detailed blog. Very informative !!

      regards,

      Praveen T

      Author's profile photo Tibin Joseph
      Tibin Joseph

      Hi Karthik,

      Good Blog!!.. :)..

      Regards,

      Tibin Joseph

       

      Author's profile photo Sharma Ashwini
      Sharma Ashwini
      Hello Karthik,
      Thanks for very informative blog .

      Regards,

      Ashwini

      Author's profile photo Axel Albrecht
      Axel Albrecht

      Hi Karthik,

      great blog.

      Minor feeback: When fetching the properties you are not using a local variable map (--> def map).

      Could you kindly add this?

      https://blogs.sap.com/2017/06/22/avoid-binding-variables-in-groovy-scripts/

      thx, Axel

      Author's profile photo Karthik Bangera
      Karthik Bangera

      Hi Axel,

       

      Thanks for the feedback, however, unfortunately I can't edit it since the other account is inactive 🙁

       

      Cheers!

      Karthik

      Author's profile photo Axel Albrecht
      Axel Albrecht

      Hi Karthik,

      are the code snippets screenshots or text?

      I can check with the channel moderators to update the blog for you. If it's text it's a simple change. If it's a screenshot, could you kindly provide some updated screenshots please?

      regards,
      Axel

      Author's profile photo Karthik Bangera
      Karthik Bangera

      Hi Axel,

      The code snippets are texts. Sure, that would be great, thanks!

      regards,

      Karthik