Unlocking Hashmap Magic: Efficient Bulk Data Lookup with Value Mapping in SAP CI using Groovy/XSLT

AmanVarshney · ‎01-31-2024

Problem Statement:

In the SAP Cloud Integration (CI) landscape, it's imperative to ensure robust data persistence and maintain data integrity across versions within the Value Mapping framework. This involves facilitating efficient bulk data lookup and accurate value validation. In scenarios where values from the source payload are unavailable during the lookup process, a seamless mechanism should be in place to replace them with empty entries, orchestrated through Groovy/XSLT transformations.

Challenges:

Bi-directional Value Mapping Storage: How to efficiently store XML field names, values, and their versions (multi-map) in a bi-directional Value Mapping system.
Efficient Lookup and Validation: Ensuring that the entire list from Value Mapping is readily accessible for lookup and validation of incoming XML field values with their existing versions while mapping.
Performance Considerations: Designing an interface that performs well for handling huge datasets, considering the potential volume of data involved.

Design Solution:

The data is stored in value mapping as below

Operations Team in future can easily validate the field values and provide maintenance. Value mapping api's can also be used to automate the maintenance.

The iflow is designed as below, for experimental purposes I have tried the approach with XSLT and Groovy both

Explanation:

I endeavored to create an integration flow (iflow) that can be triggered via HTTP, with the payload externally initiated (via tools like Postman, etc.).

The objective is to perform a lookup in the value mapping for all fields (specifically, field1 and field2) within segment1, excluding the version field. The version and field values together constitute the key for the lookup in the Value Mapping.

In the resulting output, the first occurrence of field2 within segment1 is anticipated to be empty, as it does not exist in the Value Mapping for version 1.0.0.

I have found both approaches, utilising hashmap, to be highly performant for achieving the current objective of multi-field lookup in the Value Mapping, especially when dealing with large datasets

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <segment1>
        <version>1.0.0</version>
        <field1>Abc</field1>
        <field2>Def</field2>
    </segment1>
    <segment1>
        <version>1.0.1</version>
        <field1>Abc</field1>
        <field2>Def</field2>
    </segment1>
    <segment2>
        <field3>.....</field3>
        <field4>.....</field4>
    </segment2>
</root>

Groovy Approach:

1) The received XML Payload is stored in the property for later use.

2) I used the below Value Mapping API from SAP Business Accelerator Hub to fetch the list of all values from the value mapping
/ValueMappingDesigntimeArtifacts(Id='{Id}',Version='{Version}')/ValMapSchema(SrcAgency='{SrcAgency}'...

Spoiler

Please note that for successful trigger to this API you need Process Integration Runtime credential with api plan from SAP Business Technology Platform

Please note that for successful trigger to this API you need Process Integration Runtime credential with api plan from

3) The below Groovy script function createHashmap saves all the values as hashmap in the format key=value

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.xml.*;
import java.util.regex.*;
import java.io.*;

def Message createHashmap(Message message) {
	
	def body = message.getBody(java.io.Reader);
	HashMap<String, String> hmap1 = new HashMap<String, String>();	
	map = message.getProperties();
	def Root = new XmlSlurper().parse(body);

    Root.entry.each{
        try{
            hmap1.put(it.content.properties.Value.SrcValue.text().toString(), it.content.properties.Value.TgtValue.text().toString());
        }
        catch(Exception ex){
            //do nothing, skip the record
        }
    }
	message.setProperty("hashmapOutput", hmap1);
	return message;
}

4) The main initial XML payload from the sender is set for further hashmap lookup through Groovy

5) The below groovy function lookupAndReplaceData looksup for all the fields in segment1 (except for version) for all its occurrence and validates its existence in the hashmap else sets empty for that specific field.

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.xml.*;
import java.util.regex.*;
import java.io.*;

def Message lookupAndReplaceData(Message message) {
	
	def body = message.getBody(java.lang.String);
	def messageLog = messageLogFactory.getMessageLog(message);
	map = message.getProperties();
	def hmap1 = map.get("hashmapOutput");   

   	def Root = new XmlParser().parseText(body);
   	
   	Root.'**'.findAll { it.name() == 'segment1' }.each{segment1 ->
   	
   	    def version = segment1.version[0].text().toString();
   	    
        segment1.children().each { node ->
        if (node.name() != 'version') {
            def fieldValue = node.text().toString();
            def nodeName = node.name().toString();
                try{
                    node.value = hmap1.get(nodeName+"|"+version+"|"+fieldValue)?:"";
                }catch(Exception ex){
                    node.value = "";
                }
            }
        }
	}
   	
	message.setBody(XmlUtil.serialize(Root));
   	return message;
}

The execution on SAP Integration Suite Cloud Integration is as below

Here the lookup happened for key field2|1.0.0|Def which was set as empty as it do not exist

XSLT Approach:

Explanation for Steps 1-4 remains same except the last XSLT enrichment step 6.

6) The below xslt performs the same enrichment logic as groovy. It reads the hashmapOutput property and uses map function to get the value of key, for eg. field2|1.0.0|Def

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:s0="urn:sap-com:document:sap:rfc:functions" xmlns:cpi="http://sap.com/it/" exclude-result-prefixes="cpi" version="2.0">
    
    <!-- include exchange parameter -->
	<xsl:param name="exchange"/>
    <!-- Access properties -->
    <xsl:param name="hashmapOutput"/>
    
    <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
    <xsl:strip-space elements="*"/>
	
	<xsl:template match="/">
        <!-- set headers -->
        <xsl:value-of select="cpi:setHeader($exchange, 'context', 'ModelingBasics-HeaderPropertiesInXSLT')"/>
        <xsl:value-of select="cpi:setHeader($exchange, 'content-type', 'application/xml')"/>

        <xsl:apply-templates select="node()"/>
    </xsl:template>
	
    <!-- Identity template: copy all nodes and attributes -->
    <xsl:template match="@* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()"/>
        </xsl:copy>
    </xsl:template>
    
    <!-- Modify field1 inside segment1 -->
    <xsl:template match="//*[local-name()='segment1']/*[not(local-name()='version')]">
        <xsl:copy>
            <!-- Variable declaration -->
            <xsl:variable name="nodeName" select="local-name()" />
            <xsl:variable name="version" select="../*[local-name()='version']" />
            <xsl:variable name="nodeValue" select="."/>
            
            <xsl:value-of select="map:get($hashmapOutput, concat($nodeName,'|', $version, '|', $nodeValue))" xmlns:map="java:java.util.Map"/>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

The execution on SAP Integration Suite Cloud Integration is as below

Disclaimer:

I want to clarify that the content presented here represents my attempt to enhance performance and simplify development processes. I drew inspiration from blogs like Implementing dynamic Lookups in CPI using Hashmap and lightweight XML parsers nd leveraged insights from various other sources. My aim was to extend these approaches to XSLT transformations, as highlighted in Value Mapping usage in XSLT Mappings with CPI and SAP Help Portal: Access Header and Properties in XSLT Mapping

It's important to note that there are numerous potential improvements and alternative approaches to this solution, depending on the specific use cases and requirements at hand.

Your feedback and suggestions are highly valuable to me. I would greatly appreciate any comments or insights you may have! 😊