Skip to Content
Technical Articles
Author's profile photo Eng Swee Yeoh

I *heart* Groovy mapping

Choices, choices, choices…

When working on CPI developments, there are a few options for developing mappings, i.e. the transformation of one message format to another. Cloud Integration mapping: Your options explained and compared by Morten Wittrock weighs the different options and their comparison.

On one hand, developers with a PI/PO background may gravitate towards Message Mapping due to familiarity with it.

On the other hand, developers from non-SAP background are normally more familiar with XSLT due to it being an open standard.

 

Nearly a year after my comment on Morten’s blog post, I am convinced that Groovy indeed is the way to go for anything but the simplest mapping (read: mostly 1-1) in CPI. I invite any and every CPI developer worth his/her salt to view my arguments below and consider adding this skill to their arsenal.

 

The Case for Groovy

Well, before I prove the case for Groovy, let me prove the case against its “opponents”.

i) Message Mapping (a.k.a Graphical Mapping)

  • Relies heavily on aΒ queue and context concept that I have only ever seen a handful of PI developers truly understand
  • Does not scale well with complexity (besides the concept above, Groovy is needed anyway if non standard functionality is required)
  • Complex logic is difficult to implement especially with the standard functionality. Back in PI days, it is not uncommon to see spiderweb-like mappings like below. It is difficult to decipher, harder to troubleshoot, impossible to enhance without breaking something!

  • Cannot be developed offline
  • Cannot be tested/simulated offline
  • WebUI test simulation cannot test mapping logic that accesses header/property
  • Does not scale well with increase of scenarios (how you do ensure logic for a new scenario does not break a previous scenario?)
  • I would even dare argue that this is baggage from PI days (an approach at least as old as 15 years) and best not to be used in any new CPI developments

 

ii) XSLT

  • Arguably the steepest learning curve
  • Typically limited to XML/Text to XML/Text transformation in the context of system-to-system integration (although it can handle other formats like HTML too)
  • Not easy to debug
  • To do anything substantial, a good XML editor is required and that typically comes with a price

 

Ok now, let’s now move to Groovy πŸ™‚

  • Pervasive in many areas of CPI (not just mapping) and therefore it is inevitable that a CPI developer will need Groovy irrespective of the mapping approach used
  • Capable of any to any transformation
  • Tools required for entire lifecycle of development are free or open-source
  • Scales well with complexity (due to flexibility of working directly with source code)
  • XML transformations are easy to developed (compared to Java) with the use of XmlSlurper for parsing and MarkupBuilder for generation (more details to follow below)
  • XmlSlurper has low memory footprint as it is based on SAX
  • Can be developed and tested offline
  • Mapping logic that accesses header/property can be tested via injection of these values from calling script
  • Can be debugged offline (with the help of an IDE like IntelliJ IDEA or Eclipse)
  • Transferrable skillset (i.e. to other integration solutions like Mulesoft or Dell Boomi) since payload parsing and generation are platform agnostic
  • And last but not least, it scales well with increase of scenarios when unit testing is implemented for each test case. And as a teaser, let me show you the outcome of running 8 tests in just over 3 seconds!

 

So if you are at least partially convinced or intrigued to find out more, let’s see how this can be achieved practically.

 

Parsing XML with XmlSlurper

As described in Parse XML easily with Groovy scripts, parsing XML can be achieved easily in Groovy using XmlSlurper. The contents of the fields can be accessed via dot notation. Below shows the comparison between how XML parsing is achieved in Java using DOM versus the simpler Groovy approach.

Java
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = factory.newDocumentBuilder();
Document doc = docBuilder.parse(inputStream);
NodeList nodeList = doc.getElementsByTagName("OrderNumber");
String orderNo = nodeList.item(0).getFirstChild().getTextContent();
Groovy
def root = new XmlSlurper().parse(reader)
def orderNo = root.Header.OrderNumber

Generating XML with MarkupBuilder

Similarly, MarkupBuilder in Groovy significantly reduces the line of codes for generating XML, as compared to using DOM in Java. The following example shows the comparison for generating a 2-level XML output.

Java
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = factory.newDocumentBuilder();
Document doc = docBuilder.newDocument();

Node root = doc.createElement("PurchaseOrder");
doc.appendChild(root);
root.appendChild(doc.createElement("Header"));

Transformer transformer = TransformerFactory.newInstance().newTransformer();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
transformer.transform(new DOMSource(doc), new StreamResult(baos));
String output = new String(baos.toByteArray());
Groovy
def writer = new StringWriter()
def builder = new MarkupBuilder(writer)
builder.PurchaseOrder {
  'Header'{ }
}
def output = writer.toString()

 

Sample scenario

Consider a sample integration scenario with the following requirements:-

  • Transform payload from source schema to target schema
  • Convert date format (dd-MM-yyyy to yyyyMMdd)
  • Populate target field DocumentType based on message property (that is configurable via externalised parameter) – Z001 in this example
  • Filter only items where IsBatchParent = true

Below are examples of the input payload and expected output payload:-

Input
<Order>
    <Header>
        <OrderNumber>OrderXYZ</OrderNumber>
        <Date>18-02-2019</Date>
    </Header>
    <Item>
        <ItemNumber>10</ItemNumber>
        <MaterialNumber>MatABC</MaterialNumber>
        <Quantity>57</Quantity>
        <IsBatchParent>true</IsBatchParent>
    </Item>
    <Item>
        <ItemNumber>20</ItemNumber>
        <MaterialNumber>MatABC</MaterialNumber>
        <Quantity>57</Quantity>
        <IsBatchParent>false</IsBatchParent>
    </Item>
</Order>
Output
<PurchaseOrder>
    <Header>
        <ID>OrderXYZ</ID>
        <DocumentDate>20190218</DocumentDate>
        <DocumentType>Z001</DocumentType>
    </Header>
    <Item>
        <ProductCode>MatABC</ProductCode>
        <Quantity>57</Quantity>
    </Item>
</PurchaseOrder>

 

Bringing it all together

With the basics in place, let us see how to bring it all together based on the requirements of the scenario.

Requirement Approach
Transform payload from source schema to target schema Utilise XmlSlurper to parse the input XML, and MarkupBuilder to generate the output XML
Convert date format (dd-MM-yyyy to yyyyMMdd) Use Java 8’s LocalDate to parse and format the date

def inputDate = LocalDate.parse(input, DateTimeFormatter.ofPattern('dd-MM-yyyy'))
def outputDate = inputDate.format(DateTimeFormatter.ofPattern('yyyyMMdd'))
Populate target field DocumentType based on message property Access via the Message property

Map properties = message.getProperties()
def docType = properties.get('DocType')
Filter only items where IsBatchParent = true Use GPathResult’s findAll method by providing a Closure with the appropriate filter criteria

def validItems = Order.Item.findAll { item -> item.IsBatchParent.text() == 'true' }

With all of this in place, the final source code is listed below:-

import com.sap.gateway.ip.core.customdev.util.Message
import groovy.xml.MarkupBuilder
import java.time.LocalDate
import java.time.format.DateTimeFormatter

Message processData(Message message) {
    // Access message body and properties
    Reader reader = message.getBody(Reader)
    Map properties = message.getProperties()

    // Define XML parser and builder
    def Order = new XmlSlurper().parse(reader)
    def writer = new StringWriter()
    def builder = new MarkupBuilder(writer)

    // Define target payload mapping
    builder.PurchaseOrder {
        'Header' {
            'ID'(Order.Header.OrderNumber)
            def inputDate = LocalDate.parse(Order.Header.Date.text(), DateTimeFormatter.ofPattern('dd-MM-yyyy'))
            'DocumentDate'(inputDate.format(DateTimeFormatter.ofPattern('yyyyMMdd')))
            'DocumentType'(properties.get('DocType'))
        }
        def validItems = Order.Item.findAll { item -> item.IsBatchParent.text() == 'true' }
        validItems.each { item ->
            'Item' {
                'ProductCode'(item.MaterialNumber)
                'Quantity'(item.Quantity)
            }
        }
    }

    // Generate output
    message.setBody(writer.toString())
    return message
}

 

Note: When a particular node from the source payload is used other than direct assignment to the target node, it needs to be transformed to a text field by adding .text() method at the end of the node.

 

Setting up Unit Tests

Next, to make it worth the effort, we will set up the above scenario as a test case for unit testing. Once again I will use the Spock-based approach, such that once we have a successful test case, there will be a baseline to ensure any future changes to the Groovy script would not break an existing scenario.

The development will be maintained as an IntelliJ project based on the below shown directory structure as described in my previous post, Pimp My Groovy – boosting CPI Groovy developments with IntelliJ IDEA. The script under test will be OrderToPurchaseOrder.groovy, and the Spock specification will be OrderToPurchaseOrderSpec.groovy. The input XML payload and expected XML output payload are maintained as resources in the project.

Following is the source code for the Spock specification, with the single test case. The input file is stored into the message body together with the parameter for DocType. The Groovy mapping script is then executed and the results compared with the provided output file.

package src.test.groovy

import com.sap.gateway.ip.core.customdev.processor.MessageImpl
import com.sap.gateway.ip.core.customdev.util.Message
import org.apache.camel.CamelContext
import org.apache.camel.Exchange
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.impl.DefaultExchange
import spock.lang.Shared
import spock.lang.Specification

class OrderToPurchaseOrderSpec extends Specification {

    Message msg
    Exchange exchange
    @Shared script

    def setup() {
        // Load Groovy Script
        GroovyShell shell = new GroovyShell()
        script = shell.parse(new File('src/main/resources/script/OrderToPurchaseOrder.groovy'))

        CamelContext context = new DefaultCamelContext()
        exchange = new DefaultExchange(context)
        msg = new MessageImpl(exchange)
    }

    def 'Purchase Order Mapping for Document Type Z001'() {
        given:
        //--------------------------------------------------------------
        // Initialize message with body, header and property
        def body = new File('src/test/resources/Order.xml')
        msg.setProperty('DocType', 'Z001')
        //--------------------------------------------------------------

        exchange.getIn().setBody(body)
        msg.setBody(exchange.getIn().getBody())

        when:
        // Execute script
        script.processData(msg)

        then:
        msg.getBody() == new File('src/test/resources/OrderOutput.xml').text
    }
}

Finally, once the Spock test is executed in IntelliJ, the below results are shown in the console.

 

Conclusion

Unashamedly, I love Groovy and the boundless options it provides in the context of CPI development. Specifically in the area of mapping, Groovy mappings are my first preference given the ease of implementation, supported by the right toolset. It further allows us to safeguard our development with the easy incorporation of unit testing capability.

If you are contemplating the best approach for performing mapping in CPI, I strongly recommend Groovy mapping.

 

Assigned tags

      40 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Morten Wittrock
      Morten Wittrock

      Awesome post, Eng Swee! I've linked to it from my mapping overview post.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Thanks Morten. Appreciate that!

      Author's profile photo Sriprasad Shivaram Bhat
      Sriprasad Shivaram Bhat

      Hello Eng Swee,

      Thanks for the Informative blog.Whenver it comes to comparison usually I refer to below groovy goodness blog series by mrhaki which provokes unconditional love towards groovy?

      https://mrhaki.blogspot.com/2009/10/groovy-goodness-reading-xml.html?m=1

      Regards,

      Sriprasad Shivaram Bhat

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Sriprasad

       

      Totally agree. I am still learning new things about Groovy all the time, and definitely find the Groovy Goodness examples very useful! πŸ™‚

      Author's profile photo Daniel Graversen
      Daniel Graversen

      Hi Eng,

      Yes Groovy can help out mapping in a lot of places, and I think we will see it in a lot places for mappings.

      I do think that XSLT is also a good option for more complex mappings. I have been using eclipse for it and it gives some option to run it. It does have some limitations on debugging and is possible in it.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Daniel

       

      I don't disagree that XSLT is a valid option. But for me, if a CPI developer that is new to both XSLT and Groovy and has limited time/resource to just pick up one - I would say "go for Groovy" for all the reasons I stated above.

      Author's profile photo Moe Alhennawi
      Moe Alhennawi

      Hi Eng,

      Your passion for Groovy is shining through!

      Given your experience, what would be your recommended learning resource(s) for integration-related groovy development especially for beginners?

      Thank you!

      Mohammad

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Mohammad

       

      Excellent question indeed!

       

      Personally I frequently refer to the official Apache Groovy Documentation. The sections that particularly stand out for me are:-

      Differences with Java - good for those coming from a Java background

      Style Guide - provides you guideline such that your code are indeed more Groovy than it is Java

      Processing XML - especially useful for integration related development

       

      If you are looking for books, you can consider Groovy in Action although I never got around to finishing it myself!

       

      Lastly, I'd shamelessly recommend going through my other Groovy related blog posts as they deal with Groovy scripts in the context of CPI development.

      (How) Do You Test Your Groovy Scripts?

      CPI’s Groovy meets Spock – To boldly test where none has tested before

      Pimp My Groovy – boosting CPI Groovy developments with IntelliJ IDEA

       

      Regards

      Eng Swee

       

       

      Author's profile photo Morten Wittrock
      Morten Wittrock

      I think this answer could be expanded into a very nice blog post πŸ™‚

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Or maybe a TechEd session πŸ™‚

      Author's profile photo Morten Wittrock
      Morten Wittrock

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Yes... or both! πŸ˜‰

      Author's profile photo Moe Alhennawi
      Moe Alhennawi

      Thank you Eng!

      This is a good starting point and helps narrow the sea of information available online for beginners. I won't discourage you from acting on Morten's reply here πŸ˜‰

      Mohammad

      Author's profile photo Adam Kiwon
      Adam Kiwon

      Great blog, I will add your Groovy blogs to the additional SAP training material of CLD900.

      Just to be complete: XSLT also supports XML to TXT conversion and also allows to invoke methods of external Java classes.

      Best regards, Adam

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Thanks, I've updated the part on XSLT for completeness sake.

      Author's profile photo Morten Wittrock
      Morten Wittrock

      Strictly speaking, you can also do text to text/XML/HTML transformations in XSLT πŸ˜€ You'd probably never do so, but it's possible.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Ok ok... I've reworded that section such that the focus is mainly in the context of integration.

       

      But maybe you, Daniel Graversen and/or Adam Kiwon should come up with an "I heart XSLT mapping" blog post so that XSLT feels some love too :p

      Author's profile photo Morten Wittrock
      Morten Wittrock

      Point taken πŸ™‚

      Author's profile photo Adam Kiwon
      Adam Kiwon

      There is definitely some space for XSLT in my heart πŸ™‚

      Author's profile photo Daniel Graversen
      Daniel Graversen

      Hi

      I have created this post on XSLT

      https://blogs.sap.com/2019/06/14/i-heart-xslt-mappings/

      I think XSLT also deserves some space.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Thanks Daniel for being such a good sport, and taking up the challenge! ?

      Β 

      It’s great to have such open debate on this area, and my hope is that the community will benefit from it.

      Author's profile photo Daniel Graversen
      Daniel Graversen

      You are welcome.

      Then we just miss the I *heart* message mappings :).

      Author's profile photo Morten Wittrock
      Morten Wittrock

      Except nobody does.... πŸ˜‰

      Author's profile photo Daniel Graversen
      Daniel Graversen

      so true, in a CPI context, you are right.

       

      Author's profile photo Craig Cmehil
      Craig Cmehil

      Might be of interest to you, https://blogs.sap.com/2019/06/13/submit-your-sessions-for-sap-teched-2019/ πŸ™‚

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Thanks for the info, Craig!

       

      Game on! πŸ˜‰

      Author's profile photo srinivas sistu
      srinivas sistu

      Excellent blogs... book marked and downloaded both the blogs Groovy and XSLT (https://blogs.sap.com/2019/06/14/i-heart-xslt-mappings/)

       

      Author's profile photo M. Jaspers
      M. Jaspers

      Hi Eng,

      Excellent blog and I absolutely agree with you.

      Like you, I don't have too much XSLT mapping experience so for me Groovy is a lot easier too.

      I have used it in quite some cases where I could not get the context changes correctly in the output when using message mappings, e.g. because the input data had less context changes than the output required. With Groovy I could easily keep the data together and create the correct output.

      I really also like the way you set up the unit test. I will definitely look at how to implement that too.

      Thanks,

      Martin

       

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Martin

       

      Thanks for your comment. Glad to hear that you have found Groovy to be easier - in some ways it also gives you a WYSIWYG overview of all your mapping logic.

       

      Regards

      Eng Swee

      Author's profile photo playsuji s
      playsuji s

      Hi Eng,

       

      I love this blog, it has every idea to discuss about the mapping and good to discuss to about the best (I suppose I ll never come to conclusion).

      I usually kinda use all three methods 1. Graphical 2. Groovy 3. XSLT but with majority of Graphical mapping. This blog opens my idea to explore further on XMLSlurper in the groovy script which If I think might be easier than other methods.

      Anyway, thank you very much for this blog. Its Very Helpful!

       

      Thank You,

      playsuji

      Author's profile photo Sudeep Menakuru
      Sudeep Menakuru

      Hi Eng,

      With the XMLSlurper and XMLParser, lot of functionalities are not working. For example if you use each, then you can’t break the occurrences or validations. By the moment each triggered, it will validate all the occurrences parallel. Instead why can’t we use JDOM parser and implement the scripts?

      Β 

      Thank you!

      BR,

      Sudeep Menkuru.

      Author's profile photo Michael Schliebner
      Michael Schliebner

      Thank you Eng!

       

      missing the *heart* message? <3 should do

      <3Β  πŸ˜‰

      Author's profile photo Saurabh Kumar
      Saurabh Kumar

      Thanks Eng!!

      This post is really helpful for me.

      Author's profile photo Christine Smith
      Christine Smith

      Hi Eng,

      I agree that groovy is great but I have an issue...

      I have an incoming XML via a Soap connector and I wrote some groovy to convert the body into JSON. However it is failing when I try and send it out and I think it is because it still sees the package as a SOAP package. How do I change it to be a JSON package?

      Any ideas?

      Author's profile photo Ryan Crosby
      Ryan Crosby

      Hi Eng Swee,

      I am on about day 4-5 on CPI so I'm starting out in a whole new world.Β  I'm spending a lot of my time these days reading help files, blogs, etc. and I'm interested to see how the Groovy mapping compares to the stuff I have used in the olden days with PI/PO.Β  The one thing that comes to my attention with my past experience with graphical mapping is that it is GUI based and the code is generated whereas Groovy requires the developer to write the code directly.Β  It may seem counterintuitive to say but for each and every effort I look for how it can be completed while writing as little code as possible, and use the tools that I have at my disposal.Β  I find that that was the power of graphical mapping because in the past I had some very complex mappings that if one has a strong understanding of the queues, contexts, qualifiers, etc., then it can be achieved without the spider web shown above.Β  I have a new s-user id after switching roles recently so I haven't been very active while getting my bearings.Β  Good to see some of the familiar faces paving the way on the latest tools that are out there πŸ™‚

       

      Cheers,

      Ryan Crosby

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Ryan,

       

      Glad to hear you are diving into this (not so) new world! πŸ˜‰ Yes, there are differences but after a while I'm sure you will get the hang of it.

       

      The discussion on which approach to use for mapping is a loaded one - I must admit that being a hard-core programmer myself, I come from a very "pro-code" perspective as opposed to "low/no-code". At the end of the day, it is up to each developer/organisation to decide which way to go.

       

      I will continue to try and convince those who cross my path of the Groovier way of doing things, because queues and contexts are so out-of-date and cumbersome. And I'll continue to push the boundaries of doing things better and smarter wherever it makes sense.

       

      Have fun!

      Eng Swee

      Author's profile photo T. van Rooijen
      T. van Rooijen

      Hi Eng Swee,

      I promised myself I would do my next mapping in groovy. well the time has come for another Successfactors compoundemployee mapping, those are always fun...especially in delta mode.

      Now of course I'm busy setting up the intelliJ IDE and am somewhat stuck in finding the correct sap jars (as I knew I probably would).
      I know the blog from vadim on how to get the jars and also the file explorer blog but since the jars are bundled differently I cannot findΒ  the jar/jars anymore.

      This for instance is the response on the class location for: MessageImpl:

      Fully qualified class name: com.sap.gateway.ip.core.customdev.processor.MessageImpl

      JAR file containing class file: jar:bundle://455.0:0/!/

      Also the directory /usr/sap/ljs/plugins doesn't seem to be available anymore. this is my (NEO) view of the directory structure:

      I've looked at the script API on the CPI tools page but that only seems to contain 3 classes, so that look to be nowhere near complete. I definitely miss the MessageImpl class in there.

      So at the moment I am not sure where to look anymore and could use a little nudge in the right direction πŸ™‚

      Thanks!

      Tom

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Tom

       

      Unfortunately, things have changed since this post was written. While the approach still works, it needs further tweaking based on how things are at the moment.

       

      Instead of using this approach, in our SAP PRESS E-Bite on Developing Groovy Scripts for SAP CPI, we introduced a different approach where you create your own mock version of the Message class. This is simpler than "hunting" for all the different JAR files that you need to set it up.

       

      May I suggest you check that out instead - not because I'm trying to sell the E-Bite 😝, but I truly believe it will be easier for you.

       

      Regards

      Eng Swee

      Author's profile photo T. van Rooijen
      T. van Rooijen

      Hi Eng Swee,

      That was the right nudge...especially since I already got that e-bite a while ago...

      The initial testing with the mock class already works, now on to the Spock approach where I hope and think a lot can be gained.

      Thanks again!

      Tom

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Tom,

       

      I"m pleased to hear that you got it working with the mock class. Hope our E-Bite will come in handy as you venture with more Groovy development. I hope you are able to grasp the power that Groovy provides you as you try it out on the compound employee mapping πŸ˜‰