Skip to Content
Technical Articles
Author's profile photo Dirk Ostertag

New Feature in Integration Advisor: Reordering of Source Structure

Introduction

If you are familiar with doing complex mappings in SAP Integration Suite, it might be of interest for you, that Integration Advisor Capability of SAP Integration Suite now introduces a new powerful feature called pre-transformation of source Message Implementation Guide structure. With this new feature Integration Advisor now supports the reordering of source structure before the actual mapping is taking place.

In certain integration scenarios it is inevitable to apply specific structural changes to the incoming data structure. This is usually the case if the source and target side structures differ that much, that a simple 1:1 mapping between hierarchy levels on both sides is not possible.To cope with such situations, Integration Advisor now allows to apply structure modifying operation prior to the actual mapping is taking place. This new and optional transformation is called Pre-transformation because at runtime it is applied to the message payload before the actual transformation is applied and it is applied to the Message Implementation Guide at design time also before the mapping utilizes the transformed source structure.

By adding a Pre-transformation to the Mapping Guideline it is now possible to tailor the incoming data structure to fit the structural requirements of the outgoing data structure in integration scenarios where this is required. The result of the Pre-transformation then is used in the Mapping Guideline as the source side Message Implementation Guide while the original source side MIG becomes the input for the Pre-transformation. The entire functionality, that is required to develop and simulate Mapping Guidelines with Pre-transformation is encapsulated in the new version of the Mapping Guideline Editor in Integration Advisor.

The export of runtime artifacts for SAP Cloud Integration automatically contain all relevant entities for executing the Pre-transformation along with the main mapping. Hence, it is completely transparent to the executing integration flow whether an IA mapping contains a Pre-transformation or not. No changes or extensions to any integration flow are required for the Pre-transformation to be executed along with the main transformation. This way, Pre-transformation integrates perfectly with all existing integration flow, that might already be in use for running Integration Advisor mappings.

The first two operations

The first version of Pre-transformation in Integration Advisor comes with two operations. The aim of the provided Pre-transformation operations is to encapsulate complex structural changes into one single step whenever possible.

Further operations are already on the roadmap.

Copy Referenced Node

This operation allows to create a copy of a group node into a different group node, that is not the parent of this original group node. The copy operation itself is controlled by a reference between the copied node and the receiving group node. The user is free to choose that reference nodes. As a result, only such group nodes are copied into such receiving group nodes, where the two reference nodes contain the same value. Reference nodes always must be descendants of the involved group nodes (the copied node called “referenced node” and the “receiving node”).

For instance, a source structure contains Items and corresponding Handling Units as sibling. Though the target structure requires the Items to be children of the corresponding Handling Units. Whenever it is possible in the source structure to derive a reference between the corresponding instances of Items and Handlings Units for example because both contain an ID, that allows to link them together, the operation Copy Referenced Node can be used to copy the items into the correct Handling Units.

Group by Key

The purpose of this operation is to allow grouping of group nodes according to properties of these pre-existing group nodes. This property must be an {descendant} {{leaf}} node of the group that shall be grouped and is called the “group key node”. For all instances of a group with the identical value in group key node, a new supergroup node will be created carrying a distinct group key. This new group collects all instances of the original group with one specific value in group key node. As the result of this operation, the structure will contain one instance of the newly created supergroup for each distinct value of group key node. All instances of the original group node will be moved into the super group with the value of group key corresponding to the value of the group key node of this original group.

Assuming a source structure contains an arbitrary number of handling units of different types as sibling nodes on the same hierarchy level. Further on these units can appear in arbitrary order while the target structure required such units to be in order corresponding to their type. In this case the handling unit itself would be the group to be grouped into newly to be created instances of the super group. The type of the unit would become the Id to be used for grouping. The result of the operation is one instance of the supergroup for each distinct type of handling unit. All handling units with the corresponding type will be contained in their matching supergroup.

A simplified real world example

Initial situation is like shown in the image below. The shipment on the source side contains the handling unit elements as sibling nodes whereas the target side structure requires the items to be contained in their corresponding handling units. A reference between both types of nodes can be derived by correlating HandlingUnit/unitid to Item/handlingUnitId. Further on inside a handling unit different types of Items need to be grouped by their type.

Two%20different%20structures%20that%20cant%20directly%20be%20mapped%20to%20each%20other

Two different structures that cant directly be mapped to each other

The required correlation and the structural change can be applied by using Pre-Transformation operation copy referenced node.

To achieve this, a Pre-transformation needs to be added to the Mapping Guideline on the Overview tab of the Mapping Guideline. While in edit mode clicking on the plus sign near “Pretransformation” a new Pre-transformation is added to the Mapping Guideline.

Adding%20the%20new%20PTS

Adding the new PTS

Right after clicking the add button, the Mapping Guideline editor switches to the newly added Pre-transformation showing the unchanged source structure on both sides.

After applying the operation copy referenced node to the Item Node and clicking on Execute, a copy of the original Item node is created as child of the handling unit node.

Operation%20Copy%20Referenced%20Node%20created%20a%20copy%20of%20the%20Item%20node%20inside%20the%20HandlingUnit%20node

After simulation with a payload, that contains three handling units and three different Items  from which one belongs to the first, 2 belong to the second and none belongs to the third handling unit, the result looks like in the three screenshots below.

The%20small%20item%20has%20been%20added%20to%20the%20small%20box

The small item has been added to the small box

 

The%20medium%20sized%20items%20have%20been%20added%20to%20the%20medium%20sized%20box

The medium sized items have been added to the medium sized box

The%20big%20box%20remains%20empty

The big box remains empty

Until here the first requirement of assigning each item to the corresponding handling unit is met. However, the target structure (see first screenshot) also requires items to be grouped inside the handling units according to one of their attributes. In this simplified example this attribute is the type of the item. This can easily be achieved by applying operation group by key to the Items inside the handling unit.

Items%20of%20different%20types%20are%20grouped%20inside%20the%20second%20handling%20unit.

Items of different types are grouped inside the second handling unit.

For the second handling unit which actually contains items of two different types, this leads to the creation of two distinct groups of items inside the corresponding handling unit. The fist handling unit in turn will only contain one items group, as it inly contains one distinct type of items. The third Handling unit still remains empty because it doesn’t contain any items to be grouped.

 

Conclusion

As shown in the picture below, to apply certain structural changes to the source structure of a Message Implementation Guide reduces the main mapping in this case to a simple 1:1 mapping. All this can be done without writing a single line of code. The resulting transformation automatically becomes part of the runtime artifacts and integrates seamlessly with existing integration scenarios respectively integration flows.

Simple%201%3A1%20mapping%20using%20the%20transformed%20structure%20of%20the%20original%20source%20MIG

Simple 1:1 mapping using the transformed structure of the original source Message Implementation Guide

 

Please feel free to drop any questions and comments.

For more details, please visit

Integration Advisor – integration game changer

Integration Advisor: Overview of components for building B2B integration content and further reading.

Appendix

Example data used in this example

<DispatchNotificationSource>
    <Header>
        <messageId>0815471142</messageId>
        <senderId>Sender Company</senderId>
        <recipientId>Recipient Number</recipientId>
    </Header>
    <Shipment>
        <HandlingUnit>
            <handlingUnitType>small Box</handlingUnitType>
            <unitId>001</unitId>
            <name>Small Box</name>
        </HandlingUnit>
        <HandlingUnit>
            <handlingUnitType>medium Box</handlingUnitType>
            <unitId>002</unitId>
            <name>Medium Box</name>
        </HandlingUnit>
        <HandlingUnit>
            <handlingUnitType>big Box</handlingUnitType>
            <unitId>003</unitId>
            <name>Big Box</name>
        </HandlingUnit>
        <Item>
            <itemType>small item</itemType>
            <itemId>100</itemId>
            <handlingUnitId>001</handlingUnitId>
            <name>Small Item</name>
        </Item>
        <Item>
            <itemType>small item</itemType>
            <itemId>100</itemId>
            <handlingUnitId>001</handlingUnitId>
            <name>Small Item</name>
        </Item>
        <Item>
            <itemType>light item</itemType>
            <itemId>200</itemId>
            <handlingUnitId>002</handlingUnitId>
            <name>Medium sized Item</name>
        </Item>
        <Item>
            <itemType>heavy item</itemType>
            <itemId>200</itemId>
            <handlingUnitId>002</handlingUnitId>
            <name>Another medium sized Item that is heavy</name>
        </Item>
        <Item>
            <itemType>heavy item</itemType>
            <itemId>200</itemId>
            <handlingUnitId>002</handlingUnitId>
            <name>Yet another medium sized Item that is heavy</name>
        </Item>
    </Shipment>
</DispatchNotificationSource>

Source side structure

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

    <xs:complexType name="messageHeader">
        <xs:sequence>
            <xs:element name="messageId" type="xs:string" maxOccurs="1"/>
            <xs:element name="senderId" type="xs:string" maxOccurs="1"/>
            <xs:element name="recipientId" type="xs:string" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="handlingUnit">
        <xs:sequence>
            <xs:element name="handlingUnitType" type="xs:string" maxOccurs="1"/>
            <xs:element name="unitId" type="xs:string" maxOccurs="1"/>
            <xs:element name="name" type="xs:string" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="item">
        <xs:sequence>
            <xs:element name="itemType" type="xs:string" maxOccurs="1"/>
            <xs:element name="itemId" type="xs:string" maxOccurs="1"/>
            <xs:element name="handlingUnitId" type="xs:string" maxOccurs="1"/>
            <xs:element name="name" type="xs:string" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="shipment">
        <xs:sequence>
            <xs:element name="HandlingUnit" type="handlingUnit" maxOccurs="unbounded"/>
            <xs:element name="Item" type="item" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="dispatchNotification">
        <xs:sequence>
            <xs:element name="Header" type="messageHeader" maxOccurs="1"/>
            <xs:element name="Shipment" type="shipment" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:element name="DispatchNotificationSource" type="dispatchNotification"/>

</xs:schema>

 

Target side structure

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

    <xs:complexType name="messageHeader">
        <xs:sequence>
            <xs:element name="messageId" type="xs:string" maxOccurs="1"/>
            <xs:element name="senderId" type="xs:string" maxOccurs="1"/>
            <xs:element name="recipientId" type="xs:string" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="handlingUnit">
        <xs:sequence>
            <xs:element name="handlingUnitType" type="xs:string" maxOccurs="1"/>
            <xs:element name="unitId" type="xs:string" maxOccurs="1"/>
            <xs:element name="name" type="xs:string" maxOccurs="1"/>
            <xs:element name="ItemGroup" type="itemGroup" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="itemGroup">
        <xs:sequence>
            <xs:element name="itemTypeInThisGroup" type="xs:string" maxOccurs="1"/>
            <xs:element name="Item" type="item" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>
    
    <xs:complexType name="item">
        <xs:sequence>
            <xs:element name="itemType" type="xs:string" maxOccurs="1"/>
            <xs:element name="itemId" type="xs:string" maxOccurs="1"/>
            <xs:element name="handlingUnitId" type="xs:string" maxOccurs="1"/>
            <xs:element name="name" type="xs:string" maxOccurs="1"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="shipment">
        <xs:sequence>
            <xs:element name="HandlingUnit" type="handlingUnit" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="dispatchNotification">
        <xs:sequence>
            <xs:element name="MessageHeader" type="messageHeader" maxOccurs="1"/>
            <xs:element name="Shipment" type="shipment" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:element name="DispatchNotificationTarget" type="dispatchNotification"/>

</xs:schema>

Assigned Tags

      5 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Robert Quindt
      Robert Quindt

      Hi Dirk,

      Thanks for the blog post. Is there a way to view/suggest/vote on additional/future operations?

      I like to use the Integration Advisor to create mappings as it helps to document mappings and I prefer it to graphical mappings.

      At the moment I have to use some custom pre-transformations to handle some cases that I would like to see in this new feature.

      Splitting a single node into multiple nodes. I have a "description" node with up to 4000 characters that I need to map to an IDOC E1EDKT1 segment with several E1EDKT2 segments. At the moment I use a script to split the input value into chunks of up to 70 characters and create a "descriptions" node with "description" nodes. I can then map these nodes to the E1EDKT1/E1EDKT2 structure.

      Copy values from several nodes to a single one, then split them to a specific length. In this case I have the EDIFact FTX segment which has a C108 group with up to 5 4440 fields. The xml representation shows each 4440 field with multiplicity 0..1 instead of a single 0..5 field. So it is not possible to create E1EDKT2 segments for the IDOC directly, as the multiplicities of the source and target do not match. It would be nice to have the possibility to copy information from several nodes into a temporary node or variable in the pre-transformation to split it as in the first case, or to use several input nodes with multiplicity 0..1 to create a node with multiplicity 0..n.

       

      Best regards

      Robert

      Author's profile photo Stefan Boller
      Stefan Boller

      Hi Robert,

      thanks for using Integration Advisor. Regarding your first requirement, please, try to use our String Processing. If I got you right that exactly what you need. Its purpose is to distribute the String content of one or more input nodes to several output nodes respecting the maximum lengths and cardinalities of the target nodes.

       

      Best regards

      Stefan

      Author's profile photo Stefan Boller
      Stefan Boller

      Hi Robert,

       

      even for the second requirement you can try to use the String Processing:

      How%20to%20map%20FTX%20to%20E1EDKT2

      How to map FTX to E1EDKT2

      Best regards

       

      Stefan

      Author's profile photo Robert Quindt
      Robert Quindt

      Hello Stefan,

       

      the String Processing does not work for the scanario in which I have a 4000 character field to map to a E1EDKT1 Structure, as the target structure has 0..999999999 E1EDKT2 Segments with a single 0..1 TDLINE field.

      As stated in the String Processing description "string processing function considers the length and cardinality of the target nodes to distribute lengthy source message among the target nodes" and "You use the String Processing type to map multiple source node of type String and Token to one or more target nodes of type String and Token".

      In this case the target String node TDLINE has only the cardinality 0..1 and the E1EDKT2 Segment is not a String node, so String Processing does not distribute the lengthy source node.

      For my scenario I used a xslt mapping to distribute the source element to several nodes.

      <xsl:template match="//quotes/quote/description">
            <xsl:element name="descriptions">
                  <xsl:variable name="s" select="."/>
                  <xsl:variable name="N" select="70"/>
                  <xsl:variable name="v" select="xs:integer(ceiling(string-length($s) div $N))"/>
                  <xsl:for-each select="0 to $v - 1">
                    <xsl:element name="description">
                      <xsl:value-of select="substring($s, .*$N+1, $N)"/>
                    </xsl:element>
                  </xsl:for-each> 
                </xsl:element>
        </xsl:template>

       

      best regards

      Robert

      Author's profile photo Stefan Boller
      Stefan Boller

      Hi Robert,

       

      thanks for trying it. Did you add the E1EDKT2 to the String processing like in the screenshot above as well?

       

      Best regards

      STefan