Skip to Content
Technical Articles
Author's profile photo Ryan Crosby

SAP EDI/IDoc Communication in CPI Using Bundling and Fewer Mapping Artifacts

Introduction

Not sure how many folks have reviewed the semi-recent announcement regarding Trading Partner Management in SAP Integration Suite Announcement: SAP Trading Partner Management and B2B Monitoring brand new capabilities of SAP Integration Suite is released! and were as disappointed as me with the initial product delivery.  I have two core issues with the approach:

  1. Still no support for bundling IDocs despite its longtime existence on the back-end and it’s noted in SAP Help EDI Converter – that’s ok… I’ll tackle it on my own.
  2. Bloated maintenance of mapping artifacts per partner – too many folks have missed the point on EDI communication and the proposed solution is no different.  Last I checked I’m still bound by the amount of time available in a day, and my company has at least a couple dozen EDI partners.

Stay tuned for a part II that will address incoming communication for 997s and have the groundwork for other transaction sets.

Now available (2022-05-22) – SAP EDI/IDoc Communication in CPI Using Bundling and Fewer Mapping Artifacts Part II

* Revision 1.1 – Adjustment to handle dynamic terminator and separator parameters in the XML to EDI converter using a combination of externalized parameters and groovy script that reads data from the PD – 2022-07-26

Prerequisites for Bulk and Determination of EDI Standard

The message function code is mapped to the appropriate field in the group segment while the package size handles the bundling, and in the other image the values are used to determine the appropriate X12 target without having to read from the TPM agreements.

Bulk

EDI%20Standard

Transformation Flow

The initial transformation flow is based off this older ICA template with some modifications to handle the splitting for bulk, and saving the IDoc numbers in a datastore for retrieval on incoming 997s.

Transformation%20Flow

 

Extract Basic Information and Lookup Partner Directory Information

The first two steps gather some basic IDoc information, establish the sender partner id, and generate a interchange control number, along with gathering conversion specific information and agreement data. The Groovy script taps into the partner directory to read the header information for the necessary XSLT/XSD artifacts, but with one important distinction – the central mapping artifacts are tied to our partner directory entry.  The other two less important distinctions are reading binary data in json format to get extra parameters about handling – e.g. archiving, target IDoc for incoming messages, etc. and reading agreement information regarding whether an acknowledgement is required (important for following datastore step) or any extended post processing should take place.  I have included a basic sample for both the message specific parameters and the partner specific agreement information.

    import com.sap.gateway.ip.core.customdev.util.Message;
    import java.util.HashMap;
    import com.sap.it.api.pd.PartnerDirectoryService;
    import com.sap.it.api.ITApiFactory;
    import groovy.json.*;

def Message processData(Message message) {

    def service = ITApiFactory.getApi(PartnerDirectoryService.class, null)
    if (service == null){
      throw new IllegalStateException("Partner Directory Service not found")
    }
	
    // Read partner data, and EDI message information for conversion and 
    // communication purposes
    def headers = message.getHeaders()
    def SenderPid = headers.get("SenderPid")
    def rcvprn = headers.get("SAP_IDoc_RCVPRN")
    def ReceiverPid = service.getPartnerId("SAP SE", "IDOC", rcvprn)
    message.setHeader("ReceiverPid", ReceiverPid)
    def mestyp = headers.get("SAP_IDoc_MESTYP")
    def idoctyp = headers.get("SAP_IDoc_IDOCTYP")
    def std = headers.get("SAP_IDoc_STD")
    def stdmes = headers.get("SAP_IDoc_STDMES")
    def stdvrs = headers.get("SAP_IDoc_STDVRS")
    def test = headers.get("SAP_IDoc_TEST")
    
     // Retrieve our qualifier and id for ISA envelope and pass usage indicator,
     // and also read receiving partner information
    def our_x12qual = service.getParameter("x12_qualifier", SenderPid, java.lang.String.class)
    message.setProperty("OurX12Qualifier", our_x12qual)
    def our_x12id = service.getParameter("x12_id", SenderPid, java.lang.String.class)
    message.setProperty("OurX12Id", our_x12id)
    message.setProperty("UsageIndicator", test == "X" ? "T" : "P")
    def x12qual = service.getParameter("x12_qualifier", ReceiverPid, java.lang.String.class)
    message.setProperty("X12Qualifier", x12qual)
    def x12id = service.getParameter("x12_id", ReceiverPid, java.lang.String.class)
    message.setProperty("X12Id", x12id)
    def x12SegmentTerminator = service.getParameter("x12_segment_terminator", ReceiverPid, java.lang.String.class)
    message.setProperty("X12SegmentTerminator", x12SegmentTerminator)
    def x12CompositeSeparator = service.getParameter("x12_composite_separator", ReceiverPid, java.lang.String.class)
    message.setProperty("X12CompositeSeparator", x12CompositeSeparator)
    def x12ElementSeparator = service.getParameter("x12_element_separator", ReceiverPid, java.lang.String.class)
    message.setProperty("X12ElementSeparator", x12ElementSeparator)
    def x12RepetitionSeparator = service.getParameter("x12_repetition_separator", ReceiverPid, java.lang.String.class)
    message.setProperty("X12RepetitionSeparator", x12RepetitionSeparator)
    
    // Read extra parameters for message handling - e.g. archiving, message target, etc.
    def slurper = new JsonSlurper()
    def msgParamsBinary = service.getParameter(stdmes, SenderPid, com.sap.it.api.pd.BinaryData)
    if(msgParamsBinary != null) {
      def msgParameters = slurper.parse(msgParamsBinary.getData(), "UTF-8")
      if(msgParameters.Outbound?.ArchiveMessage == "true") {
        message.setHeader("ArchiveMessage", msgParameters.Outbound.ArchiveMessage)
        def folder = headers.get("ArchiveFolder")
        message.setHeader("ArchiveFolder", folder + ReceiverPid + "/")
      } else {
        message.setHeader("ArchiveFolder", null)
      }
    }
    
    // Setup header variables from PD for conversion handling (AT items)
    def msgInfo = "ASC-X12_" + stdmes + "_" + stdvrs
    def preproc, mapping, postproc, rec_conversion
    preproc = mestyp + "." + idoctyp + "_preproc"
    if(std == "X") {
      mapping = mestyp + "." + idoctyp + "_to_" + msgInfo
      postproc = msgInfo + "_postproc"
      rec_conversion = msgInfo
    }
    message.setHeader("PREPROC_XSLT", "pd:" + SenderPid + ":" + preproc + ":Binary")
    message.setHeader("MAPPING_XSLT", "pd:" + SenderPid + ":" + mapping + ":Binary")
    message.setHeader("POSTPROC_XSLT", "pd:" + SenderPid + ":" + postproc + ":Binary")
    message.setHeader("REC_CONVERSION_XSD", "pd:" + SenderPid + ":" + rec_conversion + ":Binary")
    
    // Retrieve partner specific information regarding agreement
    // information for extended post processing and acknowledgements
    def agreementParamsBinary = service.getParameter("Agreements", ReceiverPid, com.sap.it.api.pd.BinaryData)
    if(agreementParamsBinary != null) {
      def agreementParameters = slurper.parse(agreementParamsBinary.getData(), "UTF-8")
      def msgAgreement = agreementParameters.Agreements.Outbound.find{ it.Message == msgInfo }
      message.setProperty("AcknowledgementRequired", msgAgreement?.AcknowledgementRequired)
      message.setProperty("AcknowledgementRequested", msgAgreement?.AcknowledgementRequired == "true" ? "1" : "0")
      if(msgAgreement?.DoExtendedPostProcessing == "true") {
        message.setProperty("DoExtPostprocessing", msgAgreement.DoExtendedPostProcessing)
        message.setHeader("EXT_POSTPROC_XSLT", "pd:" + ReceiverPid + ":ext_" + msgInfo + "_postproc:Binary")
      }
    }
    
    return message
}
{
    "Outbound": {
      "ArchiveMessage": "true"
    },
    "Inbound": {
      "Target": "INVOIC.INVOIC02"
    }
}
{
  "Agreements": {
    "Inbound": [],
    "Outbound": [
      {
        "Message": "ASC-X12_856_004010",
        "AcknowledgementRequired": "true",
        "DoExtendedPostProcessing": "false"
      }
    ]
  }
}
Write to Datastore (If applicable)

Responsible for saving the IDoc #s into the datastore with the interchange control number as the entity id in the event that an acknowledgement is both expected and required (Necessary for STATUS.SYSTAT01 communication later). The following shows the established configuration for the write step to the datastore, and the XSL mapping step has not been included because it is your basic run of the mill XSL transform.

Write%20to%20Datastore

 

Determine Envelope Type

The retrieval of the partner directory information is followed by the interchange mapping which has been configured as a local integration process with a general splitter and gather step sandwiching the necessary mapping steps.  It also includes one additional XSLT step not found in the older standard ICA content that manages the segment counter in the S_SE segment, and was referenced from Integration Advisor: Functions at target side – Ordinal number for line items and segment count.  Once the interchange mapping is complete the enveloped is constructed using the same mechanism as the older ICA template, but with the inclusion of a router based on version number.  The route for 004010 is the default with the expression for the other route as follows:

Envelope%20Router

 

Execute XML to EDI Converter

The EDI converter parameters for the segment terminator, and each of the qualified separators will be recorded and read from the partner directory.  Each individual parameter is externalized in the XML to EDI converter step, and configured as a property that corresponds to a partner directory read in the Lookup PD script.

Converter%20Parameters

 

Transactional Control Numbers and Total Transaction Count

An extra XSL transform that is a variation of the segment counter XSL with some additional logic to output transactional control numbers based on position + 1, and also pad ISA ids with trailing spaces.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
    <xsl:template match="/">
        <xsl:apply-templates select="*"/>
    </xsl:template>
    <!-- ============================================================================================= -->
    <!-- Template: Enter the count of segments -->
    <!-- ============================================================================================= -->
    <xsl:template match="D_97">
        <D_97>
            <xsl:value-of select="count(//*[name() = 'S_ST'])"/>
        </D_97>
    </xsl:template>
    <!-- ============================================================================================= -->
    <!-- Template: Pad IDs for both sender and receiver in ISA envelope-->
    <!-- ============================================================================================= -->
    <xsl:template match="D_I06">
       <xsl:variable name="Sid" select="."/>
       <D_I06><xsl:call-template name="padStr">
        <xsl:with-param name="str" select="$Sid"/>
        <xsl:with-param name="chr" select="' '"/>
        <xsl:with-param name="len" select="15"/>
        </xsl:call-template></D_I06>
    </xsl:template>
    <xsl:template match="D_I07">
       <xsl:variable name="Rid" select="."/>
        <D_I07><xsl:call-template name="padStr">
        <xsl:with-param name="str" select="$Rid"/>
        <xsl:with-param name="chr" select="' '"/>
        <xsl:with-param name="len" select="15"/>
        </xsl:call-template></D_I07>
    </xsl:template>
    <!-- ============================================================================================= -->
    <!-- Template: Determine Transaction Set Control Number -->
    <!-- ============================================================================================= -->
    <xsl:template match="D_329">
        <D_329><xsl:value-of select="format-number(count(../../preceding-sibling::*)+1, '000000000')"/></D_329>
    </xsl:template>
    <!-- ============================================================================================= -->
    <!-- Template: Consume and produce remaining nodes -->
    <!-- ============================================================================================= -->
    <xsl:template match="node() | @*">
        <xsl:copy>
            <xsl:apply-templates select="node() | @*"/>
        </xsl:copy>
    </xsl:template>

    <!-- ============================================================================================= -->
    <!-- Template: String Padding on Right -->
    <!-- ============================================================================================= -->
    <xsl:template name="padStr">
    <xsl:param name="str"/> 
    <xsl:param name="chr"/> 
    <xsl:param name="len"/> 
    <xsl:variable name="pad">
        <xsl:for-each select="1 to $len">
            <xsl:value-of select="$chr" />
        </xsl:for-each>
    </xsl:variable>
    <xsl:value-of select="substring(concat($str,$pad),1,$len)"/>
    </xsl:template>
</xsl:stylesheet>

 

The Last Step

The final content modifier is there to remove some header information from the envelope step where the system complains about improper CRLF characters and set a header for pid which is used in the communication flow.

Communication Flow

The communication flow is setup with a JMS queue with simple retry every 15 minutes settings and an exception sub-process to handle email alerting periodically if a message is not going out.  For the most part this flow is WYSIWYG.  Maybe you would ask… why the partner directory lookup again, but that is simply for separating transformation logic from communication logic which will be obvious in the script code.  The SFTP code is still a work in progress because I do not have a use case yet, but have seen it in the past, and the IDoc work will be referenced in part II.

Communication%20Flow

 

Partner Directory Script for Communication

Gather adapter information and, if necessary, details related to third party communication for AS2 and SFTP usage.  The final piece creates a header for the archive file name to be passed on to the flow responsible for handling that process.

    import com.sap.gateway.ip.core.customdev.util.Message;
    import java.util.HashMap;
    import com.sap.it.api.pd.PartnerDirectoryService;
    import com.sap.it.api.ITApiFactory;
    
def Message processData(Message message) {
    def service = ITApiFactory.getApi(PartnerDirectoryService.class, null) 
    if (service == null){
        throw new IllegalStateException("Partner Directory Service not found")
    }
    
    // Merge Partner Directory data into message header
    def headers = message.getHeaders()
    def pid = headers.get("pid")
    def adapter = service.getParameter("AdapterType", pid, java.lang.String.class)
    message.setProperty("adapter", adapter)
    if(adapter == "AS2") {
      def url = service.getParameter("ReceiverUrl", pid, java.lang.String.class)
      message.setProperty("Url", url)
      def recipientAS2 = service.getParameter("AS2_id", pid, java.lang.String.class)
      message.setProperty("RecipientAS2", recipientAS2)
      def publicKey = service.getParameter("PublicKeyAlias", pid, java.lang.String.class)
      message.setProperty("PublicKeyAlias", publicKey)
      def compress = service.getParameter("SAP_AS2_Outbound_Compress_Message", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Compress_Message", compress)
      def signMessage = service.getParameter("SAP_AS2_Outbound_Sign_Message", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Sign_Message", signMessage)
      def signAlgorithm = service.getParameter("SAP_AS2_Outbound_Signing_Algorithm", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Signing_Algorithm", signAlgorithm)
      def encryptMessage = service.getParameter("SAP_AS2_Outbound_Encrypt_Message", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Encrypt_Message", encryptMessage)
      def encryptAlgorithm = service.getParameter("SAP_AS2_Outbound_Encryption_Algorithm", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Encryption_Algorithm", encryptAlgorithm)
      def mdnType = service.getParameter("SAP_AS2_Outbound_Mdn_Type", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Type", mdnType)
      def mdnRequestSign = service.getParameter("SAP_AS2_Outbound_Mdn_Request_Signing", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Request_Signing", mdnRequestSign)
      def mdnSignAlgorithm = service.getParameter("SAP_AS2_Outbound_Mdn_Signing_Algorithm", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Signing_Algorithm", mdnSignAlgorithm)
      def mdnVerifySign = service.getParameter("SAP_AS2_Outbound_Mdn_Verify_Signature", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Verify_Signature", mdnVerifySign)
      def mdnRequestMic = service.getParameter("SAP_AS2_Outbound_Mdn_Request_Mic", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Request_Mic", mdnRequestMic)
      def mdnVerifyMic = service.getParameter("SAP_AS2_Outbound_Mdn_Verify_Mic", pid, java.lang.String.class)
      message.setHeader("SAP_AS2_Outbound_Mdn_Verify_Mic", mdnVerifyMic)
    } else if(adapter == "SFTP") {
      // TODO
    }
    
    // Set archive file name if archiving is activated for message
    def archive = headers.get("ArchiveMessage")
    if(archive == "true") {
      def stdmes = headers.get("SAP_EDI_Message_Type")
      def intchg = headers.get("SAP_EDI_Interchange_Control_Number")
      message.setHeader("ArchiveFile", stdmes + "_" + intchg + ".edi")
    }

    return message
}

 

Archive Handling

The default route is for no archiving, but in the event archiving is required, then sequential multicast is used with the first branch being the archive step.  This approach ensures that the archive is always saved first (internal FTP with overwrite) and you never get orphaned exchanges where no archive has been saved.  I believe it is generally required in most, if not all cases, to save several years in the case of 810s for audit purposes.

Conclusion

Not a whole lot to say here, but rather to show it can be done.  The important note being that it is not supported so take it under advisement and do so at your own risk.  I encourage folks to comment, like, and ask any questions (Some detail was glossed over for brevity).  Cheers!

References

https://blogs.sap.com/2021/12/17/announcement-sap-trading-partner-management-and-b2b-monitoring-brand-new-capabilities-of-sap-integration-suite-is-released/

https://help.sap.com/docs/CLOUD_INTEGRATION/987273656c2f47d2aca4e0bfce26c594/707973f9bfb1419eb80628755494b962.html?q=edi%20converter

https://api.sap.com/integrationflow/IDOCToX12_4010_Outbound

https://blogs.sap.com/2020/12/10/integration-advisor-functions-at-target-payload-ordinal-number-for-line-items-and-segment-count/

https://blogs.sap.com/2016/08/11/hcihcp-isidoc-adapter-deciphered-part-4-trigger-idoc-from-hci-to-sap-erp-using-basic-authentication/

Assigned Tags

      Be the first to leave a comment
      You must be Logged on to comment or reply to a post.