Skip to Content

Enabling complex data transformation within SAP NetWeaver

If you already know how to work with flat files and other input or output structures in SAP NetWeaver Exchange Infrastructure, then there is a good chance you have encountered difficulties when trying to work with very complex file structures (such as COBOL copybooks); forget about working with .pdf files, or Microsoft Word&#8482 documents, or other semi- or non-structured data – that requires custom coding. But what if you could use a graphical interface to import these, and many other structures, at design time, plus map them to the target schema, and then use any of the SAP or partner JCA-compliant adapters for transport of those structures at runtime?

SAP has partnered with Itemfield, the leader in complex data transformation technology to develop the Itemfield Conversion Agent for SAP NetWeaver. Using this technology, customers can efficiently incorporate and transform semi-structured data (HL7, HIPAA, EDI-X.12, EDIFact, COBOL copybooks, etc.), unstructured data (ASCII, EBCDIC , PDF, HTML, Microsoft Excel, Microsoft PowerPoint®, etc.), and messages based on Industry Standards (SAP IDocs, ACORD XML, ACORD AL3, IFX, FIXML, OFX, LegalXML, cXML, ebXML, RosettaNet, HL7 etc.) into SAP NetWeaver.

The solution includes the Itemfield ContentMaster Studio, a plug-in to the Eclipse development environment that allows you to implement or generate conversion procedures (“parsers”) to bring the data into NetWeaver in the desired format. For instance, suppose we have a requirement to accept a purchase order that is delivered as a PDF document, and we want to automate processing of these. Suppose, also, that we want to map the PO Data to an Orders IDOC. In the ContentMaster Studio, we would point to a sample instance of the PDF; We would open the IDOC-XML schema as our target schema; and we would specify or generate the rules for importing the PDF data into the IDOC structure. Of course there are wizards to assist in the task (for instance, to define the repetitive structures that contain the PO Line Items and map them to the appropriate IDOC segments and fields); and a function library (user-extensible) for inline transformations; and many nice features to the interface:

Itemfield ContentMaster UI

The solution will include (as of SPS15) a J2EE library that can be plugged into the NetWeaver environment, and this library can be called from a module that plugs into any of the JCA-compliant adapters (from SAP or the partner ecosystem) for SAP NetWeaver Exchange Infrastructure. So having defined our transformation rules as above, we could use the file adapter to pull the PDF PO file into the environment; and the Itemfield runtime J2EE library would bring the data into SAP NetWeaver XI in the IDOC format as we defined. Thus no further mapping would be required!

The solution works bi-directionally; that is, you can also define the transformation from XML to (semi-) (non-) structured data (via “serializers”). And Exchange Infrastructure is not the only NetWeaver component that can use the Itemfield Conversion Agent. For instance, you could use the solution to do screen-scraping of an HTML page and use it in an iView for SAP NetWeaver Portal.

I’m sure that, with a little thought, you can probably imagine many scenarios where this functionality can improve your life, make your work easier, and expand the capabilities of your SAP NetWeaver environment. You can find more information about the Itemfield Conversion Agent for SAP NetWeaver by going to the “XI” quicklink in the SAP Service Marketplace (http://service.sap.com/xi), and then navigating to “SAP XI in Detail &#8211&#8250 Connectivity &#8211&#8250 Connectivity SAP XI 3.0 &#8211&#8250 Itemfield

I will be writing more blogs about the Itemfield Conversion Agent for SAP NetWeaver, including some customer examples, in the months ahead. In the meantime, if you would like to use this functionality in your implementation, look at the site in the Service Marketplace, and by all means send me an email!

To report this post you need to login first.

10 Comments

You must be Logged on to comment or reply to a post.

  1. Anonymous
    Perhaps I missed something, but why do we need this tool? I thought XI with its vast array of Adapters would do the job of mapping and transformation. In the ERP environment, we have LSMW that allows us to do the same thing.

    Thanx in advance for your clarifications.

    (0) 
    1. Paul Medaille Post author
      Sanjay –

      Adapters are responsible for transport only, though there are elementary transformations from the non-xml to the xml data structure. For instance, in the file adapter you can convert a comma-delimited file to an xml file, but the xml format is fixed. With the SAP Conversion Agent by Itemfield, you could map that (perhpas extremely complex – say, a large cobol copybook) file to an IDOC without having to do a secondary mapping.  Transformations may be included as part of a business package, but those would be done via the graphical mapping tool (GMT), which does have limitations.  Since the customer nearly always has to modify those delivered maps (after all, who ever implements standards in a completely standard way?), they would tehn be stuck with the limitations of the GMT.  The SAP Conversion Agent by Itemfield extends the already impressive capabilities of SAP NetWeaver Exchange Infrastructure, as well as working with other components of the NetWeaver Suite.  The conversion agent allows for easy, bi-directional transformation of structures that cannot be mapped in either XSLT or the GMT, such as the pdf example I used in the blog, and allows you to do it via an intuitive graphical interface rather than with java code.  Is the conversion agent a “must-have” or a “nice-to-have?” For many customers, it is perhaps the latter; but for those customers who have non-structured interfaces (estimated to be up to 80+% of all information sources), this tool will be an absolute boon.

      (0) 
      1. Srinivasan Jaganathan
        Hi,

        The information is very interesting and useful. Is itemfield can handle ANSI.X12 format to SAP IDOC format translation. Can you explain this with small example configuration.

        Thanks

        (0) 
        1. Paul Medaille Post author
          Hello Srinivasan –

          Itemfield can handle ANSI.X12 – all message types are covered.  I think I need to do a separate blog on this topic to do it any justice.  look for one later this week or early next week!

          Paul

          (0) 
  2. Shabarish Vijayakumar
    Hi Paul,

    Suppose I need to deal with ebXML, which is the prefered adapter ?

    And since you mentioned ebXML in relation with Itemfield how much relevant is the same for interfaces involving ebXML.

    Regards,
    Shabarish

    (0) 
  3. Tirumal Pappu
    Paul –

    This is a great blog and appreciate for mentioning this.

    Now can we say that we can do the mapping of cxml to xml using this feature of Itemfield or if not what could be the other way.

    Thanks,
    KP

    (0) 
  4. Shahed Bashir
    Hi Paul,

    Very informative blog.

    Is it possible to use the Conversion Agent to transforms Windows client/server applications (i.e. screen scraping) into XML and integrate to XI?

    Thanks

    (0) 
    1. Paul Medaille Post author
      Hello Shahed –

      Sorry for the delayed response – I’ve been on vacation celebrating my birthday.

      As to your question, I assume you are thinking of scraping forms, and in that case the answer is no unless the Forms are HTML /DHTML.
      If you have a specific requirement, Itemfield has experience extracting data from VB/MS_ACCESS application not through screen scraping but using
      other available formats.

      Another approach used successfully in such application is using their default reports to extract the information from the appropriate page.

      Hope that helps!

      Paul

      (0) 

Leave a Reply