How to Use User-Module for Conversion of IDoc Messages Between Flat and XML Formats
When we exchange messages with our external business partners, IDocs are often used. The IDocs would be sent in flat IDoc format to our business partner. Conversely, when we receive messages from our external business partners, they are also often in flat IDoc format. In this blog, we will take a look at how we can use PI to integrate directly with our external business partners through flat IDoc formatted messages. PI 7.11 introduced a new Java adapter user-module which can translate between flat IDoc formatted messages and IDoc-XML formatted messages. The IDoc-XML messages, once in PI, can easily be used for mappings and configurations. The new user-module will not only validate the metadata of the IDoc, it will also check the validity of the content. In order to do so, the user-module will connect to the SAP backend system and retrieve the metadata and application-related information and check them against the IDoc message. To connect to the SAP backend system, connection information are required. Instead of entering these information in the user-module, the user-module will access them through objects in the NWA. This gives us a “cleaner” management of entering information for the user-module and also allow us to re-use existing objects in the NWA. As a result, to use the user-module, we will need to go through the following steps: * Create a JCA Connection Factory in NWA for the SAP backend system. * Configure the user-module using the objects in the above 2 steps. *User-Module Configuration* For demonstration purpose, in this blog we will create two user-modules. One is to convert IDoc XML to Flat format. The other is to convert IDoc Flat to XML format. h4. 1. Create destination information in NWA. 1. Enter NWA using the URL: http://+:+/nwa 0.1. Navigate to: Configuration Management -> Infrastructure -> Destination 
0.1. In Destination, click “Create”: 
0.1. Enter a Destination Name, and select RFC in the dropdown. Click Next. 
0.1. Select whether Load Balancing is used. In our case, without Load Balancing, we entered values for the Host server, System Number and System ID. Click Next. 
0.1. Enter authentication information to logon to the server. Click Finish. 
0.1. Do a ping test by clicking on “Ping Destination”. 
h4. 2. Create a JCA Connection Factory in NWA. 1. In NWA, navigate to: Configuration Management -> Infrastructure -> Application Resources 
2. Filter on “SAPJavaResourceAdapter15” and click the “Filter” button. Select “SAPJavaResourceAdapter15” with the JCA Resource type. Click on the tab “Dependent JCA Connection”. Click on “Copy and Add New JCA Connection Factory”. 
3. In the “New JCA Connection Factory Creation” screen, tab “Namespace”, enter a JNDI Name. In our case, I entered “CF_CU4_800”. *NOTE:* The JNDI name is the JCA Connection Factory name. 
0.1. In the tab “Configuration Properties”, enter the server logon information and click “Save”. 
0.1. You should receive the following message: 
h4. 3. Configure the user-module 1. IDoc XML to flat format conversion This is normally done when we send an IDoc to our external business partner. The IDoc is converted to flat file format. The user-module most likely is executed in the _receiver_ communication channel. The user-module can be used with any java adapter, e.g. file, JMS, JDBC, Mail, etc. To configure the user-module: 1. Go to the Module tab of the communication channel configuration in the Integration Directory. 2. Add a module name, *SAP_XI_IDOC/IDOCXmlToFlatConvertor*, before CallSapAdapter. Select from the dropdown for the Module Type: *Local Enterprise Bean*. 0.1. By default, the module key of *”1″* is assigned. You can rename it to any name. Create 3 Module Key entries in the “Module Configuration”, using the same key value. 0.1. For the Parameter Names and Parameter Values, enter the following: *SAPRelease:* SAP release of the system where the IDoc needs to be validated. *SourceDestination:* The destination name created in the NWA in step 1. *TargetJRA:* The connection factory name created in NWA in step 2. 
*Sample Test:* Add a module name, *SAP_XI_IDOC/IDOCFlatToXmlConvertor*, before CallSapAdapter. Select from the dropdown for the Module Type: *Local Enterprise Bean*. 0.1. By default, the module key of *”1″* is assigned. You can rename it to any name. Create 3 Module Key entries in the “Module Configuration”, using the same key value. 0.1. For the Parameter Names and Parameter Values, enter the following: *SAPRelease:* SAP release of the system where the IDoc needs to be validated. *TargetDestination:* The destination name created in the NWA in step 1. *SourceJRA:* The connection factory name created in NWA in step 2. 
0.1. In Destination, click “Create”: 0.1. Enter a Destination Name, and select RFC in the dropdown. Click Next. 0.1. Select whether Load Balancing is used. In our case, without Load Balancing, we entered values for the Host server, System Number and System ID. Click Next. 0.1. Enter authentication information to logon to the server. Click Finish. 0.1. Do a ping test by clicking on “Ping Destination”. h4. 2. Create a JCA Connection Factory in NWA. 1. In NWA, navigate to: Configuration Management -> Infrastructure -> Application Resources 2. Filter on “SAPJavaResourceAdapter15” and click the “Filter” button. Select “SAPJavaResourceAdapter15” with the JCA Resource type. Click on the tab “Dependent JCA Connection”. Click on “Copy and Add New JCA Connection Factory”. 3. In the “New JCA Connection Factory Creation” screen, tab “Namespace”, enter a JNDI Name. In our case, I entered “CF_CU4_800”. *NOTE:* The JNDI name is the JCA Connection Factory name. 0.1. In the tab “Configuration Properties”, enter the server logon information and click “Save”. 0.1. You should receive the following message: h4. 3. Configure the user-module 1. IDoc XML to flat format conversion This is normally done when we send an IDoc to our external business partner. The IDoc is converted to flat file format. The user-module most likely is executed in the _receiver_ communication channel. The user-module can be used with any java adapter, e.g. file, JMS, JDBC, Mail, etc. To configure the user-module: 1. Go to the Module tab of the communication channel configuration in the Integration Directory. 2. Add a module name, *SAP_XI_IDOC/IDOCXmlToFlatConvertor*, before CallSapAdapter. Select from the dropdown for the Module Type: *Local Enterprise Bean*. 0.1. By default, the module key of *”1″* is assigned. You can rename it to any name. Create 3 Module Key entries in the “Module Configuration”, using the same key value. 0.1. For the Parameter Names and Parameter Values, enter the following: *SAPRelease:* SAP release of the system where the IDoc needs to be validated. *SourceDestination:* The destination name created in the NWA in step 1. *TargetJRA:* The connection factory name created in NWA in step 2. *Sample Test:* Add a module name, *SAP_XI_IDOC/IDOCFlatToXmlConvertor*, before CallSapAdapter. Select from the dropdown for the Module Type: *Local Enterprise Bean*. 0.1. By default, the module key of *”1″* is assigned. You can rename it to any name. Create 3 Module Key entries in the “Module Configuration”, using the same key value. 0.1. For the Parameter Names and Parameter Values, enter the following: *SAPRelease:* SAP release of the system where the IDoc needs to be validated. *TargetDestination:* The destination name created in the NWA in step 1. *SourceJRA:* The connection factory name created in NWA in step 2.
Current options use ABAP mapping / ABAP Reports , and using ABAP mappings / ABAP reports can be cumbersome and this feature addresses a common issue.
I guess until we upgrade to PI7.1 we will have to live with ABAP Mappings, but when we do move to 7.1, this is the first feature we will turn on without any doubts.
Regards
Bhavesh
In which SP this scenario can be done?
Thanks,
Arivarasu S
This feature is available with PI 7.11, which is PI 7.0 EhP1. PI 7.11 was released in July, 2009.
Regards,
Bill
I check PI7.0EhP1 doesn't have this module. It's available with PI7.1Ehp1. Can we install as additional notes to PI7.0EhP1?
Regards,
Anucha
Great post! I would just like to ask/clarify some points:
Is it possible to do the XML IDOC to flat file IDOC conversion without doing any validation? This means that I will just specify this new user module in the module tab without specifying any parameters?
If I am doing a XML IDOC to flat file IDOC conversion, there will be no restriction on the output encoding of the flat file right? For instance, if I am using this new user module and I want the flat file to have an encoding of iso-8859-1, then I just need to specify it in the encoding field in the file adapter, right?
Regards,
Elbert
As of 7.11, it is not possible to not referencing the IDoc metadata and bypass the validation.
Regards,
Bill
Can we implement this module configuration even with custom IDOC/ZIDOCs to conver Custom IDOC flatfile to IDOC xml and vice versa?
and RFC destination we create in NWA,do we need to give SAP system details where IDOC has been existing/custome IDOC developed or XI/PI systems details to get the IDOC meta Data?
Regards,
Bill
and can we process multiple IDocs of same type at a time?
Thank you for your information.
..Sri
this blog describes the new module very good. Good job.
But how can we find details error information about problems in the module?
We get the message :
File processing failed with com.sap.conn.idoc.IDocIllegalTypeException: (4) IDOC_ERROR_ILLEGAL_TYPE: Segment type "E1EDP02" is not a valid child segment type. DOCUMENT: type=ORDERS05, number=0000000000398170; SEGMENT: type=ROOT, definition=ROOT
Thanks.
Regards
This is a validation error, indicating that the content of your payload does not match the IDOC difinition indicated in your configuration.
Regards,
Bill
yes indeed, but the structure of the flat Idoc which we want to convert into XML seems ok.
So we need a little bit more Logging Infos.
Regards.
Thorsten
Regards,
Bill
yes, but are nearly the same....
#2.0##2010 05 05 15:08:13:485#+0200#Error#com.sap.aii.adapter.file.File2XI#
#BC-XI-CON-AFW#com.sap.aii.af.lib#7A5216CF860BFEF7000000070041F09E#6021650000001921##com.sap.aii.adapter.file.File2XI.processFtpList()#J2EE_GUEST#0##BBD12D8C541B11DFB9B47A5216CF860B#bbd12d8c541b11dfb9b
Channel ORDRSP_SEND: Sending file failed with com.sap.conn.idoc.IDocIllegalTypeException: (4) IDOC_ERROR_ILLEGAL_TYPE: Segment type "E1EDP02" is not a valid child segment type.
DOCUMENT: type=ORDERS05, number=0000000000398170; SEGMENT: type=ROOT, definition=ROOT - continue processing#
Regards
Thorsten
The issue was the bean needs idoc segment definition and not the type . I've created a solution . See Using SAP Java Idoc Class Library in SAP PI Adapter Modules and allow segment type to be used for Idoc Flatfile to XML conversion
we also increased the trace Level via NWA to debug, now we see the Method "DefaultIdocSegment.addChild where the error occurs, but not why.
We think it is a little problem with the IdocFlatfile from our Partner, but we do not see the error in the file.
Regards
is there any way or tool to check what is wrong with the Flat Idoc file ?
Thanks for your Help.
Ragards
Thorsten
Thanks
Saravana
currently, we have the problem that we get in the Channel the Error Message: The IDOC metadata is not available in MC1.
What have we do:
We had create the Destination and the JCA Connection like in the HowTo. The JCA Connection is on status green. And the Test of the Destinations was also successfull.
What have we try:
We had try to convert a IDOC to a Flat File. The IDOC is Customer IDOC not a standard IDOC of SAP.
I had search in the SAP Notes and in the SDN Blogs about the error message but i found not anywhere a details about the error to the modul exit.
Is it possible that the Modul Exit can not works with customer idocs ?
The Customer have a SAP 4.7 and a NetWeaver 7.1 EHP1.
Have you a idea, or a side with error message and solution to the modul exit ?
Can you please share the solution for the above issue which you face. I am also getting the same in one of my implementations. I have also posted a thread for the same : Error while implementing "SAP_XI_IDOC/IDOCXmlToFlatConvertor"
Thanks!!
We have a scenario where the input IDoc flat file contains multiple IDocs. We have a limitation using this module that it works only for one IDoc.
How to convert the File having multiple IDocs.
Thanks,
Krishnaraju.
This is a limitation right now. This capability will be available with PI 7.31.
Regards,
Bill
Thanks for your reply.
Do we have any other option to execute the multiple idocs from the input flat file other than using the report "RSEINB00".
We are developing an interface with multiple IDocs in the flat file and the input file is very huge.
Appreciate your help,
Thanks.
How is the IDoc created?
Regards,
Bill
Legacy is sending huge IDoc flat files around 10GB. XI has to convert the IDoc Flat files to IDoc xml and sent to SAP to create IDocs.
Input file contains multiple Idocs in Flat file format.
Thinking whether to open FTP port in ECC to pass through the file in XI and place the Idocs flat file on SAP R/3 application directory or to place the IDoc flat file on SAP-XI directoy using NFS protocol and schedule the report "RSEINB00".
Is there any other process to execute this scenario?
Thanks,
Krishnaraju.
Regards,
Bill
We are trying to persuade the customer to split the file and send each IDoc flat file to PI.
I am getting the below exception when I tried to convert single IDoc Flat File to IDoc XML using the module "SAP_XI_IDOC/IDOCFlatToXmlConvert" in sender File adapter.
Exception:
Module Exception 'com.sap.aii.af.idoc.exception.IDOCModuleException' found, cause: java.lang.NumberFormatException: For input string: ""
I have successfully uploaded and created the IDoc with the same file from WE19 transaction.
Any idea about this Exception.
Thanks,
Krishnaraju.
We have over come the number format exception by placing unique identifier in place of Idoc number and segment relations ships.
Module is successfully converting the IDoc Flat file with multiple Idocs, we dont have this limitation in this module.
The limitation is in the format of the source file. The flat file should contain unique identifier in place of IDoc and Parent and child node relation ship should explicitly mentioned in the IDoc flat file. Without these explicit parameters Module is not converting IDoc flat file into IDoc XML.
Without mentioning these explicit parameters, WE19 trasaction is successfully uploading the Idocs.
I think Java module is considering the unique relation ship by taking these parameters and ABAP program is reading the file in sequential way and creating the IDocs by breaking the file whenever it encounters EDIDC segment.
If the source system is not a SAP system, then it would be a problem using this module to convert Falt file into IDoc XML.
Thanks for your help,
Krishnaraju.
I'm getting the same input string "" error exception and I've put a unique identifier in the EDI_DC flat file and processed, but couldnt figure out the relation ship you mentioned, that need to set up for parent and child for processing.
I'm processing ORDERS05 and have the other segments like E2EDK01 , E2EDKA1 etc.. where I can find the doc no for this segments to put the same identifier mentioned in the EDI_DC record.
Thank You in advance. Pls let me know if I'm missing some thing in assigning the parent child relationship.
Thanks,
amar--
Hi,
I am facing the same issue. Could you please let me know how you resolved it. It fails in Sender channel itself, so there is no question of using any format number in mapping.
How is the IDoc created?
Regards,
Bill
We are getting a validation error IDOC_ERROR_PARSE_FAILURE: READING_FIELD_VALUE_TAG while converting an IDoc XML to Flat file. This error is due to a field that is having character value.For numeric value it works fine. However, when I am verifying the IDoc metadata this field is of character type. Still the module gives a validation error.
Is there any workaround to overcome this error.
Regards
Pravesh
Since the XSD validation uses the IDoc metadata on the ABAP system, do you have the element attribute changed there?
Regards,
Bill
I am having the IDO file for the IDoc and this field is mentioned as character type. I don't think there is element attribute change. Still we are getting the error.
Pravesh
whats about the content lenth ? Does it fit to the field length of the IDOC ? Or are there special characters, which are not allowed in XML and/or IDOC ?
Regards
Thorsten
There are no special characters in XML. If the value of the field is say PF, it fails with error. If I replace the value like 1234, it works fine. Field length is 5 characters.
PRavesh
If the IDoc metadata in ABAP is defined correctly and you still get this error, then you should create an SAP message for development to resolve.
Regards,
Bill
Regards,
Bill
We have a scenario where we receive a EDI XML file from our customer and need to convert it to a Idoc.
Will this setup work in this scenatio as well? Which steps I would need to follow for the same?
Thanks
Diptee
No. You EDI XML is already in XML format. This module is to convert raw IDoc to XML format, and vice versa.
Regards,
Bill
Nice blogs. Worked for us.
Thank you William.
Bill,
Tried all the steps , it worked for IDOCXML to FLAT without any issues . We are getting error for FLAT to IDOCXML as below . Please advise , We are on 7.3
Information
MP: processing local module localejbs/SAP_XI_IDOC/IDOCFlatToXmlConvertor
Error
MP: exception caught with cause java.lang.ClassCastException: class java.lang.String:null incompatible with class java.util.ArrayList:null
Error
Attempt to process file failed with java.lang.ClassCastException: class java.lang.String:null incompatible with class java.util.ArrayList:null
Looks like you might have some inconsistencies of the XML from the flat file after conversion when compared to the definition of the IDoc.
Am using exactly same IDOC which is used for receiver channel, works for receiver but not for sender channel
Regards
Girish
As mentioned before, I think the error is related to the conversion of the flat file to XML. The resulting XML is inconsistent with the IDoc definition. For example, in the flat file, there might be multiple values for an element which can have only 1 value. Or, it is expecting an array of values but the flat file does not have enough values.
Hi Bill,
Its a very nice blog, I have used IdocXMLtoFlat module in one of my interface, It was working fine, until we have a requirement to include Z(Custom) segment in the Custom IDOC. Then its failed and the following error occurs in Receiver Channel.
IDOC_ERROR_PARSE_FAILURE: An IDocIllegalTypeException occurred: state=READING_FIELD_STARTTAG, charPosition=170967, lineNumber=1, columnNumber=170968, invalidChar=U+0053
This points to the Z segment added in the IDOC. I have check the meta data its up to date, shows as same as backed system structure. Please let me know your thoughts on it.
Thanks,
Hi All,
i'm using the module SAP_XI_IDOC/IDOCXmlToFlatConvertor, all it's ok except that
EDI_DC40 in the first row:
Why? Help me please.
Thanks a lot
Pasquale
hello,
i am also facing same issue, may i know how did you fix this issue?
Thanks in Advance,
Madhav
Hi William.
We are trying to implement this in our landscape. We are facing trouble in creating the Connection factory in step 2. We do not find any value when we filter on SAPJavaResourceAdapter15. We are running on PI 7.1 EHP1 SP 05. We have an evaluation system for PI which is on 7.31. Could there be any service pack conflict on 7.11 which is causing this issue?
Thanks in advance.
Hari
Hi,
Using PI 7.11 SP9, I am able to find "SAPJavaResourceAdapter15". I do not have an SP5 system. Maybe you can filter on "JCA Resource" under the "Resrouce Type" heading.
Regards,
William
hi William,
we are working on a migration project from JCAPS to PI. in which JCAPS
was generating IDOC flat file which is
containing EDIDC40 as starting tag of when each IDOC segement starts.
but when we are creating the IDOC flat file with SAP PI with
("SAP_XI_IDOC/IDOCXmlToFlatConvertor") module, we are unable to see
EDIDC40 as a start tag and attached reference file(SAP PI file.zip),
and it is also creating some unnecessary data which is not required.
can you please suggest us what should be done to create file as JCAPS
is creating?
JCAPS file screen shot
SAP PI file output screenshot
Marked are unnecessary data.. and missing EDI_DC40 in PI output file.
can you please suggest me the fix for this
Try rebooting the system and send the IDoc again. These modules access Metadata of IDoc from Java stack not on ABAP stack. After Java cache refresh, metadata will be updated when you trigger the IDoc.
We always needs to reboot the system whenever the IDoc metadata changes.
Bill,
Scenario: IDoc to File >> Convert IDoc XML to Flat File.
System: PI 7.11 patch5
Per your blog, this can be done using standard Out of the box Adapter Module 'IDOCXmlToFlatConvertor'.
We have configured the RFC destination to backend ECC system and also created the JCA Connection Factory pointing to PI System in NWA.
When we test the scenario by running the sample payload through RWB to the FIle receiver adapter with the above mentioned module, I am getting the following error:
Delivering the message to the application using connection File_http://sap.com/xi/XI/System failed, due to: com.sap.engine.interfaces.messaging.api.exception.MessagingException: com.sap.conn.idoc.IDocException: (0) IDOC_ERROR_INITIALIZATION: Exception in record factory. Check the configuration of ResourceAdapter.
Pls advice.
Wondeful blog. However, there is a restriction that segment definition is required and our messages had segment types. Solution is documented here :
Using SAP Java Idoc Class Library in SAP PI Adapter Modules and allow segment type to be used for Idoc Flatfile to XML conversion