Update 22 Jan 2016: New parameters missingLastFields and additionalLastFields to handle structure deviations between configured parameters and actual file content.

Update 26 May 2015: New parameter genericRecordType to cater for records with no key field value as shown in scenario 3. Previous scenario 3 renamed to Scenario 4.

Update 25 Mar 2015: DeepFCCBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass.

Update 17 Mar 2015: Parameter indentXML replaced by indentFactor.

Introduction

This is the second part of the two-part series on achieving file content conversion (FCC) for deep structure with a reusable custom adapter module. The first part covered the conversion of deep XML to flat file at a receiver channel.

This second part will focus on conversion of flat file to deep XML at a sender channel.

Source Code

Refer to following blog on location of source code and/or EAR deployment file.

FormatConversionBean – One Bean to rule them all!

Prerequisite

  • Each substructure type must contain a field containing the key value to enable the parser to identify lines of this substructure type
  • Each substructure type must belong to one and only one parent substructure type. The parent can be either the root element or other substructure types defined

Usage of Module in Communication Channel

Module Processing Sequence

Number Module Name Type Module Key

<Depending on position of module in chain>

1) Asynchronous scenario,

Normally before the last module in channels

2) Synchronous scenario,

Before last module to convert request payload

After last module to convert response payload

Custom_AF_Modules/FormatConversionBean Local Enterprise Bean <Any Arbitrary Value>

Module Parameter Reference

Below are the parameters for configuration of the module for flat file to deep XML conversion (converterClass = ‘com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter’). Certain parameters will automatically inherit the default values if it is not configured.

The parameters are separated into parameters that are common for this conversion type, and also parameters that are specific for field separator based or fixed length based conversion.

1) Common Parameters

Parameter Name Allowed values Default value Remarks
conversionType DeepPlain2XML Required field. Determines conversion type
converterClass com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter Required field. Determines conversion class
encoding UTF-8 Defines encoding of output plain text file
documentName Required field. Document name of root element of XML output
documentNamespace Required field. Namespace of root element of XML output
recordsetStructure Required field. List of substructure names separated by comma
genericRecordType Name of substructure used as generic type. If configured, any lines that cannot be determined via key field determination will be treated as this generic type.
keyFieldName Required field. Name of the key field that must occur in all substructures
indentXML Y, N N Determines if XML output will be indented or not
indentFactor Integer values beginning from 1 0 Determines the number of indentation spaces for each level in the XML output
rowOffset Integer values beginning from 1 0 Starting row to begin extracting content from (i.e. 0 = start from first row, 1 = start from second row)
trimContents Y, N Y Determines if all the leading and subsequent blanks for a field will be trimmed or not
<StructureName>.fieldNames Required field. Names of fields in this substructure
<StructureName>.keyFieldValue Required field. Key field value to identify this substructure
<StructureName>.parent Required field. Specifies the parent of this substructure. Value should correspond to a substructure in recordsetStructure or the value ‘Root’ to indicate that it belongs directly under the root element
<StructureName>.missingLastFields ignore,add,error ignore Behavior of conversion when there are less fields in content compared to configured fields in <StructureName>.fieldNames

  • ignore = missing fields not included in output
  • add = missing fields included with blank value in output
  • error = conversion terminated with error due to incomplete input
<StructureName>.additionalLastFields ignore,error ignore Behavior of conversion when there are more fields in content compared to configured fields in <StructureName>.fieldNames

  • ignore = additional fields not included in output
  • error = conversion terminated with error due to extraneous input
debug

Displays audit log messages of substructure and parent-child determination of each line.

WARNING: Use this only for debugging in non-productive systems

messageLog pre, post Saves a log version of the message that is viewable in Message Monitor

  • pre = saves payload before conversion
  • post = saves payload after conversion
logLocation Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated

2) Field Separator Conversion Parameters

Parameter Name Allowed values Default value Remarks
defaultFieldSeparator Defines a default field separator for all substructures that do not have <StructureName>.fieldSeparator explicitly configured
<StructureName>.fieldSeparator Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated
<StructureName>.enclosureSignBegin Specify a string that acts as a text delimiter. Field separators within such texts are ignored
<StructureName>.enclosureSignEnd If the text delimiters for the beginning and end of the text are different, specify the text delimiter for the end of the text here. If it is not populated, it uses value specified in <StructureName>.enclosureSignBegin
<StructureName>.enclosureSignBeginEscape Specify a string that replaces the text delimiter if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignBegin
<StructureName>.enclosureSignEndEscape Specify a string that replaces the text delimiter for the end of the text if it occurs within a text that it delimits. When the text is transferred the string is replaced by the value specified in <StructureName>.enclosureSignEnd. If it is not populated, it uses value soecified in <StructureName>.enclosureSignBeginEscape
<StructureName>.enclosureConversion Y, N Y
  • Y = the text delimiter is to be removed and/or the escape character is to be replaced when the files are transferred
  • N = the character is to be transferred unchanged

3) Fixed Length Conversion Parameters

Parameter Name Allowed values Default value Remarks
<StructureName>.fieldFixedLengths Integer separated by commas Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated. The number of lengths must match the number of fields listed in <StructureName>.fieldNames

Similar to standard FCC functionality, fieldSeparator parameter supports non-printable ASCII characters as follows:

  • characters encoded as hexadecimal values in the form of ‘0xHH’ (including quotation marks)

Example Scenarios

Here are some example scenarios of the behavior of the conversion based on different configuration options.

Scenario 1

3 substructures with 3 level deep structure

Field separator based conversion

Field separator – Order and Item using default comma. Delivery using tab (specified in hexadecimal)

Order uses ” as enclosure sign, and “” as enclosure escape

Item uses ” as enclosure sign but enclosure sign is not removed when transferred

Row offset provided to skip first line

Output XML indented

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName MT_DeepPlain2XML
documentNamespace urn:equalize:com
recordsetStructure Delivery,Order,Item
keyFieldName Type
Delivery.fieldNames Type,DeliveryNo
Order.fieldNames Type,DeliveryNo,OrderNo
Item.fieldNames Type,OrderNo,ItemNo,Quantity
Delivery.keyFieldValue D
Order.keyFieldValue O
Item.keyFieldValue I
Delivery.parent Root
Order.parent Delivery
Item.parent Order
Delivery.fieldSeparator ‘0x09’
defaultFieldSeparator ,
Order.enclosureSignBegin
Order.enclosureSignBeginEscape “”
Item.enclosureConversion N
Item.enclosureSignBegin
rowOffset 1
indentFactor 4

Result

Input /wp-content/uploads/2015/03/in1_662853.png
Output /wp-content/uploads/2015/03/out1_662854.png

Scenario 2

4 substructures with 3 level deep structure

Fixed length based conversion

Leading and trailing whitespaces are not trimmed

Missing fields are automatically populated with blank

Output XML indented

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName MT_DeepPlain2XML
documentNamespace urn:equalize:com
recordsetStructure Header,Delivery,Order,Item
keyFieldName Type
Header.fieldNames Type,FileName
Delivery.fieldNames Type,DeliveryNo
Order.fieldNames Type,DeliveryNo,OrderNo
Item.fieldNames Type,OrderNo,ItemNo,Quantity
Header.keyFieldValue H
Delivery.keyFieldValue D
Order.keyFieldValue O
Item.keyFieldValue I
Header.parent Root
Delivery.parent Root
Order.parent Delivery
Item.parent Order
Header.fieldFixedLengths 5,20
Delivery.fieldFixedLengths 5,10
Order.fieldFixedLengths 5,10,10
Item.fieldFixedLengths 5,10,10,10
trimContents N
indentFactor 4

Result

Input /wp-content/uploads/2015/03/in2_662870.png
Output /wp-content/uploads/2015/03/out2_662872.png

Scenario 3

Conversion where one and only one substructure type does not have key field to identify the type of record

Substructure Detail is configured as the generic record type

Sample test data for this scenario is from the following thread File Content Conversion – No Key Field in Detail Structure

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter
documentName MT_DeepPlain2XML
documentNamespace urn:equalize:com
recordsetStructure Header,Detail,Footer
genericRecordType Detail
keyFieldName Type
Header.fieldNames Type,HeaderField
Detail.fieldNames Field1,Field2,Field3,Field4,Field5
Footer.fieldNames Type,TrailerField
Header.keyFieldValue HDR
Footer.keyFieldValue TRL
Header.parent Root
Detail.parent Root
Footer.parent Root
Header.fieldFixedLengths 3,11
Detail.fieldSeparator ,
Footer.fieldFixedLengths 3,9
indentFactor 2

Result

Input

HDRXXXYYY12345

MM8100422373,M631530008  ,1530008,031615,17

MN8100424411,M631530008  ,1530008,031615,22

XY8100422373,M631530008  ,1530008,031615,17

TRLXXZZZ4321

Output /wp-content/uploads/2015/03/scenario3_710809.png

Scenario 4

Finally for the fourth scenario, we will use the same input file that was used in the XSLT approach. Actual configuration and testing screenshots are shown.

Default field separator for all substructures is used.

A log version of the message payload before conversion is saved.

Module configuration on an SFTP sender channel.

/wp-content/uploads/2015/03/module_662093.png

The ‘plain’ log version shows the flat file before conversion.

/wp-content/uploads/2015/03/pre_662068.png

The converted payload is sent to a receiver proxy without further mapping. As shown below, the target payload has been converted to a deeply nested XML structure.

/wp-content/uploads/2015/03/proxy_662067.png

The audit log shows the trace of steps being executed by the module.

/wp-content/uploads/2015/03/log_662097.png

Conclusion

With the completion of this second part for DeepFCCBean, we now have a generic, reusable and highly configurable custom adapter module that is able to handle conversion of flat file to deep XML and vice versa!

To report this post you need to login first.

20 Comments

You must be Logged on to comment or reply to a post.

  1. PAVAN Y

    Hi Eng Swee,

    Thanks for sharing the very useful information on Adapter Framework Modules

    When i trying to do the scenario 3 then it is throwing the error in channel monitor

    Error while sending message: Cannot send message to module processor com.sap.aii.af.lib.mp.module.ModuleException: Error during processing local bean: localejbs/Custom_AF_Modules/DeepFCCBean

    Error: com.sap.engine.services.jndi.persistent.exceptions720.NameNotFoundException: Path to object does not exist. First missing component is [Custom_AF_Modules], the whole lookup name is [localejbs/Custom_AF_Modules/DeepFCCBean].

    So my question is,

    Can we use this Custom Adapter Framework Modules without implementing them in the local system? If we need to implement them, then where we have to do this?


    And how can I overcome the  above error? Thanks in advance!!



    Regards

    Pavan

    (0) 
      1. Eng Swee Yeoh Post author

        Hi Praveen,

        Thank you for helping to point that out. I appreciate it. Looks like I need to update the info on my blogs so that the new name that should be used in the Module configuration is clearer.

        Pavan,

        Please use Custom_AF_Modules/FormatConversionBean as mentioned by Praveen.

        Rgds

        Eng Swee

        (0) 
    1. Eng Swee Yeoh Post author

      Hi Thorsten

      Adding a default endSeparator would be possible. I’d probably can find some time to get it done and out by sometime next week.

      <Structure>.endSeparator would be a bit more tricky as it involves revamping the simple file parser into something more elaborate. I’ll add it to the list of enhancements for this module in the future.

      I’m always up for others to collaborate on this – all the source codes are already publicly available. Let me know if you are interested.

      Rgds

      Eng Swee

      (0) 
        1. Eng Swee Yeoh Post author

          Hi Thorsten

          I’ve never had to work with flat files with end separator other than the common CRLF/LF. Can you let me know some sample files that is different? This will help me design the changes accordingly.

          Rgds

          Eng Swee

          (0) 
          1. Thorsten Düvelmeyer

            Hi Eng Swee,


            the end seperator is working. The error is because of emty segments at the end of the line – i think.


            /wp-content/uploads/2015/12/error_843646.jpg

            When i add a value before the LF it works.

            The exception is:


            Caused by: javax.ejb.EJBException: ASJ.ejb.(Failed in component: sap.com/com.equalize.xpi.af.modules.app) Exception raised from invocation of public com.sap.aii.af.lib.mp.module.ModuleData com.equalize.xpi.af.modules.util.AbstractModule.process(com.sap.aii.af.lib.mp.module.ModuleContext,com.sap.aii.af.lib.mp.module.ModuleData) throws com.sap.aii.af.lib.mp.module.ModuleException method on bean instance com.equalize.xpi.af.modules.FormatConversionBean@13d0a9ea for bean sap.com/com.equalize.xpi.af.modules.app*xml|com.equalize.xpi.af.modules.ejb.jar*xml|FormatConversionBean in application sap.com/com.equalize.xpi.af.modules.app.; nested exception is: java.lang.NullPointerException: while trying to invoke the method java.lang.String.trim() of a null object loaded from local variable ‘fieldValue’

            at com.sap.engine.services.ejb3.runtime.impl.RequestInvocationContext.proceedFinal(RequestInvocationContext.java:88)

            at com.sap.engine.services.ejb3.runtime.impl.AbstractInvocationContext.proceed(AbstractInvocationContext.java:166)

            at com.sap.engine.services.ejb3.runtime.impl.Interceptors_StatesTransition.invoke(Interceptors_StatesTransition.java:19)

            at com.sap.engine.services.ejb3.runtime.impl.AbstractInvocationContext.proceed(AbstractInvocationContext.java:179)

            at com.sap.engine.services.ejb3.runtime.impl.Interceptors_Resource.invoke(Interceptors_Resource.java:50)

            at com.sap.engine.services.ejb3.runtime.impl.AbstractInvocationContext.proceed(AbstractInvocationContext.java:179)

            at com.sap.engine.services.ejb3.runtime.impl.Interceptors_Transaction.doWorkWithAttribute(Interceptors_Transaction.java:37)

            … 49 more

            Caused by: java.lang.NullPointerException: while trying to invoke the method java.lang.String.trim() of a null object loaded from local variable ‘fieldValue’

            at com.equalize.xpi.af.modules.util.RecordTypeParameters.createNewField(RecordTypeParameters.java:201)

            at com.equalize.xpi.af.modules.util.RecordTypeParameters.extractLineContents(RecordTypeParameters.java:185)

            at com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter.extractLineToFieldList(DeepPlain2XMLConverter.java:142)

            at com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter.generateNestedContents(DeepPlain2XMLConverter.java:114)

            at com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter.parseInput(DeepPlain2XMLConverter.java:68)

            at com.equalize.xpi.af.modules.FormatConversionBean.processModule(FormatConversionBean.java:14)

            at com.equalize.xpi.af.modules.util.AbstractModule.process(AbstractModule.java:49)

            at sun.reflect.GeneratedMethodAccessor2122.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at com.sap.engine.services.ejb3.runtime.impl.RequestInvocationContext.proceedFinal(RequestInvocationContext.java:47)

            … 55 more



            (0) 
            1. Eng Swee Yeoh Post author

              Hi Thorsten

              There might be a bug, not too sure yet.

              Can you do me a favor and open a new thread on this and provide the following:-

              – Screenshot of the module parameters

              – Sample file

              Rgds

              Eng Swee

              (0) 
    1. Eng Swee Yeoh Post author

      Hi Otto

      Thanks for your comment. I haven’t used Apache Commons CSV before. I don’t mind considering it, but would you kindly elaborate further on what type of other flat files would benefit from using it?

      Thanks

      Eng Swee

      (0) 
  2. Eng Swee Yeoh Post author

    Hi Otto

    Thanks for the challenge to pick up adapter module with script support but I think I will pass at this moment – I’ve got too much on my plate to find time to look into it.

    Anyway, IMHO, I think if you already have such a solution, it would be more appropriate to share that yourself in the spirit of the SCN community instead of getting someone else to come up with the solution from scratch and share it.

    Rgds

    Eng Swee

    Edit: Hmmm… not sure why the original comment issuing the challenge was deleted.

    (0) 
    1. Eng Swee Yeoh Post author

      Hi Thorsten

      Thanks for your request. I had a look into the functionality in SAP and it is only allowed for the built-in FCC functionality in the adapter. For MessageTransformBean module, message splitting it not allowed.

      /wp-content/uploads/2016/03/split_907108.png

      One alternative is to use a similar approach as mentioned in AttachmentSplitterBean – Split attachments into child messages

      However, I need to spend some time exploring this. At the moment, for simplicity sake, the current converter does not have a “Recordset” level, so introducing that together with splitting will require significant effort and I can’t estimate how long it’ll take at the moment.

      Regards

      Eng Swee

      (0) 
  3. Jörg Hopmann

    Dear Eng Swee,

    Many thanks for your work, here is my gift to you …

    Regarding the Flat2XML-Conversion I have invested some time to re-implement com.equalize.xpi.util.converter.ConversionPlainInput  to handle any endSeparator instead of being fixed bound to newline (with the newLineReader). There was a “TODO” (also in the code remarked) still existing which I would like to fill now…

    It uses a java.util.Scanner to split the ‘lines’. This was needed for our implementation as our provider sends Messages with Key=Value&Key=Value… ampersand instead of new line.

    Here is the code which replaces your class:


    package com.equalize.xpi.util.converter;

    import java.io.IOException;
    import java.io.InputStream;
    import java.util.ArrayList;

    public class ConversionPlainInput {
    private final ArrayList<String> lineContents;

    public ConversionPlainInput(InputStream inStream, String endSeparator) throws IOException {

    java.util.Scanner scanner = new java.util.Scanner(inStream, “UTF-8”).useDelimiter(endSeparator);
    String theString = null;
    ArrayList<String> contents = new ArrayList<String>();

    while (scanner.hasNext()) {
    theString = scanner.next();
    contents.add(theString);

    }

    scanner.close();
    this.lineContents = contents;
    }

    public ConversionPlainInput(String string, String encoding, String endSeparator) throws IOException   {
    this(Converter.toInputStream(string, encoding), endSeparator);
    }

    public ConversionPlainInput(String string, String endSeparator) throws IOException {
    this(Converter.toInputStream(string), endSeparator);
    }

    public ConversionPlainInput(byte[] inputBytes, String endSeparator) throws IOException {
    this(Converter.toInputStream(inputBytes), endSeparator);
    }
    public ArrayList<String> getLineContents() {
    return this.lineContents;
    }
    }


     

    Additionally one has to replace and add few lines in public class DeepPlain2XMLConverter

    Add declaration:

    private String endSeparator;

    and

    gathering the value:

    this.endSeparator = this.param.getParameter(“endSeparator”, “\n”, true);

     

    Change the call of ConversionPlainInput because a parameter was added to

    this.plainIn = new ConversionPlainInput(inStream);

    this.plainIn = new ConversionPlainInput(inStream, this.endSeparator);

     

    By default newLine is still working as the default \n is exactly that, so the Module will work exactly the same way it did before without changing any parameter (Compatibility).

    We checked and it passed our tests.

     

    Maybe you can add this to your code one day.

    Kind regards Jörg

    (0) 

Leave a Reply