Skip to Content
Technical Articles

DeepFCCBean – The better FCC at meeting your deep (structure) needs! (Part 1 – Deep XML to Flat File)

Update 4 Jul 2019: Add parameters defaultEnclosureSign & defaultEnclosureSignEscape for CPI version.

Update 2 Nov 2018: Default parsing in CPI will be using SAX-based XmlSlurper.

Update 4 Sep 2018: Now available in CPI as well.

Update 25 Mar 2015: DeepFCCBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass.

Update 12 Mar 2015: Update links to second part of the series. Parameter for conversion type has been renamed to XML2DeepPlain. Reformat parameter reference section to split parameters applicable for field separator based and fixed length based conversion.

Introduction

In my recent two-part series on using XSLT to handle FCC of deep structures, I mentioned that even after 10 years, the standard FCC functionality provided has not changed significantly. This is unlikely to change as there are other areas of interest which SAP is focusing on these days. Borne out of my preference for reusable and configurable solutions, I decided to challenge myself to come up with an adapter module solution to hopefully fill this gap, if possible maybe once and for all!

In this two-part series, I will share about DeepFCCBean, the custom adapter module for handling conversion of flat files to deep XML structures and vice versa. As much as possible, the module’s behaviors and functionalities are similar to standard SAP’s FCC.

This first part covers deep XML to flat file conversion, while the second part covers flat file to deep XML conversion.

Source Code

Refer to following blog on location of source code and/or EAR deployment file.

FormatConversionBean – One Bean to rule them all!

Usage of Module in Communication Channel

Module Processing Sequence

Number Module Name Type Module Key

<Depending on position of module in chain>

1) Asynchronous scenario,

Normally before the last module in channels

2) Synchronous scenario,

Before last module to convert request payload

After last module to convert response payload

Custom_AF_Modules/FormatConversionBean Local Enterprise Bean <Any Arbitrary Value>

Module Parameter Reference

Below is a list of the parameters for configuration of the module for deep XML to flat file conversion (conversionType = ‘com.equalize.xpi.af.modules.deepfcc.DeepPlain2XMLConverter’). Certain parameters will automatically inherit the default values if it is not configured.

The parameters are separated into parameters that are common for this conversion type, and also parameters that are specific for field separator based or fixed length based conversion.

1) Common Parameters

Parameter Name Allowed values Default value Remarks
converterClass

PI – com.equalize.xpi.af.modules.deepfcc.XML2DeepPlainConverter

CPI – com.equalize.converter.core.XML2DeepPlainConverter

Required field. Determines conversion class
encoding UTF-8 Defines encoding of output plain text file
recordsetStructure Required field. List of substructure names separated by comma
<StructureName>.endSeparator System specific new line Defines end of record separator
useDOM Y, N N

Available only in CPI

Parser used in parsing input XML

  • N = SAX-based XmlSlurper
  • Y = Document Object Model (DOM) based parser
messageLog

pre,

post

Saves a log version of the message that is viewable in Message Monitor

  • pre = saves payload before conversion
  • post = saves payload after conversion

Available only in PI

logLocation

Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated.

Available only in PI

2) Field Separator Conversion Parameters

Parameter Name Allowed values Default value Remarks
defaultFieldSeparator

Defines a default field separator for all substructures that do not have <StructureName>.fieldSeparator explicitly configured

Note: Cannot be used together with <StructureName>.fieldFixedLengths

defaultEnclosureSign

Specify a string that acts as a text delimiter for each field. For example: ” (double quote character).

Available only in CPI

defaultEnclosureSignEscape

Specify a string that replaces the text delimiter defaultEnclosureSign if it occurs within a field that it delimits. For example: “” will replace “.

Available only in CPI

<StructureName>.fieldSeparator Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated

3) Fixed Length Conversion Parameters

Parameter Name Allowed values Default value Remarks
<StructureName>.fieldFixedLengths Integer separated by commas Either <StructureName>.fieldFixedLengths or <StructureName>.fieldSeparator must be populated
<StructureName>.fixedLengthTooShortHandling

Error,

Cut,

Ignore

Error Applicable when <StructureName>.fieldFixedLengthsis configured. Determines behavior when actual length of field exceeds corresponding length configured:

  • Error = Document processing is cancelled
  • Cut = Value is cut to the maximum permitted length
  • Ignore = The value is accepted even though its length exceeds the permitted value. Subsequent columns are moved accordingly

Similar to standard FCC functionality, fieldSeparator and endSeparator parameters support non-printable ASCII characters as follows:

  • characters encoded as hexadecimal values in the form of ‘0xHH’ (including quotation marks)
  • line break specified using ‘nl’ (including quotation marks)

Example Scenarios

Here are some example scenarios of the behavior of the conversion based on different configuration options.

Scenario 1

3 level deep input XML with delimited output

Field separator: Delivery and Order using comma. Item using tab (specified in hexadecimal)

End separator: Delivery has additional ZZZZ followed by line break. Order and Item not specified therefore using default line break

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.deepfcc.XML2DeepPlainConverter
recordsetStructure Delivery,Order,Item
Delivery.fieldSeparator ,
Delivery.endSeparator ZZZZ’nl’
Item.fieldSeparator ‘0x09’
Order.fieldSeparator ,

Result

Input

<?xml version=”1.0″ encoding=”UTF-8″?>

<ns0:MT_Deep xmlns:ns0=”urn:equalize/DeepFCC”>

<Delivery>

  <Type>D</Type>

  <DeliveryNo>Delivery2</DeliveryNo>

  <Order>

  <Type>O</Type>

  <DeliveryNo>Delivery2</DeliveryNo>

  <OrderNo>Order2_A</OrderNo>

  <Item>

    <Type>I</Type>

    <OrderNo>Order2_A</OrderNo>

    <ItemNo>10</ItemNo>

    <Quantity>90</Quantity>

  </Item>

  <Item>

    <Type>I</Type>

    <OrderNo>Order2_A</OrderNo>

    <ItemNo>20</ItemNo>

    <Quantity>80</Quantity>

  </Item>

  </Order>

</Delivery>

</ns0:MT_Deep>

Output /wp-content/uploads/2015/03/out1_659257.png

Scenario 2

3 level deep input XML with fixed length output

Length too short handling: Order set to ignore so long fields can potentially push subsequent fields. Item set to cut to maximum configured length

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.deepfcc.XML2DeepPlainConverter
recordsetStructure Delivery,Order,Item
Delivery.fieldFixedLengths 5,10
Order.fieldFixedLengths 5,5,10
Order.fixedLengthTooShortHandling Ignore
Item.fieldFixedLengths 5,5,10,10
Item.fixedLengthTooShortHandling Cut

Result

Input

<?xml version=”1.0″ encoding=”UTF-8″?>

<ns0:MT_Deep xmlns:ns0=”urn:equalize/DeepFCC”>

<Delivery>

  <Type>D</Type>

  <DeliveryNo>Delivery2</DeliveryNo>

  <Order>

  <Type>O</Type>

  <DeliveryNo>Delivery2</DeliveryNo>

  <OrderNo>Order2_A</OrderNo>

  <Item>

    <Type>I</Type>

    <OrderNo>Order2_A</OrderNo>

    <ItemNo>10</ItemNo>

    <Quantity>90</Quantity>

  </Item>

  <Item>

    <Type>I</Type>

    <OrderNo>Order2_A</OrderNo>

    <ItemNo>20</ItemNo>

    <Quantity>80</Quantity>

  </Item>

  </Order>

</Delivery>

</ns0:MT_Deep>

Output /wp-content/uploads/2015/03/out2_659300.png

Scenario 3

Finally for the third scenario, we will use the same input file that was used in the XSLT approach. Actual configuration and testing screenshots are shown.

Default field separator for all substructures is used.

A log version of the message payload after conversion is saved.

Module configuration on an SFTP receiver channel.

/wp-content/uploads/2015/03/module_659301.png

The ‘plain’ log version shows the payload after conversion of deep XML to flat file.

/wp-content/uploads/2015/03/plain_659302.png

The audit log shows the trace of steps being executed by the module.

/wp-content/uploads/2015/03/log_659315.png

Conclusion

With DeepFCCBean, content conversion of deeply nested flat files no longer require new custom development.

Do note that it is technically possible for DeepFCCBean to handle the flat files that can be converted by standard FCC. However, it is not the intention of DeepFCCBean to replace SAP’s standard functionality, but to complement it in handling deep structures. As such, for flat files that can be converted by standard FCC, it is recommended that standard FCC is used.

Upcoming will be the second part of this series covering flat file to deep XML conversion with DeepFCCBean. Watch this space!

Second part is now out at DeepFCCBean – The better FCC at meeting your deep (structure) needs! (Part 2 – Flat File to Deep XML)

24 Comments
You must be Logged on to comment or reply to a post.
  • I am genuinely excited to see Part II Eng Swee Yeoh. Well done!!

    Correct me if I am wrong, but this is likely to resolve my problem of having to build my own Java adapters for flat files that don’t match a simple header/detail/trailer pattern.

    I think I even have a potential use in the next couple of weeks.

    • Hi Jason

      Yes, the intention is to have a “build/deploy once, use unlimited times” solution to complement existing standard FCC functionality. There will be certain prerequisites for Part 2 as flat file patterns come in all shapes and sizes, but hopefully it will be able to handle most of it. Once the initial version is out, I will try my best to keep enhancing it for other feasible pattern.

      Won’t be too long till Part 2 is out. Watch this space 🙂

      Rgds

      Eng Swee

    • Hi Thorsten

      Can you please elaborate further on your requirement? If possible, please provide an example input XML and the expected output flat file.

      Regards

      Eng Swee

  • Hi Eng Swee,

    Appreciate if you could please provide it the below output is possible in FCC?

    Header A1 | Header B1 | Header C1 | Header D1 | Header E1

    Header A2 | Header B2 | Header C2 | Header D2 | Header E2

    <br>

    HeaderItem A1 | HeaderItem B1 | HeaderItem C1 | HeaderItem D1 | HeaderItem E1

    Item A1 | Item B1 | Item C1 | Item D1 | Item E1

    Item A2 | Item B2 | Item C2 | Item D2 | Item E2

    thanks for your inputs!

  • Hello

    I try the  Custom_AF_Modules/FormatConversionBean  with converterClass com.equalize.xpi.af.modules.deepfcc.XML2DeepPlainConverter but have the Problem that xxx.fieldSeparator = ‘nl’ will not work, in the outpot just ‘nl’ is adding after each field.

    What ist the trace.location name for FormatConversionBean  just for debugging the bean ?

     

    Regards

    Ralf

     

    Similar to standard FCC functionality, fieldSeparator and endSeparator parameters support non-printable ASCII characters as follows:

    • characters encoded as hexadecimal values in the form of ‘0xHH’ (including quotation marks)
    • line break specified using ‘nl’ (including quotation marks)

     

    • Headers are not supported due to the nature of the deeply nested output file. Typically, headers are only seen in simple flat files with just one structure type – and that you can use SAP’s standard functionality to achieve it.

  • Hi Eng,

    I tried to use the first example scenario on CPI tenant. I am executing the iFlow from PostMan, but I am getting an error :

    An internal server error occured: java.lang.reflect.MalformedParameterizedTypeException@ line 19 in script1.groovy.
    The MPL ID for the failed message is : null
    For more details please check tail log.

    the error is happening on cb.convert()

    def fcb = new FormatConversionBean(message.exchange, message.getProperties())messageLog.addAttachmentAsString(“BEGIN FCB”, “2”, “text/plain”);
    def output = fcb.convert()
    messageLog.addAttachmentAsString(“BEGIN FCB”, “3”, “text/plain”);
    message.setBody(output)
    return message

    Any idea what is causing this error ?

    Thanks, radim

    • Hi Radim

       

      This seems to be working fine on my tenant:-

       

      Are you using version 1.3.0 that was just released this week? If yes, then it is dependent on the Groovy version that was updated to 2.4.12 on the CPI tenant. I would have expected this update across all CPI tenants – CPITracker reports this to have happened on 24 Mar. My tenant’s build number is 3.9.13 as of the time of this comment.

       

      You can find out the Groovy version of your tenant with the following script.

      import com.sap.gateway.ip.core.customdev.util.Message;
      
      def Message processData(Message message) {
          message.setBody("Groovy version is ${GroovySystem.version}")
      
          return message
      }

       

      If you are on the older version of Groovy (2.2.1), then please download version 1.2.0 instead.

       

      Regards

      Eng Swee

  • Hi Eng,

    thanks, you was right. The Groovy version on the CPI tenant is 2.4.12. So I downloaded the version 1.2.0 and now it is working like like a charm.

    I have one more question, what would be the best strategy for processing large CSV files using XML2DeepPlainConverter convertor, as files can be pretty large (a million rows) ? We are building a iflow in the CPI to fetch data from SAP system, convert it into CSV file and store in external system.

     

    regards,

    Radim

  • Hi Eng,

    Is there a way to add text-delimiters to the output? The other converter class (DeepPlain2XMLConverter) does have enclosureSignBegin  and enclosureSignEnd when parsing CSV, but I miss that for this class, at least in the documentation.

    Great work on the bean by the way, it’s works perfectly otherwise.

     

    Kr.

    Ivo

    • Hi Ivo

       

      As it is, it does not have the functionality to add begin/end enclosing signs. I developed it following the same design as standard MessageTransformBean which also did not have such functionality. A workaround would be to add the enclosing signs during mapping step prior it is finally processed by the adapter module.

       

      It seems fair to enhance the converter class with such functionality, but I won’t be able to at the moment as I don’t have access to a PI system, so I’m not actively making changes to the code base since I can’t test them on a system.

       

      If you are interested, I’d suggest you fork the repository and make the changes accordingly.

       

      Regards

      Eng Swee

      • Hi Eng,

        I finally got to test your latest release for this bean. Thanks for adding the support for text qualifiers! They work perfectly, and it’s easy to setup.

        I now added your bean to an client’s integration and this should hit production within a week or two. I will make sure to link to your github repo and this blogpost in the script I use to do the XML2DeepCSV conversion.

        Kr.

        Ivo

        • Hi Ivo

           

          From your initial request, I wasn’t sure if you were refering to the PI version or the CPI version. Looks like you are using the CPI version, and I am glad to hear that it’s working fine with the newly added functionality. And thanks for letting me know.

           

          All the best in the go-live of your integration scenario.

           

          Regards

          Eng Swee

  • Hi Eng,

    This conversion bean is really great. I have been working with it lately with great satisfaction

    Recently I came across the next situation :

    I have an xml where one of the elements contains a 0 with leading blank spaces

    <Points> 0</Points>

    I have configured the following parameters (besides the other standard ones) :

    message.setProperty(“row.fieldNames”,”BillTo,CardID,CardName,Points,Flagdelete,Date”)
    message.setProperty(“row.fieldFixedLengths”, “6,13,50,5,1,8”)

    The output is a flat file (not all spaces are in there but it gives the idea (the ‘    0’ is written as ‘0    ‘) :

    8137591000027369233              0    130072018
    8137591000027369264              0    130072018

    So altough the field Points contains 4 spaces and a 0, the spaces are not in the output.

    Is there a solution to overcome this problem ?

    Thanks for your input

    Regards,

    Jan

    • Hi Jan

       

      Thanks for bringing up this issue. I had a look into it further and indeed it was automatically trimming away leading/trailing spaces from XML text nodes. I can’t remember why I did it that way. I looked up further into it and believe it shouldn’t have been implemented that way.

       

      I have now fixed it with a new release 2.1.1.

       

      I have also implemented an additional parameter trim – if this is set to Y, then it will behave as before.

       

      Do have a try and let me know how it goes.

       

      Best regards

      Eng Swee