Skip to Content
Author's profile photo Eng Swee Yeoh

FormatConversionBean – One Bean to rule them all!

Update 9 Sep 2018: FormatConversionBean now available in CPI as well

Update 19 May 2015: Added new converter Base64EncodeConverter

Update 30 Apr 2015: Added new converter Base64DecodeConverter

Introduction

Over the past months, I have introduced various custom adapter modules that are configurable and reusable; aiming to handle various format conversions to/from XML that are not available in standard adapter modules.

I have recently refactored the different modules and consolidated them into a single module, FormatConversionBean. The benefits of this refactoring are:-

  • Single point of entry for the different types of conversion
  • Refactoring of factory class utilizing ConverterFactory allows dynamic instantiation of converters
  • Easily extensible to new converter formats by extending class AbstractModuleConverter
  • Conforms to the Open Close Principle whereby new converters can be added without modification to existing code

Converter Class List

Below is the comprehensive list of the converter classes that are part of FormatConversionBean now. This list will be updated as and when new converter classes are introduced in the future.

Usage of Module in Communication Channel

Module Processing Sequence

Number Module Name Type Module Key

<Depending on position of module in chain>

1) Asynchronous scenario,

Normally before the last module in channels

2) Synchronous scenario,

Before last module to convert request payload

After last module to convert response payload

Custom_AF_Modules/FormatConversionBean Local Enterprise Bean <Any Arbitrary Value>

Module Configuration

For configuration of parameters, refer to parameter list of each converter.

Source Code and Deployment Archive

The Java source codes for the module are located in the following GitHub repositories.

GitHub repository for equalize-xpi-modules

The EAR file for deployment (compiled on NWDS 7.31 SP13 Patch 0) is also available from the latest repository release below. The EAR file can be deployed directly on PI/PO systems with versions that are same/higher than 7.31 SP13 or 7.4 SP8.

Latest release for repository equalize-xpi-modules

To set up the EJB/EAR projects on NWDS for further customization, refer to the following.

Further Reference

If you plan to download the source codes into your own NWDS installation to make changes and/or develop new modules, the following blog would be useful for performing testing in NWDS prior to deployment into the PI system.

Standalone testing of Adapter Module in NWDS

Guide on EAR deployment via NWDS

Deploying .EAR files Using NWDS

Assigned Tags

      50 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Avinash Ayanala
      Avinash Ayanala

      Hi Eng,

      It was really great info, the EAR file has been successfully deployed and it is working fine.Thanks a lot.

      Regards,

      Avinash

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Avinash

      Thanks for your comment. You are the first to inform me that the modules' EAR file has been successfully deployed and I really appreciate this feedback. It is good to know that it is working on other systems apart from mine.

      Rgds

      Eng Swee

      Author's profile photo Ryan Crosby
      Ryan Crosby

      Nice work Eng Swee!

      I don't have any specific needs for this now but I will have to remember it for the future.

      Regards,

      Ryan Crosby

      Author's profile photo Shaibayan Chakrabarti
      Shaibayan Chakrabarti

      Hi Eng Swee Yeoh and Avinash Ayanala ,

      I am working currently on a requirement where the input is proxy and output is a file in excel format. I am using SAP PI 7.31 .

      I have some queries related to this.

      Is the below the correct EAR that we need to deploy or it contains only the JSON2XML converter. My exact need is to convert XML to EXCEL but I want to try the FormatConversionBean as a whole since it contains the entire module and I would not have to deploy any other separately. So do we need to deploy only the below EAR file?

      com.equalize.xpi.af.modules.app.ear 

      I will provide feedback and updates once I am done.

      Please do let me know.

      Thanks,

      Shaibayan

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Shaibayan

      Your queries will be answered directly in the thread below that you opened with the same details.

      Output File in Excel format in SFTP server?

      Rgds

      Eng Swee

      Author's profile photo Shaibayan Chakrabarti
      Shaibayan Chakrabarti

      Sure Eng.

      Thanks a lot.

      Author's profile photo Juan Vasquez
      Juan Vasquez

      Hi Eng, can you help me please with this issue

      I am using the Adapter Module success, in the receiver channel

      but i need to send some header values:

      Cache-Control: no-cache

      Content-Type: application/json

      Application-Secret: 6474b25daaaaaaaaaacc7c9c892195352354facd

      where can i send it?

      what is the better channel type soap http (axis)

      the content is in the right way, you can see in the immage

      thanks a lot2015-04-28 13_43_34-Message Editor - SAP NetWeaver Administrator.png

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Juan

      Thanks for your interest in FormatConversionBean. Glad to know that the JSON transformation is working well for you.

      As to your query regarding custom HTTP headers, since it's unrelated to this module, I'd suggest that you open a new thread in the forum for this issue instead.

      As always, please provide as much details/screenshots as possible in the thread, i.e. PI/PO version & SP, scenario overview & details, adapter types, error messages, etc.

      Rgds

      Eng Swee

      Author's profile photo Yee Loon Khoo
      Yee Loon Khoo

      Hi Eng Swee,

      Feedback on my trial.

      Deployed EAR in PO server 7.31, so far tested excel2xml working well, able to convert accordingly to xml. Nice! Many end users still love excel for manual upload, this will help a lot.

      Your blog, example and github code is very useful and inspiring. Thanks again.

      Regards,

      Yee Loon

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Dear Yee Loon

      Thanks for your feedback. I am very pleased to hear that is is working well. Please feel free to let me know if you do encounter any issues if you do get around to testing the modules more.

      Best regards,

      Eng Swee

      Author's profile photo Raghuraman S
      Raghuraman S

      Excellent and very useful blog.

      Both Json to XML and XML to Json conversions are working fine.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Raghu

      Thanks for the feedback. Glad to know that the conversions are working fine 🙂

      Rgds

      Eng Swee

      Author's profile photo Varun Mukund
      Varun Mukund

      HI Eng Swee Yeoh,

      Firstly, let me congratulate you on the blog. It is awesome and very helpful.

      I deployed the module and tested the Deep flat to XML conversion. I wanted to report a small bug. On the variable for fixedFieldLengths since you are doing a number format check after removing commas, the module will throw error any time you exceed 2,147,483,647 which is the maximum value int type supports. May be you can use a string.matches("\\d+?");


      Once again, amazing work 🙂


      Regards

      Varun

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Varun

      Thanks for your kind comments and also the feedback regarding the bug.

      Would you be able to provide some more details so that I can try to reproduce it? Can you provide screenshots of the module configuration as well as the error you are getting?

      Rgds

      Eng Swee

      Author's profile photo Varun Mukund
      Varun Mukund

      Hi Eng Swee Yeoh,

      To reproduce the error, you just need to do an FCC for a fixed field length file and you need to ensure that the recordset.fixedFieldLengths parameter has a value which exceeds 2147483647 after you remove the commas. I think if you test with a file having 5 columns with each column having a fixed length of 25, you should get the error.

      The exact location of error is at the class com.equalize.xpi.af.modules.util.RecordTypeParameters on line 54. I removed the try/catch block and used the below snippet. It worked like a charm 🙂

      if(!lengthsWithoutComma.matches("\\d+?"))

      {

      throw new ModuleException("Maintain only integers separated by commas for '" + fieldFixedLengthsName);

      }

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Varun

      Ah, thanks for elaborating further. I've fixed the bug accordingly and published a new release for the EAR file as well as the source code.

      Thanks again for reporting the bug. It is much appreciated! 🙂

      Rgds

      Eng Swee

      Author's profile photo Anil Kumar Vaitla
      Anil Kumar Vaitla

      Hi Eng Swee Yeoh,

      Tried DeepPlain2XML and XML2DeepPlain in the File and JMS adapters and they worked great.

      Thanks for the detailed blog it was very helpful.

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Anil

      Really pleased to hear that they are working well. 🙂

      Rgds

      Eng Swee

      Author's profile photo Vadim Klimov
      Vadim Klimov

      Hi Eng Swee,

      Thank you for developing and sharing this module - very helpful, and your efforts are very much appreciated. Recently we used it for XML to JSON conversion in JMS based integration scenario - worked perfectly well.

      One question to you. Manifest contained in a project, specifies implementation vendor ID = "sap.com". As a result, corresponding component gets deployed with vendor = "sap.com", which is normally reserved / used by SAP standard components. Would you suggest downloading source code provided by you, adopting manifest and building deployable, or may you adjust vendor identifier to something different from "sap.com" and rebuild EAR so that it already contains custom vendor identifier?

      Regards,

      Vadim

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Vadim

      Thank you for your feedback. I'm really glad to hear that the conversion worked well.

      Regarding the Manifest file, to be honest, this is something that I'm not that familiar with. I checked both the EAR and EJB projects - there is no manifest file maintained in the EAR project and the one in the EJB project does not specify the vendor ID (very little info in that file).

      It looks like the MANIFEST.MF file within the EAR deployable file is automatically generated when I perform Export > SAP EAR file for the EAR project. I guess I can of course manually edit the entry in the generated EAR file, but do you know if there is some other automatic way to achieve this - maybe manually maintaining a manifest file in the EAR project?

      Have you adjusted the vendor identifier before for some custom adapter module that you have developed? Let me know if you do, I'm all up for modifying it accordingly for the EAR files I provide.

      Rgds

      Eng Swee

      Author's profile photo Ryan Crosby
      Ryan Crosby

      Hi Eng Swee,

      I was able to achieve this in the past by adding the following tag into the application-j2ee-engine.xml for the EAR file:

      <provider-name>idexx.com</provider-name>

      Regards,

      Ryan Crosby

      Author's profile photo Vadim Klimov
      Vadim Klimov

      Hi Eng Swee,

      I assume, if manifest files (MANIFEST.MF and SAP_MANIFEST.MF) are not present in project structure, then NWDS auto-generates them when building EJB and EAR projects, and uses standard attribute values (this is where vendor = "sap.com" comes from). Since in built EAR file, after I decompile it, I can see manifest files:

      com.equalize.xpi.af.modules.app decompiled.png

      I resolved this issue by explicit creation of manifest files within a project. I didn't create them in NWDS (but it is worth trying this - probably, it will be much simpler), but used build tools and specified what kind of manifest file I would like to assemble when building EJB project. To be more precise, I used Gradle (http://gradle.org/) in order to automate bundling the deployable. In Gradle script, I made use of Java plugin and its property manifest, where I provided attributes and their custom values which I want to put into manifest file. Usage of Gradle purely for this task may be a bit overkilling - but I used Gradle also for dependency management (especially when adapter module / custom job was using external libraries and I wanted to insert some of them, but not all, into the same deployable) and entire compilation and assembly of JARs and EARs, so manifest creation was just one of tasks of executed Gradle script.

      Regards,

      Vadim

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Vadim, Ryan

      Thank you so much for both your feedback. I've definitely learnt something new 🙂

      I've managed to adjust the value generated in the MANIFEST.MF file by adding the provider name into the application-j2ee-engine.xml file as suggested by Ryan. That worked really well and once it's deployed into the SAP system, the vendor value is reflected correctly in the different views in NWA.

      I've updated the source code and also released a new EAR file in the GitHub repository for this change.

      Thanks also for the info on Gradle. Will definitely have to look into it at some point! 😉

      Rgds

      Eng Swee

      Author's profile photo sungtae bang
      sungtae bang

      Thank you

      The information provided by you became great help to me.

      But i was occur problem in my source of NWDS

      APIAccessFactory.java and MessageLogger.java Is not find in MessageLoggerHelper.java

      Where is find JAVA Source? and .jar download url?

      Regards,

      Bang

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Bang

      Glad to know this was helpful for you.

      The pre-packaged Adapter library that comes with NWDS 7.31 are reduced versions and do not contain all classes for some packages. As such, you will need to get the full JAR file for com.sap.aii.af.ms.ifc_api.jar from your PI server.


      Refer screenshot below that shows the package and classes that are in that JAR file. This will enable MessageLoggerHelper.java to compile successfully.

      /wp-content/uploads/2016/01/lib_867776.png

      Refer to the following Wiki on where to get that file in your PI system.

      Where to get the libraries for XI development - Process Integration - SCN Wiki

      After you have retrieved that file, create a User Library in NWDS (give it any name) that includes that JAR file. Then add the newly created User Library to the EJB project's build path.

      Rgds

      Eng Swee

      Author's profile photo sungtae bang
      sungtae bang

      Dear Eng

      Thank you, kind explanation for me

      Is not found MessageLogger.class in com.sap.aii.af.ms.ifc.api.jar in PI 7.3

      So, i'm going to after annotation in MessageLogger.class

      Regards,

      Bang

      Author's profile photo Former Member
      Former Member

      Hi Bang,

      can i deploy the FormatConversionBean on our PI 7.3 SP10 ?

      Author's profile photo Former Member
      Former Member

      Hi Eng,

       

      Very nice work with providing us this wholesome solution.

      I am planning to use this solution for my below requirement. I have JSON file which i will need to read and update that data into Database tables.

       

      So can you please let me know once i read this file using File adapter and custom JSONtoXML converter module, how can i map it to the database tables?

       

      Also i will let you know once i deploy your module in our server, we are on SAP PO 7.4 SP9 how does it work out.

       

      Thanks,

      Hetal

      Author's profile photo Former Member
      Former Member

      Any idea how can i deploy the EAR file on SAP PO 7.4? I do not need to make any changes in the code you provided or any enhancements, hence if you can let me know how to deploy it directly on SAP PO.

       

      Do i still need to have NWDS to deploy the file on the server?

       

      Thanks,

      Hetal

      Author's profile photo vk k
      vk k

       

      Hi Eng Swee,

      I have been fan of this bean and I have been using this quiet a number of times.

      Recently, I ran into trouble when I am using xml2json converter in this bean, when I am passing 800 + rows of XML, total of 1700kb , it takes around 9 minutes to perform the conversion.

      Do you think it is normal ?

       

      Regards,

      Vk

       

      Author's profile photo Angelo Paolo Bandelaria
      Angelo Paolo Bandelaria

      Hi Eng Swee,

      I am facing issues in downloading the EAR file from the link provided. For some reason the download just stops. Not sure if there is an issue with github. Is there another location where this can be downloaded?

      https://github.com/engswee/equalize-xpi-modules/releases/latest

       

       

      Best Regards,

      Paolo

      Author's profile photo Angelo Paolo Bandelaria
      Angelo Paolo Bandelaria

      Nevermind. Link seems to be working now. Thank you!

      Paolo

      Author's profile photo Peter Wallner
      Peter Wallner

      Thank you Eng Swee - it works perfectly. We are on SAP PO 7.5

      Best regards, Peter

       

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Thanks for the feedback, Peter. Real glad to know that is it working fine, even on the newer version 😉

      Author's profile photo Rajesh Aravapalli
      Rajesh Aravapalli

      Hi Peter,

      How did you deploy the .ear file?

      Did you deploy the .ear file directly using SUM?

       

       

      Regards,

      Rajesh

      Author's profile photo Peter Wallner
      Peter Wallner

      Hello Rajesh,

      No for deploying - adapter modules or EAR files - I always use the NWDS.

      There is a "deploy view" in NWDS you can use also for external files such as EAR files. It works well and is really practical.

      BEst regards, Peter

      Author's profile photo Rajesh Aravapalli
      Rajesh Aravapalli

      Hi Eng Swee,

       

      We got the below error while installing  the com.equalize.xpi.af.modules.app.ear file directly   using SUM.

      Are we missing anything?

      Error:

      Component equalize.com/com.equalize.xpi.af.modules.app cannot be installed. Component com.equalize.xpi.af.modules.app is not part of any existing software components.

       

      Thanks in Advance.

       

      Regards,

      Rajesh.A

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      This EAR package is not packed as a DC that is part of a Software Component (typical of NWDI-based development).

       

      I have never used SUM to deploy it so I suggest you use other methods - Telnet, NWDS, etc.

      Author's profile photo Rajesh Aravapalli
      Rajesh Aravapalli

      Thanks Eng..I was able to deploy it using telnet.

      I am getting null error in the channel.

       

      Channel Configuration:

       

      File loaded :

       

       

       

      Regards,

      Rajesh.A

       

       

       

       

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      I'd suggest you try it on a receiver channel first and see if there are any errors logged on it which will provide more details.

      Author's profile photo Rajesh Aravapalli
      Rajesh Aravapalli

      Hi Eng Swee,

      Excel2XMLTransformer is working with xls format but I am getting the null error with 2010 xlsx format.

       

       

      Regards,

      Rajesh.A

      Author's profile photo Vikash Gupta
      Vikash Gupta

      Hello Eng Swee,

       

      Thank you for this wonderful solution. As in the steps I downloaded the EAR file from GitHub (com.equalize.xpi.af.modules.app.ear) and deployed it directly in my PO system using Deployment perspective of NWDS . I have PO 7.5 system.

      I intend to use DynamicAttributeChangeBean. But when I give this in the receiver channel module, the log shows error -

      "Unable to obtain EJB home interface [Root exception is java.lang.NoSuchMethodException: com.sap.engine.services.jndi.implclient.ClientContext.create()]"

      The Deployment seems to be correct -

       

       

      Is there any post step that needs to be done or if I have missed anything.

      Thanks in Advance.

      Regards

      Vikash

       

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Try and check in the JNDI Browser and/or Application Modules in NWA to see if the EJB Modules are listed there. I don't have access to a PI system at the moment so can't provide more details.

       

      Also, double check your module configuration (sequence, name, type, etc).

      Author's profile photo Alexandre Rezende
      Alexandre Rezende

      Tank's  Eng Swee Yeoh , this module adapter solved my problem in my current project!

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Fantastic! Great to hear that 😉

      Author's profile photo Dhanvanttri Parameswaran
      Dhanvanttri Parameswaran

      I have a Bug to report, when i am converting XML to JSON using the module, £ symbol is getting converted to �. Please help. Thanks in advance.

      Author's profile photo Smith Smith
      Smith Smith

      HI Eng,

      I am working on Proxy to REST sync interface. I have few input fields which has value '/' in it ex: ( AOUS/12 and some date fields(23/04/2020).

      at the JSON output payload it is coming as AOUS\/12 and dates are coming as 23\/04\/2020.

      there is an addition of '\' in between. I tried to declare the field as "string" in custom xml/JSON coversion rules. but still its not working. i also tried to declare the field as "Date" in the datatype but still no luck because in the mapping its correctly populated.

      the issue is coming at the REST channel while converting XML to JSON format.

      Can you please help me how to get rid of this extra '\' at the channel level.

       

      thanks

      Smith

      thanks

      Smith

      Author's profile photo Cem Unal
      Cem Unal

      Hi Eng,

      First of all thank you for this useful blog, it is very helpful.

      I have been dealing with a project which i need to send a zipped file to rest api. My problem is the content of the file must be in json.

      I am using rest receiver and "AF_Modules/MessageTransformBean" and "AF_Modules/PayloadZipBean" modules, you can see in screenshots.

      can you help me with some advice on this please?

       

      Author's profile photo Steve Lewis
      Steve Lewis

      Hello Eng

      Thank you for creating this amazing module. I have been using it successfully to convert relatively small xml files to excel.

      In one scenario I am also using the payloadZipBean module to unzip the file before it passes to the FormatConversionBean. All of this works great for small files. However, I have a case with a large xml file(200mb) which causes the Receiver File adapter to hang with the following error:

      • Adding message to blacklist. Reason: Message (in delivering state) found during startup
      • Message added to blacklist
      • Moved message to the non-delivered state. Reason: Blacklisted message (in delivering state) found during startup

      It appears the conversion is taking too long for the large file to convert and the message gets stuck in Delivering then blacklisted. If I remove the FormatConversionBean I am able to process the file successfully as an xml. Do we have a known fix for this use case?

      Thanks for reading and I look forward to your reply.

      Thanks

      Steve

      Author's profile photo Eng Swee Yeoh
      Eng Swee Yeoh
      Blog Post Author

      Hi Steve,

       

      200MB is definitely a big file, and the module has never been tested for such use case. Maybe the memory handling is not optimised for it.

       

      I am no longer actively working with PI, so I would suggest that you clone the Git repository and try to test it locally, and make any fixes as required.

       

      All the best!

      Eng Swee