FormatConversionBean – One Bean to rule them all!
Update 9 Sep 2018: FormatConversionBean now available in CPI as well
Update 19 May 2015: Added new converter Base64EncodeConverter
Update 30 Apr 2015: Added new converter Base64DecodeConverter
Introduction
Over the past months, I have introduced various custom adapter modules that are configurable and reusable; aiming to handle various format conversions to/from XML that are not available in standard adapter modules.
I have recently refactored the different modules and consolidated them into a single module, FormatConversionBean. The benefits of this refactoring are:-
- Single point of entry for the different types of conversion
- Refactoring of factory class utilizing ConverterFactory allows dynamic instantiation of converters
- Easily extensible to new converter formats by extending class AbstractModuleConverter
- Conforms to the Open Close Principle whereby new converters can be added without modification to existing code
Converter Class List
Below is the comprehensive list of the converter classes that are part of FormatConversionBean now. This list will be updated as and when new converter classes are introduced in the future.
| Converter Class | Source Format | Target Format | Reference |
|---|---|---|---|
| Excel2XMLTransformer | Excel | XML | ExcelTransformBean Part 1: Convert various Excel formats to simple XML easily |
| XML2ExcelTransformer | XML | Excel | ExcelTransformBean Part 2: Convert simple XML to various Excel formats easily |
| DeepPlain2XMLConverter | Deep Plain Text | XML | DeepFCCBean – The better FCC at meeting your deep (structure) needs! (Part 2 – Flat File to Deep XML) |
| XML2DeepPlainConverter | XML | Deep Plain Text | DeepFCCBean – The better FCC at meeting your deep (structure) needs! (Part 1 – Deep XML to Flat File) |
| JSON2XMLConverter | JSON | XML | JSONTransformBean Part 1: Converting JSON content to XML |
| XML2JSONConverter | XML | JSON | JSONTransformBean Part 2: Converting XML to JSON content |
| Base64DecodeConverter | Base64 (Plain/XML) | Binary | Base64DecodeConverter – Base64 decoding made easy! |
| Base64EncodeConverter | Binary | Base64 (Plain/XML) | Base64EncodeConverter – Base64 encoding made easy! |
Usage of Module in Communication Channel
Module Processing Sequence
| Number | Module Name | Type | Module Key |
|---|---|---|---|
|
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload |
Custom_AF_Modules/FormatConversionBean | Local Enterprise Bean | <Any Arbitrary Value> |
Module Configuration
For configuration of parameters, refer to parameter list of each converter.
Source Code and Deployment Archive
The Java source codes for the module are located in the following GitHub repositories.
GitHub repository for equalize-xpi-modules
The EAR file for deployment (compiled on NWDS 7.31 SP13 Patch 0) is also available from the latest repository release below. The EAR file can be deployed directly on PI/PO systems with versions that are same/higher than 7.31 SP13 or 7.4 SP8.
Latest release for repository equalize-xpi-modules
To set up the EJB/EAR projects on NWDS for further customization, refer to the following.
| NWDS version | Reference |
|---|---|
| 7.3x or higher | Setup com.equalize.xpi.af.modules Adapter Module project in NWDS easily using EGit |
| 7.1x | Recompile com.equalize.xpi.af.modules as EJB 2.1 modules in NWDS 7.1x |
Further Reference
If you plan to download the source codes into your own NWDS installation to make changes and/or develop new modules, the following blog would be useful for performing testing in NWDS prior to deployment into the PI system.
Standalone testing of Adapter Module in NWDS
Guide on EAR deployment via NWDS
Hi Eng,
It was really great info, the EAR file has been successfully deployed and it is working fine.Thanks a lot.
Regards,
Avinash
Hi Avinash
Thanks for your comment. You are the first to inform me that the modules' EAR file has been successfully deployed and I really appreciate this feedback. It is good to know that it is working on other systems apart from mine.
Rgds
Eng Swee
Nice work Eng Swee!
I don't have any specific needs for this now but I will have to remember it for the future.
Regards,
Ryan Crosby
Hi Eng Swee Yeoh and Avinash Ayanala ,
I am working currently on a requirement where the input is proxy and output is a file in excel format. I am using SAP PI 7.31 .
I have some queries related to this.
Is the below the correct EAR that we need to deploy or it contains only the JSON2XML converter. My exact need is to convert XML to EXCEL but I want to try the FormatConversionBean as a whole since it contains the entire module and I would not have to deploy any other separately. So do we need to deploy only the below EAR file?
com.equalize.xpi.af.modules.app.ear
I will provide feedback and updates once I am done.
Please do let me know.
Thanks,
Shaibayan
Shaibayan
Your queries will be answered directly in the thread below that you opened with the same details.
Output File in Excel format in SFTP server?
Rgds
Eng Swee
Sure Eng.
Thanks a lot.
Hi Eng, can you help me please with this issue
I am using the Adapter Module success, in the receiver channel
but i need to send some header values:
Cache-Control: no-cache
Content-Type: application/json
Application-Secret: 6474b25daaaaaaaaaacc7c9c892195352354facd
where can i send it?
what is the better channel type soap http (axis)
the content is in the right way, you can see in the immage
thanks a lot
Hi Juan
Thanks for your interest in FormatConversionBean. Glad to know that the JSON transformation is working well for you.
As to your query regarding custom HTTP headers, since it's unrelated to this module, I'd suggest that you open a new thread in the forum for this issue instead.
As always, please provide as much details/screenshots as possible in the thread, i.e. PI/PO version & SP, scenario overview & details, adapter types, error messages, etc.
Rgds
Eng Swee
Hi Eng Swee,
Feedback on my trial.
Deployed EAR in PO server 7.31, so far tested excel2xml working well, able to convert accordingly to xml. Nice! Many end users still love excel for manual upload, this will help a lot.
Your blog, example and github code is very useful and inspiring. Thanks again.
Regards,
Yee Loon
Dear Yee Loon
Thanks for your feedback. I am very pleased to hear that is is working well. Please feel free to let me know if you do encounter any issues if you do get around to testing the modules more.
Best regards,
Eng Swee
Excellent and very useful blog.
Both Json to XML and XML to Json conversions are working fine.
Hi Raghu
Thanks for the feedback. Glad to know that the conversions are working fine 🙂
Rgds
Eng Swee
HI Eng Swee Yeoh,
Firstly, let me congratulate you on the blog. It is awesome and very helpful.
I deployed the module and tested the Deep flat to XML conversion. I wanted to report a small bug. On the variable for fixedFieldLengths since you are doing a number format check after removing commas, the module will throw error any time you exceed 2,147,483,647 which is the maximum value int type supports. May be you can use a string.matches("\\d+?");
Once again, amazing work 🙂
Regards
Varun
Hi Varun
Thanks for your kind comments and also the feedback regarding the bug.
Would you be able to provide some more details so that I can try to reproduce it? Can you provide screenshots of the module configuration as well as the error you are getting?
Rgds
Eng Swee
Hi Eng Swee Yeoh,
To reproduce the error, you just need to do an FCC for a fixed field length file and you need to ensure that the recordset.fixedFieldLengths parameter has a value which exceeds 2147483647 after you remove the commas. I think if you test with a file having 5 columns with each column having a fixed length of 25, you should get the error.
The exact location of error is at the class com.equalize.xpi.af.modules.util.RecordTypeParameters on line 54. I removed the try/catch block and used the below snippet. It worked like a charm 🙂
if(!lengthsWithoutComma.matches("\\d+?"))
{
throw new ModuleException("Maintain only integers separated by commas for '" + fieldFixedLengthsName);
}
Hi Varun
Ah, thanks for elaborating further. I've fixed the bug accordingly and published a new release for the EAR file as well as the source code.
Thanks again for reporting the bug. It is much appreciated! 🙂
Rgds
Eng Swee
Hi Eng Swee Yeoh,
Tried DeepPlain2XML and XML2DeepPlain in the File and JMS adapters and they worked great.
Thanks for the detailed blog it was very helpful.
Hi Anil
Really pleased to hear that they are working well. 🙂
Rgds
Eng Swee
Hi Eng Swee,
Thank you for developing and sharing this module - very helpful, and your efforts are very much appreciated. Recently we used it for XML to JSON conversion in JMS based integration scenario - worked perfectly well.
One question to you. Manifest contained in a project, specifies implementation vendor ID = "sap.com". As a result, corresponding component gets deployed with vendor = "sap.com", which is normally reserved / used by SAP standard components. Would you suggest downloading source code provided by you, adopting manifest and building deployable, or may you adjust vendor identifier to something different from "sap.com" and rebuild EAR so that it already contains custom vendor identifier?
Regards,
Vadim
Hi Vadim
Thank you for your feedback. I'm really glad to hear that the conversion worked well.
Regarding the Manifest file, to be honest, this is something that I'm not that familiar with. I checked both the EAR and EJB projects - there is no manifest file maintained in the EAR project and the one in the EJB project does not specify the vendor ID (very little info in that file).
It looks like the MANIFEST.MF file within the EAR deployable file is automatically generated when I perform Export > SAP EAR file for the EAR project. I guess I can of course manually edit the entry in the generated EAR file, but do you know if there is some other automatic way to achieve this - maybe manually maintaining a manifest file in the EAR project?
Have you adjusted the vendor identifier before for some custom adapter module that you have developed? Let me know if you do, I'm all up for modifying it accordingly for the EAR files I provide.
Rgds
Eng Swee
Hi Eng Swee,
I was able to achieve this in the past by adding the following tag into the application-j2ee-engine.xml for the EAR file:
<provider-name>idexx.com</provider-name>
Regards,
Ryan Crosby
Hi Eng Swee,
I assume, if manifest files (MANIFEST.MF and SAP_MANIFEST.MF) are not present in project structure, then NWDS auto-generates them when building EJB and EAR projects, and uses standard attribute values (this is where vendor = "sap.com" comes from). Since in built EAR file, after I decompile it, I can see manifest files:
I resolved this issue by explicit creation of manifest files within a project. I didn't create them in NWDS (but it is worth trying this - probably, it will be much simpler), but used build tools and specified what kind of manifest file I would like to assemble when building EJB project. To be more precise, I used Gradle (http://gradle.org/) in order to automate bundling the deployable. In Gradle script, I made use of Java plugin and its property manifest, where I provided attributes and their custom values which I want to put into manifest file. Usage of Gradle purely for this task may be a bit overkilling - but I used Gradle also for dependency management (especially when adapter module / custom job was using external libraries and I wanted to insert some of them, but not all, into the same deployable) and entire compilation and assembly of JARs and EARs, so manifest creation was just one of tasks of executed Gradle script.
Regards,
Vadim
Hi Vadim, Ryan
Thank you so much for both your feedback. I've definitely learnt something new 🙂
I've managed to adjust the value generated in the MANIFEST.MF file by adding the provider name into the application-j2ee-engine.xml file as suggested by Ryan. That worked really well and once it's deployed into the SAP system, the vendor value is reflected correctly in the different views in NWA.
I've updated the source code and also released a new EAR file in the GitHub repository for this change.
Thanks also for the info on Gradle. Will definitely have to look into it at some point! 😉
Rgds
Eng Swee
Thank you
The information provided by you became great help to me.
But i was occur problem in my source of NWDS
APIAccessFactory.java and MessageLogger.java Is not find in MessageLoggerHelper.java
Where is find JAVA Source? and .jar download url?
Regards,
Bang
Hi Bang
Glad to know this was helpful for you.
The pre-packaged Adapter library that comes with NWDS 7.31 are reduced versions and do not contain all classes for some packages. As such, you will need to get the full JAR file for com.sap.aii.af.ms.ifc_api.jar from your PI server.
Refer screenshot below that shows the package and classes that are in that JAR file. This will enable MessageLoggerHelper.java to compile successfully.
Refer to the following Wiki on where to get that file in your PI system.
Where to get the libraries for XI development - Process Integration - SCN Wiki
After you have retrieved that file, create a User Library in NWDS (give it any name) that includes that JAR file. Then add the newly created User Library to the EJB project's build path.
Rgds
Eng Swee
Dear Eng
Thank you, kind explanation for me
Is not found MessageLogger.class in com.sap.aii.af.ms.ifc.api.jar in PI 7.3
So, i'm going to after annotation in MessageLogger.class
Regards,
Bang
Hi Bang,
can i deploy the FormatConversionBean on our PI 7.3 SP10 ?
Hi Eng,
Very nice work with providing us this wholesome solution.
I am planning to use this solution for my below requirement. I have JSON file which i will need to read and update that data into Database tables.
So can you please let me know once i read this file using File adapter and custom JSONtoXML converter module, how can i map it to the database tables?
Also i will let you know once i deploy your module in our server, we are on SAP PO 7.4 SP9 how does it work out.
Thanks,
Hetal
Any idea how can i deploy the EAR file on SAP PO 7.4? I do not need to make any changes in the code you provided or any enhancements, hence if you can let me know how to deploy it directly on SAP PO.
Do i still need to have NWDS to deploy the file on the server?
Thanks,
Hetal
Hi Eng Swee,
I have been fan of this bean and I have been using this quiet a number of times.
Recently, I ran into trouble when I am using xml2json converter in this bean, when I am passing 800 + rows of XML, total of 1700kb , it takes around 9 minutes to perform the conversion.
Do you think it is normal ?
Regards,
Vk
Hi Eng Swee,
I am facing issues in downloading the EAR file from the link provided. For some reason the download just stops. Not sure if there is an issue with github. Is there another location where this can be downloaded?
https://github.com/engswee/equalize-xpi-modules/releases/latest
Best Regards,
Paolo
Nevermind. Link seems to be working now. Thank you!
Paolo
Thank you Eng Swee - it works perfectly. We are on SAP PO 7.5
Best regards, Peter
Thanks for the feedback, Peter. Real glad to know that is it working fine, even on the newer version 😉
Hi Peter,
How did you deploy the .ear file?
Did you deploy the .ear file directly using SUM?
Regards,
Rajesh
Hello Rajesh,
No for deploying - adapter modules or EAR files - I always use the NWDS.
There is a "deploy view" in NWDS you can use also for external files such as EAR files. It works well and is really practical.
BEst regards, Peter
Hi Eng Swee,
We got the below error while installing the com.equalize.xpi.af.modules.app.ear file directly using SUM.
Are we missing anything?
Error:
Component equalize.com/com.equalize.xpi.af.modules.app cannot be installed. Component com.equalize.xpi.af.modules.app is not part of any existing software components.
Thanks in Advance.
Regards,
Rajesh.A
This EAR package is not packed as a DC that is part of a Software Component (typical of NWDI-based development).
I have never used SUM to deploy it so I suggest you use other methods - Telnet, NWDS, etc.
Thanks Eng..I was able to deploy it using telnet.
I am getting null error in the channel.
Channel Configuration:
File loaded :
Regards,
Rajesh.A
I'd suggest you try it on a receiver channel first and see if there are any errors logged on it which will provide more details.
Hi Eng Swee,
Excel2XMLTransformer is working with xls format but I am getting the null error with 2010 xlsx format.
Regards,
Rajesh.A
Hello Eng Swee,
Thank you for this wonderful solution. As in the steps I downloaded the EAR file from GitHub (com.equalize.xpi.af.modules.app.ear) and deployed it directly in my PO system using Deployment perspective of NWDS . I have PO 7.5 system.
I intend to use DynamicAttributeChangeBean. But when I give this in the receiver channel module, the log shows error -
"Unable to obtain EJB home interface [Root exception is java.lang.NoSuchMethodException: com.sap.engine.services.jndi.implclient.ClientContext.create()]"
The Deployment seems to be correct -
Is there any post step that needs to be done or if I have missed anything.
Thanks in Advance.
Regards
Vikash
Try and check in the JNDI Browser and/or Application Modules in NWA to see if the EJB Modules are listed there. I don't have access to a PI system at the moment so can't provide more details.
Also, double check your module configuration (sequence, name, type, etc).
Tank's Eng Swee Yeoh , this module adapter solved my problem in my current project!
Fantastic! Great to hear that 😉
I have a Bug to report, when i am converting XML to JSON using the module, £ symbol is getting converted to �. Please help. Thanks in advance.
HI Eng,
I am working on Proxy to REST sync interface. I have few input fields which has value '/' in it ex: ( AOUS/12 and some date fields(23/04/2020).
at the JSON output payload it is coming as AOUS\/12 and dates are coming as 23\/04\/2020.
there is an addition of '\' in between. I tried to declare the field as "string" in custom xml/JSON coversion rules. but still its not working. i also tried to declare the field as "Date" in the datatype but still no luck because in the mapping its correctly populated.
the issue is coming at the REST channel while converting XML to JSON format.
Can you please help me how to get rid of this extra '\' at the channel level.
thanks
Smith
thanks
Smith
Hi Eng,
First of all thank you for this useful blog, it is very helpful.
I have been dealing with a project which i need to send a zipped file to rest api. My problem is the content of the file must be in json.
I am using rest receiver and "AF_Modules/MessageTransformBean" and "AF_Modules/PayloadZipBean" modules, you can see in screenshots.
can you help me with some advice on this please?
Hello Eng
Thank you for creating this amazing module. I have been using it successfully to convert relatively small xml files to excel.
In one scenario I am also using the payloadZipBean module to unzip the file before it passes to the FormatConversionBean. All of this works great for small files. However, I have a case with a large xml file(200mb) which causes the Receiver File adapter to hang with the following error:
It appears the conversion is taking too long for the large file to convert and the message gets stuck in Delivering then blacklisted. If I remove the FormatConversionBean I am able to process the file successfully as an xml. Do we have a known fix for this use case?
Thanks for reading and I look forward to your reply.
Thanks
Steve
Hi Steve,
200MB is definitely a big file, and the module has never been tested for such use case. Maybe the memory handling is not optimised for it.
I am no longer actively working with PI, so I would suggest that you clone the Git repository and try to test it locally, and make any fixes as required.
All the best!
Eng Swee