Technical Articles
JSONTransformBean Part 2: Converting XML to JSON content
Update 23 Jun 2020: Add new parameter arrayGPathList to force JSON output in arrays using GPath notation
Update 5 Nov 2019: Add new parameter trim to trim XML text nodes
Update 1 May 2019: Add new parameter fieldConversions to generate output other than String
Update 2 Nov 2018: Default parsing in CPI will be using SAX-based XmlSlurper
Update 4 Sep 2018: Now available in CPI as well.
Update 31 Mar 2016: Enhancement to support forcing of JSON output in arrays irrespective of number of values in payload with new parameters forceArrayAll or arrayFieldList
Update 25 Mar 2015: JSONTransformBeanBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass
Introduction
This is the second part of the two-part series on conversion of JSON content with a custom adapter module. The first part covered the conversion of JSON to XML at a sender channel.
This second part will focus on conversion of XML to JSON at a receiver channel.
Source Code
Refer to following blog on location of source code and/or EAR deployment file.
FormatConversionBean – One Bean to rule them all!
Usage of Module in Communication Channel
Module Processing Sequence
Number | Module Name | Type | Module Key |
---|---|---|---|
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload |
Custom_AF_Modules/FormatConversionBean | Local Enterprise Bean | <Any Arbitrary Value> |
Module Parameter Reference
Below are the parameters for configuration of the module for XML to JSON conversion (converterClass = ‘com.equalize.xpi.af.modules.json.XML2JSONConverter’). Certain parameters will automatically inherit the default values if it is not configured.
Parameter Name | Allowed values | Default value | Remarks |
---|---|---|---|
converterClass |
PI – com.equalize.xpi.af.modules.json.XML2JSONConverter CPI – com.equalize.converter.core.XML2JSONConverter |
Required field. Determines conversion class | |
indentFactor | Integer values beginning from 1 | 0 |
Determines the number of indentation spaces for each level in the JSON output. Note: When indentation is not used, the JSON content produced is compact and ideal for lightweight data transfer |
skipRootNode | Y,N | N | Skips root XML element in JSON output |
forceArrayAll | Y,N | N | Forces all nodes and fields to be in JSON Array |
arrayFieldList |
Comma separated values of field names to be forced as JSON Array. Example: field1,field2,field3 |
||
arrayGPathList |
Available only in CPI Comma separated values of field names to be forced as JSON Array. This uses GPath notation based on the target JSON structure Example: field1,parent.field2,parent.child.field3 |
||
useDOM | Y, N | N |
Available only in CPI Parser used in parsing input XML
|
trim | Y, N | N |
Available only in CPI Trim whitespace in XML text nodes during parsing |
fieldConversions |
Available only in CPI Comma separated values of field names to be generated as a different type. Each entry is a key-value pair separated by colon (:) Supported types are: number, null, boolean Example: field1:number,field2:null,field3:boolean |
||
messageLog | pre, post | Saves a log version of the message that is viewable in Message Monitor
Available only in PI |
|
logLocation |
Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated. Available only in PI |
Example
Module parameters
Parameter Name | Parameter Value |
---|---|
converterClass | com.equalize.xpi.af.modules.json.XML2JSONConverter |
skipRootNode | Y |
indentFactor | 2 |
Result
Screenshots of actual configuration and testing are also shown below.
Module configuration on an SFTP receiver channel.
The XML input payload.
The ‘jsonContent’ log version shows the JSON content after conversion. The JSON content produced is compact since parameter indentFactor was not used.
The audit log shows the trace of steps being executed by the module.
Conclusion
With the completion of this second part for JSONTransformBean, we now have a generic and reusable custom adapter module that is able to handle conversion of JSON to XML and vice versa! Bring on the RESTful web services!
Hello
It is a nice module, but i have a problem with custom headers
Can you please help me
i need some custom headers for the call
Content-Type: application/json
Cache-Control: no-cache
Application-Secret:5554b65aaaacb2ed74cc7d9c5921aaaa2314aacd
what is the right receiver adapter soap?, http (axis)?
But I don't know here to add
Thanis a lot
It is a clear document
So good writing
it works fine
Hi Eng Swee,
Great blog... Just wanted to check whether the converter class mentioned on this page 'XML2JSONConverter' can help in forcing some of the json objects to be array values even though they are supplied with single values in some payloads?
Is it possible to force the square brackets via the module?
Hi Vipin
That feature is not available in the current XML2JSONConverter. There is no straightforward method to determine which XML single values should be JSON arrays or not. It is something that requires further thought in the design in order to incorporate it properly into such a generic converter - unfortunately, I don't have the liberty of spending time to enhance it at the moment.
Since the source code for my converters are available as open source, you can consider downloading them and enhancing the module specific to your requirement. You can refer to the following blog on how to do download and recompile it.
Recompile com.equalize.xpi.af.modules as EJB 2.1 modules in NWDS 7.1x
Rgds
Eng Swee
Hi Eng Swee,
Thanks for the response. We will try with the enhancement.
Hi Vipin
You are welcome. It should be quite straightforward to download and setup the module project in your NWDS environment. All the best in your efforts 😉
Rgds
Eng Swee
HI Eng Swee,
Thanks a lot for the excellent Blog.
I will try to use the XML2JSON Converter from this.
Can you please help me in the deployment of the same.
Where Exactly I can see the .ear file ? Currently, I am not able to find it.
Once I have it, i will try to deploy the same directly in my NWDS..
Regards,
Vk
Source Code
Refer to following blog on location of source code and/or EAR deployment file.
FormatConversionBean - One Bean to rule them all!
Probably you missed the beginning of the blog..
Location :
Release New converter Base64EncodeConverter · engswee/com.equalize.xpi.af.modules · GitHub
Hi Hareesh, Hi Eng Swee,
How can we Import the source Code on a 7.31 system? I saw a manuel for doing it with EJB 2.1 on PI 7.1, but if we have a 7.31 System we can use the EJB3 as Eng Swee used for the original development.
I am just not sure how the procedure is. Once I downloaded and extracted the source Code, do we need to create an EJB3 Project in NWDS manually and then add the source Code or is somewhere a complete NWDS Project available for Import?
HI Peter,
There is .ear available and all you need to do is just to deploy the same on your PI server using NWDS or some other means.
It is available in the very beginning of the blog.
Regards,
Vk
Hi Peter
The ready to deploy EAR file is compiled for 7.31/7.4 so you can use it directly if you do not need further customisation. As mentioned by VK, refer to the Source Code section, it points to the following blog which provides you the details for deployment. If you need to change the logic, there is also further instruction on how you can build your own project.
FormatConversionBean - One Bean to rule them all!
Rgds
Eng Swee
Hello Eng Swee,
thank you for providing this valuable Custom module!
I would like to integrate it in my synchronous Scenario:
3rd party program calls a webservice on PI via SOAP Sender channel.
This request calls a RFC in SAP ECC which gives a response.
This response should be converted from XML to JSON.
I tried to put the module in SENDER SOAP channel, but it doesn't work.
Only the Incoming Request XML was mapped to JSON and This fails of course while RFC call...
I think these modules are only processed, while message gets IN.
How can I set up the conversion to JSON for the response message before the webservice invoking program gets the response ?
Do you have an Idea?
Thank you and best regards,
Fabian
Hi Fabian
I've not used this on the sender channel for synchronous response before.
First things first, which PI version are you on because I know earlier versions like 7.11 do not support adding modules on the SOAP sender channel.
Secondly, when using modules to deal with synchronous response, the modules need to be placed after any existing standard modules in the channel. I normally place this after the standard module at the receiver channel. You can try this or even at the sender channel.
Rgds
Eng Swee
Hi Eng Swee
I am doing a REST --- PO --- ECC synchronous scenario. REST is using GET method to get some data from the ECC. In each set of data, PI is sending a url also in the payload.
When the xml is converted to JSON, the url is escaped.
http://host:port/RESTAdapter/xyz
becomes
http:\/\/host:port\/RESTAdapter\/xyz
Sender synchronous REST adapter does this. Is there any way I can pass forward slash without escaping
Regards
Midhun
Hi Eng Swee,
great work and thank you so much for the wonderful converter modules.
We are trying to use the XML to JSON conversion module in our REST adapter. The module is interpreting everything as a string and while its converting it is not considering the data types from the xml input. so the JSON is actually failing at the receiver end, because we are sending string in the integer/Boolean/float variables. And there is no type conversion at their end.
Any suggestions from your end?
Thanks,
Vijaya Palla
Hi Vijaya Palla or Sri Barghava
None of the XML to JSON converters (SAP's, Advantco's or mine) will check the field type from the XML definition. At the moment, yes my converter will output all fields as string in the JSON output.
If you already have the REST adapter, I'd suggest you use SAP's standard feature to perform enhance XML/JSON conversion that allows you to specify certain fields to be of a specific type. Refer to the following blog for more details.
REST Adapter in PI/PO: Enhanced XML/JSON Conversion
Regards
Eng Swee
Hi Eng Swee,
Thanks for the reply.
We actually went with REST adapter since, our rest service is synchronous and they are returning us JSON. so to convert that to xml, we used the SAP's standard feature of mapping conversion rules.
I am not sure if REST itself can convert the input xml to JSON before hitting the rest service. we tried that out but it didn't help us. so we had to go with the custom module given by you. But facing the schema validation issues.
So was wondering, if there is any thing that can help us generating the JSON payload using XSD schema. any help?
Thanks,
Vijaya Palla
Hi Vijaya Palla
Firstly, as mentioned above, I've provided the link that shows how further conversion enhancement can be configured in the standard SAP adapter. SAP's REST adapter provides XML to JSON functionality and I would strongly recommend that you try that out fully and raise a discussion thread in the forum in you have issues with that. The intention of my module is to cover the gaps where SAP's solution do not provide such functionality, however in your particular case there is such functionality direct from SAP.
Secondly, there is no publicly released API from SAP that provides functionality to access the XSD schema in ESR from an adapter module. As such, at the moment it's not possible to include the feature to have a JSON payload generated based on XSD schema field types.
Regards
Eng Swee
Hi Eng Swee Yeoh,
We need send values in HTTP header but my Rest adapter don't have the tab HTTP Header. Do you know how could do this with a custom module?
Thanks and regards
Hi Monica
Adapter modules don't have access to the HTTP header. You might want to update your SP level so that the version of the REST adapter has HTTP header functionality.
Hi Eng Swee,
Very nice Blog.
I am having the scenario to download the .tar.zip file using the URL( ECC --- PI --- REST) In the request we are sending the URL for file download .This ZIP file contains the .JSON file. In the request we are sending the url to download the zip file but,we are getting the erro.
We thought that develop a Module For REST receiver Channel. Hear we are facing challenges for the deveopment.
Below are my Queries.
Please help us to achive the requirment .
Regards
Ravi.S
7893366122
Hi Eng,
I encountered issue with Chinese characters like after transformation from xml to Json it shows '?' any idea about it ? could it be due to encoding in program?
Appreciate your inputs.
Thanks
Anuj
Hi All,
I have a scenario (REST–>PI–>REST) wherein PI will receive the data from sender in JSON format. Data format accepted by receiver is also a JSON format.
The input payload to PI looks like:
[{
“ClassCode”: “”,
“FSAC”: “”,
“TrainId”: “”,
“UserId”: “”,
“Equipment”: [{
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}, {
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}]
}]
When this input is passed to receiver Rest Adapter, I see that PI append root to it which is not acceptable by REST Service.
JSON at receiver adapter looks like below:
{
“root”: [{
“ClassCode”: “”,
“FSAC”: “”,
“TrainId”: “”,
“UserId”: “”,
“Equipment”: [{
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}, {
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}]
}]
}
How to ignore this root element and add outer Array[], similar to the input payload.
Thanks in advance!!
Best Regards,
Sachin Jangir
Hello Eng Swee Yeoh
I am providing it an xml where
but it is converting them to string as below
and this is what i do not want. I want them without quotes, that is integers.
Hi Anurag
This feature is not available currently. Feel free to download the source code and modify it accordingly to achieve your requirement.
Regards
Eng Swee
Hi Eng Swee Yeoh
Many thanks!!. I've modified your module as you suggests and then it's work.
Hi Eng Swee
We have a problem with special characters like umlauts like e.g. “ü” when using XML2JSONConverter.
E.g the German word “Bürste” is converted to “B?rste” in the JMS-adapter.
How does this happen?
In our scenario (ICO with IDOC-sender adapter and JMS-receiver adapter) we are actually converting the idoc-xml to json format in the JMS adapter by using the converterClass XML2JSONConverter:
JMS-Adapter:
The VO-version of the message is still correct (xml-fomat):
However the payload of the Json message version shows not the correct umlaut, which is “?” instead of “ü”:
As a matter of fact the payload is the same in the RabbitMQ-Queue-Monitor:
What have we done to narrow down the problem? Instead of a JMS-receiver adapter we’ve used a file receiver adapter.
Receiver File-Adapter with file type set to binary:
Also with Converter class:
The payload of the json-version of the message shows again “?” instead of “ü”:
Also the VO-version is correct:
But the produced file shows the correct umlaut:
When we use file type “text” in the file adapter the same output as with JMS is produced:
This leads us to the following conclusion:
File-receiver-adapter:
Obviously the file adapter works only with file type = “binary” and not with “text” in our example.
JMS receiver adapter:
In the JMS adapter the class ConvertMessageToBinary is called which should also convert to binary. But then the umlauts are still not properly translated.
Do you have an idea how we can solve our umlaut problem in the JMS receiver adapter in connection with the use of FormatConversionBean?
Thanks for your feedback and kind regards
Christof
Hi Christof
Sorry for my delayed response as I have been away.
Can you open an issue in my GitHub respository (link below) and we can discuss further how we can work collaboratively on this?
https://github.com/engswee/equalize-xpi-modules/issues
Regards
Eng Swee
Hi Eng Swee
thanks for your reply and the offer for your GitHub.
In the mean time I could solve the issue.
I've managed in the mean time to solve the issue by defining
the following parameters in the JMS receiver adapter:
JMS.Message.type=Text
JMS.TextMessage.charset=ISO-8859-1
This settings helped to get rid of the umlaut problem while
using the FormatConversionBean.
Kind regards
Christof
Hi Christof
Thanks for your response. Really glad to hear that you managed to resolve the issue. Hopefully this means it's not related to the coding in the adapter module 😉
Thanks also for sharing the solution - this will help others who may face the same issue in the future.
Regards
Eng Swee
Hi Christof Johner
Vadim Klimov and myself have been looking into this issue further and have found the root cause of the character corruption in the adapter module's logic. This issue only occurs when the PI system's default charset/encoding is not UTF-8 (typically Windows-based systems uses CP1252 encoding).
The JSON output was generated as a UTF-8 encoded string, but during conversion to bytes, it used the system's default encoding, therefore causing the corruption. I have now fixed it by specifying the encoding as UTF-8 during string to byte conversion.
You can get the latest release with this fix in the following link.
https://github.com/engswee/equalize-xpi-modules/releases/tag/1.7.1
With this, you would not need to specify the additional encoding parameters in the JMS channel.
If you do try this out, do let me know how it goes.
Regards
Eng Swee
Hi Eng Swee,
I'm currently using the Format Conversion bean to convert xml to JSON arrays.
I added module parameter Log Location as json content but I cannot see the log versions jsonCOntent pre and post.
Can you please advise.
Please ensure that the processing sequence is correct and check what is happening on the audit/message log.
Hi Eng Swee,
I placed the custom module before the REST Adapter bean.
Please see the channel monitor below. It says logging message in jsoncontent but we do not know exacty where the log is written.
Hello Eng Swee
thanks for your feedback regarding the encoding issue
from the 21.9.18. We've tested the 1.7.1 library.
Actually when we still used the 2 parameter:
-JMS.Message.type=Text
-JMS.TextMessage.charset=ISO-8859-1
the umlaut problem happened again.
Is this behaviour wantend, e.g when one still
uses the 2 parameters that the conversion
with the umlauts still fails?
Thanks for your feedback.
Kind regards
Christof
Hi Christof
If you use release 1.7.1, then the output will always be generated in UTF-8, so you do not need to additionally set the charset using those two parameters.
Matter of fact, you shouldn't as that would force the adapter to assume that the charset is ISO-8859-1 when in fact it is now UTF-8. So having those parameters is now the cause of the corruption.
Regards
Eng Swee
Hi Eng Swee,
I am using the standard xml to JSON conversion provided in the REST adapter, but during the conversion its adding special character back slash("\") to my payload.
Can you please help me how can I handle this
Regards,
Sushant
Hi Eng Swee
We think we have problem with the character * (star) when doing a conversion from xml to json using XML2JSONConverter.
It looks like that * is being lost during the xml-json-conversion. However this only applies to
the special case, when * is in a XML tag. As soon as there are more characters, like "* XX" the conversion is correct:
Do you you know this problem or how to solve our problem (be file or JMS receiver adapter) in connection with the use of FormatConversionBean?
Thanks for your feedback and kind regards
Christof
Hi Eng Swee,
I have used this bean to covert XML to JSON, the reason for using this bean was that I have special characters in my payload, which were causing a problem while using the standard REST Sender adapter conversion. But while using this in REST receiver adapter the response messages is not able to create the target JSON structure. I have used the bean after the standard module, please find attached snap for the same
Regards,
Sushant
You need to use it before the standard module.
Hello Eng Swee,
I want the conversion for response message for synchronous call, if you still need me to use it before the standard module, I will try that.
I tried using the BEAN before the standard module but still no luck.
Can you please give me some more idea on how this can be used for response message in synchronous scenarios.
Regards,
Sushant
Hi, is it possible to send a JSON field as a number value, so without " around it? Currently, every field is sent out as "field": "1". I have a requirement to send out a field as "field": 1
Hi Iddo
If you are asking for the CPI version, as a matter of fact, I happen to have code lying around (for months) in my laptop doing just that. Your comment pushed me to make it public, and I've just released version 1.4.0 with that capability.
I've yet to find time to update the blog/documentation, but basically, a new optional parameter fieldConversions is introduced, which allows you to configure the field output as number, boolean or null.
To use it, populate the parameter with key-value pairs stating the field name & field type (separated by colon). If more than one fields need to be converted, separate the pairs with comma. Example below
fieldConversions = houseNumber:number,isResident:boolean
This version is very raw with no validations on whether the parameter is populated correctly, or if the content of the field is suitable for the type (i.e. converting a field with a non-number value to a number format).
Give it a try and let me know how it goes. If there are any bugs, please raise an issue on the GitHub page.
Regards
Eng Swee
Hi Eng Swee,
That sounds awesome. This would mean that we no longer have to wait for SAP to finally fix the bug mentioned in note 2521044, which has been reported over 6 months ago.
I will test your new release in CPI this Monday and let you know the outcome!
Iddo
Hello Eng Swee Yeoh
Im using FormatConversionBean to convert message into encoded format and then i want to convert the same message in JSON format
Its file to rest scenario, no mapping is involved. I wanted to encode file in base64 and then send to API in JSON format.
I tried below method encoding is working fine but XML2JSON is not working can you please help me out to get correct output.
Channel Configuration
MessageLog
Expected output
Regards,
Chetan Risbud
Your module sequence is somewhat messed up. You have an exit after calling the Base64 Encoder. Remove line number 2 and try again.
Thanks for quick response.
Hi Eng Swee,
I am using XML to JSON conversion for single line to array conversion.
Below are the config screen shots
When I check the message in PI, I am getting below error
I request to please help in this or advise what is wrong in configuration.
-Regards
Rohit