JSONTransformBean Part 2: Converting XML to JSON content

March 18, 2015 | 5,542 Views |

Eng Swee Yeoh

SAP Cloud Platform Integration for process services

SAP Process Integration adapter module custom module ejb ejb adapter module json jsontransformbean module

Update 4 Sep 2018: Now available in CPI as well.

Update 31 Mar 2016: Enhancement to support forcing of JSON output in arrays irrespective of number of values in payload with new parameters forceArrayAll or arrayFieldList

Update 25 Mar 2015: JSONTransformBeanBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass

Introduction

This is the second part of the two-part series on conversion of JSON content with a custom adapter module. The first part covered the conversion of JSON to XML at a sender channel.

This second part will focus on conversion of XML to JSON at a receiver channel.

Source Code

Refer to following blog on location of source code and/or EAR deployment file.

FormatConversionBean – One Bean to rule them all!

Usage of Module in Communication Channel

Module Processing Sequence

Number	Module Name	Type	Module Key
<Depending on position of module in chain> 1) Asynchronous scenario, Normally before the last module in channels 2) Synchronous scenario, Before last module to convert request payload After last module to convert response payload	Custom_AF_Modules/FormatConversionBean	Local Enterprise Bean	<Any Arbitrary Value>

Number

Module Name

Type

Module Key

1) Asynchronous scenario,

Normally before the last module in channels

2) Synchronous scenario,

Before last module to convert request payload

After last module to convert response payload

Custom_AF_Modules/FormatConversionBean

Local Enterprise Bean

Module Parameter Reference

Below are the parameters for configuration of the module for XML to JSON conversion (converterClass = ‘com.equalize.xpi.af.modules.json.XML2JSONConverter’). Certain parameters will automatically inherit the default values if it is not configured.

Parameter Name	Allowed values	Default value	Remarks
converterClass	PI – com.equalize.xpi.af.modules.json.XML2JSONConverter CPI – com.equalize.converter.core.XML2JSONConverter		Required field. Determines conversion class
indentFactor	Integer values beginning from 1	0	Determines the number of indentation spaces for each level in the JSON output. Note: When indentation is not used, the JSON content produced is compact and ideal for lightweight data transfer
skipRootNode	Y,N	N	Skips root XML element in JSON output
forceArrayAll	Y,N	N	Forces all nodes and fields to be in JSON Array
arrayFieldList			Comma separated values of field names to be forced as JSON Array. Example: field1,field2,field3
messageLog	pre, post		Saves a log version of the message that is viewable in Message Monitor pre = saves payload before conversion post = saves payload after conversion Available only in PI
logLocation			Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated. Available only in PI

Example

Module parameters

Parameter Name	Parameter Value
converterClass	com.equalize.xpi.af.modules.json.XML2JSONConverter
skipRootNode	Y
indentFactor	2

Result

Input

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?>

<ns:MT_XML2JSON xmlns:ns=”urn:equalize:com”>

<title>example glossary</title>

<para>A meta-markup language, used to create markup languages such as DocBook.</para>

</GlossDef>

<GlossTerm>Standard Generalized Markup Language</GlossTerm>

<GlossSee>markup</GlossSee>

</GlossEntry>

</GlossList>

</GlossDiv>

</glossary>

</ns:MT_XML2JSON>

Output

/wp-content/uploads/2015/03/out1_666636.png

Screenshots of actual configuration and testing are also shown below.

Module configuration on an SFTP receiver channel.

The XML input payload.

The ‘jsonContent’ log version shows the JSON content after conversion. The JSON content produced is compact since parameter indentFactor was not used.

The audit log shows the trace of steps being executed by the module.

Conclusion

With the completion of this second part for JSONTransformBean, we now have a generic and reusable custom adapter module that is able to handle conversion of JSON to XML and vice versa! Bring on the RESTful web services!

To report this post you need to login first.

31 Comments

You must be Logged on to comment or reply to a post.

Juan Vasquez April 28, 2015 at 6:44 pm

Hello

It is a nice module, but i have a problem with custom headers

Can you please help me

i need some custom headers for the call

Content-Type: application/json

Cache-Control: no-cache

Application-Secret:5554b65aaaacb2ed74cc7d9c5921aaaa2314aacd

what is the right receiver adapter soap?, http (axis)?

But I don’t know here to add

Thanis a lot

like (0)
Juan Vasquez April 28, 2015 at 6:49 pm

It is a clear document

So good writing

it works fine

like (0)
Vipin Kanchan May 15, 2015 at 11:35 am

Hi Eng Swee,

Great blog… Just wanted to check whether the converter class mentioned on this page ‘XML2JSONConverter‘ can help in forcing some of the json objects to be array values even though they are supplied with single values in some payloads?

Is it possible to force the square brackets via the module?

like (0)
1. Eng Swee Yeoh Post authorMay 18, 2015 at 2:32 am
  
  Hi Vipin
  
  That feature is not available in the current XML2JSONConverter. There is no straightforward method to determine which XML single values should be JSON arrays or not. It is something that requires further thought in the design in order to incorporate it properly into such a generic converter – unfortunately, I don’t have the liberty of spending time to enhance it at the moment.
  
  Since the source code for my converters are available as open source, you can consider downloading them and enhancing the module specific to your requirement. You can refer to the following blog on how to do download and recompile it.
  
  Recompile com.equalize.xpi.af.modules as EJB 2.1 modules in NWDS 7.1x
  
  Rgds
  
  Eng Swee
  
  like (0)
  1. Vipin Kanchan May 18, 2015 at 5:03 am
    
    Hi Eng Swee,
    
    Thanks for the response. We will try with the enhancement.
    
    like (0)
    1. Eng Swee Yeoh Post authorMay 18, 2015 at 5:23 am
      
      Hi Vipin
      
      You are welcome. It should be quite straightforward to download and setup the module project in your NWDS environment. All the best in your efforts 😉
      
      Rgds
      
      Eng Swee
      
      like (0)
Former Member May 19, 2015 at 7:54 pm

HI Eng Swee,

Thanks a lot for the excellent Blog.

I will try to use the XML2JSON Converter from this.

Can you please help me in the deployment of the same.

Where Exactly I can see the .ear file ? Currently, I am not able to find it.

Once I have it, i will try to deploy the same directly in my NWDS..

Regards,

Vk

like (0)
1. Hareesh Gampa May 19, 2015 at 8:07 pm
  
  Source Code
  
  Refer to following blog on location of source code and/or EAR deployment file.
  
  FormatConversionBean – One Bean to rule them all!
  
  Probably you missed the beginning of the blog..
  
  Location :
  
  Release New converter Base64EncodeConverter · engswee/com.equalize.xpi.af.modules · GitHub
  
  like (0)
  1. Peter Chezowitch August 10, 2015 at 9:50 am
    
    Hi Hareesh, Hi Eng Swee,
    
    How can we Import the source Code on a 7.31 system? I saw a manuel for doing it with EJB 2.1 on PI 7.1, but if we have a 7.31 System we can use the EJB3 as Eng Swee used for the original development.
    
    I am just not sure how the procedure is. Once I downloaded and extracted the source Code, do we need to create an EJB3 Project in NWDS manually and then add the source Code or is somewhere a complete NWDS Project available for Import?
    
    like (0)
    1. Former Member August 10, 2015 at 2:42 pm
      
      HI Peter,
      
      There is .ear available and all you need to do is just to deploy the same on your PI server using NWDS or some other means.
      
      It is available in the very beginning of the blog.
      
      Regards,
      
      Vk
      
      like (0)
    2. Eng Swee Yeoh Post authorAugust 11, 2015 at 12:22 am
      
      Hi Peter
      
      The ready to deploy EAR file is compiled for 7.31/7.4 so you can use it directly if you do not need further customisation. As mentioned by VK, refer to the Source Code section, it points to the following blog which provides you the details for deployment. If you need to change the logic, there is also further instruction on how you can build your own project.
      
      FormatConversionBean – One Bean to rule them all!
      
      Rgds
      
      Eng Swee
      
      like (0)
Former Member October 12, 2015 at 12:44 pm

Hello Eng Swee,

thank you for providing this valuable Custom module!

I would like to integrate it in my synchronous Scenario:

3rd party program calls a webservice on PI via SOAP Sender channel.

This request calls a RFC in SAP ECC which gives a response.

This response should be converted from XML to JSON.

I tried to put the module in SENDER SOAP channel, but it doesn’t work.

Only the Incoming Request XML was mapped to JSON and This fails of course while RFC call…

I think these modules are only processed, while message gets IN.

How can I set up the conversion to JSON for the response message before the webservice invoking program gets the response ?

Do you have an Idea?

Thank you and best regards,

Fabian

like (0)
1. Eng Swee Yeoh Post authorOctober 13, 2015 at 2:42 am
  
  Hi Fabian
  
  I’ve not used this on the sender channel for synchronous response before.
  
  First things first, which PI version are you on because I know earlier versions like 7.11 do not support adding modules on the SOAP sender channel.
  
  Secondly, when using modules to deal with synchronous response, the modules need to be placed after any existing standard modules in the channel. I normally place this after the standard module at the receiver channel. You can try this or even at the sender channel.
  
  Rgds
  
  Eng Swee
  
  like (0)
Former Member January 4, 2016 at 10:34 am

Hi Eng Swee

I am doing a REST — PO — ECC synchronous scenario. REST is using GET method to get some data from the ECC. In each set of data, PI is sending a url also in the payload.

When the xml is converted to JSON, the url is escaped.

http://host:port/RESTAdapter/xyz

becomes

http:\/\/host:port\/RESTAdapter\/xyz

Sender synchronous REST adapter does this. Is there any way I can pass forward slash without escaping

Regards

Midhun

like (0)
Former Member April 7, 2016 at 8:13 pm

Hi Eng Swee,

great work and thank you so much for the wonderful converter modules.

We are trying to use the XML to JSON conversion module in our REST adapter. The module is interpreting everything as a string and while its converting it is not considering the data types from the xml input. so the JSON is actually failing at the receiver end, because we are sending string in the integer/Boolean/float variables. And there is no type conversion at their end.

Any suggestions from your end?

Thanks,

Vijaya Palla

like (0)
1. Eng Swee Yeoh Post authorApril 8, 2016 at 2:02 am
  
  Hi Vijaya Palla or Sri Barghava
  
  None of the XML to JSON converters (SAP’s, Advantco’s or mine) will check the field type from the XML definition. At the moment, yes my converter will output all fields as string in the JSON output.
  
  If you already have the REST adapter, I’d suggest you use SAP’s standard feature to perform enhance XML/JSON conversion that allows you to specify certain fields to be of a specific type. Refer to the following blog for more details.
  
  REST Adapter in PI/PO: Enhanced XML/JSON Conversion
  
  Regards
  
  Eng Swee
  
  like (0)
  1. Former Member April 8, 2016 at 1:42 pm
    
    Hi Eng Swee,
    
    Thanks for the reply.
    
    We actually went with REST adapter since, our rest service is synchronous and they are returning us JSON. so to convert that to xml, we used the SAP’s standard feature of mapping conversion rules.
    
    I am not sure if REST itself can convert the input xml to JSON before hitting the rest service. we tried that out but it didn’t help us. so we had to go with the custom module given by you. But facing the schema validation issues.
    
    So was wondering, if there is any thing that can help us generating the JSON payload using XSD schema. any help?
    
    Thanks,
    
    Vijaya Palla
    
    like (0)
    1. Eng Swee Yeoh Post authorApril 11, 2016 at 6:47 am
      
      Hi Vijaya Palla
      
      Firstly, as mentioned above, I’ve provided the link that shows how further conversion enhancement can be configured in the standard SAP adapter. SAP’s REST adapter provides XML to JSON functionality and I would strongly recommend that you try that out fully and raise a discussion thread in the forum in you have issues with that. The intention of my module is to cover the gaps where SAP’s solution do not provide such functionality, however in your particular case there is such functionality direct from SAP.
      
      Secondly, there is no publicly released API from SAP that provides functionality to access the XSD schema in ESR from an adapter module. As such, at the moment it’s not possible to include the feature to have a JSON payload generated based on XSD schema field types.
      
      Regards
      
      Eng Swee
      
      like (0)
      1. Former Member February 7, 2017 at 3:39 pm
        
        Hi Eng Swee Yeoh,
        
        We need send values in HTTP header but my Rest adapter don’t have the tab HTTP Header. Do you know how could do this with a custom module?
        
        Thanks and regards
        
        like (0)
        
        Eng Swee Yeoh Post authorFebruary 10, 2017 at 9:06 am
        
        Hi Monica
        
        Adapter modules don’t have access to the HTTP header. You might want to update your SP level so that the version of the REST adapter has HTTP header functionality.
        
        like (0)
Former Member March 7, 2017 at 10:41 am
Hi Eng Swee,

Very nice Blog.

I am having the scenario to download the .tar.zip file using the URL( ECC — PI — REST) In the request we are sending the URL for file download .This ZIP file contains the .JSON file. In the request we are sending the url to download the zip file but,we are getting the erro.

We thought that develop a Module For REST receiver Channel. Hear we are facing challenges for the deveopment.

Below are my Queries.
1. what is the input for module development as of now REST receiver channel downloading the ZIP file but while unzipping we are facing the problem.
2. How we need to map the map the download zip file to the java code which is developed in module.
3. My java team is asking how are what he need consider the input for his ejb development and how this resposne willl mapp is there any standard class or paramerte need to use ? what is it.
4. how we need to do response caputre in the module and what is standard calss?
5. we are very new to the module development .
Please help us to achive the requirment .

Regards

Ravi.S

7893366122
like (0)
Former Member March 9, 2017 at 8:11 pm

Hi Eng,

I encountered issue with Chinese characters like after transformation from xml to Json it shows ‘?’ any idea about it ? could it be due to encoding in program?

Appreciate your inputs.

Thanks

Anuj

like (0)
Former Member March 13, 2017 at 9:23 pm

Hi All,

I have a scenario (REST–>PI–>REST) wherein PI will receive the data from sender in JSON format. Data format accepted by receiver is also a JSON format.

The input payload to PI looks like:

[{
“ClassCode”: “”,
“FSAC”: “”,
“TrainId”: “”,
“UserId”: “”,
“Equipment”: [{
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}, {
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}]

}]

When this input is passed to receiver Rest Adapter, I see that PI append root to it which is not acceptable by REST Service.

JSON at receiver adapter looks like below:

{
“root”: [{
“ClassCode”: “”,
“FSAC”: “”,
“TrainId”: “”,
“UserId”: “”,
“Equipment”: [{
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}, {
“EquipmentInitials”: “”,
“EquipmentNumber”: “”,
“EquipmentDisplayType”: ” “,
“SourceId”: “”,
“SourceType”: “”
}]

}]
}

How to ignore this root element and add outer Array[], similar to the input payload.

Thanks in advance!!

Best Regards,

Sachin Jangir

like (0)

Anurag Gupta June 22, 2017 at 4:50 pm

Hello Eng Swee Yeoh

I am providing it an xml where

<carrierId>8</carrierId>
<locationId>200</locationId>
<maxDays>8</maxDays>
<minDays>8</minDays>
<supplierId>200</supplierId>

but it is converting them to string as below

“carrierId”: “8”,
“locationId”: “200”,
“maxDays”: “8”,
“minDays”: “8”,
“supplierId”: “200”

and this is what i do not want. I want them without quotes, that is integers.

like (0)

Eng Swee Yeoh Post authorJune 23, 2017 at 1:54 am

Hi Anurag

This feature is not available currently. Feel free to download the source code and modify it accordingly to achieve your requirement.

Regards

Eng Swee

like (0)

Technical ABEAM October 8, 2018 at 8:36 am

Hi Eng Swee Yeoh

Many thanks!!. I’ve modified your module as you suggests and then it’s work.

	private void constructJSONContentfromXML(JSONObject parent, XMLElementContainer element) {
		LinkedHashMap<String, JSONArray> map = new LinkedHashMap<String, JSONArray>();
		// Process all the child fields of the XML element
		for (Field childField : element.getChildFields()) {
			// Check if it is an array first
			int count = element.getChildFieldList().get(childField.fieldName);

			Object fieldContent = childField.fieldContent;
			if (fieldContent instanceof XMLElementContainer) {
				// If it is a segment, create a JSON object for it first before adding into parent
				JSONObject childjo = new JSONObject();
				constructJSONContentfromXML(childjo, (XMLElementContainer) fieldContent);
				putObjectIntoJSONObject(count, map, parent, childField.fieldName, childjo);
			} else if (fieldContent instanceof String) {
				// If it is a string, directly add it to parent
//				System.out.println("Field: " + childField.fieldName + "/ Content: " + fieldContent); // binla
				
				if (intFields.contains(childField.fieldName)) {
					Integer value = Integer.parseInt((String) fieldContent);
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, value);
				} else if (doubleFields.contains(childField.fieldName)) {
					Double value = Double.parseDouble((String) fieldContent);
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, value);
				}  else {
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, fieldContent);
				}
				
//				putObjectIntoJSONObject(count, map, parent, childField.fieldName, fieldContent); // binla
			}
			
			// If a JSONArray is created for this field and hasn't been added to the parent,
			// then add the JSONArray to the parent
			if (map.containsKey(childField.fieldName) && !parent.has(childField.fieldName)) {
				parent.put(childField.fieldName, map.get(childField.fieldName));
			}
		}
	}

like (0)

Christof Johner August 15, 2018 at 9:15 am

Hi Eng Swee

We have a problem with special characters like umlauts like e.g. “ü” when using XML2JSONConverter.

E.g the German word “Bürste” is converted to “B?rste” in the JMS-adapter.

How does this happen?

In our scenario (ICO with IDOC-sender adapter and JMS-receiver adapter) we are actually converting the idoc-xml to json format in the JMS adapter by using the converterClass XML2JSONConverter:

JMS-Adapter:

The VO-version of the message is still correct (xml-fomat):

However the payload of the Json message version shows not the correct umlaut, which is “?” instead of “ü”:

As a matter of fact the payload is the same in the RabbitMQ-Queue-Monitor:

What have we done to narrow down the problem? Instead of a JMS-receiver adapter we’ve used a file receiver adapter.

Receiver File-Adapter with file type set to binary:

Also with Converter class:

The payload of the json-version of the message shows again “?” instead of “ü”:

Also the VO-version is correct:

But the produced file shows the correct umlaut:

When we use file type “text” in the file adapter the same output as with JMS is produced:

This leads us to the following conclusion:

File-receiver-adapter:

Obviously the file adapter works only with file type = “binary” and not with “text” in our example.

JMS receiver adapter:

In the JMS adapter the class ConvertMessageToBinary is called which should also convert to binary. But then the umlauts are still not properly translated.

Do you have an idea how we can solve our umlaut problem in the JMS receiver adapter in connection with the use of FormatConversionBean?

Thanks for your feedback and kind regards

Christof

like (0)
1. Eng Swee Yeoh Post authorAugust 23, 2018 at 5:22 am
  
  Hi Christof
  
  Sorry for my delayed response as I have been away.
  
  Can you open an issue in my GitHub respository (link below) and we can discuss further how we can work collaboratively on this?
  
  https://github.com/engswee/equalize-xpi-modules/issues
  
  Regards
  
  Eng Swee
  
  like (0)
Christof Johner September 5, 2018 at 8:50 am

Hi Eng Swee

thanks for your reply and the offer for your GitHub.

In the mean time I could solve the issue.
I’ve managed in the mean time to solve the issue by defining
the following parameters in the JMS receiver adapter:
JMS.Message.type=Text
JMS.TextMessage.charset=ISO-8859-1

This settings helped to get rid of the umlaut problem while
using the FormatConversionBean.

Kind regards
Christof

like (1)
1. Eng Swee Yeoh Post authorSeptember 5, 2018 at 1:50 pm
  
  Hi Christof
  
  Thanks for your response. Really glad to hear that you managed to resolve the issue. Hopefully this means it’s not related to the coding in the adapter module 😉
  
  Thanks also for sharing the solution – this will help others who may face the same issue in the future.
  
  Regards
  
  Eng Swee
  
  like (0)
2. Eng Swee Yeoh Post authorSeptember 21, 2018 at 2:29 am
  
  Hi Christof Johner
  
  Vadim Klimov and myself have been looking into this issue further and have found the root cause of the character corruption in the adapter module’s logic. This issue only occurs when the PI system’s default charset/encoding is not UTF-8 (typically Windows-based systems uses CP1252 encoding).
  
  The JSON output was generated as a UTF-8 encoded string, but during conversion to bytes, it used the system’s default encoding, therefore causing the corruption. I have now fixed it by specifying the encoding as UTF-8 during string to byte conversion.
  
  You can get the latest release with this fix in the following link.
  
  https://github.com/engswee/equalize-xpi-modules/releases/tag/1.7.1
  
  With this, you would not need to specify the additional encoding parameters in the JMS channel.
  
  If you do try this out, do let me know how it goes.
  
  Regards
  
  Eng Swee
  
  like (0)