Skip to Content
Eng Swee Yeoh
Eng Swee Yeoh
March 18, 2015 3 minute read

JSONTransformBean Part 2: Converting XML to JSON content

Follow RSS feed
6 Likes 11,375 Views 48 Comments

Update 2 Nov 2018: Default parsing in CPI will be using SAX-based XmlSlurper

Update 4 Sep 2018: Now available in CPI as well.

Update 31 Mar 2016: Enhancement to support forcing of JSON output in arrays irrespective of number of values in payload with new parameters forceArrayAll or arrayFieldList

Update 25 Mar 2015: JSONTransformBeanBean has been refactored to be part of FormatConversionBean. Parameter conversionType replaced with converterClass. Note that some of the screenshots are not updated to reflect the new parameter converterClass

Introduction

This is the second part of the two-part series on conversion of JSON content with a custom adapter module. The first part covered the conversion of JSON to XML at a sender channel.

This second part will focus on conversion of XML to JSON at a receiver channel.

Source Code

Refer to following blog on location of source code and/or EAR deployment file.

FormatConversionBean – One Bean to rule them all!

Usage of Module in Communication Channel

Module Processing Sequence

Number Module Name Type Module Key

<Depending on position of module in chain>

1) Asynchronous scenario,

Normally before the last module in channels

2) Synchronous scenario,

Before last module to convert request payload

After last module to convert response payload

 Custom_AF_Modules/FormatConversionBean Local Enterprise Bean <Any Arbitrary Value>

Module Parameter Reference

Below are the parameters for configuration of the module for XML to JSON conversion (converterClass = ‘com.equalize.xpi.af.modules.json.XML2JSONConverter’). Certain parameters will automatically inherit the default values if it is not configured.

Parameter Name Allowed values Default value Remarks
converterClass

PI – com.equalize.xpi.af.modules.json.XML2JSONConverter

CPI – com.equalize.converter.core.XML2JSONConverter

 Required field. Determines conversion class
indentFactor Integer values beginning from 1 0

Determines the number of indentation spaces for each level in the JSON output.

Note: When indentation is not used, the JSON content produced is compact and ideal for lightweight data transfer
skipRootNode Y,N N Skips root XML element in JSON output
forceArrayAll Y,N N Forces all nodes and fields to be in JSON Array
arrayFieldList

Comma separated values of field names to be forced as JSON Array.

Example: field1,field2,field3
useDOM Y, N N

Available only in CPI

Parser used in parsing input XML

  • N = SAX-based XmlSlurper
  • Y = Document Object Model (DOM) based parser
messageLog pre, post Saves a log version of the message that is viewable in Message Monitor

  • pre = saves payload before conversion
  • post = saves payload after conversion

Available only in PI
logLocation

Name of log version when messageLog is populated. Location defaulted to value in messageLog if logLocation not populated.

Available only in PI

Example

Module parameters

Parameter Name Parameter Value
converterClass com.equalize.xpi.af.modules.json.XML2JSONConverter
skipRootNode Y
indentFactor 2

Result

Input

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?>

<ns:MT_XML2JSON xmlns:ns=”urn:equalize:com”>

<glossary>

  <title>example glossary</title>

  <GlossDiv>

   <GlossList>

    <GlossEntry>

     <SortAs>SGML</SortAs>

     <GlossDef>

      <GlossSeeAlso>GML</GlossSeeAlso>

      <GlossSeeAlso>XML</GlossSeeAlso>

      <para>A meta-markup language, used to create markup languages such as DocBook.</para>

     </GlossDef>

     <GlossTerm>Standard Generalized Markup Language</GlossTerm>

     <GlossSee>markup</GlossSee>

     <ID>SGML</ID>

     <Acronym>SGML</Acronym>

     <Abbrev>ISO 8879:1986</Abbrev>

    </GlossEntry>

   </GlossList>

   <title>S</title>

  </GlossDiv>

</glossary>

</ns:MT_XML2JSON>
Output /wp-content/uploads/2015/03/out1_666636.png

Screenshots of actual configuration and testing are also shown below.

Module configuration on an SFTP receiver channel.

/wp-content/uploads/2015/03/param2_666686.png

The XML input payload.

/wp-content/uploads/2015/03/in1_666690.png

The ‘jsonContent’ log version shows the JSON content after conversion. The JSON content produced is compact since parameter indentFactor was not used.

/wp-content/uploads/2015/03/json_666689.png

The audit log shows the trace of steps being executed by the module.

/wp-content/uploads/2015/03/log_666692.png

Conclusion

With the completion of this second part for JSONTransformBean, we now have a generic and reusable custom adapter module that is able to handle conversion of JSON to XML and vice versa! Bring on the RESTful web services!

Assigned tags

Related Blog Posts

    Related Questions

      48 Comments
      You must be Logged on to comment or reply to a post.
      • Juan Vasquez

        Hello

        It is a nice module, but i have a problem with custom headers

        Can you please help me

        2015-04-28 13_43_34-Message Editor - SAP NetWeaver Administrator.png

        i need some custom headers for the call

        Content-Type: application/json

        Cache-Control: no-cache

        Application-Secret:5554b65aaaacb2ed74cc7d9c5921aaaa2314aacd

        what is the right receiver adapter soap?, http (axis)?

        But I don’t know here to add

        Thanis a lot

      • Vipin Kanchan

        Hi Eng Swee,

        Great blog… Just wanted to check whether the converter class mentioned on this page ‘XML2JSONConverter‘ can help in forcing some of the json objects to be array values even though they are supplied with single values in some payloads?

        Is it possible to force the square brackets via the module?

        • Eng Swee Yeoh Post author

          Hi Vipin

          That feature is not available in the current XML2JSONConverter. There is no straightforward method to determine which XML single values should be JSON arrays or not. It is something that requires further thought in the design in order to incorporate it properly into such a generic converter – unfortunately, I don’t have the liberty of spending time to enhance it at the moment.

          Since the source code for my converters are available as open source, you can consider downloading them and enhancing the module specific to your requirement. You can refer to the following blog on how to do download and recompile it.

          Recompile com.equalize.xpi.af.modules as EJB 2.1 modules in NWDS 7.1x

          Rgds

          Eng Swee

      • Former Member

        HI Eng Swee,

        Thanks a lot for the excellent Blog.

        I will try to use the XML2JSON Converter from this.

        Can you please help me in the deployment of the same.

        Where Exactly I can see the .ear file ? Currently, I am not able to find it.

        Once I have it, i will try to deploy the same directly in my NWDS..

        Regards,

        Vk

      • Former Member

        Hello Eng Swee,

        thank you for providing this valuable Custom module!

        I would like to integrate it in my synchronous Scenario:

        3rd party program calls a webservice on PI via SOAP Sender channel.

        This request calls a RFC in SAP ECC which gives a response.

        This response should be converted from XML to JSON.

        I tried to put the module in SENDER SOAP channel, but it doesn’t work.

        Only the Incoming Request XML was mapped to JSON and This fails of course while RFC call…

        I think these modules are only processed, while message gets IN.

        How can I set up the conversion to JSON for the response message before the webservice invoking program gets the response ?

        Do you have an Idea?

        Thank you and best regards,

        Fabian

        • Eng Swee Yeoh Post author

          Hi Fabian

          I’ve not used this on the sender channel for synchronous response before.

          First things first, which PI version are you on because I know earlier versions like 7.11 do not support adding modules on the SOAP sender channel.

          Secondly, when using modules to deal with synchronous response, the modules need to be placed after any existing standard modules in the channel. I normally place this after the standard module at the receiver channel. You can try this or even at the sender channel.

          Rgds

          Eng Swee

      • Former Member

        Hi Eng Swee

        I am doing a REST — PO — ECC synchronous scenario. REST is using GET method to get some data from the ECC. In each set of data, PI is sending a url also in the payload.

        When the xml is converted to JSON, the url is escaped.

        http://host:port/RESTAdapter/xyz

        becomes

        http:\/\/host:port\/RESTAdapter\/xyz


        Sender synchronous REST adapter does this. Is there any way I can pass forward slash without escaping


        Regards


        Midhun

      • Former Member

        Hi Eng Swee,

        great work and thank you so much for the wonderful converter modules.

        We are trying to use the XML to JSON conversion module in our REST adapter. The module is interpreting everything as a string and while its converting it is not considering the data types from the xml input. so the JSON is actually failing at the receiver end, because we are sending string in the integer/Boolean/float variables. And there is no type conversion at their end.

        Any suggestions from your end?

        Thanks,

        Vijaya Palla

        • Eng Swee Yeoh Post author

          Hi Vijaya Palla or Sri Barghava

          None of the XML to JSON converters (SAP’s, Advantco’s or mine) will check the field type from the XML definition. At the moment, yes my converter will output all fields as string in the JSON output.

          If you already have the REST adapter, I’d suggest you use SAP’s standard feature to perform enhance XML/JSON conversion that allows you to specify certain fields to be of a specific type. Refer to the following blog for more details.

          REST Adapter in PI/PO: Enhanced XML/JSON Conversion

          Regards

          Eng Swee

          • Former Member

            Hi Eng Swee,

            Thanks for the reply.

            We actually went with REST adapter since, our rest service is synchronous and they are returning us JSON. so to convert that to xml, we used the SAP’s standard feature of mapping conversion rules.

            I am not sure if REST itself can convert the input xml to JSON before hitting the rest service. we tried that out but it didn’t help us. so we had to go with the custom module given by you. But facing the schema validation issues.

            So was wondering, if there is any thing that can help us generating the JSON payload using XSD schema. any help?

            Thanks,

            Vijaya Palla

            • Eng Swee Yeoh Post author

              Hi Vijaya Palla

              Firstly, as mentioned above, I’ve provided the link that shows how further conversion enhancement can be configured in the standard SAP adapter. SAP’s REST adapter provides XML to JSON functionality and I would strongly recommend that you try that out fully and raise a discussion thread in the forum in you have issues with that. The intention of my module is to cover the gaps where SAP’s solution do not provide such functionality, however in your particular case there is such functionality direct from SAP.

              Secondly, there is no publicly released API from SAP that provides functionality to access the XSD schema in ESR from an adapter module. As such, at the moment it’s not possible to include the feature to have a JSON payload generated based on XSD schema field types.

              Regards

              Eng Swee

      • Former Member

        Hi Eng Swee,

        Very nice Blog.

         

        I am having the scenario to download the .tar.zip file using the URL( ECC — PI — REST)  In the request we are sending the URL for file download .This ZIP file contains the .JSON  file. In the request we are sending the url to download the zip file but,we are getting the erro.

        We thought that develop a Module For REST receiver Channel. Hear we are facing challenges for the deveopment.

        Below are my Queries.

        1. what is the input for module development as of now REST receiver channel downloading the ZIP file but while unzipping we are facing the problem.
        2. How we need to map the map the download zip file to the java code which is developed in module.
        3. My java team is asking  how are what he need consider the input for his ejb development and how this resposne willl mapp is there  any standard class or paramerte need to use  ? what is it.
        4. how we need to do response caputre in the module and what is standard calss?
        5. we are very new to the module development .

        Please help us to achive the requirment .

         

        Regards

        Ravi.S

        7893366122

         

         

      • Former Member

        Hi Eng,

         

        I encountered issue with Chinese characters like after transformation from xml to Json it shows ‘?’ any idea about it ? could it be due to encoding in program?

        Appreciate your inputs.

        Thanks

        Anuj

      • Former Member

         

        Hi All,

        I have a scenario (REST–>PI–>REST) wherein PI will receive the data from sender in JSON format. Data format accepted by receiver is also a JSON format.

        The input payload to PI looks like:

        [{
        “ClassCode”: “”,
        “FSAC”: “”,
        “TrainId”: “”,
        “UserId”: “”,
        “Equipment”: [{
        “EquipmentInitials”: “”,
        “EquipmentNumber”: “”,
        “EquipmentDisplayType”: ” “,
        “SourceId”: “”,
        “SourceType”: “”
        }, {
        “EquipmentInitials”: “”,
        “EquipmentNumber”: “”,
        “EquipmentDisplayType”: ” “,
        “SourceId”: “”,
        “SourceType”: “”
        }]

        }]

        When this input is passed to receiver Rest Adapter, I see that PI append root to it which is not acceptable by REST Service.

        JSON at receiver adapter looks like below:

        {
        “root”: [{
        “ClassCode”: “”,
        “FSAC”: “”,
        “TrainId”: “”,
        “UserId”: “”,
        “Equipment”: [{
        “EquipmentInitials”: “”,
        “EquipmentNumber”: “”,
        “EquipmentDisplayType”: ” “,
        “SourceId”: “”,
        “SourceType”: “”
        }, {
        “EquipmentInitials”: “”,
        “EquipmentNumber”: “”,
        “EquipmentDisplayType”: ” “,
        “SourceId”: “”,
        “SourceType”: “”
        }]

        }]
        }

        How to ignore this root element and add outer Array[], similar to the input payload.

        Thanks in advance!!

        Best Regards,

        Sachin Jangir

      • Anurag Gupta

        Hello Eng Swee Yeoh

        I am providing it an xml where

        <carrierId>8</carrierId>
<locationId>200</locationId>
<maxDays>8</maxDays>
<minDays>8</minDays>
<supplierId>200</supplierId>

        but it is converting them to string as below 

        “carrierId”: “8”,
“locationId”: “200”,
“maxDays”: “8”,
“minDays”: “8”,
“supplierId”: “200”

        and this is what i do not want. I want them without quotes, that is integers.

        • Eng Swee Yeoh Post author

          Hi Anurag

           

          This feature is not available currently. Feel free to download the source code and modify it accordingly to achieve your requirement.

           

          Regards

          Eng Swee

          • Technical ABEAM

            Hi Eng Swee Yeoh

            Many thanks!!.  I’ve modified your module as you suggests and then it’s work.

            	private void constructJSONContentfromXML(JSONObject parent, XMLElementContainer element) {
		LinkedHashMap<String, JSONArray> map = new LinkedHashMap<String, JSONArray>();
		// Process all the child fields of the XML element
		for (Field childField : element.getChildFields()) {
			// Check if it is an array first
			int count = element.getChildFieldList().get(childField.fieldName);

			Object fieldContent = childField.fieldContent;
			if (fieldContent instanceof XMLElementContainer) {
				// If it is a segment, create a JSON object for it first before adding into parent
				JSONObject childjo = new JSONObject();
				constructJSONContentfromXML(childjo, (XMLElementContainer) fieldContent);
				putObjectIntoJSONObject(count, map, parent, childField.fieldName, childjo);
			} else if (fieldContent instanceof String) {
				// If it is a string, directly add it to parent
//				System.out.println("Field: " + childField.fieldName + "/ Content: " + fieldContent); // binla
				
				if (intFields.contains(childField.fieldName)) {
					Integer value = Integer.parseInt((String) fieldContent);
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, value);
				} else if (doubleFields.contains(childField.fieldName)) {
					Double value = Double.parseDouble((String) fieldContent);
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, value);
				}  else {
					putObjectIntoJSONObject(count, map, parent, childField.fieldName, fieldContent);
				}
				
//				putObjectIntoJSONObject(count, map, parent, childField.fieldName, fieldContent); // binla
			}
			
			// If a JSONArray is created for this field and hasn't been added to the parent,
			// then add the JSONArray to the parent
			if (map.containsKey(childField.fieldName) && !parent.has(childField.fieldName)) {
				parent.put(childField.fieldName, map.get(childField.fieldName));
			}
		}
	}
      • Christof Johner

        Hi Eng Swee

        We have a problem with special characters like umlauts like e.g. “ü” when using XML2JSONConverter.

        E.g the German word “Bürste” is converted to “B?rste” in the JMS-adapter.

        How does this happen?

        In our scenario (ICO with IDOC-sender adapter  and JMS-receiver adapter) we are actually converting the idoc-xml to json format in the JMS adapter by using the converterClass XML2JSONConverter:

        JMS-Adapter:

        The VO-version of the message is still correct (xml-fomat):

        However the payload of the Json message version shows not the correct umlaut, which is “?” instead of “ü”:

        As a matter of fact the payload is the same in the RabbitMQ-Queue-Monitor:

        What have we done to narrow down the problem? Instead of a JMS-receiver adapter we’ve used a file receiver adapter.

         

        Receiver File-Adapter with file type set to binary:

        Also with Converter class:

        The payload of the json-version of the message shows again “?” instead of “ü”:

        Also the VO-version is correct:

        But the produced file shows the correct umlaut:

        When we use file type “text” in the file adapter the same output as with JMS is produced:

        This leads us to the following conclusion:

        File-receiver-adapter:

        Obviously the file adapter works only with file type = “binary” and not with “text” in our example.

        JMS receiver adapter:

        In the JMS adapter the class ConvertMessageToBinary is called which should also convert to binary. But then the umlauts are still not properly translated.

        Do you have an idea how we can solve our umlaut problem in the JMS receiver adapter in connection with the use of FormatConversionBean?

        Thanks for your feedback and kind regards

        Christof

      • Christof Johner

        Hi Eng Swee

        thanks for your reply and the offer for your GitHub.

        In the mean time I could solve the issue.
        I’ve managed in the mean time to solve the issue by defining
        the following parameters in the JMS receiver adapter:
        JMS.Message.type=Text
        JMS.TextMessage.charset=ISO-8859-1

        This settings helped to get rid of the umlaut problem while
        using the FormatConversionBean.

        Kind regards
        Christof

        • Eng Swee Yeoh Post author

          Hi Christof

           

          Thanks for your response. Really glad to hear that you managed to resolve the issue. Hopefully this means it’s not related to the coding in the adapter module 😉

           

          Thanks also for sharing the solution – this will help others who may face the same issue in the future.

           

          Regards

          Eng Swee

           

        • Eng Swee Yeoh Post author

          Hi Christof Johner

           

          Vadim Klimov and myself have been looking into this issue further and have found the root cause of the character corruption in the adapter module’s logic. This issue only occurs when the PI system’s default charset/encoding is not UTF-8 (typically Windows-based systems uses CP1252 encoding).

           

          The JSON output was generated as a UTF-8 encoded string, but during conversion to bytes, it used the system’s default encoding, therefore causing the corruption. I have now fixed it by specifying the encoding as UTF-8 during string to byte conversion.

           

          You can get the latest release with this fix in the following link.

          https://github.com/engswee/equalize-xpi-modules/releases/tag/1.7.1

           

          With this, you would not need to specify the additional encoding parameters in the JMS channel.

           

          If you do try this out, do let me know how it goes.

           

          Regards

          Eng Swee

      • Ushasri FPS21X Reddi

         

        Hi Eng Swee,

        I’m currently using the Format Conversion bean to convert xml to JSON arrays.

        I added module parameter Log Location as json content but I cannot see the log versions jsonCOntent pre and post.

         

        Can you please advise.

         

         

         

         

      • Ushasri FPS21X Reddi

         

        Hi Eng Swee,

         

        I placed the custom module before the REST Adapter bean.

        Please see the channel monitor below. It says logging message in jsoncontent but we do not know exacty where the log is written.

      • Christof Johner

        Hello Eng Swee

        thanks for your feedback regarding the encoding issue
        from the 21.9.18. We’ve tested the 1.7.1 library.
        Actually when we still used the 2 parameter:
        -JMS.Message.type=Text
        -JMS.TextMessage.charset=ISO-8859-1
        the umlaut problem happened again.

        Is this behaviour wantend, e.g when one still
        uses the 2 parameters that the conversion
        with the umlauts still fails?
        Thanks for your feedback.

        Kind regards
        Christof

        • Eng Swee Yeoh Post author

          Hi Christof

           

          If you use release 1.7.1, then the output will always be generated in UTF-8, so you do not need to additionally set the charset using those two parameters.

          Matter of fact, you shouldn’t as that would force the adapter to assume that the charset is ISO-8859-1 when in fact it is now UTF-8. So having those parameters is now the cause of the corruption.

           

          Regards

          Eng Swee

      • Sushant Shinde

        Hi Eng Swee,

         

        I am using the standard xml to JSON conversion provided in the REST adapter, but during the conversion its adding special character back slash(“\”) to my payload.

        Can you please help me how can I handle this

        Regards,

        Sushant

         

      • Christof Johner

        Hi Eng Swee

        We think we have problem with the character * (star) when doing a conversion from xml to json using XML2JSONConverter.
        It looks like that * is being lost during the xml-json-conversion. However this only applies to
        the special case, when * is in a XML tag. As soon as there are more characters, like “* XX” the conversion is correct:

        Do you you know this problem or how to solve our problem (be file or JMS receiver adapter) in connection with the use of FormatConversionBean?

        Thanks for your feedback and kind regards
        Christof

      • Sushant Shinde

        Hi Eng Swee,

        I have used this bean to covert XML to JSON, the reason for using this bean was that I have special characters in my payload, which were causing a problem while using the standard REST Sender adapter conversion. But while using this in REST receiver adapter the response messages is not able to create the target JSON structure. I have used the bean after the standard module, please find attached snap for the same

        Regards,

        Sushant

         

          • Sushant Shinde

            Hello Eng Swee,

             

            I want the conversion for response message for synchronous call, if  you still  need me to use it before the standard module, I will try that.

            I tried using the BEAN before the standard module but still no luck.

            Can you please give me some more idea on how this can be used for response message in synchronous scenarios. 

            Regards,

            Sushant

             

      • Iddo Rijsdijk

        Hi, is it possible to send a JSON field as a number value, so without ” around it? Currently, every field is sent out as “field”: “1”. I have a requirement to send out a field as “field”: 1

        • Eng Swee Yeoh Post author

          Hi Iddo

           

          If you are asking for the CPI version, as a matter of fact, I happen to have code lying around (for months) in my laptop doing just that. Your comment pushed me to make it public, and I’ve just released version 1.4.0 with that capability.

           

          I’ve yet to find time to update the blog/documentation, but basically, a new optional parameter fieldConversions is introduced, which allows you to configure the field output as number, boolean or null.

           

          To use it, populate the parameter with key-value pairs stating the field name & field type (separated by colon). If more than one fields need to be converted, separate the pairs with comma. Example below

          fieldConversions = houseNumber:number,isResident:boolean

           

          This version is very raw with no validations on whether the parameter is populated correctly, or if the content of the field is suitable for the type (i.e. converting a field with a non-number value to a number format).

           

          Give it a try and let me know how it goes. If there are any bugs, please raise an issue on the GitHub page.

           

          Regards

          Eng Swee

          • Iddo Rijsdijk

            Hi Eng Swee,

            That sounds awesome. This would mean that we no longer have to wait for SAP to finally fix the bug mentioned in note 2521044, which has been reported over 6 months ago.

            I will test your new release in CPI this Monday and let you know the outcome!

            Iddo

      • Chetan Risbud

        Hello Eng Swee Yeoh

        Im using FormatConversionBean to convert message into encoded format and then i want to convert the same message in JSON format

        Its file to rest scenario, no mapping is involved. I wanted to encode file in base64 and then send to API in JSON format.

        I tried below method encoding is working fine but XML2JSON is not working can you please help me out to get correct output.

        Channel Configuration

        MessageLog

        Expected output

        Regards,

        Chetan Risbud