Conversion of XML to JSON using Generic Java Mapping

Update 10th July 2020 – Added functionality to convert numbers to String using the NUM_TO_STRING parameter.

Introduction:

I was recently trying to convert XML To JSON in the below format in SAP 7.4 and also trying to remove certain tags from the XML structure and force a few nodes as JSON Array.

[
{
  record : [
        { key1: value1 }
  ]
},
{
   record : [
        { key1: value1 },
        { key1: value2 }
  ]
} 
]

We can easily convert XML to JSON on the REST channel. However, there are many restrictions on it in SAP PI/PO.

Besides, in SAP PO 7.5 we can easily remove a key name by listing the tags in ‘setIgnoredElements’ on REST Receiver Channel.  However, this liberty isn’t available on SAP PI versions 7.4 & below. The Custom XML to JSON Conversion bean also doesn’t help much here, especially if you wish to remove a node from the JSON structure. Hence, I have implemented a Generic Java Mapping that can be used to achieve this and many other tasks.

This blog focus on XML to JSON conversion using Java Mapping and also provides the liberty to play around with the structure.

This can be used in any SAP PI/PO version.

Benefits of Using this Mapping :

• Converts XML to JSON
• Force convert JSONObject to JSONArray
• Hide tag names in JSON structure
• Delete key & value pair from a JSON structure
• Crop up initial tags in the JSON structure
• Convert the JSON root tag to Array
• Convert Numbers to String

Examples are illustrated further below after the implementation section.

Implementation:

Step1:- Get the JAVA Mapping JAR file

You can directly download the zip file from here and unzip it.

This zip contains 2 JAR files. Import the 2 jar files in SAP ESR as Imported Archives.

Please note the Java mapping uses org.json.XML java library to convert XML to JSON, so if it’s not in your SAP library, you will receive missing ‘org.json.XML’ error. So do import the orgXmlJson jar file also.

Step 2:- Use the java mapping in your operation mapping

Step 3:- Define the 5 Import parameters in operation mapping.

• ARRAY_NODES – Simple Type – xsd:string
• DELETE_ENTRY – Simple Type – xsd:string
• HIDE_KEYS – Simple Type – xsd:string
• LAST_LEVEL_TO_KEEP – Simple Type – xsd:int
• NUM_TO_STRING – Simple Type – xsd:string

Step 4:- Define Binding for the JAVA mapping in operation mapping as below:

Step 5:-  Assign values to these parameters in the Integrated Configuration as required

Step 6:- (Optional) If you are using REST channel, don’t forget to set the data as JSON in the Receiver channel

And you are done.

Understanding the parameters:

• ARRAY_NODES:
  • This is of String data type.
  • All tags including the root tag that are to be converted into an array can be passed here as a list separated by commas.
  • E.g. ARRAY_NODES = Root,node1,node4
  • If you don’t want to convert any node to array in JSON, then keep the field blank.

• HIDE_KEYS:
  • This is of String data type
  • If you wish to hide the tag name and pass only the equivalent values to JSON structure, then pass all the tag names as a list separated by commas to this field.
  • E.g. HIDE_KEYS = node2,root
  • If you don’t want to hide any keys in JSON, then keep the field blank.

• DELETE_ENTRY:
  • This is of String data type
  • If you wish to delete a tag and its value from the JSON structure, then pass all such tag names as a list separated by commas to this field.
  • E.g. DELETE_ENTRY = node5,node6
  • If you don’t want to delete any entries, then keep the field blank.

• LAST_LEVEL_TO_KEEP:
  • This is of integer data type
  • If you wish to delete the initial tags from XML hierarchy and retain the child nodes, then pass the level number to this field
  • Value 0 implies to not delete any level. (default value)
  • Value 1 implies removing the root node completely from the structure and keep its child node in the JSON structure.
  • E.g. LAST_LEVEL_TO_KEEP = 2 (This will remove first 2 levels from the XML/JSON structure)
  • If you don’t want to delete any entries, then set the field as 0.

•  NUM_TO_STRING:
  • This is of the boolean data type.
  • If you want to covert all the numbers to string in the output JSON structure, then set this variable as ‘true’.
  • Setting this to true will result in all the numbers (integer, float, long, short, double) to be in double-quotes in the output JSON format.
  • Default value is ‘false’.

How to set the parameter value?

Now let’s understand how you have to set the parameter values. Below are a few examples that will help you understand the same.

Sample XML :

<ns0:Root_MT xmlns:ns0="https://namespace">
	<node1>
		<node2>
		<node3>abc</node3>
		<node4>xyz</node4>
		<node5>123</node5>
		</node2>
		<node6/>
	</node1>
	<node1>
		<node2>
		<node3>abc</node3>
		<node4>xyz</node4>
		<node5>200.005</node5>
		</node2>
	</node1>
</ns0:Root_MT>

Example1: If the default values of the parameters are set.

ARRAY_NODES = “”

HIDE_KEYS = “”

DELETE_ENTRY = “”

LAST_LEVEL_TO_KEEP = 0

NUM_TO_STRING = false

{ 	"Root_MT":
	{
		"node1": [
		{  "node2": {
				"node3": "abc",
				"node4": "xyz",
				"node5": 123
			},
			"node6": ""
		},
		{  "node2": {
				"node3": "abc",
				"node4": "xyz",
				"node5": 200.005
			}
		}
		]
	},
	"xmlns:ns0"="https://namespace"
}

Example2: Hide the root tag and send data as JSON Array. Also, delete the namespace tag from the JSON structure and set node5 as JSON Array

ARRAY_NODES = “Root_MT,node5”

HIDE_KEYS = “Root_MT”

DELETE_ENTRY = “xmlns:ns0”

LAST_LEVEL_TO_KEEP = 0

NUM_TO_STRING = false

{ 	[
	{
		"node1": [
		{  "node2": {
				"node3": "abc",
				"node4": "xyz",
				"node5": 123
			},
			"node6": []
		},
		{  "node2": {
				"node3": "abc",
				"node4": "xyz",
				"node5": 200.005
			}
		}
		]
	}
	]
}

Example3: Remove the level 1 i.e the root tag level, always keep node1 as JSON Array (even if one record is found)  and hide node1 key and delete the namespace attribute. Also, convert the numbers to string format.

ARRAY_NODES = “node1”

HIDE_KEYS = “node1”

DELETE_ENTRY = “xmlns:ns0”

LAST_LEVEL_TO_KEEP = 1   (As level count starts from level 0 being root)

NUM_TO_STRING = true

 [
	{  "node2": {
			"node3": "abc",
			"node4": "xyz",
			"node5": "123"
		    },
	    "node6": ""
	},
	{  "node2": {
			 "node3": "abc",
			 "node4": "xyz",
			 "node5": "200.005"
		     }
	}
]

Source Code:

If you are importing the code from the link above, you need not bother about the source code. But just for your curiosity purpose and understanding provided the source code below. Do have a glance at it and provide suggestions and feedback.

You can also find the code here.

package com.convert;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.util.Arrays;
import java.util.List;
import java.util.ArrayList;

import org.json.XML;
import org.json.JSONArray;
import org.json.JSONObject;

import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;

public class XML2JSON extends AbstractTransformation {

	private List<String> array_nodes = new ArrayList<String>();
	private List<String> hide_keys = new ArrayList<String>();
	private List<String> delete_entry = new ArrayList<String>();
	private int last_level_to_keep = 0;
	private boolean num_to_string = false;

	public XML2JSON() {

	}

	public XML2JSON(List<String> array_nodes, List<String> hide_keys,
			List<String> delete_entry, int last_level_to_keep, 
			boolean num_to_string) {
		/*
		 * Declare constructor to define the class-level variables Input: 
		 * 1) array_nodes -> List to store Keys that should be JSONArray 
		 * 2) hide_keys -> List to store Keys that should be Hidden 
		 * 3) delete_entry -> List to store Keys, whose records are to be deleted from the structure 
		 * 4) last_level_to_keep -> integer to store maximum level of JSON Structure to be retained
		 * 5) num_to_string -> boolean variable to set if numbers are to be converted in string format
		 */

		this.array_nodes = array_nodes;
		this.hide_keys = hide_keys;
		this.delete_entry = delete_entry;
		this.last_level_to_keep = last_level_to_keep;
		this.num_to_string = num_to_string;
	}
	

	@Override
	public void transform(TransformationInput transformationInput,
			TransformationOutput transformationOutput)
			throws StreamTransformationException {

		/*
		 * This is the default method called by SAP PI Java mapping Input :
		 * InputStream and OutputStream objects
		 */

		// An info message is added to trace by calling the getTrace() method of
		// AbstractTransformation class
		getTrace().addInfo("JAVA Mapping to Convert XML to JSON Initiated");

		// Input Payload is obtained from transformationInput
		InputStream inputStream = transformationInput.getInputPayload()
				.getInputStream();

		// Input Parameters is obtained from transformationInput
		String array_nodes = transformationInput.getInputParameters()
				.getString("ARRAY_NODES");
		String hide_keys = transformationInput.getInputParameters().getString(
				"HIDE_KEYS");
		String delete_entry = transformationInput.getInputParameters()
				.getString("DELETE_ENTRY");
		int last_level_to_keep = transformationInput.getInputParameters()
				.getInt("LAST_LEVEL_TO_KEEP");
		boolean num_to_string = Boolean.parseBoolean(transformationInput.getInputParameters()
		.getString("NUM_TO_STRING"));

		// Display parameter value on trace
		getTrace().addInfo("Input Parameters listed below: \n");
		getTrace().addInfo("ARRAY_NODES : " + array_nodes);
		getTrace().addInfo("HIDE_KEYS: " + hide_keys);
		getTrace().addInfo("DELETE_ENTRY: " + delete_entry);
		getTrace().addInfo("LAST_LEVEL_TO_KEEP: " + last_level_to_keep);
		getTrace().addInfo("NUM_TO_STRING: " + num_to_string);
		
		// Output Payload Stream is obtained to send the data
		OutputStream outputStream = transformationOutput.getOutputPayload()
				.getOutputStream();
		
		// Convert Message Content type in Msg Header to application/json
		transformationOutput.getOutputHeader().setContentType("application/json");		

		// Instance of XML2JSON created and parameters passed to contructor
		XML2JSON obj = new XML2JSON(Arrays.asList(array_nodes.split(",")),
				Arrays.asList(hide_keys.split(",")), Arrays.asList(delete_entry
						.split(",")), last_level_to_keep, num_to_string);

		// Call readStreamContent() to Handle the input and output stream content
		try {
			obj.readStreamContent(inputStream, outputStream);
		} catch (UnsupportedEncodingException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	public void readStreamContent(InputStream inputStream,
			OutputStream outputStream) throws StreamTransformationException, UnsupportedEncodingException, IOException {

		/*
		 * This method Input: InputStream and OutputStream objects
		 */

		try {
			// Declare a String variable to store output json format
			String jsonPrettyPrintString = "";

			// If data is present in InputStream, it is stored in byte object
			byte[] buf = new byte[inputStream.available()];

			// Inputstream reads the data in byte format
			inputStream.read(buf);

			// A debug message is added to display the input XML
			if (getTrace() != null) {
				getTrace().addDebugMessage(
					"Input XML:\n" + new String(buf, "utf-8") + "\n ------");
			} else { // This section is added for normal JAVA Program run
				System.out.println(new String(buf, "utf-8"));
			}

			// Convert XML to JSON
			JSONObject xmlJsonObj = XML.toJSONObject(new String(buf, "utf-8"));
			// System.out.println("XML JSON ",xmlJsonObj.toString());

			// Call handleJSONData() to parse the JSON Structure to Delete a
			// record or convert it to an array
			if (array_nodes.size() > 0 || delete_entry.size() > 0)
				xmlJsonObj = handleJSONData(xmlJsonObj);

			// Convert the Output JSONObject to String
			jsonPrettyPrintString = xmlJsonObj.toString(2);

			// Remove the levels if required
			if (last_level_to_keep > 0) {
				jsonPrettyPrintString = (String) deleteLevel(xmlJsonObj, 0);
			} 

			// Hide Key names if required
			for (String text : hide_keys) {
				jsonPrettyPrintString = jsonPrettyPrintString.replaceAll("\""
						+ text + "\":", "");
			}  

			// Print the Final JSON structure in Trace
			if (getTrace() != null) {
				getTrace().addDebugMessage(
						"Output JSON:\n" + jsonPrettyPrintString + "\n ");
			} else {
				System.out.println(jsonPrettyPrintString);
			}

		
			// Convert the Output JSON Structure to bytes
			if (jsonPrettyPrintString == null
					|| jsonPrettyPrintString.length() < 1) {
				byte[] bytes = "{}".toString().getBytes("UTF-8");

				// Write output bytes to the output stream
				outputStream.write(bytes);
			}

			else {
				byte[] bytes = jsonPrettyPrintString.toString().getBytes("UTF-8");

				// Write output bytes to the output stream
				outputStream.write(bytes);
			}

		} catch (Exception e) {
			// Handle all exceptions
			if (getTrace() != null) {
				getTrace().addDebugMessage(
					"Exception while writing OutputPayload: IOException", e);
				outputStream.write("{}".toString().getBytes("UTF-8"));
				throw new StreamTransformationException(e.toString());
				
			} else
				e.printStackTrace();
		}
	}

	public JSONObject handleJSONData(JSONObject jsonObj) {
		/*
		 * Parse the JSON Structure to Delete a record or convert it to an array
		 * Input: JSONObject -> Json Sub structure to be updated Output:
		 * JSONObject -> Updated Json Sub structure with deleted records and
		 * arrays.
		 */

		try {
			// Create an array of keyset to loop further
			String arr[] = new String[jsonObj.keySet().size()];
			int k = 0;
			for (String key : jsonObj.keySet())
				arr[k++] = key;

			// Loop through all the keys in a JSONObject
			for (String key : arr) {

				// If there are records to be deleted, remove them and move to next key
				if (delete_entry.contains(key)) {
					jsonObj.remove(key);
					continue;
				}

				// If there are records to be converted to Array, convert it.
				if (array_nodes.contains(key)) {
					jsonObj = forceToJSONArray(jsonObj, key);
				}

				// If the sub node is a JSONArray or JSONObject, step inside the Object
				if (jsonObj.get(key) instanceof JSONArray) {
					JSONArray sjao = jsonObj.getJSONArray(key);
					for (int i = 0; i < sjao.length(); i++) {
						sjao.put(i, handleJSONData(sjao.getJSONObject(i)));
					}
					jsonObj.put(key, sjao);
				} else if (jsonObj.get(key) instanceof JSONObject) {
					jsonObj.put(key,handleJSONData(jsonObj.getJSONObject(key)));
				} else {
					// Convert number to String if num_to_string is set
					if (num_to_string){
						Object val = jsonObj.get(key);
						if(val instanceof Integer || val instanceof Float || val instanceof Double || val instanceof Long || val instanceof Short)
							jsonObj.put(key,jsonObj.get(key).toString());
					}
				} 
			}
		} catch (Exception e) {
			// Handle all exceptions
			if (getTrace() != null) {
				getTrace().addDebugMessage(
						"Exception while Updating Payload: ", e);
			} else
				e.printStackTrace();
		}

		return jsonObj;
	}

	public static JSONObject forceToJSONArray(JSONObject jsonObj, String key)
			throws org.json.JSONException {
		/*
		 * Force Convert a record to JSON Array Input: 1) JSONObject -> JSON Sub
		 * structure to be updated 2) key -> Key whose value is to be converted
		 * to JSONArray Output: JSONObject -> Updated Json Sub structure with
		 * deleted records and arrays.
		 */

		// Get the key value from JSONObject using opt() and not get(), as it
		// can also return null value.
		Object obj = jsonObj.opt(key);

		// If the obj doesn't exist inside my the JsonObject structure, create
		// it empty
		if (obj == null) {
			jsonObj.put(key, new JSONArray());
		}
		// if exist but is a JSONObject, force it to JSONArray
		else if (obj instanceof JSONObject) {
			JSONArray jsonArray = new JSONArray();
			jsonArray.put((JSONObject) obj);
			jsonObj.put(key, jsonArray);
		}
		// if exist but is a primitive entry, force it to a "primitive"
		// JSONArray
		else if (obj instanceof String || obj instanceof Integer) {
			JSONArray jsonArray = new JSONArray();
			jsonArray.put(obj);
			jsonObj.put(key, jsonArray);
		}
		return jsonObj;
	}

	public Object deleteLevel(JSONObject jsonObj, int current_level) {
		/*
		 * Based on the last level to be maintained, this method recursively
		 * deletes all previous levels. Input: 1) JSONObject -> Stores the json
		 * structure to be edited 2) Index -> Stores the level count to remove
		 * and is incremented as we step deeper in the levels Output: Object ->
		 * Currently returns String or JSONObject.
		 */

		// Read the first key in the JSONObject Structure
		String node = new ArrayList<String>(jsonObj.keySet()).get(0);
		// System.out.println("*" + node + "--"+
		// jo.get(node).getClass().getName());

		// Check if last level as per input is reached and return the output
		// string
		if (current_level == last_level_to_keep - 1) {
			// If instance is JSONObject, remove the key and return the output
			// string
			if (jsonObj.get(node) instanceof JSONObject) {
				return jsonObj.getJSONObject(node).toString(2);
			}
			// If instance is JSONArray, remove the key and return the output
			// string
			else if (jsonObj.get(node) instanceof JSONArray) {
				return jsonObj.getJSONArray(node).toString(2);
			}

			// If both above cases fail, invalid level was provided and return
			// empty string
			return "{}";
		}

		// If its not the last level step in deeper one level and pass Object
		// and incremented level index
		else if (jsonObj.get(node) instanceof JSONObject) {
			return deleteLevel(jsonObj.getJSONObject(node), ++current_level);
		} else if (jsonObj.get(node) instanceof JSONArray) {
			return deleteLevel(jsonObj.getJSONArray(node).getJSONObject(0),
					++current_level);
		}

		// If the object is no more an array or json object, invalid level was
		// provided and return empty string
		return "{}";
	}

}

This is how I handled my conversion using a generic and reusable custom java mapping that is able to handle XML to JSON conversion. Hopefully, this piece of code helps those who have faced similar situations.

Please do let me know if you have handled this situation in any other optimized way!!

Anyone who wishes to edit the content or convert this JAVA mapping to a custom XML to JSON conversion bean can please do the needful and share it…

And do hit the like button, if you found the blog useful..:)

Best Regards,

Kiran Roy