How to Handle Large Input/Output Data with Groovy IDE

[fatihpense]

Cloud Integration runtime or JVM doesn't have an issue with large input data. It only becomes a problem when you want to see the data in a user interface.

Sometimes the problem with the data is that it has no line breaks. If you can format the file then editors can happily show it to you. Other times, the data is still too big, and you might need a specialized editor or command-line editor just to peek inside.

If you want to simulate such large data on Groovy IDE it is a problem. Here, I will share one solution.

First of all, it works on the desktop version of Groovy IDE. We need the localness and full power of your computer.

Here is the code:

import com.sap.gateway.ip.core.customdev.util.Message;





def Message processData_wrapper(Message message) {

    String folder = message.getProperties().get("input_data_folder");



    String fileContents = new File(folder + 'toobig.input.property_file.orders.json').getText('UTF-8')

    message.setProperty("orders", fileContents);



    String fileContents2 = new File(folder + 'toobig.input.property_file.deliveries.json').getText('UTF-8')

    message.setProperty("deliveries", fileContents2);

    

    processData(message);



    // You can also write large body or property to file 

    def outputFile = new File(folder + 'toobig.output.property_file.orders.json')

    outputFile.write(message.getProperties().get("orders"))



    // Delete large output to avoid UI issues

    message.setProperty("orders", "removed");

    message.setProperty("deliveries", "removed");



    return message;

}





def Message processData(Message message) {

    // Develop your usual method.

    

    return message;

}

You can use a wrapper `processData_wrapper` method to avoid UI performance issues. On the Groovy IDE, you can use this wrapper method, and on the Cloud Integration tenant you can just use `processData` method.

It works for both input and output data. Remember to remove large data in the wrapper method. Also, if you are working with properties and don't need the data after this script step, you can just remove them in the `processData` method. You will save some memory for the rest of the flow.

You can use the same solution for dynamic data. For example, you can call an HTTP service and use the response as an input body or property.

We are using the `input_data_folder` property to avoid including this developer-machine-specific detail to integration flow in the tenant.