Skip to Content

I spent a lot of time in searching if this was actually possible, did a lot of research, tried many different methods and finally I’ve been successful.


Before I start let me give you some background;

Business Scenario


The Sender system is IBM WebSphere MQ and the receiver side is an SFTP.


So it becomes JMS to SFTP.

Untitled-1.png


Business Requirement 


The sender side is splitting each xml document to 3 mb parts. But it is splitted irregularly.


For example;


This is the structure before they split and what receiver system expects from PI.

Untitled-2.png


The xml document above becomes like below parts when client put the files to MQ.  

Untitled-3.png

Untitled-4.png

Untitled-5.png


Another requirement is, the message parts are uploaded to MQ unordered. So last message part can be at first row or first message part can be at middle rows etc.

And total size of the complete/unsplitted xml document can be max. 50+ MB.

Also quantity of message parts is unknown and most important thing is there is nothing to use to define the message parts sequence.



Solution Tries


Firstly it occurred to me that I can send message parts to ccBPM and merge them with a transformation step. Then I tried this scenario.

Untitled-7.png


Because we don’t have any possible field to use in correlation, I made dummy correlation like giving ‘sender component’.

Untitled-8.png


When I run this scenario, I noticed that incoming messages are failing on transformation step. I thought that reason of this exception is while messages goes in to mapping, they are tried to parse as if the message is xml.


By the way sender and receiver adapter validations are off.

Untitled-9.png


To achieve it I tried to add xml elements like <Envelope> and </Envelope> at beginning of the message and end of the message to convert my irregular xml message to smooth and parsable xml message.


First I tried this method using XSLT by embedding incoming message to smooth xml structure as CDATA. This was 1. step on Operation Mapping and the 2. step was my actual mapping which I merge incoming messages as below.

Untitled-10.png


This attempt failed. Because XSLT expects incoming data as xml format.


Then I used Java Mapping instead of XSLT with the same logic; embedding incoming data in to xml structure as CDATA. But this time I placed java mapping in front of the ccBPM instead of transformation step like below. This attempt failed too. 

Untitled-11.png


I realized that it mighty have impossible to handle irregular xml after message go out from sender channel.


Then I decided to write a simple custom module on JMS adapter to convert incoming data to xml message by adding xml tags beginning and end of the message.


After the start of writing EJB, an idea came; why I didn’t just make a runnable jar and call it from channel?



Solution


I made some changes on configuration and made it JMS to File as shown below.

Untitled-12.png


PI is taking message parts from MQ and putting them in a folder on itself without any conversion.


While leaving the messages to folder, there is an operating system command on receiver file adapter which works after message processing.

Untitled-13.png


With this command PI executes a script file whenever each message file created in target folder.


The script file executes a runnable jar file. But before the execution it checks whether the jar is running currently or ready to run at that moment. This check is necessary to prevent triggering jar file on every message part. It must run once when the first message part placed in the folder.


When the jar app is executed, it fetches all message part files from PI’s output folder and goes in a loop which turns number of file times.


First looks for the begin of root element if the message part contains it then it is assigned to a string. Then looks for the end of root element likewise and it is assigned to another string. Remainders is assign to another else string by concatenating each other in order and in a row.
After each assignment, currently processing file is moved to ‘processed’ folder.


At the end of the loop, which means all message part files are processed and moved from PI’s output folder, the strings assigned above are concatenated and this generates complete xml. At the same time this strings represents three main part of the complete xml which are begin part, middle parts and end part.


Then the new complete xml file is created with containing timestamp in filename to prevent duplicate situation.


The checking logic in bat file is, if jar file name is ‘ready’ then change its name to ‘running’ and execute it. Then check the folder continuously which PI puts incoming message parts, until there is no file anymore. This means jar app fetched all message parts and done its job. Then bat file changes the jar file’s name again to ‘ready’  then the script ends.


Also I made some performance tests like merging 20 files which equals to 60 mb total. During the tests I have observed that playing with big data causes Java heap size error. To achieve it I added -Xmx parameter to bat file which sets maximum java heap size.


As a result the script in front of the jar app enables to run jar app once until all message parts processed.






Script file;



@echo off
if exist "XMLMerger_ready.jar" (
rename XMLMerger_ready.jar XMLMerger_running.jar
java -Xmx512M -jar XMLMerger_running.jar
)
:loop
if not exist "T:\in\incoming\%" (
rename XMLMerger_running.jar XMLMerger_ready.jar
exit
) else (
goto :loop
)




Jar source code;


package com.xml;
/**
* @author ridvanpolat
*
*/
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Timestamp;
public class Merge {
    public static void main(String[] args) throws IOException {
 
        // wait 5 seconds to let PI to put all message parts to the incoming folder
        try {
            Thread.sleep(5000);
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
        // declarations
        java.util.Date date = new java.util.Date();
        String beginPart = "";
        String midParts = "";
        String endPart = "";
        // we will put generated xml to this path
        String path = "T:\\in\\";
        // we will fetch message parts from this path which PI puts
        File file = new File(path + "\\incoming\\");
        // check whether the path is exist or not
        if (file != null && file.exists()) {
 
            // get all files to an array, not only the filename, gets with the path
            File[] listOfFiles = file.listFiles();
            // if there is file in the folder then..
            if (listOfFiles != null) {
     
                // we will use this string to collect all parts at the end
                String completeXML = "";
     
                // run the codes below number of file times
                for (int i = 0; i < listOfFiles.length; i++) {
         
                    // this control is necessary because there might be some other non-file items in the folder, we have to process only files.
                    if (listOfFiles[i].isFile()) {
             
                        BufferedReader br = null;               
                        try {
                   
                        // for the current file. we are reading first line only, because the part files are composed of single line
                            String sCurrentLine;        
                            br = new BufferedReader(new FileReader(listOfFiles[i]));
                        
                            // if file is not empty
                            while ((sCurrentLine = br.readLine()) != null) {
       
                                // get begin part
                                if (sCurrentLine.contains("<Envelope")) {
                                    beginPart = sCurrentLine;
       
                                    // check if the message is complete xml
                                    if (sCurrentLine.contains("</Envelope")) {
                                        completeXML = sCurrentLine;
                                    }
       
                                    // get end part
                                } else if (sCurrentLine.contains("</Envelope")) {
                                    endPart = sCurrentLine;
         
                                    // get middle parts and concat
                                } else {
                                    midParts = midParts.concat(sCurrentLine);
                                }
                            }
         
                        } catch (IOException e) {
                            e.printStackTrace();
                        } finally {
                            try {
                                if (br != null) {
                                    br.close();
                                }
                            } catch (IOException ex) {
                                ex.printStackTrace();
                            }
                        }
             
                        try {
                        // move processed files from incoming folder
                            if (listOfFiles[i].renameTo(
                                    new File(
                                            path + "\\processed\\"
                                            + listOfFiles[i].getName()))) {
                                System.out.println("File is moved successful!");
                            } else {
                                System.out.println("File is failed to move!");
                            }
                        } catch (Exception e) {
                            e.printStackTrace();
                        }        
                    }
                
                // endloop
                }
            
                // concat 3 main part in order and in a row
                completeXML = beginPart.concat(midParts.concat(endPart));
            
                // for error handling from script file
                System.out.println(completeXML);
         
                BufferedWriter output = null;
                try {
           
                // get time stamp
                    String getTimestamp = new Timestamp(date.getTime()).toString();
                    // change unsupported characters for filename
                    getTimestamp = getTimestamp.replace(' ', '_').replace(':', '-').replace('.', '-');
                
                    // create new file with xml extention and timestamp to prevent duplicate situation
                    File newFile = new File(path + "\\XMLdata" + getTimestamp + ".xml");
                    output = new BufferedWriter(new FileWriter(newFile));
                
                    // push our concated data to new file
                    output.write(completeXML);
                
                } catch (IOException e) {
                    e.printStackTrace();
                } finally {
                    if (output != null) {
                        output.close();
                    }
                }
         
            }
        }
 
    }
}





I hope has been useful.

Ridvan Polat

To report this post you need to login first.

Be the first to leave a comment

You must be Logged on to comment or reply to a post.

Leave a Reply