Abstract
+
Customers often ask for massive file transfers for the interfaces affecting their systems.
This blog describes the realization of a custom adapter module in SAP PI (Process Integration) 7.1 to manage the proper acquisition
of chunk files pertaining to a single large file.
+
Referenced sites
How to Create Modules for the JEE Adapter Engine
Scope
In order to send large files, the sender system divides them into smaller chunk files (e.g.: 5 MB) to send in parallel.
Once this operation is over, an additional end-of-job file (.eoj) containing the list of chunk files is produced and sent.
Assuming that the sender system does not guarantee the sending order, the eoj file must enable the check on the proper reception of all
chunk files and determine their acquisition in the settled sequence, in order to reconstruct the original file.
The process therefore provides for the system that receives the chunk files to wait for the arrival of the end-of-job file.
Once the eoj file is obtained, the receiving system checks that all files listed have been properly received. If so, these files are
renamed to be acquired in order. Otherwise the receiving system reports the missed reception and checks again after a preset time.
Realization
Following the directions given in the document , a custom module (called Chunk Message Check - CMC) has been created for the file adapter
in SAP PI 7.1 system. This module is tailored for the FTP protocol, and it directly gets from the adapter all information necessary to access.
The only information that must be provided in input is the new extension of the chunk file to be renamed (otherwise .ch will be used by default).
In order to implement the interface that runs the CMC custom module, you just need to use objects from the Integration Builder
(design objects are not necessary). For example, you can define an integrated configuration object with a File sender and a File receiver channel,
using dummy interfaces (eg: Dummy_Out and Dummy_In) and the same business system as sender and receiver.
The file sender channel must be configured in order to acquire the end-of-job files and must provide for an adapter module CMC at the beginning
of the processing sequence, before the File adapter. You also have to insert the parameter ChunkExt to define the extension of the renamed files.
Java code
Below is the java code that implements the adapter module Chunk Message Check. i; i++)
{
newFileName = oldFileName = "";
oldFileName = ((String)chunkfiles.get(i));
newFileName = oldFileName+this.ChunkExt;
ftpClient.rename(oldFileName, newFileName);
}
audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS,
"CMC: all chunks RENAMED!");
}
else
{
//audit.addAuditLogEntry(amk, AuditLogStatus.ERROR,
//"CMC: some file MISSING ");
throw new Exception("Some chunk file missing: "counter" of "+chunkfiles.size());
}
} else {
audit.addAuditLogEntry(amk, AuditLogStatus.ERROR,
"CMC: login fail");
}
} catch (IOException ioe) {
audit.addAuditLogEntry(amk, AuditLogStatus.ERROR,
"CMC: "+ioe.getMessage());
throw new MessagingException(ioe);
} catch (Exception e){
throw new MessagingException(e);
}
finally {
try {
if(login)
{
// When logout success the logout method returns true.
boolean logout = ftpClient.logout();
if (logout) {
audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS,
"CMC: logout from FTP server ");
}
}
// Closes the connection to the FTP server
ftpClient.disconnect();
} catch (IOException e) {
throw new MessagingException(e);
}
}
}
}
Example run
A file test.eoj has been added to perform a test of the CMC module; within this file there are two file names which are the chunks
deriving from the splitting of a larger file.
The communication channel FILESenderTest reads the test.eoj file and looks for the chunks described in it.
Since there are no chunk files in the directory, the communication channel ends with processing error, showing the number of chunks detected
and the number of those expected (0/2 in this case). The channel will attempt a new check after 60 seconds, as defined in the configuration.
Below is the log concerning the steps performed by the communication channel FILESenderTest in the attempt to identify the chunks expected
in the file test.eoj.
By entering the expected chunk files (chunk1.txt and chunk2.txt), the next processing loop of the FILESenderTest channel allows
the identification of each chunk listed in the test.eoj file, and executes the renaming in order to get the chunks in the correct order by a
further standard communication channel.
In this case the FILESenderTest communication channel correctly performs the process (as it is shown below), identifying and renaming
all chunk files and moving the file test.eoj from the reading directory.
The directory now contains the chunk files properly renamed and ready to be acquired in the correct order by another
communication channel for the acquisition in Exactly Once In Order mode.
2 Comments
