SAP XI acting as a (huge) file mover

SAP XI a file mover? Yes, I know, that’s not its job.

Buy you know, customers are asking for weird things sometimes, and the good consultant should provide nice solutions!

From an EAI perspective, this task doesn’t bring any added value, I know. It’s just to have SAP XI managing the whole story, and possibily raising suitable alerts whenever something fails.

The Goal

Anyway, the goal is simple: this time we want SAP XI to move files (any kind of file, be that flat, PDF, Excel, etc., and possibily huge ones) around the network, mainly using FTP Adapter, but this is not mandatory, as you we’ll notice that this technique can be applied to any Adapter running in the Adapter Engine (thus escluding IDoc and Plain HTTP).

The Solution

From a pure XI standpoint, the implementation is straighforward.

First of all, we want the picked-up file to be written with the same name. To accomplish this task, we’ll use File Adapter Specific Attributes, as explained in XI: The same filename from a sender to a receiver file adapter – SP14.

Secondly, we don’t want to manipulate the file content anyhow (no content conversion, no mapping, no xml… nothing!). To do this, we simply create a dummy Data Type, say with just one dummy element (a string), and a dummy Message Type. This dummy Message Type will be underlying each Message Interface, both Outbound and Inbound. Communication Channels (FTP Adapter, for instance) will have no Content Conversion at all, so that each handled file is put as the XI Message Payload as is (but use Binary Mode, please).

So far, so good. If you implement this scenario (should last no more than 20 minutes for a clever XI’er), you’ll have XI acting as a “dumb” file mover.

The problem arise when you have huge files, say over 100 Mb. In this case XI is handling a big file from the Adapter Engine (“read” phase) to the Integration Engine (“message handling” phase), and then back again in the Adapter Engine (“write” phase), and the process slow down a lot. On a 2 CPUs box equipped with 4 Gb RAM, moving a 50 Mb file takes around 5 minutes. With my method, less than 30 seconds. 🙂

The Idea

So why not using compression?

Once again, the solution is a very very simple Adapter Engine Module, which zips after reading the file (and before posting to the Integration Engine pipeline URL), and unzips before writing the file to the final destination. The file content remains untouched, but the Integration Engine gets relief and the whole process gets fast and furious. Additionaly, your DB will be grateful! Consider that on common flat files the compression ratio is about 99%… That is, my 110 Mb file, after being compressed by the module, is put in the XI Message Payload as a 1,5 Mb file…!

The Code

As usual, I’m not explaining how to code a Module in NWDS and deploy: for that, look around in SDN (or in other blogs o’mine).

No “strange” libraries are needed, as the basis is represented by the standard java.util.zip package.

package biz.talentlab.sap.xi.ae.modules;

import com.sap.aii.af.mp.module.*;

import com.sap.aii.af.ra.ms.api.*;

import com.sap.aii.af.service.auditlog.*;

import java.io.*;

import java.text.*;

import java.util.*;

import java.util.zip.*;

import javax.ejb.*;

/**

• @ejbLocal <{biz.talentlab.sap.xi.ae.modules.ZipperLocal}>

• @ejbLocalHome <{biz.talentlab.sap.xi.ae.modules.ZipperLocalHome}>

• @stateless

*/

public class ZipperBean implements SessionBean {

     static final int BUFFER = 2048;

     static final String ENTRYNAME = “TheFile.any”;

     private SessionContext myContext;

     private ModuleContext mc;

     private final String auditStr = “Talentlab Zipper – “;

     public ModuleData process(ModuleContext moduleContext, ModuleData inputModuleData) throws ModuleException {

          Object obj = null;

          Message msg = null;

          Hashtable mp = null;

          AuditMessageKey amk = null;

          ModuleException mEx = null;

          try {

               obj = inputModuleData.getPrincipalData();

               msg = (Message) obj;

               if (msg.getMessageDirection() == MessageDirection.INBOUND)

                    amk = new AuditMessageKey(msg.getMessageId(), AuditDirection.INBOUND);

               else

                    amk = new AuditMessageKey(msg.getMessageId(), AuditDirection.OUTBOUND);

               mc = moduleContext;

          } catch (Exception e) {

               Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error while creating basic instances (obj,msg,amk,mp)”);

               throw mEx = new ModuleException(auditStr + “Error while creating basic instances (obj,msg,amk,mp)”);

          }

          // *** ZIPPING ***

          if (msg.getMessageDirection() == MessageDirection.OUTBOUND) {

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Zipping process started.”);

               try {

                    byte buf[] = new byte[BUFFER];

                    ByteArrayOutputStream baos = new ByteArrayOutputStream();

                    ZipOutputStream zos = new ZipOutputStream(baos);

                    ByteArrayInputStream bais = new ByteArrayInputStream(msg.getDocument().getContent());

                    zos.putNextEntry(new ZipEntry(ENTRYNAME));

                    int len;

                    while ((len = bais.read(buf)) > 0)

                         zos.write(buf, 0, len);

                    zos.closeEntry();

                    bais.close();

                    zos.close();

                    msg.getDocument().setContent(baos.toByteArray());

               } catch (Exception e) {

                    Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error in Zipping process: ” + ex2str(e));

               }

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Zipping process ended.”);

          // *** UNZIPPING ***     

          } else {

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Unzipping process started.”);

               try {

                    ZipInputStream zis = null;

                    ByteArrayInputStream bais = new ByteArrayInputStream(msg.getDocument().getContent());

                    zis = new ZipInputStream(bais);

                    int currentByte;

                    byte data[] = new byte[BUFFER];

                    ByteArrayOutputStream baos = null;

                    ZipEntry ze = zis.getNextEntry();

                    baos = new ByteArrayOutputStream();

                    while ((currentByte = zis.read(data, 0, BUFFER)) != -1) {

                         baos.write(data, 0, currentByte);

                    }

                    msg.getDocument().setContent(baos.toByteArray());

               } catch (Exception e) {

                    Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error in Unzipping process: ” + ex2str(e));

               }

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Unzipping process ended.”);

          }

          return inputModuleData;

     }

     private String mpget(String pname) {

          return mc.getContextData(pname);

     }

     private String ex2str(Exception e) {

          StringWriter strWr = new StringWriter();

          e.printStackTrace(new PrintWriter(strWr));

          return strWr.toString();

     }

     public void ejbRemove() {

     }

     public void ejbActivate() {

     }

     public void ejbPassivate() {

     }

     public void setSessionContext(SessionContext context) {

          myContext = context;

     }

     /**

     

• Create Method.

      */

     public void ejbCreate() throws CreateException {

          // TODO : Implement

     }

</textarea>

Usage

How to use it? Simply put “localejbs/