Skip to Content

SAP XI a file mover? Yes, I know, that’s not its job.

Buy you know, customers are asking for weird things sometimes, and the good consultant should provide nice solutions!

From an EAI perspective, this task doesn’t bring any added value, I know. It’s just to have SAP XI managing the whole story, and possibily raising suitable alerts whenever something fails.

The Goal

Anyway, the goal is simple: this time we want SAP XI to move files (any kind of file, be that flat, PDF, Excel, etc., and possibily huge ones) around the network, mainly using FTP Adapter, but this is not mandatory, as you we’ll notice that this technique can be applied to any Adapter running in the Adapter Engine (thus escluding IDoc and Plain HTTP).

The Solution

From a pure XI standpoint, the implementation is straighforward.

First of all, we want the picked-up file to be written with the same name. To accomplish this task, we’ll use File Adapter Specific Attributes, as explained in XI: The same filename from a sender to a receiver file adapter – SP14.

Secondly, we don’t want to manipulate the file content anyhow (no content conversion, no mapping, no xml… nothing!). To do this, we simply create a dummy Data Type, say with just one dummy element (a string), and a dummy Message Type. This dummy Message Type will be underlying each Message Interface, both Outbound and Inbound. Communication Channels (FTP Adapter, for instance) will have no Content Conversion at all, so that each handled file is put as the XI Message Payload as is (but use Binary Mode, please).

So far, so good. If you implement this scenario (should last no more than 20 minutes for a clever XI’er), you’ll have XI acting as a “dumb” file mover.

The problem arise when you have huge files, say over 100 Mb. In this case XI is handling a big file from the Adapter Engine (“read” phase) to the Integration Engine (“message handling” phase), and then back again in the Adapter Engine (“write” phase), and the process slow down a lot. On a 2 CPUs box equipped with 4 Gb RAM, moving a 50 Mb file takes around 5 minutes. With my method, less than 30 seconds. 🙂

The Idea

So why not using compression?

Once again, the solution is a very very simple Adapter Engine Module, which zips after reading the file (and before posting to the Integration Engine pipeline URL), and unzips before writing the file to the final destination. The file content remains untouched, but the Integration Engine gets relief and the whole process gets fast and furious. Additionaly, your DB will be grateful! Consider that on common flat files the compression ratio is about 99%… That is, my 110 Mb file, after being compressed by the module, is put in the XI Message Payload as a 1,5 Mb file…!

The Code

As usual, I’m not explaining how to code a Module in NWDS and deploy: for that, look around in SDN (or in other blogs o’mine).

No “strange” libraries are needed, as the basis is represented by the standard java.util.zip package.

package biz.talentlab.sap.xi.ae.modules;

import com.sap.aii.af.mp.module.*;

import com.sap.aii.af.ra.ms.api.*;

import com.sap.aii.af.service.auditlog.*;

import java.io.*;

import java.text.*;

import java.util.*;

import java.util.zip.*;

import javax.ejb.*;

/**

  • @ejbLocal <{biz.talentlab.sap.xi.ae.modules.ZipperLocal}>

  • @ejbLocalHome <{biz.talentlab.sap.xi.ae.modules.ZipperLocalHome}>

  • @stateless

*/

public class ZipperBean implements SessionBean {

     static final int BUFFER = 2048;

     static final String ENTRYNAME = “TheFile.any”;

     private SessionContext myContext;

     private ModuleContext mc;

     private final String auditStr = “Talentlab Zipper – “;

     public ModuleData process(ModuleContext moduleContext, ModuleData inputModuleData) throws ModuleException {

          Object obj = null;

          Message msg = null;

          Hashtable mp = null;

          AuditMessageKey amk = null;

          ModuleException mEx = null;

          try {

               obj = inputModuleData.getPrincipalData();

               msg = (Message) obj;

               if (msg.getMessageDirection() == MessageDirection.INBOUND)

                    amk = new AuditMessageKey(msg.getMessageId(), AuditDirection.INBOUND);

               else

                    amk = new AuditMessageKey(msg.getMessageId(), AuditDirection.OUTBOUND);

               mc = moduleContext;

          } catch (Exception e) {

               Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error while creating basic instances (obj,msg,amk,mp)”);

               throw mEx = new ModuleException(auditStr + “Error while creating basic instances (obj,msg,amk,mp)”);

          }

          // *** ZIPPING ***

          if (msg.getMessageDirection() == MessageDirection.OUTBOUND) {

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Zipping process started.”);

               try {

                    byte buf[] = new byte[BUFFER];

                    ByteArrayOutputStream baos = new ByteArrayOutputStream();

                    ZipOutputStream zos = new ZipOutputStream(baos);

                    ByteArrayInputStream bais = new ByteArrayInputStream(msg.getDocument().getContent());

                    zos.putNextEntry(new ZipEntry(ENTRYNAME));

                    int len;

                    while ((len = bais.read(buf)) > 0)

                         zos.write(buf, 0, len);

                    zos.closeEntry();

                    bais.close();

                    zos.close();

                    msg.getDocument().setContent(baos.toByteArray());

               } catch (Exception e) {

                    Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error in Zipping process: ” + ex2str(e));

               }

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Zipping process ended.”);

          // *** UNZIPPING ***     

          } else {

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Unzipping process started.”);

               try {

                    ZipInputStream zis = null;

                    ByteArrayInputStream bais = new ByteArrayInputStream(msg.getDocument().getContent());

                    zis = new ZipInputStream(bais);

                    int currentByte;

                    byte data[] = new byte[BUFFER];

                    ByteArrayOutputStream baos = null;

                    ZipEntry ze = zis.getNextEntry();

                    baos = new ByteArrayOutputStream();

                    while ((currentByte = zis.read(data, 0, BUFFER)) != -1) {

                         baos.write(data, 0, currentByte);

                    }

                    msg.getDocument().setContent(baos.toByteArray());

               } catch (Exception e) {

                    Audit.addAuditLogEntry(amk, AuditLogStatus.ERROR, auditStr + “Error in Unzipping process: ” + ex2str(e));

               }

               Audit.addAuditLogEntry(amk, AuditLogStatus.SUCCESS, auditStr + “Unzipping process ended.”);

          }

          return inputModuleData;

     }

     private String mpget(String pname) {

          return mc.getContextData(pname);

     }

     private String ex2str(Exception e) {

          StringWriter strWr = new StringWriter();

          e.printStackTrace(new PrintWriter(strWr));

          return strWr.toString();

     }

     public void ejbRemove() {

     }

     public void ejbActivate() {

     }

     public void ejbPassivate() {

     }

     public void setSessionContext(SessionContext context) {

          myContext = context;

     }

     /**

     

  • Create Method.

      */

     public void ejbCreate() throws CreateException {

          // TODO : Implement

     }

</textarea>

Usage

How to use it? Simply put “localejbs/

To report this post you need to login first.

19 Comments

You must be Logged on to comment or reply to a post.

    1. Alessandro Guarneri Post author
      And more, even if I could get a little unpopular:
      1. your opinion (just as any opinion) about what is better should be confirmed by actual benchmarks
      2. usually I don’t find Michal’s weblogs so interesting (just as other people could think the same about mine): for instance, in the blog you mention, what’s the real added value? I’m not that kind of newbie XI guy that needs a blog to figure out how a standard and documented feature works. But I know that Michal’s contribution in SDN is really appreciated, so I respect its job.

      Regards,
      Alex

      (0) 
      1. Stefan Grube
        Hi Alessandro,

        I find your blog very interesting, otherwise I wouldn’t have written a comment for it 😉
        The argument with the benchmark beats me. I only considered the implementation time.

        Regards
        Stefan

        (0) 
      2. Stefan Grube
        Hi Alessandro,

        I find your blog very interesting, otherwise I wouldn’t have written a comment for it 😉
        The argument with the benchmark beats me. I only considered the implementation time.

        Regards
        Stefan

        (0) 
      3. Michal Krawczyk
        Hi Alex,

        >>>>2. usually I don’t find Michal’s weblogs so interesting

        I don’t see a problem with that Alex 🙂

        I don’t write blogs for XI experts as:
        a) they don’t need blogs basically as you’ve noticed
        b) I want to write blogs that MANY people read & undestand (something like MC Donalds of integration  LOL:)

        BTW
        on the other hand I find your blogs very interesting
        (and always wait for a new episode)
        even though I almost never use them in real projects

        Regards,
        michal

        (0) 
          1. Alessandro Guarneri Post author
            I’m not really catching the meaning of your comment, Henrique…
            If you mean that I wasn’t polite, well, I don’t think so: I just expressed my technical opinion.
            I am used to say what I think. That’s it.
            (0) 
    2. Sergio Locatelli
      You’re right, the proposal appears as crazy method to use XI.
      But this kind of method is frequently required from the customer.
      Could be, as example, after/before some steps of a more complex integration scenario. XI must sinchronize all the steps of that scenario (simple ftp of file included).

      The zip/unzip feature could be real intresting in this direction.

      Regards,
      Sergio

      (0) 
  1. ranga rajan
    Hi ,
          
         Excellent !. Blog is very good. I have doubt in receiver determination . As u specified i have not used any interface mapping , message mapping .

             I have deployed the Module .It picking up file  and zip  that file also . But while in receiver determination  , without interface mapping how can i assign the Receiver ?

            It would be tahnkful if u have replied !

    Best Regards.,
    V.Rangarajan

    Regards.,
    V.Rangarajan

    (0) 
    1. Alessandro Guarneri Post author
      Hi,
      Well, in Rcv Det. you do have both an Interface Determination and Rcv Agreement, so that you determine both the Rcv System and the Rcv Msg Itf.
      What you don’t need is the Interface Mapping in the Interface Determination. That’s it!

      Regards,
      Alex

      (0) 
  2. Dev Noronha
    This is an excellent solution and it works great with my FTP/FTPS channels. However, when I use it on an advantco sftp channel, it get
    Error when sending message from source file ‘/public/temp/test.txt’: com.sap.engine.services.ts.transaction.TxRollbackException: Current transaction is marked for rollback

    Actually, this error comes with any adapter module i use on sftp. Any idea?

    (0) 
  3. senthilprakash selvaraj
    How much time does the Module take to ZIP a file of say 100MB before it sends to IE for Pipeline steps.

    This is a crucial factor to be considered. because compressing a 100MB file is not a Joke :).

    Do let me know.

    Regards,
    Senthilprakash

    (0) 

Leave a Reply