Skip to Content
Author's profile photo Former Member

File/FTP Adapter – Large File Transfer (Chunk Mode)

This blog is the extension of following blogs.


Whenever large (binary) file needs to be transferred by using PI, there might be a chance for Out of Memory error which might results in server restart.  Technically, it often fails in the sender adapter.


The most popular work around is to split the file in to multiple parts in Sender Adapter and Combining them in Receiver Adapter (via OS Scripts)

From PI 7.30 onwards File/FTP adapter natively supports transferring large size files by splitting them in to smaller chunks based on the configured size. Each chunk will be processed as an individual XI Message in sender adapter and all the chunks are combined based on the sequence in receiver adapter. Since all the chunks have to be combined in the actual order, QoS EOIO has to be used to enable this feature.


Configuration Options

Select ‘EOIO’ as a QoS under Processing tab.


After that select ‘Advanced Mode‘ under ‘Advanced’ Tab

Option 1

You can choose the required chunk size option from the given list of 1, 2, 5, 10, 20 and 50 MB.


Option 2

Custom values which are not possible in default options can also be configured in additional parameters.

Parameter Name

Possible Values


Non emptry string  will be considered as true


Positive Numeric  Value


Processing in Sender Adapter

  • Basic Mode will be enabled only if QoS is EOIO
  • Chunk Mode related Advanced Mode parameters will be ignored if QoS  not EOIO
  • If chunkSizeKB is not defined /incorrect value, default is 10KB
  • If both are configured, Basic mode will have higher precedence

Sender Adapter will create XI message with additional dynamic headers for each chunk and forwarded to Messaging System for IRD processing.


For this example, chunk size is configured as 2 MB and the below input file (~4.5 MB) is used.


For demo purpose and to identify/analyze how the actual split happens, the input file is created with the data like below ( 1 to 600000)


As per the configuration, three chunks should be created with the size of 2 MB, 2MB and ~500 KB.


Chunk 1 (2 MB)

Transferring first 2 MB


Chunk 2 (2 MB)

Transferring next 2 MB (Total 4 MB)


Chunk 3 (~500 KB)

Transferring the remaining ~500 KB (Total ~4.5 MB)


Dynamic Headers

The entire chunk mode related dynamic headers will be set under the namespace

Header Value
ChunkStart Start of the Chunk (Position of bytes)
ChunkMode Active/End
ChunkKey Unique Key (Same for all the chunks for a particular file)

The corresponding screenshots for all 3 chunks are given below.

First Chunk


Next Chunk


Last Chunk


If multiple files sent from different file sender channels, receiver adapter uses the chunk key to identify the proper output file(for combining). During the creation(and append) of output file, it uses the chunk key like below.

In Temp mode, the temp files won’t be deleted till the last chunk is received.

Overall Control Flow


Recovery Mode

If the input file is very large, it would normally take more time to be transferred completely.  If there is any server failure in the middle, the transfer can resumed from the chunk it stopped (Not from the beginning)

Individual Payloads after Split

Though this feature operates only on binary mode, text file is used to analyze how the split happens during chunk creation.

Payload of Chunk 1


Payload of Chunk 2

Payload of Chunk 3

As per the above screenshots, the split never cosiders the payload. It’s just a binary split. So the following limitations would apply

  • Only for File Sender to File Receiver
  • No Mapping
  • No Content Based Routing
  • No Content Conversion
  • No Custom Modules

Technically, other type of receiver adapters can be used. But it won’t have the option to combine the chunks.

Does any special configuration required in File Receiver?

No. File receiver will automatically identify & combine the chunks using dynamic headers.

How to choose the chunk size? Smaller chunk size ensures more reliability?

Yes.  But significantly affects the performance. Smaller chunk size creates more chunks (more XI messages)


If chunks are stuck in the EOIO queue, it can be monitored with


Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Hello, The blog links in your blog point to the same blog. Is this feature available on 7.1? I was under the impression that this is a feature available only from 7.31

      A very helpful feature! Great blog with good explanation!

      Author's profile photo Former Member
      Former Member
      Thanks. Updated the links.
      Author's profile photo Former Member
      Former Member
      This feature can be found in 7.1 latest SPs. But as of now, officially supported only in 7.3+
      Author's profile photo Former Member
      Former Member
      Thanks for the blog Siva...

      Can u tell me for which latest SP's it will supoort in PI 7.1.

      Author's profile photo Shabarish Nair
      Shabarish Nair
      So this means we cant use it in productive PI 7.11 environments?

      Well that would be a shame, isnt it!

      Author's profile photo Former Member
      Former Member
      Yes. But we can use it at our own risk.
      Author's profile photo Bhavesh Kantilal
      Bhavesh Kantilal
      Does this feature continue to support ASMA - FileName? Can I use it to transfer files from source to target with same original filename?

      I see no reason it shouldn't but would be good if this can be confirmed.


      Author's profile photo Former Member
      Former Member
      It will perfectly work with ASMA.
      Author's profile photo Former Member
      Former Member

      not able to view the screeb shots can you correct the screens shots



      Author's profile photo Former Member
      Former Member

      Hi Sreeram,

      Due to SDN to  SCN migration, few screenshots are corrupted. I will check with SCN team.



      Author's profile photo Former Member
      Former Member

      Hi Siva, All

      All images should be fine now.



      Author's profile photo Jose Augastine
      Jose Augastine


      Still some images are missing.



      Author's profile photo Former Member
      Former Member

      Hi Jose, We will try to recover the missing images.

      Author's profile photo Former Member
      Former Member

      Very good information. One additional point leading us to move to PI 7.31 in the near future i guess.

      Author's profile photo RAVI REDDY

      hi thanks for innovative information

      thanks & regards

      E.Ravi chandra reddy

      Author's profile photo Martin Babayan
      Martin Babayan


      Have you ever been able to do the same within ICO?

      I configured a classical scenario in Pi 7.40 and it worked fine. Then I decided to migrate it ICO (just for test) and now it's failing with error in sender CC "Unable to parse XML message payload". But why it is trying to parse the file?

      Will appreciate your input.

      Author's profile photo Otto Frost
      Otto Frost

      You need to use XI3.0 interface or dummy interface.

      With other interface types it tries to parse the payload to find out the operation.

      Author's profile photo Former Member
      Former Member

      Hi Siva,

      Chunk mode will be only useful for file to file scenarios? not useful for File to Proxy interfaces? Please let me know



      Author's profile photo Former Member
      Former Member


      Have the same/similar question. File to Proxy/Proxy to File scenarios.



      Author's profile photo G SHIVA
      G SHIVA

      Hi Siva ,

      Can we append two files at receiver side, when we are using QOS : EOIO for sender communication channel with advanced selection for source option (when are try to pick multiple files from multiple directories??)


      Author's profile photo Former Member
      Former Member

      I've tried using this temp mode together with chunk mode and it is working on File to File Scenario but not on File to SFTP scenario.

      on SFTP it always overwrites the file.  Is there a way to use this function in SFTP Receiver?

      Author's profile photo Arkesh Sharma
      Arkesh Sharma

      Hi Everyone,

      I wish to know if the chunking will work for XML files? I tried it and it is giving me invalid XML formats for my large files.

      Kindly advise in this regard.




      Author's profile photo Laxman Molugu
      Laxman Molugu

      Thanks for sharing, I used this chunk mode it worked great. But I added ZipBean Module in the chain that worked very well. The Note "Custom Modules can't be used" - does this mean ZipBean a standard module?

      Please let me know if any SAP's relevant adapter Modules can be used with this chunk mode option