Skip to Content

How to handle large volume of data for Proxy-to-File scenario

1.0 Introduction

We have seen in many scenarios in which we need to send more than 100 Mb data (XML payload volume) using proxy where we are facing system error due to large data volume. The error we are facing for message size approx. around 100 Mb is attached below.


For this kind of situation we have come up with a solution which is dealt in details in the following points.

2.0 Business Requirements

We had a requirement to send very large amount of data for Proxy-to-File scenario. To do so, the ABAP program will split the data in small packets (around 22 Mb XML payload size) and sent via EOIO, check the below link to ensure the message sent via proxy as EOIO:

In PI we will create a temporary folder. This temporary folder will contain target file, in which the data from proxy will be appended in correct sequence until the end of data is not reached. The moment last part of proxy data is appended in target file, a batch file will be created. This batch file will move the target file from the temporary folder location to the required target location then remove the temporary folder and at the end delete itself (batch file). So the batch file is not visible to the user. Check the below points for details.

This architecture cannot be done with AAE as “Maintain order at runtime” can only be maintained in classical interface determination.

3.0 Methodology

STEP 1          Create the design object in ESR comprising data types, message types, service interfaces, message mapping and operational mapping.

  • First create source data type (for proxy structure) and two target data types(one for target file structure and another for batch file)
  • In source data type, three extra mandatory fields (CurrentFileNo, TotalFileNo and filename) apart from proxy structure needs to be added. CurrentFileNo to get current file number, TotalFileNo to get total number of file split and filename to get name of the file with timestamp.)
  • The first target data type contains file structure along with “Folder_Details” comprising “FileNameScheme” and “TargetDirectory”.
  • The second target data type is created for batch file processing.
  • Create one split mapping between one source message type and two target messages as shown in below attachment:

           Mapping logic for creating target message one:


  • Value mapping is defined to return target file location, using this location we are creating a temporary folder (starting with “Temp_”) in which the temporary target file is to be appended until we get the full data for a particular proxy call./wp-content/uploads/2012/12/3_168107.png     
  • All other target fields are mapped with source message according to the business rule.
  • The mapping used for target message two (for batch file creation) are attached below:


The target message uses DynamicConfiguration UDF for creating file name and directory in batch file communication channel.

Mapping logic for Target node of target message two (batch file):


Mapping logic for Line1:


Display Queue for Line1, displays the batch file run time command:


Mapping logic for Line2:


Display Queue for Line2,displays the batch file run time command:


Mapping logic for Line3:


Display Queue for Line3, displays the batch file run time command:


STEP 2          Create the configuration objects in ID comprising communication channels, receiver determination, interface determination, receiver agreement, value mapping objects and configuration scenario.

  • Two receiver communication channels of adapter type file are required, one for target file and another one for batch file processing.
  • The first communication channel (for main file) is configured as:


The target directory and file name scheme is provided by using variable substitution set in Advanced tab.


The file construction mode is set as Append in Processing tab, so that data from each proxy call get appended in file in temporary folder until the final proxy call.


The Content Conversion parameters are set as per the target structure. To handle the extra line and space which is generated due Folder_Details node in target structure fixedLengthTooShortHandling parameter is used here.


The second communication channel (for batch file) is configured as:


Here in Advanced tab the Adapter-Specific Message Attributes properties are enabled so that we can provide file name and directory from message mapping (using UDF) itself same as first communication channel. Thus File Access Parameters in Target tab is set as *.


The Content Conversion Parameter is defined as:


The Processing tab plays very important part in this communication channel to run operating system command, in this case %F to get file name along with full file path.


The value mapping object is created in ID to get folder location of temporary folder:


Now create the other configuration objects in conventional manner receiver determination, interface determination and receiver agreements for both receiver type communication channels.

4.0 Unit Testing

STEP 1          Here we are testing the scenario with two parts (i.e TotalFileNo = 2) of proxy data.

STEP 2          In the first (i.e. CurrentFileNo = 1) proxy call a temporary folder is created in the target location. Go to the SXMB_MONI transaction and monitor the message.



For second (i.e. CurrentFileNo = 2 and TotalFileNo =2) call, target message two is created in mapping:


The target message two (batch file one) is displayed in below screen shot.


Since this is the second proxy (i.e. CurrentFileNo = 2 and TotalFileNo = 2) call so temporary folder is deleted and the file is moved to target location.



You must be Logged on to comment or reply to a post.
    • In classical configuration, inside interface determination we have option “Maintain Order at Runtime” but in ICO we can not maintain order.

  • Hi Sabyasachi,

    Could you give a further explanation of the .bat file? How to trigger the .bat file?

    What is the context in the .bat file? How is the data file moved out the temp folder?


    Kevin Song

    • 1. Data file will be moved to temporary folder. Temporary folder name will be “Temp_” concatenate with file name excluding file extension. Suppose file name is abc201210191223123.txt, then temporary folder name will be Temp_abc201210191223123. Inside this temporary folder “abc201210191223123.txt” file will be created in append mode using the first communication channel. In this communication channel file and directory name will be populated using variable substitution.

      2. For the final proxy call (i.e currentFileNo = totalFileNo, here it is 2) the .bat file will be triggered. Batch file name will be populated in the second file communication channel using UDF (dynamic configuration).

      3. Batch file will be triggered from command line from the second communication channel. Here we are using %F (absolute file name including path) in the Run Operating System Command After Message Processing.

      4. This batch file have 3 command lines.

      • First command will move the final data file to actual target folder.
      • Second command will delete the temporary folder.
      • Third command line will delete the batch file itself.

      5. Batch file contain is given below.

      • MOVE /Y D:\Test_Data\Test_Scenario\Temp_FILE_TEST_01\*.txt D:\Test_Data\Test_Scenario
      • RMDIR /S /Q D:\Test_Data\Test_Scenario\Temp_FILE_TEST_01
      • DEL /Q D:\Test_Data\Test_Scenario\FILE_TEST_01.bat