Dynamic File Splitting using SAP Data Services

There might be a requirement where you need to split bulk table data into several files for loading further into other systems. Instead of creating single file and executing the job several times, in Data Services we can automate this process to split the files dynamically based on no of files and records required.

So lets assume we have a table T1 contains 10 Million records and we are required to split into 10,000 chunks each.

Overview of dynamic file splitting process

1) Adda new column to the table and populate with sequential numbers. This will be used to identify the chunks of records.

2) Create a script to declare and initialize variables for file count(ex 50),records count(ex 10000) etc.

3) Create a WHILE loop to run for no of times as many files are required.

4) Create a new DF inside the loop to split the records and push to file format.

5) Create a post processing script to increment the variable values.

Sample working demo of this process is shown in this video.