Migration Cockpit – Parallelization of XML file loads

Hello SAP S/4HANA Cloud Community,

this would be my first blog here, when I would like share some insight on how to simultaneously transfer multiple XML files with the S/4HANA Migration Cockpit. It should improve the throughput of the data load during migration.

Loading one large XML file may take up too much time and the system doesn’t parallelize the processing by default. Therefore If you would like to achieve parallel processing keep reading! To introduce this functionality for your migration object, perform the following steps:

1. Instead of creating one big file, split your data in several (n) files.
2. Open fiori app “Migrate Your Data” and the object where you would like to introduce parallelization
3. Upload all (n) files and set the status to “active”
4. Click “Edit” and in the field “Max. Data Transfer Jobs” enter a number of jobs (m) higher than 1 in order to transfer (n) files in parallel. Note: m <= n.

The number entered here will allow the system to trigger multiple jobs for data transfer. Therefore if several files are being transferred at the same time, those will be processed by the system in parallel if this attribute (“Max. Data Transfer Jobs”) is higher than 1. Otherwise only a single load job will be used to process all files.

Please note that your system is configured with a certain number of available batch job processes. For S/4HANA Cloud, the migration cockpit will only allow up to 80% of these processes (number m) to be used for the migration object.

Avoid running parallel processing using several migration projects and objects at the same time if the total number of batch jobs exceeds the available system resources.

Example:
You have split one large file into three smaller files, and you have set the “Max. Transfer data jobs” to value ‘3’. The SAP S/4HANA migration cockpit will try to migrate all three files in parallel using background jobs.

Restrictions:

• This feature is not supported for migration object “Cost Center” because of the hierarchy group creations.
• When splitting the data to multiple files, note the requirements detailed in KBA 2719524 for the XML size limit.
• It is not recommended to migrate multiple migration objects simultaneously using several projects as this will only occupy job resources.
• In SAP S/4HANA Cloud the number of available batch processes is normally 10 per instance (AppServer).
• Usually, for quality environments, there is one AppServer (10 jobs available) and for productive environment 2 Application Servers (20 Jobs available). The migration cockpit will only allow up to 80% of these processes to be used for the migration object.
• If you try to migrate several migration objects in parallel by using multiple migration projects, the system could run out of batch job processes and will queue the jobs. The result will be that the migration remains at a low percentage level because jobs are waiting in the queue to be executed.
• We therefore recommend that you avoid running too many batch jobs in parallel – this will not improve the performance of the data migration.

Further Recommendations:

• Do not start the migration of all objects in a migration project in parallel. There is a sequence for migrating objects (because of the dependencies between the migration objects). The sequence of the migration objects can be found in the migration object documentation.
• Parallelization is meant for migration objects for which high data volumes need to be migrated.
• We recommend preventing a situation where many people are working in parallel on separate migration projects, or where one user triggers multiple activities for several projects in same data migration context. Uploading multiple objects in parallel can flood the job queue. In our experience, it delivers better results to migrate one object with full number of jobs, complete the data migration and then start with the migration of the next object. Do this instead of distributing the number of free jobs and start several objects with high data volumes in parallel.
• Distributing the migration objects into different migration projects will not help regarding performance. On the contrary, you will lose the advantage of the central cross-object value mapping that the migration cockpit provides.
• Even if you assign a higher number of data transfer jobs that could run in parallel, the effect in the performance depends on the overall load of the system. If you have leveraged parallelization and have considered the recommendations above and still run into performance issues, you should know how many additional activities are going on in the system that also fills the process queue.
• If you experience performance problems, consider executing the data migration in a time period when the system has a low workload.

Kind Regards,
Zsombor Danka
SAP S/4HANA Cloud Product Support