Parallel processing of background jobs in SAP

[Former Member]

Parallel Processing reduces the time taken by usual background jobs in SAP by optimally utilizing the available background work processes in the server.

Why parallel processing?

• When the volume of data to process is too large in SAP, background jobs are created to run reports or interfaces. But as the business grows, the volume of data increases and such jobs will take many hours to complete.

• A business scenario for example - A job to post invoices in SAP system from legacy system can be executed only after creating the job to post customer master data is complete. Hence, there exists a dependency. In such a scenario, if the first job takes many hours, then it will also affect the start of the second job. Hence, there is a need to reduce the time taken by the background job to complete. This can be achieved by optimally utilizing the work process available in the application server.

Assume a scenario, where a new job has to be run, to complete a tedious task which normally takes 2 days to complete. When this job is running, there can be a number of background process available in the system, which is unutilized. If some of these free work processes (say 3 more) are allocated to work in the currently running job, the time taken by the main job can be reduced to 1/4th of its original time.

What is work process in SAP?

Before understanding this concept, the basic understanding on work process is required. Work processes are be available in the application server. Each work process will do a particular kind of task as its name indicate. The two work process which are required to understand this concept are,

• Dialog work processes will wait for user input and it executes the tasks  Ex: Creation of new Customer through foreground
• Background work processes (BGD WP) can execute the task without the dialog input from the user. Ex : Background jobs

Whenever a background job is created, a background work process (BGD WP) will be allocated by the system. Therefore, each background job created in SM37 will consume one background work process. The number of such work processes available in any system would be directly proportional to that system’s hardware capacity.

Concept

The idea here is to reduce the time taken by the job to complete and to optimally utilize the work processes available in the system.

Assume a scenario, where a new job has to be run to complete a tedious task which normally takes 2 days to complete. When this job is running, there can be a number of background process available in the system, which is unutilized. If some of these free work processes (say 3 more) are allocated to work in the currently running job, the time taken by the main job can be reduced to 1/4th of its original time.

Calculation using example:

Time taken by 1 job = 2 days

Assign 3 more jobs dynamically. Now 4 jobs will run parallel to do a single task. Hence time taken to complete the task would be only 0.5 days.

Flowchart explaining the normal and parallel processing is given below,

Technical Solution:

Consider an example scenario where the program has to create some 60,000 Customers by posting some 60,000 IDOCS.

The following are the steps involved in implementing parallel processing of background jobs.

Step 1: Calculate the number of background work process (BGD) that is available in the system using FM ‘TH_WPINFO’. Here, as shown in below example 5 BGD processes are available in status ‘WAITING’ which is available for consumption). These jobs are free to utilize.

                                                 

Step 2: Decide on the number of child jobs that you need for processing. As shown in the above screen shot, there are 5 BGD WPs available for consumption. When you execute the long running program in background, it will consume 1 BGD WP. Hence, there are 4 more remaining jobs,
out which we can use a few to help this main job i.e. the long running job. Say, if we utilize 25% of the remaining job (i.e. by creating one more child job), we can reduce the total time taken by the long running main job by 50%, since 2 jobs will be working on the same task (Assume the main job was supposed to post some 60,000 IDOCs, now since two jobs are actively working and posting IDOCs, each job will post only 30,000 IDOCs and both will complete this activity in more or less the same time).

Step 3: Assuming that for posting around 60,000 IDOCs, there should be around 60,000 entries in the internal table in the main program. This 60,000 entries should be divided based on the number of child jobs to be created. In the scenario explained in step 2, there were 2 jobs (1 main + 1 child job) which we allocated to work on this task. Hence, the internal table with 60,000 records needs to be dynamically divided into 2 internal tables. The first internal table with 30,000 records will be processed by the child job, the remaining will be processed by the main job.

Step 4: Have the main processing logic in a separate report program (in this case the posting of IDOCs logic will be there in a separate program). The first internal table with 30,000 records needs to be passed to this report which will do the posting of IDOC alone. The passing can be either through report selection screen or through intermediate file in AL11 path (based on the volume of records that needs to be passed and
developer's convenience). Follow the below approach to call the report and for creating a child job.

                                                     

Step 5: Once the 30,000 records are moved to child job for processing, start processing the remaining records in main job itself. After completing the processing of 30,000 records, check if the child job has completed the processing using FM ‘BP_JOB_STATUS_GET’. We can also, get the status of the records processed by the child job using FM ‘BP_JOBLOG_READ’. Based on this read further necessary action if needed, can be taken.

Step 6: Consolidate the status from child job and main job and display the status as final message, which can be read from main job’s log.

The above scenario was explained in such a way that we utilize one extra job than normal. But in case if we can afford 4 child jobs, the totally 5 jobs will work together to post the IDOCs. This will considerably reduce the total time taken and also, it would result in optimal utilization of existing hardware capacity as well.

Pros

• Reduces the total time taken to complete a task considerably, by optimally utilizing systems hardware capacity
• Since, external switches are available (through application table) to control the number of BGD WPs a job can use, it provides control to
  the operation team to prioritize and complete the important jobs quickly.
• No additional hardware is added to make jobs faster. However, addition of hardware capacity can give more BGD WPs to the system which in turn will help in parallel processing.

Cons

• Of course this process has its limitations. But care must be taken in allocating WPs to any job. Over prioritizing can starve other jobs
  and in worst case may also affect the creation of new jobs.
• Knowledge of available BGD WPs in system will become mandatory for operations team who till now might not have considered this
  before running the jobs.

Suggestions to improve this document is most welcome.