Skip to Content
Technical Articles
Author's profile photo Dinesh Naidu

Parallel Processing for Huge Number of Records In Internal Table

In some situations, Processing huge number of records in Internal table could take a lot of time. This in turn reduces performance of the entire system.This is where the concept of Parallel Processing comes into the picture. This Blog explains how to use Parallel Processing technique to Process huge volume of data in a much efficient manner.

Basically we are going to split the entire data into small packets and then process them individually by using Function Modules and Work Processes.

1) Create a Custom RFC Function Module

 

*Import Parameter IM_TAB type CCRCTT_KNA1 (search table type of any table using “where used list” icon/button)

*Export Parameter EX_TAB type CCRCTT_KNA1 to export data through Internal Table

*In source code we will just push data from IM_TAB to EX_TAB (You can write your own logic here based on your requirements)

 

2) Create a Z/Y Program.

   *Top_Include

REPORT zdev_paralell_procs.

DATA : lt_kna1 TYPE TABLE OF kna1, 
       lt_kna1_temp TYPE TABLE OF kna1,
       lt_kna1_final TYPE TABLE OF kna1,
       ls_kna1_final TYPE kna1,
       lv_free_wp TYPE i,
       lv_sucess TYPE i,
       lv_task(2) TYPE c.

 

*Main Program.

INCLUDE zdev_paralell_procs_top .     " Top Include
INCLUDE zdev_paralell_procs_f01 .     " FORM Include

START-OF-SELECTION.
  "Get_Data(Customers) from KNA1...
  SELECT * FROM kna1 UP TO 19000 ROWS INTO TABLE lt_kna1.    "You Can Fetch As Much You Whant

  "FM to get active work process...  (Go To T-code SM50 and SM51 to View and Process "Work Process"
  CALL FUNCTION 'SPBT_INITIALIZE'
    IMPORTING
      free_pbt_wps                   = lv_free_wp            "It will give the Number of Free Work Process
    EXCEPTIONS
      invalid_group_name             = 1
      internal_error                 = 2
      pbt_env_already_initialized    = 3
      currently_no_resources_avail   = 4
      no_pbt_resources_found         = 5
      cant_init_different_pbt_groups = 6
      OTHERS                         = 7.
  IF sy-subrc <> 0.
    MESSAGE text-001 TYPE 'S' DISPLAY LIKE 'E'.              "Message If No Free Work Process
  ENDIF.
  """Acess Half of the Work Process...                       
  lv_free_wp = lv_free_wp / 2.
  ""Total records in Internal Table...
*******@Dont'n Mind About "Inline Declaration :-)******
  DESCRIBE TABLE lt_kna1 LINES DATA(lv_records).
  "Split the Volume of Data divide by Free Work Process...
  DATA(lv_split) = lv_records / lv_free_wp.
  DATA(lv_split_temp) = lv_split.                            "Temporary Store
  "missed records...
"""11/2 = 5.5 It will take 5. So, will get missed record
  DATA(lv_missed) = lv_split * lv_free_wp.

  IF lv_records GT lv_free_wp.              "Only if the records greater than work process
    DO .
      IF lt_kna1 IS INITIAL.
        EXIT.
      ELSE.
        lv_task = sy-index.                                  "Storing a uniqe number for a task in FM        
        CONCATENATE 'D' lv_task INTO lv_task.                "uniqe number like 'D1'
        CONDENSE lv_task NO-GAPS.
        IF sy-index = 1 AND lv_records LT lv_missed.         "If total records less than lv_missed 
          lv_split = lv_split + ( lv_missed - lv_records ).  "difference add to lv_split
        ELSEIF sy-index = 1 AND lv_records GT lv_missed.     "If lv_missed is less than lv_records
          lv_split = lv_split + ( lv_records - lv_missed ).  "difference add to lv_split
        ELSE.
          lv_split = lv_split_temp.                          "store old value from temp.
        ENDIF.
        APPEND LINES OF lt_kna1 FROM 1 TO lv_split TO lt_kna1_temp.    "push part of the data to IT.
        CALL FUNCTION 'ZDEV_PARALELL_PROCS' STARTING NEW TASK lv_task DESTINATION IN GROUP DEFAULT
          PERFORMING get_data ON END OF TASK                 "With form routin 'Get_Data' using IT
          EXPORTING
            im_tab = lt_kna1_temp.                           "pass internal table
        IF sy-subrc = 0.
          CLEAR lt_kna1_temp.                                "clear temp IT
          DELETE lt_kna1 FROM 1 TO lv_split.                 "Clear part of data from IT KNA1
        ENDIF.
      ENDIF.
    ENDDO.
    ""Wait until processed data get..
    WAIT UNTIL lv_sucess = lv_free_wp.                     "Wait until every Work Process get processed
  ELSE.                                               
    CALL FUNCTION 'ZDEV_PARALELL_PROCS'
      EXPORTING
        im_tab = lt_kna1_temp
      IMPORTING
        ex_tab = lt_kna1_final.
  ENDIF.

 

*Form_Include

*&  Include           ZDEV_PARALELL_PROCS_F01
*&      Form  GET_DATA
*&---------------------------------------------------------------------*
FORM get_data USING lv_task.
  RECEIVE RESULTS FROM FUNCTION 'ZDEV_PARALELL_PROCS'
  IMPORTING                                       "Data getting back from FM
    ex_tab = lt_kna1_temp.
  lv_sucess = lv_sucess + 1.                      "Store total WP excecuted times value
  APPEND LINES OF lt_kna1_temp TO lt_kna1_final.  "Store data to final IT   
  CLEAR lt_kna1_temp.
ENDFORM.
NOTE : You can use TIMESTAMP to see the duration of execution. 


CLICK HERE to know more about WORK PROCESS SM50.

This Technique is useful when we are processing large number of records, but when are processing a considerably lesser number of records, then this technique is not really recommended. If there are any changes that need to made the document, kindly put them in the comments below, I will edit the Blog accordingly.

Assigned tags

      12 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Matthias Bucher
      Matthias Bucher

      Hi Dinesh,

      maintain a RFC Server Group "ggg" with RZ12 and put as many appl. Server and WPs into it

      CALL FUNCTION ... DESTINATION IN GROUP ggg

      Split the input in small packages ...

      • you may query the RFC Group for available resources once at process begin
        (but this may be useless if another process uses same RFC group just 1 sec. later as you queried the free ressources)
      • you may just use a fixed value which is small enough to allocate all WPs
        • try 1 and measure the RFC admin overhead
        • try lines(Input) and measure it
        • try lines(Input) / avail WPs and measure it

      Then let SAP manage the execution of RFCs in this Group without having fear that your RFC will flood the system up to rdisp/ max RFC quota.

      Best regards,
      Matthias

      Author's profile photo Jelena Perfiljeva
      Jelena Perfiljeva

      I wish this comment was a blog. 🙂

      Author's profile photo Dinesh Naidu
      Dinesh Naidu
      Blog Post Author

      Hey Matthias Bucher,

      Firstly, "Apology for delay in response"

      And thanks for warm suggestion to mentioned Inputs.

      Pleasure to get your reviews on my blogs 🙂

      Author's profile photo Michelle Crapo
      Michelle Crapo

      Nice job on a first time blog.  I like examples and the description of why the program was needed.

      My comment is the same as above.  My big concern is that you can bring a system to a standstill without letting SAP manage the execution.  You might not see it testing on development or quality if you don't have that many users on the system.

      Additional Information:

      https://help.sap.com/doc/saphelp_nw70/7.0.31/en-US/fa/096e92543b11d1898e0000e8322d00/content.htm?no_cache=true

      Author's profile photo Dinesh Naidu
      Dinesh Naidu
      Blog Post Author

      Hello Michelle Crapo,

      Sorry for late response and I glad that you liked my first blog.

      More to explore 🙂

      Author's profile photo Joe Valliparampil
      Joe Valliparampil

      Nice.

      There are a few demo programs in the SAP system, search with ‘SPTA*’ and you will see them. These use the ‘SPTA’ function modules.

      Author's profile photo Dinesh Naidu
      Dinesh Naidu
      Blog Post Author

      Hey Joe, thanks for the helpful inputs 😉

      Author's profile photo Matthew Billingham
      Matthew Billingham

      Can you explain what you means

      Processing huge number of records in Internal table could take a lot of time. This in turn puts immense stress on database

      Why should internal table processing affect the database?

       

      There is a bad error in your code. You've defined variables with the L prefix, which in most places is used to indicate "local". Your variables are, however, not local.

      You might want to check out the LINES function, rather than using DESCRIBE. FORMs are obsolete - better to use classes and methods. There is an addition to CALL FUNCTION that will trigger a method call rather than a PEFORM.

      Author's profile photo Dinesh Naidu
      Dinesh Naidu
      Blog Post Author

      Hello Matthew,

      Thank you for your feedback. This is my first blog and this kind of mistakes happens. So, as a associate consultant we guy's try to learn better things from senior's like you. Only pointing out the mistakes will not entertainment us 🙂

      Author's profile photo Matthew Billingham
      Matthew Billingham

      I'm trying to help you, not entertain you. And yes, shock you into realising you could do things better in order to motivate you to make changes. I was hoping to enter into a dialogue, so I reiterate:

      Can you explain what you means

      Processing huge number of records in Internal table could take a lot of time. This in turn puts immense stress on database

      Why should internal table processing affect the database?

      As far as learning, I've given you learning points - you just need to implement them

      There is a bad error in your code. You’ve defined variables with the L prefix, which in most places is used to indicate “local”. Your variables are, however, not local.

      So fix it. Remove the l_ prefixes. Replace them with g_

      You might want to check out the LINES function, rather than using DESCRIBE. FORMs are obsolete – better to use classes and methods. There is an addition to CALL FUNCTION that will trigger a method call rather than a PERFORM.

      1. Read the documentation for LINES and use it instead of DESCRIBE

      2. Replace all your FORMs with a class and methods.

      3. Change your CALL FUNCTION so that when the FM is finished, it calls a method. You can find out exactly how by reading the keyword functionality.

      Do this, and you'll have learned something and overall you'll have improved as a developer. I may even add a like to your blog. Sorry I don't sugar coat my criticism - but it is constructive.

       

      Author's profile photo Dinesh Naidu
      Dinesh Naidu
      Blog Post Author

      Hey Matthew,

      Those are some "Words or Sentence" Mistakes. And yes I will continuously try to improve myself.

      Thanks for your helpful points. Keep helping/motivating others !!! 🙂

      Author's profile photo Matthew Billingham
      Matthew Billingham

      I find it sad that you're apparently not motivated sufficiently to update your blog to implement the other suggestions. But thank-you for correcting your opening paragraph.