Skip to Content
Technical Articles

And now, a real world case of parallel processing

Intro

 

Throughout my career as an ABAP developer I found very limited real world cases of parallel processing in ABAP. Truth is, 90% of the times, poor performance is coming from an unreasonable number of accesses to the database, normally in the order of several thousands of accesses, which is very easily fixable by changing the selection strategy, and the other 9.9% of the time, from not defining internal tables properly (as sorted or hashed tables when suitable), etc. Only very rarely there comes a situation where performance can be best improved by proper parallel processing, and I think that when these situations finally come around, the developer forgets that this is possible, it simply doesn’t come to the developer’s mind, or the developer is lazy to educate him or herself about it.

So, this blog post serves mainly to raise awareness once again to the fact that parallel processing is possible in SAP ABAP and actually quite easy, and maybe it will help someone getting it to work, even if I think the technical part is not very challenging. The “inspiration” for this came from the following link:

  • https://help.sap.com/doc/saphelp_nw70/7.0.31/en-US/fa/096e92543b11d1898e0000e8322d00/content.htm?no_cache=true

 

Coming to the topic at hand, at my company we bought and installed a piece of software called “Transport Profiler” (hopefully this is not confidential information) which checks all transport requests which are planned to be imported into production, to check for integrity, missing dependencies, or any issues that might prevent a smooth import. The software is sometimes a bit unstable, but it has become quite helpful, I can recommend it.

Anyway, as we have several productive systems, to make it practical for the operator who is checking the results, we created a program that could be executed from a central system (Solution Manager in our case) and trigger the API for this Transport Profiler for each productive system, gather the results and display them to the operator. Typically, this would be done sequentially, but it’s an excellent opportunity for parallel processing, so I took this opportunity.

 

Implementation

 

The first step is creating an RFC that you will need to propagate for each system, or for other cases, for example if you need to perform some expensive calculations that you’d like to do in parallel, you need to create a remote-enabled function module anyway, because that’s simply how this works.

 

Figure 1 – The function that will be called in parallel

 

I will simply copy/paste the coding now that I wrote to call this in parallel. As in my situation we only need 3 processes, and we pretty much always have enough processes available, my resource handling is atrocious. In another case, I created a proper loop which will wait for a few second and try a few times to get an available process. If it continues to fail for a long time, it gives up. You can get an “inspiration source” in the link above.

 

The idea is simple… call the function module to initialize the environment, whatever that means… and from this initialization you can get the number of processes available. This could be helpful if you want to guarantee, for example, that you don’t completely drain the resources of the system and you always leave enough margin for other activities. Following, you call the remote-enabled function module which is responsible for performing the calculations or whatever you want to do in parallel, specifying the subroutine which is responsible for gathering the results. You can use the “taskname” counter to identify this, or you can simply include some identification in the results of the function module, it’s a design decision, however you feel most comfortable. The code snippets should be enough for you to understand how this is working, if necessary.

If you need more information or something is not clear, feel free to ask.

 

FUNCTION ZSOL_CHECK_TP_PROD.
*"----------------------------------------------------------------------
*"*"Local Interface:
*"  IMPORTING
*"     VALUE(IT_TRKORRS) TYPE  TRKORRS
*"  EXPORTING
*"     VALUE(EV_ERROR_OCCURRED) TYPE  FLAG
*"     VALUE(ET_TP_RESULTS) TYPE  ZSOL_TP_PROD_RESULTS_TT
*"----------------------------------------------------------------------

  DATA lt_systems TYPE tt_system.
  DATA lv_taskname(4) TYPE n VALUE '0001'.
  DATA ls_check_results LIKE LINE OF gt_check_results.
  DATA lv_progress_text TYPE syst_ucomm.
  DATA lv_percentage TYPE i.

  PERFORM clear_global_variables.
  CLEAR:
        ev_error_occurred,
        et_tp_results.

*  Something to check?!
  CHECK it_trkorrs IS NOT INITIAL.

  PERFORM get_systems
    CHANGING
      lt_systems.

*  Any system to check?!
  CHECK lt_systems IS NOT INITIAL.

  PERFORM initialize_pbt_environment
    CHANGING ev_error_occurred.
  CHECK ev_error_occurred IS INITIAL.

  LOOP AT lt_systems INTO DATA(ls_system).

*    Save information of taskname and system
    CLEAR ls_check_results.
    ls_check_results-taskname = lv_taskname.
    ls_check_results-check_system = ls_system-system.
    ls_check_results-check_client = ls_system-client.
    INSERT ls_check_results INTO TABLE gt_check_results.

    PERFORM start_trans_profiler_remote
      USING
        lv_taskname
        ls_system
        it_trkorrs.

    ADD 1 TO lv_taskname.
    ADD 1 TO gv_snd_jobs.

  ENDLOOP.

  DO.

    IF gv_rcv_jobs >= gv_snd_jobs.
      EXIT.
    ENDIF.

    WAIT UP TO 1 SECONDS.

    CHECK gv_number_of_systems IS NOT INITIAL "CYA
      AND gv_rcv_jobs > 0.

    lv_percentage = gv_rcv_jobs * 100 / gv_number_of_systems.
    lv_progress_text = |Results received from { gv_rcv_jobs } servers.|.
    CALL FUNCTION 'SAPGUI_PROGRESS_INDICATOR'
      EXPORTING
        percentage = lv_percentage
        text       = lv_progress_text.

  ENDDO.

  et_tp_results = gt_check_results.

ENDFUNCTION.

Code snippet 1 – Code of the main function module

 

*----------------------------------------------------------------------*
***INCLUDE LZSOL_TPF01.
*----------------------------------------------------------------------*
*&---------------------------------------------------------------------*
*&      Form  GET_SYSTEMS
*&---------------------------------------------------------------------*
*       Get systems from customizing
*----------------------------------------------------------------------*
FORM get_systems
  CHANGING
    ct_systems TYPE tt_system.

  DATA lt_values TYPE TABLE OF zsol_attr_value.
  DATA ls_system LIKE LINE OF ct_systems.

  CLEAR ct_systems.

  SELECT attrval FROM zsol_setting
    INTO TABLE lt_values
    WHERE area = 'TRANSPORT_PROFILER'
      AND attrname = 'SYSTEMS_FOR_PROD_CHECK'.
  CHECK sy-subrc = 0.

  LOOP AT lt_values INTO DATA(value).

    CLEAR ls_system.

    SPLIT value AT '.' INTO ls_system-system ls_system-client.

    INSERT ls_system INTO TABLE ct_systems.

  ENDLOOP.

  gv_number_of_systems = lines( ct_systems ).

ENDFORM.
*&---------------------------------------------------------------------*
*&      Form  RETRIEVE_INFO
*&---------------------------------------------------------------------*
*       text
*----------------------------------------------------------------------*
FORM retrieve_info USING iv_taskname.

  DATA lv_contains_error TYPE flag.

  READ TABLE gt_check_results ASSIGNING FIELD-SYMBOL(<lf_results>)
    WITH KEY taskname = iv_taskname.
  CHECK sy-subrc = 0.

  RECEIVE RESULTS FROM FUNCTION 'ZSOL_RUN_TP_API'
    IMPORTING
      ev_contains_error = lv_contains_error
      ev_inspection_id  = <lf_results>-inspection_id
    EXCEPTIONS
      communication_failure = 1
      system_failure        = 2.
  IF sy-subrc <> 0.
    <lf_results>-check_result = 'E'.
    RETURN.
  ENDIF.

  ADD 1 TO gv_rcv_jobs.

  IF lv_contains_error = abap_true.
    <lf_results>-check_result = 'E'.
  ELSE.
    <lf_results>-check_result = abap_false.
  ENDIF.

ENDFORM.
*&---------------------------------------------------------------------*
*&      Form  INITIALIZE_PBT_ENVIRONMENT
*&---------------------------------------------------------------------*
*       Initialize the PBT environment, whatever it is
*----------------------------------------------------------------------*
FORM initialize_pbt_environment
  CHANGING
    pv_error_occurred TYPE flag.

  CALL FUNCTION 'SPBT_INITIALIZE'
    EXPORTING
      group_name                  = gc_server_group
    EXCEPTIONS
      pbt_env_already_initialized = 1
      OTHERS                      = 4.
  IF sy-subrc > 1.
    pv_error_occurred = abap_true.
  ENDIF.

ENDFORM.
*&---------------------------------------------------------------------*
*&      Form  START_TRANS_PROFILER_REMOTE
*&---------------------------------------------------------------------*
*       Start Transport Profiler in the remote system
*----------------------------------------------------------------------*
FORM start_trans_profiler_remote
  USING
    iv_taskname TYPE n
    is_system   TYPE ty_system
    it_trkorrs  TYPE trkorrs.

  DATA lv_excp_flag.      "Number of RESOURCE_FAILUREs
  DATA lv_progress_text TYPE syst_ucomm.

  zcl_utilities=>get_rfc_destination(
    EXPORTING
      im_tech_sys_name = is_system-system    " Extended System ID
      im_client        = is_system-client    " The ABAP Client
      im_rfc_type      = zcl_utilities=>co_rfc_type-trusted    " RFC Destination Type
    IMPORTING
      ex_rfc_dest      = DATA(ls_trusted_rfc_dest)    " Logical Destination (Specified in Function Call)
  ).

  PERFORM filter_irrelevant_trkorrs
    USING
      is_system
    CHANGING
      it_trkorrs.

  CALL FUNCTION 'ZSOL_RUN_TP_API'
    STARTING NEW TASK iv_taskname
    DESTINATION IN GROUP gc_server_group
    PERFORMING retrieve_info ON END OF TASK
    EXPORTING
      it_trkorrs            = it_trkorrs
      iv_rfcdest            = ls_trusted_rfc_dest
    EXCEPTIONS
      communication_failure = 1 "MESSAGE lv_msg
      system_failure        = 2 "MESSAGE lv_msg
      resource_failure      = 3.  "No work processes are
  IF sy-subrc <> 0.
    MESSAGE 'All servers currently busy' TYPE 'S'. "i837. "All servers currently busy.
    "Wait for replies to asynchronous RFC calls.  Each
    "reply should make a dialog work process available again.
    IF lv_excp_flag = space.
      lv_excp_flag = 'X'.
      WAIT UNTIL gv_rcv_jobs >= gv_snd_jobs UP TO '1' SECONDS.
    ELSE.
      WAIT UNTIL gv_rcv_jobs >= gv_snd_jobs UP TO '5' SECONDS.
      IF sy-subrc = 0.
        CLEAR lv_excp_flag.
      ELSE.  "No replies
        "Endless loop handling
        ...
      ENDIF.
    ENDIF.
  ENDIF.

  lv_progress_text = |Transport Profiler started on { gv_snd_jobs + 1 } servers.|.
  CALL FUNCTION 'SAPGUI_PROGRESS_INDICATOR'
    EXPORTING
      percentage = 1
      text       = lv_progress_text.

ENDFORM.
*&---------------------------------------------------------------------*
*&      Form  CLEAR_GLOBAL_VARIABLES
*&---------------------------------------------------------------------*
*       text
*----------------------------------------------------------------------*
FORM clear_global_variables.

  CLEAR:
        gv_snd_jobs,
        gv_rcv_jobs,
        gt_check_results,
        gv_number_of_systems.

ENDFORM.
*&---------------------------------------------------------------------*
*&      Form  FILTER_IRRELEVANT_TRKORRS
*&---------------------------------------------------------------------*
*       It could be that some transport requests were already imported
*       or don't exist. Let's not run transport profiler on them
*----------------------------------------------------------------------*
FORM filter_irrelevant_trkorrs
  USING
    is_system TYPE ty_system
  CHANGING
    ct_trkorrs TYPE trkorrs.

  DATA lt_trkorrs LIKE ct_trkorrs.

  LOOP AT ct_trkorrs INTO DATA(lv_trkorr).

    TRY.

        DATA(tr_order) = NEW zcl_sol_trorder( lv_trkorr ).

        DATA(cofile) = tr_order->read_cofile( ).

      CATCH zcx_bobj.
        CONTINUE.
    ENDTRY.

    DATA system LIKE LINE OF cofile-systems.
*    This is NOT very elegant
    CASE is_system-system.
      WHEN 'QG1'.
        READ TABLE cofile-systems INTO system " ASSIGNING FIELD-SYMBOL(<lfs_system>)
          WITH KEY systemid = 'PG1'.
      WHEN 'T19'.
        READ TABLE cofile-systems INTO system
          WITH KEY systemid = 'P19'.
      WHEN 'T33'.
        READ TABLE cofile-systems INTO system
          WITH KEY systemid = 'P33'.
    ENDCASE.
    CHECK sy-subrc = 0.
    "Has the transport request already been imported?
    READ TABLE system-steps TRANSPORTING NO FIELDS
      WITH KEY stepid = 'I'.
    CHECK sy-subrc <> 0.
*        No, so add it
    APPEND lv_trkorr TO lt_trkorrs.

  ENDLOOP.

  ct_trkorrs[] = lt_trkorrs[].

ENDFORM.

Code Snippet 2 – Subroutines

 

Conclusions

 

Parallel processing is not so hard to implement. In situations where users are waiting for a longer period, and there’s not much room for improvement in other ways, for example if we are talking about contacting several systems via RFC, or processing many sales orders which have to execute the pricing procedure each time, etc., parallel processing is an effective way of reducing the waiting times for the user, thus giving the illusion of improved performance, even though this should *not* be used as a replacement for proper database access strategies and proper coding.

10 Comments
You must be Logged on to comment or reply to a post.
  • Nice blog. I have used several times parallel processing and not because of a lot database fetches. It’s a bit to common to say 90%, I think it depends more on the complexity and the freedom to think out of the box to have more useable implemetations. The parallel processing to pay technical mistakes is making you pay a lot money more for the landscape and I’m pretty sure there will be a time when a smart person recognize it 😉

    ~Florian

    PS: Do you really use performs nowadays?

  • hmm….  performs?

    I have used parallel processing fairly often. One particular case was a very large and complex extractor to BW. On a forty dialog processes system, it would use 2/3 of those available. The problem with extractors that use parallel processing is that, due to the way they work, you can’t test them easily through RSA3.

    I have a class somewhere that makes the actual technical parts quite easy.

     

  • Something to remember;

    Using RFC processes to generate parallel work processes uses Dialog Work processes. This can get in the way of other Dialog processes like your users (and other standard processes that use parallel RFC processes like KKS1 (variance calculation), RVKRED* processing, your BW extracts, etc)

    You can use RFC load balancing to ensure that these RFC processes are restricted to a particular instance, or are limited in number if you’re a single instance system, so that you have at least dialog processes available to logon and monitor your paralellism …

    An alternative I have used is to submit the parallel processes as a series of jobs; these use the BG Work processes.

    https://help.sap.com/doc/abapdocu_751_index_htm/7.51/en-us/abapsubmit_via_job.htm

    hth

    • Great, thank you for your input.

      Is it trivial to get the results of the jobs you submit, or did you use it as a “fire and forget” application?

      In my case, I need the results to display to the user. Is it possible using the background jobs easily or would you have to use some interim database to get these results?

      Thank you,
      Bruno

      • Hi Bruno,

        I haven’t used it as anything m,ore than a “fire-and-forget” application. Given that parallel processing is used to minimize elapsed time of otherwise long running jobs or processes, I’ve not really considered using this approach for returning data to a users screen. The output is either written to spool, or to a table that can be viewed to examine progress (i.e. something a bit more informative than looking at SM37 and seeing whether the bg job has started, completed or abended).

        if you use a table to view the progress, make sure the table is keyed on something unique to each job; you don’t want to trigger a bunch of jobs that just end up waiting behind a DBMS or SAP lock !!

        For the progress table, in my implementations the background job created a key of process name (you could use start date / time, but we ended up using this for multiple processes) and two fields for the range (i.e. the start and end document number) being processed. The data field is the current document being processed.

        In most cases we did update the data field for each document processed, when it was critical that we knew where the job stopped if there was an ABEND, but the important thging is to make sure you update it once per COMMIT, so that the “progress table” reflects the state of tghe rest of the database

        The user can view the table via SE16N or a standalone transaction / program; the functional consultants can edit the table via SM30 for reruns etc

        HTH

        • Hi again Martin!

          Ah, I see. The case I am sharing is about code checking tools, and the users for this (normally ABAP developers) are waiting for the results “online”. Basically, at this time, we are running 3 independent tools (Code Inspector, Code Profiler and Transport Profiler), and the waiting times were something between a couple of seconds to a minute or two, depending on the size of the transport requests. This is why I invested the time to run these tools in parallel, so basically the waiting time will match the running time for the slowest tool, instead of being the sum of all three.

          Nonetheless, the idea you shared seems to be a better approach for background applications, where people can check the results asynchronously at a later time.

          Thanks again.

  • Nice blog, I use similar techniques relatively often.

    I have 2 issues though that I couldn’t find a solution for:

    1. detect when a child process dies. Best I found is a timeout. Did consider a watchdog in shared memory but seems overkill, and besides might not get updated for legitimate reasons (perhaps cl_gui_timer could fix that, not sure if it will interrupt a process)
    2. workprocesses and user sessions are used. The hard limit of how many you can use is quite small. No big deal for a single report doing this, but doesn’t scale

    I worked on a product that does similar things with jobs, but is complicated (need to store stuff in database for each and then join them) and works terribly for interactive stuff