It is a common programming pattern to fill a character string in iterative steps, either using the concatenation operator && or character string templates |…|:

DO|WHILE|LOOP|SELECT …

  str = str && …

  str = |{ str }…|.

ENDDO|END|ENDWHILE|ENDSELECT …

Bad News

Both assignments have a string expression as an RHS (right hand side) . What does that mean? As a rule, when assigning an expression, a temporary or intermediate result is created that must be copied to the LHS (left hand side). Since the length of the intermediate result increases with each loop pass and there is a copy operation for each loop pass, the runtime dependency from the number of loop passes is quadratic. Not good.

Good News

Since this is a well known fact, there is an internal optimization for all concatenations that look as follows:

str = str && dobj1 && dobj2 && … .

str = |{ str }…{ dobj1 [format_options] }…{ dobj2 [format_options] }…|.

As long

  • as the target string str occurs at the RHS only once and as the leftmost operator,
  • and as long as there are no formatting options for str ,
  • and as long as there are no other expressions or function calls involved in the RHS

no intermediate result is created but the characters are concatenated directly to the target string str. This prevents the quadratic dependency of the runtime from the number of iterations. The same is true for the CONCATENATE statement.


With other words, no problems in writing something like this:


DATA(html) = `<html><body><table border=1>`.

DO … TIMES.

  html = |{ html }<tr><td>{ sy-index }</td></tr>|.

ENDDO.

html = html && `</table></body></html>`.


There is a simple data object sy-index concatenated to a target string html and the optimization takes place.


Bad News


The optimization only takes place for the simple cases above!


And this is the trap.


You loose the optimization, if you loop over concatenations that look as follows:


str = str && … && meth( … ) && … .

str = str && … && str && … .


str = |{ str }…{ expr( … ) }…|.


str = |{ str format_options }…|.


str = |{ str }…{ str }…|.


As long as you do such a concatenation outside of loops, no problem. But inside of loops and for a large number of iterations you can quickly experience really large runtimes.


There is in fact a problem, writing something looking as harmless like this:


DATA(html) = `<html><body><table border=1>`.

DO … TIMES.

  html = |{ html }<tr><td>{

            CONV string( ipow( base = sy-index exp = 2 ) )

            }</td></tr>|.

ENDDO.

html = html && `</table></body></html>`.

Concatenating the ipow expression to str breaks the optimization.


Good News


Now that you know the problem, you can easily circumvent it. Normally we use expressions to get rid of helper variables. But in connection with loops helper variables can be a good thing. You already know that you use  them for calculating results that are constant within a loop. Now you learn, that you should also use them for concatenating expressions or functions to strings:

DATA(html) = `<html><body><table border=1>`.

DATA square type string.

DO ... TIMES.

   square = ipow( base = syindex exp = 2 ).

   html = |{ html }<tr><td>{ square }</td></tr>|.

ENDDO.

html = html && `</table></body></html>`.

By assigning the ipow expression to helper variable square and concatenating that to str,  the optimization takes place again. Try it yourself and see what happens for large numbers of iterations with and without optimization!

Last but not least, what is said above for loops realized by control statements is also true for FOR loops in expressions:

DATA(result) =

  REDUCE string( INIT s = “

                 FOR i = 1 UNTIL i > …

                 NEXT s = s && CONV string( i ) ).

vs.

DATA(result) =

  REDUCE string( INIT s = “

                 FOR i = 1 UNTIL i > …

                 LET num = CONV string( i ) IN

                 NEXT s = s && num ).

Only by using the helper variable num declared in a LET expression the optimization is enabled. The example without helper variable shows a quadratic increase of runtime with the number of iterations.

The last example is directly taken from the recent documentation. But have you been aware of that?

To report this post you need to login first.

16 Comments

You must be Logged on to comment or reply to a post.

  1. Suhas Saha

    Super content Horst! Thanks for highlighting this trap.

    But have you been aware of that?

    Not in my wildest dreams. On our ABAP740(SP12) system, it gives a performance improvement of almost 500 😯

    2016-08-15 10_04_52-Laufzeitanalyse_ Messung anzeigen.jpg

    (1) 
  2. Christian Dr. Drumm

    Hi Horst,

    thanks for this post. Good to know. I wouldn’t have expected any performance traps in the string concatenation. For me this breaks the POLS (https://de.wikipedia.org/wiki/Principle_of_Least_Surprise) 😯

    However, I’m wondering why the ABAP runtime doesn’t handle those cases

    DATA(html) = `<html><body><table border=1>`.

    DO … TIMES.

      html = |{ html }<tr><td>{

                CONV string( ipow( base = sy-index exp = 2 ) )

                }</td></tr>|.

    ENDDO.

    html = html && `</table></body></html>`.

    My naive assumption would be that it should be pretty easy to automatically perform the optimization introduced by the additional variable. 😕

    Christian

    (0) 
    1. Horst Keller Post author

      Yep, it’s a pity. I wrote the blog despite the fact that it will leave a bad impression. But better to know the flaws than hiding them …

      (1) 
      1. Shai Sinai

        Is there any special reason that the internal optimization cannot handle expressions the same as variables?

        And another case: When target string str occurs at the RHS only once and as the rightest operator?

        (0) 
        1. Horst Keller Post author

          Is there any special reason that the internal optimization cannot handle expressions the same as variables?


          Everything that potentially depends on LHS leads to a copy of LHS before using it. ABAP does not look into the expression.


          as the rightest operator?


          Only concatenations at the right end of str. str is listed leftmost.

          (0) 
  3. Aasim Khan

    I’m on SAP 750/003.

    While trying out this performance trap in my system, I wondered if fetching the data from AMDP and concatenating at DB level would have any performance improvement. But I was shocked to see the results.

    And here’s the SAT screenshot.

    amdp.JPG

    Below is the program:

    REPORT ztest_sm.

    PARAMETERS: p_vbeln TYPE vbeln_va.

    TYPES:

      BEGIN OF ty_salesord,

        vbeln TYPE vbeln_va,

        posnr TYPE posnr_va,

        matnr TYPE matnr,

      END OF ty_salesord,

      tt_salesord TYPE STANDARD TABLE OF ty_salesord.

    DATA:

      lv_oldstr TYPE string,

      lv_newstr TYPE string,

      lv_finstr TYPE string,

      lv_matnr  TYPE matnr,

      li_data   TYPE tt_salesord.

    START-OF-SELECTION.

      PERFORM f_get_data CHANGING li_data.

      PERFORM f_old_concat USING li_data.

      PERFORM f_new_concat USING li_data.

      PERFORM f_imp_concat USING li_data.

      PERFORM f_amd_concat USING li_data.

    *&———————————————————————*

    *&      Form  F_GET_DATA

    *&———————————————————————*

    *       text

    *———————————————————————-*

    *      <–P_LI_DATA  text

    *———————————————————————-*

    FORM f_get_data  CHANGING fp_li_data TYPE tt_salesord.

      SELECT FROM vbak AS a

        JOIN vbap AS b

          ON b~vbeln = a~vbeln

      FIELDS a~vbeln,

             b~posnr,

             b~matnr

       WHERE a~vbeln = @p_vbeln

       GROUP BY a~vbeln,

                b~posnr,

                b~matnr

        INTO TABLE @fp_li_data.

    ENDFORM.

    *&———————————————————————*

    *&      Form  F_OLD_CONCAT

    *&———————————————————————*

    *       text

    *———————————————————————-*

    *      –>P_LO_DREF  text

    *———————————————————————-*

    FORM f_old_concat  USING    fp_li_data TYPE tt_salesord.

      WRITE:/ |Convetional concatenation|.

      LOOP AT fp_li_data INTO DATA(lw_salesord).

        CONCATENATE lv_oldstr

                    lw_salesord-matnr

               INTO lv_oldstr

          SEPARATED BY space.

      ENDLOOP.

      WRITE:/ lv_oldstr.

    ENDFORM.

    *&———————————————————————*

    *&      Form  F_NEW_CONCAT

    *&———————————————————————*

    *       text

    *———————————————————————-*

    *      –>P_LI_DATA  text

    *———————————————————————-*

    FORM f_new_concat  USING    fp_li_data TYPE tt_salesord.

      WRITE:/ |New concatenation|.

      LOOP AT fp_li_data INTO DATA(lw_salesord_n).

        lv_newstr = lv_newstr &&

                    | |       &&

                    lw_salesord_n-matnr.

      ENDLOOP.

      WRITE:/ lv_newstr.

    ENDFORM.

    *&———————————————————————*

    *&      Form  F_IMP_CONCAT

    *&———————————————————————*

    *       text

    *———————————————————————-*

    *      –>P_LI_DATA  text

    *———————————————————————-*

    FORM f_imp_concat  USING    fp_li_data TYPE tt_salesord.

      WRITE:/ |Fine tuned concatenation|.

      LOOP AT fp_li_data INTO DATA(lw_salesord_f).

        lv_matnr  = lw_salesord_f-matnr.

        lv_finstr = lv_finstr &&

                    | |       &&

                    lv_matnr.

      ENDLOOP.

      WRITE:/ lv_finstr.

    ENDFORM.

    *&———————————————————————*

    *&      Form  F_AMD_CONCAT

    *&———————————————————————*

    *       text

    *———————————————————————-*

    *      –>P_LI_DATA  text

    *———————————————————————-*

    FORM f_amd_concat  USING    fp_li_data TYPE tt_salesord.

      WRITE:/ |AMDP concatenation|.

      NEW zcl_collect_amdp( )->meth_concat_material(

        EXPORTING

          im_mandt = sy-mandt

          im_vbeln = p_vbeln

        IMPORTING

          et_string = DATA(li_concat)

      ).

      DATA(lv_amdstr) = VALUE string( li_concat[ 1 ]-str OPTIONAL ).

      WRITE:/ lv_amdstr.

    ENDFORM.

    And here’s the AMDP.

    method.JPG

    I’m still a newbie and not quite aware of the intricacies involved in AMDP.

    Thanks!

    PS: Apologies for the text formatting as I’m not aware of how to paste programs with ABAP fonts on SCN.

    (1) 
    1. Horst Keller Post author

      Hi Aasim,

      • Did you measure several times? When using an AMDP procedure for the first time, it must be created on the DB.
      • For the sake of completeness, you might also measure the concatenation in Open SQL with && or Open SQL function CONCAT or in ABAP CDS with CDS function CONCAT.
      • In order to check if it is an AMDP or a SQLScript issue, you can try to call the procedure with ADBC or EXEC SQL. If it is slow there too, it is the DB itself.

      Horst

      (0) 
      1. Aasim Khan
        • Did you measure several times? When using an AMDP procedure for the first time, it must be created on the DB.

        Yes, I did. The SAT result shows the performance time in improving trend, however, as compared to OpenSQL, AMDP still took a lot of time.

        • For the sake of completeness, you might also measure the concatenation in Open SQL with && or Open SQL function CONCAT or in ABAP CDS with CDS function CONCAT.

        As suggested, I used CONCAT function, && operator in OpenSQL and CONCAT in CDS and AMDP. AMDP take more time as shown below:

        /wp-content/uploads/2016/08/1_1022397.jpg

        /wp-content/uploads/2016/08/2_1022398.jpg
        /wp-content/uploads/2016/08/3_1022399.jpg

        /wp-content/uploads/2016/08/4_1022400.jpg

        • In order to check if it is an AMDP or a SQLScript issue, you can try to call the procedure with ADBC or EXEC SQL. If it is slow there too, it is the DB itself.

        SAT for ADBC – Same query that I wrote in AMDP which performed better but openSQL still gave good results.

        5.JPG

        Thank you for your inputs!

        I’ll open a thread soon on this topic

        (0) 
        1. Horst Keller Post author

          Since you are so active  …

          Measure each access in DO loops using GET RUN TIME FIELD and calculate the mean or the minimal run times. I still can’t believe the bad results for AMDP.

          Horst

          (0) 
  4. Laurin Kirbach

    Nice post, Horst. Always very informative to read your articles!

    Maybe a little offtopic but also regarding LET… Wouldn’t it be a nice idea to implement a statement like C#’s using for opening a local context? I’m often facing the scenario, that I have to pass a variable to several consecutive operations (e.g. handling XML nodes in loops) and explicitly clean up my workspace. Could be a bit neater in modern ABAP… 😉

    For example:

         WITH x = expr IN

        

         ENDWITH.

         * x is not available anymore

    Honestly, I haven’t checked whether it is possible to create such a function without wasting too much performance.


    Kind regards

    (0) 
    1. Horst Keller Post author

      If I understand it correctly, it is about local variables for a context opened by WITH.

      Unfortunately, I’m afraid I will never live to see that coming. “Modern ABAP”, as nice as it is, is just a doily (can I say that?) placed over the old ABAP.

      B.t.w., there will be WITH. – ENDWITH. in an upcoming release. But in Open SQL …

      (0) 
  5. Loyd Enochs

    Horst, thank you for enlightening us to this!  Like many others have commented, it never occured to me that this was an issue.

    Thank you for your research and your insights.

    (and fig leaf also works for doily  😆 )

    (0) 
    1. Horst Keller Post author

      and fig leaf also works for doily

      To be fair, it is more than that. But unfortunately it is out of question to fundamentally change the underlying engine itself. Such changes would be incompatible and for that, there are simply no stakeholders.

      (0) 
      1. Loyd Enochs

        Of course, and I completely understand why there are no stakeholders for throwing away “old ABAP”, introducing a new language and forcing every customer on the planet to rewrite all their Z_ programs.  Not to mention rewritting and retesting every SAP standard program.

        The liability insurance alone would rival the GDP of most nations 🙂 .

        (0) 

Leave a Reply