Skip to Content
Actions

Releasing Internal Table Memory

February 18, 2015 | 2,129 Views |
Follow

It is a well known fact, that you release the memory occupied by an internal table using either CLEAR or FREE, where FREE releases also the initial memory area. You normally use CLEAR, if you want to reuse the table and you use FREE, if you  really want to get rid of it and don’t want to refill it later on. Assigning an initial internal table to a filled internal table does also release the target table’s memory in the same way as CLEAR does.

But last week a colleague pointed out to me that it is not such a well known fact that deleting lines of internal tables with DELETE normally does not release the memory occupied by the deleted lines. Instead, there seem to be people deleting lines of internal tables in order to release memory. Therefore as a rule:

Deleting lines of an internal table using the DELETE stament does not release the table’s memory.

For an internal table that was filled and where all lines are deleted using the DELETE statement the predicate IS INITIAL in fact is true. But the internal table is only initial regarding the number of lines but not regarding the memory occupied. You can check that easily using the memory analysis tools of the ABAP debugger.

So far so good. For releasing the memory of an internal table you use CLEAR or FREE and you do not simply DELETE all lines.

But what about the use case, where you want to delete almost all lines from a big internal table and to keep the rest? After deleting, the internal table occupies much more memory than needed for its actual lines. If memory consumption is critical, you might want to get rid of the superfluous memory occupied by such an internal table. How to do that?

Spontaneous idea:

DELETE almost_all_lines_of_itab.

DATA buffer_tab LIKE itab.
buffer_tab = itab.
CLEAR itab.
itab =  buffer_tab.
CLEAR buffer_tab.

Bad idea! Check it in the ABAP Debugger. Due to table sharing, after assigning itab to buffer_tab, buffer_tab is pointing to the same memory area as itab. Assigning buffer_tab back to itab after clearing itab is simply an effectles roundtrip and you gain nothing.

Improved idea:

DELETE almost_all_lines_of_itab.

DATA buffer_tab LIKE itab.
buffer_tab = VALUE #( ( LINES OF itab ) ).
CLEAR itab.
itab =  buffer_tab.
CLEAR buffer_tab.

Now it works! Instead of copying itab to buffer_tab you can transfer the lines of itab sequentially to the initial target table and the memory is not shared. Before 7.40, SP08 you have to use INSERT LINES OF itab INTO TABLE buffer_tab instead of the VALUE expression, of course.

What also works for the use case is:

DELETE almost_all_lines_of_itab.

DATA buffer_tab LIKE itab.
buffer_tab = itab.
INSERT dummy_line INTO TABLE buffer_tab.
DELETE buffer_tab WHERE table_line = dummy_line.
CLEAR itab.
itab =  buffer_tab.
CLEAR buffer_tab.

By inserting a dummy line into buffer_tab and deleting it again, the table sharing is canceled and buffer_tab is built from scratch (but only, if it needs considerably less memory than before; otherwise it is copied and nothing is gained again).

Ingenious minds might also find the following ways:

DELETE almost_all_lines_of_itab.

DATA buffer_string TYPE xstring.
EXPORT itab TO DATA BUFFER buffer_string.
CLEAR itab.
IMPORT itab FROM DATA BUFFER buffer_string.
CLEAR buffer_string.

or even

DELETE almost_all_lines_of_itab.

CALL TRANSFORMATION id SOURCE itab = itab
                       RESULT XML DATA(buffer_string).
CLEAR itab.
CALL TRANSFORMATION id SOURCE XML buffer_string
                       RESULT itab = itab.

CLEAR buffer_string.

Yes, those work too, but put some GET RUN TIME FIELD statements around them to see that those are not the best ideas …

To report this post you need to login first.

52 Comments

You must be Logged on to comment or reply to a post.

  1. Amit Sawant

    Nice blog, this would surely improve my programming skills.

    cant we get a conditional CLEAR statement… like CLEAR WHERE  🙂

    that would help a lot

    (0) 
    1. Horst Keller Post author

      cant we get a conditional CLEAR statement… like CLEAR WHERE

      In fact, I discussed that with the developers already. One might also consider a statement RELEASE MEMORY or something like that. Also implicit mechanisms for releasing superfluous memory are under consideration. But for the time being, we have to live with the workarounds.

      (0) 
      1. Fred Verheul

        Horst Keller wrote: … Also implicit mechanisms for releasing superfluous memory are under consideration.

        Indeed, in 2015 I would expect not to have to deal with this anymore as a programmer.

        Still a very nice blog post though! Thanks for explaining.

        Fred

        (0) 
      1. Nikolay Evstigneev

        That’s very interesting. I thought they are stored as an array so that if you know the address of itab’s beginning, table length and the number of lines you can say itab’s end in memory.

        Really cool post 🙂

        (0) 
        1. Horst Keller Post author

          thought they are stored as an array so that if you know the address of itab’s beginning, table length and the number of lines you can say itab’s end in memory.

          No that’s how strings are handled. But internal table are managed either by their table index or the hash engine depending on the table type. When adressing an internal table line either the index or the hash management points to the real location. So, if you delete lines even from an index table there is no physical memory table that is shrunk, but the index is reorganized.

          (0) 
          1. Nikolay Evstigneev

            Frankly speaking, I haven’t even thought about it until today, it seemed to be so simple.

            Can you say the name of the mechanism that “spreads” the tables over allocated memory? Couldn’t google it at once, I’d like to read more about how it works.

            (0) 
  4. Suhas Saha

    Hi Horst,

    How would you do it in a productive code? Would you write this code inside a macro? For e.g., mac_save_itab_memory 😉

    BR,

    Suhas

    (0) 
    1. Horst Keller Post author

      LOL …

      But seriously one might ask why there isn’t a wrapper for this. On the other hand, just look how long nobody never ever has cared for that. Maybe memory isn’t such important any more.

      (0) 
      1. Uwe Fetzer

        That.

        And IMHO the Garbage Collector should be responsible to release unused space in this case. Would be difficult to define the correct rules for that however (memory consumption vs. performance).

        (0) 
          1. Uwe Fetzer

            Releasing unused memory in the stack isn’t called Garbage Collection? Learned something new (again). My education is too long ago it seams…

            (0) 
            1. Horst Keller Post author

              Releasing unused memory in the stack isn’t called Garbage Collection?

              Didn’t say so. I onyl said what the ABAP Garbage Collector in its current state does.

              (0) 
      2. Loyd Enochs

        Horst said: “Maybe memory isn’t such important any more.”

        It used to be important, and I’m willing to bet it will be again when Code-to-Data is misunderstood and everything gets pushed to the database servers 😉

        I used to teach ABAP back in 3.x and 4.x days, and the CLEAR v REFRESH v FREE was a 15 minute topic.

        (0) 
  5. Nick U

    Suppose we have the following abap code which consists of two small procedures:

    form form1.

         perform form2.

    endform.

    form form2.

         data itab type standard table of …

         …populate itab from db tables…

         …process itab lines (but there is no “clear itab” or “free itab” statements)…

    endform.

    Question is: will the memory occupied by itab be freed when execution flow exits from form2 (and, on the next step, from form1)? Do I need to manually call “free itab” in such cases?

    Thank you for useful blog.

    (0) 
  6. Eric Nilson

    So if one uses statement clear internal_table.  Then one should also after the internal_table is not to be used again, have the statement free internal_table.

    Correct?

    Also I am usually using refresh internal_table.   Does this free any memory?

    (0) 
    1. Horst Keller Post author

      You don’t have to use FREE after CLEAR. You can use FREE directly. But normally FREE is not necessary, only if you desperatly seek for additional memory.

      REFRESH itab is exactly the same as CLEAR itab[]. It adresses always the table body and never the obsolete header line. Since we don’t work with header lines any more (duh!). You need neither REFRESH nor itab[] any more. Simply CLEAR is OK.

      (0) 
    1. Horst Keller Post author

      I’m with SAP for 20 years now, always writing ABAP documentation, and added that stuff only recently for the next upcoming release, also embarrassing. And that’s not the only thing,  😕 …

      (0) 
  12. Iurii Slobodchikov

    Hi, Horst! Is there an event in SAP when Garbage Collector end its work?

    I have a class which collects references to data in a static table and when this data is released reference becomes incorrect and the dump is occured.

    I want to subscribe to this event and delete table lines with an inappropriate references.

    (0) 
    1. Horst Keller Post author

      Nothing that I know of. And in fact it is not wanted too. The garbage collector should be transparent for the normal ABAP developer.

      (0) 
  14. Sandra Rossi

    For fun, below, 3 procedures to shrink an internal table, based on Horst algorithm ; there’s only the “dummy line” the developer has to worry about (PS: it could be automatically generated, let’s wait for the next mad developer to make it 😛 ):

    DATA dummy_line TYPE sbook.

DATA gt_sbook TYPE TABLE OF sbook.
SELECT * FROM sbook INTO TABLE gt_sbook.
DELETE gt_sbook FROM 2.
PERFORM shrink_standard_itab USING dummy_line CHANGING gt_sbook.

DATA gt_sbook_sort TYPE SORTED TABLE OF sbook WITH NON-UNIQUE KEY carrid connid.
SELECT * FROM sbook INTO TABLE gt_sbook_sort.
DELETE gt_sbook_sort FROM 2.
PERFORM shrink_sorted_itab USING dummy_line CHANGING gt_sbook_sort.

DATA gt_sbook_hash TYPE HASHED TABLE OF sbook WITH UNIQUE KEY carrid connid fldate bookid.
SELECT * FROM sbook INTO TABLE gt_sbook_hash.
DELETE gt_sbook_hash WHERE carrid > 'AA' .
PERFORM shrink_hashed_itab USING dummy_line CHANGING gt_sbook_hash.

assert 1 = 1. " dummy line for break point eventually


FORM shrink_standard_itab USING dummy_line CHANGING itab TYPE STANDARD TABLE.
  DATA ref_buffer_tab TYPE REF TO data.
  FIELD-SYMBOLS <buffer_tab> TYPE STANDARD TABLE.

  CREATE DATA ref_buffer_tab LIKE itab.
  ASSIGN ref_buffer_tab->* TO <buffer_tab>.
  <buffer_tab> = itab.
  APPEND dummy_line TO <buffer_tab>.
  DELETE <buffer_tab> INDEX sy-tabix.
  CLEAR itab.
  itab =  <buffer_tab>.
  CLEAR <buffer_tab>.
ENDFORM.

FORM shrink_sorted_itab USING dummy_line CHANGING itab TYPE SORTED TABLE.
  DATA ref_buffer_tab TYPE REF TO data.
  FIELD-SYMBOLS <buffer_tab> TYPE SORTED TABLE.

  CREATE DATA ref_buffer_tab LIKE itab.
  ASSIGN ref_buffer_tab->* TO <buffer_tab>.
  <buffer_tab> = itab.
  READ TABLE <buffer_tab> FROM dummy_line TRANSPORTING NO FIELDS. "binary search implicite.
  INSERT dummy_line INTO <buffer_tab> INDEX sy-tabix.
  IF sy-subrc <> 0.
    " Error DUMMY_LINE badly chosen TODO handle this case
  ELSE.
    DELETE <buffer_tab> INDEX sy-tabix.
    CLEAR itab.
    itab =  <buffer_tab>.
  ENDIF.
  CLEAR <buffer_tab>.
ENDFORM.

FORM shrink_hashed_itab USING dummy_line CHANGING itab TYPE HASHED TABLE.
  DATA ref_buffer_tab TYPE REF TO data.
  FIELD-SYMBOLS <buffer_tab> TYPE HASHED TABLE.

  CREATE DATA ref_buffer_tab LIKE itab.
  ASSIGN ref_buffer_tab->* TO <buffer_tab>.
  <buffer_tab> = itab.
  INSERT dummy_line INTO TABLE <buffer_tab>.
  IF sy-subrc <> 0.
    " Error DUMMY_LINE badly chosen TODO handle this case
  ELSE.
    DELETE TABLE <buffer_tab> FROM dummy_line.
    CLEAR itab.
    itab =  <buffer_tab>.
  ENDIF.
  CLEAR <buffer_tab>.
ENDFORM.
    (0) 
      1. Sandra Rossi

        For the dummy line, there may be an issue only if the table has a unique key (primary/secondary): an initial line might not work if there’s an “initial” key. It’s why I let the developer decide and transmit the dummy line to the procedure.

        (0) 
        1. Shai Sinai

          I guess you can solve it by a generic DELETE/INSERT method:

          METHOD shrink.

            DATA: ld_table_line TYPE REF TO data.

            FIELD-SYMBOLS: <lt_table_std> TYPE STANDARD TABLE,

                                           <ls_table> TYPE any.

            DATA: lo_tabledescr TYPE REF TO cl_abap_tabledescr.

            CREATE DATA ld_table_line LIKE LINE OF ct_table.

            ASSIGN ld_table_line->* TO <ls_table>.

            lo_tabledescr ?= cl_abap_tabledescr=>describe_by_data( ct_table ).

            LOOP AT ct_table INTO <ls_table>.

              EXIT.

            ENDLOOP.

            DELETE TABLE ct_table FROM <ls_table>.

            CASE lo_tabledescr->table_kind.

              WHEN cl_abap_tabledescr=>tablekind_std.

                ASSIGN ct_table TO <lt_table_std>.

                INSERT <ls_table> INTO TABLE <lt_table_std> INDEX 1.

              WHEN OTHERS.

                INSERT <ls_table> INTO TABLE ct_table.

            ENDCASE.

          ENDMETHOD.

          (0) 
          1. Sandra Rossi

            how stupid I am 🙂 Thanks, you’re right ! We just need to delete one line and recreate it at the same place (instead of the inverse sequence insert a dummy line + delete it), and it’s solved and simplified a lot. Your code just misses the important part for shrinking the memory, i.e. the internal table needs first to be copied to a new one.

            (0) 
            1. Shai Sinai

              You’re right. I left that part for you 😛

              P.S.

              1. I wonder what table type we should use for the temporary table: Standard table or original table type?

              Standard table might simplify the logic, but I’m not sure about the performance aspects (Whether better or worse).

              2. How does table assignment (itab2[] = itab1[]) handled in memory?

              Is the memory assigned to itab2 being first released in such case?

              (0) 
              1. Sandra Rossi

                Here is the new easy-to-use code, no more “dummy line”, and only one procedure as you suggested (easy to change it to a method 😉 ):

                DATA gt_sbook TYPE TABLE OF sbook.
SELECT * FROM sbook INTO TABLE gt_sbook.
DELETE gt_sbook FROM 2.
PERFORM shrink_itab CHANGING gt_sbook.

DATA gt_sbook_sort TYPE SORTED TABLE OF sbook WITH NON-UNIQUE KEY carrid connid.
SELECT * FROM sbook INTO TABLE gt_sbook_sort.
DELETE gt_sbook_sort FROM 2.
PERFORM shrink_itab CHANGING gt_sbook_sort.

DATA gt_sbook_hash TYPE HASHED TABLE OF sbook WITH UNIQUE KEY carrid connid fldate bookid.
SELECT * FROM sbook INTO TABLE gt_sbook_hash.
DELETE gt_sbook_hash WHERE carrid > 'AA' .
PERFORM shrink_itab CHANGING gt_sbook_hash.

assert 1 = 1. " dummy line for break point eventually

FORM shrink_itab CHANGING itab TYPE ANY TABLE.
  DATA: ref_buffer_tab  TYPE REF TO data,
        lrw_buffer_line TYPE REF TO data,
        lo_tabledescr   TYPE REF TO cl_abap_tabledescr.
  FIELD-SYMBOLS: <buffer_tab_hashed> TYPE HASHED TABLE,
                 <buffer_tab_index>  TYPE INDEX TABLE,
                 <buffer_line>  TYPE ANY,
                 <buffer_line2> TYPE ANY.

  IF itab IS INITIAL.
    FREE itab.
    RETURN.
  ENDIF.

  CREATE DATA ref_buffer_tab LIKE itab.

  CREATE DATA lrw_buffer_line LIKE LINE OF itab.
  ASSIGN lrw_buffer_line->* TO <buffer_line>.

  lo_tabledescr ?= cl_abap_tabledescr=>describe_by_data( itab ).
  CASE lo_tabledescr->table_kind.
    WHEN cl_abap_tabledescr=>tablekind_std
          OR cl_abap_tabledescr=>tablekind_sorted.
      ASSIGN ref_buffer_tab->* TO <buffer_tab_index> .
      <buffer_tab_index> = itab.
      READ TABLE <buffer_tab_index> INTO <buffer_line> INDEX 1.
      DELETE <buffer_tab_index> INDEX 1.
      INSERT <buffer_line> INTO <buffer_tab_index> INDEX 1.
      itab = <buffer_tab_index>.
    WHEN cl_abap_tabledescr=>tablekind_hashed.
      ASSIGN ref_buffer_tab->* TO <buffer_tab_hashed>.
      <buffer_tab_hashed> = itab.
      " get last line (so that the resulting shrinked itab has same LOOP AT order)
      LOOP AT <buffer_tab_hashed> ASSIGNING <buffer_line2>.
      ENDLOOP.
      <buffer_line> = <buffer_line2>.
      DELETE TABLE <buffer_tab_hashed> FROM <buffer_line>.
      INSERT <buffer_line> INTO TABLE <buffer_tab_hashed>.
      itab = <buffer_tab_hashed>.
    WHEN OTHERS. "impossible - raise exception
  ENDCASE.

ENDFORM.
                (0) 
                1. Shai Sinai

                  I think it closes the issue 🙂

                  BTW, Why do you read the last record in hashed table (LOOP … ENDLOOP) instead of the first one? Isn’t it a redundant loop?

                  Horst,

                  do you have any input for my previous thoughts?

                  1. I wonder what table type we should use for the temporary table: Standard table or original table type? Standard table might simplify the logic, but I’m not sure about the performance aspects (Whether better or worse).

                  2. How does table assignment (itab2[] = itab1[]) handled in memory? Is the memory assigned to itab2 being first released in such case?

                  (0) 
                    1. Shai Sinai

                      Thanks for the reference.

                      I am aware of the sharing mechanism, but I’m not sure how the memory is handled for the old table (i.e. What happens to the “old” memory allocated to itab2 when itab2[] = itab1[] is executed).

                      (0) 
                  2. Sandra Rossi

                    Shai Sinai wrote:

                    BTW, Why do you read the last record in hashed table (LOOP … ENDLOOP) instead of the first one? Isn’t it a redundant loop?

                    As I said in the comment above the loop 😉 : “get last line (so that the resulting shrinked itab has same LOOP AT order)”. As this procedure is to be a utility, and work in all situations, there is the case the developer wants to LOOP AT the hashed table, which loops at the lines according to the chronology they have been added to the itab. So the only way to keep the order is to work with the last line: the last line will remain the last line added. From a performance point of view, we may assume that it won’t take too long as the shrink should be done on an internal table with few lines, as Horst assumed in his original post.

                    (0) 
    1. Marius Piedallu van Wyk

      Here is the code: (link should be more up-to-date)

      CLASS cl_demo DEFINITION.
  PUBLIC SECTION.
    CLASS-METHODS compact CHANGING ct_tab TYPE ANY TABLE.
ENDCLASS.

CLASS cl_demo IMPLEMENTATION.
  METHOD compact.
    FIELD-SYMBOLS: <lt_buffer>      TYPE ANY TABLE,
                   <lt_buffer_std>  TYPE STANDARD TABLE,
                   <ls_buffer>      TYPE any,
                   <ls_buffer_prev> TYPE any.

    DATA: ltr_buffer TYPE REF TO data,
          lsr_buffer TYPE REF TO data,
          l_kind     TYPE c LENGTH 1.

    " simple case:
    IF ct_tab IS INITIAL.
      FREE ct_tab.
      RETURN.
    ENDIF.

    CREATE DATA ltr_buffer LIKE ct_tab.

    DESCRIBE TABLE ct_tab KIND l_kind.
    CASE l_kind.
      WHEN sydes_kind-standard.

        ASSIGN ltr_buffer->* TO <lt_buffer_std>.
        <lt_buffer_std> = ct_tab.

        " this will create/copy a new instance of the table
        APPEND INITIAL LINE TO <lt_buffer_std>.
        DELETE <lt_buffer_std> INDEX sy-tabix.

        ct_tab = <lt_buffer_std>.

      WHEN OTHERS. " hased / sorted (unique/non-unique)

        ASSIGN ltr_buffer->* TO <lt_buffer>.
        <lt_buffer> = ct_tab.

        LOOP AT <lt_buffer> ASSIGNING <ls_buffer>.
          " this is done only once: work with a actual line copy
          CREATE DATA lsr_buffer LIKE LINE OF ct_tab.
          ASSIGN lsr_buffer->* TO <ls_buffer_prev>.
          <ls_buffer_prev> = <ls_buffer>.

          " this will create/copy a new instance of the table (slow for sorted/hashed)
          DELETE TABLE <lt_buffer> FROM <ls_buffer_prev>.
          INSERT <ls_buffer_prev> INTO TABLE <lt_buffer>.

          EXIT.
        ENDLOOP.

        ct_tab = <lt_buffer>.

    ENDCASE.

    " For immediate garbage collection (when trying to prevent fragmentation):
    " Run: cl_abap_memory_utilities=>do_garbage_collection( ).

  ENDMETHOD.

ENDCLASS.
      (0) 
      1. Sandra Rossi

        Thanks for this OO version of the code! I think that there’s an issue for non-unique sorted tables: DELETE TABLE … FROM … may delete several lines with the same key, but you re-insert only the first line. I guess you should delete and insert one line of sorted tables by using the index.

         

        (0) 
  16. Oliver Sviszt

    Hi Horst,

     

    Time is flying… any news on  “RELEASE MEMORY”? 🙂

     

    Excellent article by the way – but unfortunately in real life it is practically impossible to always consider this. It’s just a question of time time that such a situation will go unnoticed… … at least until the first tsv_tnew_page_alloc_failed dumps arrive. 🙂

    I am just debugging code (SAP standard by the way) where there is a deep nested structure of tables within tables, the inner tables being empty, but still allocating gigabytes of memory in total.

     

    Oliver

     

    (0) 
    1. Sandra Rossi

      You should contact the SAP support for issues with the standard. It’s unusual to first build a big internal table and then delete most of lines, it’s usually done by inserting the few concerned lines into the table. Maybe it’s due to the logic of your SAP application, which one is it?

      (0) 
      1. Oliver Sviszt

        Hi Sandra,

        We have a support ticket open with SAP, but such a memory issue is always hard to proove that it is a problem with the program, and not just simply too much data / too small memory.

        During debugging we found that in a specific scenario all data from an internal table (part of a deep structure) is deleted with the DELETE statement, and thus the memory is not deallocated. This happens in standard code within the class handling characteristic relationships in BW IP. (Class CL_RSPLS_CR_MAPPER)

        I think this shows that it is quite easy to step into this trap when writing your code.

        Actually (some of) the records are deleted from table X and instead added to table A. The only chance to deallocate would be to copy all records from X into A or B, free X, copy back B to X, and free B – i guess?

         

        (1) 
          1. Oliver Sviszt

            I like your subroutine, the trick with the dummy change to force a real copy of data instead of reference based reuse is a nice idea.

            But still I think this is something that should be supported at least by a specific statement – and/or even better an automatism that would keep statistics of row count vs. allocated size and automatically do a clean-up if it is ‘worth the effort’.

            I am sure that that an internal command doing this directly in the kernel can be programmed more efficiently than anything we try to hack together as a workaround. Besides kernel code being faster, your (and probably any) ABAP solution has the problem of temporarily using ‘double’ memory because the table if first copied, before the original is freed. While a kernel based compression and deallocation could actually be done ‘in place’ within the memory already allocated.

            Particularly in BW IP tables might get huuuuge – gigabytes! – and one more additional copy might easily lead to a page allocation failed dump.

            (0) 
            1. Sandra Rossi

              That’s the idea from Horst Keller as presented in his blog post, but just wrapped in a generic way!

              You’re definitely right for the kernel solution, but based on Horst post not saying a word about this, it’s probably not planned.

               

              (0) 
  17. Philipp Hinnah

    Hello,

    is the memory released directly at that point in time when the command line “FREE my_itab.” is executed or is it released later, e. g. after leaving a FUNCTION or METHOD.

    The background of my question: assume, that we are within one METHOD and build up an internal table locally in that method and FREE it again in the same method. Would memory then be released and be directly available after the code line saying “FREE my_itab.”?

    (0) 

Leave a Reply