Releasing Internal Table Memory
It is a well known fact, that you release the memory occupied by an internal table using either CLEAR or FREE, where FREE releases also the initial memory area. You normally use CLEAR, if you want to reuse the table and you use FREE, if you really want to get rid of it and don’t want to refill it later on. Assigning an initial internal table to a filled internal table does also release the target table’s memory in the same way as CLEAR does.
But last week a colleague pointed out to me that it is not such a well known fact that deleting lines of internal tables with DELETE normally does not release the memory occupied by the deleted lines. Instead, there seem to be people deleting lines of internal tables in order to release memory. Therefore as a rule:
Deleting lines of an internal table using the DELETE stament does not release the table’s memory.
For an internal table that was filled and where all lines are deleted using the DELETE statement the predicate IS INITIAL in fact is true. But the internal table is only initial regarding the number of lines but not regarding the memory occupied. You can check that easily using the memory analysis tools of the ABAP debugger.
So far so good. For releasing the memory of an internal table you use CLEAR or FREE and you do not simply DELETE all lines.
But what about the use case, where you want to delete almost all lines from a big internal table and to keep the rest? After deleting, the internal table occupies much more memory than needed for its actual lines. If memory consumption is critical, you might want to get rid of the superfluous memory occupied by such an internal table. How to do that?
Spontaneous idea:
DELETE almost_all_lines_of_itab.
DATA buffer_tab LIKE itab.
buffer_tab = itab.
CLEAR itab.
itab = buffer_tab.
CLEAR buffer_tab.
Bad idea! Check it in the ABAP Debugger. Due to table sharing, after assigning itab to buffer_tab, buffer_tab is pointing to the same memory area as itab. Assigning buffer_tab back to itab after clearing itab is simply an effectles roundtrip and you gain nothing.
Improved idea:
DELETE almost_all_lines_of_itab.
DATA buffer_tab LIKE itab.
buffer_tab = VALUE #( ( LINES OF itab ) ).
CLEAR itab.
itab = buffer_tab.
CLEAR buffer_tab.
Now it works! Instead of copying itab to buffer_tab you can transfer the lines of itab sequentially to the initial target table and the memory is not shared. Before 7.40, SP08 you have to use INSERT LINES OF itab INTO TABLE buffer_tab instead of the VALUE expression, of course.
What also works for the use case is:
DELETE almost_all_lines_of_itab.
DATA buffer_tab LIKE itab.
buffer_tab = itab.
INSERT dummy_line INTO TABLE buffer_tab.
DELETE buffer_tab WHERE table_line = dummy_line.
CLEAR itab.
itab = buffer_tab.
CLEAR buffer_tab.
By inserting a dummy line into buffer_tab and deleting it again, the table sharing is canceled and buffer_tab is built from scratch (but only, if it needs considerably less memory than before; otherwise it is copied and nothing is gained again).
Ingenious minds might also find the following ways:
DELETE almost_all_lines_of_itab.
DATA buffer_string TYPE xstring.
EXPORT itab TO DATA BUFFER buffer_string.
CLEAR itab.
IMPORT itab FROM DATA BUFFER buffer_string.
CLEAR buffer_string.
or even
DELETE almost_all_lines_of_itab.
CALL TRANSFORMATION id SOURCE itab = itab
RESULT XML DATA(buffer_string).
CLEAR itab.
CALL TRANSFORMATION id SOURCE XML buffer_string
RESULT itab = itab.
CLEAR buffer_string.
Yes, those work too, but put some GET RUN TIME FIELD statements around them to see that those are not the best ideas …
Nice blog, this would surely improve my programming skills.
cant we get a conditional CLEAR statement... like CLEAR WHERE ๐
that would help a lot
In fact, I discussed that with the developers already. One might also consider a statement RELEASE MEMORY or something like that. Also implicit mechanisms for releasing superfluous memory are under consideration. But for the time being, we have to live with the workarounds.
Indeed, in 2015 I would expect not to have to deal with this anymore as a programmer.
Still a very nice blog post though! Thanks for explaining.
Fred
Nice Block......... thank you for sharing this
Is it intentional that DELETE does not release the memory? Better performance, or difficult re-allocation of memory?
The internal table lines are spreaded so to say randomly over the chunks of allocated memory.
That's very interesting. I thought they are stored as an array so that if you know the address of itab's beginning, table length and the number of lines you can say itab's end in memory.
Really cool post ๐
No that's how strings are handled. But internal table are managed either by their table index or the hash engine depending on the table type. When adressing an internal table line either the index or the hash management points to the real location. So, if you delete lines even from an index table there is no physical memory table that is shrunk, but the index is reorganized.
Frankly speaking, I haven't even thought about it until today, it seemed to be so simple.
Can you say the name of the mechanism that "spreads" the tables over allocated memory? Couldn't google it at once, I'd like to read more about how it works.
Oh no, that's very internal and subject to change.
You find small glimpses of it here.
Hi Horst,
How would you do it in a productive code? Would you write this code inside a macro? For e.g., mac_save_itab_memory ๐
BR,
Suhas
LOL ...
But seriously one might ask why there isn't a wrapper for this. On the other hand, just look how long nobody never ever has cared for that. Maybe memory isn't such important any more.
That.
And IMHO the Garbage Collector should be responsible to release unused space in this case. Would be difficult to define the correct rules for that however (memory consumption vs. performance).
In its current state, garbage collector handles heap objects only.
Releasing unused memory in the stack isn't called Garbage Collection? Learned something new (again). My education is too long ago it seams...
Didn't say so. I onyl said what the ABAP Garbage Collector in its current state does.
Horst said: "Maybe memory isn't such important any more."
It used to be important, and I'm willing to bet it will be again when Code-to-Data is misunderstood and everything gets pushed to the database servers ๐
I used to teach ABAP back in 3.x and 4.x days, and the CLEAR v REFRESH v FREE was a 15 minute topic.
Suppose we have the following abap code which consists of two small procedures:
form form1.
perform form2.
endform.
form form2.
data itab type standard table of ...
...populate itab from db tables...
...process itab lines (but there is no "clear itab" or "free itab" statements)...
endform.
Question is: will the memory occupied by itab be freed when execution flow exits from form2 (and, on the next step, from form1)? Do I need to manually call "free itab" in such cases?
Thank you for useful blog.
Well, an internal table inside a procedure (next time, you use METHOD, ENDMETHOD ๐ ) is part of the stack and not of the heap, therefore ...
So if one uses statement clear internal_table. Then one should also after the internal_table is not to be used again, have the statement free internal_table.
Correct?
Also I am usually using refresh internal_table. Does this free any memory?
You don't have to use FREE after CLEAR. You can use FREE directly. But normally FREE is not necessary, only if you desperatly seek for additional memory.
REFRESH itab is exactly the same as CLEAR itab[]. It adresses always the table body and never the obsolete header line. Since we don't work with header lines any more (duh!). You need neither REFRESH nor itab[] any more. Simply CLEAR is OK.
Hi Horst,Thanks for sharing this with the community. Good to know.
Thanks
Hakim
Good stuff, Horst! Always a delight to get more insight into how to code ABAP well.
Thanks,
Thorsten
Good Post!
Thanks for sharing this useful info..
I've been doing it wrong all these years. How embarrassing. ๐ณ
I'm with SAP for 20 years now, always writing ABAP documentation, and added that stuff only recently for the next upcoming release, also embarrassing. And that's not the only thing, ๐ ...
Good Stuff Horst.
Thanks for sharing.
Hi, Horst! Is there an event in SAP when Garbage Collector end its work?
I have a class which collects references to data in a static table and when this data is released reference becomes incorrect and the dump is occured.
I want to subscribe to this event and delete table lines with an inappropriate references.
Nothing that I know of. And in fact it is not wanted too. The garbage collector should be transparent for the normal ABAP developer.
To be honest, so far I have not cared to much about releasing internal table memory.
Still, nice to know!
For fun, below, 3 procedures to shrink an internal table, based on Horst algorithm ; there's only the "dummy line" the developer has to worry about (PS: it could be automatically generated, let's wait for the next mad developer to make it ๐ ):
Hmm, INSERT INITIAL LINE? Just a hunch, I'll check tomorrow.
For the dummy line, there may be an issue only if the table has a unique key (primary/secondary): an initial line might not work if there's an "initial" key. It's why I let the developer decide and transmit the dummy line to the procedure.
I guess you can solve it by a generic DELETE/INSERT method:
METHOD shrink.
DATA: ld_table_line TYPE REF TO data.
FIELD-SYMBOLS: <lt_table_std> TYPE STANDARD TABLE,
<ls_table> TYPE any.
DATA: lo_tabledescr TYPE REF TO cl_abap_tabledescr.
CREATE DATA ld_table_line LIKE LINE OF ct_table.
ASSIGN ld_table_line->* TO <ls_table>.
lo_tabledescr ?= cl_abap_tabledescr=>describe_by_data( ct_table ).
LOOP AT ct_table INTO <ls_table>.
EXIT.
ENDLOOP.
DELETE TABLE ct_table FROM <ls_table>.
CASE lo_tabledescr->table_kind.
WHEN cl_abap_tabledescr=>tablekind_std.
ASSIGN ct_table TO <lt_table_std>.
INSERT <ls_table> INTO TABLE <lt_table_std> INDEX 1.
WHEN OTHERS.
INSERT <ls_table> INTO TABLE ct_table.
ENDCASE.
ENDMETHOD.
how stupid I am ๐ Thanks, you're right ! We just need to delete one line and recreate it at the same place (instead of the inverse sequence insert a dummy line + delete it), and it's solved and simplified a lot. Your code just misses the important part for shrinking the memory, i.e. the internal table needs first to be copied to a new one.
You're right. I left that part for you ๐
P.S.
1. I wonder what table type we should use for the temporary table: Standard table or original table type?
Standard table might simplify the logic, but I'm not sure about the performance aspects (Whether better or worse).
2. How does table assignment (itab2[] = itab1[]) handled in memory?
Is the memory assigned to itab2 being first released in such case?
Here is the new easy-to-use code, no more "dummy line", and only one procedure as you suggested (easy to change it to a method ๐ ):
I think it closes the issue ๐
BTW, Why do you read the last record in hashed table (LOOP ... ENDLOOP) instead of the first one? Isn't it a redundant loop?
Horst,
do you have any input for my previous thoughts?
Maybe you should refer to this - sharing - ABAP Keyword Documentation.
Thanks for the reference.
I am aware of the sharing mechanism, but I'm not sure how the memory is handled for the old table (i.e. What happens to the "old" memory allocated to itab2 when itab2[] = itab1[] is executed).
As I said in the comment above the loop ๐ : "get last line (so that the resulting shrinked itab has same LOOP AT order)". As this procedure is to be a utility, and work in all situations, there is the case the developer wants to LOOP AT the hashed table, which loops at the lines according to the chronology they have been added to the itab. So the only way to keep the order is to work with the last line: the last line will remain the last line added. From a performance point of view, we may assume that it won't take too long as the shrink should be done on an internal table with few lines, as Horst assumed in his original post.
Good point.
My solution is pretty similar, and I've taken some bits from your advice as well:
http://teuton.blogspot.com/2017/10/compacting-internal-memory-of-sap.html
I tried to make it generic for all types of tables (standard, hashed, sorted unique/non-unique)
Here is the code: (link should be more up-to-date)
Thanks for this OO version of the code! I think that there's an issue for non-unique sorted tables: DELETE TABLE ... FROM ... may delete several lines with the same key, but you re-insert only the first line. I guess you should delete and insert one line of sorted tables by using the index.
Thanks for this insight! I'm wondering if, even though the delete might delete all the entries as you said, that the loop may still have a reference to those and will loop over them yet. In which case subsequent deletes may delete all again (but not do anything since they were all deleted already), and then re-insert it anyway.
It may still work...
Hi Horst,
Time is flying... any news onย "RELEASE MEMORY"? ๐
Excellent article by the way - but unfortunately in real life it is practically impossible to always consider this. It's just a question of time time that such a situation will go unnoticed... ... at least until the firstย tsv_tnew_page_alloc_failed dumps arrive. ๐
I am just debugging code (SAP standard by the way) where there is a deep nested structure of tables within tables, the inner tables being empty, but still allocating gigabytes of memory in total.
Oliver
You should contact the SAP support for issues with the standard. It's unusual to first build a big internal table and then delete most of lines, it's usually done by inserting the few concerned lines into the table. Maybe it's due to the logic of your SAP application, which one is it?
Hi Sandra,
We have a support ticket open with SAP, but such a memory issue is always hard to proove that it is a problem with the program, and not just simply too much data / too small memory.
During debugging we found that in a specific scenario all data from an internal table (part of a deep structure) is deleted with the DELETE statement, and thus the memory is not deallocated. This happens in standard code within the class handling characteristic relationships in BW IP. (Class CL_RSPLS_CR_MAPPER)
I think this shows that it is quite easy to step into this trap when writing your code.
Actually (some of) the records are deleted from table X and instead added to table A. The only chance to deallocate would be to copy all records from X into A or B, free X, copy back B to X, and free B - i guess?
Yes. There's a utility subroutine to shrink X, in the comment September 2, 2016 at 3:18 pm.
I like your subroutine, the trick with the dummy change to force a real copy of data instead of reference based reuse is a nice idea.
But still I think this is something that should be supported at least by a specific statement - and/or even better an automatism that would keep statistics of row count vs. allocated size and automatically do a clean-up if it is 'worth the effort'.
I am sure that that an internal command doing this directly in the kernel can be programmed more efficiently than anything we try to hack together as a workaround. Besides kernel code being faster, your (and probably any) ABAP solution has the problem of temporarily using 'double' memory because the table if first copied, before the original is freed. While a kernel based compression and deallocation could actually be done 'in place' within the memory already allocated.
Particularly in BW IP tables might get huuuuge - gigabytes! - and one more additional copy might easily lead to a page allocation failed dump.
That's the idea fromย Horst Keller as presented in his blog post, but just wrapped in a generic way!
You're definitely right for the kernel solution, but based on Horst post not saying a word about this, it's probably not planned.
Hello,
is the memory released directly at that point in time when the command line "FREE my_itab." is executed or is it released later, e. g. after leaving a FUNCTION or METHOD.
The background of my question: assume, that we are within one METHOD and build up an internal table locally in that method and FREE it again in the same method. Would memory then be released and be directly available after the code line saying "FREE my_itab."?