Skip to Content

This is going to be one of those weird blog posts, that basically convey just a single piece of information, but since the new SAP Community platform doesn’t really care for WIKI/document style knowledge storage, I had to come up with a click-bait title to make it a blog post.

I’ve written before on installing HANA cockpit and yesterday I fired it up once again.
Snooping around a bit in the tracefiles (running two HANA 2 instances + XSA stacks on my one 32GB NUC always leaves room for questions like “how much actual processing memory do we have left now?“) I found this jewel of an “error curtain”:

[26144]{-1}[-1/-1] 2017-09-12 13:30:34.208039 w Basis ELF.cpp(00770) :
[26144]{-1}[-1/-1] 2017-09-12 13:30:34.208039 w Basis ELF.cpp(00771) :
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpythonbase.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libxsengine.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libstatisticsserver2.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libstatisticscommon.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdblivecache.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpackman.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpackmansqldriver.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbxsdbcserver.so

[... Buzz-Lightyear continuation mode ... ]
[... TO INFINITY ... AND BEYOND ....]

Strip it!

To the average advanced computer user like myself, this looks like error messages.

Yes, these messages are marked as ‘w’ like wwwwwarning and the whole system still seems to be doing its thing, but personally, I don’t really like when I see tons of such messages and find precisely zero documents, SAP notes, KBAs or other information sources that can tell me what they mean.
I have the same issue with looking into the CONSOLE app of my MacBook by the way… #who_thinks_of_the_poor_end_user_nerd?

After some HANA source code reading (that really helps every now and then, especially when the trace file message neatly points to C++ file ELF.cpp, lines 770 and 771) I figured out two things:

  • this is really just an info-warning written out when shared libraries are loaded by a HANA process. That usually happens once, at process startup, e.g. when the nameserver starts up.
    and
  • this specific code has been around for several years now. Yet, I have not seen these messages before, which means that something else (aka not this piece of code) must have changed.

The next best thing from here was to look into the trace files of my other HANA 2 installation.
Guess what! No such messages in any of the trace files.

Ok, I thought, so maybe there is something different with the HANA installation that comes with HANA cockpit.

Fighting with ELFes

(sadly, she has nothing to do with this…) (image http://lotr.wikia.com/wiki/Arwen)

Let’s have a look into that anyway.

The warning message listed the shared objects (.so) files that it seems to have issues with, one of them was libhdbpythonbase.so.
Checking this in my HANA Cockpit installation with the Linux tool `file` shows:

file /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so
/usr/sap/H4C/HDB96/exe/libhdbpythonbase.so:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
, dynamically linked
, BuildID[sha1]=cafedde1f0db3527e5a8f3f69fc75019287f4196
, stripped <<< !

Compare this with my other HANA 2 installation:

file /usr/sap/S20/HDB20/exe/libhdbpythonbase.so
/usr/sap/S20/HDB20/exe/libhdbpythonbase.so:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
, dynamically linked
, BuildID[sha1]=1089da0d45a4fc59a631f0877b0f2786a5282937
, not stripped <<< !

Aha! So, the files actually have been “stripped”.

What that means is that human readable information, so called symbols, about the compiled program in the .so files have been removed.
This information is not required to run the program but is extremely useful for debugging and problem analysis.
Without the information, a crash dump would only contain memory addresses for functions and since dynamic loading is used, these addresses change every time the program is run.
On the other hand, with symbols in the files, a stack trace or a crash dump literally print out the names of the involved functions and methods.
This is what makes the problem analysis actually possible in most cases.

Huh? I mean why?

Peachy“, you think, “why on earth would then somebody strip this important info from the files“?
Great question!

The reason is the file size for the .so files.
Symbols can take quite some space in them. This alone might not be a problem since disk space is cheap, but the symbols also get loaded/mapped into memory when the shared object file is used.

And if there is a single precious resource on HANA systems it is the main memory.

How much can we save with this stripping?
First the unstripped version:

ls -lh /usr/sap/S20/HDB20/exe/libhdbpythonbase.so
-r-xr-xr-x 1 s20adm sapsys 8.0M Jul 25 05:11 /usr/sap/S20/HDB20/exe/libhdbpythonbase.so

Now the stripped version:

ls -lh /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so
-r-xr-xr-x 1 h4cadm sapsys 2.2M Jun 26 19:42 /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so

Boom!

Down from 8MB to 2.2MB!
And that is just one of 418 shared objects
(counted on my HANA 2 system via “ls -l *.so | wc -l“).

In total that amounts to quite a difference between the binary sizes for the normal HANA installation and the HANA Cockpit one:

“Normal HANA 2”
ls -lh /usr/sap/S20/HDB20/exe/ | grep total
total 5.2G
“HANA Cockpit”
ls -lh /usr/sap/H4C/HDB96/exe/ | grep total
total 1.7G

Obviously, it has been a goal for the HANA cockpit team to make the additional overhead of having its own HANA instance as small as possible.
There is actually another case where memory usage of HANA needed to be minimized: the HANA Express edition.

After quickly firing up my HXE virtual machine (gee, I really love my NUC system for making this so easy) I checked whether it also uses symbol stripping and lo and behold, it does:

“HANA Express edition”
ls -lh /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so
-r-xr-xr-x 1 hxeadm sapsys 2.2M Jul 24 19:16 /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so

file /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so
/usr/sap/HXE/HDB90/exe/libhdbpythonbase.so:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
, dynamically linked
, BuildID[sha1]=1089da0d45a4fc59a631f0877b0f2786a5282937
, stripped <<< !

For those interested to learn more about symbols and shared libraries here are some pointers:

https://www.akkadia.org/drepper/dsohowto.pdf
https://unix.stackexchange.com/questions/116327/loading-of-shared-libraries-and-ram-usage
https://stackoverflow.com/questions/7683630/library-symbols-and-user-symbols-in-elf

Finally, to go really down into the deep end of this rabbit hole, can we actually see that HANA uses less memory now?

Yes, we can!

The first option, on OS level:

“HANA Express”
hxeadm@hxehost:/usr/sap/HXE/HDB90> HDB info
USER PID PPID %CPU VSZ RSS COMMAND
[...]
hxeadm 1707 1 0.0 21628 2792 sapstart pf=/usr/sap/HXE/SYS/profile/HXE_HDB90_hxehost
hxeadm 1716 1707 0.0 217104 60060 \_ /usr/sap/HXE/HDB90/hxehost/trace/hdb.sapHXE_HDB90 ...
hxeadm 1739 1716 9.5 6880612 5089404 \_ hdbnameserver
[...]

We take the process ID of the nameserver and use the `pmap` tool with it.
As the man page explains, pmap displays information about process memory mappings.

hxeadm@hxehost:/usr/sap/HXE/HDB90> pmap -x 1739

1739: hdbnameserver
START            SIZE       RSS      PSS    DIRTY SWAP SWAPPSS PERM MAPPING
000055b587d67000 1932K    1928K    1928K      12K   0K      0K r-xp .../hdbnameserver
000055b587f4b000  248K     248K     248K     248K   0K      0K r--p .../hdbnameserver
000055b587f89000    4K       4K       4K       4K   0K      0K rw-p .../hdbnameserver
[...]
00007f0be3758000  712K     520K     288K       8K   0K      0K r-xp .../libhdbaflsdk.so
00007f0be380a000   16K      16K      16K      16K   0K      0K r--p .../libhdbaflsdk.so
00007f0be380e000    4K       4K       4K       4K   0K      0K rw-p .../libhdbaflsdk.so
[...]
00007f0c54af6000    4K       4K       4K       4K   0K      0K rw-p [anon]
00007ffccc580000  136K      68K      68K      68K   0K      0K rw-p [stack]
00007ffccc5bc000   12K       0K       0K       0K   0K      0K r--p [vvar]
00007ffccc5bf000    8K       8K       0K       0K   0K      0K r-xp [vdso]
ffffffffff600000    4K       0K       0K       0K   0K      0K r-xp [vsyscall]
Total:        6880616K 5092916K 4825683K 4633372K   0K      0K

5870672K writable-private, 992656K readonly-private, 17288K shared, and 4621300K referenced

(full path to objects /hana/shared/HXE/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeac)

The output from pmap is usually very long and wide, so I cut out a lot here and shortened the file path.
The important bit is 17288K (~16 MB) shared in the last line.

s20adm@skullbox:/usr/sap/S20/SYS/exe/hdb> HDB info
USER PID PPID %CPU VSZ RSS COMMAND
[...]
s20adm 20398 1 0.0 21600 228 sapstart pf=/usr/sap/S20/SYS/profile/S20_HDB20_skullbox.lab.cat5
s20adm 20407 20398 0.0 322428 580 \_ /usr/sap/S20/HDB20/skullbox.lab.cat5/trace/hdb.sapS20_HDB20 ...
s20adm 20429 20407 11.8 9131476 2749264 \_ hdbnameserver
[...]

s20adm@skullbox:/usr/sap/S20/SYS/exe/hdb> pmap -x 20429
20429: hdbnameserver
START            SIZE  RSS  PSS  DIRTY SWAP SWAPPSS PERM MAPPING
000000679a5a6000 1932K 240K 240K    8K   4K      4K r-xp .../hdbnameserver
000000679a78a000 248K   24K  24K   24K 224K    224K r--p .../hdbnameserver
000000679a7c8000 4K      4K   4K    4K   0K      0K rw-p .../hdbnameserver
[...]
00007fd36a355000 76K     0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a368000 32K     0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a370000 648K    0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a412000 1368K   0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a568000 116K    0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a585000 156K    0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
00007fd36a5ac000 80K     0K   0K    0K   0K      0K r--s .../libhdbaflsdk.so
[...]
00007fd46f0d7000 4K      4K   4K    4K   0K      0K rw-p [anon]
00007ffdd3c51000 136K   28K  28K   28K  40K     40K rw-p [stack]
00007ffdd3dd6000 12K     0K   0K    0K   0K      0K r--p [vvar]
00007ffdd3dd9000 8K      8K   0K    0K   0K      0K r-xp [vdso]
ffffffffff600000 4K      0K   0K    0K   0K      0K r-xp [vsyscall]
Total:          8925232K 2753308K 2717475K 2491696K 3128688K 3023716K

5720584K writable-private, 994368K readonly-private, 2210280K shared, and 2569084K referenced

(full path to objects /hana/shared/S20/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeac)

Same story here, but here we have 2210280K (2.1 GB) shared data.

The HANA way

Another approach to compare the memory usage is of course to query the HANA monitoring views:

select
name,
round (value/1024/1024, 2) as value_MB
from "PUBLIC"."M_MEMORY";
                               HANA 2    HANA Express
NAME                          VALUE_MB       VALUE_MB
SYSTEM_MEMORY_SIZE           32063.550      23546.860
SYSTEM_MEMORY_FREE_SIZE        272.140        246.680
PROCESS_MEMORY_SIZE           8652.050       6717.480
PROCESS_RESIDENT_SIZE         2110.720       4960.260 <<< ~ Linux RSS
PROCESS_CODE_SIZE             2017.950          3.580 <<< ~ Linux shared
PROCESS_STACK_SIZE             146.50          99.690
PROCESS_ALLOCATION_LIMIT     28857.190      21192.170
GLOBAL_ALLOCATION_LIMIT      28857.190      21192.170
[...]

More about HANA memory can be found in SAP Note 1999997 – FAQ: SAP HANA Memory

If you see warning about the “using only dynamic symbol table” now, you can put this into perspective.

There you go, now you know.

Cheers,

Lars

To report this post you need to login first.

4 Comments

You must be Logged on to comment or reply to a post.

    1. Lars Breddemann Post author

      Yep, you’re right!

      The HANA binaries should get/are accounted for in the PROCESS_CODE_SIZE line. But (!) PROCESS_CODE_SIDE and PROCESS_RESIDENT_SIZE are measures that overlap!

      See https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management#21049737 for a good explanation here.

      If I get that right, then what HANA does to get these numbers is to look at the memory mapping (similar to my pmap example) and check the sizes of the executable segments. This includes the opened shared libraries/objects that have been loaded at that point in time.

       

      (0) 

Leave a Reply