Auto-Tune Your Buffers With Kernel 742
A couple of decades ago, a fellow by the name of Andy Hildebrand was a geophysicist working in the field of oil exploration. He developed a digital signal processing algorithm to interpret sound waves that could help locate oil deposits. It didn’t take him long to realize this algorithm could be applied to the human voice as well, pitch-shifting to correct for off-key singing. The result, in 1998, was Cher’s hit Believe, and the wildly successful software Auto-Tune. Today you would be hard pressed to find a pop song that doesn’t use Auto-Tune, whether to intentionally distort or to subtly correct. Auto-Tune has become a critical part of the toolkit in any recording studio, and is even being built in to electric guitars for realtime perfect pitch.
(“Autotuneevo6” by Source. Licensed under Fair use via Wikipedia.)
Just like a pop star or audio engineer, the ability to autotune is now part of the toolkit for the Basis administrator, too. No, I don’t mean that a bunch of SAP geeks are forming a rock band (although TechEd attendees might get to see some jamming on stage), but rather you can now use the power of algorithms to set many of your memory and buffer parameters automatically.
- Extended Memory Pool
- Export/Import Buffer
- Program Buffer
- Table Buffer (previously Generic Key Buffer)
- Single Records Buffer
- Nametab Buffers
- Other Parameters
- Comparing NetWeaver 7.4 and 7.01 Systems
- More Information
Zero Administration Memory Management
Automatically adjustable memory parameters have been around, at least for Windows-based ABAP systems, for quite a while now. There’s nothing really new there. Since at least R/3 4.6b the basic parameters for controlling work process memory have either been automated, or defaulted high enough that, given the typical amounts of physical memory in servers of the day, the limits would not be reached. Parameters that Unix admins tweaked regularly were, for Windows admins, practically invisible.
But the term Zero Administration Memory Management was still a bit of a misnomer, as the system administrator still needed to monitor and manually adjust quite a few parameters to control the nametab buffer, the program buffer, the generic and single record table key buffers, among others. Various Notes detailed recommended starting values, and then over a period of days, weeks, or months, you would adjust to keep hit ratios high and swap counts low.
In other words, there was still a fair amount of manual tuning to get the buffers just right, although as the cost of physical memory came down and modern servers were built to accommodate much more of it, one could argue that you just throw more memory in the system and make all the buffers really large and be done with it.
This approach lacks a certain amount of elegance and finesse, though, and could still leave you open to provisioning too much memory for one buffer when it could have been used more effectively elsewhere.
Enter NetWeaver ABAP 7.4, and more specifically 7.4 SR2 and its attendant kernel version, 742.
Each new kernel release has introduced new parameters, or modified the defaults in older ones, and with each release the recommendations from SAP for initial parameter settings have changed slightly.
The 740, 741, and 742 kernels, however, have introduced a new level of automation for a number of the parameters Basis admins have become used to configuring, through the use of a cascading series of formulas. Each parameter derives its value from another parameter, and all of the automated parameters ultimately find their origin from the well-known and easily configurable PHYS_MEMSIZE.
You can easily look up these formulas, either in Note 2085980 (New features in memory management as of Kernel Release 7.40), or in your own 7.4 system with transaction code RZ11 or report RSPARAM. Here, however, I’ll try to interpret the formulas in plain English and show how each value derives from the previous parameter.
The venerable PHYS_MEMSIZE, which defines how much of main memory, in megabytes, should be made available to the SAP instance, has not changed. It still defaults to the amount of physical RAM installed in the server, and in the case of a dedicated application server should generally be left unset. Doing so means that most of the other parameters will automatically adjust themselves whenever you add additional memory to the server, without changing a single setting.
If your server runs both a database and application instance (in old terminology, a Central Server), then you should still follow the old advice about setting PHYS_MEMSIZE to restrict how much memory SAP will use so that enough is reserved for the database. Likewise if you run two or more SAP instances on the same host: you will set this parameter for each instance to appropriately divide up the memory between them. However, the beauty here is that, in most cases, this is the only parameter you need to adjust.
Extended Memory Pool
The Extended Memory Pool, or how much memory will actually be used by the system, is a dynamically variable amount during system operation, starting with the value of em/initial_size_MB and then increasing automatically when system demand requires it. In prior releases, the initial size of this pool was equal to PHYS_MEMSIZE, but now it starts at 70% of that amount, with a maximum size of 500 GB. This parameter, em/initial_size_MB, then becomes the starting point for most of the buffer size calculations.
The exception is the Export/Import Buffer, which is controlled by rsdb/obj/buffersize and rsdb/obj/max_objects. The size of this buffer is set to 1% of PHYS_MEMSIZE, with a minimum of 4 MB, and the max_objects value is set to 25% of the buffersize, with a minimum of 2000 objects.
The Program Buffer, controlled via abap/buffersize, and often the bane of many a Basis admin’s existence in years gone by, is typically the largest buffer in the system. Previously defaulted to about 300 MB, it now defaults to 15% of the initial extended memory pool (em/initial_size_MB), rounded up to the nearest multiple of 4 MB.
Table Buffer (previously Generic Key Buffer)
Controlled via zcsa/table_buffer_area and zcsa/db_max_buftab, this buffer previously defaulted to about 30 MB. Now it defaults to 10% of the initial extended memory pool (em/initial_size_MB), but with a minimum of ~30 MB and a maximum of ~3 GB. The number of objects that can be stored in the buffer (zcsa/db_max_buftab) previously defaulted to 5000, but now is a function of the buffer size divided by 5120, and with a minimum of 20,000 objects.
Single Records Buffer
This buffer doesn’t show up in ST02 anymore, but it still exists. It’s size, defined by rtbb/buffer_length, used to be about 10 MB but is now 10% of the Table Buffer. Likewise, the number of object entries, defined by rtbb/max_tables, used to be 500 but is now 10% of the number of entries allowed in the Table Buffer.
This is a collection of four related buffers.
Table Definition Buffer
The number of table definitions (i.e., which fields make up the table, etc) that can be buffered is controlled by rsdb/ntab/entrycount. Previously set at 200,000, it now takes its value from zcsa/db_max_buftab, i.e. the same number of table definitions can be buffered as the number of table generic keys. This parameter, the number of entries in the Table Definition Buffer, then goes on to also define the number of entries allowed in the other three Nametab buffers, as well as form the basis for the size of those buffers.
Field Definition Buffer
This is controlled via rsdb/ntab/ftabsize, and is often the second largest buffer in the system (after the Program Buffer). Previously defaulted to ~250 MB, it now defaults to 1 KB per allowed object (i.e., rsdb/ntab/entrycount), to a maximum of ~500 MB.
Short NTAB Buffer
This buffer contains a derived summary of table and field definitions and is controlled via rsdb/ntab/sntabsize. Previously defaulted to ~15 MB, now it is 10% of the Field Definition Buffer size.
Initial Record Layouts Buffer
This buffer contains the record layout based upon the field types and is controlled via rsdb/ntab/irbdsize. Previously defaulted to ~30 MB, now it is 20% of the Field Definition Buffer size.
Not every buffer parameter has such a helpful formula, so there are still some buffers you must manually tune yourself, if needed. These have not changed their defaults from kernel 722.
rsdb/cua/buffersize (~3 MB) and rsdb/cua/max_objects (25% of rsdb/cua/buffersize, minimum of 2000; this is a new parameter).
zcsa/presentation_buffer_area (~4 MB) and sap/bufdir_entries (2000).
zcsa/calendar_area (~500 KB) and zcsa/calendar_ids (200).
Online Text Repository (OTR) Buffer
rsdb/otr/buffersize_kb (~4 MB) and rsdb/otr/max_objects (2000).
Export/Import Shared Memory (ESM) Buffer
rsdb/esm/buffersize_kb (~4 MB) and rsdb/esm/max_objects (2000).
Comparing NetWeaver 7.4 and 7.01 Systems
Let’s compare how these new defaults work in two identical systems, each with 64 GB of physical RAM.
|Parameter||Units||NetWeaver 7.4||NetWeaver 7.01|
Obviously, the new defaults result in some significantly increased initial values!
Let’s look at the same comparison, but this time between two NetWeaver 7.4 systems, one with the same 64 GB of RAM, and one with 8 GB of RAM (to see the effect of minimums and maximums in the ranges).
|Parameter||Units||64 GB RAM||8 GB RAM|
Does this mean you never again have to tune these parameters? Not at all. However, it means you are likely to start from a more reasonable base value, minimizing the chance of your system performing poorly on day one due to insufficient buffer space. You still must watch the buffers over time, and you can still adjust these parameters manually. Be aware, however, that when you adjust one that is “upstream” it may have ripple effects “downstream.”
If you find a large number of parameters need increasing, consider increasing the “upstream” parameter(s) and letting the effect cascade. If you have manually set PHYS_MEMSIZE to something other than (less than) your physical RAM, consider adjusting just this one parameter first.
Note 2085980: New features in memory management as of Kernel Release 7.40
Note 88416: Zero administration memory management for the ABAP server
Table buffer parameter (zcsa/db_max_buftab & zcsa/table_buffer_area) and Single records buffer parameter (rtbb/max_tables) can be changed to the dimensioning of the new table buffer by explicitly setting the parameter below and remove above old prameters:
fyi - 2103827 - Profile parameters for table buffer as of SAP Kernel Release 7.40
Interesting. That explains a few things, and it definitely simplifies the way the buffer is used, but instead of simplifying the scenario for setting the parameters, I think it may complicate them (if set at all). For one thing, my own NW 7.4 SP11 (kernel 742 pl 200) system behaves almost but not exactly as described in the Note (and furthermore in the Notes 1864189 and 1926986 linked from that Note). The Table Buffer does indeed show a number of directory entries that matches the formula from Note 2103827. The allocated size is close to, but actually slightly larger than, the size from the formula in the Note. So, I see that behavior, but something else is at play here for the allocated size to add a bit more to it.
The two new parameters are set to the "default" in my system (it is an upgraded system from NW 7.00 with kernel 7.22, actually). However, I have also not specifically set the old parameters, and they do show as calculated according to the formulas.
Given this information, I think my general advice would be to continue to not set any of these six parameters (zcsa/table_buffer_area, zcsa/db_max_buftab, rtbb/buffer_length, rtbb/max_tables, rsdb/tbi_buffer_area_MB, and rsdb/tbi_dir_entries) unless needed, but instead to first see if tweaking PHYS_MEMSIZE makes more sense. If not, then my question would be whether tweaking rsdb/tbi_buffer_area_MB makes sense, or if it would be better to instead tweak the old zcsa/table_buffer_area. My thinking here is how zcsa_db_max_buftab and rtbb/max_tables are now influenced by zcsa/table_buffer_area, and that would in turn influence rsdb/tbi_dir_entries. If, however, you manually set rsdb/tbi_buffer_area_MB, does that influence rsdb/tbi_dir_entries, or are you now required to again manually adjust it? Perhaps some experimentation is in order here.
Regardless, I see I must update the section in the blog about the Table Buffer.
Thanks for leading me to this Note, Nicholas!