Memory – is it Real, or is it Virtual?
We’re in the midst of a hardware upgrade, replacing the previous generation with the next one, going from partly virtualized to 100% virtualized for all of our SAP systems, and many non-SAP systems. I’m working on a blog on that project, currently waiting for a few trends to shake out so I can share what we’ve learned. In addition to that project, we’re upgrading SCM from 4.1 to 7.0 (also possibly known as SCM 2007 — I am not really sure as the OSS notes mainly say 7.0). Doing these both simultaneously can make benchmarking more challenging than either in isolation.
[see prior blog on SCM performance monitoring – Measuring SCM ATP Workload Impact on R/3 ]
As part of our quality assurance testing, we’ve brought in a consultant (Suresh R.) who has been reviewing how our systems and applications look prior to the production upgrade to SCM 7. Right now, QA is on new hardware (IBM Power6), with SCM 7.0, and production is on new hardware, with SCM 4.1. Comparing the two environments is tricky, especially as we’ve generally found volume testing with APO/GATP to be unreliable as a predictor of actual workloads.
Suresh wrote a first draft report analyzing our memory allocations, focused more on the SCM side of the suite than the LiveCache side. Application components such as real time pricing/availability checks and batch runs such as back order processing are mission critical, as we’ve learned during earlier software stability battles. I offered to review his recommendations, and set up an hour to chat with him and the Basis team upgrade lead.
Back In The Day
10 years ago or so, as SAP released a new version of their enterprise software, you got a nice (though general) note describing how much more hardware you were likely to need to throw in the data center to keep the same performance level as the old version. If it was 10% more CPU, and you had 5 already running, you needed one more. Or maybe two. Better get two just in case. Memory? Same deal. Have 2GB? Better go to 3GB. Or 4GB to be safe. And the notes were generally helpful, though you must always (and I mean always) have your own measurements to satisfy both your users and your financial approval chain. Just pointing to an SAP note doesn’t cut it.
- 323263 – Resource Requirements for Release 4.6 C SR1
- 517085 – Resource Requirements for SAP R/3 Enterprise 47×110
- 752532 – Resource Requirements for SAP R/3 Enterprise 47×200
- 778774 – Resource requirements for SAP Enterprise Core Component 5.0
- 901070 – Resource requirements for SAP Enterprise Core Component 6.0
- 1311835 – Resource requirements for SAP ERP Central Component 6.0 EHP4
Are there nice, neat recommendations for SCM? Don’t bet on it. Here’s what I have found (without reading the Upgrade Guide – I’ll leave that until I can’t sleep one night):
- 869651 – Prerequisites for upgrading to SCM 5.0
- 1021662 – Release Restrictions for SCM 2007 [replaced by 1413545]
- 1413545 – “The requested SAP Note is either in reworking or is released internally only” – Ooops.
We got into a discussion about how much memory is needed on the DB/CI node. Let’s see, there are 2 Oracle databases (one for ABAP, one for Java stack); there’s the ABAP stack, the Java stack, the O/S, and oh yeah, the Oracle shadow processes.
|Oracle DBs – shared pool and block buffers||6|
|Shadow processes (100 @ 100MB each||10|
Except I don’t believe these numbers for a second. Suresh did a calculation, and came up with 15 GB, which I still think is too high. He was looking at vmstat, which told him how much memory had been touched, not what we needed. I’d look at this resource, for example, to see how to interpret vmstat:
“When determining if a system might be short on memory or if some memory tuning needs to be done, run the vmstat command over a set interval and examine the pi and po columns on the resulting report.”
“constantly non-zero, there might be a memory bottleneck”
The key phrase here is “might be”. While it’s great to keep your finger on the pulse of the system, ask the patient how they are doing. In this case, compare batch runs before and after the software upgrade. If they are about the same or better (adjusting for hardware differences, easier said than done), great. If they are worse, find out if the application is suffering from memory faults or not. Ask the application team when they are running tests and monitor the system closely.
My sense it that we still need work configuring the parallelism of SCM, including RFC groups and other workload balancing efforts. If I see memory interfering with that. I can ask for more. But in the meantime, I’m unconvinced the hardware is at fault.
AIX specific memory and tuning notes