SMP 3.0: Analyze and adjust JVM heap memory
All.
I have done some findings in the area of JVM memory analysis and adjustment that I wanted to share.
Background
In the Agentry Server prior to SMP one had the option to adjust the JVM heap memory size of each Agentry Server by adjusting the corresponding parameters of the Agentry.ini file.
The parameters were:
[Java-1]
initialHeapSize=
maxHeapSize=
Now on SMP 3.0 the Agentry Server(s) are using the same runtime as the SMP, and you can no longer set memory allocation in the Agentry.ini (as it no longer exist). And you cannot set it in the Management Cockpit either, eventhough you have the [Java-1] section in your Backend configuration part of your Application definition.
So how do you analyse and adjust the JVM heap memory of the SMP 3.0 ?
Analyze
Please note that the SMP 3.0 is using the SAP Java runtime JVM.
Fortunately a simple Analyzer tool is included in the SMP installation.
The tool is located in <SMP_HOME>\sapjvm_8\bin\jconsole.exe
If you start the tool directly, it will display Java processes executing in your user, and you can connect directly. Normally you would have to start the SMP by Go.bat.
If your SMP server is running as a service it will normally be executing under a different user (smpServiceUser), and you will need to start jconsole a little differently.
Start the Task Manager, and determine the process ID of the java.exe process hosting your SMP platform.
In my case the SMP java.exe is executing under process ID 1632.
Now open a command prompt and navigate to the folder where jconsole is located.
Issue the command “jconsole <PID>”, where <PID> is the process ID of the SMP java process.
You will get this warning, but just click “Insecure connection”.
You are located directly on the host so I don’t see any problem in this.
You will now get an overview, and you can navigate to multiple tabs for various information.
On the “VM Summary” tab, you can find info of the current and maximum heap size, as well as all kinds of other info regarding the VM.
Note:
In Windows (if you have no other Java processes running) you can use this script (attached) to start the JConsole view:
set SMP_ROOT=<Path-to-SMP-root>
cd /d %SMP_ROOT%\sapjvm_8\bin
for /f “tokens=2” %%i in (‘tasklist /nh /fi “imagename eq java.exe” ‘) do set PID=%%i
jconsole %PID%
Adjusting memory
If your server is running out of heap memory (java.lang.OutOfMemoryError), you may want to increase the heap memory allocation on the SMP.
Startup parameters for the JVM is set in the file: <SMP_HOME>\Server\props.ini
Default memory parameters are:
-Xms1024m Initial Heap size
-Xmx2048m Max Heap size
-XX:PermSize=256M Initial memory allocation for permanent objects
-XX:MaxPermSize=512M Max memory allocation for permanent objects
By adjusting the value of Xmx you can change the max allowed memory allocation for the JVM.
It is good practice to set Xms to 50% of the Xmx value (as far as I know).
A restart of the SMP server is needed for the changes to take effect.
I hope this is helpfull.
Søren Hansen
Soren,
Thanks for writing this up!
--Bill
Søren Hansen
Above article is really helpful. We are currently experiencing SMP server issue for which we need to restart the SMP server. As per customer, by looking at logs, issue is related to heap memory.
By looking at Analyzer tool, i can see below details;
Current heap size: 483,493 kbytes
Maximum heap size: 1,864,192 kbytes
but looking at props.ini, default values are:
-Xms1024m
-Xmx2048m
My question is what are the factors to keep in mind before increasing the Xmx value?
RAM size is 4 GB
Regards,
JK
Jitendra,
I know it will depend on the type of applications you are running on the server but 4GB is probably at the very low end. Have you done a server sizing for the types of applications you are running?
Upgrading the memory may also be needed to address the problem.
--Bill
Hi JK.
I must admit that I am not an expert on this topic, but with only 4 GB of physical RAM, I doubt that it would be adviceable to increase the Heap Memory size.
I agree with Bill, that 4GB sounds to be very low.
Remember that the JVM will allocate more memory, than just the Heap memory.
Check Configuration and Setup of SAP JVM - AS Java (Application Server Java) - SAP Library for details of the JVM memory allocation.
I think that one should be careful with increasing Heap Memory above 4GB, but on 64-bit systems I think this is an OK setting (provided you have available RAM for it).
So if you could get 8 or perhaps 16 GB of RAM, I think there is no problem in increasing Heap memory to 4GB. Values above this would require further analysis. Remember that high Heap Memory, will increase the time for Garbage Collection. And full GC's may include periods where the JVM is not responding, so you want these to finish as fast as possible.
I hope this helps.
Søren Hansen
Hi guys,
I have this issue in a server type production with 24GB of memory. I did the tuning of JVM parameters at props.ini file as SAP documentation suggested and.. just a couple of hours after the change, the server collapsed and crashed. I have only running o that server SAPIM application over a SMP 3.0 SP3 with Oracle database. The server crash is not all the time only once or twice per month. What may be wrong here?
Best regards to all,
MC
cc: Bill Froelich Michael Appleby Jitendra Kansal
Hi Mariana,
Please create a new Discussion marked as a Question. Comments to a Blog (or Document) is not the right vehicle for asking questions as the results are not easily searchable. Once your issue is solved, a Discussion with make the results much more visible to others experiencing a similar problem. If a blog or document is related, put in a link.
NOTE: Getting the link is easy enough for both the author and Blog. Simply MouseOver the item, Right Click, and select Copy Shortcut. Paste it into your Discussion. You can also click on the url after pasting. Click on the A to expand the options and select T (on the right) to Auto-Title the url.
Thanks, Mike (Moderator)
SAP Technology RIG
Hi Jitendra,
Back in the old days of NW Portal (J2EE 1.4) I struggled with that as well.
The developers gave me a hint that latter became an SAP Note on the recommend settings for the JVM used by Portal.
I think we can trace a parallel approach to SMP as it uses Java just the same as Portal
So here is my two cents on this topic:
1) It doesn't matter if your server has a huge amount of memory to be allocated by the JVM. You should not allocate more memory than it can manage in a full GC cycle.
2) It never hurts to say that the JVM must fit entirely in the free memory (the memory must be enough to handle OS and applications/services running on it and the JVM altogether without swapping - so no tricks with virtual memory allowed here).
3) The full GC Cycle should not take more than 10s to complete. If a full GC is less than 10s, than a Minor CGs will be faster so no worries about it.
Basically what this means is that a JVM should be as large as the CPU can handle the CGs times to maintain the JVM at usable state. Keep in mind that during a Full GC the JVM freezes all running threads to ensure memory integrity. If your CPU can handle the full GC under 10s with 24GB of RAM that should be just fine. However, if you have a machine with loads of memory but no CPU to handle it, than it would be wise to distribute it between several server nodes instead of a single instance. Let's say your server starts increasing the GC times above 10s with a heap gratear than 12GB and your server has 24GB than it is time to think about splitting that machine in two - simply because you won't be able to use the other 12GB within that same box.
The JConsole is one approach to monitor the JVM's Heap, CPU and GC Times. However, most companies that implement SAP already have Wily Introscope installed. The advantage is that you can monitor the SMP instance constantly and look at peak hours later on to better determine the best Heap size for your SMP usage. It is way much better using the historical data of a server being monitored than doing it "manually" like I explained above. However, even with such tools, this is an empirical task which involves several restarts and assertions. I believe that you can setup a JVM parameter to display full GC times as well, but the graph is much too cool.
Regards,
Ivan
Starting with SMP 3.0 Server SP05 you can now adjust the JVM parameters directly from the Management Cockpit.
--Bill
Hello Søren,
I had follow your document and modified props.ini file as below.
...............................
#jvm
-XX:+HeapDumpOnOutOfMemoryError
-XX:+DisableExplicitGC
-Xms2048M
-Xmx4096M
-XX:PermSize=256M
-XX:MaxPermSize=512M
-XX:ErrorFile=./log/error.log
-XX:HeapDumpPath=./log/heap_dump.hprof
-Djava.io.tmpdir=./work/tmp
-Djava.endorsed.dirs=lib/endorsed
-Dosgi.logfile=./log/osgi.log
-DsecretKey=aNwS5L1xgu
-DsecretKeylength=128
-Dfile.encoding=UTF-8
#main
org.eclipse.equinox.launcher.Main
#program
-console localhost:2401
................................
My machine is loaded with 16giga RAM. I see the values are reverting to -Xms1024M/-Xmx2048M after restarting the SMP service (....\SAP\MobilePlatform3\Server\go-service.bat).
Could you/anyone let us know cause for this. Do i need follow any addition steps to make this change.
I had tried the steps posted in Heap size issue for JVM which doesn't help me.
Regards
Pavan.Venkatt
Hello All,
I have modified the values in SMP cockpit. This change directly updated the props.ini file. I can see the updated values being stable after restarting the service also.
Thanks All.
Pavan.Venkatt
All,
There may be questions about the SMP 3.0 SP09 documentation stating that the Java heap size min and max should be the same.
(See link -> http://help.sap.com/saphelp_smp309svr/helpdata/en/4c/3382a34c6147f8a64c00a2bfd95068/content.htm)
For explanation about the red entry (picture above), see link below.
http://service.sap.com/sap/support/notes/2321376
Cheers,
Mark Pe
SAP Platinum Support Engineer
Soren,
In the case of 20 concurrent users,the “Java.exe” process usage is Ninety-nine percent,
but looking at props.ini, default values are:
-Xms8192M
-Xmx16384M
-XX:PermSize=256M
-XX:MaxPermSize=512M
Could you let me know cause for this
-kevin