Skip to Content

The SAP Memory Analyzer team gets asked from time to time about perm space issues and how to solve them. Unfortunately, HPROF binary heap dumps, which are the basis for the SAP Memory Analyzer, don’t contain much valuable information from perm space. The only thing you get are class names and some few bits of information on the classes themselves like their inheritance hierarchy or their defining classloader, but that’s mainly it.

However, there is one query that might help you: Open Query Browser (fourth button on the tool bar) -> Core -> Duplicate Classes. This lists you all classes loaded twice (more precise: all classes with the same package and class name and their corresponding class loader). What also might help (a bit) is to use grouping by class loader in the class histogram or in the dominator tree. Usually perm space problems also manifest in heap.

But if it is just about big classes, then you have a problem. You have a problem fixing it, too, because you can’t do much about big classes. What we found out, though: If you use lots of generators for a model/proxy-driven approach, then you might save a lot of memory by improving the generator templates (and factor out some often repeating code) or, if this suits your use case, deploy them only once instead of with every consumer.

Last but not least, try jmap. This tool is by Sun and may help you with your problem. For us it turned out to be not precise enough as the accounting of the memory that goes to the various perm space pools is pretty poor. Another group at SAP improved jmap for the SAP JVMs. There the accounting of memory to a class is now much more useful, but as said, you have this option only if you run your application on the SAP JVM.

If you even have a heap dump from a SAP Application Server you could use some special features within the SAP Memory Analyzer, e.g. Leaking Loaders (again Open Query Browser -> SAP -> Leaking Loaders) gives you all class loaders that shouldn’t be in memory anymore, but which are. We know about them because we know in our own application server where to look for the still registered class loaders. All other class loaders in heap are unregistered ones which are still hold in memory “somehow”. The SAP Memory Analyzer helps you e.g. with the Paths From GC Roots query to show you this “somehow” in detail. One reason for this behavior could be a poor shutdown of the corresponding service or application. Unfortunately we can’t tell that for an arbitrary heap dump, because this is application server specific.

I hope this was somehow helpful to you. Tracking down perm space issues is kind of hard with heap dumps only, but there are options worth exploring.

To report this post you need to login first.

13 Comments

You must be Logged on to comment or reply to a post.

  1. Chuck Nelson
    Verdan,

    Sun provided various OQL commands to help deduce permanent generation leaks using these JHAT OQL commands – – these same commands fail using SAP OQL – – do you know what the equivalent commands would be?

    http://www.stpmag.com/issues/stp-2007-05.pdf

    OQL QUERY 1

    select map(sort(map(heap.objects(‘java.lang.ClassLoader’),
    ‘{ loader: it, count: it.classes.elementCount }’), ‘lhs.count < rhs.count’),
    ‘toHtml(it) + “
    “‘)

    OQL QUERY 2

    (function() {
    var histo = new Object();
    // replace parameter to findObject() with your object’s
    address
    map(heap.livepaths(heap.findObject(‘0x0f054b90’)),
    function(path) {
    for (var i = 0; i < path.length; i++) {
    var desc = toHtml(path[i]);
    if (desc in histo) {
    histo[desc]++;
    } else {
    histo[desc] = 1;
    }
    }
    });
    var str = “”;
    for (i in histo) {
    str += “”;
    str += “”;
    }
    str += “

    ” + i + “ ” + histo[i] +

    “;
    return str;
    })();

    Thanks, Chuck

    (0) 
    1. Andreas Buchen
      Hi Chuck,

      both things can be done in the Memory Analyzer without coding any OQL.

      ad Query 1:

      In the histogram (in fact, in any view) just select “Group by Class Loader” from the drop-down box and it shows you which class loaders loader how many classes. Drill-down into the tree to see what classes have been loaded.

      ad Query 2:

      I guess this snippet tries to extract the relevant path to the GC roots, e.g. tries to find out why a particular class loader is not garbage collected. Try right-clicking on the object and select “Path to GC Roots -> exclude weak/soft references…”. This will give you a tree view of the shortest paths. The path segments closest to your objects are the ones you wants you want to investigate further.

      Kind regards,

         – Andreas.

      (0) 
      1. Chuck Nelson
        Andreas,

        How does that correlate to the internal class representation (e.g., Klass) and the ‘objects’ on the permanent generation?

        For instance, if I group by classloader and examine the sun.misc.Launcher@AppClassloader object (I’m assuming this is an object since it has a hexadecimal address?) – – it in turn has say 100,000 objects listed in the SAP Memory Analyzer. Does the sun.misc.Launder@AppClassloader object and all of it’s 100,000 ‘referent’ objects have a 1:1 mapping for the internal class representation in the permanent generation? In other words, if I see a large amount of heap consumed by a large number of objects does that roughly correlate to a large amount of consumption in the permanent generation or do you only have a class ‘instance’ represented in permanent generation even though that class may actually have say hundreds of objects allocated on the heap

        Am I confused?

        Let me know,

        Thanks again, Chuck

        (0) 
        1. Andreas Buchen
          Hi Chuck,

          the perm space contains information like methods, constant methods, symbols, method data, object and type array class information. This is information on a class level. So it does not make a difference, if there is one instance or 100 instances of a class, the perm space should be the same.

          You are absolutely right, grouping the histogram by class loader just shows you the number of instances per class loader.

          What you really want is the number of classes loaded per class loader. This is what this OQL snippet will give you:

          SELECT cl.@classSpecificName, cl.@definedClasses.size() FROM INSTANCEOF java.lang.ClassLoader c

          Sorry a/b confusing you.

          Andreas.

          (0) 
          1. Nirmal Kumar
            I have a doubt in understanding the permgen space.

            There are 3 kinds of Generations in the VM memory:
            1.young generation(holds recently created objects)
            2.tenured generation(holds those objects which survived (multiple) major Garbage Collections) 
            3.permanent generation(It is used to store class and method data as well as interned strings)

            Does that mean all the three generations are part of the HEAP?

            If yes then if we increase permGen space then the total available Heap size for (young and tenured generation) will shrink.

            Because I think the equation is like:
            Total Heap Size = youngGen + tenuredGen + permGen

            Am I correct?

            But if this is the case then if we have set max. heap size say 512MB then we cannot set max permGen space to 512MB.

            or heap space is different than permGen space?

            And finally is there a way throgh which we can stop the increasing PermGen? I heard of Leakage Classloaders…..

            (0) 
            1. Vedran Lerenc Post author
              Hi

              yes all three generations are part of the heap, but this is the process heap that is meant (JVM process itself). This process heap is split into Java Heap and Perm Spaces. That’s why heap used sometimes in this meaning (process overall heap memory) and in the other meaning (Java heap memory available to the program running inside the virtual machine).

              To answer your other concrete question, the VM parameters are specific to the Java Heap and Perm Space respectively, i.e. heap does not include perm space. If you set heap to 512 M and perm to 512 M, you need 1024 M overall.

              About the leaking classloaders. Yes, this wastes perm space and the Eclipse Memory Analyzer helps pinpointing this. Example: You deploy an application 10 times and register every time on appl startup a listener somewhere centrally or an object through JNDI. If you forget to unregister properly in appl shutdown, you end up having all your classes loaded 10 times – not to mention the illegal state which you have.

              Hope this was of some help,
              Regards,

              Vedran

              (0) 
              1. Nirmal Kumar
                Hi Vedran,

                Thanks a lot for clearing my doubt about PermGen.

                So now:
                HEAP= Java Heap(YoungGen+TenuredGen) + PermGen

                And we can increase either of the “Java Heap” or “PermGen” by VM parameters separately.

                The problem I am having is that our application has certain memory bottlenecks.

                I am using SAP Memory Analyzer to analyze the heap at certain inportant scenarios.

                The application runs on Tomcat 5 & jdk 1.5

                We need heap dumps at each of the identified test cases i.e. Heap Dump on Demand.

                I am using jmap in jdk 1.5.0_15 using the following command:
                   jmap -heap:format=b 26915

                The problem with jmap in jdk 1.5.0_15 is that it always generates the heap as a binary format with .bin extension(heap.bin)
                And this heap dump taken with jmap in jdk 1.5.0_15 cannot be open or analyzed with any of the tools like SAP Memory Analyzer.

                More ever we cannot migrate from JDK 1.5 to JDK 1.6

                Is there another way to take the “Heap Dump on Demand” so that I can use that heap dump in SAP Memory Analyzer?

                I know that jmap on jdk 1.6 using the following command generates the correct heap dump:
                  jmap -dump:format=b,file=HeapDump.hprof 3312

                Here in this case we can specify the file too.
                So i get the heap dump in “HeapDump.hprof” file in the current directory.

                This file gets opened in SAP Memory Analyzer and thereafter we can start our analysis.

                But the problem is that I have to use jdk 1.5 only….

                Any help will be appreciated.

                Thanks & Regards,
                Nirmal

                (0) 
                  1. Nirmal Kumar
                    Hi Vedran,

                    Thanks again for helping me out.

                    I am now using Eclipse Memory Analyzer instead of SAP Memory Analyzer since it has option for analyzing Heap Dumps taken from JDK 1.4 or 1.5 or higher..

                    Earlier I was using SAP Memory Analyzer which could not open heap dump taken from the following command :
                       jmap -heap:format=b 26915
                    But Eclipse Memory Analyzer can easily open and analyze the same heap dump.

                    Hopefully this tool will help us to analyze the memory related problems.

                    Regards,
                    Nirmal

                    (0) 
                    1. Vedran Lerenc Post author
                      Hi,

                      the Eclipse Memory Analyzer IS the SAP Memory Analyzer. SAP donated it and continues its development in the Eclipse community. It is, so to speak, just newer, more feature-rich, etc.

                      Regards,

                      Vedran

                      (0) 
                      1. Nirmal Kumar
                        Hi,

                        Well I thought these two are different….
                        Thanks for updating me.

                        I have just started to analyze heap dumps and shall start by identifying if my application has leakage classloaders or not as I am getting OutOfMemory PermGen space error..

                        Thanks & Regards,
                        Nirmal

                        (0) 
    2. Chuck Nelson
      Andreas,

      How does that correlate to the internal class representation (e.g., Klass) and the ‘objects’ on the permanent generation?

      For instance, if I group by classloader and examine the sun.misc.Launcher@AppClassloader object (I’m assuming this is an object since it has a hexadecimal address?) – – it in turn has say 100,000 objects listed in the SAP Memory Analyzer.   Does the sun.misc.Launder@AppClassloader object and all of it’s 100,000 ‘referent’ objects have a 1:1 mapping for the internal class representation in the permanent generation?   In other words, if I see a large amount of heap consumed by a large number of objects does that roughly correlate to a large amount of consumption in the permanent generation or do you only have a class ‘instance’ represented in permanent generation even though that class may actually have say hundreds of objects allocated on the heap

      Am I confused?

      Let me know,

      Thanks again, Chuck

      (0) 
  2. Nirmal Kumar
    The String objects that are interned also go into the PermGen space.

    Is there any tool in JDK 1.5 (beside jconsole) or any Profiling tool to see these interned Strings?

    I m using Eclipse MAT to analyze heap dumps but I also want to see how many interned Strings are there in the PermGen space as i am getting OutOfMemory:PermGen space error in my application.

    Any help…

    Thanks & Regards
    Nirmal

    (0) 

Leave a Reply