Monthly Archives: April 2016

Oracle Cloud – Querying Glassfish Memory Usage

In a previous post I posted about a Java Heap memory issue we were having with Glassfish.

One of the ways we tracked down the cause of the issue and by how much to increase the memory by was to use the asadmin command to generate a memory report – which is a really useful way to see a snapshot of how your Glassfish instance is consuming memory.

You can run a command like

./asadmin generate-jvm-report --type memory

which should give output similar to

[oracle@ahi-dhh-prod bin]$ ./asadmin generate-jvm-report --type memory
Enter admin user name> admin
Enter admin password for user "admin">
The uptime of Java Virtual Machine: 47 Hours 2 Minutes 44 Seconds
Memory Pool Name: PS Eden Space
Memory that Java Virtual Machine initially requested to the Operating System: 122,683,392 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 239,599,616 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 264,241,152 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 45,212,424 Bytes

Memory Pool Name: PS Survivor Space
Memory that Java Virtual Machine initially requested to the Operating System: 20,447,232 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 2,097,152 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 2,097,152 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 312,976 Bytes

Memory Pool Name: Code Cache
Memory that Java Virtual Machine initially requested to the Operating System: 2,555,904 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 17,956,864 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 50,331,648 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 17,726,208 Bytes

Memory Pool Name: PS Perm Gen
Memory that Java Virtual Machine initially requested to the Operating System: 67,108,864 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 99,614,720 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 201,326,592 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 99,278,352 Bytes

Memory Pool Name: PS Old Gen
Memory that Java Virtual Machine initially requested to the Operating System: 326,631,424 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 441,974,784 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 536,870,912 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 372,432,888 Bytes


Name of the Garbage Collector: PS MarkSweep
Number of collections occurred using this garbage collector: 244 Bytes
Garbage Collection Time: 37 Seconds 768 Milliseconds
Name of the Garbage Collector: PS Scavenge
Number of collections occurred using this garbage collector: 54,911 Bytes
Garbage Collection Time: 330 Seconds 761 Milliseconds

Heap Memory Usage:
Memory that Java Virtual Machine initially requested to the Operating System: 489,924,096 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 683,671,552 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 716,177,408 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 418,149,376 Bytes

Non-heap Memory Usage:
Memory that Java Virtual Machine initially requested to the Operating System: 69,664,768 Bytes
Memory that Java Virtual Machine is guaranteed to receive from the Operating System: 117,571,584 Bytes
Maximum Memory that Java Virtual Machine may get from the Operating System: 251,658,240 Bytes. Note that this is not guaranteed.
Memory that Java Virtual Machine uses at this time: 117,004,560 Bytes

Approximate number of objects for which finalization is pending: 0


Command generate-jvm-report executed successfully.

As you can see there’s a lot of output here, but I’ve highlighted the section which shows how much Heap Memory we’re currently using and what the maximums are set to. By regularly monitoring these values we were able to hone in on how much memory was the optimal amount to allocate to the Glassfish instance.

Oracle Cloud – Glassfish Heap Memory Issues

I recently encountered some issues with our Oracle Cloud Glassfish server encountering out of memory issues.

This manifested itself by Glassfish becoming unrepsonsive and eventually crashing, digging into the logfile we found entries like this

server.log_2016-04-08T18-32-27:[#|2016-04-04T18:21:56.472+0000|SEVERE|oracle-glassfish3.1.2|null|_ThreadID=74;_ThreadName=Thread-2;|Java heap space
server.log_2016-04-08T18-32-27:java.lang.OutOfMemoryError: Java heap space

After a lot of Googling, I found that you can find out the current Java settings that are being used by Glassfish by running the list-jvm-options command, like this:

[oracle@prod bin]$ ./asadmin list-jvm-options
Enter admin user name> admin
Enter admin password for user "admin">
-XX:MaxPermSize=192m
-XX:PermSize=64m
-client
-Djava.awt.headless=true
-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
-XX: UnlockDiagnosticVMOptions
-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dosgi.shell.telnet.port=6666
-Dosgi.shell.telnet.maxconn=1
-Dosgi.shell.telnet.ip=127.0.0.1
-Dgosh.args=--nointeractive
-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
-Dfelix.fileinstall.poll=5000
-Dfelix.fileinstall.log.level=2
-Dfelix.fileinstall.bundles.new.start=true
-Dfelix.fileinstall.bundles.startTransient=true
-Dfelix.fileinstall.disableConfigSave=false
-XX:NewRatio=2
-Xmx128m
Command list-jvm-options executed successfully.

You can see a lot of info here, the Heap memory parameter is the Xmx one, which we can adjust by deleting the current setting:

./asadmin delete-jvm-options --target server-config -- '-Xmx128m'

and then assigning a new value

./asadmin delete-jvm-options --target server-config -- '-Xmx256m'

 

Then we restarted the Glassfish server and haven’t seen the issue occur since.
It’s important to not just blindly choose a value here, you need to understand why you’re running out of heap memory and not just increase it for the sake of it (but that’s a post for another day).