Monitoring JBoss/Wildfly, Tomcat and other application servers with JMXetric

NOTE: this page has been updated now there has been feedback about JBoss issues and improvements to integration with JBoss now make it significantly easier to integrate. It is no longer necessary to customize JBoss logger settings in the JVM command line in order to use JMXetric with JBoss if you use JMXetric 1.0.6 or later and set initialdelay to a suitable value.

JMXetric is the Java JMX profiling agent for Ganglia. Let's have a quick look at how conveniently it can be integrated with an application server.

A more thorough analysis of gmetric4j and JMXetric can be found in the sections I contributed to the new O'Reilly book Monitoring with Ganglia

General comments about JVM instrumentation and the JVM boot classpath

Monitoring agents (there are a number of them out there) are loaded at an early stage by the JVM and it needs to have access to any of their dependencies through the boot classpath.

JMXetric explicitly declares it's dependencies in MANIFEST.MF:

  Boot-Class-Path: oncrpc-1.0.7.jar gmetric4j-1.0.4.jar

As no absolute path is specified, the JVM will look for those JARs in the same directory where the jmxetric.jar is located.

Furthermore, the JVM will expect the exact filenames to match. If other filenames are used (e.g. for a newer version of gmetric), then the JVM command line needs to specify the filenames and locations, for example:

  -Xbootclasspath/a:/opt/ganglia/jmxetric/gmetric4j-1.0.4.jar

Starting JBoss 7 or Wildfly (JBoss 8) with JMXetric

To start with, unpack a vanilla JBoss distribution:

$ mkdir ~/opt $ cd ~/opt $ tar xzf ~/Downloads/jboss-as-7.1.0.Final.tar.gz

Now get the JMXetric code and put it, together with dependencies, in the same directory:

$ mkdir -p ~/opt/ganglia/jmxetric $ cd ~/opt/ganglia/jmxetric $ wget http://search.maven.org/remotecontent?filepath=info/ganglia/jmxetric/jmxetric/1.0.6/jmxetric-1.0.6.jar $ wget http://search.maven.org/remotecontent?filepath=info/ganglia/gmetric4j/gmetric4j/1.0.4/gmetric4j-1.0.4.jar $ wget http://search.maven.org/remotecontent?filepath=org/acplt/oncrpc/1.0.7/oncrpc-1.0.7.jar

Set up a config file for JMXetric:

$ mkdir -p ~/opt/ganglia/jmxetric/etc $ cd ~/opt/ganglia/jmxetric/etc $ wget https://raw.github.com/ganglia/jmxetric/master/etc/jmxetric.xml $ cp jmxetric.xml jmxetric-jboss.xml $ vi jmxetric-jboss.xml

Review the prefix you want to use for the metrics, change it from the default value, ProcessName to something like jboss7 or wildfly8. Also make sure the setting of initialdelay is longer than the time it takes for the application server to start see this issue for details):

                %lt;jmxetric-config>
                        <jvm process="jboss7"/>
                        <sample initialdelay="20" delay="20">

and further down in the file, insert the following code:

                <!-- report web server request count -->
		<mbean name="jboss.as:subsystem=web,connector=http" pname="http">
			<attribute name="requestCount" type="int32" slope="positive"/>
		</mbean>

Finally, tell JBoss about the agent:

$ cd ~/opt/jboss-as-7.1.0.Final $ vi bin/standalone.conf

and apply the following changes to standalone.conf:

--- bin/standalone.conf.orig	2014-03-04 10:06:50.357481691 +0100
+++ bin/standalone.conf	2014-03-04 10:06:33.117472047 +0100
@@ -46,10 +46,17 @@
 #
 # Specify options to pass to the Java VM.
 #
+JMXETRIC_CFG="${HOME}/opt/ganglia/jmxetric/etc/jmxetric-jboss.xml"
+JMXETRIC_ARGS="host=239.2.11.71,mode=multicast,port=8649,wireformat31x=true,config=${JMXETRIC_CFG}"
+JMXETRIC_JVM_ARGS=" \
+  -Xbootclasspath/a:/usr/share/java/oncrpc.jar \
+  -Xbootclasspath/a:/usr/share/java/gmetric4j.jar \
+  -javaagent:/usr/share/java/jmxetric.jar=${JMXETRIC_ARGS}"
 if [ "x$JAVA_OPTS" = "x" ]; then
    JAVA_OPTS="-Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
    JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
    JAVA_OPTS="$JAVA_OPTS -Djboss.server.default.config=standalone.xml"
+   JAVA_OPTS="$JAVA_OPTS ${JMXETRIC_JVM_ARGS}"
 else
    echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
 fi

Now go ahead and start JBoss, you should see a log message informing you that JMXetric is operational:

$ cd ~/opt/jboss-as-7.1.0.Final
$ chmod a+x bin/*.sh
$ bin/standalone.sh


=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: ...
.
.
.
=========================================================================

JMXetricAgent instrumented JVM, see https://github.com/ganglia/jmxetric
12:59:34,536 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:59:34,617 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:59:34,653 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
.
.
.

I hope this example is useful for you. Please feel free to share recipes you develop for collecting useful JVM metrics, and also any custom charts/reports through collaboration with the Ganglia community mailing list, the Ganglia wiki or your own blog.