Lately I’ve been doing a lot of performance testing. It’s fun to watch the metrics in real time and test the limits of the system but when things goes wrong it can be a real challenge to find the root cause. There are so many variables you can isolate. It’s easy to say “oh let me just try one more thing” and fall down a rabbit hole of trial and error. Oftentimes the hardest part is taking a step back and reevaluating the situation.
Over the last week I’ve been looking at some very suspicious memory use in a Kubernetes pod. The service in question was using an awful lot of memory and didn’t appear to be releasing any of it. I began to worry it was due to a memory leak or some nasty threading; two problems I want nothing do to with. Countless configuration changes were tested and the system was boiled down to the simplest possible setup. Every day I tried new things and before I knew it I was doing a lot more than just making minor changes to the configuration. It was time to go back to the drawing board.
I have an embarrassing thing to share. I’ve spent the last 5 years developing in Java and have never used a profiler. My poor track record of configuring things and lack of basic networking knowledge has successfully scared me into never touching a profiler. It’s amazing how far you can get with so little in your toolbox. I must be super persistent, or just really stubborn. It’s likely the latter. I am a developer after all.
At this point I had enough. It was time to put my big girl pants on and look at what Java was doing under the covers. Better late than never, right?
Below are the steps I took to create a simple Java app, deploy it to Docker, and attach the VisualVM profiler on my Mac. I choose VisualVM because it uses Java Management Extensions (JMX). JMX is part of Java’s built in tools making it a free, easy, and standard way to monitor Java applications. There’s very little needed to get basic functionality up and running. There are other great paid profilers out there that do much more such as YourKit and JProfiler. JConsole is another good one that’s very similar to VisualVM but doesn’t do quite as much.
Step 1: Create a deployable JAR
Start with this super basic daemon that prints “Hello World” every 2 seconds.
com/test/helloworld/HelloWorld.java
package com.test.helloworld; public class HelloWorld { public static void main(final String[] args) { while (true) { try { System.out.println("Hello World"); Thread.sleep(2000); } catch (Exception e) { break; } } } }
I’m often in an IDE and will use Eclipse’s ‘Export’ tool to create jars but if you want to get back to basics, or if you have never done manually then check out my post on how to create a JAR the old fashioned way.
Step 2: Package the JAR into a Docker image
To package the jar into a Docker image we need to create a Dockerfile. The example below uses relative paths so if you are copying it word for word make sure your jar is in the same directory as the Dockerfile.
Dockerfile
FROM openjdk:8-alpine ADD HelloWorld.jar HelloWorld.jar CMD java -jar HelloWorld.jar
openjdk is used for the base image and the HelloWorld jar has been added to it. Now we need to turn JMX on. Add the following properties to the run command so the jar is exposed for remote management. In this example I have chosen port 9010. Make sure whatever port you pick is unused.
CMD java \ -Dcom.sun.management.jmxremote=true \ -Dcom.sun.management.jmxremote.local.only=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Djava.rmi.server.hostname=localhost \ -Dcom.sun.management.jmxremote.port=9010 \ -Dcom.sun.management.jmxremote.rmi.port=9010 \ -jar HelloWorld.jar
Note that there is no security here so don’t open any ports like this when you expose your code to the world. For more information on implementing JMX security mechanisms, see this documentation.
Lastly, at the Docker container level expose a port for the JMX (Remote Method Invocation aka RMI) connections. This will give VisualVM (the JMX client that you run locally) something to connect to.
EXPOSE 9010
The final Dockerfile looks like this:
FROM openjdk:8-alpine ADD HelloWorld.jar HelloWorld.jar EXPOSE 9010 CMD java \ -Dcom.sun.management.jmxremote=true \ -Dcom.sun.management.jmxremote.local.only=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Djava.rmi.server.hostname=localhost \ -Dcom.sun.management.jmxremote.port=9010 \ -Dcom.sun.management.jmxremote.rmi.port=9010 \ -jar HelloWorld.jar
Now we can build the Docker image. If you haven’t started Docker yet, do so now. I use Docker for Mac and really like it. You can use Docker Machine if you have an older operating system but Docker for Mac is so easy that I wouldn’t recommend using Docker Machine unless I had to.
Run this command from the same directory as the jar and Dockerfile. Here I have named the image ‘helloworld’.
docker build -t helloworld .
As you can see it made its way to the local registry:
docker images REPOSITORY TAG IMAGE ID CREATED SIZE helloworld latest 665cf7fb36d2 15 seconds ago 102MB
Now we can start the container. The most important thing is to forward port 9010, the JMX port that we opened up the Dockerfile.
docker run -d -p 9010:9010 --name helloworld helloworld
-d runs the container in detached mode, and –name assigns a name to the container so we aren’t assigned one of those awesome but not very useful generated Docker names.
Verify that the container is up and running:
docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ffc9f4a96631 helloworld "/bin/sh -c 'java ..." 9 seconds ago Up 8 seconds 0.0.0.0:9010->9010/tcp helloworld
Step 3: Connecting to VisualVM
Download VisualVM if you haven’t already.
Open VisualVM and enter “localhost:9010” as a new local JMX connection.
It will show up as a new connection on the panel to the left and that’s it! Have fun diving deeper into garbage collection, thread usage, the CPU, the heap, metaspace, and the rest of the goodies VisualVM has to offer.
