heap

Introduction to Java heap tuning – Some easy steps to improve response times

At our current project we wanted to upgrade our EC2 instance to a newer family and generation to improve response times and make our application start faster. This simple task started with blindly increasing the heap size and ended with counting strings on the heap.

A few months ago our cpu usage was running a bit hight and we thought it was caused by garbage collection (since the memory usage was high). We increased the heap size of some jvm’s and our problems seemed solved. A few weeks later we wanted to upgrade our r4.large (15.25GB RAM) instances to to m5.large (8 GB RAM). The m5 instances are cheaper, have better response times with our application and start our application faster. Due to the earlier ‘fix’ we were about 1GB short to run on a 8GB machine. Since the cpu usage didn’t change much after the ‘fix’ I decided to use a less blunt weapon.

A bit of background information: we’re running about 15 jvm’s on a single node and most interactions with those jvm’s are stateless.

Before you start your tuning you should equalize the -Xms and -Xmx (starting and max heap size) to make sure you’re measuring the right thing. Another thing to keep in mind is to keep the memory usage below 90%. We haven’t figured out yet what happens at this limit (why it is 90% and whether it applies to your environment), but it makes the measurements less reliable (and you need some slack anyway for OS caching and buffering).

JStat

My starting point was the excellent article ‘How to Tune Java Garbage Collection’. The article left us with 4 simple rules supplemented with my own rule (the last one).

1. Minor GC is processed quickly (within 50 ms).
2. Minor GC is not frequently executed (about 10 seconds).
3. Full GC is processed quickly (within 1 second).
4. Full GC is not frequently executed (once per 10 minutes).
5. GC time should not exceed 1% cpu usage.

These rules are just rough guidelines and are a starting point to begin the tuning. In practice you will probably find your own numbers that match your application.

The GC numbers can be found by executing the jstat command with a pid as parameter (ie jstat -gc 13214)

An example result :

The columns are explained here.

The average minor GC time (1) is YGCT divided by YGC. Full is FGCT divided by FGC (3).
For the other times you need the uptime of the jvm (so NOT the uptime of your machine).
Minor frequency (2) is the uptime in seconds divided by YGC. Full is the uptime in minutes divided by FGC (4).
The cpu usage (5) is GCT divided by the jvm uptime.

Visualizing of JStat output

Since our application is an application for high school students our peak load is between 8:30 and 14:00, so it makes sense to tune the application for this period. You can achieve this by running jstat twice, once at 8:30 and once at 14:00 and subtract the differences (the -t option will add a timestamp to the result). Now you also can calculate the cpu usage for GC during that period.

Another useful tool is jstatplot. This tool draws a plot of the statistics. When you want to see things about young collection make sure the frequency is high enough to be useful (about 200ms).

This image is output from our application. The red color is the old space. As you can see it’s pretty nice saw tooth and it’s relatively small compared to the young space.

Changing the ratio young/old

The heap space is roughly divided in young and old space. Objects with a short life time live in the young space, older objects live (or are promoted) to the old space. Our application is stateless, so we can expect relatively more young objects.

The first step in tuning the ratio is -XX:NewRatio. The default ratio is 2:1 (2 parts old, 1 part young). This effectively means 66.7% of the space is dedicated to old objects. The only improvement we can make here is changing 2 to 1 for a 1:1 ratio, or 50% each.

To tune more accurately (or if 50% is still way too much) you can use -XX:NewSize where you can set an exact amount of MB’s. Note that this number cannot be changed, so it’s wise to have -Xms and -Xmx the same size. Another warning is that when some fool decides to increase the heap size he probably forgets to change the -XX:NewSize.

Verify your changes

When the numbers of the 5 numbers from jstat are within the margins it’s also wise to keep track of the total cpu and memory usage of your node and response times of your application. Since we’re running 15 jvm’s and our performance is more than acceptable we decided to level all the numbers so each application spends about the same time garbage collecting. This probably can be improved with some nifty machine learning algorithms, but since we’re running on EC2 it is probably cheaper to just add another machine.

Conclusion

Don’t blindly increase the heap size, it can actually harm your response times. Always verify your changes (or better, do A/B testing).
Don’t go overboard with tuning every MB of the heap. Keeping things fool proof is usually a better choice (-XX:NewRatio for example grows with the heap, -XX:NewSize doesn’t, but is more accurate). Also keep in mind that devops-time is often more expensive than adding another node.

I also did some experiments with the G1 garbage collector and -XX:+UseStringDeduplication (after analysing a heap dump), but this didn’t improve things (but they might in your situation, so keep it in mind as a next step).

Sources

Sun Java System Application Server Enterprise Edition 8.2 Performance Tuning Guide
Advanced JVM and GC tuning
Heap tuning parameters
Java SE Tools reference