Why HTTP/2 with TLS is not supported properly in Java 8 – And what you can do ab...

Why HTTP/2 with TLS is not supported properly in Java 8 – And what you can do about it

Recently I got a bit frustrated that Undertow is still the only HTTP/2 server in Java that properly supports HTTP/2 with TLS. Today I found out I was being unreasonable. After watching a talk it appeared there’s a good reason for it: TLS in Java cancels a large part of the latency improvement of HTTP/2.

updated @ 17-nov-2018 : This article is covers Java 8, with higher versions things should run way more smoothly.

We recently decided to create our first micro service with HTTP/2 support. 90% of our current micro services run on Spring-boot with an nginx proxy for TLS offloading. Since Spring usually picks very reasonable defaults we started of with the default Tomcat implementation (note that you need at least version 1.4.0 of Spring boot to get Tomcat with HTTP/2 support).
After a lot of experimenting and puzzling the only web server that supported HTTP/2 over TLS was Undertow. I saw I lot of plain text (non-TLS) support, but always thought that was a bad idea.

h2 vs h2c

All major browsers that support HTTP/2 only support h2, which is HTTP/2 over TLS. Plain text (or clear text, hence the c in h2c) is not supported. At first this might sound strange, but this helps the adoption of https. H2c was developed for the communication between web server and proxy. Sending HTTP over plain text is usually a bad idea because someone with a packet sniffer can easily listen in on your connection (for example on unprotected wifi). So why is h2 support so limited in Java?

Don’t let java do the TLS offloading

After watching a talk by Simone Bordet things made a lot more sense. According to Simone the preferred way to setup an application server is a java web server with h2c and an HAProxy for TLS offloading.
Since the java implementation is “Memory hungry and not that efficient” HAProxy is a better choice to do the offloading.

After watching the talk, HAProxy seemed the way to go. I still was very hesitant to use it since the logo and web site look like they were never updated since Google was launched (that was in 1998). But looks can be deceiving.

Setup HAProxy and Undertow (or Tomcat)

To setup HAProxy the Jetty documentation is excellent. Just follow the instructions until the paragraph of the Jetty setup. Mac users can install HAProxy by running ‘brew install haproxy‘ (but still need the configuration and certificates).

To setup spring-boot use the github project by Toshiaki Maki. Exclude all ssl entries from the application.properties and change the port number to 8282 (this is the port number used in the HAProxy configuration).
Do not follow the README.md since we don’t need the -Xbootclasspath because we’re using plain text instead of SSL, you can just run the DemoHttp2Application from your IDE.

I also tested the configuration with Tomcat and that works fine. Make sure you use version 1.4.x or higher of spring-boot and ‘overwrite’ the bean in DemoHttp2Application :

I gave Jetty a shot, but most hits on Google go back to my own blog and this stuff shouldn’t be more than a few lines of extra code in my opinion. So kudos to Undertow and Tomcat for making it so easy. Note that switching implementations probably needs a restart of HAProxy since I got strange results capturing packets with Wireshark (probably because HTTP/2 keeps a TCP connection open for a while).

To run everything start the DemoHttp2Application from your IDE just like a normal Java class.
Now HAProxy has to be started. On mac os X the command is :
sudo haproxy -f haproxy.cfg

Since you’re using port 443 for SSL you need sudo rights. When this is not possible pick a port higher than 1024.

Big warning

One of the biggest pitfalls in the proxy+server setup is to don’t have HTTP/2 traffic between servers. In your browser everything looks fine and your Java web server isn’t very clear about what kind of HTTP it’s serving. It degrades so gracefully to HTTP/1.1 you won’t even notice.
The best thing to do is test your setup with wireshark. It might scare you at first, but in my experience it’s your best bet to make sure there is HTTP/2 everywhere. You can add a filter : ‘tcp.port==8282‘, this will only show you the traffic between HAProxy and your Java web server.

A capture of a wrong setup looks like this :

The red flag you’re looking for is HTTP/1.1, so that’s quite easy to spot.

An HTTP/2 capture is a bit more opaque :

What you have to look for here are the words in capitals that are in the TOC at chapter 6 of the RFC.
And of course the word Magic is a great help, if you inspect this packet it will mention HTTP/2.0.

Ready for a test drive

Now that you’ve set up everything it’s time for a test drive. First test things in your browser or with Curl and then hook up Wireshark to make sure communication between server and proxy is also HTTP/2 (check my old blogs about HTTP/2 for explanations and tools).

A new tool I came across was h2c it’s basically the same as Chrome net internals, but more structured and with colors!

So is it faster with HAProxy?

Yes! I only ran a very rudimentary test, but with significant results. Setting up SSL takes about 3.2ms with HAProxy and about 27ms with Undertow. This is a factor 8.5! Or about 24ms in absolute numbers, so that’s pretty hard to ignore.

I restarted the server 10 times so I got a fresh connection and looked in the Chrome web inspector to see how long the SSL handshake took. In the screenshot below the handshake took 3.10ms.

Conclusion

We thought we didn’t need nginx anymore and that was a right thought. Unfortunately it is replaced by HAProxy and is it still possible to capture packets in plain text, which is a small security risk. The latency improvement however is so big that I think it’s an easy sell. Remember to keep HAProxy and your web server very close to minimize the risk of packet sniffing.
The HAProxy setup is very basic, when you’re going to do real things take a look at the Mozilla SSL Configuration Generator. That’s a good starting point for a secure setup.

Sources

HTTP 2 0 & Java: Current Status by Simone Bordet (talk @ Devoxx)
https://webtide.com/http2-with-haproxy-and-jetty/
https://www.eclipse.org/jetty/documentation/current/http2-configuring-haproxy.html
https://tools.ietf.org/html/rfc7540
http://stackoverflow.com/questions/38612704/enable-http2-with-tomcat-in-spring-boot
https://github.com/bclozel/http2-experiments/blob/master/pom.xml
https://http2.github.io/faq/
HTTP/2 proxy support in nginx