AOT Compilation of Spring Boot: Behind the Scenes

The recently released Excelsior JET 15.3 has introduced out-of-the-box support for Spring Boot applications. Turning your Spring Boot jar into an optimized native binary has become as easy as adding a few lines to yourpom.xml/build.gradle or invoking the command:

jc MySpringBootApp.jar

where jc is the Excelsior JET AOT compiler.

In this article, we would like to share the technical problems we had to solve while adding that feature.

What Is Great with Spring Boot

The Spring Boot project wires together various pieces of the modern Java technologies stack. Using the Spring Boot auto-configuration feature, you can combine all those Kafka, Elasticsearch, Hibernate, Reactive WebFlux, Docker, and other buzzwords into a working piece of software with minimum effort.

On top of that, you can deploy your application as a single artifact called Spring Boot executable archive and run it with a simple command:

java –jar MySpringBootApp.jar

No wonder that “Make jar not war!”, the slogan promoted by Josh Long at almost every Java conference in the world, has become an unofficial motto of Spring Boot.

All that made Spring Boot, enhanced with the Spring Cloud project, a de-facto standard platform for modern Java applications built using the micro-service architecture.

To us, Spring Boot presented a great opportunity to test how all these modern Java technologies work together on the Excelsior JET JVM.

Spring Boot Executable Archive

Let us take a closer look  at what a Spring Boot executable archive essentially is.

If we talk about a jar, it has the following layout starting with Spring Boot 1.4:

BOOT-INF/classes
BOOT-INF/lib

Finally, the root directory contains Spring Boot loader classes and the main class set in META-INF/MANIFEST.MF is org.springframework.boot.loader.JarLauncher .

A war looks pretty similar with the difference that instead of BOOT-INF it has the WEB-INF subdirectory, just like standard .war files, the main class is org.springframework.boot.loader.WarLauncher , and servlet container jars reside in the WEB-INF/lib-provided subdirectory.  That structure enables you to run such a war file either within a servlet container or as a stand-alone executable jar with the following command:

java –jar MySpringBootApp.war

An interesting question is how Spring Boot tells the JVM to look for classes in those BOOT-INF/WEB-INF subdirectories. The standard application classloader only seeks classes from the root directories of classpath entries (jars), and the only jar which the classpath contains in this case is the Spring Boot executable archive itself.

Of course, this is accomplished through creating a custom classloader on startup of a Spring Boot application. That classloader loads classes from the subdirectories of BOOT-INF/WEB-INF .

Therefore, the first problem that we had faced with the implementation of our Spring Boot support was how to handle classes loaded by the Spring Boot custom classloader in our AOT compiler.

Custom Classloaders and AOT

An AOT compiler can easily compile classes from the application classpath and platform classes, as their class loading logic is totally known ahead of time. However, many modern applications may only have a small bootstrap jar in their classpath, which, in turn, creates custom classloaders that load all other application classes. All modern Java Web servers and frameworks such as OSGi are designed this way. Can an AOT compiler handle such classes? The answer is “yes and no”.

Surely, an AOT compiler cannot know how to resolve references between classes that are loaded by custom classloaders in the general case . And without resolving those references, it cannot perform inline substitution of methods between such classes, which, in turn, significantly reduces the efficiency of many other optimizations, and the performance of the resulting native binaries leaves a lot to be desired.

However, if the AOT compiler has been “taught” about the classloaders of a particular framework, and how class reference resolution behaves in them, it can compile applications based on that framework with all optimizations enabled.

Excelsior JET has “known” the classloading logic of Apache Tomcat and Equinox OSGi for quite a long time. So it can compile Tomcat Web applications and Eclipse RCP applications entirely ahead-of-time, in spite of the fact that application classes in those frameworks are loaded by their custom classloaders.

Thus, in order to add support for Spring Boot, we had to teach our AOT compiler and runtime routines that are responsible for loading AOT-compiled classes to recognize the behavior of a new class loader – the Spring Boot class loader. Fortunately, unlike the frameworks we had supported earlier, Spring Boot creates only one class loader and its logic is simple. Basically, it is the well-known URLClassloader with special URL handler support for referencing jars inside an executable archive by URL.

It took me a few days to add such a support to the Excelsior JET engine. Then it was able to compile all classes of a Spring Boot executable jar to a native executable that run flawlessly.

Dynamic Class Loading and AOT

What makes Spring Boot great is its auto-configuration feature. However, it does not come for free. On startup, Spring Boot generates tons of proxy classes for auto-configuration that are loaded dynamically. For instance, the classic Petclinic Spring Boot application generates and loads 538 classes while starting!

Fortunately, the Excelsior JET Runtime includes a JIT compiler to handle the dynamically generated classes. Most classes are still compiled and aggressively optimized with the AOT compiler.

First Problem Running an AOT-compiled Spring Boot Application

As I said above, it was quite easy to add support for the Spring Boot class loader to Excelsior JET, and after AOT compilation some sample Spring Boot applications ran flawlessly.  However, for deployment to other systems you need to package the native binary together with the required Excelsior JET Runtime files. And the subsequent run of a packaged application had failed with the following Exception:

Application run failed
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class

To understand the reason we need to learn how Spring works internally.

Dependency Injections in Spring

Any Spring Boot application is a Spring application as well. One of the main features of Spring is dependency injection, responsible for auto-wiring your Spring components together. In order to do this, Spring reads the respective annotations of your class files and understands what components to take and in which components they should be injected.

Spring can still read application class-files right after AOT compilation because the class files are left untouched by the Excelsior JET AOT compiler. That’s why natively compiled Spring Boot applications worked perfectly on the system on which they were produced. However, the Excelsior JET packager intentionally omits the original class files from the deployment package by default. This is how Excelsior JET provides one of the key benefits, IP protection: you deploy the resulting native binary without the original class-files, making all those Java decompilers useless against your application. As a result, reverse engineering of your Java applications becomes much harder.  But still, what to do with Spring in this case? It obviously depends on the availability of class files at run time.

Synthetic Class Files

Honestly speaking, Spring is not the only one Java framework that reads class files at run time. We had first encountered that problem with the JavaServer Pages (JSP) technology back in the day. A JSP container translates .jsp files into servlet class files. Usually .jsp files are translated into .java files first and then the javac compiler get invoked to obtain the class files. As .jsp files can reference application classes, javac needs the respective class files to resolve such references. To solve the problem we initially suggested to our users to include the original class-files in the deployment package. However, that had of course undermined our IP protection feature. Therefore, we came up with  another solution.

What information does javac actually retrieve from class files in the JSP scenario? Obviously, it does not need their bytecode because javac does not do inline code substitution. All it needs is symbolic information about the respective classes: what access modifiers, methods, fields, etc., they have. All this information is actually available from the reflection information of the class.

Thus Excelsior JET can generate a class file based on reflection information that would contain all symbolic information of the original class file, including annotations, but not the Java bytecode instructions. We call such class files synthetic or virtual. Then it can provide those class files to javac or other Java code which needs to read the original class files at run time. This is the solution for the problem with such frameworks.

But, surprisingly, it did not work for Spring Boot at all. To understand the cause we should go deeper into the implementation of synthetic class files in Excelsior JET.

How does a Java code retrieve the class file of a class? It searches for it in the respective classpath entry. If the classpath entry is a directory, the class file is then read directly from the file system; and if it is a zip/jar, the class file is accessed via the standard Java zip/jar implementation. As Excelsior JET is a complete Java SE implementation, we have full control over the internals of platform classes providing file system and jar/zip file access routines, including java.util.jar.JarFile . The Excelsior JET Runtime simply intercepts application attempts to access its own class files and generates the respective synthetic ones.

So, why did it not work for Spring Boot?  The answer is because Spring Boot has its own implementation of JarFile , and Excelsior JET was not aware of it! Spring Boot needs it to have random access to the contents of jar files inside a Spring Boot executable archive. The standard platform class JarFile does not provide that feature, so Spring Boot includes an enhanced version of that class in order to avoid unpacking of the embedded jar files on startup. (On-the-fly unpacking is a more common way to access embedded jar files, for instance, that’s how it is implemented in Tomcat, but it makes startup longer.)

As Spring Boot relies on its own, non-standard JarFile implementation, our trick did not quite work. Moreover, implementations of JarFile may differ across Spring Boot versions, so we cannot just substitute our own instead. So, how did we solve this problem?

We have noticed that, even though the implementation in Spring Boot is custom, it still has the standard interface – essentially it inherits java.util.jar.JarFile and overrides its methods. We also know which method is responsible for retrieving the contents of an entry from the jar file. Thus, the solution was to inject code that handles generation of synthetic class files into the beginning of that method during AOT compilation. Once we had implemented this solution, sample Spring Boot applications began to work after Excelsior JET packaging as well.

Two More Cases with JET-Compiled Spring Boot Applications

Once synthetic class files had started working with Spring Boot, I began compiling and running the entire set of Spring Boot samples . The set contains 80+ Spring Boot applications testing all modern Java technologies such as Kafka, Elasticsearch, Cassandra, Neo4J, etc. It was interesting to see how our product would handle them. Most of those applications just ran flawlessly, but I discovered two more issues in Excelsior JET while testing. Let’s consider them.

Case 1: Is The Synthetic Class File Readable?

The first case was an application failing to start with

org.springframework.beans.factory.NoSuchBeanDefinitionException

while the same application worked fine on the HotSpot VM.

Spring looks for bean definitions by parsing class files again. By the way, Spring can parse the same class file three times (!) for different reasons. However, we had synthetic class files working now, so what was the problem?

Bean definition candidates are selected by the ClassPathScanningCandidateComponentProvider class of the spring-context module. Let’s look at the code of the scanCandidateComponents method:

if (resource.isReadable()) {
      . . . 
    candidates.add(beanDefinition);
      . . . 
} else if (traceEnabled) {
    this.logger.trace("Ignored because not readable: " + resource);
}

After enabling tracing, I discovered that all synthetic class files were ignored by scanCandidateComponents method “because not readable”!

Then I went deeper. Why were they not readable? Other Spring components just read them without problems!

So I’ve found the actual implementation of the isReadable() method. Its code contains the following:

URL url = this.getURL();
if (!ResourceUtils.isFileURL(url)) {
    URLConnection con = url.openConnection();
       .  .  .  
    int contentLength = con.getContentLength();
    if (contentLength > 0) {
        return true;
    } else if (contentLength == 0) {
        return false;

If a resource content length is zero, then, according to Spring, the resource is not readable.

It may seem not obvious, but the content length of synthetic class files was zero. Why? This sounds reasonable: synthetic class files are virtual, they do not exist in reality, and, as a result, they do not have any content until someone tries to read them. A kind of a quantum effect :).  However, Spring does not even try to read them in this case. This also sounds logical.

How did we solve this problem? We made content length of synthetic class files equal to 1 (one) and that was enough!

Case 2: Parameter Names

After fixing the first case, almost all Spring Boot samples started working. All except one that failed to run again with the following:

***************************
APPLICATION FAILED TO START
***************************
 
Description:
 
Parameter 0 of method _relProvider in org.springframework.hateoas.config.HateoasConfiguration required a single bean, but 3 were found:
    - entityLinksPluginRegistry: defined by method 'entityLinksPluginRegistry' in classpath resource [org/springframework/hateoas/config/EntityLinksConfiguration.class]
    - relProviderPluginRegistry: defined by method 'relProviderPluginRegistry' in classpath resource [org/springframework/hateoas/config/HateoasConfiguration.class]
    - linkDiscovererRegistry: defined in null
 
 
Action:
 
Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed

What was the problem now? Spring uses parameter names to select beans for parameters. Since Java 8, names of parameters are available via reflection (as well as in class files) if you use the –parameters switch of javac . The latest Maven compiler plugin emits parameter names by default if you set sourceLevel to 1.8 for your project.

If parameter names persist in the original class files, then Spring works fine with Excelsior JET.

Honestly speaking, we had often been presented with similar problems when customers tried to compile Spring applications. Usually they were easy to fix using the -parameters javac switch.

However, here we had a problem that could not be fixed in such a manner. The failed application contained a dependency on the spring-hateoas component, which, in turn, uses the Spring auto-wiring facility, but its classes do not have the MethodParameters class file attribute. Of course, everything works if you recompile the dependency with method parameters, but who will try to recompile external dependencies? And the second question: why does the application work on HotSpot?

It turned out that if a class file does not contain method parameters, Spring has a backup path to retrieving their names. In such a case it reads the LocalVariableTable class file attribute.

LocalVariableTable is also an optional class file attribute. Its original purpose is to provide local variable names to Java debuggers. However, Spring discovers parameter names from that attribute if the MethodParameters attribute is absent.

LocalVariableTable is the attribute of Java bytecode while MethodParameters is the attribute of a method.

As you may recall, synthetic class files do not contain Java bytecode, so there were no LocalVariableTable attributes.

That’s why the above example did not work with JET while working fine with HotSpot: spring-hateoas component class files contain debug information, but do not have method parameter names.

So, how did we fix the problem? Well, I was slightly incorrect saying that synthetic class-files do not contain Java bytecode. Non-abstract methods actually do have some bytecode, as that is required by the JVM specification. And the bytecode is always

aconst_null
athrow

Thus, to solve the problem we started attaching the LocalVariableTable attribute to that bytecode in synthetic class files, containing only parameter names, so that Spring could detect them. Sounds weird, but it really worked!

Finally I’ve got the whole set of Spring Boot samples working fine with Excelsior JET.

Spring Boot, Microservices … and Compilation Time

When we say Spring Boot, we mean microservices, and when we say Java microservices – we mean Spring Boot. Spring Boot has a standard platform for microservices called Spring Cloud . The project also has a bunch of examples . So, the next logical step for me was to start compiling and launching those examples. I tried a few of them and they all worked fine.

Did I encounter any problems during that testing? Yes, compilation time was one of the biggest problems. A typical “microservice” contains 30K+ classes in its dependencies which is actually more than the entire Java SE platform! Excelsior JET includes pre-compiled platform classes, so they do not get re-compiled each time. However, all dependencies classes have to be compiled, so compiling one microservice was taking 30m-1.5h and 6-7 hours were needed to compile them all. I’ve heard that applications based on the microservice architecture often contain tens of microservices, which would have pushed AOT compilation times into tens of hours…

Fortunately, starting from version 15 Excelsior JET has incremental compilation, so if you modify your microservice, a re-build won’t need that much time (typically, it takes not more than 7-8 minutes).

AWT, JavaFX, and Spring Boot

Spring Boot is all about Java back-ends, right? A Spring Boot app is unlikely to have a GUI based on AWT/Swing or JavaFX, thought I.

However, I’ve noticed that a typical Spring Boot application depends on both AWT/Swing and JavaFX. That means that the resulting deployment package would contain these components. For a conventional JVM, omitting those components became possible only since Java 9 (via jlink ), but Excelsior JET users had had a similar feature for years. JavaFX and AWT/Swing are the biggest components in the Java SE so it is desirable to get rid of them when they are not required. Nonetheless, why does Spring Boot depend on them in the first place?

Probably you’ve watched the great talk by Josh Long – “ The Beautiful Application ”, where Josh showed an ascii art banner (the one you can see on Spring Boot startup) exclaiming “How beautiful it is!”. Now you need to know that this “beauty” does not come for free.

Actually, Spring Boot can render an arbitrary image file into ascii art to show a custom banner. This rendering is done by Java2D/AWT! So the AWT dependency is present in Spring Boot just because of the banner !

If you use a custom banner you cannot get rid of the dependency, but if you don’t use it, AWT classes are not loaded at run time and you can omit AWT from the deployment package. In Excelsior JET, omitting is not so easy. because Spring Boot classes reference AWT classes statically, but it is possible.

Okay, AWT gets sucked in because of the banner, but what the hell is JavaFX doing here?

Actually, JavaFX is referenced not by all Spring Boot applications, but only by those that use Hibernate. But, you know, almost all Spring Boot applications use Hibernate. So yes, Hibernate depends on JavaFX! Why? May be it can supply DB rows into JavaFX components directly? I do not know. However, here is the code that actually adds the dependency:

static {
    LinkedHashSet specDefinedExtractors = new LinkedHashSet();
    if (isJavaFxInClasspath()) {
        specDefinedExtractors.add(ObservableValueValueExtractor.DESCRIPTOR);
        specDefinedExtractors.add(ListPropertyValueExtractor.DESCRIPTOR);
   .  .  .
private static boolean isJavaFxInClasspath() {
    return isClassPresent("javafx.beans.value.ObservableValue", false);
}

So, if JavaFX is present in the JRE, Hibernate will always load that component! However, it of course works fine without JavaFX, so you just need to remove the latter from the JRE. The Excelsior JET Runtime always has JavaFX available for applications by default at the moment, so Hibernate always finds it. It is complicated to hide JavaFX from Excelsior JET Runtime but I was able to get rid of this component in a sample Spring Boot application. If you would like to know the details, please feel free tocontact us.

So, What About the Startup Time and Performance?

Well, I hope the article has been interesting to you so far. However, you may be asking: ok, it was an interesting read, but why would I use this tool?

First, if you are selling your Spring Boot application as a boxed solution, you may want to protect its code against reverse engineering, and Excelsior JET can help you with this.

However, most Spring Boot applications are deployed into well-controlled environments, such as clouds and do not necessarily require protection. So, what about the startup time and performance of AOT-compiled Spring Boot applications?

That is an interesting question! However, this article has turned out to be quite long, so I decided to talk about the startup time and performance in my future blog posts. Stay tuned!

Conclusion

Out-of-the box support of Spring Boot has just appeared in Excelsior JET. As for any new feature, we do not actually know how it will be used. Thus, any of your feedback on this subject is very welcome!

In the end, I would like to say that we have done a lot of work to support Spring Boot: almost all components of the Excelsior JET toolchain are now aware of Spring Boot: AOT compiler, runtime, graphical front-ends and Maven/Gradle plugins. By the way, look at the Spring Boot icon on the Welcome Screen of Excelsior JET Control Panel. How beautiful it is!

Links

HOWTO: Natively Compile a Spring Boot Application: https://www.excelsiorjet.com/kb/42/

JIT vs. AOT: Unity And Conflict of Dynamic and Static Compilers: https://www.youtube.com/watch?v=CZVa4PBv2qg