Thoughts on the JPMS open letter

Jigsaw is the name of a the project to build a Java Platform Module System (JPMS) into the Java 9 platform. Yesterday some of the members of the JPMS expert group published an open letter that, to not put too fine a point on it, damns the design of the new system and states that nobody should use it.

I’m not really involved with Jigsaw or Java module systems so I have no dog in this fight, but as I’ve been reading the Jigsaw mailing lists from the start and my name appeared incidentally in the letter (due to having written an email with suggestions), I figure I can provide an alternative, perhaps more neutral perspective.

Background

The document is very long. It makes many points, some of which I agree with. But I think it is worth reading with a skeptical eye. Although it lists the authors at the top, a casual reader can be forgiven for not knowing that many of them work on what are effectively competitors to Jigsaw like JBoss Modules and OSGi.

I don’t mean to engage in ad-hominem argumentation as the points are technical in nature, and I’ll get to them in a moment, but it’s worth considering if it’s possible for the Java team to integrate into the JDK a competitor to existing projects without triggering these sorts of gripes … gripes that often boil down to “we were here first, so what we did is best practice and you should do it too”.

Let me give an example. The very first paragraph in the letter states that Jigsaw is a “cleanroom reinvention” of a system that is “wholly untried in any real world application”. It goes on to say that “many use cases” that are “widely implemented” today are not possible under Jigsaw. That phrasing could apply to literally any new product that enters a market in which there are more complex incumbents.

Unfortunately, this is one of several parts of the document that made me uneasy. Jigsaw has had a multi-year development and feedback period in which the views of other module system creators have been led to substantial design changes again and again. Feedback has been taken from anyone who turned up, even Java nobodies like myself. So given how often Red Hat’s feedback has affected the design of Jigsaw, it seems both petty and wrong to describe it as “cleanroom”, implying it was developed in isolation from the outside world.

The module system market

It’s important to understand the market context in which this dispute is occurring.

There are no module systems in Java I would describe as successful. The most well known is probably OSGi and that’s using a generous definition of well known. OSGi is extremely complicated and whenever I’ve looked at it, I’ve found it very hard to get started. This isn’t just my view. I mentioned OSGi to my team the other week. One of my most experienced engineers said he’d also looked at it some time ago and been surprised at how complicated it seemed to be.

Partly that’s because it’s a spec with a variety of implementations, so for example, going to the OSGi website will not present you with a tutorial on getting started but rather conference adverts and marketing material for the governing consortium. Following links with enticing sounding names like “Where to start” simply gives you a bunch of links to off-site tutorials written by implementation vendors, none of which are any good. There is also a list of five different books on the topic. This developer experience is not excellent. But mostly it’s complicated because its design is genuinely enormous.

Given that a large part of the criticism in this document can be phrased as “Jigsaw isn’t the same as our own module systems” and given the lack of broad market success of the existing module systems that are out there (sorry guys!), I think we have to cut the Oracle team some slack. They’re attempting to build a module system for the masses, which may entail not supporting out of the box every possible use case that has been put into OSGi or JBoss Modules over the years. And in fact as can be seen in the spec discussion page, many of the features that are described as “critical deficiencies” have been dropped, simply on the grounds that they add too much complexity and risk. Given the poor tooling, poor documentation and vertical learning curve that typifies this space, it is hard to categorically condemn limiting the scope of a new module system.

There are far too many points made in the document for me to address them all, so I’ll pick a selection and bucket them into strong and weak (obviously, in my own humble opinion).

Weak criticisms

Descriptor format

The authors argue that Jigsaw has made a mistake by using binary module descriptors. Their argument is:

Binary descriptor formats are considered an architectural regression. Text-based descriptor formats are easier to read, modify, and programmatically manipulate using standard tools.

This is a statement of their own personal opinions. Text based configuration formats and protocols have a host of downsides that are simply ignored here, beyond the simple cost of parsing them. Would the authors also argue for .class files to be replaced with text files? If not, why not, as the arguments they make here would appear to apply to literally any file format or protocol.

JBoss uses XML to define modules. XML has gone out of vogue in recent years for valid reasons: it is a format that is a good fit for neither humans nor machines. OSGi uses the MANIFEST “Key: a=b, c=d” type format. I’ve found neither to be especially readable. Jigsaw uses source code with a grammar designed specifically for specifying modules, an approach I’d argue has better usability than a generic syntax like XML or manifest files, and it’s translated at build time to binary which is a much better fit for high performance virtual machines written in C++. Given that the user has to compile source code to class files anyway, this is hardly a major imposition.

Binary vs text is a bit like spaces vs tabs — we aren’t going to resolve the question of which is better here. But by phrasing a nuanced design choice as if it were an unquestioned statement of fact, the document undermines its own authority (and needlessly so, as there are far more important issues with Jigsaw than this).

Versioning

The document says:

Module version strings in Jigsaw are constrained by a format which does not reflect any current versioning best practices. The implication is that they incompatible with most (if not all) existing Java-based versioning schemes.

I don’t understand this point. The currently supported format is specified here. To me that looks pretty complete, and certainly can accommodate almost all versioning schemes I’ve seen, including the stupid ones.

We can compare Jigsaw to OSGi and JBoss. OSGi’s approach to versioning is really crazy — it doesn’t only version modules but also sub-packages within modules, and it imposes a significantly more strict version string pattern than Jigsaw does. I wanted to write about what JBoss Modules requires but unfortunately the documentation doesn’t seem to say, it just refers to “slots” which can apparently be any arbitrary string. In fact the JBoss Modules docs are so sparse as to be nearly useless: e.g. the “Executable modules” page is empty and the introduction page hasn’t been updated since 2011.

Given the dubious approach taken by Jigsaw’s competition in this area, perhaps they should be less categorical with their condemnation of what are — again — nuanced design choices on which reasonable people can disagree.

The kill switch

Because of the scale of the compatibility problems, the JDK has lately added an option to blanket-disable the additional reflection security capabilities, which further acknowledges the difficulties with the dramatic nature of this change. The change itself introduces a new problem: log messages are emitted to the error stream regardless of any application use of that stream.

This is weak. There are several switches. The —-permit-illegal-access option is intended to help developers figure out where their code is crossing module boundaries in ways that aren’t declared, so it prints when that occurs because it’s a diagnostic option. But by using the —-add-opens and —-add-exports flags, you can override the module system in the same way, without printing any warnings, in order to encode exceptions and module patches into your program long term. This hardly seems like a big deal.

The visual comparison

This kind of tick-box table that lists only features that your product happens to have, regardless of importance, is a classic marketing technique. It’s also sketchy in a deeper way. Their table says “Theoretically Possible to AOT-compile” and gives all module systems a tick. But Jigsaw based apps actually are possible to AOT compile, not just theoretically but in practice, using the new jlink tool that is a key part of Jigsaw’s value proposition (it’s kind of a static linker for Java).

Phrasing this as “theoretically possible” and awarding their own systems a tick, when it should really be “Supports AOT compilation of modular apps” and only Jigsaw gets a tick …. well, to me that seems like a low blow.

The tweet

In a short section that argues Jigsaw’s backwards compatibility is insufficient the following quote is provided as evidence:

“Ok. Fuck Jigsaw compatibility. If doing so requires use of Java 9 tools, I’ll have zero-fucks level interesting in support for next 2+ years” — Tatu Saloranta (Author of Jackson, WoodStox, ClassMate, TStore)

This quote isn’t talking about backwards compatibility, so doesn’t actually support the point being made, but still, I was curious as to the context because Jackson is an important library. Sadly it turns out the quote is a tweet without context. I’m unsure why anyone would expect a feature added in Java 9 to not require using Java 9 tools, but at any rate, Tatu’s heartfelt argument is slightly undermined by his Twitter bio, which says “Rant First, Analyse Later! Then think”. Maybe this is one of those times.

There are enough real points to debate about Jigsaw’s design that argument by tweet is not necessary.

Stronger points

• One-classloader-per-app (by default). Concealed package conflicts are probably going to be a pain. Of course, they don’t work properly now either because classpath orderings are frequently unstable and cannot be relied on, but an ideal module system would actually solve problems like these, not just flag them as errors at startup. Often Jigsaw seems to default to simply surfacing previously unstable configurations as startup errors, which feels like tiny progress compared to making the problem go away. The other problems in this section, like multiple incompatible versions of the same module, are all related. In fairness, supporting the case of multiple versions in the same process is not ruled out entirely: Jigsaw team have said it may be implemented in a future release. And the implementation actually can do it with additional outside assistance, lending credibility to this position.
• Resource handling. It looks like a bit of a mess.
• Module naming. The aesthetic consistency of having module identifiers look the same as Java package identifiers is clear, but there’s already a whole universe of module names out there that look like Maven coordinates. Jigsaw picks the Java style and then comes up with various heuristics to try and map from what’s actually being used back to what Jigsaw uses, which just adds complexity: the lack of which is a major part of the justification for doing Jigsaw in the first place (vs just adopting OSGi wholesale).
• The fate of javax.annotations. I guess any attempt to modularise the JDK would have this sort of problem. It’s not a reason to use other systems, which don’t even try.

There are many other points that one could list as either strong or weak, depending on how much you value specific features and current ways of doing things, and how much you assume a version 1.0 should accomplish (i.e. is it everything or is it OK to add features in future versions).

Conclusion

Final thoughts. Jigsaw doesn’t adopt a subset of OSGI’s/JBoss’ features and call it a day. It brings major new features to the table as well. The document simply doesn’t mention them at all, which is a pity. For instance:

• Modularising your app using Jigsaw lets you use JLink, a tool that can produce a minimised and heavily optimised “statically linked” JVM/app combination, removing the need for your users to provide their own JVM.
• JLink can use the statically specified, strict nature of Jigsaw modules to figure out what modules your app needs and then do ahead of time compilation of the entire thing to native code. No other module system can do this.
• Jigsaw modularises the JDK itself. That is a significant chunk of work that other module systems haven’t tackled (though they theoretically could have done, as the JDK is open source).
• The tooling has been extended, for example, javadoc now knows how to draw module diagrams and how to generate API docs that show module boundaries. Tooling is often a weakness of other module systems.
• Although technically Jigsaw is an implementation of a spec (the JPMS), the Java platform has always managed to avoid the UX nightmare that OSGi has got: Jigsaw is as well documented as you’d expect from the JDK, there are plenty of tutorials you can use to learn it, and it strikes a similar feature/complexity balance to the rest of the Java platform.

Without a doubt, modularising the Java ecosystem will take years and involve breaking more than a few eggs along the way. I am unconvinced this process would be easier if Jigsaw was simply a clone of OSGi or JBoss Modules. To get developers to modularise their apps it isn’t enough to simply make a module system. You need to make it simple enough for busy developers to digest and you need to provide some big feature carrots to motivate the work.

Having read the entire document and written out this giant response, I actually find myself gaining sympathy for Jigsaw instead of losing it. The team chose to compete with existing solutions rather than simply adopt them because the existing solutions have been rejected by the market. This means challenging some of the design decisions other module system experts made, and will inevitably cause friction. That does not mean Jigsaw is a failure. Telling users they should avoid using it under any circumstances — as this document does — makes me perceive it as a subtle, somewhat underhanded form of competitive marketing. “Everyone should use OSGi or JBoss” is simply not a workable alternative and indeed, the authors don’t quite go there. Instead they give the impression they’d prefer the Java ecosystem simply not modularise at all.

In the end I think Jigsaw may fail to gain adoption … but it still has a much better chance of succeeding than the incumbents.