JEP draft: Add detailed message to NullPointerException describing what is null

Summary

NullPointerExceptions are freqently encountered developing or maintaining a Java application. NullPointerExceptions often don't contain a message. This complicates finding the cause of the exception.

This JEP proposes to enhance the exception text to tell what was null and which action failed.

Motivation

Currently, a common NullPointerException does not contain a helpful message. The following code:

a.to_b.to_c = null;
a.to_b.to_c.to_d.num = 99;

will just print:

java.lang.NullPointerException

This does not help to find out what was null. Was it a ? a.to_b ? a.to_b.to_c ?

A message like

'a.to_b.to_c' is null. Can not read field 'to_d'.

will immediately tell where the exception was thrown. For more message examples see [1].

The same algorithm can be used to improve other messages:

a[i][j][k]; // Index j is out-of-bounds

currently prints

java.lang.ArrayIndexOutOfBoundsException: Index 99 out of bounds for length 2

This message does not tell which index is out of bounds. This can be improved to print:

java.lang.ArrayIndexOutOfBoundsException: a[i][j]: Index j = 99 out of bounds for length 2

Description

Basic algorithm to compute the message

Java code like a.to_b.to_c is compiled to several bytecode instructions. When the exception is raised, the original Java code is not available, but the bytecodes. Given the bytecodes of the method in which the exception was raised and the precise bytecode where it happened, information about the exeption context can be derived and printed.

If an exception is raised, the precise position in the bytecode, i.e. the instruction that caused the exception, is known by the virtual machine. This is stored in the 'backtrace' datastructure of a Throwable object which is held in a field private to the jvm implementation. It also contains a reference to the method that was executed [2].

To assemble a string as a.to_b.to_c the bytecodes must be visited in reverse execution order starting at the bytecode that raised the exception.

Given a bytecode, one knows which expression stack slots it pops, i.e., it is known which slot contains the null value. If one knows which bytecode pushed the null value, one can print the message. Going several steps backwards allows more comprehensive strings as, e.g., a.to_b.to_c .

Given the bytecode, it is not obvious which previous instruction pushed the null value. To find out about this, a simple data flow analysis is run on the bytecodes.

This data flow analysis walks the bytecodes forward simulating the execution stack. The simulated stack does not contain the values computed by the bytecodes, but information about which bytecode pushed the value to the stack. Therefore this analysis terminates quickly. The analysis runs until the information for the bytecode that raised the exception is available. Thus, it stops before analysing the whole method [3].

With this information, the message can be assembled.

The message consists of two parts.

Looking at the bytecode that failed to execute the intended action can be described. Knowing the semantics of bytecodes, a message for each can be formulated [4].

Looking at the bytecodes that pushed the null value onto the stack, a message telling which entity is null can be assembled. This is the essential part of the message [5].

Computing the second part of these messages can fail due to insufficient information (e.g. for expression (a ? b : c).d it can not be computed where the null reference came from). In this case the message part is omitted.

Different bytecode variants of a method

Due to redefinition and bytecode modification different variants of the same method can be used by the jvm implementation. The bytecode used can differ from the bytecodes in the original class file that was loaded. The analysis must operate on the very bytecode that has been executed when the exception was raised. The bytecode index stored in the exception must point to the instruction that caused the exception.

Using the method attached to the backtrace, this is guaranteed by construction. The method representation referenced from the backtrace is always the right one.

Computation overhead

NullPointerExceptions are thrown frequently. Each time a NullPointerException object is created. Usually, exception messages are computed at Object creation time and stored in the private field 'detailMessage' of Throwable. As computing the NullPointerException message proposed here is a considerable overhead, this would slow down throwing NullPointerExceptions.

Fortunately, most Exceptions are discarded without looking at the message. I.e, if we delay computing the message until it is actually accessed, we omit the effort for all the exceptions that are discarded without printing the message.

A Throwable gives access to the stackTrace, a representation of the stack trace when the exception was thrown. This consists of a lot of Java Objects. Here too, it would be costly to build this information at object construction time. Therefore the information is stored in an intermediate format in the internal 'backtrace' field. Only if the stackTrace is accessed from Java, the actual Java objects are created. This is the same approach as this JEP proposes for the NullPointerException message.

If the message is printed based on the backtrace, it must be omitted if the top frame was dropped from the backtrace. This happens for generated methods that are marked as hidden. This fact must be preserved until the message is printed.[6]

Message persistance

Usually, an exception message is passed to the constructor of Throwable that writes it to its private field 'detailMessage'. If we compute the message only on access, we can no more pass it to the Throwable constuctor. As the field is private, there is no natural way to store the message in it.

There are several possibilities to overcome this:

• make detailMessage package-private.
• use a shared secret to write it
• write to the detailMessage field via JNI

An alternative is to recomputed the message each time it is accessed. Presumably, the common use case is to get it once to write it to some output, so the additional overhead of recomputing the message should be encountered rarely. Following this approach, npe.getMessage() == npe.getMessage() will not hold, as it is the case for other exceptions.

The 'backtrace' field can be cleared after the stackTrace has been computed. E.g., if a Throwable is serialized, the backtrace is gone. If the backtrace is lost, the message can no more be computed. It must either be computed and stored to some field, or the bytecode index and method must be preserved. If this is not done, npe.getMessage() will first deliver a useful message, and on later calls it will return null for the message.

For serialization, this can be solved implementing writeReplace().

Message content

The proposed generated message should only be printed if the NullPointerException is raised by the runtime. If the exception is explicitly constructed, it makes no sense to add the message. It will be misleading, as no real NullPointerException was encountered.

The message can not regain the original code 1:1. The message should try though to give as much resemblance to code as possible. This makes it easy to understand and compact.

The message will mention information from the code like class names, method names, field names and variable names. It must be decided whether these need to be quoted or not. Other exception messages, as IllegalAccessError, don't quote class names etc.

Method signatures should be printed in Java syntax. Currently, they are printed in the compressed format. I propose to change this in a follow up, as many other exceptions use the compressed format in their messages. Again, see IllegalAccessError.

Local variable names are retrieved from the debug information. If a class file does not contain this information, a replacement must be found. A first proposal is to use <local1> , <local2> , <parameter1> etc.

Alternatives

The current proposal is to implement this in the Java runtime in C++ accessing directly the available datastructures in the metaspace.

Alternatively, this can be implemented in Java. Basic functionality is available in the class library, e.g. StackWalker to retrieve the bytecode index and ASM to analyze bytecodes.

This adds further overhead though, as datastructures must be replicated in Java.

Also, ASM and StackWalker do not support the full functionality needed. StackWalker must be called in the NullPointerException constructor. For performance reasons, this is not desirable, see above. StackWalker could be changed to operate on the backtrace information captured in the Throwable, all needed information should be available. This requires changes to StackWalker, Throwable and implementation of C++ code.

ASM does not have a notion of bytecode indexes. I.e., given a bytecode index, you can not access the corresponding bytecode. It is straight forward to extend ASM to compute this information when parsing the bytecodes. Complex bytecodes as tableswitch must be handled.

ASM should operate on the actual bytecode used. A file

containing the class might not be available or hard to locate. The bytecode run by the virtual machine may be modified in many ways. It might be possible to use the information returned by JvmtiClassFileReconstitutor.

Testing

A test checking for regressions of the messages will be implemented.

The basic implementation is in use in SAP's internal Java virtual machine since 2006. We run all jtreg tests, many jck tests and various other tests on the current implementation. No issues were encountered so far.

Risks and Assumptions

This imposes overhead on retrieving the message of a NullPointerMessage.

The risk of breaking something in the virtual machine is low. The feature can be implemented completely on top of existing data structures without changeing them. Only writing the 'detailMessage' field of Throwable late is a considerable change.

The implementation needs to be extended if new bytecodes are added to the Bytecode specification.

The concern was raised that printing field names or local names might impose a security problem. If so, this should be addressed more globaly. Other exceptions already print field information (IllegalAccessError). Class and method names are contained in the stack trace. Printing field names in NullPointerExceptions should not impose any new security problems.

Dependencies

None. A stable and tested prototype exists.

References

A prototype has been proposed in 8218628: Add detailed message to NullPointerException describing what is null. .

Awebrev of the prototype is available.

[1] Messages if classfiles contain debug info. Messages if classfiles contain no debug info.

[2] java_lang_Throwable::get_method_and_bci(...) retrieves the needed information of the top frame.

For the following references see prims.cpp: JVM_GetExtendedNPEMessage(...) in the webrev:

[3] The data flow analysis of the bytecodes is implemented in the TrackingStackCreator constructor.

[4] The message part for the failed action is computed by TrackingStackCreator.get_null_pointer_slot(...) .

[5] The message part describing the null entity is computed by TrackingStackCreator.get_source(...) .

[6] A solution is proposed in8221077: No NullPointerException message if top frame is hidden.