Sandboxed execution of native code with GraalVM

Safe and sandboxed execution of native code

GraalVM is a high-performance runtime for different programming languages. Currently, it supports the JVM languages, JavaScript, Ruby, R, Python, and even native languages. To support a programming language GraalVM needs only an interpreter. At runtime, it uses partial evaluation to generate machine code and execute programs very efficiently.

One particularly interesting language supported in GraalVM is an LLVM bitcode interpreter which can execute LLVM bitcode in a managed environment. It is useful for running unmanaged languages like C and C++ on top of GraalVM, running native extensions for dynamic languages like Ruby, R, or Python, and letting the native code participate in the polyglot world when mixing these.

In this article, we explore a new feature of the Enterprise Edition of GraalVM: sandboxed execution of native code. It not only enables native code to participate in the polyglot programs, passing and receiving the data from any other supported language, but it also executes all native code in a managed environment with additional safety guarantees: catching illegal pointer accesses, accessing arrays outside of the bounds, and so on.

Imagine you have a hello world C code like this:

You can compile it to LLVM bitcode, and run it on GraalVM.

Note: All examples in this blog post assume you have a copy of GraalVM EE (RC8 or later) downloaded, with the $GRAALVM_HOME environment variable pointing to where it is extracted. You also need to have the clang compiler installed.

$ clang -c -g -O1 -emit-llvm hello.c
$ $GRAALVM_HOME/bin/lli hello.bc
Hello, World!

But of course, C being an unsafe language, you can also do things like this:

We compile this code to bitcode:

$ clang hello-bug.c -o hello-bug
hello-bug.c:4:39: warning: format specifies type 'int *' but the argument has type 'int' [-Wformat]
  printf("number of arguments: %n\n", argc);
                               ~~     ^~~~
1 warning generated.

Let’s pretend we didn’t see the compiler warning (or maybe it just got lost in pages of other build output, or it was even turned off), and run the code (without GraalVM for now):

$ ./hello-bug
Segmentation fault (core dumped)

Whoops. What happened?

The problem is the mistyped format specifier %n instead of %d . The %n format specifier stores the string length until now into the provided pointer. We're passing an int . Of course, this crashes. An int is not a valid pointer, but printf tries to write to it anyway. Since C is an unsafe language, the compiler will happily generate code that will pass an int where a pointer should go. At runtime, values are not typed, and a pointer is just another number, so the code inside printf has no way to detect the problem.

In this particular case, the compiler gave us a warning. However, this is only because printf is a well-known library function and the compiler knows what arguments you're supposed to pass. But for your own functions, you're on your own.

Now let’s try the same program with GraalVM. First, we compile our code to LLVM bitcode using Clang with the -c and -emit-llvm options. We also use the debug and optimization flags recommended for compiling C-family code for execution on GraalVM, -g -O1 (see here ).

$ clang -c -g -O1 -emit-llvm hello-bug.c
[...]

Running this code:

$ $GRAALVM_HOME/bin/lli hello-bug.bc
Segmentation fault (core dumped)

It still crashes the VM. Why is that?

When GraalVM runs LLVM bitcode, the bitcode is executed as-is. Memory is still allocated on the native heap and external functions are called as they are. If the function is not available as LLVM bitcode, e.g. printf from the system libc , it is just called through the native interface.

GraalVM has some mechanisms for polyglot interop, so if a pointer refers to a managed object (e.g. a JavaScript object), the access can’t crash the VM. However, a native pointer is still just a number, which refers to some memory location on the native heap. GraalVM can’t tell in a reliable way whether a pointer to native memory is valid. And, of course, once you start calling real native code outside of GraalVM like the printf function in libc , all bets are off.

Because of these issues, GraalVM can’t really make any additional security guarantees in this case.

Sandboxed mode

GraalVM EE contains an experimental sandboxed mode for executing LLVM bitcode. In that mode, all memory allocations from LLVM bitcode are done on the managed heap. This mode has various advantages and limitations, but the main benefit is that you can’t crash the VM anymore:

$ $GRAALVM_HOME/bin/clang-sandboxed -c hello-bug.c -o hello-bug-sandboxed.bc
[...]
$ $GRAALVM_HOME/bin/lli --llvm.sandboxed hello-bug-sandboxed.bc
number of arguments: Illegal pointer access: 0x0000000000000001
        at <llvm> printf_core(Unknown)
        at <llvm> vfprintf(/b/b/e/main/sulong-managed/mxbuild/linux-amd64/projects/com.oracle.truffle.llvm.managed.libraries.bitcode/musl-1.1.19/src/stdio/vfprintf.c:1:0)
        at <llvm> printf(/b/b/e/main/sulong-managed/mxbuild/linux-amd64/projects/com.oracle.truffle.llvm.managed.libraries.bitcode/musl-1.1.19/src/stdio/printf.c:1:0)
        at <llvm> main(hello-bug.c:4:57)
        at <llvm> @_start(libsulong.bc:@_start:1:0-19)

( clang-sandboxed is just a wrapper script for clang , we'll get into why it is needed later)

On the surface, the result doesn’t look that much different: The program crashes. But the important thing is: Now it’s crashing with an exception instead of a segfault. So for example, when embedding this code in a server application, it is possible to catch this exception rather than letting the whole server go down:

$ $GRAALVM_HOME/bin/javac SandboxedTest.java
$ $GRAALVM_HOME/bin/java SandboxedTest
number of arguments: something went wrong: org.graalvm.polyglot.PolyglotException: Illegal pointer access: 0x0000000000000001

So how does that work?

In the previous section, we identified two crucial problems with memory safety:

• We want to call native library functions.
• We can’t tell the difference between pointers and numbers.

Let’s address these points one at a time.

Sandboxed libraries

Calling code in native libraries defeats the purpose of having a sandbox. Native code can do anything it wants, possibly even break out of the sandbox. Because of that, calling native code can’t be allowed in a sandboxed environment. So we need to run all library code inside of GraalVM, down to the standard C library. For that to work, we need all libraries in LLVM bitcode format.

Currently, GraalVM only ships a standard C library, musl-libc , pre-compiled to LLVM bitcode. If the program to be executed requires additional libraries, they can be either linked statically to the program with llvm-link , or loaded dynamically with the --lib option of GraalVM's lli command (see the Reference Manual ).

The standard C library is using syscalls to communicate with the operating system. To make it impossible to break out of the sandbox, GraalVM has to virtualize these syscalls, redirecting them to the equivalent GraalVM or Java interfaces. That way, all the standard security features of GraalVM and of the underlying Java VM (e.g. security managers) also work for LLVM code.

To make this approach more scalable, we decided to virtualize the syscalls of a single platform only, x86_64 Linux . To the LLVM bitcode, it will always look like it is running on x86_64 Linux, regardless of the actual platform you’re running the code on. This also means that all bitcode needs to be compiled for this platform.

The standard C library shipped with GraalVM is also compiled for that platform. We ship the correct header files for that libc as well. The clang-sandboxed (see example above) wrapper script can be used to produce bitcode files for the correct platform. It calls clang with the compiler flags for selecting the target platform and using the correct header files.

People familiar with GraalVM might have noticed in the embedding example that in sandboxed mode, it is no longer necessary to call allowNative(true) on the context builder:

Context.Builder builder = Context.newBuilder().allowIO(true).option("llvm.sandboxed", "true");

On the other hand, that means you need to supply bitcode versions of all other libraries that your code depends on. Calling code in real native libraries do not work in sandboxed mode, since that would allow breaking out of the sandbox.

Memory sandboxing

Now that we have all code running purely inside of GraalVM, we have control over everything that is run. Now we can tackle the more interesting problem of memory safety.

When sandboxed mode is enabled, this does three things:

• All memory allocations are done on the managed heap.
• Access to native memory is forbidden.
• Access to native libraries is forbidden.

Since all allocations are on the managed heap, memory access violations are impossible by construction. The LLVM language implementation itself is written completely in Java, so the running LLVM bitcode is now as safe as running Java code. For example, even if we have a bug in the LLVM language implementation and miss some out-of-bounds check, we still can not access memory in an uncontrolled way, we still get at least an ArrayIndexOutOfBoundsException from the VM.

This does not just prevent crashes, it can also prevent more subtle bugs.

Consider the following C code (a variant of the usual “don’t use gets” bug):

If we run this code with too much input, it overflows.

$ clang overflow.c -o overflow
$ echo "123456789" | ./overflow
$ echo $?
0

But it doesn’t actually crash.

We have just overwritten some random memory. In this small example, overwriting random memory doesn’t break anything, since we haven’t actually used any other memory. In a real-world program, this will just crash later. In the worst case the program never crashes but allows an attacker to take over the system, or leak confidential data.

Let’s try the same program with GraalVM in sandboxed mode:

$ $GRAALVM_HOME/bin/clang-sandboxed -c overflow.c -o overflow-sandboxed.bc
$ echo "123456789" | $GRAALVM_HOME/bin/lli --llvm.sandboxed overflow-sandboxed.bc
Out-of-bounds access (object size in bytes: 8, accessed byte index: 8)
        at <llvm> vfscanf(/b/b/e/main/sulong-managed/mxbuild/linux-amd64/projects/com.oracle.truffle.llvm.managed.libraries.bitcode/musl-1.1.19/src/stdio/vfscanf.c:1:0)
        at <llvm> vscanf(/b/b/e/main/sulong-managed/mxbuild/linux-amd64/projects/com.oracle.truffle.llvm.managed.libraries.bitcode/musl-1.1.19/src/stdio/vscanf.c:1:0)
        at <llvm> scanf(/b/b/e/main/sulong-managed/mxbuild/linux-amd64/projects/com.oracle.truffle.llvm.managed.libraries.bitcode/musl-1.1.19/src/stdio/scanf.c:1:0)
        at <llvm> main(overflow.c:6:83)
        at <llvm> @_start(libsulong.bc:@_start:1:0-19)

The out-of-bounds access throws an exception.

Limitations

As already mentioned, the most important current limitation of sandboxed mode is not being able to call real native libraries. All running code must be available in LLVM bitcode form and executed by GraalVM.

Another limitation is that GraalVM currently only ships with a standard library for C. Other LLVM languages (e.g. C++) may only work if you manually supply their standard library in bitcode form.

Also note that this feature is still experimental and not yet complete. Only the most important syscalls are implemented. For example, regular file and console IO should work, but memory mapped files and network related syscalls are not implemented yet.

As a workaround, you can use polyglot interoperability with other languages (e.g. Java) for any missing features.

What else can you do with this?

In principle, as long as it doesn’t require native access, anything that works in GraalVM should also work in LLVM sandboxed mode.

In particular, all polyglot interoperability features should also work in sandboxed mode. Interoperability can even work better in sandboxed mode. One limitation of LLVM interop in non-sandboxed mode is that you can not store pointers to foreign objects in native memory. Since in sandboxed mode the “native” allocations are actually in managed memory, this limitation does not apply anymore.

Conclusion

Enterprise Edition of GraalVM introduced the experimental option --llvm.sandboxed to run LLVM bitcode in sandboxed mode. In this mode, all access to native libraries and native memory is forbidden. All memory operations occur on managed objects on the managed heap of the VM, where the standard Java memory protection guarantees apply. In this mode, memory violations by user code are reported as exceptions instead of crashing the VM.

You can get the GraalVM binary from the website: graalvm.org/downloads . We encourage you to experiment with this new feature and report any feedback or bugs you encounter on Github .