Rethinking JavaScript Test Coverage

This blog was written by Benjamin Coe , Product Manager at npm, Inc. and lead maintainer of yargs and Istanbul. It covers work that has gone into introducing native code coverage support to Node.js.

TLDR: You can now expose coverage output through Node.js by setting the environment variable NODE_V8_COVERAGE to the directory you would like coverage data output in. The tool c8 can be used to output pretty reports based on this coverage information.

A Bit of History: How Test Coverage Has Historically Worked

In JavaScript code, coverage has traditionally been facilitated by a clever hack: tools like Istanbul and Blanket parse JavaScript code inserting counters that (ideally) don’t change the original behavior of the application, for instance:

function foo (a) {
  if (a) {
  // do something with 'a'.
  } else {
  // do something else.
  }
}

gets rewritten as:

function foo(a) {
  cov_2mofekog2n.f[0]++;
  cov_2mofekog2n.s[0]++;
  if (a) {
    // do something with 'a'.
    cov_2mofekog2n.b[0][0]++;
  } else {
    // do something else.
    cov_2mofekog2n.b[0][1]++;
  }
}

cov_2mofekog2n.f[0]++ indicates that the function foo was executed, cov_2mofekog2n.s[0]++ indicates that a statement within this function was called, and cov_2mofekog2n.b[0][0]++ and cov_2mofekog2n.b[0][1]++ indicate that branches were executed. Based on these counts, reports can be generated.

This transpilation approach works, but has shortcomings:

• Tools like Istanbul need to play catch up with the evolving JavaScript language and sometimes lag behind on language features; here’s a pull request adding object-spread syntax to Istanbul from September 2017, several months after the feature became widely available .
• Introducing counters on every line of an application significantly impacts performance (Node.js’ own test suite runs about 300% slower when instrumented).
• It’s difficult to insert counters into a codebase without accidentally changing its behavior .

I found myself wishing there was a better way to collect code coverage…

Code Coverage in V8

I was chatting with Bradley Farias in August of 2017 about ESM module support in Node.js as ESM modules presented a problem for Istanbul. The problem? Bradley’s rewrite of Node.js’ loader to support ESM modules no longer supported hooking require statements, this made it difficult for Istanbul to detect that an ESM module had been loaded and instrument it. I made a strong case for adding this functionality back, Bradley had another suggestion:

What if we leverage V8’s new built in coverage functionality?

Using coverage built directly into the V8 engine could address many of the shortcomings facing the transpilation-based approach to code coverage. The benefits being:

• Rather than instrumenting the source-code with counters, V8 adds counters to the bytecode generated from the source-code. This makes it much less likely that the counters alter your program’s behavior.
• Counters introduced in the bytecode don’t impact performance as negatively as injecting counters into every line of the source (I noticed a 20% slowdown in Node.js’ suite vs 300%).
• As soon as new language features are added to V8, they would be immediately available for coverage.

I proceeded to investigate using Node.js’ inspector module for collecting coverage directly from V8; there were some hiccups:

• Timing issues with the inspector made it so only function-level coverage could be enabled (you couldn’t collect coverage for block-level statements: if statements, while statements, switch statements).
• Block coverage was missing some features: || expressions, && expressions.
• The dance to get the inspector up and running felt overly complex. You needed to start your program, enable the inspector, hook into it, and dump the coverage report.

These challenges aside, using V8’s coverage via the inspector felt promising. I was left wanting to help see things over the finish line.

Moving Towards a Proof of Concept

I reached out to Jakob Gruber on the V8 team regarding the bugs I was seeing integrating V8 coverage with Node.js. The folks at Google were also excited to see coverage support in Node.js and pitched in to start addressing bugs almost immediately.

After a discussion with several V8 maintainers , we determined that there was in fact a mechanism for enabling block level coverage:

--inspect-brk
Runtime.runIfWaitingForDebugger
Runtime.executionContextDestroyed

I tested the approach outlined above and it worked!

I next asked Jakob if I could pitch in and start implementing some of the missing coverage functionality in V8. With the patient help of several folks on the V8 team, I was able to implement support for || and && expressions . This was my first time writing C++ in years and was a ton of fun for me.

At this point we had detailed V8 coverage information being output, but no easy way to output human readable reports. Two npm modules were written to facilitate this:

• v 8-to-istanbul , which converts from V8 format coverage output to Istanbul format.
• c8 , which pulls together the entire inspector dance into a single command, sot you can collect coverage by simply running c8 node foo.js .

Leveraging these new libraries, we were finally able to see coverage reports!

code coverage report, based on output from Node.js

This was a exciting milestone, but I still wasn’t satisfied. Here’s why:

process.exit(0)

Node-Core Implementation

I had an epiphany, what if Node.js could be placed into a mode that always dumped coverage?

• This would mean that another process wouldn’t need to connect to the inspector session and kick off coverage tracking.
• It would put us in a position to better detect when Node.js was shutting down, such that we could capture process.exit(0) and process.kill events.

In conversation with Anna Henningsen , it turned out that the Node.js inspector implementation lent itself to my idea:

1. The inspector is actually always running in most environments, the websocket interface is just not enabled.
2. There is an internal inspector protocol available that can interact with the inspector without creating a socket connection.

Excited, I sat down and implemented V8 test coverage as a feature of Node.js itself . Here’s what it looks like:

• In Node.js >=10.10.0 you can now set the environment variable NODE_V8_COVERAGE to a directory, this will result in V8 coverage reports being output in this location.
• The tool c8 now simply enables the NODE_V8_COVERAGE environment variable, consumes the V8 coverage data, and outputs pretty reports.

Next Steps

You can start using the Node.js built in coverage reporting today:

1. Make sure you’ve upgraded to Node.js 10.10.0 .
2. Install the c8 tool, which can be used to convert V8 coverage output into human readable reports.
3. Use c8 to execute your application, e.g., c8 node foo.js .

This feature is brand new, and I’d love for JavaScript communities to help see things over the finish line.

You can help by:

• Playing with the Node.js built in coverage and filing any bugs you find.
• Submitting patches for existing bugs .
• Spreading the word about this awesome new tool that’s available for JavaScript testing.