Rust: integrating LLVM source-base code coverage with GitLab

Rust: integrating LLVM source-base code coverage with GitLab

Earlier this year, the Rust compiler gained support for LLVM source-base code coverage. This feature is called source-base because it operates on AST and preprocessor information directly, producing more precise coverage data compared to the traditional gcov coverage technique.

GitLab provides built-in integration of coverage information allowing for example reviewers to check if a MR is changing tested code or if it's increasing or decreasing the total coverage of the project. In this post we'll explain how to setup a CI job in a Rust project to feed source-base coverage information to GitLab.

Generating coverage profiles

The frst step is to add a new job to your CI pipeline, which will take care of generating the coverage reports. See the GitLab documentation if your project does not have any CI setup yet.

LLVM source-base code coverage instrumentation is currently only available in Rust nightly so our job will use an image providing this version of the compiler. It also needs the llvm-tools-preview component.

In order to generate the code coverage information, called raw profile, we need to set the environment variable RUSTFLAGS="-Zinstrument-coverage". By default, the profile is saved to a file called default.profraw. This will be a problem if we have multiple tests as each one will override the profile of the previous one. To avoid this, we'll also define LLVM_PROFILE_FILE with a generic pattern so each test will save its profile to its own file: LLVM_PROFILE_FILE="coverage-%p-%m.profraw".

Once we have setup these variables we just need to run the tests as usual. Here is how a coverage CI job would look like:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test

Generating coverage report

Now that the Rust compiler has generated the coverage profiles we can generate a report. This is done using grcov from Mozilla which is installed using cargo install.

We can then use it to generate a html report that we'll export as a job artifact. Here is an example of such report for the zbus crate.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
  artifacts:
    paths:
      - 'coverage'

We want our coverage report to cover only the code of our current crate, not its dependencies. This is achieved by passing the --ignore "*cargo*" flag to grcov to exclude code from the cargo registry. Depending of your exact setup you may have to adjust it. I'd suggest to first generate a report without any --ignore and then tweak it to exclude all the code not from your crate.

GitLab Test Coverage Visualization

The html report we just generated is very handy for manually checking which part of the code is covered by tests but is not usable directly by GitLab. To do so we need to feed it a cobertura report which is currently not supported by grcov.

Fortunately, it's possible to generate a lcov report and then convert it to cobertura as suggested in the ticket.

Passing this report to GitLab as a reports artifact will enable Test Coverage Visualization allowing reviewers to easily check if a MR is changing tested or untested code.

Here is the updated job:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Coverage parsing metric

Finally we want to tell GitLab the number of covered lines of code and the total number of lines so it can compute the coverage rate.

As grcov cannot easily produce this information yet we'll use the lcov tool:

    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov

The last step is to tell GitLab how to extract those numbers from the job logs. This is done in the CI/CD settings page, Test coverage parsing section, by setting this regular expression:

\s*lines\.*:\s*([\d\.]+%)

CI pipelines and merge requests will now display the coverage rate of the branch:

Conclusion

The flexibility of grcov and GitLab allowed our coverage job to provide:

• a full html report as an artifact;
• a cobertura report for GitLab integration;
• coverage rate metrics to GitLab.

We had to work around some current grcov limitations by using external tools to convert and parse the reports. Once grcov will have gained support for these formats the whole process will become much more straightforward.

Here is the final job, you can also check the CI of zbus and gstreamer-rs for real-life examples.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Note that this job will build and download the reporting tools at each run. A future improvement for projects running a lot of coverage reports would be to build a seperate docker container with all the tooling preinstalled, as we did in zbus.