Problem

At my current client we have been heavily investing in UI tests.
We make extensive use of Firebase TestLab to run our instrumented tests twice a day.

Until recently we ran our UI tests without the need for any mocking.
This also meant we have huge maintenance to keep the pipeline green 🟢.
In our effort to keep the benefits of UI tests while at the same time increase the stability, we converted a big chunk of our test cases to test fragments in isolation.

Since we have good experiences using MockK for our unit tests we decided to use the Android-MockK dependency for our instrumented tests.

When running the tests locally they passed but, running them on Firebase they consistently failed with the following cryptic stack trace:

io.mockk.MockKException: Missing calls inside verify { ... } block.
at io.mockk.impl.recording.states.VerifyingState.checkMissingCalls(VerifyingState.kt:52)
at io.mockk.impl.recording.states.VerifyingState.recordingDone(VerifyingState.kt:21)
at io.mockk.impl.recording.CommonCallRecorder.done(CommonCallRecorder.kt:47)
at io.mockk.impl.eval.RecordedBlockEvaluator.record(RecordedBlockEvaluator.kt:60)
at io.mockk.impl.eval.VerifyBlockEvaluator.verify(VerifyBlockEvaluator.kt:30)
at io.mockk.MockKDsl.internalVerify(API.kt:118)
at io.mockk.MockKKt.verify(MockK.kt:149)
at io.mockk.MockKKt.verify$default(MockK.kt:146)
at com.example.our.app.OurUITest.method(OurUITest.kt:75)

This trace didn’t make any sense since we were using a mock inside the verify block. Hence why it worked locally. 🤔
But why did it not work on Firebase testlab?

We use this script to launch our UI tests on Firebase.

gcloud firebase test android run \
  --app testlab/build/outputs/apk/debug/testlab-debug.apk \
  --test $MODULE_PATH/build/outputs/apk/androidTest/debug/*.apk \

By default when you don’t select a specific device/emulator it will use a Pixel 2 with API 27. Comparing the default Firebase configuration we noticed the API level was different from our local emulator, which was using API 29.
Luckily we can specify the API level in the gcloud command.

gcloud firebase test android run \
  --app testlab/build/outputs/apk/debug/testlab-debug.apk \
  --test $MODULE_PATH/build/outputs/apk/androidTest/debug/*.apk \
--device version=29,model=Pixel2

When we ran our test on Firebase with API 29 the tests started passing again.

Why? 🤔
And how can we make our instrumented tests run on lower API levels? 🤔

A-ha moment

Turning to Google yielded an open feature request (#182) on the MockK issue board. Turns out mocking on the JVM is entirely different than mocking during instrumentation.
What’s more, is the ability to mock final classes during instrumented tests is dependent on the API level you are running.

Just like every modern Android codebase, we adopted Kotlin as our programming language of choice for Android Development.
By default every Kotlin class you create is final by default.

It is even mentioned in the MockK documentation

Implementation is based on dexmaker project. With Android P, instrumentation tests may use full power of inline instrumentation, so object mocks, static mocks and mocking of final classes are supported. Before Android P, only subclassing can be employed, so you will need the ‘all-open’ plugin.

Verifying this we can indeed see that io.mockk:mockk-android is using dexmaker. Which is capable of generating .dex files at runtime.

+--- io.mockk:mockk-android:1.10.6
    +--- org.jetbrains.kotlin:kotlin-stdlib:1.5.0
    +--- io.mockk:mockk:1.10.6
    +--- io.mockk:mockk-agent-android:1.10.6
    |    +--- com.linkedin.dexmaker:dexmaker:2.21.0
    |    \--- org.objenesis:objenesis:2.6
    \--- io.mockk:mockk-agent-api:1.10.6

Unfortunately, we can not use the dexmaker library for lower API versions.

This functionality requires OS APIs which were introduced in Android P and cannot work on older versions of Android.

If we wish to enable mocking support for lower API levels we have two options:

• Kotlin all-open plugin
• DexOpener

You can read more about Kotlin all-open plugin in this blogpost as we’ll be focusing on using DexOpener in this blogpost.

DexOpener to the rescue

If you want to skip ahead; I created a dummy project where you can see a full implementation of the DexOpener library.

DexOpener describes themselves as:

An Android library that provides the ability to mock your final classes on Android devices.

Exactly what we need!
Adding the dependency is easy.

Add https://jitpack.io to your repositories block

allprojects {
repositories {
        maven { url 'https://jitpack.io' }
google()
        mavenCentral()
        jcenter()
}
}

Add the dependency itself to the module.

androidTestImplementation 'com.github.tmurakami:dexopener:2.0.5'

Inside your package under the androidTest directory add a new class DexOpeningTestRunner .

As you can see from the code, we only activate DexOpener when we detect that we are running on < Android P since MockK will use DexMaker for higher API levels.

The last step is to enable our custom instrumentation runner in the build.gradle

Bonus Multi-module

What might not be immediately obvious is the package directive of the custom testrunner is very important to be placed correctly.

Consider you have a multi-module project, you probably want to share this test runner with other modules.
In which case you will have to ensure the DexOpeningTestRunner is placed in a package used by all the modules.

You can find a full demo project on Github.

Happy testing