Whole API Testing with Reflection

Whole API Testing with Reflection

Suppose you’re writing a library, and you want to test an assertion that logically applies to the entire library. For example, you might want to ensure that all enumerables produced by the library fail eagerly instead of lazily. Or maybe you want to check that all collections returned by the library are neutral instead of null.

One way to test these assertions over the entire library is via reflection. In this post, I’ll show you how.

Writing an API Test

In my recent spare time I’ve been working on PickleJar, which I mentioned last week. PickleJar allows you to create and combine “jars”, which allow you to parse and pack data via a description of the serialized format.

One important property that a jar should satisfy is “round tripping”. When a jar packs a value into data, and is then used to parse that data back into a value, all of the data should be consumed and the original value should be reproduced.

Lets write a test to check that all jars exposed by PickleJar’s API satisfy the round tripping property.

In PickleJar, all jars are created via the static Jar class, which contains property getters used to access primitive jars (e.g. Jar.Int32LittleEndian and Jar.UTF8) as well as methods to augment and combine jars (e.g. jar.RepeatNTimes(5) and jar.NullTerminated())

Lets start by getting all of the primitive jars. The following code uses reflection to iterate over Jar‘s public static properties, dynamically invoke each property’s getter, and store the resulting jars:

private static readonly IEnumerable<object> ApiJarGetters = 
    typeof(Jar) // <-- api's main class, exposes primitive jar getters and factory methods
    .GetProperties(BindingFlags.Static | BindingFlags.Public)
    .Select(e => e.GetValue(null))
    .ToArray();

We also want to test the augmented jars produced by Jar‘s methods. We can get all of the methods in the same way that we got all of the properties, but invoking methods is more complicated because they take parameters.

First, we need to deal with generic parameters. The following code will get all methods used to make jars and, when it finds a generic method, try setting all of the type parameters to int or to string:

private static IEnumerable<MethodInfo> FillInGenericParameters(MethodInfo method) {
    if (!method.IsGenericMethodDefinition) {
        yield return method;
        yield break;
    }

    var allIntTypeArgs = method.GetGenericArguments().Select(_ => typeof(int)).ToArray();
    yield return method.MakeGenericMethod(allIntTypeArgs);

    var allStringTypeArgs = method.GetGenericArguments().Select(_ => typeof(string)).ToArray();
    yield return method.MakeGenericMethod(allStringTypeArgs);
}

private static readonly IEnumerable<MethodInfo> ApiJarMakers = 
    typeof(Jar)
    .GetMethods(BindingFlags.Static | BindingFlags.Public)
    .SelectMany(FillInGenericParameters)
    .ToArray();

Note that, although using all ints and all strings is sufficient for our purposes here, in other cases we’d need something a bit more flexible.

Second, we need to deal with normal parameters. In order to invoke a method like RepeatNTimes<string>, which expects a jar for strings and an integer count, we need a string jar instance and an integer value before we can invoke the method.

To generate argument values we’ll use a method that takes a type and returns test values of that type. When the type is a jar, we can use the primitive jars we’ve already extracted. For other types we’ll use hardcoded values. There’s definitely nicer ways to generate these values (e.g. whatever Pex does, also see Haskell’s QuickCheck), but for now we’ll settle for simple:

private static IEnumerable<object> ChooseTestValues(Type type) {
    if (type == typeof(int))
        return new object[] { -100, -1, 0, 1, 2, 100 };
    if (type == typeof(long))
        return new object[] { -100L, -1L, 0L, 1L, 2L, 100L };
    ...
    if (type == typeof(IReadOnlyList<int>))
        return new object[] { new int[0], new[] { -1 }, new[] { 2, 3, 5, 7 } };
    if (type == typeof(Func<int, bool>))
        return new object[] { new Func<int, bool>(e => e % 2 == 0) };
    ...
    if (type == typeof(string))
        return new[] { "a", "bra", "ca", "da" };
    ...
    var matchingJars = ApiJarGetters.Where(type.IsInstanceOfType).ToArray();
    if (matchingJars.Length > 0) return matchingJars;
    throw new Exception(type.ToString());
}

Now, given a method to invoke, we can choose test values to provide for each argument. We’ll invoke the method with each possible combination of argument values. Sometimes these invocations will fail (e.g. because we passed a negative integer as a count argument), but when one succeeds the result is a jar we can test.

Putting it all together, we get a (presumably) representative sample of the jars that can be produced by the API. Here’s the code:

private static IEnumerable<object> JarsExposedByPublicApi() {
    var derivedJars = (from jarMaker in ApiJarMakers
                       from args in jarMaker.GetParameters()
                                            .Select(e => ChooseTestValues(e.ParameterType))
                                            .AllChoiceCombinationsVolatile()
                       let e = InvokeWithDefaultOnThrow(() => jarMaker.Invoke(null, args))
                       where e != null
                       select e
                      ).ToArray();
    (derivedJars.Length > 0).AssertTrue();

    return ApiJarGetters.Concat(derivedJars);
}

Side note: Because we’re only using primitive jars when a method asks for a jar, we’re not testing doubly-augmented jars like Jar.Utf8.NullTerminated().RepeatNTimes(2). Another omission is jars created by instantiating classes, such as new Jar.Builder {{"x", Jar.Float32}, {"y", Jar.Float32}}.Build(). Handling these cases involves the exact same concepts used to invoke methods on the primitive jars. I won’t bother covering them for this example.

With our somewhat-representative-sample-of-jars-that-can-be-created-by-the-API method in hand, all we need is a method to check the round tripping property and a test that feeds all of the jars into the check method:

[TestMethod]
public void TestApiHasOnlyValidJars() {
    foreach (dynamic jar in JarsExposedByPublicApi()) {
        AssertJarCanParseWhatItPacks(jar);
    }
}

private static void AssertJarCanParseWhatItPacks<T>(IJar<T> jar) {
    foreach (var item in ChooseTestValues(typeof(T)).Cast<T>()) {
        var itemData = jar.Pack(item);
        var parsed = jar.Parse(new ArraySegment<byte>(itemData));
        itemData.Length.AssertEquals(parsed.Consumed);
        item.AssertSimilar(parsed.Value);
    }
}

Done. The above test will fail if we implement and expose a jar with a dumb mistake that prevents round-tripping.

Benefits

I’ve found having tests that cover the entire API to be really useful.

The main downside I’ve encountered is artificial code coverage. Because API tests spider the entire API, most code ends up being marked as ‘covered’ regardless of whether or not it’s been properly tested. A distinction between coverage by API tests, which care about general semantics, and coverage by unit tests, which care about specific semantics, would be really useful.

The upsides have more than outweighed the downsides, so far. When I make a stupid mistake, there’s this wonderful tendency for it to be caught right away.

For example, I made a stupid mistake while implementing the null terminated jar (augments a jar so that it only parses data up to a null terminator). I forgot to append the null terminator when packing. Within three seconds of making the mistake, before I had even saved, the API test (being continuously run by NCrunch, which is amazing) had been run and caught the bug:

Awesome.

Summary

You can test that a correctness property applies to an entire API by using reflection to iterate over all things exposed by the API, and testing that they each satisfy the property. This is extremely useful.

I wouldn’t be surprised if this sort of functionality was already part of testing frameworks. I looked a bit, but google results related to testing+reflection are obsessed with other things (using reflection to violate encapsulation), and relevant questions on StackOverflow have answers that essentially say “use reflection” instead of giving a tool recommendation.

Discuss on Reddit

Comments are closed.

Twisted Oak Studios offers consulting and development on high-tech interactive projects. Check out our portfolio, or Give us a shout if you have anything you think some really rad engineers should help you with.

Archive