GotW #97 Solution: Assertions (Difficulty: 4/10)

Assertions have been a foundational tool for writing understandable computer code since we could write computer code… far older than C’s assert() macro, they go back to at least John von Neumann and Herman Goldstine (1947) and Alan Turing (1949). [1,2] How well do we understand them… exactly?

1. What is an assertion, and what is it used for?

An assertion documents the expected state of specific program variables at the point where the assertion is written, in a testable way so that we can find program bugs — logic errors that have led to corrupted program state. An assertion is always about finding bugs, because something the programmer thought would always be true was found to actually be false (oops).

For example, this line states that the program does not expect min to exceed max, and if it does the code has a bug somewhere:

If in this example min did exceed max, that would mean we have found a bug and we need to go fix this code.

GUIDELINE: Assert liberally. [3] The more sure you are that an assertion can’t be false, the more valuable it is if it is ever found to be false. And in addition to finding bugs today, assertions verify that what you believe is “obviously true” and wrote correctly today actually stays true as the code is maintained in the future.

GUIDELINE: Asserted conditions should never have side effects on normal execution. Assertions are only about finding bugs, not doing program work. And asserted conditions only evaluated if they’re enabled, so any side effects won’t happen when they’re not enabled; they might sometimes perform local side effects, such as to do logging or allocate memory, but the program should never rely on them happening or not happening. For example, adding an assertion to your code should never make a logically “pure” function into an impure function. (Note that “no side effects on normal execution” is always automatically true for violation handlers even when an assertion system such as proposed in [4] allows arbitrary custom violation handlers to be installed, because those are executed only if we discover that we’re in a corrupted state and so are already outside of normal execution. [5] For conditions, it’s up to us to make sure it’s true.)

2. C++20 supports two main assertion facilities… For each one, briefly summarize how it works, when it is evaluated, and whether it is possible for the programmer to specify a message to be displayed if the assertion fails.

assert

The C-style assert is a macro that is evaluated at execution time (hopefully enabled at least during your unit testing! see question 4) if NDEBUG is not set. The condition we pass it must be a compilable boolean expression, something that can be evaluated and converted to bool.

It doesn’t directly support a separate message string, but implementations will print the failed condition itself, and so a common technique is to embed the information in the condition itself in a way that doesn’t affect the result. For example, a common idiom is to append &&"message"  (a fancy way of saying &&true):

static_assert

The C++11 static_assert is evaluated at compile time, and so the condition has to be a “boolean constant expression” that only refers to compile-time known values. For example:

It has always supported a message string, and C++17 made the message optional.

Bonus: [[assert: ?

Looking forward, a proposed post-C++20 syntax for assertions would support it as an attribute, which has a number advantages including that it’s not a macro. [4] This version would be evaluated at execution time if checking is enabled. Currently that proposal does not have an explicit provision for a message and so programmers would use the && "message" idiom to add a message. For example:

3. If an assertion fails, what does that indicate, and who is responsible for fixing the failure?

A failed assertion means that we checked and found the tested variables to be in an unexpected state, which means at least that part of the program’s state is corrupt. Because the program should never have been able to reach that state, two things are true:

• There is a program bug, possibly in the assertion itself. The first place to look for the bug is in this same function, because if prior contracts were well tested then likely this function created the first unexpected state. [5]
• The program cannot recover programmatically by reporting a run-time error to the calling code, because by definition the program is in a state it was not designed to handle, so the calling code isn’t ready for that state. It’s time to terminate and restart the program. (There are advanced techniques that involve dumping and restarting an isolated portion of possibly tainted state, but that’s a system-level recovery strategy for an impossible-to-handle fault, not a handling strategy for run-time error.) Instead, the bug should be reported to the human developer who can fix the bug.

GUIDELINE: Don’t use assertions to report run-time errors. For example, don’t use an assertion to check that a remote host is available, or that the user types valid input. Yes, std::logic_error was originally created to report bugs (logic errors) using an exception, but this is now widely understood to be a mistake; don’t follow that pattern.

Referring to this example:

In this code, if the assertion at line A is false, that means what the function actually did before the assertion doesn’t match what the assertion condition expected, so there is a bug somewhere in this function — either before or within the assertion.

This demonstrates why assertions are primarily about eliminating bugs, which is why we test…

4. Are assertions primarily about checking at compile time, at test time, or at run time? Explain.

Assertions are primarily about finding bugs at test time. But assertions can also be useful at other times because of some well-known adages: “the sooner the better,” “better late than never,” and “never [will] I ever.”

Bonus points for pointing out that there is also a fourth time in the development cycle I didn’t list in the question, when assertions can be profitably checked. Here they are:

Of course this can be even more nuanced. For example, you might make different decisions about enabling assertions if your “run time” is an end user’s machine, or a server farm, or a honeypot. Also, checking isn’t free and so you may enable run-time checking for severe classes of bugs but not others, such as that an operating system component may require checking in production for all out-of-bounds violations and other potential security bugs, but not non-security classes of bugs.

First, “the sooner the better”: It’s always legal and useful to find bugs as early as possible. If we can find a bug even before actually executing a compiled test, then that’s wonderful. This is a form of shift-left. We love shift-left. There are two of these times in the graphic:

• (Earliest, best) Edit time: By using a static analysis tool that is aware of assert and can detect some violations statically, you can get some diagnostics as you’re writing your code, even before you try to compile it! Note that to recognize the assert macro, you want to run the static analyzer in debug mode; analyzers that run after macro substitution won’t see an assert condition when the code is set to make release builds since the macro will expand to nothing. Also, usually this kind of diagnostic uses heuristics and works on a best-effort basis that catches some mistakes while not diagnosing others that look similar. But it does shift some diagnostics pretty much all the way left to the point where you’re actually writing the code, which is great when it works… and you still always have the next three assertion checking times available as a safety net.
• (Early) Compile time: If a bug that depends only on compile-time information can be detected at compile time even before actually executing a compiled test, then that’s wonderful. This is one reason static_assert exists: so that we can express tests that are guaranteed to be performed at compile time.

Next, the primary target:

• Test time: This is the main time tests are executed. It can be during developer-machine unit testing, regression testing, build-lab integration testing, or any other flavor of testing. The point is to find bugs before they escape into production, and inform the programmer so they can fix their bug.

Finally, “better late than never” (safety net) or “never [will] I ever” (intolerable severe condition):

• (Late) Run time: Even after we release our code, it can be useful to have a way to enable checking at run time to at least log-and-continue (e.g., using facilities such as [6] or [7]). One motivation is to know if a bug made it through testing and out into the field and get better late-debug diagnostics; this is sometimes called shift-right but I think of it as much as being about belt-and-suspenders. Another motivation is to ensure that severe classes of bugs ensure execution will halt outright if we cannot tolerate continuing after such a fault is detected.

Importantly, in all cases the motivation is still debugging: Findings bugs early is still debugging, just better (sooner and less expensive). And finding bugs late that escaped into production is still debugging, just worse (later and more expensive). Each of these times is a successive safety net for bugs that make it past the earlier times.

Because at run time we may want to log a failed assertion, our assertion violation handler should be able to USE-A logging system, but the relationship really is USES-A. An assertion violation handling system IS-NOT-A general-purpose logging system, and so a contracts language feature shouldn’t be designed around such a goal. [5]

Finally, speaking of run time: Note that it can be useful to write an assertion, and also write code that does some handling if the assertion is false. Here’s an example from [8]:

You might see this pattern written interleaved as follows to avoid duplicating the condition, and this is one of the major patterns that leads to writing assert(!"message"):

At first this may look like it’s conflating the distinct “bug” and “error” categories we saw in Question 3’s table. But that’s not the case at all, it’s actually deliberately using both categories to implement “defense in depth”: We assert something in testing to minimize actual occurrences, but then in production still provide fallback handling for robustness in case a bug does slip through, for example if our test datasets didn’t exercise the bug but in production we hit some live data that does.

Notes

With thanks to Wikipedia for the first two references.

[1] H. H. Goldstine and J. von Neumann. “Planning and Coding of problems for an Electronic Computing Instrument” (Report on the Mathematical and Logical Aspects of an Electronic Computing Instrument, Part II, Volume I, p. 12; Institute for Advanced Study, April 1947).

[2] Alan Turing. “Checking a Large Routine” (Report of a Conference on High Speed Automatic Calculating Machines, pp. 67-9, June 1949).

[3] H. Sutter and A. Alexandrescu. C++ Coding Standards (Addison-Wesley, 2004). Item 68, “Assert liberally to document internal assumptions and invariants.”

[4] G. Dos Reis, J. D. Garcia, J. Lakos, A. Meredith, N. Myers, and B. Stroustrup. “P0542: Support for contract based programming in C++” (WG21 paper, June 2018). To keep it as legal compilable (if unenforced) C++20 for this article I modified the syntax from : to ( ). That’s not a statement of preference, it’s just so the examples can compile today to make them easier to check.

[5] Upcoming GotWs will cover postconditions, preconditions, invariants, and violation handling.

[6] G. Melman. spdlog: Fast C++ logging library (GitHub).

[7] Event Tracing for Windows (ETW) (Microsoft, 2018).

[8] H. Sutter. “P2064: Assumptions” (WG21 paper, 2020).

Acknowledgments

Thank you to the following for the comments on drafts of this article: Joshua Berne, Gábor Horváth, Andrzej Krzemieński, Andrew Sutton, and Reddit user “evaned.”