on how to write good tests in COMP1511/COMP1917
a reply-to-public, originally written 2018-05-20 on OpenLearning

So is it ok to have one assert at the end, if in the log you can find all the errors and what went wrong? (which I believe is more helpful) or do I have to change it?

tl;dr: use lots of assertions to fail fast and fail hard on test failures.

Using something like assert(3) makes it easy to find the point where a test failure occurred. Sure, this kills the process immediately; however, with a debugger or similar attached, it’s easy to work backwards from the point of the assertion failure to what actually went wrong in the operation under test.

Killing the process under test as soon as it starts misbehaving seems like a counter-productive idea. And, yes, you get more information if you allow the processes under test to continue, but how meaningful is that data after the point of the first test failure?

If that test failure causes (or is a result of) some unpredictable or undefined internal state change, allowing the tests to continue doesn’t produce any useful output; if anything, that’s potentially more misleading.

One of the disputed tests in the last round was due to catastrophic memory corruption, and by allowing that test to continue, you allowed it to reach a point where the standard library’s memory management assertions failed, instead of quickly and robustly spotting errors.

So, after all that, sure, testing is hard.

On the other hand, here’s my sure-fire recipe to testing reliably and robustly.

  1. Set up a known, deterministic “state of the world” (i.e., everything that’s relevant; here, that’s just a Game ADT instance, but in other systems or environments you may have to think about how much is included in that).

    Students have been known to use rand(3) or scanf(3) to populate their test worlds; these don’t produce worlds that are deterministic or known, respectively.

  2. Pick an operation (which may be more than one function call!), and work out what the “state of the world” would be after doing that.

    This gives you an idea of the results you need to assure; not only the results of potentially only the function call, but also probing through other information that’s known. Don’t forget to check everything that’s relevant – not only things that changed, but things that didn’t (and shouldn’t) change.

  3. Between these two points where everything is well known, actually perform the operation under test.

  4. Rinse and repeat until the end of time, or you get bored, whichever comes first.

It’s important to make sure you totally separate all the worlds you create, and to only perform one testing operation on a world. If you want to observe the effect of a sequence of multiple operations \(a,b,c,d\), how do you know that \(d\) would work if \(a\) or \(b\) failed?

Instead, consider that testing the sequence of operations \(a,b,c,d\) is the same as setting up a world where \(a,b,c\) have already happened, and checking what would happen after \(d\) occurs.

You may also like my write-up on Black- and white-box tests, or my collection of test1511 testing hacks for first- and second-year student code.