test1511

download / changelog / in the wild / documentation

This file contains several horrible hacks to make it easier to write tests for student code. I’ve collected together many interesting bits of cruft along the way.

• The most useful thing here is the TEST macro, which takes two or more arguments: first, an expression to test; and following that, a constant format string and format arguments. It either prints to stderr and runs assert(3), or produces a stdout complying with the Test Anything Protocol, TAP:

   TEST (6 * 9 == 42,
       "life, the universe, everything (base 10s)");
   TEST (6 * 9 == 54,
       "life, the universe, everything (base 13s)");
  

  In assert(3) mode, on stderr:

   -- t1: life, the universe, everything (base 10s)
   Assertion failed
   [and, if that didn't happen:]
   -- t2: life, the universe, everything (base 13s)
  

  In TAP mode, on stdout:

   not ok 1 # life, the universe, everything (base 10s)
   ok 2 # life, the universe, everything (base 13s)
  

• For testing floating-point operations, I have feq, because == doesn’t do what you expect. It’s a static inline function, and works a treat with the TEST macro.

   TEST (feq (0.0, 0.0, 1),
       "0.0 == 0.0 (± 1.0)")
  

• I have have_symbol, a function that tests if a symbol exists in an executable; it uses dlopen(3), dlsym(3), and ugly hacks. It may not be very portable…

   const char *sym = "imageAsBMP";
   if (! have_symbol (sym))
       errx (EX_SOFTWARE, "this test needs '%s'", sym);
  

• Various memory protection checks, wrapping ASan’s interface in a slightly more pleasant (and less ifdef-heavy) one.

  • mem_address_is_poisoned :: Addr# -> Bool
  • mem_region_is_poisoned :: (Addr#, Word#) -> Bool
  • mem_poison_region :: (Addr#, Word#) -> ()
  • mem_unpoison_region :: (Addr#, Word#) -> ()
  • mem_describe_address :: Addr# -> IO ()

  And, to use it:

   String s = newString ("");
   destroyString (s);
   TEST (mem_address_is_poisoned (s),
       "s is now gone")
  

• TODO: write some linked-list test goop.

changelog

• 2014-05-?? Jashank Jeremy z5017851@cse.unsw.edu.au
  • (… the distant past: “test_tap.h”, a terrible set of hacks that have evolved into the current TEST macro.)
• 2017-09-27 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • Refactor TEST macro into the two forms here: one that produces Test Anything Protocol (TAP) output, and one that assert(3)’s and dumps information on stderr.
• 2017-10-10 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • Add feq, to do floating-point equality. https://wiki.jashankj.space/CSE/Hints/COMP1511FloatEqual/
• 2017-10-19 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • Add have_symbol, to do symbol presence analysis in ADT implementations using dlopen and dlsym.
• 2017-10-28 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • mem_* wrappers around the #ifdef mess for sanitizers.
• 2018-05-17 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • Add diagnostic -Wno-unused-function.
  • Move header guard within -Wno-unused-macros
  • List a canonical source for test1511.h! Given how often I use it for various student-code-testing projects, it makes sense to publish it on my website.

where you’ve seen it

• This was used in COMP2521 18x1, where we leveraged the TAP output and Perl’s prove, so our assignment dryruns could effectively be

  make test{Game,Drac,Hunter}View CC=3c &&
  prove -f ./test{Game,Drac,Hunter}View
  

• This was used in COMP1511 17s2, where we developed it to write tests for, student submissions and sample solutions, adding hacks and quirks along the way.

documentation

TEST

• macro TEST (status, descr):
  Test a thing.

  status: Test ‘passed’ if a true value.
  descr: Short description of test.

• global variable _test_:
  the number of tests that have been run

• global variable _fail_:
  the number of tests that have failed.

Floating-point equality

• function feq (double x, double y, double eps) -> bool:
  Floating-point equality. Compares the left-hand side with the right-hand side, with a margin of error around the right.

  x: Left-hand side
  y: Right-hand side
  eps: Margin of error (taken on the right-hand side)

Symbol existence

• function have_symbol (const char *sym) -> bool:
  Symbol existence. Does the named symbol exist?

  Uses dlopen(3) and dlsym(3). May need -ldl, maybe also -rdynamic…? dcc’s ASan dependency may give us -rdynamic, so we’d only need -ldl on silly platforms.

  sym: A symbol name to resolve in the currently-executing process.

AddressSanitizer shims

If we’re running with the Address Sanitizer, we can use its exposed interface to do memory checks. Wrap the useful bits of ASan in a slightly more pleasant and less ifdef-heavy interface.

For more details, see <sanitizer/asan_interface.h>. (llvm/projects/compiler-rt/include/sanitizer/asan_interface.h)

• function mem_address_is_poisoned (void *addr) -> bool:
  Report if this address is “poisoned” – whether a one-byte read will cause a memory error, either by ASan tracking or not.

  addr: the address to check

• function mem_region_is_poisoned (void *addr, size_t size) -> bool:
  Report if the region of memory from addr to addr + size is poisoned, as above.

  addr: the base address to check
  size: the size of the region to check

• function mem_poison_region (void *addr, size_t size):
  Mark this region of memory as “poisoned”.

  addr: the base address to mark
  size: the size of the region to mark

• function mem_unpoison_region (void *addr, size_t size):
  Mark this region of memory as not poisoned.

  addr: the base address to unmark
  size: the size of the region to unmark

• function mem_describe_address (void *addr) -> IO ():
  Print out what’s known about the given address.

  addr: the address to report on