assay

• A first taste
• Loading
• Defining tests
  • Assertions
  • Structure
  • Teaching assay about your predicates
• Running from the REPL
  • The debugger
• ASDF test-op
• The command line
• Parallel runs
• Mocks
• Open questions and notes
• Migrating from Parachute
• License

assay is a zero-dependency test framework for SBCL, with the ergonomics of Go's testing, Rust's harness, Tasty and JUnit/AssertJ, plus a Mockito-style mocking layer. It is two files in one system: an agent can vendor assay.lisp alone and get everything except mocks.

A first taste

(defpackage #:parser/test (:use #:cl #:assay #:parser))
(in-package #:parser/test)

(deftest parse-numbers
  "Decimal integers, with and without sign."
  (is (= 42 (parse "42")))
  (is (= -1 (parse "-1")))
  (signals parse-error (parse "4x2")))

(deftest (large-input :timeout 2 :tags (:slow))
  (let ((input (make-string 1000000 :initial-element #\9)))
    (must (parse input) "must not blow the stack")
    (is (= 1000000 (digit-count (parse input))))))

CL-USER> (assay:run :parser/test)

parser/test  ✓  2 passed   (3 ms)
ok: 2 tests   (3 ms)
#<SUMMARY 2 tests, all passed>

One macro, is, does the work AssertJ needs a fluent API for: it reads its own source form, evaluates each operand once, and on failure shows every non-literal operand with its value plus a one-line explanation.

CL-USER> (is (= -1 (parse "-1")))   ; parse returns 1 here
✗ (is (= -1 (parse "-1")))
      (parse "-1") ⇒ 1
      differ by 2

A bare (is ...) at the REPL, outside any run, just prints what it would have recorded, so you can try the framework without writing a test.

Loading

assay is plain ASDF with no dependencies beyond SBCL and its contribs. Put the checkout where ASDF can find it and:

(asdf:load-system "assay")

Requires SBCL 2.4 or later. There is no portability layer; assay uses SBCL's threads, timeouts, backtraces and alien calls directly.

Defining tests

A test is a named body of code registered in a global table at load time; the package it was defined in is its group. There are no suites, fixtures or dependencies between tests.

(deftest name body...)
(deftest (name &key timeout tags skip wrap serial) [docstring] body...)

Option	Meaning
:timeout	interrupt the body after this many seconds; the result is an error
:tags	a list of keywords, for run's :tags and :exclude
:skip	a string; the test does not run and its status is :skip
:wrap	with- macro heads, each wrapping the body: (:wrap ((with-db)))
:serial	the test never runs on a worker thread (see Parallel runs)

The displayed name is package/name, both downcased. A leading string followed by more forms is the description. Redefining a test keeps its place in run order. deftest returns the name, like defun.

(find-test name) returns the test or NIL, (remove-test name) unregisters it, and (clear-tests package) unregisters every test in a package.

Assertions

All assertions record and continue; only must stops the test.

Form	Meaning
(is form [msg args...])	record, continue; show the operands on failure
(must form [msg args...])	like is, but abort the test on failure
(signals type form &key matching)	FORM must signal a condition of TYPE; returns it
(finishes form)	FORM must return without an error; returns its value
(fail control args...)	record a failure unconditionally
(skip &optional reason)	end the test with status :skip
(note control args...)	attach a note, shown only if the test fails
(abort-test)	end the innermost test or subtest now
(approx= a b &key tolerance)		a - b	<= tolerance; has an explainer

Message arguments are evaluated only on failure, so (is x "got ~a" (expensive)) costs nothing when it passes.

Structure

subtest is Go's t.Run: a named child with its own result and abort boundary. cases is the table-driven idiom on top of it.

(deftest arithmetic
  (cases (expr expected)
      '(("1+1" 2) ("2*3" 6) ("7/2" 7/2))
    (is (= expected (eval-string expr)))))

Each row becomes a subtest named from the variables, e.g. expr="7/2" expected=7/2. There is no fixture mechanism: a with- macro plus unwind-protect is the fixture, and :wrap applies one to a whole body.

Teaching assay about your predicates

explain is a generic function dispatched on the operator of a failed function-call form. Add a method to explain your own predicates:

(defmethod assay:explain ((op (eql 'vec3=)) values)
  (destructuring-bind (a b) values
    (format nil "components differ: ~a" (map 'list #'- a b))))

A method returns one line without a trailing newline, or NIL when it has nothing useful to say. The shipped methods turn the five minutes everyone loses to (eq "a" "a") into one line:

CL-USER> (is (eq (copy-seq "a") (copy-seq "a")))
✗ (is (eq (copy-seq "a") (copy-seq "a")))
      (copy-seq "a") ⇒ "a"
      (copy-seq "a") ⇒ "a"
      EQUAL but not EQ (distinct objects); did you mean EQUAL?

Running from the REPL

(run &optional (what t)
     &key reporter verbose debug parallel
          tags exclude fail-fast shuffle seed signal-on-failure)

what is a designator or a list of them, selected in definition order, each test once:

what	Selects
t	every registered test
a keyword/package/string	the tests in that package
a non-keyword symbol	the test with that name
a string with *	tests whose package/name matches the glob
a test object	itself, registered or not

:tags keeps tests carrying a listed tag; :exclude drops tests carrying one; exclusion wins. :verbose t prints a line per test, not only failures. :fail-fast stops after the first failure. :shuffle t runs in a random order and prints the seed; :seed n reproduces one.

run returns a summary; (passed-p summary) is the bottom line. *last-summary* holds the most recent top-level run, and (run-failed) re-runs just the tests that failed, errored or did not run:

(run :my-project/test)        ; everything
(run-failed :verbose t)       ; only what went wrong last time, verbosely
(run "my-project/parse-*")    ; a glob

The debugger

*debug* (and run's :debug) is NIL, :errors or t. With :errors, an unhandled error in a test enters the debugger from the handler, before the stack unwinds, so the real frames are there. With t, a failed assertion enters the debugger too. The restarts, innermost first, are continue (record and go on), abort-test (end this test), retry (run it again) and skip-test. C-c reaches the debugger as usual; it is not an error and assay does not catch it.

ASDF test-op

Wire test-op to assay so asdf:test-system runs the suite and fails the build when a test does:

(defsystem "my-project/test"
  :depends-on ("my-project" "assay")
  :pathname "tests"
  :components ((:file "my-project-test"))
  :perform (test-op (o c)
             (symbol-call :assay :run :my-project/test :signal-on-failure t)))

:signal-on-failure t makes run signal tests-failed after reporting, which test-op turns into a failed build.

The command line

main parses a list of strings, runs, prints, and exits 0 when everything passed, 1 when it did not, 2 for a usage error. A shell entry point:

sbcl --non-interactive \
     --eval '(asdf:load-system "my-project/test")' \
     --eval '(assay:main)' \
     --end-toplevel-options my-project/test --parallel

Argument	Meaning
pkg, pkg/name, pkg::name, glob	which tests to run (case-insensitive)
-v, --verbose	a line per test
--fail-fast	stop after the first failure
--parallel[=N], --serial	parallelism
--shuffle, --seed N	random order, or reproduce one
--tags a,b, --exclude a,b	tag filters
--no-capture, --no-color, --ascii	output tweaks
--help	usage, exit 0

Parallel runs

Parallelism is per test, opt-in per run, opt-out per test. (run :parallel t) uses one worker thread per processor; :parallel N uses N; *parallel* is the default. The non-serial tests go onto a queue the workers drain; then the :serial tests run on the calling thread. The final report is identical to a serial run's, because it is printed from the summary in definition order.

ok: 128 tests   (310 ms, 8 workers)

A test is parallel-safe when it touches only its own data or rebinds specials with let. A test that setf\s a global, writes a shared file, or uses with-mocks is not, and must carry :serial t:

(deftest (writes-the-cache-file :serial t)
  ...)

with-mocks on a worker thread is an error that names the test and the fix, so you find out the first time you run the suite in parallel.

Mocks

with-mocks swaps global fdefinition\s for its body, records every call, and restores them however the body exits. The syntax is flet's. Verification is is with three functions, so a failed verification reads like every other failure. Because it replaces global definitions, a test that mocks must be :serial t.

(deftest (welcome-email :serial t)
  (with-mocks ((fetch-user (id) (make-user :name "Kim" :email "kim@example.com"))
               send-email)                       ; a bare name records and returns NIL
    (notify-user 42)
    (is (called-p 'fetch-user 42))
    (is (called-p 'send-email "kim@example.com" (any 'string)))
    (is (= 1 (call-count 'send-email)))))

Function	Meaning
(call-original args...)	inside a stub, the function the mock replaced
(calls &optional name)	argument lists, oldest first (or every call with NAME off)
(call-count name)	how many times NAME was called
(called-p name pattern...)	some call matched PATTERN (or called at all)
(any &optional type)	matcher: any argument of TYPE
(satisfying predicate)	matcher: any argument the predicate accepts

A stub that applies call-original is a spy; (calls) compared with equal is InOrder and verifyNoMoreInteractions; (calls 'name) is ArgumentCaptor. A mock object for a CLOS protocol is a subclass with defmethod\s, plus with-mocks on the generic function when calls should be recorded.

Open questions and notes

These are points where SBCL's reality, or a tension inside the design, led to a decision worth recording.

• Showing constant-foldable operands. SBCL's constantp folds (+ 1 1) to a constant, which would hide it on failure. assay treats only atoms (via constantp) and quote~/~function~/~lambda forms as literals, so (is (= 1 (+ 1 1))) still shows (+ 1 1) ⇒ 2.
• Keeping the test frame on the stack. A test body ending in a tail call would lose its (test name) frame to tail-call optimization, and with it the backtrace's trim boundary. deftest adds a trailing nil so the frame survives. A test body that provably never returns (ends in error) may draw a one-off "deleting unreachable code" compiler note; it is harmless.
• inlinep. On SBCL 2.6.9 (sb-int:info :function :inlinep name) returns the symbol cl:inline, not the keyword :inline the design assumed; assay compares against the symbol.
• First-load style-warnings. Compiling and loading any macro-defining system in one image makes SBCL report "redefining … in DEFMACRO" style-warnings, because defmacro defines the macro at compile time and again at load time. This is not specific to assay; a second load from the cached fasls is silent, and assay's own code produces no other warnings.
• Mocking a narrowly-typed function. If SBCL has inferred a function's return type at a call site (for instance because it returns a single constant), a mock that returns a different type can trip that call site's type check. Real code rarely returns a single constant, but mock such a function through a wrapper of your own if you hit it.
• Accessor docstrings. Every exported function, macro, variable, generic function and class or condition type carries its own docstring. The slot accessors (test-name, result-status, event-run and the rest) are documented in their class's or condition's docstring, the usual Common Lisp way, rather than each carrying a separate string.
• Out of scope, left for later: a JUnit-XML reporter for CI, golden-file assertions, a property-testing provider, with-methods via add-method~/~remove-method, running a test on C-c C-c, expected-failure marking, and detection of unused stubs.

Migrating from Parachute

Parachute	assay
(define-test name ...)	(deftest name ...)
(is = a b)	(is (= a b))
(true x)	(is x)
(false x)	(is (not x))
(of-type type x)	(is (typep x 'type))
(fail ...)	(fail ...) (unchanged)
:parent	dropped; use subtest to nest
:fix	a let, or deftest's :wrap

License

MIT. See LICENSE.