Running

run selects tests, runs them, reports through a reporter, and returns a summary.

(run &optional (what t)
     &key reporter verbose debug parallel
          tags exclude fail-fast shuffle seed signal-on-failure)

It is also callable as (run), (run what) or (run what :key value ...); what defaults to t.

Selecting tests

what is a designator or a list of designators; the selection is their union in definition order, each test at most once.

what	Selects
t	every registered test
a keyword, a package, or a package-naming string	the tests in that package
a non-keyword symbol	the test with that name (an unknown name is an error)
a string containing *	tests whose package/name matches the glob (* matches any run of characters, including /; a pattern without / matches the test name alone)
a test object	itself, registered or not

(run)                      ; every test
(run :parser/test)         ; one package
(run 'parse-numbers)       ; one test
(run "parser/*")           ; a glob
(run (list :parser/test :lexer/test))

:tags keeps only tests carrying at least one listed tag; :exclude removes tests carrying any; exclusion wins. Tags are compared with eq.

(run t :exclude '(:slow))
(run :parser/test :tags '(:regression))

Options

Option	Meaning
:reporter	a reporter instance; defaults to a tree-reporter
:verbose	t makes the default reporter print a line per test
:debug	NIL, :errors or t — see below; forces one worker
:parallel	NIL, t (one per processor) or a positive integer; see Parallel
:fail-fast	stop taking new tests after the first failure
:shuffle	run in a random order and record the seed
:seed	reproduce a shuffled order
:signal-on-failure	signal tests-failed after reporting if the run did not pass

The summary

run returns a summary:

Reader	Value
summary-tests	the selection, in run order
summary-results	one result per test that ran, in finishing order
summary-seed	the shuffle seed, or NIL
summary-workers	the worker count
summary-duration	the whole run's wall time, in seconds
(passed-p summary)	true when every selected test passed or was skipped

A summary prints its bottom line:

#<SUMMARY 34 tests, all passed>
#<SUMMARY 34 tests, 2 failed, 1 error>
#<SUMMARY 34 tests, 3 not run, 1 failed>

Each result answers to result-name, result-parent, result-status (:pass, :fail, :error or :skip), result-assertions, result-children, result-notes, result-condition, result-backtrace, result-output, result-skip-reason and result-duration; (failed-p result) is true for :fail or :error.

Re-running what failed

*last-summary* holds the summary of the most recent top-level run (a run started inside a test does not set it). run-failed re-runs exactly the tests that failed, errored or did not run last time, passing its keyword arguments on to run:

(run :my-project/test)
(run-failed :verbose t)        ; just the broken ones, verbosely

Nested runs

Runs nest — assay tests itself with its own runner. Events carry the summary of the run that produced them, so an outer reporter ignores an inner run's events; an inner run's result binding, restarts and handlers are its own.

The debugger

*debug* (and run's :debug) is NIL, :errors or t.

• :errors — an unhandled error in a test enters the debugger from the handler, before the stack unwinds, so the real frames are there.
• t — a failed assertion enters the debugger too, on the assertion-failed condition, whose report is the assertion block.

Any non-NIL value forces a single worker. The restarts available, innermost first:

Restart	Effect
continue	record the failure or error and go on
abort-test	end the innermost test or subtest
retry	run this test again from the start
skip-test	give this test status :skip and move on

C-c reaches the debugger as usual; sb-sys:interactive-interrupt is not an error and assay does not catch it.

Next: Command Line.