Events and reporters

Everything a reporter might want to know is a condition the runner signals and never errors. A reporter is an object with report methods for the events it cares about. The debugger is just one more handler on the same conditions.

The events

(define-condition event ()
  ((run :reader event-run)))                 ; the summary of the run that produced it
(define-condition test-event (event)
  ((result :reader event-result)))
(define-condition test-started  (test-event) ())
(define-condition test-finished (test-event) ())
(define-condition assertion-failed (test-event)
  ((assertion :reader event-assertion)))
(define-condition test-error (test-event)
  ((condition :reader event-condition)))
(define-condition run-finished (event)
  ((summary :reader event-summary)))

• test-started when a test or subtest begins; test-finished when it ends, however it ended.
• assertion-failed from the assertion machinery, before the failure's effect on must.
• test-error from the error handler, with the stack still intact.
• run-finished once, after every result is in, carrying the complete summary.

event-run identifies the run that produced the event, which is what makes nested runs work: a reporter is delivered only its own run's events.

The report protocol

(defgeneric report (reporter event)
  (:method ((reporter reporter) (event event)) nil))   ; report only what you want

• tree-reporter (:verbose boolean) prints the Output Format.
• quiet-reporter prints nothing.

Delivery is serialized: run wraps every report call in one mutex, so a reporter needs no locking of its own, even under a parallel run. A report method must not itself signal an event.

A custom reporter

Count outcomes as they happen:

(defclass counting-reporter (assay:reporter)
  ((passed :initform 0 :accessor passed)
   (failed :initform 0 :accessor failed)))

(defmethod assay:report ((r counting-reporter) (e assay:test-finished))
  (case (assay:result-status (assay:event-result e))
    (:pass (incf (passed r)))
    ((:fail :error) (incf (failed r)))))

(defmethod assay:report ((r counting-reporter) (e assay:run-finished))
  (format t "~&~d ok, ~d broken~%" (passed r) (failed r)))

(assay:run :my-project/test :reporter (make-instance 'counting-reporter))

You can also skip reporters entirely and wrap run in your own handler-bind on these conditions — a JUnit-XML writer, a progress bar, or a notifier is a handful of methods.

The debugger

*debug* is one more handler on the same conditions. With :errors, the test-error handler enters the debugger before the stack unwinds; with t, the assertion-failed handler does too, so the condition you see is the assertion, whose report is the block from Output Format. The restarts are listed under Running.

Back to Home.