Errors in Jobs

A job runs on a worker, but the code that cares about its errors usually runs elsewhere. cl-threads makes sure a job's error reaches someone: whoever waits, an error handler, or you in the debugger. The pool's :on-error option chooses between two modes.

`:on-error`	What a job's error does
`:signal` (default)	The job is skipped and the other jobs run. The error goes to the group's error handler at once, if it has one; otherwise it is kept, and `wait-for-jobs` or `parallel-for` signals a `job-error` carrying it.
`:debug`	The debugger opens in the worker, with the restarts `retry-job` and `skip-job`. For interactive development.

`:signal`: errors come to whoever waits

(handler-case
    (th:parallel-for *pool* 0 10
                     (lambda (start end)
                       (declare (ignore end))
                       (when (= start 5) (error "Chunk ~D failed." start)))
                     :chunk-size 1)
  (th:job-error (condition)
    (th:job-error-condition condition)))
;; => #<SIMPLE-ERROR "Chunk 5 failed.">

The failing job is skipped. The other jobs run to the end before the error is signalled, so parallel-for and wait-for-jobs never return while jobs are still running.
The waiter gets a job-error. job-error-condition is the job's own error, and the report shows both:
```
A job of #<JOB-POOL 4 workers> signalled an error.
  The job's error: Chunk 5 failed.
```
If several jobs fail, the first error is kept and the rest are dropped.
After wait-for-jobs has signalled, the group is clear of the error and can be reused.
The pool goes on working.

job-error is a threads-error, which is an error.

Error handlers

A group's error handler is a function of the condition. It gets each of the group's errors as it happens, on the worker, and the error is then not kept for wait-for-jobs:

(let ((group (th:make-job-group *pool*
                                :error-handler (lambda (condition)
                                                 (log-error "Asset failed: ~A" condition)))))
  (th:submit-job group (lambda () (error "Could not read ~A." "missing.png")))
  (th:wait-for-jobs group))     ; returns normally; the handler logged the error

The handler:

runs on the worker, at once, while the failing job's dynamic extent is still active, so it can inspect the stack or log it;
is called once per failing job, possibly from several workers at once, so it must be safe to call concurrently: guard any shared log with a lock;
must not signal an error itself.

Jobs nobody waits for

Jobs submitted to a group that nobody waits for have no waiter to receive their errors. Without an error handler, their errors are kept in the group and never seen. Give such groups a handler.

The easiest way is a default for the whole pool. make-job-pool's :error-handler is the handler of every group the pool makes, unless the group is given another:

(defvar *pool*
  (th:make-job-pool :error-handler (lambda (condition)
                                     (log-error "Background job failed: ~A" condition))))

(th:make-job-group *pool*)                      ; uses the pool's handler
(th:make-job-group *pool* :error-handler #'my-handler) ; uses MY-HANDLER
(th:make-job-group *pool* :error-handler nil)   ; no handler: errors go to WAIT-FOR-JOBS

Use :error-handler nil for groups that someone does wait for and that should signal there.

parallel-for always signals. It always waits, so its errors always come to its caller as a job-error, whatever the pool's error handler.

Where the job was submitted	Error goes to
`parallel-for`	the caller of `parallel-for`, as a `job-error`
a group with an error handler	the handler, at once
a group without one	`wait-for-jobs`, as a `job-error`

`:debug`: errors open the debugger

(defvar *pool* (th:make-job-pool :on-error :debug))

With :on-error :debug, nothing catches a job's error before the debugger: it opens in the worker, with the job's stack, while any thread waiting for the job goes on waiting. Two restarts are available:

Restart	Does
`retry-job`	Runs the job again from the start, for example after you fixed and recompiled the function it calls.
`skip-job`	Skips the job. The jobs waited for with it still run, and the waiter carries on.

This is the mode for interactive development: fix the bug, choose retry-job, and the frame that was waiting carries on as if nothing happened.

retry-job and skip-job are also functions that invoke their restarts, as continue does its own, so code such as a handler inside the job can choose one:

(handler-bind ((error (lambda (condition)
                        (if (transient-p condition)
                            (th:retry-job condition)
                            (th:skip-job condition)))))
  ...)

Each takes an optional condition, to choose a restart for that condition, and returns nil if there is no such restart.

In :debug mode, error handlers and job-error are not used.

Errors cl-threads signals itself

These are threads-errors, signalled in the calling thread:

Situation	Message
Giving work to a destroyed pool	`#<JOB-POOL ...> has been destroyed: it runs no more jobs.`
Destroying a pool from one of its workers	`#<JOB-POOL ...> cannot be destroyed by one of its own workers, which would wait for itself.`
`worker-local-value` outside the pool's workers	`WORKER-LOCAL-VALUE was called outside the workers of #<JOB-POOL ...>, which the value belongs to.`
A thread joining itself	`#<THREAD ...> cannot join itself: it would wait forever. Join it from another thread.`

threads-error-message returns the message. Arguments of the wrong type signal a type-error.

cl-threads

Reference