stift

• Installing
• The stiftfile
• Choosing commits
• Workspaces
• Commands
• Starting Lisps
• Running tests
• Building programs
• Checking what systems need
• The cache
• Vendoring
• From Lisp
• Tests
• Licence

stift is a package manager for Common Lisp, after Go's modules. A project names the git repositories it requires in its stiftfile, each pinned to a commit. stift fetches them into a cache of its own, sets ASDF up to find exactly them, and runs Lisps and test suites with them. There is no registry and no mirror: every project comes from its own repository. stift needs only ASDF, UIOP and git.

A project in the directory hello, a git repository:

;;; hello.asd
(defsystem "hello"
  :depends-on ("alexandria")
  :components ((:file "hello"))
  :perform (test-op (op c)
             (unless (equal '(1 2 3) (uiop:symbol-call :hello :numbers))
               (error "hello's tests failed."))))

;;; hello.lisp
(defpackage #:hello (:use #:cl) (:export #:numbers))
(in-package #:hello)
(defun numbers () (alexandria:flatten '(1 (2 (3)))))

cd hello
stift init example.org/me/hello
stift get https://gitlab.common-lisp.net/alexandria/alexandria.git
stift exec sbcl --non-interactive --eval '(asdf:load-system "hello")' \
                --eval '(print (hello:numbers))'
stift test --lisp sbcl --lisp ccl

stift init names the project by its repository's host and path, and stift get records the commit that alexandria's default branch points to, in the stiftfile:

;;; -*- mode: lisp-data -*-
(:project "example.org/me/hello"
 :require (("https://gitlab.common-lisp.net/alexandria/alexandria.git"
            :branch "master"
            :commit "…")))

stift exec starts SBCL with ASDF set up to find hello and that commit of alexandria, and nothing else; stift test runs hello's test suite in a new SBCL and a new CCL, and says how each went:

ok    hello  sbcl  0.4s
ok    hello  ccl   1.3s

Installing

stift is a Lisp program built into an executable with SBCL. It needs git 2.29 or later, Git for Windows on Windows, and nothing else.

Each release, on stift's releases page, carries the executable for Linux on x86-64, built with the SBCL release that stift's suite passed on just before. It needs glibc 2.38 or later, as Debian 13, Ubuntu 24.04 and Fedora 39 have. stift-VERSION-linux-x86_64.tar.gz holds stift, LICENSE and README.org in a directory of the same name, and stift-VERSION-linux-x86_64.tar.gz.sha256 holds its SHA-256 checksum, which sha256sum -c checks in the directory of both. Unpacked, stift goes into a directory on the PATH, such as ~/.local/bin.

Anywhere else, as on Windows, one command builds stift from a clone and installs it, in sh, cmd.exe or PowerShell alike:

git clone https://fakehacker.cc/dxtr/stift.git
cd stift
sbcl --script install.lisp

install.lisp builds bin/stift, or bin\stift.exe on Windows, and copies it into the directory XDG_BIN_HOME names, or else .local/bin in the home directory, or into the directory named after it, as in sbcl --script install.lisp /usr/local/bin. It says so when that directory is not on the PATH. Once stift is installed, stift build in the clone rebuilds bin/stift.

The executable is built with SBCL even for running CCL, because CCL's run-program starts every program in a process group of its own, which leaves an interactive Lisp in the background of the terminal. The Lisps stift runs may be SBCL or CCL.

The stiftfile

A project's stiftfile sits at its root. It is Lisp data, which stift reads with a reader of its own: lists, strings, the keys below, integers, t, nil and comments, and nothing else. It never evaluates or interns anything, so reading another repository's stiftfile is safe.

It holds one property list:

• :project NAME comes first and names the project: the host and path of its repository, as below.

• :require (ENTRY ...) holds an entry (URL :commit ID ...) for each project required: the project at URL, at the commit ID, a full commit ID, or any commit that contains it. An entry's keys:

  • :branch NAME or :tag NAME, the ref the commit was read from, which update follows;
  • :test t, on a project only this project's own tests need;
  • :indirect t, on the entries tidy writes: projects that only the projects this one requires need;
  • :exclude ("DIR/" ...), directories of that project, relative to its root, whose systems stift leaves out, such as a copy of another library inside it.
• :replace (ENTRY ...) holds an entry (URL ...) for each project whose files it chooses, whatever is required of it: :with URL2 :commit ID takes a commit of another repository, such as a fork; :commit ID alone takes that commit; :local DIR takes the files of a directory, relative to the project's. :branch or :tag beside :commit is the ref for update to follow.

:replace, :exclude and requirements marked :test count only in the project being built, the main project: a library cannot rearrange the builds of the projects that use it, and nobody who uses it needs its test framework.

stift's commands keep the direct requirements first and the indirect ones after them, each sorted by name, and the replacements sorted by name. They write the file entry by entry: an entry they do not change keeps its text, the comments above an entry and after it on its line stay with it, and what they write is laid out the same by SBCL and CCL.

A project's stiftfile in a repository it requires counts too. Of pyssel's, which requires cl-gfx and cl-windowing, both of which require cl-ffi, stift tidy makes:

;;; -*- mode: lisp-data -*-
(:project "fakehacker.cc/dxtr/pyssel"
 :require (("https://fakehacker.cc/dxtr/cl-gfx.git" :branch "main"
            :commit "5e03cf5ae3ff322ec143280a69680ed4b13ce233")
           ("https://fakehacker.cc/dxtr/cl-windowing.git" :branch "main"
            :commit "e96c076a03a83f9cf9646d78918b6a8697f62f82")
           ("https://github.com/Shinmera/parachute.git" :branch "master"
            :commit "9a6679e611925dfb59067393c5b7996f69501aa6" :test t)
           ("https://fakehacker.cc/dxtr/cl-ffi.git" :branch "main"
            :commit "b7248691986fdeb2e7b4c0bcb7463ea3375089bb"
            :indirect t)))

A project's name is the host and path of its repository, such as fakehacker.cc/dxtr/cl-ffi, or the path of a repository on this machine. URLs with the same host and path name the same project: https://fakehacker.cc/dxtr/cl-ffi.git and git@fakehacker.cc:dxtr/cl-ffi are both fakehacker.cc/dxtr/cl-ffi. A name without the URL's scheme and .git keeps no more than it needs, and unlike a short name, such as cl-ffi, it cannot be taken by another project. stift fetches from https://, ssh://, USER@HOST:PATH and file:///PATH URLs. It refuses http:// and git://, over which a branch could be forged on the way.

Choosing commits

stift chooses as Go chooses versions, by minimal version selection, with git's history in place of version numbers. From the main project, it reads the stiftfile of each project required, at each commit required of it, and follows the requirements there; then it chooses, for each project, the commit required of it that contains all the others. No newer commit comes in unless something requires it, and two machines with the same stiftfiles choose the same commits.

When no commit required of a project contains the others, the requirements have diverged, across a fork or a rewritten branch. stift stops and names each requirement and who made it. The main project settles it by requiring a commit that contains them all, such as a merge, or by choosing one with a replacement (URL :commit ID).

The main project's stiftfile holds the whole build list: stift tidy writes each project it does not require itself as an :indirect entry, at the commit chosen, marks :test those only its test requirements bring in, removes the indirect entries nothing needs any more, and moves any requirement up to the commit chosen for its project. get, remove and update tidy too; the entries without :indirect are added and removed only by get and remove.

Before it builds, stift checks that the stiftfile holds the build list its requirements make: that no commit it lists requires a project it does not list, or a commit its commit for that project does not contain. Otherwise it stops and asks for stift tidy, which it never runs unasked. It also stops when two projects of the build list provide one system, which ASDF would choose between without a word, and when a project's stiftfile names another project than the one required: a fork or a mirror, which a :replace puts in the original's place.

Workspaces

A workspace builds checkouts of several projects together, while they are changed together. Its file, stiftwork, names the members' directories:

;;; -*- mode: lisp-data -*-
(:use ("cl-ffi" "cl-ft2" "cl-harfbuzz" "pyssel"))

cd ~/code/lisp/engine
stift work init cl-ffi cl-ft2 cl-harfbuzz pyssel

stift finds the stiftwork in the directory a command runs in or the nearest one above it. In a workspace, every member is a main project: its requirements, test requirements included, are chosen from together, and its checkout stands in for it, uncommitted changes and all, whatever commit other members require of it. The members' stiftfiles need not be tidy. stift exec, test and list in any member, or at the workspace's root, work with the workspace's build list; at the root, stift test tests every member. A project inside the workspace's directory that is not a member is an error. A stiftwork may also hold a :replace list, as a stiftfile does, whose entries win over the members'.

get, update, remove and tidy ignore the workspace: they change one project's stiftfile from pins alone, so that what is committed builds without the workspace, as CI builds it. --no-workspace leaves the workspace out for one command, and STIFT_WORKSPACE names another stiftwork, or off for none.

stift work sync moves the members' requirements on each other to the commits their checkouts hold, once those commits are pushed, and tidies each stiftfile it changes. It refuses a member whose commit is on no branch or tag of its remote, or does not contain the commit required now, and then changes nothing. Committing and pushing stay with you: commit and push the stiftfiles it names, in the order it gives, and run it again until nothing is left.

A stiftwork is not meant to be committed.

Commands

• stift init [NAME] writes a stiftfile for the project in the current directory, named NAME: the host and path of its repository, or a URL of it, by default that of the git remote origin.
• stift get [--test] URL[@REF] requires a project at the commit REF names: a branch or tag name, refs/heads/NAME or refs/tags/NAME, or a full or abbreviated commit ID. Without REF it takes the repository's default branch. A requirement of the same project is replaced, and the stiftfile is tidied.
• stift remove URL stops requiring a project, and tidies the stiftfile: a project the build list still needs stays, as an indirect entry.
• stift tidy brings the stiftfile's build list up to date (see Choosing commits), and prints what it changed.
• stift update [URL ...] moves requirements along their refs, or only those of the projects named, then tidies. A :branch entry goes to the branch's newest commit; a :tag entry to the highest tag of the same major version, among the tags that read as versions, such as v1.2.3 and 1.2.3, whose commit contains its own; an entry with neither stays. It prints the range each moved over, with the git log command that shows it. A tag that points elsewhere than recorded, or a branch that no longer contains its commit, stops it.
• stift list prints the build list: the project, and each project of its build with its commit and ref.
• stift why PROJECT-OR-SYSTEM prints the chains of requirements that bring a project, or the project that provides a system, into the build list.
• stift graph prints every requirement met, a line each: the project and commit that requires, and the project and commit required.
• stift download fetches the build list into the cache, as for working offline or for a CI cache.
• stift vendor copies the build list into the project's vendor/, which stift then builds from (see Vendoring).
• stift verify [--remote] checks that the copies in vendor/, or the cache's checkouts, hold their commits' files, and with --remote that the branches and tags recorded still hold their commits (see Vendoring).
• stift exec PROGRAM [ARG ...] runs a program with the build list (see Starting Lisps) and exits with its status.
• stift test [--lisp LISP] [SYSTEM ...] runs test suites (see Running tests).
• stift build [--lisp LISP] [--compress] [SYSTEM] builds a system as asdf:make does, an executable for one, compressed with --compress where the Lisp can (see Building programs).
• stift check [--lisp LISP] loads the .asd files in a Lisp and reports the systems they need that the build list lacks (see Checking what systems need).
• stift work init [DIR ...] writes a stiftwork in the current directory, whose members are the projects in DIR ...; stift work use DIR ... and stift work drop DIR ... add and remove members; stift work sync moves the members' pins on each other (see Workspaces).
• stift setup prints the path of the build list's setup file.
• stift clean [--fasls] empties the cache, or only its fasls.

Before the command come the options -C DIR, to work as if started in DIR; --offline, to use only what the cache holds; --no-workspace, to leave the workspace out; and -v, to print each git command before it runs. stift finds the project's stiftfile in the current directory or the nearest one above it. It exits with 0 on success, 1 after an error, whose message says what to do, and 2 when it does not understand its command line.

Starting Lisps

stift exec sbcl and stift exec ccl start the Lisp without its init files and load the build list's setup file before anything else, after SBCL's runtime options. Loading it sets ASDF's source registry to the build list's directories and nothing else, and puts their fasls in a directory of the build list's own. Init files are skipped because Quicklisp, loaded from one, adds a search of its own to ASDF, and a system missing from the build list would then come from Quicklisp without a word.

The setup file signals an error when something else would find systems, as asdf:*central-registry* or Quicklisp's search would, with a continue restart that removes them; and when a system of the build list is loaded already from another directory, since loaded code cannot be swapped.

Any other program gets the build list in the environment variables ASDF reads, CL_SOURCE_REGISTRY and ASDF_OUTPUT_TRANSLATIONS, so that a Lisp it starts finds it, and the setup file's path in STIFT_SETUP:

stift exec make test

The terminal's Ctrl-C goes to the program, and stift waits on. In Emacs, SLY starts its Lisp through stift when its program is ("stift" "exec" "sbcl"), in sly-lisp-implementations. A Lisp that runs already loads the setup file itself:

(load (uiop:run-program '("stift" "setup") :output '(:string :stripped t)))

Running tests

stift test runs each system's test suite with asdf:test-system, in a new Lisp for each implementation given with --lisp, SBCL by default. With no system named, it tests the project's primary systems, those its .asd files are named after; at a workspace's root, those of every member.

ASDF defines no result for test-op. As its manual suggests, a suite fails when an error escapes asdf:test-system: its test-op must signal when its tests fail, as this one does with Parachute.

:perform (test-op (op c)
           (let ((report (symbol-call :parachute :test :hello/test
                                      :report (find-symbol "SUMMARY" :parachute))))
             (unless (eq :passed (symbol-call :parachute :status report))
               (error "hello's tests failed."))))

A system that does not load fails too. stift prints a line for each suite at the end, and exits with 1 when any failed. Ctrl-C stops the suite that runs, and the rest do not run.

Building programs

stift build builds a system as asdf:make does, in a new Lisp with the build list, SBCL unless --lisp names CCL. asdf:make performs the build operation the system's defsystem names: program-op writes an executable to :build-pathname, which starts in the function :entry-point names. With these lines in hello's defsystem, and a function hello:main:

  :build-operation "program-op"
  :build-pathname "bin/hello"
  :entry-point "hello:main"

$ stift build
Built hello with sbcl: /home/me/hello/bin/hello

With no system named, it builds the one system of the project's .asd files that names a build operation, and says what to do when none does, or several do. The executable holds all it loaded, so it runs without stift and without the cache. The Lisp's own output, the compiler's included, goes to the terminal. The build writes its files anew each time, even when an earlier build's look up to date.

--compress compresses the executable, or the image image-op writes, on an SBCL built with core compression, as Debian's is. It takes less space on disk and more in memory, and starts later: the whole image is decompressed into memory when it starts, where an uncompressed one is read in as it is used. For a small program, 11 MB against 37 MB on disk, 45 MB against 20 MB of memory, and 33 ms against 3 ms to start. A Lisp without core compression, such as the SBCL releases of sbcl.org or CCL, builds the executable uncompressed, and stift says so:

$ stift build --compress
stift: SBCL 2.6.9 was built without core compression, so hello was built uncompressed.  An SBCL built with it, as Debian's is, compresses.
Built hello with sbcl: /home/me/hello/bin/hello

Checking what systems need

The stiftfiles say which projects a project requires, not which systems its systems need: that is in its .asd files, which are code. stift check starts a Lisp, SBCL unless --lisp says otherwise, with the build list set up, loads every .asd file of the main projects, and follows ASDF's depends-on and defsystem-depends-on from each system they define, test systems included, and from each system those need:

$ stift check
The system "trivial-indent", which "documentation-utils" depends on, is in no project of the build list.
stift cannot know where a system lives: add the repository that provides each that is missing with stift get URL.

It reports systems no project of the build list provides, with what needs them; systems of a main project that use a project its stiftfile requires only as an :indirect entry, which tidy may drop, or, in a workspace, not at all; requirements of a main project that none of its systems needs on this platform; and .asd files that cannot be loaded. Dependencies under (:feature ...) are followed whatever the platform, as ASDF keeps them as written, while #+ in an .asd file is settled as it is read, so CI runs check on each platform. Package-inferred systems are followed as ASDF infers them. check changes nothing, and exits with 1 when a build would fail: a system missing, a project a stiftfile lacks, or an .asd file that cannot be loaded.

The cache

The cache is ~/.cache/stift/ on Linux; STIFT_CACHE moves it. It holds a bare repository for each project, each commit checked out once in src/NAME@COMMIT/, a setup file for each build list, and fasls in a directory for each build list and implementation. Sources are shared by every build list that has their commit, and fasls only by identical build lists, so a fasl is never loaded against other commits than it was compiled with.

Any number of processes may use the cache at once. stift never collects its repositories' garbage, so a commit a force-push removed stays for the pins that name it. --offline or STIFT_OFFLINE forbids the network: a commit the cache lacks is then an error. STIFT_GIT names the git program, and STIFT_WORKSPACE the workspace (see Workspaces).

Vendoring

stift vendor copies the build list into the project, so that a clone of it builds without the network and without the repositories it requires, should one of them disappear or a force-push leave a pinned commit on no branch:

stift vendor
git add vendor
git commit -m "Vendor the build list"

Each project of the build list goes into vendor/HOST/PATH/: the files of its commit, as git checks them out, modes and all. vendor/projects records each one's URL, commit, tree and directory, and vendor/.gitattributes and vendor/.gitignore keep the copies whole in a commit: their line endings as they are, whatever core.autocrlf says, and every file in, whatever the project's own .gitignore says. A project that a :replace takes from a directory stays where it is. stift vendor copies the build list from the stiftfile's pins, whatever workspace the project is in. Run it again whenever the stiftfile changes: it copies the new build list, and removes the copies of projects that left it.

While vendor/projects exists and no workspace is in use, stift builds from the copies: exec, test, build, check, setup and list need neither the network nor the cache's repositories, while why and graph still read the requirements' stiftfiles from the cache. When the stiftfile's requirements or replacements have changed since stift vendor ran, stift stops with an error that asks for stift vendor. In a workspace, the members' checkouts and the cache count, not vendor/. The vendor/ of a project with a stiftfile is stift's: its systems are never the project's own.

The copies belong to stift. Code you change goes elsewhere in the tree, named by a replacement (URL :local "third-party/foo/"), or lives in a repository of its own, named with :with. stift verify checks that each copy holds its commit's files and nothing else:

$ stift verify
The copy of example.org/me/lib, /home/me/hello/vendor/example.org/me/lib/, is not the files of commit 1a2b3c4d5e6f:
  modified  lib.lisp
  added     notes.txt
The copies in vendor/ belong to stift: run stift vendor to copy the commit again, and put changes of your own in a replacement (URL :local DIR) instead.

It compares each copy with the tree recorded for it, which it finds in the cache, in the project's own repository once vendor/ is committed as stift made it, or else by fetching the commit. File modes count, except on Windows, where files have no executable bit and a copy keeps the modes of its commit. Without vendor/projects, or in a workspace, verify checks the cache's checkouts of the build list instead. stift verify --remote also checks over the network that each tag recorded still points to its commit, and each branch recorded still contains its commit. verify changes nothing, and exits with 1 when it finds anything.

From Lisp

Each command is a function of the stift package, which takes the project's directory as :directory and a settings object, made by make-settings, as :settings: initialize-project, add-requirement, remove-requirement, tidy-manifest, update-requirements, build-list, requirement-chains, requirement-graph, download, setup-file, exec-program, run-tests, build-system, check-systems, vendor-projects, verify-projects, initialize-workspace, add-workspace-members, remove-workspace-members, sync-workspace and clean-cache. stift/cli:run runs a command line and returns its exit status. make-settings takes :workspace: :search, the default, :off, or a stiftwork's pathname.

(stift:add-requirement "https://gitlab.common-lisp.net/alexandria/alexandria.git"
                       :ref "master" :directory "hello/")
(mapcar #'stift:entry-identity (stift:build-list :directory "hello/"))
;; => ("example.org/me/hello" "gitlab.common-lisp.net/alexandria/alexandria")

Errors are conditions under stift-error, whose message says what went wrong and what to do: manifest-error, with the stiftfile's line and column; url-error; git-error; ref-error; missing-commit; no-manifest; diverged-requirements; inconsistent-manifest; system-collision; name-mismatch; moved-tag; rewritten-branch; workspace-error, with the stiftwork's line and column when it is about its text; and vendor-mismatch, when vendor/projects does not match the stiftfile or cannot be read.

Settings, manifests, requirements, replacements, workspaces, build lists, edges, changes, findings, discrepancies and test results never change and may be shared between threads. Any number of threads may use the cache at once; functions that write a project's stiftfile or vendor/, or a workspace's stiftwork, must not run at the same time for it.

Tests

stift's own suite needs git, and sbcl and ccl on the PATH, but no network: every repository it uses, it makes. It runs on Linux and on Windows, where CCL's wx86cl64.exe answers to ccl when copied to ccl.exe beside it, with its image as ccl.image. stift runs it:

bin/stift test --lisp sbcl --lisp ccl

Licence

MIT; see LICENSE.