dxtr/stift

Fork 0

Bundle resources alongside a built executable (:bundle) #1

New issue

Open

opened 2026-10-18 04:36:20 +02:00 by dxtr · 0 comments

dxtr commented

2026-10-18 04:36:20 +02:00

Owner

Problem

stift build produces an executable (via asdf:make → program-op), but a
runnable program usually needs non-Lisp files beside it: shared libraries the
binding layer loads (e.g. a Windows freetype.dll), fonts, data files,
shaders, licence texts — arbitrarily many things. Today stift copies none of
them, so producing a working, shippable artifact needs an out-of-band step.

stift already knows each build-list project's pinned, verify-able checkout,
so it is well placed to copy declared files out of those trees and lay them out
beside the build output — reproducibly, with provenance.

This should be a general "lay these files out beside the artifact"
mechanism, not a native-library feature. A bundled DLL is just the special case
whose destination is the executable's own directory.

Proposal: a `:bundle` key in the stiftfile

A project declares files it carries that should ship beside an executable built
from it — or from anything that requires it:

(:project "example.org/me/game"
 :require (…)
 :bundle (("native/windows-x64/freetype.dll" :when :windows)   ; → beside the exe
          ("assets/fonts/"                    :to "fonts/")      ; → fonts/…
          ("data/game.dat"                    :to "data/game.dat")
          ("third-party/zlib/COPYING"         :to "licences/zlib.txt")))

Each entry is (SOURCE &key to when):

SOURCE — a file or directory (copied recursively), relative to the
declaring project's root. Copied from the pinned tree (cache / vendor/ /
workspace member), preserving file modes.
:to — destination relative to the built executable's directory,
cp/install-like: trailing slash = copy into that directory keeping the
basename; no slash = exact path (rename); omitted = basename at the root (so a
bare DLL lands beside the exe). A directory source places its contents at the
destination.
:when — optional condition, tested against the build host, in cl-ffi's
clause vocabulary: :windows, :linux, :darwin, :unix, and (:or …),
(:and …), (:not …). Kept to OS/arch keywords so stift can evaluate it
itself without consulting the build Lisp.

The anchor is the directory of the primary build output, so this works for
program-op and image-op alike, and is orthogonal to --compress and
SBCL-vs-CCL.

Composition

:bundle composes up the build list like :require (and unlike
:replace/:exclude, which are main-project-only): a library that needs a
resource shipped declares it, and it propagates to any artifact built from
something that uses it. Entries are inert data; sources come only from pinned,
verify-able trees; stift copies plain files, no eval.

Collisions

When two projects target the same destination (two freetype.dlls at the root,
two fonts/Inter.ttf), stift stops and names both contributors, as it already
does for system collisions. No automatic per-project namespacing — it would
produce paths the program can't predict. A main-project override to resolve a
collision is a possible later refinement.

When it runs

stift build bundles by default — the point is a runnable artifact, and on
Windows the exe won't start without its DLLs beside it. --no-bundle produces
the bare binary. A separate stift bundle (assemble/refresh without rebuilding)
is a possible addition, not needed first.

Boundary: runtime discovery is the program's job

stift guarantees a known directory layout beside the artifact and nothing more.
Finding the files at runtime is the program's concern: for shared libraries it
is the OS loader plus a by-name fallback in the binding; for everything else the
program computes paths from its own executable location (uiop:argv0 / its
truename's directory). stift never learns what a bundled file is for.
Application-level policy — third-party notice assembly, resource lookup — layers
on top.

Open question: the rebuild record

stift build writes its outputs anew each time, so bundling must remove files a
previous build bundled but the current one does not (as stift vendor removes
copies of projects that left the list). That needs stift to remember what it
placed. The same record would give verify-style provenance and let a program
enumerate bundled resources. Placement is undecided:

a small manifest in the output directory — robust, serves runtime
enumeration, but ships a stift metadata file in the distributable;
a record in stift's cache keyed to the build — invisible, but fragile if the
output directory is moved.

Notes

Touches the stiftfile format and stift build's public behaviour.
Needs tests, a README section, and a wiki page.

## Problem `stift build` produces an executable (via `asdf:make` → `program-op`), but a runnable program usually needs non-Lisp files beside it: shared libraries the binding layer loads (e.g. a Windows `freetype.dll`), fonts, data files, shaders, licence texts — arbitrarily many things. Today stift copies none of them, so producing a working, shippable artifact needs an out-of-band step. stift already knows each build-list project's pinned, `verify`-able checkout, so it is well placed to copy declared files out of those trees and lay them out beside the build output — reproducibly, with provenance. This should be a **general** "lay these files out beside the artifact" mechanism, not a native-library feature. A bundled DLL is just the special case whose destination is the executable's own directory. ## Proposal: a `:bundle` key in the stiftfile A project declares files it carries that should ship beside an executable built from it — or from anything that requires it: ```lisp (:project "example.org/me/game" :require (…) :bundle (("native/windows-x64/freetype.dll" :when :windows) ; → beside the exe ("assets/fonts/" :to "fonts/") ; → fonts/… ("data/game.dat" :to "data/game.dat") ("third-party/zlib/COPYING" :to "licences/zlib.txt"))) ``` Each entry is `(SOURCE &key to when)`: - **SOURCE** — a file or directory (copied recursively), relative to the *declaring* project's root. Copied from the pinned tree (cache / `vendor/` / workspace member), preserving file modes. - **`:to`** — destination relative to the built executable's directory, `cp`/`install`-like: trailing slash = copy into that directory keeping the basename; no slash = exact path (rename); omitted = basename at the root (so a bare DLL lands beside the exe). A directory source places its contents at the destination. - **`:when`** — optional condition, tested against the build host, in cl-ffi's clause vocabulary: `:windows`, `:linux`, `:darwin`, `:unix`, and `(:or …)`, `(:and …)`, `(:not …)`. Kept to OS/arch keywords so stift can evaluate it itself without consulting the build Lisp. The anchor is the directory of the primary build output, so this works for `program-op` and `image-op` alike, and is orthogonal to `--compress` and SBCL-vs-CCL. ## Composition `:bundle` composes up the build list like `:require` (and unlike `:replace`/`:exclude`, which are main-project-only): a library that needs a resource shipped declares it, and it propagates to any artifact built from something that uses it. Entries are inert data; sources come only from pinned, `verify`-able trees; stift copies plain files, no eval. ## Collisions When two projects target the same destination (two `freetype.dll`s at the root, two `fonts/Inter.ttf`), stift stops and names both contributors, as it already does for system collisions. No automatic per-project namespacing — it would produce paths the program can't predict. A main-project override to resolve a collision is a possible later refinement. ## When it runs `stift build` bundles by default — the point is a runnable artifact, and on Windows the exe won't start without its DLLs beside it. `--no-bundle` produces the bare binary. A separate `stift bundle` (assemble/refresh without rebuilding) is a possible addition, not needed first. ## Boundary: runtime discovery is the program's job stift guarantees a known directory layout beside the artifact and nothing more. Finding the files at runtime is the program's concern: for shared libraries it is the OS loader plus a by-name fallback in the binding; for everything else the program computes paths from its own executable location (`uiop:argv0` / its truename's directory). stift never learns what a bundled file is *for*. Application-level policy — third-party notice assembly, resource lookup — layers on top. ## Open question: the rebuild record `stift build` writes its outputs anew each time, so bundling must remove files a previous build bundled but the current one does not (as `stift vendor` removes copies of projects that left the list). That needs stift to remember what it placed. The same record would give `verify`-style provenance and let a program enumerate bundled resources. Placement is undecided: - a small manifest in the output directory — robust, serves runtime enumeration, but ships a stift metadata file in the distributable; - a record in stift's cache keyed to the build — invisible, but fragile if the output directory is moved. ## Notes - Touches the stiftfile format and `stift build`'s public behaviour. - Needs tests, a README section, and a wiki page.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

dxtr/stift#1

No description provided.

Rows
Columns