Generator and Registry

cl-vk's bindings are generated from the Vulkan API registry, vk.xml, by a generator of its own, the system cl-vk/generator. The generated code is committed, so using cl-vk needs neither the generator nor any XML. This page is for working on cl-vk itself: updating the registry, changing what is generated, or reading the registry from Lisp.

The registry files

registry/ holds the files the bindings are generated from, kept in the repository so that regenerating gives the same output on every machine:

• vk.xml: the Vulkan API registry, from Vulkan-Headers v1.4.365.
• video.xml: the video codec definitions vk.xml refers to, from the same release.

Both are under Apache-2.0 OR MIT, as stated in each file.

The reader's output for this vk.xml has been checked against the C headers of the same release, vulkan_core.h and vulkan_beta.h: every enum value and constant matches, and every struct member, command and function pointer has exactly the type GCC sees in the header.

What is generated

File	Contents
registry/vulkan.lisp	The raw layer, the package cl-vk/ffi.
registry/structs.lisp	The package cl-vk/structs, a description of every struct's fields, a keyword for every enum value, and for each enum the type of its keywords. Read by the idiomatic layer's struct macros.
tests/layout.c	A C program that prints the size, alignment and member offsets of every struct.

The bindings cover every API version and every extension that is not specific to a platform, plus the extensions of the platforms the engine targets: "wayland", "xcb", "xlib", "xlib_xrandr" and "win32". Provisional extensions are left out. Starting from what those require, the selection follows every type, command and constant a definition refers to, across vk.xml and video.xml. Definitions nothing requires, such as those only Vulkan SC uses, are left out.

Regenerating

After changing the generator or updating the registry files:

(asdf:load-system :cl-vk/generator)
(cl-vk/generator:regenerate-bindings)
;; => (#P".../registry/vulkan.lisp" #P".../registry/structs.lisp" #P".../tests/layout.c")

Tests check that each committed file matches what the generator produces, so forgetting to regenerate fails the suite. The output is laid out by hand rather than by the pretty printer, whose layout can change between SBCL releases, so every release produces the same bytes.

bindings-text, struct-descriptions-text and layout-program-text return each file's text without writing it. generate-bindings, generate-struct-descriptions and generate-layout-program write to a stream, from registries you read yourself: a list of the registry of vk.xml and the registry of video.xml, in that order.

Updating the registry

1. Copy registry/vk.xml and registry/video.xml from the new Vulkan-Headers release.

2. Regenerate, as above.

3. Compile the layout program against the C headers of the same release, and save its output:

   cc -std=c11 -I Vulkan-Headers/include -o layout tests/layout.c
   ./layout > tests/layout.sexp
   

4. Run the test suite. The layout test compares every struct's cl-ffi layout with tests/layout.sexp, and the generator tests compare the generated files with the committed ones.

Reading the registry from Lisp

(asdf:load-system :cl-vk/generator)
(use-package :cl-vk/generator)

(let* ((registry (read-registry
                  (asdf:system-relative-pathname "cl-vk" "registry/vk.xml")))
       (application-info (find "VkApplicationInfo" (registry-types registry)
                               :key #'type-definition-name :test #'string=)))
  (values (mapcar #'feature-name
                  (remove-if #'feature-internal-p (registry-features registry)))
          (mapcar #'struct-member-name (struct-type-members application-info))))
;; => ("VK_VERSION_1_0" "VK_VERSION_1_1" "VK_VERSION_1_2" "VK_VERSION_1_3"
;;     "VK_VERSION_1_4")
;;    ("sType" "pNext" "pApplicationName" "applicationVersion" "pEngineName"
;;     "engineVersion" "apiVersion")

read-registry reads a file; parse-registry reads a string. Both return a registry, and signal a registry-error that says where a problem is:

registry/vk.xml: type VkS: expected an array length for a, found "]"

registry-error-source, registry-error-context (outermost first, such as ("type VkS")) and registry-error-message take it apart.

The registry model

The generator works in two stages. The reader (generator/reader.lisp) parses the XML into a model of plain, read-only structs, keeping what belongs to Vulkan and leaving out what is marked for Vulkan SC. The generator (generator/selection.lisp, generator/generator.lisp, generator/structs.lisp) reads only the model.

Names in the model are the registry's own ("VkInstance", "vkCreateInstance", "VK_SUCCESS"); turning them into Lisp names is the generator's job. The model is immutable and may be shared between threads.

Struct	Holds
registry	registry-platforms, registry-tags (author tags such as "KHR"), registry-types, registry-constants, registry-enums, registry-commands, registry-features, registry-extensions
c-type	A parsed C type: c-type-name, c-type-const-p, c-type-pointers (one entry per pointer level, each saying whether it is const)
c-declaration	A parsed declaration: c-declaration-name, -type, -dimensions, -bit-width. struct-member and param include it, with optional, len and the like
type-definition	The base of the type structs: external-type, header, define, basetype, bitmask, handle, enum-type, funcpointer, struct-type (unions too, with struct-type-union-p) and type-alias
enumeration	An enum type's values, enumeration-entries, each an enum-value. Values that API versions and extensions add are merged in, offsets computed into integers, and each records its origin
constant	A constant with a Lisp value: (~0ULL) becomes #xFFFFFFFFFFFFFFFF, 1000.0F becomes 1000.0f0, an extension name becomes a string
command, command-alias	A command's result, parameters, success and error codes
feature, extension, requirement	API versions and extensions, with what they require. Dependency expressions are parsed into lists such as (:or "VK_VERSION_1_1" "VK_KHR_get_physical_device_properties2")
platform	A platform and the C macro that guards it

The model's deftypes: type-category, bit-width (32 or 64), extension-kind (:instance or :device), dependency, constant-value and dimension.

Every struct's readers are exported from cl-vk/generator, and each struct's docstring describes its slots.

A model built by hand needs what every registry has, each part of the type its slot declares: a part of another type signals a type-error on entry. A constructor called without one of those parts signals a registry-error that names the keyword to give:

MAKE-COMMAND needs :RESULT: every command of a registry has one.

Replacing the XML library

The reader is the only part of the generator that uses an XML library, xmls, so swapping it means rewriting one file. C syntax (declarations, literals and dependency expressions) is parsed in generator/c-syntax.lisp, which knows nothing about XML. The tests in tests/reader-test.lisp and tests/c-syntax-test.lisp define what any replacement must produce.