cl-vk

• Status
• Usage
• The raw layer
• Structs
• The generator
• Regenerating
• Registry files
• Development
• License

Common Lisp bindings for Vulkan, generated from the Vulkan API registry. Vulkan 1.4 is the minimum version: the bindings are generated from the latest registry, request 1.4 at run time and detect anything newer.

cl-vk is made of these systems:

• cl-vk: the Lisp-idiomatic API. It returns Lisp values, turns VkResult error codes into conditions, fills structs by field name with sType set automatically, and dispatches device commands through the device's own function pointers.
• cl-vk/ffi: the raw layer underneath, on cl-ffi, a direct transcription of the Vulkan C API, generated from the registry. It does no error checking beyond argument types, and no memory management.
• cl-vk/generator: a build-time tool that reads the registry (vk.xml) and generates the bindings. The generated code is committed, so loading cl-vk needs neither the generator nor any XML.

Only the Vulkan loader shared library is needed at run time (libvulkan.so.1, vulkan-1.dll). No C compiler or Vulkan headers are needed to build.

Status

The raw layer is complete: every type, constant and command of Vulkan 1.0 to 1.4 and of every extension, except those for platforms the engine does not target (Android, Metal and others) and provisional ones. The idiomatic layer covers structs, instances, physical devices, devices and their queues, surfaces, swapchains, fences and semaphores.

Usage

(asdf:load-system :cl-vk)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:vk #:cl-vk) (#:vks #:cl-vk/structs)))
(in-package #:example)

(vk:with-instance (instance :extensions '("VK_KHR_surface" "VK_EXT_headless_surface"))
  (let ((gpu (first (vk:physical-devices instance)))
        (surface (vk:create-headless-surface instance)))
    (unwind-protect
         (vk:with-device (device gpu :extensions '("VK_KHR_swapchain")
                                     :features '((vks:physical-device-vulkan13-features
                                                  :dynamic-rendering :synchronization2)))
           (let ((swapchain (vk:create-swapchain device surface :extent '(800 600))))
             (prog1 (list (vk:physical-device-type gpu)
                          (vk:swapchain-format swapchain)
                          (vk:swapchain-extent swapchain))
               (vk:destroy-swapchain swapchain))))
      (vk:destroy-surface surface))))
;; => (:CPU :B8G8R8A8-SRGB (800 600))   ; on Mesa's lavapipe

Instances. create-instance takes the Vulkan version (1.4 by default), :extensions and :layers. It checks them against what is installed before creating anything, and signals an unsupported-error that names what is missing. With :validation t it also enables the Khronos validation layer. Its messages are queued and returned by take-debug-messages, oldest first. :validation-checks adds checks the layer leaves out by default, and turns validation on: '(:synchronization) checks the barriers between commands, shader accesses through descriptors included, and '(:best-practices) uses that are valid but slow or fragile.

Physical devices and devices. physical-devices describes each GPU: name, type, API version, queue families and extensions. physical-device-feature-p asks about one feature, such as (vks:physical-device-vulkan13-features :dynamic-rendering). create-device takes the queues to make as (family count) (by default one graphics queue), :extensions, and :features as lists of a feature struct and its fields. Unsupported requests are reported before anything is created. device-queue returns a queue: by default the first one, or the one :family and :index name.

Surfaces and swapchains. A display backend creates a VkSurfaceKHR for its window and hands it to adopt-surface; create-headless-surface makes one that presents nowhere, for tests. surface-capabilities, surface-formats and surface-present-modes ask what a surface supports. create-swapchain chooses sensible defaults: an sRGB format, :fifo, and one image more than the minimum. Each frame:

• acquire-next-image signals a semaphore or fence when the image is ready. It returns the image's index and a status: :success, :suboptimal, :out-of-date, :timeout or :not-ready.
• present waits for a semaphore and returns :success, :suboptimal or :out-of-date.

On :out-of-date or :suboptimal, call recreate-swapchain. Neither function allocates.

Errors. A failing Vulkan call signals a result-error, which names the command and the VkResult and says what it means. Its subtypes are out-of-date-error, surface-lost-error and device-lost-error. check does the same for results of the raw commands. Every error is a vk-error, and misuse that no other condition names, such as asking a device for a queue it was not created with, is one whose message says what to write instead. An argument of the wrong type, such as a queue request that is not (family count), is a type-error. Every function declares its arguments' types, so SBCL refuses a wrong one on entry, and warns when it compiles a call with a constant of the wrong type.

Lifetimes. Each object has a destroy- function, and the main ones a with- macro. Destroying an object while objects made from it are alive, such as an instance with a live device, signals live-children-error. Using a destroyed object signals destroyed-object-error.

Threads. An instance or device may be used from several threads, but creating and destroying the objects made from it must happen on one thread at a time. A queue and a swapchain belong to one thread at a time: the render thread. Each object's documentation states its rules.

The raw layer. raw-handle returns the Vulkan handle of any object. instance-commands and device-commands return the command tables. With them, any command of cl-vk/ffi can be called on cl-vk's objects. swapchain-images returns a swapchain's images as raw VkImage handles, since cl-vk does not wrap images: code that draws into them, such as a GPU backend, makes their views through cl-vk/ffi. These escape hatches signal destroyed-object-error for a destroyed object.

The raw layer

(asdf:load-system :cl-vk/ffi)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:vk-ffi #:cl-vk/ffi)))
(in-package #:example)

(let ((global (vk-ffi:load-global-commands)))
  (cl-ffi:with-foreign-object (version :uint32)
    (vk-ffi:vk-enumerate-instance-version global version)
    (let ((version (cl-ffi:mem-ref version :uint32)))
      (list (vk-ffi:vk-api-version-major version)
            (vk-ffi:vk-api-version-minor version)))))
;; => (1 4)

cl-vk/ffi is generated from the registry. Names are the C names spelled the Lisp way:

C	Lisp
vkCreateInstance	function vk-create-instance
VkApplicationInfo	struct vk-application-info, as (:struct vk-application-info)
pNext, a member	slot :p-next
VK_SUCCESS, VK_WHOLE_SIZE	constants +vk-success+, +vk-whole-size+
VkResult, VkQueueFlags	foreign types vk-result, vk-queue-flags
PFN_vkAllocationFunction	foreign type pfn-vk-allocation-function
VK_MAKE_API_VERSION	function vk-make-api-version

The package uses no other package, so that it can define names such as type and format of its own. Refer to it through a package-local nickname, as above. Slots are keywords, so bind them with explicit pairs: (cl-ffi:with-foreign-slots ((s-type :s-type) (p-next :p-next)) info (:struct vk-ffi:vk-application-info) ...).

Commands are called through a command table, passed first. A table holds the function pointers of one level of the API:

• load-global-commands: vkCreateInstance and the vkEnumerateInstance* commands.
• load-instance-commands takes a VkInstance: its commands and those of its physical devices.
• load-device-commands takes the instance and a VkDevice: the commands of the device, its queues and command buffers. The pointers come from vkGetDeviceProcAddr, so calls go straight to the driver.

A command missing from the table, because the instance or device was created without the API version or extension that provides it, signals command-unavailable instead of crashing. command-available-p asks first. Each command checks the types of its arguments, and a call allocates nothing. Values are passed as integers and pointers; nothing is translated.

Structs are cl-ffi structs whose layouts were checked against C (see below). (structure-type 'vk-application-info) returns the value its sType must hold. cl-ffi has no bit-fields, so each storage unit of C bit-fields is an integer slot, :bit-field-0 and so on. bit-field and (setf bit-field) read and write the fields by name.

Structs

Filling Vulkan structs takes most of the code of calling Vulkan. cl-vk fills them by field name, from Lisp values:

(asdf:load-system :cl-vk)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:vk #:cl-vk) (#:vks #:cl-vk/structs) (#:vk-ffi #:cl-vk/ffi)))
(in-package #:example)

(vk:with-structs ((app vks:application-info
                       :application-name "example"
                       :api-version vk-ffi:+vk-api-version-1-3+)
                  (info vks:instance-create-info
                        :application-info app
                        :enabled-extension-names '("VK_KHR_surface")))
  (list (vk:struct-ref info vks:instance-create-info :enabled-extension-count)
        (vk:struct-ref app vks:application-info :application-name)))
;; => (1 "example")

with-structs binds each variable to a struct on the stack, zeroed and with its sType set, and those of the structs embedded in it, such as VkComputePipelineCreateInfo's stage, for the extent of its body. Structs are named by the symbols of cl-vk/structs: VkInstanceCreateInfo is vks:instance-create-info. Fields are keywords named after the members, without the p or pp of pointers: pApplicationName is :application-name.

Field	Value
integer, float	the number
VkBool32	a generalized boolean
enum	a keyword (:b8g8r8a8-srgb, :fifo) or an integer
flags	a keyword, a list of keywords ((:color-attachment :transfer-dst)) or an integer
handle	a raw handle, anything raw-handle accepts, or nil for none
string	a string, a foreign pointer, or nil
array	a list or vector; its count field is set from the length
struct by value	a field list, :image-extent (:width 800 :height 600)
pointer to a struct	a struct bound earlier, or nil
:next	a struct or a list of structs, chained in order

Enum keywords drop the type's prefix, its own vendor tag and _BIT: VK_PRESENT_MODE_FIFO_KHR is :fifo and VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT is :color-attachment. Each enum's keywords are a type of cl-vk/structs, named as the structs are: (typep :fifo 'vks:present-mode-khr) is true. A name that would be a symbol of Common Lisp keeps its prefix, so the type of VkFormat is vks:vk-format. Arrays that share a count, such as :wait-semaphores and :wait-dst-stage-mask of vks:submit-info, must have the same length.

A binding with several field lists makes an array of structs, and (variable (name count)) an array of count zeroed ones; either can be given to an array field of a later binding, which then knows its length. struct-aref points at one element.

For structs reused every frame, allocate-struct and free-struct manage the memory, and set-struct fills fields without allocating anything: strings and arrays are then given as foreign pointers, with their counts. struct-ref reads a field back as a Lisp value, and setf of it writes one.

The macros know the struct and its fields when they expand. An unknown struct, field or keyword is a compile-time error that lists what is valid, and constant keywords cost nothing at run time. A value of the wrong kind signals struct-value-error, which names the struct and the field.

The generator

(asdf:load-system :cl-vk/generator)

(use-package :cl-vk/generator)

(let* ((registry (read-registry
                  (asdf:system-relative-pathname "cl-vk" "registry/vk.xml")))
       (application-info (find "VkApplicationInfo" (registry-types registry)
                               :key #'type-definition-name :test #'string=)))
  (values (mapcar #'feature-name
                  (remove-if #'feature-internal-p (registry-features registry)))
          (mapcar #'struct-member-name (struct-type-members application-info))))
;; => ("VK_VERSION_1_0" "VK_VERSION_1_1" "VK_VERSION_1_2" "VK_VERSION_1_3"
;;     "VK_VERSION_1_4")
;;    ("sType" "pNext" "pApplicationName" "applicationVersion" "pEngineName"
;;     "engineVersion" "apiVersion")

The generator works in two stages:

1. The reader (generator/reader.lisp) parses a registry file into a registry model of plain, read-only structs. It keeps what belongs to Vulkan and leaves out what is marked for Vulkan SC. It also signals a registry-error that says where a problem is, such as registry/vk.xml: type VkS: expected an array length for a, found "]".
2. The generator (generator/selection.lisp, generator/generator.lisp) reads only the model. It selects what Vulkan's API versions and the included extensions require, following every type, command and constant they refer to across vk.xml and video.xml. It then writes define-constant, define-enum, define-struct, define-command and similar forms, which the macros of cl-vk/ffi-support expand.

The model has:

• C declarations, already parsed. A struct member or command parameter is a c-declaration of a c-type: a type name, its constness, one entry per pointer level, array dimensions and a bit-field width. The C text is not kept.
• Types, one struct per category: external-type, header, define, basetype, bitmask, handle, enum-type, funcpointer, struct-type (unions too) and type-alias.
• Complete enums. Each enum type's values are an enumeration. The values that API versions and extensions add are merged into its enumeration-entries, offsets computed into plain integers, and each value records its origin.
• Constants with Lisp values: (~0ULL) becomes #xFFFFFFFFFFFFFFFF, 1000.0F becomes 1000.0f0, and an extension name becomes a string.
• API versions (feature) and extensions, with their requirements. Dependency expressions are parsed into lists such as (:or "VK_VERSION_1_1" "VK_KHR_get_physical_device_properties2").

Names in the model are the registry's own ("VkInstance", "vkCreateInstance", "VK_SUCCESS"). Turning them into Lisp names is the generator's job. A few definitions in the registry are used only by Vulkan SC. The requirements of the API versions and extensions say which ones Vulkan uses.

The reader is the only part of the generator that uses an XML library (xmls), so swapping that library means rewriting one file. C syntax (declarations, literals and dependency expressions) is parsed in generator/c-syntax.lisp, which knows nothing about XML. The tests in tests/reader-test.lisp and tests/c-syntax-test.lisp define what any replacement must produce.

Regenerating

After changing the generator or updating the registry files:

(asdf:load-system :cl-vk/generator)
(cl-vk/generator:regenerate-bindings)

This rewrites three files, and tests check that each matches what the generator produces:

• registry/vulkan.lisp: the raw layer.
• registry/structs.lisp: what the idiomatic layer knows about each struct and enum. It defines the package cl-vk/structs, whose symbols name the structs (instance-create-info), and describes each field: its keyword without the pointer prefix (:enabled-extension-names), the kind of value, and which field holds an array's length. It also gives every enum value a keyword: VK_PRESENT_MODE_FIFO_KHR is :fifo, and VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT is :color-attachment; and every enum a type of its keywords, such as vks:present-mode-khr.
• tests/layout.c: the C layout program, described below. The output is laid out by hand rather than by the pretty printer, whose layout can change between SBCL releases, so every release produces the same bytes.

tests/layout.c prints the size, alignment and member offsets of every struct of the core API and the video headers. Its output, tests/layout.sexp, is committed, and a test compares cl-ffi's layouts with it. After a registry update, compile it against the C headers of the same Vulkan-Headers release and save its output:

cc -std=c11 -I Vulkan-Headers/include -o layout tests/layout.c
./layout > tests/layout.sexp

Registry files

registry/ holds the registry files that the bindings are generated from. Keeping them in the repository means regenerating gives the same output on every machine:

• vk.xml: the Vulkan API registry, from Vulkan-Headers v1.4.365.
• video.xml: the video codec definitions that vk.xml refers to, from the same release.

The reader's output for this vk.xml has been checked against the C headers of the same release, vulkan_core.h and vulkan_beta.h. Every enum value and constant matches. Every struct member, command and function pointer has exactly the type GCC sees in the header. tests/layout.sexp comes from the same headers on x86-64.

Both files are under Apache-2.0 OR MIT, as stated in each file.

Development

Dependencies are pinned in the stiftfile with stift, and tests use parachute.

stift test

The tests need the Vulkan loader and a Vulkan 1.4 driver with VK_EXT_headless_surface. A software driver is enough, so no GPU is needed: on Debian/Ubuntu, install libvulkan1 and mesa-vulkan-drivers, which provides lavapipe. When the Khronos validation layer is installed (vulkan-validationlayers), the tests run with validation and fail if it reports an error.

License

MIT. See LICENSE. The files in registry/ are under their own licence, stated in each file.