Grammar and Generator

clsl's SPIR-V tables, which clsl/spirv's functions such as instruction-opcode and enumerant-capabilities read, are generated from the SPIR-V grammar that Khronos publishes in JSON.

The grammar files

The grammar is vendored in grammar/: spirv.core.grammar.json, SPIR-V 1.6, revision 7, and extinst.glsl.std.450.grammar.json, the extended instruction set GLSL.std.450, unmodified from Khronos's SPIRV-Headers, with their licence, MIT, in grammar/LICENSE.

clsl/generator reads them and writes spirv/grammar.lisp, which is committed, so loading clsl needs no JSON library. Never edit that file by hand. After changing a grammar file or the generator, run

(asdf:load-system :clsl/generator)
(clsl/generator:regenerate-grammar)

which writes spirv/grammar.lisp and returns its pathname, and commit both. The test committed-grammar fails when the committed file is not what the generator writes. grammar-text returns the text the generator writes, without writing it:

(string= (clsl/generator:grammar-text)
         (uiop:read-file-string (asdf:system-relative-pathname "clsl/spirv" "spirv/grammar.lisp")))
;; => T

The generator

The generator reads JSON with jzon, in generator/json.lisp alone, which turns it into plain Lisp data: changing the JSON library is rewriting that file. read-json reads a file in UTF-8 or a string of JSON:

JSON	Lisp
an object	an equal hash table from its keys, as strings, to values
an array	a simple vector
a string	a simple string
an integer	an integer
another number	a double-float
true, false	:true and :false
null	:null

(asdf:load-system :clsl/generator)

(gethash "a" (clsl/generator:read-json "{\"a\": [1, 2.5, true, null]}"))
;; => #(1 2.5d0 :TRUE :NULL)
;;    T

The grammar reader turns that data into a model of plain structs, which never changes once read: a grammar of instructions and operand kinds, the kinds' enumerants, and the operands of instructions and enumerants; and an extended-grammar for an extended instruction set. Names in the model are the grammar's own, such as "OpTypeInt" and "GLCompute":

(let ((grammar (clsl/generator:read-grammar
                (asdf:system-relative-pathname "clsl" "grammar/spirv.core.grammar.json"))))
  (list (clsl/generator:grammar-major-version grammar)
        (clsl/generator:grammar-minor-version grammar)
        (clsl/generator:grammar-revision grammar)
        (length (clsl/generator:grammar-instructions grammar))
        (length (clsl/generator:grammar-operand-kinds grammar))))
;; => (1 6 7 877 72)

(let ((glsl (clsl/generator:read-extended-grammar
             (asdf:system-relative-pathname "clsl" "grammar/extinst.glsl.std.450.grammar.json")
             "GLSL.std.450")))
  (list (clsl/generator:extended-grammar-name glsl)
        (length (clsl/generator:extended-grammar-instructions glsl))))
;; => ("GLSL.std.450" 81)

read-grammar and read-extended-grammar read a file; parse-grammar and parse-extended-grammar take what read-json returned. lisp-name gives the keyword's name clsl uses for a name of the grammar: its words joined by hyphens, in lower case. A word starts at an underscore, which becomes a hyphen; at a capital after a small letter; at a capital that starts a capitalised word, as in "FAdd"; and at a run of two or more capitals after a digit, as in "Float16NV", but not at one, as in "Sampled1D". A few names this rule splits wrongly, such as "DPdx", are renamed. The tables drop an instruction's Op first.

(mapcar #'clsl/generator:lisp-name
        '("FOrdLessThan" "GLCompute" "ImageMSArray" "FAdd" "Float16NV" "Sampled1D" "DPdx" "2D"))
;; => ("f-ord-less-than" "gl-compute" "image-ms-array" "f-add" "float16-nv" "sampled1d" "dpdx" "2d")

write-grammar writes the tables of a grammar and its extended grammars to a stream, and signals a grammar-error if two names the tables keep apart would become the same keyword.

Errors

The generator signals a grammar-error, its own condition, when a grammar file is not well-formed JSON, or does not follow the format of the SPIR-V grammar: a missing field, a field of the wrong type, or a field the generator does not know. A field it does not know may be one a newer grammar added, which the generator has to learn before it can read that grammar. grammar-error-source is the file being read, or nil for a string; grammar-error-context where in the grammar the problem is, outermost first, as strings such as "instruction OpTypeInt" and "operand 2"; and grammar-error-message what is wrong.

(handler-case (clsl/generator:parse-grammar
               (clsl/generator:read-json "{\"magic_number\": \"0x07230203\"}"))
  (clsl/generator:grammar-error (error)
    (princ-to-string error)))
;; => "The field \"operand_kinds\" is missing."

Updating the grammar

The grammar files stay as Khronos publishes them. A newer grammar replaces them in grammar/; regenerate-grammar then writes the tables from it, or signals a grammar-error naming a field the generator has to learn first. The grammar files, the generated spirv/grammar.lisp and any change to the generator are committed together, and the suite run, committed-grammar among it.