This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
Cookbook
Complete recipes. Each was run against DejaVu Sans; outputs shown are real. They are written in a package that gives cl-harfbuzz, cl-ft2 and cl-icu local nicknames:
(asdf:load-system :cl-harfbuzz/ft2) ; loads cl-harfbuzz and cl-ft2
(asdf:load-system :cl-icu)
(defpackage #:cookbook
(:use #:cl)
(:local-nicknames (#:hb #:cl-harfbuzz) (#:ft2 #:cl-ft2) (#:icu #:cl-icu)))
(in-package #:cookbook)
- Measuring text
- Drawing a line into a bitmap
- Shaping a bidi paragraph
- Hit testing and carets
- Line breaking without re-shaping everything
- Font fallback
- OpenType features
- Reusing a buffer
- Fonts from memory
- Color emoji
- Variable fonts
- Glyph outlines for a GPU renderer
Measuring text
The advance width of a run is the sum of its advances:
(defun text-width (font text &rest args)
(reduce #'+ (apply #'hb:shape-text font text args) :key #'hb:shaped-glyph-x-advance))
(hb:with-face (face "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf")
(hb:with-font (font face)
(text-width font "Hello, world"))) ; => 12132 font units
;; at 16 px: (* 12132 16/2048) ; => 3033/32 ≈ 94.8 px
(defun text-width-px (font text pixels &rest args)
"The advance width of TEXT shaped with FONT, in pixels at PIXELS per EM."
(* (apply #'text-width font text args)
(/ pixels (hb:face-units-per-em (hb:font-face font)))))
This includes kerning and ligatures, which summing font-glyph-advance per character would not. For the ink extent instead (where pixels actually land), offset each glyph's font-glyph-extents box by its pen position. Line height comes from font-extents: ascender - descender + line-gap.
Drawing a line into a bitmap
With cl-harfbuzz/ft2, positions are in pixels and line up with cl-ft2's bitmaps. Mind the y axes: HarfBuzz offsets and ft2:bitmap-top are y-up, the canvas is y-down.
(defun draw-line (face text canvas-width baseline)
"Return a 2D octet array with TEXT drawn on BASELINE (pixels from the top)."
(let ((canvas (make-array (list (* 2 baseline) canvas-width)
:element-type '(unsigned-byte 8) :initial-element 0)))
(cl-harfbuzz/ft2:with-ft2-font (font face)
(let ((pen-x 0) (pen-y 0))
(loop for g across (hb:shape-text font text)
do (let* ((bitmap (ft2:glyph-index-bitmap face (hb:shaped-glyph-index g)))
(x0 (+ (round (+ pen-x (hb:shaped-glyph-x-offset g)))
(ft2:bitmap-left bitmap)))
(y0 (- baseline
(round (+ pen-y (hb:shaped-glyph-y-offset g)))
(ft2:bitmap-top bitmap)))
(data (ft2:bitmap-data bitmap)))
(dotimes (row (array-dimension data 0))
(dotimes (col (array-dimension data 1))
(let ((x (+ x0 col)) (y (+ y0 row)))
(when (array-in-bounds-p canvas y x)
(setf (aref canvas y x)
(max (aref canvas y x) (aref data row col)))))))
(incf pen-x (hb:shaped-glyph-x-advance g))
(incf pen-y (hb:shaped-glyph-y-advance g))))))
canvas))
(ft2:with-library (lib)
(ft2:with-face (face lib "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf")
(setf (ft2:face-size face) 24)
(draw-line face "Hello" 100 24)))
Rounding each glyph's position to whole pixels is the simplest choice. For better spacing, keep the fractional pen and render at subpixel offsets (cl-textrender's raster backend uses quarter-pixel positions), or round only the pen start of each run.
Shaping a bidi paragraph
HarfBuzz shapes one direction and one script at a time. cl-icu splits a paragraph into bidi runs (in visual order) and script runs; shape each, passing the whole text so context crosses run boundaries:
(defun shape-paragraph (font text)
"Shape TEXT into a list of glyph vectors, in visual order."
(loop for bidi in (icu:bidi-runs text)
nconc (let ((script-runs (icu:script-runs text :start (icu:bidi-run-start bidi)
:end (icu:bidi-run-end bidi))))
;; Script runs come in logical order; an RTL run displays them reversed.
(when (eq (icu:bidi-run-direction bidi) :rtl)
(setf script-runs (reverse script-runs)))
(loop for run in script-runs
collect (hb:shape-text font text
:start (icu:script-run-start run)
:end (icu:script-run-end run)
:direction (icu:bidi-run-direction bidi)
:script (icu:script-run-script run))))))
(mapcar (lambda (v) (map 'list #'hb:shaped-glyph-cluster v))
(shape-paragraph font "abc سلام def"))
;; => ((0 1 2 3) (7 5 4) (8 9 10 11))
Draw the vectors left to right, one after another. In the Arabic run, characters 5 and 6 (ل and ا) formed the lam-alef ligature, so cluster 6 does not appear.
For multi-line text, break lines first (cl-icu's line breaking, with widths from shaping), then call icu:bidi-runs per line with :start/:end, since bidi reordering is per line. A full implementation, with font fallback and styled spans, is in cl-textrender.
Hit testing and carets
Clusters map glyphs back to characters. Group adjacent glyphs by cluster to get each cluster's horizontal extent:
(defun cluster-boundaries (glyphs)
"Return a list of (CLUSTER X-START X-END), one per cluster, in visual order."
(let ((x 0) (result '()))
(loop for g across glyphs
for c = (hb:shaped-glyph-cluster g)
do (if (and result (= c (first (first result))))
(incf (third (first result)) (hb:shaped-glyph-x-advance g))
(push (list c x (+ x (hb:shaped-glyph-x-advance g))) result))
(incf x (hb:shaped-glyph-x-advance g)))
(nreverse result)))
(cluster-boundaries (hb:shape-text font "office"))
;; => ((0 0 1253) (1 1253 3233) (4 3233 4359) (5 4359 5619))
A ligature cluster covers several characters (here 1, 2, 3: "ffi"). Without ligature caret data, split it evenly:
(defun character-at (glyphs text-length x)
"Return the index of the character under X, splitting ligatures evenly."
(let* ((clusters (cluster-boundaries glyphs))
(sorted (sort (mapcar #'first clusters) #'<)))
(loop for (c x0 x1) in clusters
when (and (<= x0 x) (< x x1))
do (let* ((next (or (find-if (lambda (n) (> n c)) sorted) text-length))
(n (- next c))
(part (floor (* n (- x x0)) (- x1 x0))))
(return (+ c part))))))
(let ((g (hb:shape-text font "office")))
(list (character-at g 6 1300) (character-at g 6 2000) (character-at g 6 3000)))
;; => (1 2 3)
Caveats: for RTL runs, a cluster's characters run right to left within its box; place carets only at grapheme boundaries (cl-icu); with :monotone-graphemes marks share their base's cluster, which is usually what you want for carets. cl-textrender implements all of this (hit-test, move-caret, selection rectangles).
Line breaking without re-shaping everything
Shape a whole paragraph once, then break it into lines. When the break falls before a glyph that is not :unsafe-to-break, the shaped glyphs on each side are still correct; otherwise, re-shape the two lines separately.
(defun safe-break-p (glyphs cluster)
"True if the shaped GLYPHS may be split before CLUSTER without re-shaping."
(let ((g (find cluster glyphs :key #'hb:shaped-glyph-cluster)))
(and g (not (member :unsafe-to-break (hb:shaped-glyph-flags g))))))
(let ((g (hb:shape-text font "AV Wa")))
(list (safe-break-p g 1) (safe-break-p g 2) (safe-break-p g 3)))
;; => (NIL T T) ; between A and V is kerned; after it is safe
(Line breaking itself, i.e. where breaks are allowed, is UAX #14: cl-icu.)
For an editor that re-shapes after each edit, shape with :flags '(:produce-unsafe-to-concat) and re-shape only from the last :unsafe-to-concat-free glyph before the edit to the first one after it.
Font fallback
A character the font lacks shapes to glyph 0. Find those clusters and shape them with another font:
(defun missing-clusters (glyphs)
(remove-duplicates
(loop for g across glyphs
when (zerop (hb:shaped-glyph-index g))
collect (hb:shaped-glyph-cluster g))))
(missing-clusters (hb:shape-text font (format nil "ab~Cc" (code-char #xE000))))
;; => (2)
It is usually better to choose fonts before shaping, per character run, with font-char-glyph as a coverage test ((hb:font-char-glyph font #\U+4E2D) → nil for DejaVu Sans), and to keep a combining mark with its base's font. cl-textrender has an explicit fallback list.
OpenType features
(hb:shape-text font "office" :features '("-liga")) ; no standard ligatures
(hb:shape-text font "AVATAR" :features '("kern=0")) ; no kerning
(hb:shape-text font "Hello" :features '("smcp")) ; small caps, if the font has them
(hb:shape-text font "0123" :features '("tnum")) ; tabular figures for tables
(hb:shape-text font "1/2" :features '("frac")) ; fractions
(hb:shape-text font "Hello" :features '("smcp[1:]")) ; small caps after the first letter
(hb:shape-text font "a" :features '("salt=2")) ; second stylistic alternate
Check what a font offers with (hb:face-feature-tags face) and (hb:face-feature-tags face :table :gpos).
Reusing a buffer
shape-text makes a buffer per call. To shape many runs on one thread, reuse one:
(defun shape-many (font strings)
(hb:with-buffer (buffer)
(loop for s in strings
collect (progn (hb:clear-buffer buffer)
(hb:buffer-add-text buffer s)
(setf (hb:buffer-language buffer) "en")
(hb:guess-buffer-properties buffer)
(hb:shape font buffer)
(hb:buffer-glyphs buffer)))))
(mapcar #'length (shape-many font '("one" "two" "three"))) ; => (3 3 5)
clear-buffer keeps flags and cluster level; buffer-glyphs copies the results, so they survive the next clear-buffer.
To shape again and again without allocating, as a label whose numbers change, read the glyphs where they are, one at a time, before the next clear-buffer: (hb:buffer-glyph buffer index) returns a glyph's index, cluster, advances and offsets as six values. Clearing, adding text, setting the direction and the script, guessing the rest, shaping without features and reading the glyphs so allocate nothing.
(dotimes (index (hb:buffer-length buffer))
(multiple-value-bind (glyph cluster x-advance) (hb:buffer-glyph buffer index)
…))
Fonts from memory
Pass an octet vector, e.g. a font embedded in your application or downloaded:
(let ((octets (with-open-file (in path :element-type '(unsigned-byte 8))
(let ((v (make-array (file-length in) :element-type '(unsigned-byte 8))))
(read-sequence v in)
v))))
(hb:with-face (face octets)
(hb:face-glyph-count face))) ; => 6253
The octets are copied, so the vector may be reused. WOFF and WOFF2 must be converted first: make-face reports them as such (see Faces).
Color emoji
Test whether a glyph has color data, then draw it with glyph-paint:
(defun color-glyph-p (font glyph)
(or (hb:glyph-color-paint-p font glyph) ; COLRv1
(hb:glyph-color-layers font glyph) ; COLRv0
(hb:glyph-svg font glyph) ; SVG
(hb:glyph-png font glyph))) ; CBDT / sbix (set font-ppem first)
To pick emoji vs. text presentation, append U+FE0F (emoji) or U+FE0E (text) to the character and let the font's cmap variation sequences decide, then fall back to an emoji font for clusters whose glyph is not a color glyph. Emoji ZWJ sequences (👩👩👧) become one glyph through the font's ligatures, so shape the whole sequence with the emoji font.
Variable fonts
(hb:with-face (face "Inter.ttf")
(hb:with-font (font face)
(loop for weight from 100 to 900 by 100
collect (progn (setf (hb:font-axis-coordinate font "wght") weight)
(text-width font "Hello")))))
Changing the variation is cheap; each font has its own, so keep one font per weight you use concurrently. For an ft2-font, set the axis through either library; see Font variations.
Glyph outlines for a GPU renderer
Shape, then fetch each distinct glyph's outline once:
(defun outlines-for (font text)
(let ((cache (make-hash-table)))
(loop for g across (hb:shape-text font text)
for id = (hb:shaped-glyph-index g)
unless (gethash id cache)
do (setf (gethash id cache) (hb:glyph-outline font id)))
cache))
Outlines of a plain font are in font units: build your SDF/MSDF atlas or Slug curve textures once per glyph and scale at draw time. cl-textrender provides a per-thread glyph cache and cubic-to-quadratic conversion on top of this.