This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Shaping

Shaping turns a run of characters into positioned glyphs: choosing glyphs from the cmap, composing and decomposing (é ↔ e + ◌́), applying GSUB substitutions (ligatures, contextual and positional forms, Indic reordering) and GPOS positioning (kerning, mark attachment, cursive attachment).

There are two ways in:

• shape-text: text in, glyph vector out, in one call. Use it unless you need to reuse a buffer or read its properties.
• A buffer: make-buffer → buffer-add-text → set properties → shape → buffer-glyphs.

shape-text

shape-text font text &key (start 0) end direction script language features cluster-level flags → vector of shaped-glyph

Shapes the characters of text from start to end (the rest of text is context, see below) and returns a fresh vector.

Keyword	Default	Meaning
:start, :end	whole string	the part to shape; characters outside it are shaping context
:direction	guessed from the text	:ltr, :rtl, :ttb, :btt
:script	guessed from the text	ISO 15924, e.g. "Arab"
:language	none ("und")	BCP 47; selects the font's language-specific forms (locl)
:features	()	list of feature strings
:cluster-level	:monotone-graphemes	see cluster levels
:flags	()	see buffer flags

Language. Without :language, shape-text shapes with the font's default forms (language "und"); it never takes the language from the process locale, so results do not depend on the machine. With it, the font's localized forms are used:

(hb:shape-text font "шрифт" :language "sr")   ; Serbian forms of б, г, д, п, т in fonts that have them
(hb:shape-text font "ı i İ" :language "tr")

Empty text gives an empty vector.

shaped-glyph

A read-only struct, one per output glyph, in visual order.

Reader	Value
shaped-glyph-index	glyph index in the font (0 is .notdef: the character is missing)
shaped-glyph-cluster	index in the text of the first character the glyph came from
shaped-glyph-x-advance, shaped-glyph-y-advance	how far to move the pen after this glyph, in the font's units (y up)
shaped-glyph-x-offset, shaped-glyph-y-offset	where to draw the glyph relative to the pen (y up); nonzero for marks, kerning on some fonts, vertical text
shaped-glyph-flags	list of :unsafe-to-break, :unsafe-to-concat, :safe-to-insert-tatweel

The pen model: start at the origin; for each glyph, draw it at pen + offset, then add the advance to the pen. The sum of advances is the run's width (horizontal) or height (vertical).

(hb:shape-text font "AV")
;; => #(#S(SHAPED-GLYPH :INDEX 36 :CLUSTER 0 :X-ADVANCE 1270 :Y-ADVANCE 0 :X-OFFSET 0 :Y-OFFSET 0 :FLAGS NIL)
;;      #S(SHAPED-GLYPH :INDEX 57 :CLUSTER 1 :X-ADVANCE 1401 :Y-ADVANCE 0 :X-OFFSET 0 :Y-OFFSET 0
;;                      :FLAGS (:UNSAFE-TO-BREAK)))

Glyph flags

Flag	Meaning
:unsafe-to-break	Breaking the text before this glyph's cluster (between the previous cluster and this one) and shaping the two halves separately could give a different result. Above, kerning between A and V makes the break unsafe. If a glyph lacks this flag you may reuse the shaped glyphs on both sides of it unchanged after a line break; otherwise re-shape.
:unsafe-to-concat	Only reported with the :produce-unsafe-to-concat buffer flag. Shaping the two sides separately and concatenating could differ from shaping them together. Used to reshape only part of a line after an edit. :unsafe-to-break implies :unsafe-to-concat, so a glyph flagged unsafe to break is also unsafe to concatenate.
:safe-to-insert-tatweel	Only with :produce-safe-to-insert-tatweel. Arabic justification may insert a tatweel (kashida) before this glyph without breaking joining.

Buffers

make-buffer → buffer · with-buffer (var) &body body · destroy-buffer buffer · buffer-live-p buffer

A buffer holds text, then, after shape, glyphs. Reuse a buffer across many shapes to avoid allocation: clear-buffer it between runs. Buffers are not thread-safe.

(hb:with-buffer (buffer)
  (hb:buffer-add-text buffer text :start 10 :end 20)   ; the rest is shaping context
  (setf (hb:buffer-direction buffer) :rtl
        (hb:buffer-script buffer) "Arab"
        (hb:buffer-language buffer) "ar")
  (hb:shape font buffer :features '("kern=0" "ss01"))
  (hb:buffer-glyphs buffer))

buffer-add-text buffer text &key (start 0) end → buffer

Appends the characters of text from start to end. The characters before start and after end are passed as context: they are not shaped, but they influence the shaping of the characters next to them. This matters for Arabic and other joining scripts, and for contextual features, whenever you shape a run that is part of a longer paragraph: pass the whole paragraph and the run's bounds rather than (subseq text start end).

Each character's cluster is its index in text, so clusters of a run starting at 10 begin at 10:

(hb:shape-text font "abc" :start 1 :end 2)   ; => #(#S(SHAPED-GLYPH :INDEX 69 :CLUSTER 1 ...))

Bad bounds signal harfbuzz-error:

END is 9, past the end of the text, which has 3 characters.
START and END are character indexes into the text, with
0 <= START <= END <= its length.

Text is passed as Unicode code points (char-code of each character), so any Lisp string works.

clear-buffer buffer → buffer

Empties text and glyphs and unsets direction, script and language, but keeps flags and cluster level. Use between runs.

reset-buffer buffer → buffer

Returns the buffer to the state of a new one, including flags and cluster level.

buffer-length buffer → integer

The number of characters, or after shaping the number of glyphs.

buffer-content-type buffer → keyword

:unicode (holds text), :glyphs (shaped), or :invalid (empty).

(hb:with-buffer (b)
  b                                   ; #<CL-HARFBUZZ:BUFFER invalid 0 {...}>
  (hb:buffer-add-text b "Hello")      ; #<CL-HARFBUZZ:BUFFER unicode 5 {...}>
  (hb:guess-buffer-properties b)
  (hb:shape font b))                  ; #<CL-HARFBUZZ:BUFFER glyphs 5 {...}>

buffer-handle buffer → foreign pointer

The raw hb_buffer_t* for cl-harfbuzz/ffi.

Buffer properties

Direction, script and language describe the run (HarfBuzz calls them segment properties). They must be set before shaping, explicitly or with guess-buffer-properties.

Accessor (setf-able)	Values	Unset
buffer-direction	:ltr, :rtl, :ttb, :btt	nil
buffer-script	ISO 15924 string: "Latn", "Arab", "Deva", "Hani" …	nil
buffer-language	BCP 47 string: "en", "ar", "zh-Hant" …	nil

Setting nil unsets the property. The four directions make up the type direction, and a script is the type script, a string of four letters; anything else but nil is a type-error.

guess-buffer-properties buffer → buffer

Fills in whichever properties are unset: script from the first character with a strong script, direction from the script, and language as "und" (undetermined), which shapes with the font's default forms. Properties you set beforehand are kept.

Unlike HarfBuzz's hb_buffer_guess_segment_properties, it never takes the language from the process locale, so shaping gives the same result on every machine. Set buffer-language for language-specific forms.

(hb:buffer-add-text b "Hello")
(hb:guess-buffer-properties b)
(list (hb:buffer-direction b) (hb:buffer-script b) (hb:buffer-language b))
;; => (:LTR "Latn" "und")

Guessing only looks at the buffer's own characters. For mixed text it picks one script for the whole buffer: itemize first (see Core concepts).

script-direction script → :ltr, :rtl or nil

The natural horizontal direction of an ISO 15924 script, or nil for the few historic scripts written either way:

(hb:script-direction "Arab")   ; => :RTL
(hb:script-direction "Latn")   ; => :LTR
(hb:script-direction "Ital")   ; => NIL   (Old Italic)

Unknown and neutral scripts ("Zyyy") give :ltr.

shape

shape font buffer &key features → buffer

Shapes the buffer's text with font, replacing it with glyphs. The buffer remembers the font, so buffer-glyphs reports in that font's units. Signals harfbuzz-error for an invalid feature string or if HarfBuzz runs out of memory.

buffer-glyphs buffer → vector of shaped-glyph

The shaped glyphs. Signals harfbuzz-error if the buffer holds text that has not been shaped:

The buffer holds text, not glyphs: it has not been shaped.
Shape it with SHAPE first, or shape text in one call with SHAPE-TEXT.

An empty buffer gives an empty vector.

buffer-glyph buffer index → index, cluster, x-advance, y-advance, x-offset, y-offset

The glyph at index of a shaped buffer, as six values: those of the shaped-glyph buffer-glyphs gives for it, but its flags. Where buffer-glyphs makes a vector of structs, it allocates nothing, so code that shapes into one buffer again and again, as text laid out again does, allocates nothing at all. Signals harfbuzz-error for a buffer that has not been shaped, and for an index not below buffer-length:

The buffer holds 6 glyphs, so it has none at 6.
Give an index below BUFFER-LENGTH.

Features

Features are strings in HarfBuzz's syntax (hb_feature_from_string):

String	Meaning
"liga", "+liga"	turn liga on (value 1)
"-liga"	turn liga off
"kern=0", "kern=1"	set a value; 0 is off
"aalt=2", "salt=3", "cv01=2"	choose alternate n
"smcp[3:5]"	apply to characters 3 to 4 only (end exclusive)
"smcp[3:]", "smcp[:5]", "smcp[3]"	open ranges; a single character
"ss01"	stylistic set 1

Ranges are cluster indexes, i.e. indexes into your text. Features not mentioned keep their default: HarfBuzz enables the shaper's defaults (ccmp, locl, rlig, calt, liga, clig, kern, mark, mkmk, plus script-specific ones such as init/medi/fina for Arabic). Discretionary features (dlig, smcp, onum, tnum, ssNN, …) are off until you ask. Features the font does not have are silently ignored; check with face-feature-tags.

(hb:shape-text font "office" :features '("-liga"))        ; no ffi ligature
(hb:shape-text font "1234" :features '("tnum" "onum"))   ; tabular old-style figures
(hb:shape-text font "AV" :features '("kern=0"))         ; no kerning

An invalid string signals harfbuzz-error:

"bogus feature!" is not a valid feature.
Write features in HarfBuzz's syntax: "liga" or "+liga" to turn one
on, "-liga" to turn it off, "aalt=2" for a value, "smcp[3:5]" for a range.
(HarfBuzz: hb_feature_from_string failed)

Cluster levels

buffer-cluster-level buffer → keyword — setf-able

How HarfBuzz merges clusters:

Level	Behaviour
:monotone-graphemes (default)	Marks and other grapheme extenders are merged into their base's cluster; clusters are monotone (increasing for LTR, decreasing for RTL). Good for cursor movement by grapheme.
:monotone-characters	Marks get their own clusters, but clusters stay monotone. Finer hit testing inside graphemes.
:characters	Clusters are not merged or made monotone; each glyph keeps the cluster of the character it came from. For tools that need exact character-glyph correspondence.

The three levels make up the type cluster-level; anything else is a type-error.

Note that normalization happens regardless of level: e + U+0301 with a font that has é comes out as one glyph with cluster 0 at every level.

Buffer flags

buffer-flags buffer → list — setf-able

Flag	Effect
:bot	the text is at the beginning of text (paragraph start); affects some contextual shaping
:eot	the text is at the end of text
:preserve-default-ignorables	show default-ignorable characters (ZWJ, ZWNJ, ZWSP, variation selectors …) as their glyphs instead of hiding them
:remove-default-ignorables	remove them from the output entirely, instead of leaving zero-width space glyphs
:do-not-insert-dotted-circle	do not insert ◌ before a mark with no base (broken clusters)
:verify	have HarfBuzz check its unsafe-to-break/concat results by re-shaping (slow; for debugging)
:produce-unsafe-to-concat	report :unsafe-to-concat on glyphs
:produce-safe-to-insert-tatweel	report :safe-to-insert-tatweel on glyphs

By default a default-ignorable is shaped as a zero-width space glyph:

(map 'list #'hb:shaped-glyph-index (hb:shape-text font (format nil "a~Cb" (code-char #x200B))))
;; => (68 3 69)
(map 'list #'hb:shaped-glyph-index (hb:shape-text font (format nil "a~Cb" (code-char #x200B))
                                                  :flags '(:remove-default-ignorables)))
;; => (68 69)

The flags make up the type buffer-flag; an unknown flag is a type-error. clear-buffer keeps the flags; reset-buffer clears them.

Vertical text

Set the direction to :ttb. Advances are then in y (negative, since y is up) and offsets move each glyph from the vertical origin (top centre) to the horizontal one HarfBuzz draws from:

(map 'list (lambda (g) (list (hb:shaped-glyph-x-advance g) (hb:shaped-glyph-y-advance g)
                             (hb:shaped-glyph-x-offset g) (hb:shaped-glyph-y-offset g)))
     (hb:shape-text font "ab" :direction :ttb))
;; => ((0 -2384 -627 -1751) (0 -2384 -650 -1955))

The vert/vrt2 features are applied automatically. Deciding which characters are upright and which are rotated (UTR #50) is your job, or cl-textrender's.