cl-icu

• Usage
  • Laying out a paragraph
  • Errors
• API overview
• ICU versions
• Not covered
• Development
• License

Common Lisp bindings for ICU's text services: the pieces of text layout that come before shaping, plus normalization, case mapping and character properties.

With cl-harfbuzz and cl-ft2 it forms a text stack:

1. cl-icu splits text into paragraphs, directional runs and script runs, and finds line breaks.
2. cl-harfbuzz shapes each run into positioned glyphs.
3. cl-ft2 renders the glyphs.

cl-icu does not depend on either.

The package cl-icu works on Lisp strings, and every position it returns is a character index into them. ICU itself works in UTF-16, where a character such as an emoji takes two units; cl-icu converts the offsets back. Scripts are ISO 15924 codes such as "Latn", which is the form HarfBuzz takes. Directions and property values are keywords. ICU failures signal an icu-error condition. The raw bindings are in cl-icu/ffi. The examples use icu as a local nickname for cl-icu.

Usage

(asdf:load-system :cl-icu)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:icu #:cl-icu)))
(in-package #:example)

;; Bidirectional text: runs in visual order, each with its direction.
(icu:bidi-runs "abc אבג 123 def")
;; => (#S(BIDI-RUN :START 0 :END 4 :DIRECTION :LTR :LEVEL 0)
;;     #S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)   ; the number
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)    ; the Hebrew
;;     #S(BIDI-RUN :START 11 :END 15 :DIRECTION :LTR :LEVEL 0))

;; Script runs, for shaping each script separately.
(icu:script-runs "Hello Мир 日本語")
;; => (#S(SCRIPT-RUN :START 0 :END 6 :SCRIPT "Latn")
;;     #S(SCRIPT-RUN :START 6 :END 10 :SCRIPT "Cyrl")
;;     #S(SCRIPT-RUN :START 10 :END 13 :SCRIPT "Hani"))

;; Line break opportunities, :soft or :hard (mandatory).
(icu:line-breaks "Hello world")
;; => (#S(LINE-BREAK :POSITION 6 :KIND :SOFT) #S(LINE-BREAK :POSITION 11 :KIND :SOFT))

(icu:graphemes "é👨‍👩‍👧")       ; => ("é" "👨‍👩‍👧") -- what a reader sees as characters
(icu:words "It's 42 cats.")           ; the words It's, 42, cats, with kinds :letter, :number
(icu:normalize "é")            ; => "é"
(icu:upcase "straße")                 ; => "STRASSE"
(icu:upcase "istanbul" :locale "tr")  ; => "İSTANBUL"
(icu:char-property #\一 :east-asian-width)  ; => :WIDE

Laying out a paragraph

1. Find line breaks. Use line-breaks on the paragraph and choose the breaks with your own fitting logic.
2. Reorder each line. Call (bidi-runs paragraph :start line-start :end line-end). Passing the whole paragraph with the line's bounds resolves the line in context, and lines must be reordered separately (UAX #9). Trailing whitespace on a line is moved to the paragraph's level.
3. Shape each run. Split each directional run further with script-runs, then shape the pieces with cl-harfbuzz's shape-text using :start, :end, :direction and :script.

Errors

(handler-case (icu:bidi-runs "abc" :start 2 :end 9)
  (icu:icu-error (e)
    (princ-to-string e)))
;; => "END is 9, past the end of the text, which has 3 characters.
;;     START and END are character indexes into the text, with
;;     0 <= START <= END <= its length."

icu-error is signalled for failures ICU reports, for arguments cl-icu rejects itself, such as an index past the end of the text or an unknown property name, and when the ICU the process finds cannot be loaded or lacks a function cl-icu needs. Its report says what went wrong, then what to do. icu-error-message and icu-error-hint return the two parts. When ICU reported the error, icu-error-function and icu-error-name say which function failed and how (:memory-allocation-error, …); otherwise they are nil. An argument of the wrong type, such as a keyword that is not one of the documented choices, signals a standard type-error.

API overview

• Versions: icu-version, unicode-version.
• Bidirectional text (UAX #9): bidi-base-direction, bidi-paragraphs (→ bidi-paragraph), bidi-levels, bidi-runs (→ bidi-run), and bidi-visual-order and bidi-visual-positions, which map character positions between logical and visual order. Base direction, the type bidi-direction, is :auto (the first strong character decides, defaulting to left to right), :auto-rtl, :ltr or :rtl.
• Scripts: char-script, char-script-extensions, script-name, script-runs (→ script-run). Common and inherited characters join the run around them, and brackets go with the run they were opened in, as in ICU's own script run iterator.
• Text boundaries (UAX #29, UAX #14): text-boundaries (:grapheme, :word, :line or :sentence, the type boundary-type), graphemes, line-breaks (→ line-break), words (→ word), sentences. ICU uses dictionaries for languages written without spaces (Thai, Japanese, …). :locale tailors the rules.
• Normalization: normalize (:nfc, :nfd, :nfkc, :nfkd or :nfkc-casefold, the type normalization-form), normalizedp.
• Case mapping: upcase, downcase, titlecase (with :locale), and foldcase (with :turkic).
• Characters: char-property (any binary or enumerated Unicode property, named as in the UCD, the type unicode-property), char-general-category, char-bidi-class, char-unicode-name, unicode-name-char, char-mirror. These and the script functions take a character or its code point, the type code-point.

Without :locale, ICU's language-neutral root rules apply, not the process's locale, so results don't depend on the machine.

ICU versions

ICU exports its C functions with its major version appended (ubidi_open_78 for ICU 78). cl-icu finds that suffix in whichever ICU the process loads, when cl-icu is loaded and again when an image saved with it starts. So an ICU upgrade needs no recompile, and a saved image works with the ICU of the machine it runs on. Any ICU from 52 on works; only libicuuc is used, and no C compiler is needed.

On Windows, cl-icu uses the icu.dll that ships with Windows 10 version 1903 and later, so there is nothing to install.

Either way, the ICU version, and the Unicode version it supports, are the system's. icu-version and unicode-version report them.

When a saved image starts, cl-ffi loads ICU again, and cl-icu looks its functions up in it.

Not covered

Collation, transliteration, formatting (dates, numbers, messages), regular expressions, charset conversion, IDNA, and locale data. These live mostly in libicui18n and are outside text layout.

Development

Dependencies are pinned in the stiftfile with stift, and tests use parachute.

stift test

The tests need only ICU (Debian/Ubuntu: libicu-dev, or the versioned runtime package such as libicu74; Windows: nothing).

License

MIT. See LICENSE.