Getting started

A guided tour. All output below is real, from ICU 78.3 (Unicode 17.0).

(asdf:load-system :cl-icu)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:icu #:cl-icu)))
(in-package #:example)

(icu:icu-version)   ; => "78.3"

The examples in this wiki use the local nickname icu for cl-icu, and hb for cl-harfbuzz where it appears.

1. Bidirectional text

Text that mixes left-to-right and right-to-left scripts is displayed in an order that differs from the order it is stored in. bidi-runs splits it into directional runs, in visual order (left to right on the screen):

(icu:bidi-runs "abc אבג 123 def")
;; => (#S(BIDI-RUN :START 0 :END 4 :DIRECTION :LTR :LEVEL 0)     ; "abc "
;;     #S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)    ; "123"
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)     ; "אבג "
;;     #S(BIDI-RUN :START 11 :END 15 :DIRECTION :LTR :LEVEL 0))  ; " def"

START and END are character indexes into the string. Inside the Hebrew phrase, the number stays left to right but comes before the Hebrew word on screen. A shaper (cl-harfbuzz) shapes each run with its direction. See Bidirectional text.

2. Script runs

A shaper also needs to know the script. script-runs splits text into runs of one script, letting spaces, digits and punctuation join the run around them:

(icu:script-runs "Hello Мир 日本語")
;; => (#S(SCRIPT-RUN :START 0 :END 6 :SCRIPT "Latn")
;;     #S(SCRIPT-RUN :START 6 :END 10 :SCRIPT "Cyrl")
;;     #S(SCRIPT-RUN :START 10 :END 13 :SCRIPT "Hani"))

Scripts are ISO 15924 codes, the form HarfBuzz takes. See Scripts.

3. Line breaks

line-breaks finds where a line may be broken (:soft) and where it must be (:hard), following UAX #14:

(icu:line-breaks (format nil "Hello world, how~%are you?"))
;; => (#S(LINE-BREAK :POSITION 6 :KIND :SOFT)  #S(LINE-BREAK :POSITION 13 :KIND :SOFT)
;;     #S(LINE-BREAK :POSITION 17 :KIND :HARD) #S(LINE-BREAK :POSITION 21 :KIND :SOFT)
;;     #S(LINE-BREAK :POSITION 25 :KIND :SOFT))

A break at position p means a line may end before character p. Which opportunities you use is your choice: measure the text and pick the last one that fits. Languages written without spaces are handled with dictionaries:

(mapcar #'icu:line-break-position (icu:line-breaks "สวัสดีครับ"))   ; Thai: => (6 10)

4. What a reader calls a character

A Lisp character is a Unicode code point, but a reader sees grapheme clusters: a letter with accents, a flag, a family emoji:

(icu:graphemes "é👨‍👩‍👧🇸🇪!")    ; "é" written as e + U+0301
;; => ("é" "👨‍👩‍👧" "🇸🇪" "!")

Move carets, delete and truncate by graphemes, never inside one. See Text boundaries.

5. Words and sentences

(icu:words "It's 42 cats.")
;; => (#S(WORD :START 0 :END 4 :KIND :LETTER) #S(WORD :START 5 :END 7 :KIND :NUMBER)
;;     #S(WORD :START 8 :END 12 :KIND :LETTER))

(icu:sentences "Hi there. How are you? Fine!")
;; => ("Hi there. " "How are you? " "Fine!")

6. Normalization and case

The same text can be encoded in several ways (é as one code point, or e + combining accent). Normalize before comparing or storing:

(string= "é" (format nil "e~C" (code-char #x301)))                  ; => NIL
(string= "é" (icu:normalize (format nil "e~C" (code-char #x301))))  ; => T

Case mapping follows the full Unicode rules and, when asked, a language's rules:

(icu:upcase "straße")                  ; => "STRASSE"
(icu:upcase "istanbul" :locale "tr")   ; => "İSTANBUL"
(icu:titlecase "ijsland" :locale "nl") ; => "IJsland"
(icu:foldcase "Straße")                ; => "strasse"   for caseless comparison

See Normalization and case.

7. Character properties

Any binary or enumerated Unicode property, by its Unicode name:

(icu:char-general-category #\A)                      ; => :UPPERCASE-LETTER
(icu:char-property #\一 :east-asian-width)           ; => :WIDE
(icu:char-property (code-char #x1F600) :emoji)       ; => T
(icu:char-unicode-name (code-char #x1F600))          ; => "GRINNING FACE"

See Character properties.

8. Putting it together

To lay out a paragraph:

1. Break it into lines. Call line-breaks on the paragraph and choose breaks by measuring the text, for example by shaping it with cl-harfbuzz.
2. Reorder each line. Call bidi-runs with the whole paragraph and the line's :start and :end. Lines must be reordered separately.
3. Shape each piece. Cut each directional run at the script runs and shape every piece with cl-harfbuzz's shape-text, passing :start, :end, :direction and :script.

The Cookbook has the complete code.