This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Core concepts

Lisp strings in, character indexes out

Every cl-icu function takes ordinary Lisp strings, and every position it returns is a character index into the string you passed.

ICU itself works in UTF-16, where a character outside the Basic Multilingual Plane, such as most emoji, takes two code units. cl-icu converts each string to UTF-16 for ICU and maps ICU's offsets back to character indexes, so you never see a UTF-16 offset:

(icu:bidi-runs (format nil "a~Cאb" (code-char #x1F600)))   ; a, 😀, alef, b
;; => (#S(BIDI-RUN :START 0 :END 2 :DIRECTION :LTR :LEVEL 0)   ; "a😀": 2 characters
;;     #S(BIDI-RUN :START 2 :END 3 :DIRECTION :RTL :LEVEL 1)
;;     #S(BIDI-RUN :START 3 :END 4 :DIRECTION :LTR :LEVEL 0))

Character indexes can be passed straight to subseq, to cl-harfbuzz's :start/:end, and back to cl-icu.

Arguments that name a single character take either a Lisp character or a code point integer (0 to #x10FFFF), the type code-point.

Ranges and context

Several functions take :start and :end (default: the whole string). They are character indexes with 0 ≤ start ≤ end ≤ length; anything else signals an icu-error saying which bound is wrong.

What the range means depends on the function:

• bidi-runs, bidi-levels: the range is a line of the paragraph in the string. The whole paragraph is resolved, so the line gets the levels it has in context, and then the line is reordered on its own. The line must lie within one paragraph.
• script-runs: only that part of the string is split.

Results are plain Lisp values

Results are fresh lists, vectors, strings and read-only structs (bidi-run, bidi-paragraph, script-run, line-break, word). Nothing holds on to ICU objects: each call creates whatever ICU needs and frees it before returning. There is nothing to release and no with- macros.

Concept	Lisp form	Examples
script	ISO 15924 code, a 4-letter string	"Latn", "Arab", "Hani", "Zyyy" (Common), "Zinh" (Inherited)
direction	keyword	:ltr, :rtl; base directions also :auto, :auto-rtl
property value	keyword from the value's long Unicode name	:uppercase-letter, :left-to-right, :wide
property name	keyword or string, matched loosely	:east-asian-width, "East_Asian_Width", "ea"
locale	ICU locale ID string, with optional @ keywords	"tr", "ja@lb=strict", "en@ss=standard"
boundary type	keyword	:grapheme, :word, :line, :sentence
normalization form	keyword	:nfc, :nfd, :nfkc, :nfkd, :nfkc-casefold

Locales

Functions whose results depend on language take :locale: the boundary functions (line-breaks, words, sentences, text-boundaries) and the case functions (upcase, downcase, titlecase).

• Without :locale, ICU's language-neutral root rules apply, never the process's locale. Results don't depend on the machine or the user's environment.
• A locale is an ICU locale ID: a language ("tr"), optionally with region ("pt_BR"), and optional keywords after @ that tailor the rules ("ja@lb=strict").
• An unknown locale is not an error. ICU falls back to the nearest locale it has, ultimately the root rules, and cl-icu does not report the fallback. A misspelt locale silently gives root behaviour.
• Locale only changes the rules that differ for that language. Dictionaries for languages written without spaces (Thai, Lao, Khmer, Burmese, Chinese, Japanese) are used whatever the locale.

The locale-specific behaviours worth knowing are listed on Text boundaries and Normalization and case.

Unicode and ICU versions

cl-icu uses the system's ICU, so results follow that ICU's Unicode version: new scripts, emoji, property values and rule changes arrive with ICU upgrades, without recompiling anything.

(icu:icu-version)       ; => "78.3"
(icu:unicode-version)   ; => "17.0"

Tests and stored data that depend on exact Unicode behaviour should record these. See ICU versions and saved images for how one compiled cl-icu works with any ICU from 52 on.

Threads

• Every cl-icu function may be called from any number of threads at once. Each call works on its own ICU objects, and the shared state (ICU's data, normalizer instances, cl-icu's function table) is only read.
• cl-icu/ffi:load-icu is the exception: it rewrites the function table, so call it only while no other thread is using cl-icu. cl-icu itself calls it at load time and when a saved image starts, before user threads run.

Cost

Each call converts its text to UTF-16, opens the ICU object it needs (a break iterator, a bidi object), walks the text once and frees everything. Time is linear in the length of the text. As an order of magnitude, measured with ICU 78 and SBCL 2.6 on a desktop machine, for a 9,000-character paragraph:

Call	Time
line-breaks, bidi-runs, normalize	about 0.1 ms
words, text-boundaries (graphemes), script-runs	about 0.2–0.3 ms
line-breaks on 40 characters	a few µs
char-property, char-script	under 0.5 µs

char-property, char-general-category, char-bidi-class and char-script allocate nothing once a property and its values have been asked for: cl-icu keeps the keywords and script codes it has found.

So call each function once per paragraph and keep the results, rather than once per character or per line candidate. For example, compute script-runs and text-boundaries for a paragraph once and intersect them with each line's bidi-runs, as the Cookbook does.