cl-icu

Common Lisp bindings for ICU's text services: the steps of text layout that come before shaping (bidirectional text, script runs, line breaking), plus grapheme, word and sentence boundaries, normalization, case mapping and Unicode character properties.

(asdf:load-system :cl-icu)

(defpackage #:example
  (:use #:cl)
  (:local-nicknames (#:icu #:cl-icu)))
(in-package #:example)

(icu:bidi-runs "abc אבג 123 def")
;; => (#S(BIDI-RUN :START 0 :END 4 :DIRECTION :LTR :LEVEL 0)
;;     #S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)    ; "123"
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)     ; the Hebrew
;;     #S(BIDI-RUN :START 11 :END 15 :DIRECTION :LTR :LEVEL 0))

(icu:graphemes "é👨‍👩‍👧")     ; => ("é" "👨‍👩‍👧")
(icu:upcase "straße")        ; => "STRASSE"

Two layers

Package	What it is
cl-icu	The Lisp-idiomatic API. It takes Lisp strings and returns character indexes into them, whatever ICU's UTF-16 offsets were. Scripts are ISO 15924 codes ("Latn"), directions and property values are keywords, and ICU failures signal an icu-error.
cl-icu/ffi	The raw layer: the ICU4C functions cl-icu uses, under their C names without ICU's version suffix (ubidi_setPara_78 → ubidi-set-para), and the C constants they take.

Most programs only need cl-icu. See the FFI layer for the rest. The examples in this wiki use icu as a local nickname for cl-icu, as above.

Contents

Start here

• Installation: requirements, Linux, macOS, Windows
• Getting started: a guided tour
• Core concepts: character indexes, locales, versions, threads, cost

Reference

• Bidirectional text: bidi-runs, bidi-levels, bidi-visual-order, bidi-visual-positions, bidi-paragraphs, bidi-base-direction (UAX #9)
• Scripts: char-script, char-script-extensions, script-name, script-runs
• Text boundaries: graphemes, line-breaks, words, sentences, text-boundaries (UAX #29, UAX #14)
• Normalization and case: normalize, normalizedp, upcase, downcase, titlecase, foldcase
• Character properties: char-property, char-general-category, char-bidi-class, names, char-mirror
• Errors: icu-error and type-error
• ICU versions and saved images: how cl-icu finds ICU's versioned symbols
• FFI layer: the raw cl-icu/ffi package
• API index: every exported symbol, alphabetically

Guides

• Cookbook: laying out a paragraph with cl-harfbuzz, wrapping, carets, word selection, caseless matching, terminal widths, identifiers
• Scope and limitations: what is not covered, and pitfalls
• Development: tests, CI

At a glance

• Runtime requirement: ICU's common library, libicuuc, version 52 or newer, found at load time. No headers, no C compiler. On Windows, the icu.dll that ships with Windows 10 version 1903 and later.
• Dependencies: cl-ffi, uiop. The tests also use parachute.
• Lisp: SBCL. cl-icu has no implementation-specific code, which is all cl-ffi's.
• License: MIT.
• Source: https://fakehacker.cc/dxtr/cl-icu

Related projects

cl-icu is the first layer of a text stack. It depends on neither of the others:

1. cl-icu splits text into paragraphs, directional runs and script runs, and finds line breaks.
2. cl-harfbuzz shapes each run into positioned glyphs.
3. cl-ft2 rasterizes the glyphs.

cl-textrender builds text layout on cl-icu and cl-harfbuzz.