This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

FFI layer

cl-icu/ffi holds the ICU4C functions cl-icu uses, under their C names in Lisp spelling and without ICU's version suffix (ubidi_setPara_78 → ubidi-set-para), the C types they take, and the C constants. It does no error checking, no string conversion and no memory management. Use it for ICU features the idiomatic layer does not wrap; see the ICU4C API reference for the semantics.

Binding more functions: deficu

deficu c-name lisp-name return-type &body arguments

Defines lisp-name as the ICU function c-name, given without its version suffix. arguments and return-type are as for cl-ffi:defcfun, and the function's ftype is declaimed from them as defcfun declaims it, so SBCL checks a call where it is compiled. The function is looked up under the suffix of the loaded ICU, now and again whenever load-icu runs, so it keeps working across ICU versions and in saved images. Use it rather than cl-ffi:defcfun, which would hard-code one version's name.

Example: char-property cannot read the Age property, but ICU's u_charAge can:

(cl-icu/ffi:deficu "u_charAge" u-char-age :void
  (c cl-icu/ffi:uchar32) (version-array :pointer))

(defun char-age (char)
  "Return the Unicode version that introduced CHAR, e.g. \"6.1\"."
  (cl-ffi:with-foreign-objects ((version :uint8 4) (string :char 20))
    (u-char-age (char-code char) version)
    (cl-icu/ffi:u-version-to-string version string)
    (values (cl-ffi:foreign-string-to-lisp string))))

(char-age #\a)                   ; => "1.1"
(char-age (code-char #x20AC))    ; € => "2.1"
(char-age (code-char #x1F600))   ; 😀 => "6.1"

cl-icu loads only libicuuc. To bind functions from libicui18n (collation, formatting, transliteration, …) on Linux or macOS, load that library yourself first, in the same version:

(cl-ffi:load-foreign-library
 (format nil "libicui18n.so.~A" (subseq cl-icu/ffi:*symbol-suffix* 1)))   ; "libicui18n.so.78"
(cl-icu/ffi:deficu "ucol_countAvailable" ucol-count-available :int32)
(ucol-count-available)   ; => 145

On Windows, icu.dll already contains libicui18n's functions.

Passing text

ICU takes text as UTF-16 (UChar*) with a length in code units. cl-ffi can encode a Lisp string as UTF-16:

(cl-icu/ffi:deficu "u_countChar32" u-count-char32 :int32 (s :pointer) (length :int32))

(defun count-code-points (text)
  (cl-ffi:with-foreign-string ((pointer bytes) text :encoding :utf-16le :null-terminated-p nil)
    (u-count-char32 pointer (floor bytes 2))))

(count-code-points (format nil "a~Cb" (code-char #x1F600)))   ; => 3

Offsets ICU returns are UTF-16 code unit offsets, not Lisp character indexes; convert them yourself if the text has characters above U+FFFF. (cl-icu's own conversion is internal.)

Error codes

ICU reports errors through a UErrorCode out-parameter: initialise it to +u-zero-error+ (0); after the call, a positive value is an error, a negative one a warning. u-error-name gives its C name:

(cl-icu/ffi:u-error-name 7)    ; => "U_MEMORY_ALLOCATION_ERROR"

Loading ICU

Symbol	What it is
libicuuc	the cl-ffi foreign library: libicuuc on Unix and macOS, icu.dll on Windows
load-icu	load libicuuc if needed, find its symbol suffix and look up every deficu function. Runs when cl-icu is loaded and when a saved image starts; call it yourself only after closing and reopening ICU, while no other thread uses cl-icu. Signals an icu-error if ICU cannot be loaded or lacks a function.
probe-symbol-suffix	return the suffix the loaded ICU appends: "" or "_" and the major version; signals an icu-error if the library is no ICU from 50 to 99
*symbol-suffix*	the suffix load-icu found, e.g. "_78"; nil before it has run

See ICU versions and saved images.

Types

Type	C type	cl-ffi type
uchar	UChar, a UTF-16 code unit	:uint16
uchar32	UChar32, a code point	:int32
ubool	UBool	:int8
uerror-code	UErrorCode	:int

Functions

Area	Functions
versions, errors	u-get-version, u-get-unicode-version, u-version-to-string, u-error-name
bidi (ubidi.h)	ubidi-open-sized, ubidi-close, ubidi-set-para, ubidi-set-line, ubidi-get-direction, ubidi-get-base-direction, ubidi-get-para-level, ubidi-get-levels, ubidi-count-runs, ubidi-get-visual-run, ubidi-count-paragraphs, ubidi-get-paragraph-by-index
properties (uchar.h)	u-char-mirror, u-get-bidi-paired-bracket, u-char-name, u-char-from-name, u-get-property-enum, u-get-property-name, u-has-binary-property, u-get-int-property-value, u-get-int-property-max-value, u-get-property-value-name
scripts (uscript.h)	uscript-get-script, uscript-get-script-extensions, uscript-has-script, uscript-get-short-name, uscript-get-name, uscript-get-code
boundaries (ubrk.h)	ubrk-open, ubrk-close, ubrk-first, ubrk-next, ubrk-get-rule-status
normalization (unorm2.h)	unorm2-get-nfc-instance, unorm2-get-nfd-instance, unorm2-get-nfkc-instance, unorm2-get-nfkd-instance, unorm2-get-nfkc-casefold-instance, unorm2-normalize, unorm2-is-normalized
case (ustring.h)	u-str-to-upper, u-str-to-lower, u-str-to-title, u-str-fold-case

Each is documented as "Call the ICU function name."; their arguments follow the C prototypes, and their ftypes are declaimed from the foreign types, as deficu does for every function.

Constants

Constant	Value	Meaning
+u-zero-error+	0	no error
+u-buffer-overflow-error+	15	the output buffer was too small; the return value is the size needed
+ubidi-default-ltr+, +ubidi-default-rtl+	#xFE, #xFF	paragraph level: from the text, defaulting to LTR or RTL
+ubidi-ltr+, +ubidi-rtl+, +ubidi-mixed+, +ubidi-neutral+	0–3	UBiDiDirection values
+ubrk-character+, +ubrk-word+, +ubrk-line+, +ubrk-sentence+	0–3	break iterator types
+ubrk-done+	−1	end of iteration
+ubrk-line-hard+	100	line rule statuses from here up are mandatory breaks
+ubrk-word-number+, +ubrk-word-letter+, +ubrk-word-kana+, +ubrk-word-ideo+	100, 200, 300, 400	lower bounds of the word rule status ranges
+u-bpt-none+, +u-bpt-open+, +u-bpt-close+	0, 1, 2	UBidiPairedBracketType values
+uscript-common+, +uscript-inherited+, +uscript-invalid-code+	0, 1, −1	UScriptCode values
+u-unicode-char-name+, +u-char-name-alias+	0, 3	UCharNameChoice
+u-short-property-name+, +u-long-property-name+	0, 1	UPropertyNameChoice
+uchar-invalid-code+	−1	not a property
+uchar-binary-start+, +uchar-int-start+, +uchar-mask-start+	0, #x1000, #x2000	where binary, enumerated and mask properties start in UProperty
+uchar-bidi-paired-bracket-type+	#x1015	the UProperty Bidi_Paired_Bracket_Type, whose values are the +u-bpt-…+ constants
+u-fold-case-default+, +u-fold-case-exclude-special-i+	0, 1	case folding options; the second is Turkic folding