This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Normalization and case

Normalization

Unicode can encode the same text in more than one way. "é" is either the single code point U+00E9 or e followed by U+0301 COMBINING ACUTE ACCENT; Hangul syllables can be precomposed or spelt out in jamo. The strings look identical but are not string=. Normalization (UAX #15) converts text to one chosen form, so that equal-looking text compares equal.

normalize text &key (form :nfc) → string

Form	Result	Use it for
:nfc (default)	canonical composition: precomposed characters where they exist	storing and exchanging text; what most text already is
:nfd	canonical decomposition: base letters followed by combining marks	processing marks separately, accent-insensitive matching
:nfkc	compatibility composition: NFC, and also folds formatting variants (ligatures, full-width forms, circled and superscript digits, …) to plain characters	identifiers, search keys
:nfkd	compatibility decomposition	as NFKC, decomposed
:nfkc-casefold	NFKC plus case folding and removal of default-ignorable characters (Unicode's NFKC_Casefold)	case-insensitive identifiers and keys in one step

(icu:normalize (format nil "e~C" (code-char #x301)))              ; => "é" (U+00E9)
(map 'list #'char-code (icu:normalize "é" :form :nfd))            ; => (101 769)
(map 'list #'char-code (icu:normalize "가" :form :nfd))           ; => (4352 4449)  jamo
(icu:normalize "fiＡ①²" :form :nfkc)                               ; => "fiA12"
(icu:normalize "Straße Dž" :form :nfkc-casefold)                   ; => "strasse dž"
(map 'list #'char-code (icu:normalize (string (code-char #x212B)))) ; Å ANGSTROM SIGN => (197)

Canonical forms (NFC, NFD) never change how text looks. Compatibility forms (NFKC, NFKD) can: "fi" becomes "fi", "²" becomes "2". Use them for matching and identifiers, not for text you will display again.

The five forms make up the type normalization-form; an unknown form is a type-error.

normalizedp text &key (form :nfc) → boolean

True if text is already in form; cheaper than normalizing and comparing.

(icu:normalizedp "abc")                                           ; => T
(icu:normalizedp (format nil "e~C" (code-char #x301)))            ; => NIL
(icu:normalizedp (format nil "e~C" (code-char #x301)) :form :nfd) ; => T

Normalization and shaping

You do not need to normalize text before shaping it: HarfBuzz composes and decomposes as the font requires. Normalize when you compare, search, sort into keys, or store text.

Case mapping

Unicode case mapping is not character by character: the result can be longer than the input (ß → SS), and some mappings depend on context (Greek final sigma) or language (Turkish dotted I). The case functions handle all of these.

All of them take a string and return a new string. upcase, downcase and titlecase take :locale; without it the language-neutral root rules apply, never the process's locale.

upcase text &key locale → string

(icu:upcase "straße")                  ; => "STRASSE"
(icu:upcase "džemal")                  ; => "DŽEMAL"
(icu:upcase "istanbul")                ; => "ISTANBUL"
(icu:upcase "istanbul" :locale "tr")   ; => "İSTANBUL"   dotted capital İ
(icu:upcase "ὀδός")                    ; => "ὈΔΌΣ"
(icu:upcase "ὀδός" :locale "el")       ; => "ΟΔΟΣ"       Greek drops accents in capitals

downcase text &key locale → string

(icu:downcase "ΟΔΟΣ")                  ; => "οδος"        final sigma ς at the end of a word
(icu:downcase "İSTANBUL")              ; => "i̇stanbul"    i + U+0307 COMBINING DOT ABOVE
(icu:downcase "İSTANBUL" :locale "tr") ; => "istanbul"

titlecase text &key locale → string

The first letter of each word in title case, the rest in lower case. Words are found with the word break rules.

(icu:titlecase "hello wORLD")             ; => "Hello World"
(icu:titlecase "don't stop")              ; => "Don't Stop"
(icu:titlecase "džemal")                  ; => "Džemal"   title case is not upper case for digraphs
(icu:titlecase "ijsland")                 ; => "Ijsland"
(icu:titlecase "ijsland" :locale "nl")    ; => "IJsland"  Dutch IJ

Title-casing every word is a typographic convention of English headlines; check that it suits the language before applying it.

foldcase text &key turkic → string

Case folding maps text to a form for caseless comparison: two strings are equal ignoring case when their foldings are string=. It differs from downcase: it is language-independent, it folds ß to "ss" and final σ to σ, and it is stable.

(icu:foldcase "Straße")                       ; => "strasse"
(string= (icu:foldcase "MASSE") (icu:foldcase "Maße"))   ; => T
(icu:foldcase "ΣΊΣΥΦΟΣ")                      ; => "σίσυφοσ"   no final ς
(icu:foldcase "I")                            ; => "i"
(icu:foldcase "I" :turkic t)                  ; => "ı"   dotless, as in Turkish and Azerbaijani
(icu:foldcase "İ" :turkic t)                  ; => "i"

For fully correct caseless matching, combine folding with normalization; see Caseless comparison in the Cookbook.

Locales

Locale	Rules
"tr", "az"	dotted and dotless i: i ↔ İ, ı ↔ I
"lt"	Lithuanian keeps the dot of i under accents in lower case
"el"	Greek removes accents when upper-casing
"nl"	Dutch title-cases the digraph IJ together

Other locales use the root rules. An unknown locale is not an error; it falls back to the root rules.