Text boundaries

ICU's break iterators find four kinds of boundaries in text:

Type	What it finds	Standard
:grapheme	grapheme clusters: what a reader perceives as one character	UAX #29
:word	word boundaries, for selection, search and counting	UAX #29
:line	line break opportunities	UAX #14
:sentence	sentence boundaries	UAX #29

For languages written without spaces between words (Thai, Lao, Khmer, Burmese, Chinese, Japanese), ICU uses dictionaries to find word and line boundaries.

text-boundaries text type &key locale → list of integers

The type boundaries of text as character indexes, always starting with 0 and ending with its length. type is required: :grapheme, :word, :line or :sentence.

(icu:text-boundaries (format nil "e~Cx" (code-char #x301)) :grapheme)   ; e + U+0301, x => (0 2 3)
(icu:text-boundaries "Hello world" :word)                  ; => (0 5 6 11)
(icu:text-boundaries "Hello world" :line)                  ; => (0 6 11)
(icu:text-boundaries "Hello world. Bye." :sentence)        ; => (0 13 17)
(icu:text-boundaries "" :grapheme)                         ; => (0)

Consecutive pairs are the segments. With :word they include the spaces and punctuation between words, which words leaves out. The four keywords make up the type boundary-type; an unknown type is a type-error.

graphemes text → list of strings

text split into grapheme clusters:

(icu:graphemes "é👨‍👩‍👧🇸🇪!")
;; => ("é" "👨‍👩‍👧" "🇸🇪" "!")
(icu:graphemes "क्षि")         ; Devanagari conjunct with vowel sign: => ("क्षि")
(icu:graphemes "👍🏽")          ; emoji with skin tone modifier: => ("👍🏽")
(icu:graphemes "")             ; => ()

;; Hangul written as conjoining jamo ᄀ ᅡ ᆨ is one grapheme, as is CR LF:
(length (icu:graphemes (format nil "~C~C~Ca" (code-char #x1100) (code-char #x1161)
                               (code-char #x11A8))))                    ; => 2
(length (icu:graphemes (format nil "~C~Ca" #\Return #\Newline)))       ; => 2

Graphemes are the unit for caret movement, deletion, selection, truncation and counting "characters" in a user interface. A single Lisp character can be part of a grapheme (e + U+0301), and one grapheme can be many Lisp characters (a family emoji is five).

To get indexes instead of strings, use (text-boundaries text :grapheme).

line-breaks text &key locale → list of line-break

The line break opportunities in text, in order. Each is a break before character position: a line may end there.

Reader	Value
line-break-position	character index; the line before it ends at position
line-break-kind	:soft where a line may be broken, :hard where it must be

(icu:line-breaks (format nil "Hello world, how~%are you?"))
;; => (#S(LINE-BREAK :POSITION 6 :KIND :SOFT)  #S(LINE-BREAK :POSITION 13 :KIND :SOFT)
;;     #S(LINE-BREAK :POSITION 17 :KIND :HARD) #S(LINE-BREAK :POSITION 21 :KIND :SOFT)
;;     #S(LINE-BREAK :POSITION 25 :KIND :SOFT))

• A soft break after a space falls after the space: the space belongs to the line before and hangs at its end, outside the measured width (see trim-end in the Cookbook).
• Hard breaks come after line feed, CR LF, U+2029 and the other mandatory breaks. A text that ends with a newline ends with a hard break at its length.
• The last break is always at the end of the text. An empty text has no breaks.

Some of the UAX #14 rules, as ICU applies them:

Text	Break positions	Why
"Pay 1,000 now"	4 10 13	no break inside a number
"(a) b"	4 5	none inside brackets
"well-known e-mail"	5 11 13 17	after hyphens
"see https://example.com/a/b"	4 12 24 26 27	after / in URLs
"a" NBSP "b c"	4 5	U+00A0 NO-BREAK SPACE prevents a break
"word" ZWSP "word"	5 9	U+200B ZERO WIDTH SPACE allows one
"日本語の文章です。"	1 2 3 4 5 6 7 9	between CJK characters, but not before 。
"สวัสดีครับ" (Thai)	6 10	at word boundaries found by the dictionary
"안녕하세요 세계" (Korean)	1 2 3 4 6 7 8	between Hangul syllables, as UAX #14 allows

For Korean set in the modern style, where lines break only at spaces, keep only the breaks that follow white space (ICU has no option for this).

words text &key locale → list of word

The words of text, in order: the segments between word boundaries that ICU classifies as words. The spaces, punctuation and symbols between them are left out; (text-boundaries text :word) gives every segment.

Reader	Value
word-start, word-end	character indexes
word-kind	:letter, :number, :kana or :ideographic

(icu:words "It's 42 cats.")
;; => (#S(WORD :START 0 :END 4 :KIND :LETTER)    ; "It's"
;;     #S(WORD :START 5 :END 7 :KIND :NUMBER)    ; "42"
;;     #S(WORD :START 8 :END 12 :KIND :LETTER))  ; "cats"

(length (icu:words "It's 42 cats."))   ; => 3
(icu:words "... !?")                   ; => ()

Rules worth knowing: apostrophes stay inside words ("don't"), hyphens split them ("e-mail" is the words e and mail), decimal numbers stay whole ("3.14"), and "U.S.A." is the word U.S.A, without the final period.

Dictionary-based languages:

(mapcar #'icu:word-end (icu:words "สวัสดีครับ"))         ; Thai => (6 10)
(mapcar #'icu:word-end (icu:words "ຂອບໃຈຫຼາຍ"))          ; Lao => (5 9)
(mapcar #'icu:word-end (icu:words "សួស្តីពិភពលោក"))       ; Khmer => (6 13)
(mapcar #'icu:word-end (icu:words "中华人民共和国"))       ; Chinese => (2 4 7)
(icu:words "日本語の文章です")                            ; Japanese: 4 words, all :IDEOGRAPHIC

ICU's dictionary engine for Chinese and Japanese reports its words, including katakana words, as :ideographic; with ICU 78, :kana does not occur in practice.

sentences text &key locale → list of strings

text split into sentences, each with its trailing spaces:

(icu:sentences "Hi there. How are you? Fine!")
;; => ("Hi there. " "How are you? " "Fine!")

The default rules end a sentence at a period followed by a space and a capital letter, even after an abbreviation such as "Mr.". The ss=standard locale keyword suppresses breaks after the language's common abbreviations:

(icu:sentences "Mr. Smith left. Bye.")                          ; => ("Mr. " "Smith left. " "Bye.")
(icu:sentences "Mr. Smith left. Bye." :locale "en@ss=standard") ; => ("Mr. Smith left. " "Bye.")
(icu:sentences "Dr. Müller kam. Er ging." :locale "de@ss=standard")
;; => ("Dr. Müller kam. Er ging.")

Locales

Without :locale, the language-neutral root rules apply. A locale tailors them, mostly through ICU's @ keywords:

Locale	Effect
"ja@lb=strict"	strict Japanese line breaking: no break before small kana (ぁ ゃ ッ …) and prolonged sound marks
"ja@lb=loose", "ja@lb=normal"	looser variants, allowing breaks before small kana
"ja@lw=phrase"	phrase-based Japanese line breaking: "私はラーメンが好きです。" breaks only at 2, 7 and 12 instead of after almost every character
"en@ss=standard", "de@ss=standard", …	suppress sentence breaks after common abbreviations

(mapcar #'icu:line-break-position (icu:line-breaks "ぁぃゃ" :locale "ja@lb=strict"))   ; => (3)
(mapcar #'icu:line-break-position (icu:line-breaks "ぁぃゃ" :locale "ja@lb=loose"))    ; => (1 2 3)

ICU silently ignores keywords it does not implement ("ko@lw=keepall" changes nothing) and falls back to the root rules for locales it does not know.

Grapheme indexes and the other layers

• Bidi: caret positions are grapheme boundaries in logical order; bidi-levels tells you where they appear visually.
• Shaping: HarfBuzz's clusters merge graphemes further (a ligature covers several graphemes). Place carets inside a ligature by splitting it evenly between its graphemes, as cl-textrender does.
• Line breaking: never break inside a grapheme. line-breaks never does; when a word is too long for any line and you must break inside it, break at a grapheme boundary.