This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Bidirectional text

cl-icu implements the Unicode Bidirectional Algorithm (UAX #9) through ICU's ubidi. Text is stored in logical order (the order it is typed and read); the algorithm finds the visual order in which to display it, when it mixes left-to-right scripts (Latin, Cyrillic, CJK, …) with right-to-left ones (Hebrew, Arabic, Syriac, Thaana, N'Ko, …).

The examples use אבג (Hebrew alef, bet, gimel) as right-to-left text.

Concepts

• Paragraphs. The algorithm works paragraph by paragraph. A paragraph ends after a paragraph separator: line feed, carriage return (CR LF counts as one), U+2029 PARAGRAPH SEPARATOR, and the other characters of bidi class B (U+001C–U+001E, U+0085).
• Base direction (paragraph level). Each paragraph has a direction, usually taken from its first strong character (a letter with an inherent direction).
• Embedding levels. Every character gets a level: even levels are left to right, odd levels right to left. The paragraph level is 0 (LTR) or 1 (RTL). Left-to-right text inside a right-to-left paragraph gets level 2, a number inside Hebrew gets level 2, and so on.
• Runs. Consecutive characters at the same level form a run. Reordering reverses runs by level; the result is the runs in visual order, each with a single direction, which is what a shaper needs.
• Lines. Reordering happens per line, after line breaking. Trailing white space on a line goes to the paragraph level (UAX #9 rule L1), so it ends up at the line's end.

Base directions

The :direction argument of bidi-paragraphs, bidi-levels and bidi-runs sets each paragraph's base direction:

:direction	Meaning
:auto (default)	each paragraph's first strong character decides; left to right if it has none
:auto-rtl	likewise, but right to left if it has none
:ltr	every paragraph is left to right
:rtl	every paragraph is right to left

These four keywords make up the type bidi-direction; anything else is a type-error. Use :auto for text whose direction you don't know; force :ltr or :rtl when the context decides, such as a text field in a right-to-left user interface.

bidi-base-direction text → :ltr, :rtl or nil

The direction of text's first strong character, ignoring neutrals such as digits, spaces and punctuation; nil if it has none.

(icu:bidi-base-direction "abc")       ; => :LTR
(icu:bidi-base-direction "123 אבג")   ; => :RTL
(icu:bidi-base-direction "123")       ; => NIL

Use it to align a text field or label by its content (see the Cookbook).

bidi-paragraphs text &key (direction :auto) → list of bidi-paragraph

text's paragraphs. Each paragraph includes its separator.

Reader	Value
bidi-paragraph-start, bidi-paragraph-end	character indexes; end is after the separator
bidi-paragraph-direction	:ltr or :rtl
bidi-paragraph-level	the paragraph level, 0 or 1

(icu:bidi-paragraphs (format nil "abc~%אבג~Cx~C~%y" (code-char #x2029) #\Return))
;; => (#S(BIDI-PARAGRAPH :START 0 :END 4 :DIRECTION :LTR :LEVEL 0)    ; "abc" LF
;;     #S(BIDI-PARAGRAPH :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)    ; "אבג" U+2029
;;     #S(BIDI-PARAGRAPH :START 8 :END 11 :DIRECTION :LTR :LEVEL 0)   ; "x" CR LF
;;     #S(BIDI-PARAGRAPH :START 11 :END 12 :DIRECTION :LTR :LEVEL 0)) ; "y"

(icu:bidi-paragraphs (format nil "123~%abc") :direction :auto-rtl)
;; => (#S(BIDI-PARAGRAPH :START 0 :END 4 :DIRECTION :RTL :LEVEL 1)    ; no strong character
;;     #S(BIDI-PARAGRAPH :START 4 :END 7 :DIRECTION :LTR :LEVEL 0))

An empty string has no paragraphs.

bidi-runs text &key (direction :auto) (start 0) end → list of bidi-run

The directional runs of text from start to end, in visual order: the first run is leftmost on screen. With start and end, they are a line (see Lines); without them, every paragraph of text is reordered by itself (see Text with several paragraphs).

Reader	Value
bidi-run-start, bidi-run-end	character indexes into text
bidi-run-direction	:ltr or :rtl
bidi-run-level	the run's embedding level

(icu:bidi-runs "abc אבג 123 def")
;; => (#S(BIDI-RUN :START 0 :END 4 :DIRECTION :LTR :LEVEL 0)     ; "abc "
;;     #S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)    ; "123"
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)     ; "אבג "
;;     #S(BIDI-RUN :START 11 :END 15 :DIRECTION :LTR :LEVEL 0))  ; " def"

Draw the runs left to right in that order. Within a right-to-left run, characters are drawn right to left: shape the run with direction :rtl and the shaper takes care of it.

Brackets pair up and take the direction of their contents and surroundings (UAX #9 rule N0):

(icu:bidi-runs "אבג (abc) אבג")
;; => (#S(BIDI-RUN :START 8 :END 13 :DIRECTION :RTL :LEVEL 1)    ; ") אבג"
;;     #S(BIDI-RUN :START 5 :END 8 :DIRECTION :LTR :LEVEL 2)     ; "abc"
;;     #S(BIDI-RUN :START 0 :END 5 :DIRECTION :RTL :LEVEL 1))    ; "אבג ("

An empty range gives ().

Lines

To reorder one line of a paragraph, pass the whole paragraph as text and the line's bounds as start and end:

(defparameter *text* "abc אבג 123 ד def")
(icu:bidi-runs *text* :start 4 :end 12)    ; the line "אבג 123 "
;; => (#S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)    ; "123"
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1)     ; "אבג "
;;     #S(BIDI-RUN :START 11 :END 12 :DIRECTION :LTR :LEVEL 0))  ; the trailing space

The paragraph is resolved as a whole, so the line's characters get the levels they have in context; then the line is reordered by itself, with trailing white space moved to the paragraph level (here, the final space went to level 0, at the line's right end). This is why lines must be reordered separately: reordering the whole paragraph and then cutting it into lines gives the wrong order.

The line must lie within one paragraph. A line may end just after the paragraph separator. A range that crosses a paragraph boundary signals icu-error:

Characters 1 to 4 span more than one paragraph: a paragraph ends at 3.
Pass the bounds of one line, which lies in one paragraph; find the
paragraphs with BIDI-PARAGRAPHS.  To reorder the whole text, omit START and END.

Text with several paragraphs

Called without start and end on a text with several paragraphs, bidi-runs reorders each paragraph by itself, as one line, and returns the paragraphs' runs one after another in logical order. No run spans two paragraphs, and the result is the same as calling bidi-runs on each paragraph's bounds:

(icu:bidi-runs (format nil "abc~%אבג def") :direction :rtl)
;; => (#S(BIDI-RUN :START 3 :END 4 :DIRECTION :RTL :LEVEL 1)     ; 1st paragraph: the LF,
;;     #S(BIDI-RUN :START 0 :END 3 :DIRECTION :LTR :LEVEL 2)     ;   then "abc"
;;     #S(BIDI-RUN :START 8 :END 11 :DIRECTION :LTR :LEVEL 2)    ; 2nd paragraph: "def",
;;     #S(BIDI-RUN :START 4 :END 8 :DIRECTION :RTL :LEVEL 1))    ;   then "אבג "

Use this for text displayed with one paragraph per line, such as a multi-line label without wrapping. When you wrap lines, call bidi-runs once per line instead.

bidi-levels text &key (direction :auto) (start 0) end → octet vector

The resolved embedding level of each character from start to end. Element i is the level of character start + i. start and end are a line, as for bidi-runs.

(icu:bidi-levels "abc אבג 123 ד def")
;; => #(0 0 0 0 1 1 1 1 2 2 2 1 1 0 0 0 0)
(icu:bidi-levels "abc " :direction :rtl)                  ; => #(2 2 2 1)
(icu:bidi-levels "abc אבג 123 ד def" :start 4 :end 12)   ; => #(1 1 1 1 2 2 2 0)

Levels tell you which characters are right to left (odd levels), for example to mirror them. To map positions between logical and visual order, use the next two functions.

bidi-visual-order text &key (direction :auto) (start 0) end → vector

The indexes of the characters of text from start to end in visual order, left to right on the page: element v is the index into text of the character drawn at visual position v. start, end and direction are as for bidi-runs, which it follows: a line of a paragraph, or without bounds each paragraph reordered by itself, one after another.

(icu:bidi-visual-order "abc אבג 123 def")
;; => #(0 1 2 3 8 9 10 7 6 5 4 11 12 13 14)
(icu:bidi-visual-order *text* :start 4 :end 12)     ; the line from above
;; => #(8 9 10 7 6 5 4 11)

Within a right-to-left run, grapheme clusters are reversed, not characters: each keeps its characters in logical order (UAX #9 rule L3), so a combining mark stays after its base, as a display that draws characters in sequence needs:

(icu:bidi-visual-order (format nil "~C~C~C" (code-char #x5D0) (code-char #x5B8) (code-char #x5D1)))
;; alef, qamats (a vowel point on the alef), bet => #(2 0 1): bet, alef, qamats

The result is a (simple-array fixnum (*)). Use it to write text in visual order for a display without bidi support (see Visual order for plain text).

bidi-visual-positions text &key (direction :auto) (start 0) end → vector

The inverse: the visual position of each character from start to end. Element i is where character start + i is drawn, counting from 0 at the left.

(icu:bidi-visual-positions "abc אבג")   ; => #(0 1 2 3 6 5 4)

Use it to find where a logical position, such as a cursor's, is displayed; together with bidi-visual-order it moves a cursor left and right on screen (see Moving on screen in bidirectional text).

Explicit formatting characters

The algorithm honors Unicode's directional formatting characters, which authors and programs insert to fix the display of mixed text:

Characters	Effect
U+200E LRM, U+200F RLM, U+061C ALM	invisible strong characters, e.g. to keep a trailing "!" with an Arabic word
U+2066 LRI, U+2067 RLI, U+2068 FSI … U+2069 PDI	isolate the enclosed text: it gets its own direction and does not affect its surroundings. Prefer these when inserting a user-supplied string into a message.
U+202A LRE, U+202B RLE, U+202D LRO, U+202E RLO … U+202C PDF	older embeddings and overrides

(icu:bidi-levels (format nil "a~Cb c~Cd" (code-char #x2067) (code-char #x2069)))   ; RLI "b c" PDI
;; => #(0 0 2 2 2 0 0)

The formatting characters stay in the text and appear in the runs. They are default-ignorable: a shaper such as HarfBuzz draws nothing for them.

Mirroring

The algorithm does not change characters. Characters with the Bidi_Mirrored property, such as ( ) [ ] { } < > « », must be drawn mirrored in right-to-left runs, so that an opening parenthesis still opens. HarfBuzz does this itself when it shapes a run with direction :rtl. To display text without a shaper, replace each such character with char-mirror (see Character properties and the Cookbook).

With cl-harfbuzz

Each bidi-run is shaped with its direction:

(dolist (run (icu:bidi-runs text :start line-start :end line-end))
  (hb:shape-text font text :start (icu:bidi-run-start run) :end (icu:bidi-run-end run)
                           :direction (icu:bidi-run-direction run)))

Cut each run further at script boundaries, as the Cookbook shows, since HarfBuzz shapes one script at a time.