Skip to content

Fix Unicode parser correctness across historical and current releases #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mathiasbynens merged 45 commits into from
Oct 25, 2026
Merged

Fix Unicode parser correctness across historical and current releases #110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
d7c5ce9
Fix algorithmic Hangul decomposition mappings
JLHwung Oct 24, 2026
0fcdd58
Parse Unicode property records independently of separator spacing
JLHwung Oct 24, 2026
370bbff
Preserve six-digit code points in legacy property ranges
JLHwung Oct 24, 2026
2f1132a
Honor historical Common defaults for the Script property
JLHwung Oct 24, 2026
51d8c08
Canonicalize historical Script property value names
JLHwung Oct 24, 2026
0a601a1
Canonicalize binary property aliases such as STerm
JLHwung Oct 24, 2026
bcf6560
Preserve Indic_Conjunct_Break as an enumerated property
JLHwung Oct 24, 2026
f573453
Exclude NFKC simple casefold mappings from binary properties
JLHwung Oct 24, 2026
eec7af7
Derive implicit joining types for historical ArabicShaping data
JLHwung Oct 24, 2026
789253e
Infer mapping declaration value types from parsed data
JLHwung Oct 24, 2026
6f920e4
Apply versioned bidi class defaults to unlisted code points
JLHwung Oct 25, 2026
9416471
Generate formal names for algorithmic character ranges
JLHwung Oct 25, 2026
a8b6dbc
Normalize Unicode data line endings before parsing
JLHwung Oct 25, 2026
231c4bb
Canonicalize historical White_space property names
JLHwung Oct 25, 2026
fa99759
Preserve noncharacter records in legacy Unicode properties
JLHwung Oct 25, 2026
7ec9f84
Parse SpecialCasing comments and whitespace safely
JLHwung Oct 25, 2026
5c64d10
Accept flexible separators in emoji and bidi bracket data
JLHwung Oct 25, 2026
51db4f3
Strip inline comments before interpreting Unicode data fields
JLHwung Oct 25, 2026
954fcbb
Declare the generated root index as a default export
JLHwung Oct 25, 2026
77d35e9
Generate the No_Block default for gaps between blocks
JLHwung Oct 25, 2026
6f33a74
Apply the simple titlecase fallback for Unicode 4 and later
JLHwung Oct 25, 2026
e723c07
Preserve explicit Non_Joining records in the default set
JLHwung Oct 25, 2026
44481fb
Ignore whitespace-only lines and indented parser comments
JLHwung Oct 25, 2026
6483160
Split case-folding expansions on whitespace runs
JLHwung Oct 25, 2026
458c85e
Split script extension values on whitespace runs
JLHwung Oct 25, 2026
e82c9ef
Accumulate discontiguous ranges in legacy Unicode blocks
JLHwung Oct 25, 2026
b570656
Generate correction aliases from historical two-field data
JLHwung Oct 25, 2026
dc18217
Strip inline comments before parsing simple case mappings
JLHwung Oct 25, 2026
b1c5a5b
Ignore diagnostic comparisons in legacy Unicode property dumps
JLHwung Oct 25, 2026
5457e06
Skip historical Quick_Check values in normalization properties
JLHwung Oct 25, 2026
666e247
Assign the Unicode 1.1.5 Unified Han range
JLHwung Oct 25, 2026
2e0a911
Let the narrower block entry win in legacy block data
JLHwung Oct 25, 2026
7a7548d
Trim trailing spaces from character names
JLHwung Oct 25, 2026
bd41213
Assign the Unicode 1.1.5 CJK compatibility ideographs
JLHwung Oct 25, 2026
e06c8bb
Assign supplementary private-use planes before Unicode 3.0.1
JLHwung Oct 25, 2026
dbe4426
Generate the missing Unicode 1.1 Han character names
JLHwung Oct 25, 2026
3d2b3e3
Use PropList-3.1.1 for Unicode 3.1.1
JLHwung Oct 25, 2026
2f7fb60
Apply the AL default for unlisted assigned Line_Break code points
JLHwung Oct 25, 2026
b34b766
Add Unicode 3.0 Line_Break data
JLHwung Oct 25, 2026
92562cd
Add composition exclusions for Unicode 3.0 through 4.0.1
JLHwung Oct 25, 2026
43e83a1
Add Bidi mirroring data for Unicode 3.0.1 through 4.0.1
JLHwung Oct 25, 2026
3d6da4e
Add Joining_Type data for Unicode 2.0 through 4.0.1
JLHwung Oct 25, 2026
9813bb1
Reuse PropList-2.1.2 for Unicode 2.1.5
JLHwung Oct 25, 2026
37bd6b5
Preserve ZERO WIDTH JOINER in legacy joining defaults
JLHwung Oct 25, 2026
45d0131
Confine historical data workarounds to the versions that need them
JLHwung Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,314 changes: 2,314 additions & 0 deletions data/10.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,378 changes: 2,378 additions & 0 deletions data/11.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,420 changes: 2,420 additions & 0 deletions data/12.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,419 changes: 2,419 additions & 0 deletions data/12.1.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,466 changes: 2,466 additions & 0 deletions data/13.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,524 changes: 2,524 additions & 0 deletions data/14.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,516 changes: 2,516 additions & 0 deletions data/15.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,518 changes: 2,518 additions & 0 deletions data/15.1.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,579 changes: 2,579 additions & 0 deletions data/16.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,614 changes: 2,614 additions & 0 deletions data/17.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

2,647 changes: 2,647 additions & 0 deletions data/18.0.0-derived-bidi-class.txt
View file
Open in desktop

Large diffs are not rendered by default.

134 changes: 134 additions & 0 deletions data/2.0.14-arabic-shaping.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Unicode; Name; Link; Link Group
0622; MADDAH ON ALEF; R; ALEF
0623; HAMZAH ON ALEF; R; ALEF
0624; HAMZAH ON WAW; R; WAW
0625; HAMZAH UNDER ALEF; R; ALEF
0626; HAMZAH ON YA; D; YA
0627; ALEF; R; ALEF
0628; BAA; D; BAA
0629; TAA MARBUTAH; R; TAA MARBUTAH
062A; TAA; D; BAA
062B; THAA; D; BAA
062C; JEEM; D; HAA
062D; HAA; D; HAA
062E; KHAA; D; HAA
062F; DAL; R; DAL
0630; THAL; R; DAL
0631; RA; R; RA
0632; ZAIN; R; RA
0633; SEEN; D; SEEN
0634; SHEEN; D; SEEN
0635; SAD; D; SAD
0636; DAD; D; SAD
0637; TAH; D; TAH
0638; DHAH; D; TAH
0639; AIN; D; AIN
063A; GHAIN; D; AIN
0640; TATWEEL; C; <no shaping>
0641; FA; D; FA
0642; QAF; D; QAF
0643; CAF; D; CAF
0644; LAM; D; LAM
0645; MEEM; D; MEEM
0646; NOON; D; NOON
0647; HA; D; HA
0648; WAW; R; WAW
0649; ALEF MAQSURAH; R; ALEF MAQSURAH
064A; YA; D; YA
0671; HAMZAT WASL ON ALEF; U; <no shaping>
0672; WAVY HAMZAH ON ALEF; R; ALEF
0673; WAVY HAMZAH UNDER ALEF; R; ALEF
0674; HIGH HAMZAH; U; <no shaping>
0675; HIGH HAMZAH ALEF; R; ALEF
0676; HIGH HAMZAH WAW; R; WAW
0677; HIGH HAMZAH WAW WITH DAMMAH; R; WAW
0678; HIGH HAMZAH YA; D; YA
0679; TAA WITH SMALL TAH; D; BAA
067A; TAA WITH 2 DOTS VERTICAL ABOVE; D; BAA
067B; BAA WITH 2 DOTS VERTICAL BELOW; D; BAA
067C; TAA WITH RING; D; BAA
067D; TAA WITH 3 DOTS ABOVE DOWNWARD; D; BAA
067E; TAA WITH 3 DOTS BELOW; D; BAA
067F; TAA WITH 4 DOTS ABOVE; D; BAA
0680; BAA WITH 4 DOTS BELOW; D; BAA
0681; HAMZAH ON HAA; D; HAA
0682; HAA WITH 2 DOTS VERTICAL ABOVE; D; HAA
0683; HAA WITH MIDDLE 2 DOTS; D; HAA
0684; HAA WITH MIDDLE 2 DOTS VERTICAL; D; HAA
0685; HAA WITH 3 DOTS ABOVE; D; HAA
0686; HAA WITH MIDDLE 3 DOTS DOWNWARD; D; HAA
0687; HAA WITH MIDDLE 4 DOTS; D; HAA
0688; DAL WITH SMALL TAH; R; DAL
0689; DAL WITH RING; R; DAL
068A; DAL WITH DOT BELOW; R; DAL
068B; DAL WITH DOT BELOW AND SMALL TAH; R; DAL
068C; DAL WITH 2 DOTS ABOVE; R; DAL
068D; DAL WITH 2 DOTS BELOW; R; DAL
068E; DAL WITH 3 DOTS ABOVE; R; DAL
068F; DAL WITH 3 DOTS ABOVE DOWNWARD; R; DAL
0690; DAL WITH 4 DOTS ABOVE; R; DAL
0691; RA WITH SMALL TAH; R; RA
0692; RA WITH SMALL V; R; RA
0693; RA WITH RING; R; RA
0694; RA WITH DOT BELOW; R; RA
0695; RA WITH SMALL V BELOW; R; RA
0696; RA WITH DOT BELOW AND DOT ABOVE; R; RA
0697; RA WITH 2 DOTS ABOVE; R; RA
0698; RA WITH 3 DOTS ABOVE; R; RA
0699; RA WITH 4 DOTS ABOVE; R; RA
069A; SEEN WITH DOT BELOW AND DOT ABOVE; D; SEEN
069B; SEEN WITH 3 DOTS BELOW; D; SEEN
069C; SEEN WITH 3 DOTS BELOW AND 3 DOTS ABOVE; D; SEEN
069D; SAD WITH 2 DOTS BELOW; D; SAD
069E; SAD WITH 3 DOTS ABOVE; D; SAD
069F; TAH WITH 3 DOTS ABOVE; D; TAH
06A0; AIN WITH 3 DOTS ABOVE; D; AIN
06A1; DOTLESS FA; D; FA
06A2; FA WITH DOT MOVED BELOW; D; FA
06A3; FA WITH DOT BELOW; D; FA
06A4; FA WITH 3 DOTS ABOVE; D; FA
06A5; FA WITH 3 DOTS BELOW; D; FA
06A6; FA WITH 4 DOTS ABOVE; D; FA
06A7; QAF WITH DOT ABOVE; D; QAF
06A8; QAF WITH 3 DOTS ABOVE; D; QAF
06A9; OPEN CAF; D; GAF
06AA; SWASH CAF; D; SWASH CAF
06AB; CAF WITH RING; D; GAF
06AC; CAF WITH DOT ABOVE; D; CAF
06AD; CAF WITH 3 DOTS ABOVE; D; CAF
06AE; CAF WITH 3 DOTS BELOW; D; CAF
06AF; GAF; D; GAF
06B0; GAF WITH RING; D; GAF
06B1; GAF WITH 2 DOTS ABOVE; D; GAF
06B2; GAF WITH 2 DOTS BELOW; D; GAF
06B3; GAF WITH 2 DOTS VERTICAL BELOW; D; GAF
06B4; GAF WITH 3 DOTS ABOVE; D; GAF
06B5; LAM WITH SMALL V; D; LAM
06B6; LAM WITH DOT ABOVE; D; LAM
06B7; LAM WITH 3 DOTS ABOVE; D; LAM
06BA; DOTLESS NOON; D; NOON
06BB; DOTLESS NOON WITH SMALL TAH; D; NOON
06BC; NOON WITH RING; D; NOON
06BD; NOON WITH 3 DOTS ABOVE; D; NOON
06BE; KNOTTED HA; D; KNOTTED HA
06C0; HAMZAH ON HA; R; TAA MARBUTAH
06C1; HA GOAL; D; HA GOAL
06C2; HAMZAH ON HA GOAL; R; HAMZAH ON HA GOAL
06C3; TAA MARBUTAH GOAL; R; HAMZAH ON HA GOAL
06C4; WAW WITH RING; R; WAW
06C5; WAW WITH BAR; R; WAW
06C6; WAW WITH SMALL V; R; WAW
06C7; WAW WITH DAMMAH; R; WAW
06C8; WAW WITH ALEF ABOVE; R; WAW
06C9; WAW WITH INVERTED SMALL V; R; WAW
06CA; WAW WITH 2 DOTS ABOVE; R; WAW
06CB; WAW WITH 3 DOTS ABOVE; R; WAW
06CC; DOTLESS YA; D; YA
06CD; YA WITH TAIL; R; ALEF MAQSURAH
06CE; YA WITH SMALL V; D; YA
06D0; YA WITH 2 DOTS VERTICAL BELOW; D; YA
06D1; YA WITH 3 DOTS BELOW; D; YA
06D2; YA BARREE; R; YA BARREE
06D3; HAMZAH ON YA BARREE; R; YA BARREE
06D5; AE; U; <no shaping>

134 changes: 134 additions & 0 deletions data/2.1.2-arabic-shaping.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Unicode; Name; Link; Link Group
0622; MADDAH ON ALEF; R; ALEF
0623; HAMZAH ON ALEF; R; ALEF
0624; HAMZAH ON WAW; R; WAW
0625; HAMZAH UNDER ALEF; R; ALEF
0626; HAMZAH ON YA; D; YA
0627; ALEF; R; ALEF
0628; BAA; D; BAA
0629; TAA MARBUTAH; R; TAA MARBUTAH
062A; TAA; D; BAA
062B; THAA; D; BAA
062C; JEEM; D; HAA
062D; HAA; D; HAA
062E; KHAA; D; HAA
062F; DAL; R; DAL
0630; THAL; R; DAL
0631; RA; R; RA
0632; ZAIN; R; RA
0633; SEEN; D; SEEN
0634; SHEEN; D; SEEN
0635; SAD; D; SAD
0636; DAD; D; SAD
0637; TAH; D; TAH
0638; DHAH; D; TAH
0639; AIN; D; AIN
063A; GHAIN; D; AIN
0640; TATWEEL; C; <no shaping>
0641; FA; D; FA
0642; QAF; D; QAF
0643; CAF; D; CAF
0644; LAM; D; LAM
0645; MEEM; D; MEEM
0646; NOON; D; NOON
0647; HA; D; HA
0648; WAW; R; WAW
0649; ALEF MAQSURAH; R; ALEF MAQSURAH
064A; YA; D; YA
0671; HAMZAT WASL ON ALEF; U; <no shaping>
0672; WAVY HAMZAH ON ALEF; R; ALEF
0673; WAVY HAMZAH UNDER ALEF; R; ALEF
0674; HIGH HAMZAH; U; <no shaping>
0675; HIGH HAMZAH ALEF; R; ALEF
0676; HIGH HAMZAH WAW; R; WAW
0677; HIGH HAMZAH WAW WITH DAMMAH; R; WAW
0678; HIGH HAMZAH YA; D; YA
0679; TAA WITH SMALL TAH; D; BAA
067A; TAA WITH 2 DOTS VERTICAL ABOVE; D; BAA
067B; BAA WITH 2 DOTS VERTICAL BELOW; D; BAA
067C; TAA WITH RING; D; BAA
067D; TAA WITH 3 DOTS ABOVE DOWNWARD; D; BAA
067E; TAA WITH 3 DOTS BELOW; D; BAA
067F; TAA WITH 4 DOTS ABOVE; D; BAA
0680; BAA WITH 4 DOTS BELOW; D; BAA
0681; HAMZAH ON HAA; D; HAA
0682; HAA WITH 2 DOTS VERTICAL ABOVE; D; HAA
0683; HAA WITH MIDDLE 2 DOTS; D; HAA
0684; HAA WITH MIDDLE 2 DOTS VERTICAL; D; HAA
0685; HAA WITH 3 DOTS ABOVE; D; HAA
0686; HAA WITH MIDDLE 3 DOTS DOWNWARD; D; HAA
0687; HAA WITH MIDDLE 4 DOTS; D; HAA
0688; DAL WITH SMALL TAH; R; DAL
0689; DAL WITH RING; R; DAL
068A; DAL WITH DOT BELOW; R; DAL
068B; DAL WITH DOT BELOW AND SMALL TAH; R; DAL
068C; DAL WITH 2 DOTS ABOVE; R; DAL
068D; DAL WITH 2 DOTS BELOW; R; DAL
068E; DAL WITH 3 DOTS ABOVE; R; DAL
068F; DAL WITH 3 DOTS ABOVE DOWNWARD; R; DAL
0690; DAL WITH 4 DOTS ABOVE; R; DAL
0691; RA WITH SMALL TAH; R; RA
0692; RA WITH SMALL V; R; RA
0693; RA WITH RING; R; RA
0694; RA WITH DOT BELOW; R; RA
0695; RA WITH SMALL V BELOW; R; RA
0696; RA WITH DOT BELOW AND DOT ABOVE; R; RA
0697; RA WITH 2 DOTS ABOVE; R; RA
0698; RA WITH 3 DOTS ABOVE; R; RA
0699; RA WITH 4 DOTS ABOVE; R; RA
069A; SEEN WITH DOT BELOW AND DOT ABOVE; D; SEEN
069B; SEEN WITH 3 DOTS BELOW; D; SEEN
069C; SEEN WITH 3 DOTS BELOW AND 3 DOTS ABOVE; D; SEEN
069D; SAD WITH 2 DOTS BELOW; D; SAD
069E; SAD WITH 3 DOTS ABOVE; D; SAD
069F; TAH WITH 3 DOTS ABOVE; D; TAH
06A0; AIN WITH 3 DOTS ABOVE; D; AIN
06A1; DOTLESS FA; D; FA
06A2; FA WITH DOT MOVED BELOW; D; FA
06A3; FA WITH DOT BELOW; D; FA
06A4; FA WITH 3 DOTS ABOVE; D; FA
06A5; FA WITH 3 DOTS BELOW; D; FA
06A6; FA WITH 4 DOTS ABOVE; D; FA
06A7; QAF WITH DOT ABOVE; D; QAF
06A8; QAF WITH 3 DOTS ABOVE; D; QAF
06A9; OPEN CAF; D; GAF
06AA; SWASH CAF; D; SWASH CAF
06AB; CAF WITH RING; D; GAF
06AC; CAF WITH DOT ABOVE; D; CAF
06AD; CAF WITH 3 DOTS ABOVE; D; CAF
06AE; CAF WITH 3 DOTS BELOW; D; CAF
06AF; GAF; D; GAF
06B0; GAF WITH RING; D; GAF
06B1; GAF WITH 2 DOTS ABOVE; D; GAF
06B2; GAF WITH 2 DOTS BELOW; D; GAF
06B3; GAF WITH 2 DOTS VERTICAL BELOW; D; GAF
06B4; GAF WITH 3 DOTS ABOVE; D; GAF
06B5; LAM WITH SMALL V; D; LAM
06B6; LAM WITH DOT ABOVE; D; LAM
06B7; LAM WITH 3 DOTS ABOVE; D; LAM
06BA; DOTLESS NOON; D; NOON
06BB; DOTLESS NOON WITH SMALL TAH; D; NOON
06BC; NOON WITH RING; D; NOON
06BD; NOON WITH 3 DOTS ABOVE; D; NOON
06BE; KNOTTED HA; D; KNOTTED HA
06C0; HAMZAH ON HA; R; TAA MARBUTAH
06C1; HA GOAL; D; HA GOAL
06C2; HAMZAH ON HA GOAL; R; HAMZAH ON HA GOAL
06C3; TAA MARBUTAH GOAL; R; HAMZAH ON HA GOAL
06C4; WAW WITH RING; R; WAW
06C5; WAW WITH BAR; R; WAW
06C6; WAW WITH SMALL V; R; WAW
06C7; WAW WITH DAMMAH; R; WAW
06C8; WAW WITH ALEF ABOVE; R; WAW
06C9; WAW WITH INVERTED SMALL V; R; WAW
06CA; WAW WITH 2 DOTS ABOVE; R; WAW
06CB; WAW WITH 3 DOTS ABOVE; R; WAW
06CC; DOTLESS YA; D; YA
06CD; YA WITH TAIL; R; ALEF MAQSURAH
06CE; YA WITH SMALL V; D; YA
06D0; YA WITH 2 DOTS VERTICAL BELOW; D; YA
06D1; YA WITH 3 DOTS BELOW; D; YA
06D2; YA BARREE; R; YA BARREE
06D3; HAMZAH ON YA BARREE; R; YA BARREE
06D5; AE; U; <no shaping>

Loading
Loading