Skip to content

Fix Unicode parser correctness across historical and current releases - #110

Merged
mathiasbynens merged 45 commits into
node-unicode:mainfrom
JLHwung:fix/unicode-parser-correctness
Oct 25, 2026
Merged

mathiasbynens merged 45 commits into
node-unicode:mainfrom
JLHwung:fix/unicode-parser-correctness

Conversation

@JLHwung

@JLHwung JLHwung commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Disclaimer: This PR is created with the help of LLM. The PR description is generated by LLM and edited by myself.

Summary

Several parsers omit implicit values, misinterpret historical formats, or generate incomplete property data. This PR corrects those behaviors across all 35 supported Unicode releases, from 1.1.5 to 18.0.0. It also adds the UCD files that older releases published but the resource list never included.

  • Apply version-specific defaults for bidi classes, scripts, joining types, blocks, and line breaks.
  • Generate algorithmic character names and Hangul decompositions, and restore omitted historical Han and private-use ranges.
  • Preserve Indic_Conjunct_Break as an enumerated property, and exclude mapping and Quick_Check records from binary properties.
  • Handle comments, whitespace, line endings, aliases, and legacy range formats correctly.
  • Add missing historical resources, and preserve one-way bidi mirroring mappings.
  • Correct the generated TypeScript declarations for mapping values and default exports.

Historical workarounds are restricted to the versions that require them, so parsing current releases skips them. Each commit is one self-contained fix with its own regression test. Reviewing commit by commit should be easier than reviewing the combined diff.

Changes to current output

Most fixes affect historical releases only, but some change the generated data for current versions too. I generated the Unicode 18.0.0 package from main and from this branch. These are the only differences:

Area main This branch
Bidi_Class 801,723 unlisted code points have no value Unlisted code points get the @missing defaults from DerivedBidiClass.txt: L, R, AL, BN, or ET
Decomposition_Mapping/Canonical 2,081 mappings, with no Hangul syllables 13,253 mappings, adding the 11,172 algorithmic Hangul decompositions
Names Range placeholders such as CJK Ideograph Extension A, plus surrogates and private use Formal names such as CJK UNIFIED IDEOGRAPH-3400; code points without a Name are omitted
Block No No_Block value No_Block covers the gaps between blocks
Indic_Conjunct_Break Generated as a Binary_Property/InCB set Enumerated property with the values Consonant, Extend, Linker, and None
Binary_Property/NFKC_Simple_Casefold Generated as a binary property Excluded, because it's a mapping
index.d.mts Declares named exports that index.mjs doesn't provide Declares the default export that index.mjs has

Historical fixes

Defaults

  • Before Unicode 5.0, Script defaults to Common, because Unknown didn't exist yet.
  • Unicode 3.x applies the UAX #‌9 Bidi_Class defaults to unassigned code points.
  • Releases with only ArabicShaping.txt derive implicit Transparent as Mn + Cf minus the joining controls. ZERO WIDTH JOINER counts as Join_Causing.
  • Before Unicode 4.0, unlisted assigned code points get Line_Break AL, not XX.
  • From Unicode 4.0 on, an empty simple titlecase mapping falls back to the uppercase mapping. Earlier releases keep the identity mapping of titlecase digraphs.

Aliases and names

  • Historical spellings are canonicalized, such as White_space, STerm, and uppercase script names like LATIN.
  • The Quick_Check records NFC_NO and NFC_MAYBE and the FNC alias from Unicode 3.1.0 through 4.0.0 are no longer binary properties.
  • Two-field NameAliases.txt records from before Unicode 6.1 are treated as corrections.
  • Trailing spaces in two Unicode 4.0.0 names, for U+02AE and U+0615, are trimmed.

Omitted ranges and legacy formats

  • Unicode 1.1.5 lists U+4E00 only as a "representative" record and omits the compatibility ideographs. Following CJKXREF.TXT, 4E00..9FA5 and F900..FA2D are now assigned and named.
  • Unicode 2.0 through 3.0.0 omit the plane 15 and 16 private-use characters that their private-use high surrogates encode. They're now restored as Private_Use, matching the Unicode 3.0.1 documentation.
  • Old PropList files keep six-digit ranges and noncharacter records, and the 2.1.8 "Discrepancy" sections are skipped.
  • Discontiguous legacy blocks are accumulated. Where the Unicode 2.0 blocks file overlaps at U+FEFF, the narrower Specials entry wins, as in the corrected 2.1.9 file.
  • The Unicode 3.0 LineBreak.txt lists ranges as First and Last records, which are now expanded.

Bidi mirroring

The writer assumed every mapping had a matching reverse mapping. The 3.2 and 4.0 data contain two that don't, U+2A2D→U+2A2C and U+2A2E→U+2A2D, which it would have dropped. One-way mappings are now preserved. Output for symmetric data, including every current release, is byte-for-byte unchanged.

Missing resources added

A release that didn't republish a file reuses the previous release's copy, as 3.0.1 already did for Blocks.

Type Versions added
derived-bidi-class 3.2.0 to 18.0.0
name-aliases 5.0.0 to 6.0.0
arabic-shaping 2.0.14 to 3.1.1
derived-joining-type 3.1.1 to 4.0.1
composition-exclusions 3.0.0 to 4.0.1
bidi-mirroring 3.0.1 to 4.0.1
line-break 3.0.0, 3.0.1
properties 2.1.5, reusing PropList-2.1.2

Unicode 3.1.1 now uses the published PropList-3.1.1.txt instead of the 3.1.0 file. That corrects Hex_Digit and adds ASCII_Hex_Digit.

Notes for reviewers

  • Version guards. The new utils.isVersionBefore helper restricts each historical workaround to the releases that require it. Adding the guards left every parser's output identical for every release.
  • CJKXREF.TXT is not vendored. Its license forbids redistribution, and this package publishes data/. The code links to it as the source for the 1.1.5 ranges instead.
  • DerivedJoiningType-3.1.0 is not used. It omits Transparent and ZERO WIDTH JOINER, so 3.1.0 derives Joining_Type from ArabicShaping instead. From 3.1.1 to 4.0.1, the derived files and the ArabicShaping result match exactly.

Preserve historical identity mappings for titlecase digraphs while honoring the standardized uppercase fallback for null titlecase fields.

Assisted-By: GPT-6
Issue: node-unicode#110
Normalize line guards before splitting fields and recognize padded blank lines when skipping legacy property sections. Check every parser against padded input for all supported Unicode versions.

Assisted-By: GPT-6
Issue: node-unicode#110
Retain all Specials records, including U+FEFF, while preserving the overlaps present in the historical source files.

Assisted-By: GPT-6
Issue: node-unicode#110
Register and bundle NameAliases.txt for Unicode 5.0 through 6.0, and interpret records without a type field as correction aliases.

Assisted-By: GPT-6
Issue: node-unicode#110
Keep comments out of the final UnicodeData field so empty titlecase mappings retain their defaults. Check comment invariance across every parser and Unicode version.

Assisted-By: GPT-6
Issue: node-unicode#110
Stop collecting property members at Discrepancy sections and resume at the next property heading. This removes diagnostic-only code points from Unicode 2.1.8 Math, Alphabetic, and Diacritic sets.

Assisted-By: GPT-6
Issue: node-unicode#110
Unicode 3.1.0 through 4.0.0 record Quick_Check values in
DerivedNormalizationProperties.txt as `NFC_NO`, `NFC_MAYBE`, and similar
names instead of `NFC_QC; N`. Skip them like their modern equivalents, so
they no longer appear as binary properties.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
UnicodeData-1.1.5 lists only U+4E00 as `<CJK IDEOGRAPH REPRESENTATIVE>`,
so the gap logic reported 4E01..9FA5 as unassigned. The Unicode 1.1 Han
cross-reference table (CJKXREF.TXT) assigns the whole 4E00..9FA5 range,
so give the representative record that range.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
The Unicode 2.0 blocks file lists FE70..FEFF as Arabic Presentation
Forms-B and FEFF..FEFF as Specials, so U+FEFF belonged to both blocks in
Unicode 2.0.14 through 2.1.8. Apply narrower ranges last so the more
specific Specials entry wins, matching the corrected Unicode 2.1.9 file.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
UnicodeData-4.0.0 has trailing spaces after the names of U+02AE and
U+0615, which leaked into the generated name keys.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
UnicodeData-1.1.5 omits F900..FA2D entirely, so the gap logic reported
them as unassigned. The Unicode 1.1 Han cross-reference table
(CJKXREF.TXT) lists the whole range, and Unicode 2.0 lists it as Lo.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
UnicodeData from Unicode 2.0 through 3.0.0 lists the Private Use High
Surrogates but not the planes 15 and 16 private-use characters their
surrogate pairs encode, so those planes were reported as unassigned.
Unicode 3.0.1 added the ranges to the file, and its documentation
describes them as the characters those surrogate pairs represent.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
UnicodeData-1.1.5 uses one representative record and omits the compatibility ideographs. Generate individual formal names for the 20,902 unified and 302 compatibility ideographs listed in CJKXREF.

Assisted-By: GPT-6
Issue: node-unicode#110
Unicode 3.1.1 published an updated PropList-3.1.1.txt, but the resources
reused PropList-3.1.0.txt. The update removes the mathematical digits
1D7CE..1D7FF from Hex_Digit and adds ASCII_Hex_Digit, so Unicode 3.1.1
had the wrong Hex_Digit set and no ASCII_Hex_Digit.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
Before Unicode 4.0, LineBreak.txt gives unlisted assigned characters the
value AL, and only unassigned code points XX. The parser gave every
unlisted code point XX, so the private-use code points in Unicode 3.1.0
through 3.2.0 had Line_Break=Unknown instead of Alphabetic.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
Unicode 3.0.0 published LineBreak-5.txt, and 3.0.1 did not update it.
That file lists ranges the way UnicodeData does, with separate First and
Last records, so expand those ranges in the parser.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
Unicode published CompositionExclusions files from 3.0.0 on, but the
resources only included them from 4.1.0. Unicode 3.1.1 and 4.0.1 did
not update the file, so they reuse the previous release's version.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
Unicode published BidiMirroring files from 3.0.1 on, but the resources
only included them from 4.1.0. Unicode 3.1.0, 3.1.1, and 4.0.1 did not
update the file, so they reuse the previous release's version.

The 3.2 and 4.0 files map U+2A2D to U+2A2C, which itself maps to U+2A2B.
The writer assumed every mapping had a matching reverse mapping, so it
would have dropped both one-way mappings in that chain. It now emits
those mappings separately; output for symmetric data is unchanged.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
Unicode published ArabicShaping from 2.0 on and DerivedJoiningType from
3.1.0 on, but the resources only included ArabicShaping from 3.2.0 and
DerivedJoiningType from 4.1.0. Add ArabicShaping-1 through -4, reusing
each file for the releases that did not update it, and DerivedJoiningType
for 3.1.1 through 4.0.1.

DerivedJoiningType-3.1.0 is left out: it lists only C, D, and R, without
the Transparent code points or ZERO WIDTH JOINER, so 3.1.0 derives its
values from ArabicShaping-3 instead. Where both sources exist, the
derived files match the ArabicShaping fallback exactly.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
The Unicode 2.1.5 update did not ship a PropList, so the 2.1.2 file still
applied, but the resources left 2.1.5 without binary properties. Reuse
it the same way 2.1.5 already reuses the Unicode 2.0 blocks file.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
ArabicShaping files before Unicode 3.1.1 omit joining controls. The fallback excluded ZWJ from Transparent but placed it in Non_Joining, despite its join-causing semantics. Seed Join_Causing with U+200D and allow explicit shaping records to override the defaults.

Cover both joining controls and neighboring joining types across historical and derived-data releases. Only U+200D changes in the eight releases from Unicode 2.0.14 through 3.1.0.

Assisted-By: GPT-6
Issue: node-unicode#110
Add utils.isVersionBefore and guard each workaround for historical data
files with the range of releases it applies to, so parsing current
versions skips them:

- case-insensitive property names such as White_space: before 3.2.0
- FNC and NFC_NO-style Quick_Check records: before 4.0.1
- uppercase Scripts.txt names such as LATIN: before 4.0.1
- trailing spaces in UnicodeData names: before 4.0.1
- two-field NameAliases records: before 6.1.0
- the LineBreak AL default for assigned code points: before 4.0.0
- First/Last range records in LineBreak: before 3.1.0

Every parser produces identical output for every version in data/.

Assisted-By: Claude Opus 5.5
Issue: node-unicode#110
@mathiasbynens
mathiasbynens force-pushed the fix/unicode-parser-correctness branch from 9948735 to 45d0131 Compare October 25, 2026 06:46
@mathiasbynens
mathiasbynens merged commit 45d0131 into node-unicode:main Oct 25, 2026
@mathiasbynens

Copy link
Copy Markdown
Collaborator

Woah! Great set of fixes. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants