Repository navigation
Fix Unicode parser correctness across historical and current releases - #110
Merged
mathiasbynens merged 45 commits intoOct 25, 2026
Merged
Conversation
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Preserve historical identity mappings for titlecase digraphs while honoring the standardized uppercase fallback for null titlecase fields. Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Normalize line guards before splitting fields and recognize padded blank lines when skipping legacy property sections. Check every parser against padded input for all supported Unicode versions. Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Assisted-By: GPT-6 Issue: node-unicode#110
Retain all Specials records, including U+FEFF, while preserving the overlaps present in the historical source files. Assisted-By: GPT-6 Issue: node-unicode#110
Register and bundle NameAliases.txt for Unicode 5.0 through 6.0, and interpret records without a type field as correction aliases. Assisted-By: GPT-6 Issue: node-unicode#110
Keep comments out of the final UnicodeData field so empty titlecase mappings retain their defaults. Check comment invariance across every parser and Unicode version. Assisted-By: GPT-6 Issue: node-unicode#110
Stop collecting property members at Discrepancy sections and resume at the next property heading. This removes diagnostic-only code points from Unicode 2.1.8 Math, Alphabetic, and Diacritic sets. Assisted-By: GPT-6 Issue: node-unicode#110
Unicode 3.1.0 through 4.0.0 record Quick_Check values in DerivedNormalizationProperties.txt as `NFC_NO`, `NFC_MAYBE`, and similar names instead of `NFC_QC; N`. Skip them like their modern equivalents, so they no longer appear as binary properties. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
UnicodeData-1.1.5 lists only U+4E00 as `<CJK IDEOGRAPH REPRESENTATIVE>`, so the gap logic reported 4E01..9FA5 as unassigned. The Unicode 1.1 Han cross-reference table (CJKXREF.TXT) assigns the whole 4E00..9FA5 range, so give the representative record that range. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
The Unicode 2.0 blocks file lists FE70..FEFF as Arabic Presentation Forms-B and FEFF..FEFF as Specials, so U+FEFF belonged to both blocks in Unicode 2.0.14 through 2.1.8. Apply narrower ranges last so the more specific Specials entry wins, matching the corrected Unicode 2.1.9 file. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
UnicodeData-4.0.0 has trailing spaces after the names of U+02AE and U+0615, which leaked into the generated name keys. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
UnicodeData-1.1.5 omits F900..FA2D entirely, so the gap logic reported them as unassigned. The Unicode 1.1 Han cross-reference table (CJKXREF.TXT) lists the whole range, and Unicode 2.0 lists it as Lo. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
UnicodeData from Unicode 2.0 through 3.0.0 lists the Private Use High Surrogates but not the planes 15 and 16 private-use characters their surrogate pairs encode, so those planes were reported as unassigned. Unicode 3.0.1 added the ranges to the file, and its documentation describes them as the characters those surrogate pairs represent. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
UnicodeData-1.1.5 uses one representative record and omits the compatibility ideographs. Generate individual formal names for the 20,902 unified and 302 compatibility ideographs listed in CJKXREF. Assisted-By: GPT-6 Issue: node-unicode#110
Unicode 3.1.1 published an updated PropList-3.1.1.txt, but the resources reused PropList-3.1.0.txt. The update removes the mathematical digits 1D7CE..1D7FF from Hex_Digit and adds ASCII_Hex_Digit, so Unicode 3.1.1 had the wrong Hex_Digit set and no ASCII_Hex_Digit. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
Before Unicode 4.0, LineBreak.txt gives unlisted assigned characters the value AL, and only unassigned code points XX. The parser gave every unlisted code point XX, so the private-use code points in Unicode 3.1.0 through 3.2.0 had Line_Break=Unknown instead of Alphabetic. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
Unicode 3.0.0 published LineBreak-5.txt, and 3.0.1 did not update it. That file lists ranges the way UnicodeData does, with separate First and Last records, so expand those ranges in the parser. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
Unicode published CompositionExclusions files from 3.0.0 on, but the resources only included them from 4.1.0. Unicode 3.1.1 and 4.0.1 did not update the file, so they reuse the previous release's version. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
Unicode published BidiMirroring files from 3.0.1 on, but the resources only included them from 4.1.0. Unicode 3.1.0, 3.1.1, and 4.0.1 did not update the file, so they reuse the previous release's version. The 3.2 and 4.0 files map U+2A2D to U+2A2C, which itself maps to U+2A2B. The writer assumed every mapping had a matching reverse mapping, so it would have dropped both one-way mappings in that chain. It now emits those mappings separately; output for symmetric data is unchanged. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
Unicode published ArabicShaping from 2.0 on and DerivedJoiningType from 3.1.0 on, but the resources only included ArabicShaping from 3.2.0 and DerivedJoiningType from 4.1.0. Add ArabicShaping-1 through -4, reusing each file for the releases that did not update it, and DerivedJoiningType for 3.1.1 through 4.0.1. DerivedJoiningType-3.1.0 is left out: it lists only C, D, and R, without the Transparent code points or ZERO WIDTH JOINER, so 3.1.0 derives its values from ArabicShaping-3 instead. Where both sources exist, the derived files match the ArabicShaping fallback exactly. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
The Unicode 2.1.5 update did not ship a PropList, so the 2.1.2 file still applied, but the resources left 2.1.5 without binary properties. Reuse it the same way 2.1.5 already reuses the Unicode 2.0 blocks file. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
ArabicShaping files before Unicode 3.1.1 omit joining controls. The fallback excluded ZWJ from Transparent but placed it in Non_Joining, despite its join-causing semantics. Seed Join_Causing with U+200D and allow explicit shaping records to override the defaults. Cover both joining controls and neighboring joining types across historical and derived-data releases. Only U+200D changes in the eight releases from Unicode 2.0.14 through 3.1.0. Assisted-By: GPT-6 Issue: node-unicode#110
Add utils.isVersionBefore and guard each workaround for historical data files with the range of releases it applies to, so parsing current versions skips them: - case-insensitive property names such as White_space: before 3.2.0 - FNC and NFC_NO-style Quick_Check records: before 4.0.1 - uppercase Scripts.txt names such as LATIN: before 4.0.1 - trailing spaces in UnicodeData names: before 4.0.1 - two-field NameAliases records: before 6.1.0 - the LineBreak AL default for assigned code points: before 4.0.0 - First/Last range records in LineBreak: before 3.1.0 Every parser produces identical output for every version in data/. Assisted-By: Claude Opus 5.5 Issue: node-unicode#110
mathiasbynens
force-pushed
the
fix/unicode-parser-correctness
branch
from
October 25, 2026 06:46
9948735 to
45d0131
Compare
Collaborator
|
Woah! Great set of fixes. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: This PR is created with the help of LLM. The PR description is generated by LLM and edited by myself.
Summary
Several parsers omit implicit values, misinterpret historical formats, or generate incomplete property data. This PR corrects those behaviors across all 35 supported Unicode releases, from 1.1.5 to 18.0.0. It also adds the UCD files that older releases published but the resource list never included.
Indic_Conjunct_Breakas an enumerated property, and exclude mapping and Quick_Check records from binary properties.Historical workarounds are restricted to the versions that require them, so parsing current releases skips them. Each commit is one self-contained fix with its own regression test. Reviewing commit by commit should be easier than reviewing the combined diff.
Changes to current output
Most fixes affect historical releases only, but some change the generated data for current versions too. I generated the Unicode 18.0.0 package from
mainand from this branch. These are the only differences:mainBidi_Class@missingdefaults fromDerivedBidiClass.txt: L, R, AL, BN, or ETDecomposition_Mapping/CanonicalNamesCJK Ideograph Extension A, plus surrogates and private useCJK UNIFIED IDEOGRAPH-3400; code points without a Name are omittedBlockNo_BlockvalueNo_Blockcovers the gaps between blocksIndic_Conjunct_BreakBinary_Property/InCBsetConsonant,Extend,Linker, andNoneBinary_Property/NFKC_Simple_Casefoldindex.d.mtsindex.mjsdoesn't provideindex.mjshasHistorical fixes
Defaults
Common, becauseUnknowndidn't exist yet.ArabicShaping.txtderive implicitTransparentas Mn + Cf minus the joining controls. ZERO WIDTH JOINER counts asJoin_Causing.AL, notXX.Aliases and names
White_space,STerm, and uppercase script names likeLATIN.NFC_NOandNFC_MAYBEand theFNCalias from Unicode 3.1.0 through 4.0.0 are no longer binary properties.NameAliases.txtrecords from before Unicode 6.1 are treated as corrections.Omitted ranges and legacy formats
Private_Use, matching the Unicode 3.0.1 documentation.LineBreak.txtlists ranges as First and Last records, which are now expanded.Bidi mirroring
The writer assumed every mapping had a matching reverse mapping. The 3.2 and 4.0 data contain two that don't, U+2A2D→U+2A2C and U+2A2E→U+2A2D, which it would have dropped. One-way mappings are now preserved. Output for symmetric data, including every current release, is byte-for-byte unchanged.
Missing resources added
A release that didn't republish a file reuses the previous release's copy, as 3.0.1 already did for Blocks.
derived-bidi-classname-aliasesarabic-shapingderived-joining-typecomposition-exclusionsbidi-mirroringline-breakpropertiesUnicode 3.1.1 now uses the published
PropList-3.1.1.txtinstead of the 3.1.0 file. That corrects Hex_Digit and adds ASCII_Hex_Digit.Notes for reviewers
utils.isVersionBeforehelper restricts each historical workaround to the releases that require it. Adding the guards left every parser's output identical for every release.data/. The code links to it as the source for the 1.1.5 ranges instead.