Skip to content

Inflection-215 Minimize ICU4C data and code footprint via icu_filters.json and UCONFIG flags - #216

Merged
nciric merged 2 commits into
unicode-org:mainfrom
nciric:Inflection-215-icu-data-filter
Oct 24, 2026
Merged

nciric merged 2 commits into
unicode-org:mainfrom
nciric:Inflection-215-icu-data-filter

Conversation

@nciric

@nciric nciric commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #215

Summary

Provides an ICU Data Build Tool filter configuration template (icu_filters.json.in) paired with a generator script (tools/scripts/generate_icu_filters.py), along with UCONFIG_* compile-time flags to minimize both ICU4C's data and code footprint for Unicode Inflection.

Changes

  1. icu_filters.json.in + tools/scripts/generate_icu_filters.py + .gitignore:
    • Populates localeFilter.includelist dynamically from resources/org/unicode/inflection/locale/supported-locales.properties so there is a single source of truth for supported locales.
    • Strips unused ICU data trees (coll_tree, coll_ucadata, conversion_mappings, translit, zone_tree, zone_supplemental, unit_tree, lang_tree, region_tree, curr_supplemental, confusables, stringprep, unames, and non-word break iterator rules/dictionaries/LSTM/adaboost models) and retains only the resources required by Inflection.
  2. .github/workflows/cmake-multi-platform.yml:
    • Generates icu_filters.json via tools/scripts/generate_icu_filters.py and builds ICU 79.1 on both ubuntu-latest and macos-latest with ICU_DATA_FILTER_FILE and CPPFLAGS="-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_IDNA=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_USE_ML_PHRASE_BREAKING=1 -DUCONFIG_NO_FILTERED_BREAK_ITERATION=1 -DUCONFIG_NO_SERVICE=1".
    • Uses NPROC=$(getconf _NPROCESSORS_ONLN) on both Linux and macOS.
  3. test/src/inflection/util/LocaleUtilsTest.cpp (LocaleUtilsTest#testMinimalICUData):
    • Dedicated unit test verifying that every supported locale in LocaleUtils::getSupportedLocaleList() has all ICU data and APIs required by Inflection (ures_open, ulocdata_getExemplarSet, ListFormatter, RuleBasedNumberFormat(URBNF_SPELLOUT), NumberFormatter, SimpleDateFormat, PluralRules, BreakIterator::createWordInstance, Normalizer2, CaseMap, and ubidi_*).
  4. Documentation (README.md, documents/how_to_add_new_language.md, documents/how_to_update_icu.md):
    • Documents how to generate icu_filters.json and build a minimal ICU4C with both ICU_DATA_FILTER_FILE and UCONFIG_* flags.

Size Comparison (ICU 79.1 on Linux x86_64)

Library Default Unfiltered (Stripped) Filtered + UCONFIG_* (Stripped) Reduction
libicudata.so.79.1 ~32.3 MB 1.62 MiB (~310 KB compressed) -95.0%
libicui18n.so.79.1 3.52 MiB 2.86 MiB -18.8%
libicuuc.so.79.1 2.19 MiB 1.98 MiB -9.6%
Total (data + i18n + uc) ~38.0 MiB 6.46 MiB -83.0%

@nciric
nciric requested a review from grhoten October 23, 2026 14:58
Comment thread icu_filters.json Outdated
Comment thread .github/workflows/cmake-multi-platform.yml
Comment thread .github/workflows/cmake-multi-platform.yml Outdated
@nciric nciric changed the title Inflection-215 Add ICU data filter configuration and verify in CI Inflection-215 Minimize ICU4C data and code footprint via icu_filters.json and UCONFIG flags Oct 23, 2026
@nciric nciric self-assigned this Oct 23, 2026

@grhoten grhoten left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dedicated unit test verifying that every supported locale in LocaleUtils::getSupportedLocaleList() has all ICU data and APIs required by Inflection (ures_open, ulocdata_getExemplarSet, ListFormatter, RuleBasedNumberFormat(URBNF_SPELLOUT), NumberFormatter, SimpleDateFormat, PluralRules, BreakIterator::createWordInstance, Normalizer2, CaseMap, and ubidi_*).

I think your AI agent is hallucinating. This code does not use ubidi_*, nor does the new test test it. The changes seem fine otherwise.

I am curious to see what data is left, but it's not a blocking request for merging.

git clone --depth 1 --branch release-${ICU_MAJOR}.${ICU_MINOR} https://github.com/unicode-org/icu.git
cd icu/icu4c/source
ICU_DATA_FILTER_FILE="${GITHUB_WORKSPACE}/icu_filters.json" \
CPPFLAGS="-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_IDNA=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_USE_ML_PHRASE_BREAKING=1 -DUCONFIG_NO_FILTERED_BREAK_ITERATION=1 -DUCONFIG_NO_SERVICE=1" \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! I made a bad suggestion. I recommend removing the -DUCONFIG_USE_ML_PHRASE_BREAKING=1 part.

@nciric nciric closed this Oct 24, 2026
@nciric nciric reopened this Oct 24, 2026
@nciric
nciric merged commit 9e4f625 into unicode-org:main Oct 24, 2026
19 checks passed
@nciric
nciric deleted the Inflection-215-icu-data-filter branch October 24, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minimize ICU4C data and code footprint via icu_filters.json and UCONFIG flags

2 participants