Repository navigation
Inflection-215 Minimize ICU4C data and code footprint via icu_filters.json and UCONFIG flags - #216
Conversation
…rties and add UCONFIG flags
grhoten
left a comment
There was a problem hiding this comment.
Dedicated unit test verifying that every supported locale in
LocaleUtils::getSupportedLocaleList()has all ICU data and APIs required by Inflection (ures_open,ulocdata_getExemplarSet,ListFormatter,RuleBasedNumberFormat(URBNF_SPELLOUT),NumberFormatter,SimpleDateFormat,PluralRules,BreakIterator::createWordInstance,Normalizer2,CaseMap, andubidi_*).
I think your AI agent is hallucinating. This code does not use ubidi_*, nor does the new test test it. The changes seem fine otherwise.
I am curious to see what data is left, but it's not a blocking request for merging.
| git clone --depth 1 --branch release-${ICU_MAJOR}.${ICU_MINOR} https://github.com/unicode-org/icu.git | ||
| cd icu/icu4c/source | ||
| ICU_DATA_FILTER_FILE="${GITHUB_WORKSPACE}/icu_filters.json" \ | ||
| CPPFLAGS="-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_IDNA=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_USE_ML_PHRASE_BREAKING=1 -DUCONFIG_NO_FILTERED_BREAK_ITERATION=1 -DUCONFIG_NO_SERVICE=1" \ |
There was a problem hiding this comment.
Oops! I made a bad suggestion. I recommend removing the -DUCONFIG_USE_ML_PHRASE_BREAKING=1 part.
Fixes #215
Summary
Provides an ICU Data Build Tool filter configuration template (
icu_filters.json.in) paired with a generator script (tools/scripts/generate_icu_filters.py), along withUCONFIG_*compile-time flags to minimize both ICU4C's data and code footprint for Unicode Inflection.Changes
icu_filters.json.in+tools/scripts/generate_icu_filters.py+.gitignore:localeFilter.includelistdynamically fromresources/org/unicode/inflection/locale/supported-locales.propertiesso there is a single source of truth for supported locales.coll_tree,coll_ucadata,conversion_mappings,translit,zone_tree,zone_supplemental,unit_tree,lang_tree,region_tree,curr_supplemental,confusables,stringprep,unames, and non-word break iterator rules/dictionaries/LSTM/adaboost models) and retains only the resources required by Inflection..github/workflows/cmake-multi-platform.yml:icu_filters.jsonviatools/scripts/generate_icu_filters.pyand builds ICU 79.1 on bothubuntu-latestandmacos-latestwithICU_DATA_FILTER_FILEandCPPFLAGS="-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_IDNA=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_USE_ML_PHRASE_BREAKING=1 -DUCONFIG_NO_FILTERED_BREAK_ITERATION=1 -DUCONFIG_NO_SERVICE=1".NPROC=$(getconf _NPROCESSORS_ONLN)on both Linux and macOS.test/src/inflection/util/LocaleUtilsTest.cpp(LocaleUtilsTest#testMinimalICUData):LocaleUtils::getSupportedLocaleList()has all ICU data and APIs required by Inflection (ures_open,ulocdata_getExemplarSet,ListFormatter,RuleBasedNumberFormat(URBNF_SPELLOUT),NumberFormatter,SimpleDateFormat,PluralRules,BreakIterator::createWordInstance,Normalizer2,CaseMap, andubidi_*).README.md,documents/how_to_add_new_language.md,documents/how_to_update_icu.md):icu_filters.jsonand build a minimal ICU4C with bothICU_DATA_FILTER_FILEandUCONFIG_*flags.Size Comparison (ICU 79.1 on Linux x86_64)
UCONFIG_*(Stripped)libicudata.so.79.1libicui18n.so.79.1libicuuc.so.79.1data+i18n+uc)