gh-152905: Decode LC_TIME items in nl_langinfo() from glibc wide data

[serhiy-storchaka]

On glibc, decode the LC_TIME items from the wide (_NL_W*) locale data, so the result no longer depends on the LC_CTYPE encoding.

The wide constant is always _NL_W + the narrow name, so it is filled into langinfo_constants[] by token pasting — one scan yields both the item and its wide form. ERA has no wide counterpart and keeps the narrow path.

_{🤖 Generated with Claude Code}

• Issue: locale.nl_langinfo(): decode LC_TIME items from glibc wide locale data #152905

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33518029 | 📁 Comparing d338d0b against main (31864bd)

  🔍 Preview build  

66 files changed · + 1 added · ± 65 modified

+ Added

• library/asyncio-tools.html

± Modified

• deprecations/index.html
• deprecations/pending-removal-in-3.21.html
• extending/first-extension-module.html
• extending/newtypes_tutorial.html
• faq/programming.html
• howto/descriptor.html
• library/ast.html
• library/asyncio-graph.html
• library/asyncio.html
• library/codecs.html
• and 55 more...

[vstinner]

On my Fedora 44, this change is not only technical, it does actually change nl_langinfo() output on multiple locales.

Some examples.

• Locale ast_ES.iso885915: MON_10: "d'ochobre" => MON_10: 'd’ochobre' (different quote: U+0027 => U+2019)
• Locale br_FR.iso885915@euro: D_T_FMT: "D'ar %A %d a viz %B %Y %T" => 'Dʼar %A %d a viz %B %Y %T' (different quote)
• Locale es_ES.iso885915@euro: AM_STR: 'a.\xa0m.' => 'a.\u202fm.'
• Locale oc_FR.iso88591: MON_4: "d'abril" => 'd’abril' (different quote)
• Locale ro_RO.iso88592: DAY_3: 'marţi' => DAY_3: 'marți' (U+0163 => U+021b)
• Locale yi_US: ABDAY_2: "מאָנ'" => "מאָנ'" (U+05d0 U+05b8 => U+fb2f)

I wrote this script to dump all nl_langinfo() values of all locales on Linux:

import locale
import subprocess

def get_all_locales():
    cmd = ['locale', '-a']
    proc = subprocess.run(cmd, stdout=subprocess.PIPE, text=True)
    stdout = proc.stdout
    return stdout.splitlines()

langinfo_constants = [
    "DAY_1",
    "DAY_2",
    "DAY_3",
    "DAY_4",
    "DAY_5",
    "DAY_6",
    "DAY_7",
    "ABDAY_1",
    "ABDAY_2",
    "ABDAY_3",
    "ABDAY_4",
    "ABDAY_5",
    "ABDAY_6",
    "ABDAY_7",
    "MON_1",
    "MON_2",
    "MON_3",
    "MON_4",
    "MON_5",
    "MON_6",
    "MON_7",
    "MON_8",
    "MON_9",
    "MON_10",
    "MON_11",
    "MON_12",
    "ABMON_1",
    "ABMON_2",
    "ABMON_3",
    "ABMON_4",
    "ABMON_5",
    "ABMON_6",
    "ABMON_7",
    "ABMON_8",
    "ABMON_9",
    "ABMON_10",
    "ABMON_11",
    "ABMON_12",
    "RADIXCHAR",
    "THOUSEP",
    "CRNCYSTR",
    "D_T_FMT",
    "D_FMT",
    "T_FMT",
    "AM_STR",
    "PM_STR",
    "CODESET",
    "T_FMT_AMPM",
    "ERA",
    "ERA_D_FMT",
    "ERA_D_T_FMT",
    "ERA_T_FMT",
    "ALT_DIGITS",
    "YESEXPR",
    "NOEXPR",
    "_DATE_FMT",
]
langinfo_constants = [
    name
    for name in langinfo_constants
    if hasattr(locale, name)
]
langinfo_constants.sort()

all_locales = get_all_locales()
all_locales.sort()

for loc in all_locales:
    title = f"Locale {loc}"
    print(title)
    print("=" * len(title))
    print()

    locale.setlocale(locale.LC_ALL, loc)

    for name in langinfo_constants:
        key = getattr(locale, name)
        value = locale.nl_langinfo(key)
        print(f'{name}: {value!r}')
    print()

print(f"Total: nl_langinfo() values: {len(langinfo_constants)} per locale")

[vstinner]

You should also test _DATE_FMT, no?

Why not testing ERA_D_FMT, ERA_D_T_FMT and ERA_T_FMT?

[vstinner]

Please mention that ERA has no wide character variant and so is not test.

[vstinner]

You might remove max_count of decode_strings() since it's no longer needed.

[vstinner]

Except of ERA, no?

[vstinner]

On my Fedora 44, this change is not only technical, it does actually change nl_langinfo() output on multiple locales.

I'm fine with the change anyway. But since the nl_langinfo() output changes on some locales, I would prefer to not backport this change.

[serhiy-storchaka]

On my Fedora 44, this change is not only technical, it does actually change nl_langinfo() output on multiple locales.

I know, this is a point. Our strftime wraps wcsftime instead of strftime if possible. On gcc it produces results consistent with wide nl_langinfo. If we re-implemented it in Python (there are such plans), we need a wide nl_langinfo.

On other hand, I found these discrepancies when tried to use nl_langinfo in strptime. strptime should be permissive in any case, accept the output of wcsftime and strftime, it should normalize apostrophes, etc, and current code already do this (or many of this, I need to check my non-merged patches).

[vstinner]

LGTM.

I just have a minor coding style suggestion.

test__locale pass on my Fedora 44.

[vstinner]

Nitpick: you can replace loc with legacy_locale here:

        for legacy_locale in legacy_locales:
            locs = (legacy_locale.partition('.')[0] + '.UTF-8', legacy_locale)

And replace for l in locs: with for loc in locs: below.

[vstinner]

Nice change. It's a little bit surprising to use a private API of glibc, but IMO it's worth it.