Add an iconv-based codec engine to support all system locale encodings

[serhiy-storchaka]

Feature or enhancement

Python ships a fixed set of built-in codecs.
When the system locale uses an encoding none of them covers, codecs.lookup() / str.encode() / bytes.decode() raise LookupError: unknown encoding, and Python may fail to even start, aborting in init_fs_encoding when it cannot find a codec for the filesystem encoding.

These are ordinary, standard locales — zh_TW.euctw, hy_AM.armscii8, ka_GE.georgianps, and others ship with glibc and work in every other program.
Every such encoding is already known to the C library's iconv().

Proposal

Add an iconv-based codec engine, so Python can use any encoding iconv() knows, even without a dedicated built-in codec.

• A last-resort search function, so it never shadows a built-in codec — it only catches otherwise-unknown names.
• An explicit iconv: prefix (e.g. "iconv:latin1") forces the engine even when a built-in exists.
• Full codec API: stateless encode/decode with all error handlers, incremental codecs, and stream reader/writer.

Design

• C: _PyUnicode_DecodeIconv / _PyUnicode_EncodeIconv in Objects/unicodeobject.c, exposed as _codecs.iconv_decode / _codecs.iconv_encode.
  The pivot is native-endian UTF-32 (a raw array of Py_UCS4), so the conversion is independent of wchar_t's representation (correct even where wchar_t is not Unicode, e.g. Solaris/AIX) and keeps a 1:1 code-point/position mapping for error handlers.
• Python: Lib/encodings/_iconv_codecs.py plus a search function in Lib/encodings/__init__.py, mirroring _win_cp_codecs.py.
• Build: configure gains an iconv check defining HAVE_ICONV.

Prior art

gh-37800 ("Universal Unicode Codec for POSIX iconv", 2003) added an iconvcodec module, but it was removed after causing crashes (gh-37835, gh-37840, gh-37869) and build failures across Unixes (gh-38012 IRIX, gh-38019/38020 Solaris, gh-38033 Tru64, gh-38068 HP-UX, gh-38240 OpenBSD, gh-37865 Cygwin); follow-ups gh-38224/gh-38255/gh-38297 stalled.
This proposal avoids those pitfalls: no separate extension module (the conversion lives in the core unicodeobject.c), gated on a configure check, and pivoting through UTF-32 rather than wchar_t.

It is the POSIX counterpart of gh-123803, which made arbitrary Windows code pages usable as cpXXX encodings by delegating to the OS API.

Availability

Which encodings iconv provides is platform-dependent (glibc, musl, the citrus *BSDs and macOS/GNU libiconv all differ); OpenBSD, whose iconv has only UTF-8, gains nothing.
The feature is active only where iconv() is present, and transparent when absent.

Issues this resolves

• Python cannot start in locale zh_TW.euctw #91992 — Python cannot start in an unsupported locale (EUC-TW)
• Python does not support the GEORGIAN-PS charset #63658 — GEORGIAN-PS charset (a crash)
• missing vietnamese codec TCVN 5712:1993 in Python #65280 — Vietnamese TCVN 5712:1993
• Encode to EBCDIC doesn't take into account conversion table irregularities #74771 — EBCDIC cp1047 conversion table irregularities
• Add KOI8-RU as a known encoding #49464 (KOI8-RU) and Adding new CNS11643, a *huge* charset, support in cjkcodecs #46342 (CNS 11643) — requested charsets iconv provides
• relates to When Python is started on AIX in the IBM-943 (Japanese shift_jis) locale, it aborts. #130228 — abort in the AIX IBM-943 locale

Linked PRs

• gh-152997: Support system locale encodings via an iconv-based codec #153001

[BHUVANSH855]

Hi @serhiy-storchaka,

I'm interested in working on this feature and have been studying the current implementation before starting.

From exploring the existing code, my understanding of the intended architecture is:

• Add HAVE_ICONV detection during configuration.
• Implement _PyUnicode_EncodeIconv() and _PyUnicode_DecodeIconv() in Objects/unicodeobject.c.
• Expose them through _codecs.iconv_encode() and _codecs.iconv_decode().
• Add Lib/encodings/_iconv_codecs.py, similar to _win_cp_codecs.py.
• Register it as a last-resort search function in Lib/encodings/__init__.py, while allowing the iconv: prefix to explicitly select the iconv implementation.

I also noticed that CPython already uses iconv() in Python/fileutils.c for non-Unicode wchar_t conversion, so I wanted to check whether any of that implementation is expected to be reused or whether the new codec engine should be implemented independently.

Before I start implementing, does this align with the intended design, or are there any architectural considerations or preferred implementation details that I should keep in mind?

Thanks!

[serhiy-storchaka]

Caveats and limitations

By default the engine is a last-resort fallback that never shadows a built-in codec; the behavioral differences below apply only when it is forced over a built-in name via the iconv: prefix.

Availability (platform-dependent).

• Provided encodings and their spelling differ across glibc / musl / citrus *BSD / macOS libiconv (glibc has latin1, not latin-1); not portable.
• OpenBSD (UTF-8 only) and Solaris/illumos are inert. Solaris has no UTF-32 pivot — a UTF-8-pivot fallback for it could be added later.

Less strict than the built-ins.

• iconv:utf-8 etc. accept sequences CPython rejects (e.g. overlong forms) and report error positions differently.
• iconv:utf-16/utf-32 add and expect a BOM.
• UCS-2 aliases UTF-16 on some platforms (citrus *BSD), so iconv:ucs-2 there adds a BOM and combines surrogate pairs instead of keeping each 16-bit unit independent.
• Incremental/stream codecs don't keep the built-ins' fine-grained state (e.g. one BOM across chunks), so chunk-boundary results can differ.

Interop.

• Multibyte iconv-only encodings can't be used as XML document encodings: with no expat decoding table, xml.parsers.expat cleanly rejects them (never mis-parses). Tables for known multibyte encodings could be added later.
• iconv-backed incremental encoders/decoders can't be pickled or copied.

Performance. Each stateless call opens and closes an iconv descriptor (like the Windows code-page codec); no caching.

[serhiy-storchaka]

@BHUVANSH855, thanks, and sorry for the overlap — this is already implemented in PR #153001.

The iconv() in Python/fileutils.c is unrelated: it is the HAVE_NON_UNICODE_WCHAR_T_REPRESENTATION fallback for platforms where wchar_t is not Unicode. The new codec deliberately avoids wchar_t and pivots through UTF-32 instead, so none of it is reused.