[dependabot]

Bumps chardet from 4.0.0 to 7.7.0.

Release notes

Sourced from chardet's releases.

7.7.0

Big release: raising max_bytes to cover a huge input no longer makes detect() slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all!

Performance

• Raising max_bytes on large inputs no longer gets slow. On compiled wheels, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged.
• The pure-Python wheel on CPython scores bigram models through operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster. (#404, thanks @​deadstrobe5)
• Fixed three scans that went quadratic on hostile input: the UTF-7 + scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed <meta and <?xml tags in the markup scan, and the EBCDIC charset-declaration regex. (#403, thanks @​Samin061)

Bug Fixes

• Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and <meta charset> declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (EUC-JP, SHIFT_JIS, Big5, ISO-2022-JP) are only used when those narrower codecs decode it too. (#394 and #396, thanks @​Samin061)
• Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a charset= inside another attribute (like a refresh URL) no longer beats a real <meta charset>, quoted or spaced content charsets work, and an ASCII page declaring utf-16le no longer comes back as UTF-16-LE. (#389 and #398, thanks @​SashaMIT and @​xiehuanyi)
• PEP 263 cookies resolve the way tokenize.detect_encoding resolves them, including only counting a line-2 cookie when line 1 is blank or a comment. (#391, thanks @​xiehuanyi)
• Fixed BOM-bearing UTF-32 coming back as UTF-16 when max_bytes cuts a code unit. (#405, thanks @​AryanRG)
• Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104.
• Fixed a mostly-ASCII Windows-1252 file with a single Ö detecting as HP-Roman8, and English with a lone É detecting as MacRoman. (#383, thanks @​aadsm)
• Fixed null-separated records like find -print0 output coming back as UTF-16-LE with Chinese as the language.
• prefer_superset=True no longer swaps in a code page that can't decode the data, and now respects include_encodings/exclude_encodings. (#393, thanks @​00200200)
• chardetect and UniversalDetector no longer mistake input cut exactly at the byte cap for the whole stream. (#392, thanks @​00200200)
• lookup_encoding's cache is capped at 1024 entries, since charset names come from untrusted markup. (#390, thanks @​Samin061)
• Plus smaller fixes to detect_all() ranking, EBCDIC XML MIME types, and the deprecated chardet.equivalences shim. The changelog has the full list.

Improvements

• New -m/--mime-type flag for chardetect shows the MIME type next to the encoding, like file -i. (#384, thanks @​smi11 for the request)
• Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it.
• EncodingInfo.is_multibyte is now is_multibyte_cjk. The old name still works with a DeprecationWarning and goes away in 8.0.
• pipeline/markup.py is now compiled by mypyc (15 modules total), and CI enforces 100% test coverage.

Full Changelog: chardet/chardet@7.6.0...7.7.0

7.6.0

Big release: a Cython scoring kernel joins mypyc in compiled wheels, every model retrained on a deduplicated corpus, UTF-7 fixed in both directions, and a guarantee that detect() never returns an encoding that can't decode your complete input.

Performance

• Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14. _kernel.py stays plain Python (PyPy and pure wheels run it interpreted, unchanged), and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in ~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks: HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true.
• Added support for CPython 3.15, including the free-threaded build. No code changes were needed.

Bug Fixes

• detect() no longer returns an encoding that cannot decode the input it was given (#380, thanks @​yarikoptic). When the whole input has been examined and the winner's only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer.
• Fixed delimited ASCII data like |NAME,+LAY| misdetecting as UTF-7 (#371 follow-up, thanks @​agreenburg). The whole buffer must now actually decode as UTF-7, and a lone shifted character must land in a plausible script range.
• Signed UTF-7 no longer reads as ASCII: the BOM stage recognizes the four UTF-7 signature prefixes when the rest of the buffer decodes as UTF-7.
• Fixed short apostrophe-heavy English being labeled Scottish Gaelic or Breton: a rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language (ADR-0005).
• Fixed Hungarian text losing to a Czech reading in confusion rescoring; tied pairs are compared only under language models both encodings have.
• Fixed space-padded text matching a degenerate Serbian model at high confidence; statistical scoring now skips repeated-whitespace bigrams. This also fixes windows-1251 logs misdetecting as windows-1250 (#379).
• Fixed EBCDIC text being invisible to the early pipeline stages, and the last two EBCDIC sibling misdetections.
• Fixed training normalization gaps that starved ISO-8859-16 and the 26 pre-euro encodings at exactly their distinguishing bytes.

... (truncated)

Changelog

Sourced from chardet's changelog.

7.7.0 (2026-10-17)

Features:

• Added a -m/--mime-type flag to the chardetect CLI, which shows the detected MIME type next to the encoding, like file -i. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude, [#384](https://github.com/chardet/chardet/issues/384) <https://github.com/chardet/chardet/issues/384>_)

Accuracy:

• Visual-order Hebrew is now detected. The ISO-8859-8 model trains on both bidi conventions, so 1990s visual-order pages come back as ISO-8859-8 while logical-order text still prefers Windows-1255. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
• cp864 Arabic now trains on contextually shaped text in visual order, the way real cp864 systems stored it, instead of isolated letter forms in logical order. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)

Performance:

• Raising max_bytes on large inputs no longer gets slow. On the compiled build, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.23s for cp1252 and 0.56s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, so the answer still decodes it. The other stages stop at a 256 KB evidence cap, which sits above the default max_bytes, so default calls are unchanged, and the default 200 KB cap is still the fastest path. ADR-0006 has the design. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
• The UTF-7 + scan is no longer quadratic on line-wrapped base64 like PEM certificates, MIME attachments, and JWTs. 128 KiB of it went from 5.4 seconds to 1.2 milliseconds. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
• A head full of unclosed <meta or <?xml tags no longer makes the markup scan quadratic, taking it from about 5 ms to about 20 µs. A literal < inside an attribute value now hides that one tag. (Dan Blanchard <https://github.com/dan-blanchard>_ via Claude)
• An EBCDIC charset= or encoding= followed by whitespace and no value no longer makes the declaration scan backtrack quadratically. A crafted 4 KB document used to cost detect() about 60 ms. (Samina <https://github.com/Samin061>, [#403](https://github.com/chardet/chardet/issues/403) <https://github.com/chardet/chardet/pull/403>)
• The pure-Python install on CPython scores bigram models with operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster across the test corpus. Compiled builds and PyPy keep their existing paths. (Afonso Ribeiro <https://github.com/deadstrobe5>_)

... (truncated)

Commits

• 7a87294 Release 7.7.0
• 1785c56 docs: label the base64 large-input rows as ascii with a footnote
• 4ec13a0 test: take the best of three cold starts in the cold-vs-warm check
• d7be8c7 docs: re-measure the large-input table for 7.7.0
• d6dcc41 docs: update benchmark numbers for 7.7.0
• cc5924a Rename EncodingInfo.is_multibyte to is_multibyte_cjk
• e0a7ff9 docs: derive the multi-byte column from the codec
• a3699c9 Fix UTF-32 BOM detection for truncated max_bytes windows (#405)
• 89cab01 Score bigram models through itemgetter on interpreted CPython (#404)
• ea3ef5f Fix quadratic backtracking in the EBCDIC charset-declaration scan (#403)
• Additional commits viewable in compare view

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 70682b2b-539e-48ff-bb3e-b56a975c8e62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}