imaplib: search/sort/thread don't encode criteria to the declared CHARSET

[serhiy-storchaka]

IMAP4.search(charset, *criteria) (and sort, thread, and the uid SORT/THREAD variants) send a CHARSET <name> token but serialize the criteria strings with the connection encoding (bytes(arg, self._encoding), ASCII by default). So the declared charset and the actual byte-encoding of the criteria are decoupled: a non-ASCII str criterion with a charset raises UnicodeEncodeError, and the only way to search non-ASCII text today is to pass the criteria as pre-encoded bytes and remember to match them to the charset by hand.

This is a Python 2 legacy. The module was written when str was 8-bit and search criteria were byte strings passed straight through to the socket -- no encoding step was needed, and CHARSET was just an opaque label forwarded to the server. Under Python 3 that model leaves str criteria unencoded (or failing), so the CHARSET argument no longer does what it appears to.

str criteria should be encoded to the declared charset. bytes criteria -- necessary for server-only charsets that Python has no codec for (e.g. the ones this module has always forwarded blindly) -- should pass through unchanged.

Linked PRs

• gh-153494: Encode imaplib search criteria to the declared CHARSET #153495

[serhiy-storchaka]

This is a bug, but it was in Python 3 from the beginning. And it's effect is that you simply cannot use charset with string arguments (you can use it with bytes, which is not documented), not that you get wrong result or unexpected error (it always fail). So for users it looks like a new feature.