Repository navigation
xml.etree.ElementTree: file source must be binary for non-UTF-8 encodings #99064
Description
Documentation
In the documentation for xml.etree.ElementTree.parse it says for the first argument source: "... source is a filename or file object containing XML data. ..."
But for file objects this does not work in all (expected) cases. A hint like the following is missing:
For file objects containing XML data with non-ASCII and non-UTF-8 encoding (e.g. ISO 8859-1), the file must have been opened in binary mode.
Otherwise (if opening the file in ASCII mode, regardless of the specified encoding) non-ASCII characters are not read correctly. (see this question on stackoverflow and also the attached files in test_parseXml.zip for reproducing the problem)
Here is an excerpt of the attached test code:
import xml
# ok
with open('test_ISO-8859-1.xml', 'rb') as fileInBinary:
root = xml.etree.ElementTree.parse(fileInBinary).getroot()
print(root.attrib['attributeWithUmlauts'])
# garbage
with open('test_ISO-8859-1.xml', 'r', encoding='ISO-8859-1') as fileInAscii:
root = xml.etree.ElementTree.parse(fileInAscii).getroot()
print(root.attrib['attributeWithUmlauts'])giving the following output:
äöü
äöü
Linked PRs
- gh-99064: Added warnings for file objects with non-UTF8 content #123887
- gh-99064: Ignore the encoding declaration when parsing decoded text #156734
- [3.15] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156750
- [3.14] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156751
- [3.13] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156752
Activity
pythongh-99064: Added warnings for file objects with non-UTF8 content
vstinner commentedon Oct 16, 2024
It's unclear to me which encodings should emit a warning or not. For example:
serhiy-storchaka commentedon Aug 31, 2026
This is not a documentation issue.
ElementTree.parse()is simply wrong here, and only in the C accelerator; every other way of parsing the same text is correct:XMLParser.feed()encodesstrdata as UTF-8 and callsXML_SetEncoding(parser, "utf-8"), so the encoding declared in the document, which no longer applies to already decoded text, is ignored._parse_whole(), the fast path whichparse()uses when no parser is given, does the same encoding but not the override. It got its handling ofstrin 2012 (bpo-14246), and the override was added tofeed()in 2013 (bpo-16986) without touching it.PR #156734 adds the same override there. @vstinner, this answers your question about which encodings should warn: none of them. The text is already decoded by the file object, so its declaration is irrelevant by that point, and there is nothing left to warn about. A text source declaring UTF-16 or UTF-32 also stops raising
ParseError, matchingfeed().gh-99064: Ignore the encoding declaration when parsing decoded text (G…
[3.14] gh-99064: Ignore the encoding declaration when parsing decoded…
[3.13] gh-99064: Ignore the encoding declaration when parsing decoded…
pythongh-99064: Ignore the encoding declaration when parsing decoded …
[3.15] gh-99064: Ignore the encoding declaration when parsing decoded…