Repository navigation
ElementTree parser limitation of input string size #83895
Description
| BPO | 39714 |
|---|---|
| Nosy | @scoder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-02-21.17:59:58.910>
labels = ['expert-XML', 'type-bug', '3.9', '3.10']
title = 'ElementTree parser limitation of input string size'
updated_at = <Date 2020-09-08.04:55:41.878>
user = 'https://bugs.python.org/AnanthVijalapuram'bugs.python.org fields:
activity = <Date 2020-09-08.04:55:41.878>
actor = 'scoder'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['XML']
creation = <Date 2020-02-21.17:59:58.910>
creator = 'Ananth Vijalapuram'
dependencies = []
files = []
hgrepos = []
issue_num = 39714
keywords = []
message_count = 2.0
messages = ['362418', '376545']
nosy_count = 2.0
nosy_names = ['scoder', 'Ananth Vijalapuram']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39714'
versions = ['Python 3.9', 'Python 3.10']Linked PRs
- gh-83895: Accept input larger than 2 GiB in the C implementation of ElementTree #156746
- [3.15] gh-83895: Accept input larger than 2 GiB in the C implementation of ElementTree (GH-156746) #156771
- [3.14] gh-83895: Accept input larger than 2 GiB in the C implementation of ElementTree (GH-156746) #156772
- [3.13] gh-83895: Accept input larger than 2 GiB in the C implementation of ElementTree (GH-156746) #156773
Activity
AnanthVijalapuram commentedon Feb 21, 2020
I am trying to parse a very large XML file. Here is the output:
scoder commentedon Sep 8, 2020
I'd suggest feeding the data into the parser in chunks, or letting it read from a file-like object, or something like that.
Also, you probably want to do incremental processing on the data (see the XMLPullParser and iterparse), because reading 3.5GB of XML data into an in-memory tree can easily result in 10x the memory usage. You may have 40GB of RAM on your machine, but even then, I'd still recommend processing the data in incrementally.
[-]ElementTree limitation[/-][+]ElementTree parser limitation of input string size[/+][-]ElementTree limitation[/-][+]ElementTree parser limitation of input string size[/+]vadimkantorov commentedon Dec 14, 2023
This error is hit when reading wiktionary dumps :(
scoder commentedon Dec 15, 2023
I personally don't think we should do something about this. It's very easy to work around the failure by using the much more efficient
parse()function instead of first reading so much data into a string in memory and then parsing from that. That just uselessly wastes both memory and time. Whether it's a good idea to parse the whole document into memory at all is then yet another question. Most people should be better off usingiterparse()or theXMLPullParserinstead.If nothing else, the failure at least makes users aware that they are not doing something recommended.
vadimkantorov commentedon Dec 15, 2023
For modern big machines with dozens gigabytes of memory, it might be not so senseless (and be a good trade-off for its simplicity). If it's an int32 limitation, lifting it to int64 might be good
serhiy-storchaka commentedon Aug 31, 2026
PR #156746 removes the limitation.
The C implementation raised
OverflowErrorbecause Expat takes the length as anint.xml.parsers.expathas fed Expat in chunks since bpo-17089 (2013), so the pure Python implementation of ElementTree already accepted input larger than 2 GiB; only the accelerator refused it:@scoder, your advice stays right:
parse()oriterparse()is better than reading gigabytes into a string first. Butxml.parsers.expataccepts such input, so this was not a deliberate limitation, and the accelerated module should not be able to do less than the pure Python one. When I fixedpyexpatin bpo-17089 I could not test this half of it: machines had 2 GiB of RAM. The new test needs 6 GiB and is skipped unless-Mis given.Chunking is not slower for ordinary documents either: 1 GiB with a million elements is parsed in 2.37 s in chunks of 1 MiB and in 2.60 s in a single call.
Merge branch 'main' into pythongh-83895-large-input
gh-83895: Accept input larger than 2 GiB in the C implementation of E…
[3.14] gh-83895: Accept input larger than 2 GiB in the C implementati…
[3.13] gh-83895: Accept input larger than 2 GiB in the C implementati…
scoder commentedon Sep 1, 2026
Change looks good and helpful to me. Thanks for implementing it.
pythongh-83895: Accept input larger than 2 GiB in the C implementatio…
[3.15] gh-83895: Accept input larger than 2 GiB in the C implementati…