gh-158858: Stop re-copying the tarfile stream buffer for every block

[jonbaldie]

Summary

_Stream.__write writes one block and then slices it off the front of buf, which copies everything left. For a chunk of c bytes that's about c²/(2·bufsize) bytes copied. At the default 16 KiB copy buffer you can't see it. Set copybufsize to a few MiB and a bigger buffer makes w| writes slower.

Now it writes each whole block straight from buf, then keeps the remainder:

 self.buf += s
-while len(self.buf) > self.bufsize:
-    self.fileobj.write(self.buf[:self.bufsize])
-    self.buf = self.buf[self.bufsize:]
+if len(self.buf) > self.bufsize:
+    end = (len(self.buf) - 1) // self.bufsize * self.bufsize
+    for i in range(0, end, self.bufsize):
+        self.fileobj.write(self.buf[i:i + self.bufsize])
+    self.buf = self.buf[end:]

end is picked so the same 1..bufsize bytes stay behind as before. The file object gets the same bytes blocks, in the same order.

Evidence

One 64 MiB member written with mode='w|' to BytesIO, then read back and checked. 3.16 dev build, macOS arm64, median of 3, two alternating runs:

copybufsize	before	after
16 KiB (default)	13.8 ms	13.8–15.5 ms (noise)
1 MiB	53–56 ms	9.6–11.4 ms
4 MiB	212–220 ms	10.4–21.1 ms

I can't count slice copies from a test, so the new test test_stream_write_large_chunks guards behaviour instead: with copybufsize set to a 256 KiB member, every write() the file object sees is bytes and exactly RECORDSIZE long, and the archive reads back the same.

I also ran old and new _Stream against each other on 1,200 random chunk sequences, using tar, gz, bz2 and xz with bufsize 1, 7, 512 and 10240. The list of write() calls was identical every time. test_tarfile + test_shutil: 1,016 run, all pass.

Merge Danger

Door: two-way

Same output, same write calls. Only the internal buffer handling changes.

Blast Radius: stdlib

Any tarfile stream write (w|, w|gz, and so on) goes through this code. Custom file objects still get bytes blocks of the same size.

AI disclosure: the profiling, the patch and the equivalence check were done with AI tools (Codex, Claude).

🤖 Generated with Claude Code

[picnixz]

While the patch is correct, we don't want fully automated PRs opened automatically. So please avoid opening further PRs until they are merged. In particular, I would advise that you first trim down the description by yourself without ANY agent behind.