tarfile stream writes get slower as copybufsize grows

[jonbaldie]

Bug report

Bug description:

tarfile._Stream.__write writes one block and then does self.buf = self.buf[self.bufsize:], so it copies the whole remaining buffer for every block. Each chunk passed in costs O(chunk² / bufsize). At the default copybufsize (16 KiB) that doesn't matter, but raise it and stream writes slow down a lot:

import io, tarfile, time
data = b'x' * (64 << 20)
for c in (16 << 10, 1 << 20, 4 << 20):
    out = io.BytesIO(); t = time.perf_counter()
    with tarfile.open(fileobj=out, mode='w|', copybufsize=c) as tf:
        ti = tarfile.TarInfo('f'); ti.size = len(data)
        tf.addfile(ti, io.BytesIO(data))
    print(c >> 10, 'KiB', f'{(time.perf_counter() - t) * 1e3:.0f} ms')

On main (3.16 dev, macOS arm64), 64 MiB takes 14 ms at 16 KiB, 55 ms at 1 MiB, and 215 ms at 4 MiB. A bigger copy buffer should be faster, not slower.

Fix: write each whole block straight from buf, then keep only the remainder.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-158858: Stop re-copying the tarfile stream buffer for every block #158859