Skip to content

0.8.0: write without fsync or OS locks, file I/O sized apart, create timed out - #13

Merged
davidackerman merged 2 commits into
mainfrom
write-defaults-0.8
Oct 23, 2026
Merged

davidackerman merged 2 commits into
mainfrom
write-defaults-0.8

Conversation

@davidackerman

Copy link
Copy Markdown
Member

0.8.0: faster writes from the contexts the package makes, and the create open timed out. No placement change and no unit table change. It requires tensorstore 0.1.68 or later (was 0.1.60), the first release with file_io_locking.

All of this was measured on 2026-10-23 on real CellMap data, against zarr-python 2.18 and 3.1, on local NVMe and NFS.

fsync and locking. shared_context and capped_context now set file_io_sync: false and file_io_locking: {"mode": "none"}. fsync=True and locking="os" restore tensorstore's defaults.

  • fsync cost 6.5 ms per chunk on local disk and nothing on NFS, which commits a file on close. zarr-python and funlib never synced.
  • "os" locking cost about 10 ms per chunk on NFS: a 64³ label chunk took 21.7 ms with it and 12.0 ms without, against zarr-python 2's 12.7 ms.
  • What "none" keeps and gives up. A whole-unit write stays atomic, as a temporary file renamed into place. Only concurrent read-modify-writes of one unit lose their guarantee, and Behaviour 13 never gave that.
  • Why not "lockfile". A lock file left behind by a killed worker blocks every later write to that unit.

File I/O threads. The default context's file I/O pool is max(thread_budget(), 8); decoding stays at thread_budget().

  • File I/O threads mostly wait on storage, so they take no core from the job.
  • Eight 64³ label chunks written to NFS from one decoding thread took 166–176 ms with one file I/O thread and 83–89 ms with 4 to 16.
  • capped_context(file_io=) sizes the file I/O pool separately. Its default is unchanged.

Before and after, one-thread workers (OMP_NUM_THREADS=1), default context:

0.7.1 0.8.0 zarr-python 2.18
NFS, one 64³ label chunk 23.0 ms 12.2 ms 13.1 ms
NFS, 8-chunk label block 159 ms 66.5 ms 102 ms
NFS, 8-chunk EM block 330 ms 202 ms
Local disk, 8-chunk label block 51.5 ms 4.1 ms 2.9 ms

Create open timed out (cellmap-analyze's question).

  • The open in wa.create is now timed out and retried under retries.
  • Suppose a retry finds that the array was already made by an earlier attempt of its own, one that timed out. It adopts it after the usual comparison, even under exists="error".
  • That open runs on a child context that keeps "os" locking, so create-if-absent stays atomic.

Tests.

  • 189 pass locally (1 opt-in stress test skipped). They also pass on tensorstore 0.1.68 with Python 3.10.
  • The opt-in lost-wakeup stress test passes: the watched context stranded 0 tasks.
  • ruff passes, and mkdocs build --strict is clean.

https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64

…eate (0.8.0)

Measured 2026-10-23 against zarr-python on real CellMap data: tensorstore's fsync cost
6.5 ms a chunk on local disk and nothing on NFS; its "os" file locking about 10 ms a
chunk on NFS; and one file I/O thread made a one-thread worker write a block's chunk
files one NFS round trip after another. Contexts the package makes now set
file_io_sync false and file_io_locking "none" (fsync=True, locking="os" restore them;
whole-unit writes stay atomic), and the default context's file I/O pool is at least 8
while decoding stays at thread_budget(). capped_context gains file_io=. From one-thread
workers: an NFS label block 159 to 66.5 ms (zarr-python 2: 102), a local one 51.5 to
4.1 ms.

The create open is timed out under retries too (cellmap-analyze's question); a retry
that finds the array its own timed-out attempt made adopts it after the usual
comparison, and that open keeps OS locking so create-if-absent stays atomic.

tensorstore>=0.1.68, the first release with file_io_locking.

Claude-Session: https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64
@davidackerman
davidackerman merged commit 6eea888 into main Oct 23, 2026
4 checks passed
@davidackerman
davidackerman deleted the write-defaults-0.8 branch October 23, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant