adapter: keep timestamp oracle round trips off the coordinator loop for object memory arrangements

[leedqin]

Motivation

#37455 moved arrangement size history snapshotting's persist reads off the coordinator loop, but three timestamp oracle round trips stayed on it. The oracle is backed by the metadata store, so each call is a network round trip with unbounded tail latency. After the feature was re-enabled in staging, arrangement_sizes_snapshot showed ~976ms p95 on the Slow Coordinator Messages panel, against a target of <10ms for anything on the coordinator task.

The remaining round trips:

• schedule_arrangement_sizes_collection awaited peek_local_write_ts on entry. Sleeps are capped at MAX_SLEEP, so this re-entered every 60 seconds even while collection was disabled, and again on every skip path.
• arrangement_sizes_write awaited get_local_write_ts once per cycle to allocate a write timestamp used only as a display stamp, never applied to a write.
• All four handlers were dispatched as futures via .boxed_local().await.

Changes

1. The oracle peek moves into the task that already sleeps and snapshots. Only the dyncfg read and the organization_id-seeded offset derivation stay on the loop.
2. collection_ts is stamped from the snapshot's own read_ts and carried on Message::ArrangementSizesWrite, rather than allocating a second timestamp in the write handler. read_ts is the state the rows describe, is monotone across restarts via the oracle, and matches the startup prune cutoff, which is already read_ts based.
3. With no remaining await points, all four handlers become sync fns and the dispatch arms call them directly.

The result is structural rather than incremental: a sync fn cannot await, so this path can no longer block the loop regardless of future edits, and the constraint is enforced by the compiler rather than by review.

Tests

No tests added or modified. This is a threading change with no intended behavior change, and it is covered by the existing arrangement_sizes unit tests, test/testdrive/arrangement-sizes.td, and test/restart workflow_arrangement_sizes_stale_snapshot_after_restart.

One behavioral nuance for reviewers: because the snapshot now runs entirely off-thread, a replica can go offline while it is in flight, so arrangement_sizes_write revalidates introspection freshness before appending and drops records from replicas that are no longer trusted.

Follow-up

collection_ts is no longer a uniquely allocated timestamp, so two snapshots landing in the same millisecond would share one. Not reachable at the hourly production cadence.

Verification that this actually clears the spike is post-deploy: re-enable arrangement_size_history_collection_interval in staging and confirm the arrangement_sizes_* series stay flat, with oracle tail latency appearing only in arrangement_sizes_collection_time_seconds.

🤖 Generated with Claude Code