Skip to content

perf(nfs): read each RPC record with one socket read - #642

Merged
yifanxuaaa merged 6 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:feat/lp73-ask3-nfs-requests-v2
Nov 14, 2026
Merged

yifanxuaaa merged 6 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
jiangkoumo:feat/lp73-ask3-nfs-requests-v2

Conversation

@jiangkoumo

@jiangkoumo jiangkoumo commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Tier: P1 (engine/crates/layerfs-nfs) ; risks: risk:kernel ; gate: no (the EINTR loop restarts an interrupted read syscall, as std's read_exact and the base's write_record already do; it does not retry a failed operation) ; de-escalated: none
Docs: docs/architecture/engine/kernel-routes/nfs.md

Summary

The loopback NFS server read every RPC record with two socket reads: one for the 4-byte record mark, one for the body. This PR keeps a small read-ahead buffer per connection so the mark and body usually arrive in one read, and retries reads interrupted by EINTR.

Changes

  • engine/crates/layerfs-nfs/src/rpc.rs: buffered record reader with read-ahead; EINTR retry; large records still read directly.
  • engine/crates/layerfs-nfs/src/server.rs: connection loop uses the buffered reader.
  • tests/protocol.rs, tests/support/client.rs: unit tests for split marks, multi-record buffers, large records and EINTR.
  • docs/architecture/engine/kernel-routes/nfs.md: describes the read path.

Results (exploratory diagnostic, native macOS, NFSv3 loopback mount; counted with a tool not in this diff at an earlier head, not reproducible from this PR. The reproducible count is tests/protocol.rs::a_whole_call_costs_one_read)

  • Socket reads per request: 2.00 -> 0.89 to 1.00 across five scenarios, 3 runs each; writes per request unchanged at 1.00.
  • Total socket calls over the five scenarios: 112,791 -> 73,318 (-35%).
  • verify.sh output identical to baseline.
  • No wall-clock timing was collected, so this PR claims fewer syscalls, not a measured speedup.

Verification

  • macOS: cargo test -p layerfs-nfs 33 passed; clippy --all-targets -D warnings clean.
  • Linux: fmt, clippy (layerfs-nfs, layerfs-daemon) clean; layerfs-nfs 32 passed; layerfs-daemon failures identical to baseline 58c548b3e (FUSE/cgroup permission environment).

Base: layerfs_v2_0_0 at 58c548b3e.

`read_record` read a record with two `read_exact` calls: the four-byte
record-marking header, then the body. The connection now keeps the bytes one
`read` delivered in a per-connection `Records` buffer and frames the record from
them, so a call whose fragments arrived together costs one read instead of two.
A record larger than the 8 KiB read-ahead (a WRITE of a transfer window) is read
with `read_exact` into a buffer of its own, so its bytes are not copied twice; a
fragment chain is assembled from buffered bytes first and from the stream after.
The receive buffer is one read-ahead when idle and is given back after a large
record, so it never holds a record's size.

Measured on the host's native NFSv3 mount (macOS client, one Workspace, 1000
objects per scenario, server-side counters, two runs each): socket reads per
served record 2.00 -> 1.00 (create 2.00 -> 0.91, mkdir 2.00 -> 0.90, stat
2.00 -> 1.00, rm 1.00, rmdir 1.00). Requests per file are unchanged within the
client's own run-to-run variance (create 11.00/11.20 -> 11.28/11.28, rmdir
6.94/7.13 -> 7.03/7.04).

`Records` replaces `read_record`. The test client and the two framing tests hold
one `Records` per connection, which is what a stream that may deliver two records
in one segment needs.

Production LOC: 156309 -> 156409 (delta +100)
scope: first-party production source under core/crates/*/src, engine/crates/*/src,
control/crates/*/src and api/crates/*/src, counted with tools/production_loc.py
at parent 58c548b and at the committed tree; tests, docs and tools excluded.
rpc.rs 178 -> 277 (+99), server.rs 284 -> 285 (+1).
- Records::receive retries on io::ErrorKind::Interrupted, matching the
  baseline read_exact and write_record semantics, and preserves already
  buffered bytes.
- Records::large resets self.valid = 0 without re-allocating an 8 KiB
  buffer, keeping the existing buffer allocated once up to READ_AHEAD;
  removes the dead branch in Records::consume.
- tests/protocol.rs adds unit tests covering partial reads (chunked
  header, small body, large body, fragment chain), back-to-back small and
  large record coalescence from a single buffer, multi-fragment chains
  exceeding RECORD_LIMIT, truncated headers and bodies (UnexpectedEof), and
  retry across signal interruptions.

Production LOC: 156409 -> 156410 (delta +1)
scope: first-party production source under core/crates/*/src, engine/crates/*/src,
control/crates/*/src and api/crates/*/src, counted with tools/production_loc.py
at parent e2f131b and at the committed tree; tests, docs and tools excluded.
rpc.rs 277 -> 278 (+1).
@jiangkoumo
jiangkoumo marked this pull request as ready for review November 10, 2026 11:09
Copilot AI balanced review requested due to automatic review settings November 10, 2026 11:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Review (static, at c8e6f55). Approve after small fixes. I traced the Records framing in rpc.rs and found no bug:

  • a header split across reads;
  • several records in one read;
  • a large single fragment, read straight from the stream;
  • fragment chains, which never over-read into the next record;
  • the 2 MiB limit, checked per fragment and as a running total;
  • EOF mid-header, which returns UnexpectedEof.

Only Interrupted is restarted, which is a syscall restart, not an operation retry.

To fix

  1. The performance numbers don't follow the measurement policy (docs/general/agent-measurement-policy.md §3.1/§3.6).
    • The commit says "two runs each" and the body "3 runs each"; the policy is one sample per arm.
    • The socket-read counter isn't in the diff, so "112,791 -> 73,318" can't be reproduced.
    • There are no receipts, pinned identities or command, and it was measured at e2f131b, not at the head.
    • Please label these as diagnostic counts, or record them properly.
  2. docs/architecture/engine/kernel-routes/nfs.md:31: the Records::read diagram line is one character short, so the box border is misaligned.

Nits

  • The first commit message says the buffer "is given back after a large record"; the second commit keeps it. Squash or reword.
  • No test covers the boundary 4+len == READ_AHEAD (an 8188-byte body). The code handles it.

Merges cleanly on the current base and with #636, #637 and #644.

Production LOC: 156410 -> 160078 (delta +3668)
first-parent comparison: parent c8e6f55 (156410) -> this merge tree (160078),
counted with tools/production_loc.py; scope is first-party production source under
core/crates/*/src, engine/crates/*/src, control/crates/*/src and api/crates/*/src,
with tests, docs and tools excluded. All of the growth arrives from
upstream/layerfs_v2_0_0 (af2ad92); the branch's own two commits are unchanged
by this merge.
The connection diagram's `Records::read` line was one character short, so the
box border did not line up with the rest of the frame.

The paragraph under it now states what a read costs and points at the case that
shows it: one socket read per call that arrived in one piece, two for a record
past the read-ahead. It also records that a blocking read a signal interrupts is
retried and keeps the bytes already received, which the framing change of the
preceding commits introduced.

Production LOC: 160078 -> 160078 (delta 0)
scope: first-party production source under core/crates/*/src, engine/crates/*/src,
control/crates/*/src and api/crates/*/src, counted with tools/production_loc.py
at the merge tree 3b5d99c and at this committed tree; tests, docs and tools
excluded. This commit changes documentation only.
`4 + len == READ_AHEAD` is the last body the read-ahead covers: an 8188-byte
body fills the 8 KiB exactly, so the record must be framed from the buffer and
must not be handed to the own-buffer path a larger record takes. The new test
frames it in pieces of 1, 3, 4, 8188 and 8192 bytes, alone and after a small
record, and checks the next record still ends the stream orderly.

The second test carries the read-count diagnostic the preceding commits quoted
from a mount, so the count is reproducible on any platform: a counting stream
sees one read for a call that arrived in one piece, one read for two records in
one segment, one read for a body that fills the read-ahead exactly, and two for
a body past it (the buffered prefix and one exact read).

Production LOC: 160078 -> 160078 (delta 0)
scope: first-party production source under core/crates/*/src, engine/crates/*/src,
control/crates/*/src and api/crates/*/src, counted with tools/production_loc.py
at c26456a and at this committed tree; tests, docs and tools excluded. This
commit changes tests only.
Resolved docs/architecture/engine/kernel-routes/nfs.md on the new Ephemeral-AI-Lab#671 section layout while keeping the Records::read server diagram line and the RPC framing/read-ahead/read-count/EINTR paragraph.

Production LOC: 160078 -> 160078 (delta 0)
first-parent comparison: parent 36e2abb (160078) -> this merge tree (160078), counted with tools/production_loc.py; scope is first-party production source under core/crates/*/src, engine/crates/*/src, control/crates/*/src and api/crates/*/src (tests, docs and tools excluded). Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
@jiangkoumo

Copy link
Copy Markdown
Collaborator Author

更新到 e6e9ac0:

验证(Linux,cargo +1.85.1 --locked,head e6e9ac0):

  • layerfs-nfs 34 passed(protocol 15 个),clippy -D warnings 干净
  • slow 146/146,mount 二进制 45 过 0 失败
  • PG:persistence 25、store-service 6、project 1、api 2、cli 2、control 3,全过
  • fast 1904 个里 1903 过、1 个失败:layerfs-cli exit_codes stdin_and_from_feed_the_content_and_json_positionals 报 Broken pipe。本 PR 没改 layerfs-cli;这是 --from 跳过 stdin 后父进程仍写管道的并发竞态,同一 head 单独重跑通过。
  • fmt --check 只剩上游基线 api/crates/layerfs-cli/tests/concurrent_reads.rs:16

未跑:macOS 挂载侧计数复测。

@yifanxuaaa

Copy link
Copy Markdown
Contributor

Review record

  • Head reviewed: e6e9ac05f6. It merges into layerfs_v2_0_0 at 836036d2a1 without conflicts.
  • Tier: P1, risk:kernel. Gate: no. The EINTR loop restarts an interrupted syscall, as read_exact and the base's write_record do; it is not an operation retry.
  • Read: the whole buffered record reader. That covers record marks at offset 0, fragment chains through take, the per-fragment and total limit (checked before allocation), the fixed 8 KiB buffer, the large-record path when part of the body is already buffered, and EOF in a mark or body. No defect found. Behaviour change: a 1–3 byte partial mark at EOF is now UnexpectedEof instead of Ok(None). That is harmless, because the server ends the connection either way. Tests: no assertion was weakened; the moved assertions are carried into Records.
  • Re-run at the PR head merged with 836036d2a1, on macOS with Rust 1.85.1:
    • tools/check_product_boundary.py: PASS.
    • cargo test -p layerfs-nfs (--no-run first, 100 s limit): 35/35 pass.
    • cargo clippy -p layerfs-nfs --all-targets -- -D warnings: clean.
  • Linux mount proof: the author reports it; no reviewer has re-run it. Waived by the owner for this merge.
  • Not checked: the macOS native client with pipelined calls, and the server answer path beyond the reader.
  • Nit: commit e2f131b166 says the buffer "is given back after a large record". c8e6f5530e reversed that; the squash replaces the message.

Reviewer: Claude Opus 5.5 (independent-context review, then spot-checked)

@yifanxuaaa
yifanxuaaa merged commit d2f1429 into Ephemeral-AI-Lab:layerfs_v2_0_0 Nov 14, 2026
yifanxuaaa added a commit to jiangkoumo/layerfs that referenced this pull request Nov 14, 2026
…#663, Ephemeral-AI-Lab#642) into fix/578-sync-volume

Conflict resolved in engine/crates/layerfs-store-service/tests/support/installed_store.rs:
this branch's fixture layout is kept (the Store in a private directory/store,
the fixture root left traversable for mounts), and the duplicate
private_directory helper Ephemeral-AI-Lab#663 added is dropped.

Production LOC (first parent 11b30f0): 160034 -> 160621 (delta +587, the base's changes brought in); scope core 39787, engine 74475 -> 74765, control 39796 -> 40093, api 5976; reference absent (retired R9c; last counted 65,417 at tree 498dd19); method tools/production_loc.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jiangkoumo added a commit to jiangkoumo/layerfs that referenced this pull request Nov 14, 2026
Clean merge: no code or documentation conflict, so no resolution was
needed. The upstream side brings Ephemeral-AI-Lab#678, Ephemeral-AI-Lab#679/Ephemeral-AI-Lab#680, Ephemeral-AI-Lab#663, Ephemeral-AI-Lab#642, Ephemeral-AI-Lab#681 and
Ephemeral-AI-Lab#647.

Production LOC: 160158 -> 160802 (delta +644)
Scope and method: first-parent comparison at 67559fd vs this merge tree
via python3 -B tools/production_loc.py over first-party product
Rust/SQL/JavaScript only (reference absent, retired R9c).
jiangkoumo added a commit to jiangkoumo/layerfs that referenced this pull request Nov 14, 2026
Brings the Ephemeral-AI-Lab#136 helper reaping work onto upstream after the reviewer's
first pass: Ephemeral-AI-Lab#656 force-stop of machines and native Sandboxes, Ephemeral-AI-Lab#647 macOS
export-volume flush and exact Busy, Ephemeral-AI-Lab#682 image-loaded Sandboxes, Ephemeral-AI-Lab#681 GC
design, Ephemeral-AI-Lab#642 one-read NFS records.

Conflicts were documentation only, and are resolved into upstream's
structure:
- engine/daemon/control-calls.md: upstream's macOS `sync_volume_np` flush
  row kept; this branch's Attach (mount table read once, `tracked`),
  Unmount (mount-table settlement), ForceUnmount (finishing an export kept
  at Detach) and Helper run (retained queue) rows merged into upstream's
  table.
- engine/kernel-routes/nfs.md: upstream's Commit line kept inside this
  branch's Attach/Unmount/ForceUnmount diagram.
Code merged without conflicts; Ephemeral-AI-Lab#656 touches Sandbox/machine force-stop and
the Store lane, not the NFS export teardown, so the two do not overlap in
the data path. `cargo +1.85.1 check --locked --all-targets` on the changed
crates passes at this tree.

Durable: NOT_RUN - disabled by owner until explicit reauthorization.
Production LOC: 160218 -> 163746 (delta +3528); first parent b49690d.
  core 39787 -> 39787, engine 74627 -> 75078, control 39828 -> 42804,
  api 5976 -> 6077. Reference absent (retired R9c; last counted 65,417 at
  tree 498dd19). Method: python3 tools/production_loc.py --root
  <git archive of b49690d> and --root <the merged working tree>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants