Repository navigation
Speed up hosted relay transfers while preserving Raft authority - #2162
santoshkumarradha wants to merge 11 commits into
Conversation
|
Agent review round 1 — architecture Reviewed the design against the quorum/revocation ruling. Rejected a direct local directory writer: concurrent identity writers still require consensus. The implementation keeps mutation commits and uses an invalidated, short-lived confirmation of clean applied state for reads. Fresh singleton provenance is immutable; an established voting configuration cannot be reduced during recovery. No remaining architecture blocker was found within this review. |
|
Agent review round 2 — correctness Review found that a compacted snapshot could exclude a newly admitted learner, causing repeated full snapshot transfers. Learner admission now refreshes the snapshot before catch-up and retains the resulting durable effects. The archived large-body cases pass, including restart and staging cuts. Additional checks cover every formation turn and refusal to recreate a lost published member. Reviewed production revision: a8005c4. |
|
Agent review round 3 — security and cost Review found two gaps: a replicated revoke could bypass public-handler watch closure, and an immediately rejected upload could release its in-flight slot before storage finished. Committed-apply notifications now close watches and held polls; upload tracking lasts until actual handler completion. Both findings have red-to-green regressions. Existing quorum refusal, replay and revocation assertions remain. Validation at a8005c4: 166 relay proofs, the separate actor proof, 248 unit tests, and fresh affected Go-package checks pass. Performance acceptance and remaining end-to-end fixture work are still in progress; these agent reviews do not authorize merging. |
|
Agent follow-up review of |
|
Acceptance update: the full normal relay proof suite passes 167/167 against A live two-device chat continuation succeeded after abruptly stopping the source application, preserving history and a hash-verified workspace file. However, a larger real-conversation test has exposed intermittent 503 responses during the initial directory read, before any file transfer, including samples while the source is still running. This remains an acceptance blocker; the PR stays draft and is not ready to merge. The earlier five matched public-internet transfer pairs for |
|
Acceptance update for
Mutations remain consensus commits. The live read-interruption fix retries only marked read-certificate interruptions, restarts the whole directory read, and never replays a mutation or blob I/O. Quorum loss, expired authority, activation fencing, lost-store and revocation assertions pass unchanged. |
Test mechanisms and preserved contractsThe fixture changes do not relax quorum, revocation, replay, fencing, memory admission or erasure assertions. They remove outdated setup and transport assumptions so those properties run against the consensus implementation.
The separate Raft fixtures explicitly form the existing three-voter topology before injecting faults intended for that topology. Cold-birth capacity accounting counts actual singleton voters without lifting its memory budget; restored groups still require their actual quorum. Crash cut markers follow the same durable boundaries. Replay capture begins before the first request and preserves the original command across formation, avoiding a vacuous replay assertion. Test doubles add only the new presence/read-fence interface members. New cases verify lease expiry, activation fencing, stale signatures, off-path presence, singleton formation/recovery and committed revoke interruption through observable interfaces. Watch E2E exposed and fixed two implementation errors rather than changing their assertions: duplicate direct-path version broadcasts, and closing rotated sockets before sending their final committed version. The committed-apply notifier now owns one version notification before closure; successful direct revokes also send the existing revoked event. Generic replicated revoke closure and held-I/O interruption remain intact. The aggregate final HTTP E2E suite passed on the combined source at Final aggregate verification (2026-11-02): complete HTTP E2E exit 0 against frozen combined worker SHA256 21c0b8ace6503589454b59ecd7c440dba6fc89ec0ea521c1d2d9adc2e9b30652 and WASM 5f0bb4a5810e9f72468b91339ac2b2f716281cfc76829daa5af7c9512edd64fe. All API, race, Flight, dedup, caps, pairing, link, watch, lease/idle, rotation, newcomer and persistence restart stages passed. Archive: e2e-final-combined.log and e2e-final-combined-worker-logs. Two separately counted local Wrangler transport retries; no application-status retries. Capacity proof: all 40 live, creation span 5258 ms, earliest expiry 594877 ms remaining, extra request 503. Public performance runs do not use this retry facility. |
|
Cross-platform chat acceptance depends on Furrow #23, which fixes materialization of a non-restorable Unix socket marker. The pinned 0.3.5 binary fails the minimal public reproduction; the reviewed candidate passes, including unchanged concurrent-writer and symlink protections. The candidate engine can be used for isolated end-to-end validation without merging or releasing it. This PR's dependency pin has not changed. Any successful candidate-engine handoff must be labelled as requiring that upstream fix; shipped 0.3.5 cannot be claimed to pass it. The acceptance gate is a concrete normal-app journey over the public internet: original device unavailable, real chat history and task state restored, a verified file read and edit on the receiving device, and successful publication. A download timing alone does not satisfy the gate. Intermittent live directory refusals also remain under investigation; CodeAF stays draft until repeated actual moves are reliable. |
|
Actual application acceptance update at The same private 2,278-message chat (~5.6 MB of work files, three saved tasks) was transferred through public HTTPS between two computers after stopping the source application and its background processes. Original user data was left unchanged. Both sides used the same clean native client and the explicitly recorded candidate for Furrow #23; this is not a result for the currently pinned engine alone.
These are one cold-receiver run per side, not medians or a repeatable speedup claim. Receivers had fresh disposable profiles; no chat payload or engine store was copied between computers outside the public relay. All 17 work files and the full transcript matched SHA-256; saved task contents matched after excluding derived local runtime paths. Git internals and those local paths were rebuilt during arrival. Fetch is nested in staging/preflight, so phase measurements must not all be added as disjoint stages. Cold movement made no new model calls. A separate edited-chat return succeeded on the clean candidate with the previous application stopped: 5.925 s application pipeline, all 44 workspace files, updated transcript and task state matched exactly, including the new file created on the other computer. The actual receiving agent read that file through its read tool and returned the correct contents. All five calls for this verification requested and actually served OpenRouter Repeated-move gate remains open. The next return completed download, acquisition and restoration but failed during local placement after 9.892 s. The production bucket calculation produced a 259-byte filename component ( The five-pair file-transfer measurements for c498 remain labeled c498. They will not be relabeled as bcc results. Final repeated measurements, first-model/tool latency, publication, error counts and architecture notes will be updated after the final client/relay candidate is validated. Conversation-first loading/prefetch is not implemented or measured yet. |
|
Actual repeated-move gate passes at client Six actual TUI moves alternated receiving computers, with the previous application and its background processes stopped before each confirmation. All used the same published head, without intervening model turns or file edits. After EVERY move, all 44 files, the complete transcript, and task state matched exactly. No continuation failed or required a retry.
Returning-device confirmation→visible: median 7.813 s, range 5.168–10.272 s, n=6. Separate from the single cold comparison. Some phases overlap; do not sum every column. Two driver setup faults before continuation began were preserved separately; earlier failed application attempts remain documented. After move six, the actual agent read the new verification file in the relocated workspace and returned its correct contents. First verified read-tool result: 3.964 s after prompt submission. Final reply: 7.929 s after submission. All four calls requested and actually served OpenRouter Hot fetch took 3.6–10.7 ms. The remaining cost includes directory authority and preserving apparent local changes. The omitted socket causes a false local-work plan; Furrow #23 now includes Final relay validation: 176/176 normal proof tests, full aggregate HTTP conformance, and current PR CI checks pass. Rotation kept its original two-second grace. The aggregate counted three known local transport retries separately and zero application-status retries; the earlier overloaded grace-window failure is preserved. No shared relay deployment or merge. |
Storage follow-up: balanced actual-chat timingA warm-cache comparison used the same published conversation snapshot, the same native client revision
All 12 moves completed exactly once, preserved workspace content outside derived Git internals, full conversation history and saved-task domain state, retained the externally pinned published head, and made zero model calls. Local runtime paths and Git internals are rebuilt, so this comparison does not claim byte identity for those derived files. Verdict: no demonstrated end-to-end speed improvement from the storage follow-up. Ranges overlap, and both arms still preserve a regular-file difference at The follow-up is justified by its public socket correctness regression and by removing a false same-head socket restore plan. It must not be advertised as a measured whole-journey speedup. Furrow CI passes on Linux, macOS and Windows. The final relay transfer comparison is running separately. Perceived-speed implication: hot snapshot fetch is already milliseconds. Selected-chat prefetch could hide cold fetch time, but it cannot remove authority checks or local-work preservation. Conversation-first selective network hydration and background file access are not implemented by these PRs. |
Final internet transfer comparison — clean candidateFull cold download and materialization: 9.578 → 3.423 seconds median (64.3% lower), five matched pairs. The ranges do not overlap.
Method and correctness. Two physical computers alternate sender direction across five baseline/candidate pairs. Each side starts a fresh disposable identity and transfers 2,000 high-entropy files totaling 52,428,800 plaintext bytes through public HTTPS. Baseline source is Each side completed 160 HTTP requests: all 2xx, zero request/body errors, with public-edge response identifiers on every completed request. No application-status retries. Another 40 requests per side were deliberately cancelled: baseline 22 non-success statuses/32 request-or-body errors; candidate 1 non-success/35 errors. These cancellation outcomes are excluded from completed-request success counts and are not hidden. Unchanged-head reuse makes zero payload requests; it is a cache check, not transfer throughput. Resumed downloads can redownload frames; no efficient byte-range-resume claim. Limiter and relevance. The cold payload-request span fell from median 8.329 to 1.715 seconds. Local seal and hash verification stayed similar. This supports less time awaiting relay/network operations, but does not separate WAN latency from server coordination. Independent source activity was recorded and pairs were interleaved. This is an actual full-file restoration benchmark; actual TUI chat movement and model/tool timing are recorded separately in the six-move acceptance and the balanced storage comparison. Usable transferred project. The actual copied web workspace was also served and opened in a real browser on the receiving device. All three HTML pages reached complete rendering, with no JavaScript exceptions, no external network requests and successful project-resource responses. The browser's optional favicon request returned 404; no project asset was missing. The restored JavaScript file passed its syntax check. Private screenshots are retained locally and are not published because they contain user workspace material. This functional check is separate from benchmark timing. Limits. Chat-size/team-size/task-count scaling and fleet concurrency are not established by these results. The real chat has 2,278 messages and three saved historical tasks. “Previous device unavailable” here means its complete disposable application process group was stopped, not physical power-off or a deliberately elected native-leader failure. |
|
@AbirAbbas please review this PR and Furrow #23 for merge. The review package is complete: final five-pair internet measurements, actual repeated chat movement and agent file actions, balanced storage comparison, and test-fixture/security rationale. Current CI passes on both PRs. Merge order matters: Furrow review/merge/release first; then update CodeAF's released-engine pin and rerun normal installed-engine acceptance before merging this PR. Native tests currently use the fixed Furrow candidate; the default 0.3.5 pin is unchanged. Raft mutation ordering, no-quorum refusal, revocation enforcement and concurrent-file protections remain required. The measured improvement is full restored file transfer. Actual chat movement and receiving-side read/edit/publish are validated, but instant conversation-first network hydration, arbitrary lazy file access and team-size scaling are not delivered by this PR. |
Summary
Cold file download and restoration drops from 9.58 to 3.42 seconds median across five matched public-internet pairs, with every received file verified.
Consensus remains the sole directory writer. Confirmed authority is invalidated before member work and activation changes; unavailable quorum fails closed. New read fallback restarts the complete operation and pins applied state. Fix watch registration races, exact link replay/preflight, interrupted-anchor recovery and oversized copied-workspace names.
Evidence
Merge Danger
Door: one-way
Singleton birth persists immutable provenance. Older consensus code refuses that configuration; rollback needs compatible consensus code.
Blast Radius: hosted
Changes authority reuse, first-use formation, presence and committed-change notifications. Shared development and production have not been changed.
Release integration gate: Furrow #23 must be reviewed, merged and released; then update CodeAF's currently unchanged 0.3.5 pin and verify the normal installed-engine path before merging this PR for native chat acceptance. Candidate-engine testing is not proof that the default packaged dependency is fixed. Request review now; do not bypass this release gate.