Skip to content

Speed up hosted relay transfers while preserving Raft authority - #2162

Open
santoshkumarradha wants to merge 11 commits into
devfrom
fix/hosted-relay-fastpath
Open

santoshkumarradha wants to merge 11 commits into
devfrom
fix/hosted-relay-fastpath

Conversation

@santoshkumarradha

@santoshkumarradha santoshkumarradha commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Cold file download and restoration drops from 9.58 to 3.42 seconds median across five matched public-internet pairs, with every received file verified.

mutation → Raft commit → durable apply → revoke watches / held polls
read → confirmed applied authority → local view
interrupted authority → fresh safe ReadIndex → restart complete read
blob I/O → authorize once → applied-index / authority recheck
presence → bounded coalescing → asynchronous maintenance
fresh empty identity → singleton commit → background joint-consensus formation
repeated chat movement → bounded destination names → restored workspace

Consensus remains the sole directory writer. Confirmed authority is invalidated before member work and activation changes; unavailable quorum fails closed. New read fallback restarts the complete operation and pins applied state. Fix watch registration races, exact link replay/preflight, interrupted-anchor recovery and oversized copied-workspace names.

Evidence

  • Before: baseline cold restoration median 9.578 seconds, range 9.423–10.360. After: 3.423 seconds, range 3.093–3.923. Five matched pairs transfer 2,000 high-entropy files / 50 MiB between two physical computers through public HTTPS. Each side completes 160 HTTP operations, all successful with zero errors; intentional cancellation is counted separately. Released Furrow 0.3.5 is identical on both file-transfer sides. All stages, ranges and method.
  • A real 2,278-message chat with three saved historical tasks moves back and forth six times with the previous application's process group stopped. All 44 workspace-file hashes, full history and saved task state match on every move. Actual agent read/edit/publish and receiving-side readback pass; the transferred web project renders on the receiving device. These native tests use the recorded Furrow design language: the ground ladder and the calm palette (contract) #23 candidate. Actual chat acceptance.
  • Socket-plan follow-up is a correctness fix: its balanced 12-move comparison demonstrates no end-to-end latency improvement. Git differences are still preserved. No selected-chat prefetch, selective lazy network hydration, fleet-concurrency or independent team-size scaling claim. Balanced comparison.
  • Validation: 248 unit tests, 176 normal relay proofs, separate actor proof, complete HTTP E2E including unchanged lease renewal/expiry, revocation, rotation and restart persistence; final native continuation regression and PR CI pass. Three known local conformance-transport losses were retried separately; application statuses were not retried. Security-property assertions remain unchanged. Fixture mechanism changes and preserved contracts.

Merge Danger

Door: one-way

Singleton birth persists immutable provenance. Older consensus code refuses that configuration; rollback needs compatible consensus code.

Blast Radius: hosted

Changes authority reuse, first-use formation, presence and committed-change notifications. Shared development and production have not been changed.

Release integration gate: Furrow #23 must be reviewed, merged and released; then update CodeAF's currently unchanged 0.3.5 pin and verify the normal installed-engine path before merging this PR for native chat acceptance. Candidate-engine testing is not proof that the default packaged dependency is fixed. Request review now; do not bypass this release gate.

Copy link
Copy Markdown
Member Author

Agent review round 1 — architecture

Reviewed the design against the quorum/revocation ruling. Rejected a direct local directory writer: concurrent identity writers still require consensus. The implementation keeps mutation commits and uses an invalidated, short-lived confirmation of clean applied state for reads. Fresh singleton provenance is immutable; an established voting configuration cannot be reduced during recovery. No remaining architecture blocker was found within this review.

Copy link
Copy Markdown
Member Author

Agent review round 2 — correctness

Review found that a compacted snapshot could exclude a newly admitted learner, causing repeated full snapshot transfers. Learner admission now refreshes the snapshot before catch-up and retains the resulting durable effects. The archived large-body cases pass, including restart and staging cuts. Additional checks cover every formation turn and refusal to recreate a lost published member. Reviewed production revision: a8005c4.

Copy link
Copy Markdown
Member Author

Agent review round 3 — security and cost

Review found two gaps: a replicated revoke could bypass public-handler watch closure, and an immediately rejected upload could release its in-flight slot before storage finished. Committed-apply notifications now close watches and held polls; upload tracking lasts until actual handler completion. Both findings have red-to-green regressions. Existing quorum refusal, replay and revocation assertions remain. Validation at a8005c4: 166 relay proofs, the separate actor proof, 248 unit tests, and fresh affected Go-package checks pass. Performance acceptance and remaining end-to-end fixture work are still in progress; these agent reviews do not authorize merging.

Copy link
Copy Markdown
Member Author

Agent follow-up review of ca15a2543a → b2a0a59d9e: Standards 0 findings; Spec 0 findings. The transport change removes recovery reports only for anchors absent from the actual durable configuration. Configured voters, learners, outgoing voters, and next learners retain failure reporting; unavailable evidence refuses recovery rather than guessing membership. Authority checks and required consensus barriers remain intact. Unit suite: 248/248 passed. Fresh public-internet before/after transfer measurements for this revision are in progress; prior revision measurements are not treated as evidence for the new revision. This review is not merge authorization.

Copy link
Copy Markdown
Member Author

Acceptance update: the full normal relay proof suite passes 167/167 against afcd239b8 (including the original quorum, revocation, replay, and ownership assertions). Unit suite: 248/248. The maintenance-read quiescence regression was observed failing before its fix and the original cold-sleep journey then passed 10 fresh runs.

A live two-device chat continuation succeeded after abruptly stopping the source application, preserving history and a hash-verified workspace file. However, a larger real-conversation test has exposed intermittent 503 responses during the initial directory read, before any file transfer, including samples while the source is still running. This remains an acceptance blocker; the PR stays draft and is not ready to merge.

The earlier five matched public-internet transfer pairs for b2a0a59d9e verified all 2,000 files / 50 MiB on both sides. Median full download/materialization was 8.816s → 3.330s (ranges 8.075–9.861s / 2.823–3.648s). These numbers belong to that measured revision; they are not a final performance claim for the current head. Completed HTTP operations had zero errors; deliberate cancellation outcomes were counted separately. Final measurements will follow the whole device-switching journey, including time to the first usable file action.

Copy link
Copy Markdown
Member Author

Acceptance update for 4b9e716250:

  • 248 unit tests, 173 relay proof tests, and the separate actor proof pass. The aggregate HTTP E2E run also passes, including the unchanged lease renewal/expiry test, idle socket handling, rotation erasure, and real process-restart persistence checks. Two known local development-proxy connection-loss retries were counted separately; application statuses were never retried.
  • Live public-internet continuation with the previous device stopped is still not accepted. A copied real chat has 2,278 messages and approximately 5.6 MB of workspace files. One diagnostic attempt completed download in 3.377 seconds and ownership acquisition in 0.589 seconds, but restoration refused; total time to refusal was 6.793 seconds. These are diagnostic phase measurements of a failed attempt, not a successful handoff latency or comparative speed claim.
  • A minimal reproduction with the pinned Furrow 0.3.5 engine independently fails when materializing a snapshot containing a Unix socket marker into an empty destination. A separate engine fix is being prepared with the existing file and symlink interference protections retained. This PR remains draft until the real chat opens and a filesystem action succeeds.
  • Five alternating before/after public-internet transfer pairs are being repeated against the final relay implementation. Earlier measurements from b2a0a59d9e are historical and are not attributed to the current revision.

Mutations remain consensus commits. The live read-interruption fix retries only marked read-certificate interruptions, restarts the whole directory read, and never replays a mutation or blob I/O. Quorum loss, expired authority, activation fencing, lost-store and revocation assertions pass unchanged.

Copy link
Copy Markdown
Member Author

Test mechanisms and preserved contracts

The fixture changes do not relax quorum, revocation, replay, fencing, memory admission or erasure assertions. They remove outdated setup and transport assumptions so those properties run against the consensus implementation.

Fixture Mechanism correction Property retained or strengthened
Local E2E harness Build the shared WASM once; isolate independent scenarios in fresh persistence namespaces so old alarms cannot reactivate unrelated groups. Explicit restart tests retain their original namespace. Original application limits remain unchanged; process-restart durability still crosses real worker restarts. Opt-in development-proxy retries match only its known connection-loss error, never application responses; public measurements use no such retries.
API/race and watch inputs Forced takeover supplies the observed holder/fence; revoke and archive send the empty body required by the unchanged V1 decoder. Link timestamp forgery uses the actual device fields, and repeated approvals reuse the same grant. Original success, stale-fence, exact version/event and close-code assertions remain. Additional assertions reject blind takeover, changed-grant replay and invalid approvals that could reserve another identity's pending request.
Flight and link preparation Prepare identities before beginning the held-body stopwatch or short-lived link request. Original isolation threshold, upload barrier, authorization, expiry and refusal assertions remain unchanged.
Link capacity Only the isolated capacity fixture uses the production pending-request TTL. Its prior three-second fixture expired requests while forty durable creations took roughly five seconds, so an additional create correctly succeeded. Keys are prepared first, all forty requests must still be live, and the next create must return 503/full. Capacity 40, per-IP distribution, request rates, all 201 assertions and 503/full remain. Every other link fixture, including expiry, retains its original three-second TTL. The capacity fixture adds an explicit all-forty-live assertion.
Watch transport and handshake Use the existing WebSocket client with a 200 ms TCP teardown guard; keep actual server close codes. Re-pin only two legacy-vouch header assertions to 101, presence support and no legacy socket-vouch claim, matching the existing consensus watcher. Disabling real 4401 server closes leaves the original assertion pending; the timer cannot synthesize success. All event/version/4401/4410 assertions, hold parsing and the real lease renewal and expiry test lasting 217 seconds remain.

The separate Raft fixtures explicitly form the existing three-voter topology before injecting faults intended for that topology. Cold-birth capacity accounting counts actual singleton voters without lifting its memory budget; restored groups still require their actual quorum. Crash cut markers follow the same durable boundaries. Replay capture begins before the first request and preserves the original command across formation, avoiding a vacuous replay assertion. Test doubles add only the new presence/read-fence interface members. New cases verify lease expiry, activation fencing, stale signatures, off-path presence, singleton formation/recovery and committed revoke interruption through observable interfaces.

Watch E2E exposed and fixed two implementation errors rather than changing their assertions: duplicate direct-path version broadcasts, and closing rotated sockets before sending their final committed version. The committed-apply notifier now owns one version notification before closure; successful direct revokes also send the existing revoked event. Generic replicated revoke closure and held-I/O interruption remain intact.

The aggregate final HTTP E2E suite passed on the combined source at 4b9e716250, including the unchanged lease expiry, rotation erasure and real process-restart checks. Two known local development-proxy connection-loss retries were counted separately. No application statuses were retried. The final readiness gate also passed at that revision with the exact pinned engine.

Final aggregate verification (2026-11-02): complete HTTP E2E exit 0 against frozen combined worker SHA256 21c0b8ace6503589454b59ecd7c440dba6fc89ec0ea521c1d2d9adc2e9b30652 and WASM 5f0bb4a5810e9f72468b91339ac2b2f716281cfc76829daa5af7c9512edd64fe. All API, race, Flight, dedup, caps, pairing, link, watch, lease/idle, rotation, newcomer and persistence restart stages passed. Archive: e2e-final-combined.log and e2e-final-combined-worker-logs. Two separately counted local Wrangler transport retries; no application-status retries. Capacity proof: all 40 live, creation span 5258 ms, earliest expiry 594877 ms remaining, extra request 503. Public performance runs do not use this retry facility.

Copy link
Copy Markdown
Member Author

Cross-platform chat acceptance depends on Furrow #23, which fixes materialization of a non-restorable Unix socket marker. The pinned 0.3.5 binary fails the minimal public reproduction; the reviewed candidate passes, including unchanged concurrent-writer and symlink protections.

The candidate engine can be used for isolated end-to-end validation without merging or releasing it. This PR's dependency pin has not changed. Any successful candidate-engine handoff must be labelled as requiring that upstream fix; shipped 0.3.5 cannot be claimed to pass it.

The acceptance gate is a concrete normal-app journey over the public internet: original device unavailable, real chat history and task state restored, a verified file read and edit on the receiving device, and successful publication. A download timing alone does not satisfy the gate. Intermittent live directory refusals also remain under investigation; CodeAF stays draft until repeated actual moves are reliable.

Copy link
Copy Markdown
Member Author

Actual application acceptance update at bcc181b877 (still draft; repeated switching is not yet accepted).

The same private 2,278-message chat (~5.6 MB of work files, three saved tasks) was transferred through public HTTPS between two computers after stopping the source application and its background processes. Original user data was left unchanged. Both sides used the same clean native client and the explicitly recorded candidate for Furrow #23; this is not a result for the currently pinned engine alone.

Measured milestone Baseline 20eb6952de Candidate bcc181b877
Confirm move → history visible in actual TUI 9.469 s 5.639 s
Application continuation pipeline 9.300 s 5.418 s
Directory lookup 3.001 s 0.891 s
Fetch 4.190 s 1.980 s
Acquire ownership 0.854 s 0.735 s
Restore 0.186 s 0.336 s
Arrival hooks 0.746 s 0.400 s
Filing 0.243 s 0.340 s

These are one cold-receiver run per side, not medians or a repeatable speedup claim. Receivers had fresh disposable profiles; no chat payload or engine store was copied between computers outside the public relay. All 17 work files and the full transcript matched SHA-256; saved task contents matched after excluding derived local runtime paths. Git internals and those local paths were rebuilt during arrival. Fetch is nested in staging/preflight, so phase measurements must not all be added as disjoint stages. Cold movement made no new model calls.

A separate edited-chat return succeeded on the clean candidate with the previous application stopped: 5.925 s application pipeline, all 44 workspace files, updated transcript and task state matched exactly, including the new file created on the other computer. The actual receiving agent read that file through its read tool and returned the correct contents. All five calls for this verification requested and actually served OpenRouter deepseek/deepseek-v4.1-flash, HTTP 200 with no errors. This return run has no valid click-to-visible measurement: its observer missed the arrival modal; the pipeline duration and restored data are independently verified. The earlier actual write/readback used c498 with diagnostic instrumentation and is a separate functional result.

Repeated-move gate remains open. The next return completed download, acquisition and restoration but failed during local placement after 9.892 s. The production bucket calculation produced a 259-byte filename component (ENAMETOOLONG). This is being isolated at the existing real continuation seam; it must be fixed and repeated moves verified before calling this smooth. Failed runs remain in the evidence and are not replaced with successful retries.

The five-pair file-transfer measurements for c498 remain labeled c498. They will not be relabeled as bcc results. Final repeated measurements, first-model/tool latency, publication, error counts and architecture notes will be updated after the final client/relay candidate is validated. Conversation-first loading/prefetch is not implemented or measured yet.

Copy link
Copy Markdown
Member Author

Actual repeated-move gate passes at client e888e8420c, clean relay bcc181b877, and Furrow candidate 0cdbe22 (dependency unmerged/unreleased).

Six actual TUI moves alternated receiving computers, with the previous application and its background processes stopped before each confirmation. All used the same published head, without intervening model turns or file edits. After EVERY move, all 44 files, the complete transcript, and task state matched exactly. No continuation failed or required a retry.

Move Confirm→visible Pipeline Directory Fetch Acquire Restore Hooks Filing Dirty Kept upload
1 9.463 s 9.084 s 1.598 s 0.007 s 0.767 s 0.323 s 0.547 s 0.486 s 1.256 s 3.777 s
2 10.272 s 9.912 s 4.156 s 0.004 s 0.694 s 0.453 s 0.217 s 0.540 s 0.237 s 3.135 s
3 7.399 s 6.996 s 1.250 s 0.011 s 0.958 s 0.314 s 0.351 s 1.231 s 1.325 s 1.251 s
4 6.290 s 6.083 s 1.010 s 0.004 s 0.568 s 0.403 s 0.685 s 0.507 s 0.227 s 2.210 s
5 8.227 s 7.856 s 2.385 s 0.009 s 0.773 s 0.330 s 0.612 s 0.628 s 1.291 s 1.521 s
6 5.168 s 4.954 s 0.531 s 0.004 s 0.566 s 0.395 s 0.432 s 1.135 s 0.162 s 1.230 s

Returning-device confirmation→visible: median 7.813 s, range 5.168–10.272 s, n=6. Separate from the single cold comparison. Some phases overlap; do not sum every column. Two driver setup faults before continuation began were preserved separately; earlier failed application attempts remain documented.

After move six, the actual agent read the new verification file in the relocated workspace and returned its correct contents. First verified read-tool result: 3.964 s after prompt submission. Final reply: 7.929 s after submission. All four calls requested and actually served OpenRouter deepseek/deepseek-v4.1-flash, HTTP200, no errors. Operator delay before submitting is excluded; these are not click-to-answer times.

Hot fetch took 3.6–10.7 ms. The remaining cost includes directory authority and preserving apparent local changes. The omitted socket causes a false local-work plan; Furrow #23 now includes 25beda15 to remove that plan while retaining real file, symlink and metadata differences. Public regression RED→GREEN, unchanged interference protections and optimized same-head checks pass. Actual timing improvement is not yet claimed; balanced measurements are next.

Final relay validation: 176/176 normal proof tests, full aggregate HTTP conformance, and current PR CI checks pass. Rotation kept its original two-second grace. The aggregate counted three known local transport retries separately and zero application-status retries; the earlier overloaded grace-window failure is preserved. No shared relay deployment or merge.

Copy link
Copy Markdown
Member Author

Storage follow-up: balanced actual-chat timing

A warm-cache comparison used the same published conversation snapshot, the same native client revision e888e8420c, optimized immutable engine binaries, and an ABBA schedule repeated three times. Receiving platforms alternate: six moves per engine, three per platform. Each timed move stopped the previous application's disposable process group before confirmation.

Engine Moves Confirmation → chat visible, median Range
Initial socket restoration fix, 0cdbe22 6 5.491 s 4.209–6.695 s
Socket-plan follow-up, 25beda1 6 6.037 s 5.782–8.347 s

All 12 moves completed exactly once, preserved workspace content outside derived Git internals, full conversation history and saved-task domain state, retained the externally pinned published head, and made zero model calls. Local runtime paths and Git internals are rebuilt, so this comparison does not claim byte identity for those derived files.

Verdict: no demonstrated end-to-end speed improvement from the storage follow-up. Ranges overlap, and both arms still preserve a regular-file difference at .git/COMMIT_EDITMSG; the difference can be content or metadata. No Git files were ignored to force a faster result. Shared host load was recorded.

The follow-up is justified by its public socket correctness regression and by removing a false same-head socket restore plan. It must not be advertised as a measured whole-journey speedup. Furrow CI passes on Linux, macOS and Windows. The final relay transfer comparison is running separately.

Perceived-speed implication: hot snapshot fetch is already milliseconds. Selected-chat prefetch could hide cold fetch time, but it cannot remove authority checks or local-work preservation. Conversation-first selective network hydration and background file access are not implemented by these PRs.

Copy link
Copy Markdown
Member Author

Final internet transfer comparison — clean candidate

Full cold download and materialization: 9.578 → 3.423 seconds median (64.3% lower), five matched pairs. The ranges do not overlap.

Stage Baseline median [min–max], seconds Candidate median [min–max], seconds Runs per side
First identity setup 6.325 [5.869–7.457] 2.624 [2.006–3.841] 5
Seal locally 0.776 [0.740–0.804] 0.795 [0.715–0.866] 5
Upload and publish 28.929 [24.725–30.493] 17.277 [13.555–22.381] 5
Full cold download + materialize 9.578 [9.423–10.360] 3.423 [3.093–3.923] 5
Resume after deliberate interruption 12.746 [11.574–14.832] 3.546 [2.563–3.668] 5
Receiving device registration 0.855 [0.633–1.297] 0.788 [0.623–1.655] 10
Unchanged-head local reuse 0.898 [0.871–1.089] 0.976 [0.886–1.081] 10
Unchanged upload 0.068 [0.045–0.112] 0.064 [0.045–0.115] 5

Method and correctness. Two physical computers alternate sender direction across five baseline/candidate pairs. Each side starts a fresh disposable identity and transfers 2,000 high-entropy files totaling 52,428,800 plaintext bytes through public HTTPS. Baseline source is 20eb6952de; candidate is e888e8420c, whose hosted production bytes are unchanged from bcc181b877. Both file-transfer sides use released Furrow 0.3.5, separating the relay comparison from the native-chat socket dependency. The candidate deployment stayed unchanged throughout. Every cold, recovered and repeated fetch verifies every file by SHA-256.

Each side completed 160 HTTP requests: all 2xx, zero request/body errors, with public-edge response identifiers on every completed request. No application-status retries. Another 40 requests per side were deliberately cancelled: baseline 22 non-success statuses/32 request-or-body errors; candidate 1 non-success/35 errors. These cancellation outcomes are excluded from completed-request success counts and are not hidden.

Unchanged-head reuse makes zero payload requests; it is a cache check, not transfer throughput. Resumed downloads can redownload frames; no efficient byte-range-resume claim.

Limiter and relevance. The cold payload-request span fell from median 8.329 to 1.715 seconds. Local seal and hash verification stayed similar. This supports less time awaiting relay/network operations, but does not separate WAN latency from server coordination. Independent source activity was recorded and pairs were interleaved. This is an actual full-file restoration benchmark; actual TUI chat movement and model/tool timing are recorded separately in the six-move acceptance and the balanced storage comparison.

Usable transferred project. The actual copied web workspace was also served and opened in a real browser on the receiving device. All three HTML pages reached complete rendering, with no JavaScript exceptions, no external network requests and successful project-resource responses. The browser's optional favicon request returned 404; no project asset was missing. The restored JavaScript file passed its syntax check. Private screenshots are retained locally and are not published because they contain user workspace material. This functional check is separate from benchmark timing.

Limits. Chat-size/team-size/task-count scaling and fleet concurrency are not established by these results. The real chat has 2,278 messages and three saved historical tasks. “Previous device unavailable” here means its complete disposable application process group was stopped, not physical power-off or a deliberately elected native-leader failure.

@santoshkumarradha santoshkumarradha changed the title Amortize hosted relay authorization and defer peer formation Speed up hosted relay transfers while preserving Raft authority Nov 2, 2026
@santoshkumarradha
santoshkumarradha marked this pull request as ready for review November 2, 2026 18:54

Copy link
Copy Markdown
Member Author

@AbirAbbas please review this PR and Furrow #23 for merge. The review package is complete: final five-pair internet measurements, actual repeated chat movement and agent file actions, balanced storage comparison, and test-fixture/security rationale. Current CI passes on both PRs.

Merge order matters: Furrow review/merge/release first; then update CodeAF's released-engine pin and rerun normal installed-engine acceptance before merging this PR. Native tests currently use the fixed Furrow candidate; the default 0.3.5 pin is unchanged. Raft mutation ordering, no-quorum refusal, revocation enforcement and concurrent-file protections remain required.

The measured improvement is full restored file transfer. Actual chat movement and receiving-side read/edit/publish are validated, but instant conversation-first network hydration, arbitrary lazy file access and team-size scaling are not delivered by this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants