[rushd] Wire layer: protocol, transport & presentation (WS1) - #5922
Open
Sean Larkin (TheLarkInn) wants to merge 7 commits into
Open
[rushd] Wire layer: protocol, transport & presentation (WS1)#5922Sean Larkin (TheLarkInn) wants to merge 7 commits into
Sean Larkin (TheLarkInn) wants to merge 7 commits into
Conversation
Add the engine-agnostic rushd wire layer as three new 0.x packages plus additive rush-lib engine instrumentation: - @rushstack/rush-daemon-protocol: frame taxonomy (0x01 control-json, 0x02/0x03 log-stdout/stderr, 0x04 stdin, 0x05 event), length-prefixed binary codec, DAEMON_PROTOCOL_VERSION, hello/version negotiation with typed mismatch errors, and per-subscription verbosity filtering at serialization. Event envelope mirrors the @rushstack/reporter contract as a placeholder pending its merge (#5858). - @rushstack/rush-daemon-transport: workspace-key hashing, per-user runtime-dir socket/pipe paths, net listener/connector with backpressure, and PID/lockfile stale-socket reclaim. - @rushstack/rush-terminal-renderer: client reporter host with StreamCollator-backed per-op collation (byte-parity with legacy), per-client verbosity, and FORCE_COLOR/COLUMNS child-env threading. - rush-lib: optional internal IOperationGraphEventSink dual-emit (structured status/activity/header events + per-op raw output tap) with byte-identical legacy output. - build-tests/rushd-wire-e2e-test: cross-layer conformance suite (golden parity over a real socket, raw-stream integrity, verbosity isolation, failure propagation). - rigs: shared strict-codegen ESLint mixin for the new packages, with inline suppression disabled (noInlineConfig). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Review pass over the wire-layer packages (#5922): Protocol (@rushstack/rush-daemon-protocol): - kind/payload naming throughout (frames + control messages); control messages are a uniform { kind, payload } discriminated union so kind reads stay monomorphic. - Uint8Array wire payloads instead of Buffer, so the protocol is platform-agnostic and drops the @types/node peer dependency. - FrameDecoder uses ECMAScript private fields and accumulates received bytes in a SegmentBuffer (no per-push Buffer.concat); payloads copy out once per completed frame. - encodeDaemonFrames returns a Uint8Array[] (no batch concat); the transport writes parts sequentially. - Containment checks use numeric ranges/Sets; the event-type list is as-const with the union derived from it (list and type cannot drift). - Error codes are a plain string union; DaemonProtocolError accepts { cause } per the standard Error convention. - decodeDaemonEventFrame structurally validates envelopes (typed error instead of routing malformed input); new isDaemonEventEnvelope guard. - Envelope optional fields moved to the end of the layout; log chunk encoding measures the id once and allocates the payload once; TS target ES2022. Transport (@rushstack/rush-daemon-transport): - Decoder/handler failures in the socket callback now fail the connection closed instead of crashing the daemon (new ConnectionRobustness test). - Reclaim is serialized through a dedicated .reclaim mutex (wx create, dead-PID steal) so a concurrent starter cannot unlink a socket another process just bound; the daemon lockfile is written after bind so a stale record never reads as a live owner. Renderer/e2e: track the renames and Uint8Array decode boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
rush retest runs with warnings-as-failures; the reviewer-requested as-const literal lists tripped the friendly-locals @typescript-eslint/typedef rule (variableDeclaration). Annotate DAEMON_EVENT_TYPES and DAEMON_CONTROL_MESSAGE_KINDS with explicit literal-tuple types and derive the unions from them, preserving the single-source-of-truth invariant (adding a member requires updating the tuple annotation, which the compiler enforces via the derived union) with zero lint warnings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
The renderer pipeline normalizes newlines to the OS default (CRLF on Windows) via colorsNewlinesTransform, but the e2e test sink compared the raw LF-carrying chunks, so the byte-parity assertions failed only on Windows. Normalize the captured golden to the OS newline before comparing so the test is platform-correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
The previous OS-newline normalization mapped every LF to CRLF, producing CRCRLF on Windows for text that already carried CRLF. Normalize only lone-LF newlines (leave existing CRLF intact) so the golden comparison is correct on both platforms. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Collapse existing CRLF to LF before re-applying the OS newline, so the golden comparison is correct on Windows (no CRCRLF) and a no-op on POSIX. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #5894 (rushd epic). Implements the WS1 workstream of #5896 on top of current
main.Summary
rushd needs an engine-agnostic wire layer that every client speaks, with presentation moved out of the engine. This PR lands that layer as three new standalone 0.x packages (merged incrementally, per the epic's strategy) plus additive-only instrumentation in rush-lib:
@rushstack/rush-daemon-protocol- the frame taxonomy (0x01control-json,0x02/0x03log-stdout/stderr,0x04stdin,0x05event), a length-prefixed binary codec with a split/coalesce-safe streaming decoder and max-frame guard, an exportedDAEMON_PROTOCOL_VERSION, hello/helloAck version negotiation with a typedProtocolVersionMismatchError, and a pure per-subscription verbosity filter applied at event serialization.@rushstack/rush-daemon-transport- workspace-key hashing (sha256(canonicalRepoRoot + rushVersion + startupOptions), NUL-separated, order-insensitive options), per-user runtime-dir socket/pipe path derivation ($XDG_RUNTIME_DIRon POSIX,\\.\pipe\rushd-<key>on Windows), anetlistener/connector with drain-aware backpressure, and PID/lockfile handling with two-factor stale-socket reclaim (dead PID and failed connect probe, so a live but busy daemon is never reclaimed underneath itself).@rushstack/rush-terminal-renderer- the CLI client's presentation host:DaemonRendererHosthostsStreamCollatorclient-side for faithful per-operation collation (byte-parity with the legacy in-process pipeline, headers included), applies quiet/verbose per client without mutating shared state, and computes per-requestFORCE_COLOR/COLUMNSfor child spawns (non-TTY children get neither; ambient values are stripped first, so no cross-client contamination).IOperationGraphEventSinkonOperationGraph- every operation registration, status transition,==[ x of y ]==header, andColorize.*status line dual-emits as a structured event (operation-scoped and stream-tagged exactly where legacy writes them), each operation's raw stdout/stderr is tapped id-tagged upstream of the quiet-mode discard, and an explicit stream-closed event marks end-of-output. Terminal output is byte-identical with or without a sink.build-tests/rushd-wire-e2e-test- cross-layer conformance: engine dual-emit -> frames -> real socket pair -> renderer, asserting golden byte-parity with the in-process legacy output (normal and quiet, success and failure), exact per-operation raw bytes (unicode + ANSI), and per-client verbosity subsets.Details
0x05event frame carries a placeholder envelope that mirrorsIReporterEventEnvelopefield-for-field, plus the closed core event union andrushd.*namespaced extension validation. A type-level test pins the placeholder key set so the swap to a referenced@rushstack/reporterimport is mechanical. The renderer'sIDaemonRenderermirrorsIReporter's method signatures so the realdefault/ai/plaintextreporters drop in unchanged.rushd.operation-stream-closedextension event carries the authoritative close, avoiding late-write ordering bugs.requiredscopedactivityChangedevents (never filtered) while the client drops raw stdout chunks at display time.strict-codegenESLint mixin inlocal-node-rig(complexity 3, max-depth 3, 30-line functions, 100-line files, 4 params, no magic numbers, nullish coalescing, import hygiene, no eval) withnoInlineConfig: truesoeslint-disablecomments are themselves errors, plus a per-packageAGENTS.mdcodifying the no-suppression policy. Only rules already present in the repo's toolchain are enabled.rush-libdependency. API Extractor reports are committed for all three.Deferred to reporter reconciliation (after #5865-#5868 merge)
0x05payload becomes a directIReporterEventEnvelopereference (acceptance item: "referenced, not re-declared").default/ai/plaintextreporters (acceptance item: "no re-implementation").How it was tested
OperationGraphEventSinksuite (transitions, headers, activity, quiet tap, byte-identical output with/without sink); full existing suite 736/736 unchanged.rush test --to rushd-wire-e2e-testgreen end to end;rush test --from @microsoft/rush-lib(147 operations across all dependents) green except pre-existing@rushstack/heftand@rushstack/node-core-libraryfailures that reproduce identically on pristinemainin this environment.rush change --verifypasses; change files included for all touched packages.Impacted documentation
N/A