Skip to content

[rushd] Wire layer: protocol, transport & presentation (WS1) - #5922

Open
Sean Larkin (TheLarkInn) wants to merge 7 commits into
mainfrom
thelarkinn-rushd-wire-layer-ws1
Open

[rushd] Wire layer: protocol, transport & presentation (WS1)#5922
Sean Larkin (TheLarkInn) wants to merge 7 commits into
mainfrom
thelarkinn-rushd-wire-layer-ws1

Conversation

@TheLarkInn

Copy link
Copy Markdown
Member

Part of #5894 (rushd epic). Implements the WS1 workstream of #5896 on top of current main.

Note: two acceptance criteria are intentionally deferred until the @rushstack/reporter stack (#5865-#5868) merges - see "Deferred to reporter reconciliation" below. This PR is complete and green on its own; it does not close #5896.

Summary

rushd needs an engine-agnostic wire layer that every client speaks, with presentation moved out of the engine. This PR lands that layer as three new standalone 0.x packages (merged incrementally, per the epic's strategy) plus additive-only instrumentation in rush-lib:

  • @rushstack/rush-daemon-protocol - the frame taxonomy (0x01 control-json, 0x02/0x03 log-stdout/stderr, 0x04 stdin, 0x05 event), a length-prefixed binary codec with a split/coalesce-safe streaming decoder and max-frame guard, an exported DAEMON_PROTOCOL_VERSION, hello/helloAck version negotiation with a typed ProtocolVersionMismatchError, and a pure per-subscription verbosity filter applied at event serialization.
  • @rushstack/rush-daemon-transport - workspace-key hashing (sha256(canonicalRepoRoot + rushVersion + startupOptions), NUL-separated, order-insensitive options), per-user runtime-dir socket/pipe path derivation ($XDG_RUNTIME_DIR on POSIX, \\.\pipe\rushd-<key> on Windows), a net listener/connector with drain-aware backpressure, and PID/lockfile handling with two-factor stale-socket reclaim (dead PID and failed connect probe, so a live but busy daemon is never reclaimed underneath itself).
  • @rushstack/rush-terminal-renderer - the CLI client's presentation host: DaemonRendererHost hosts StreamCollator client-side for faithful per-operation collation (byte-parity with the legacy in-process pipeline, headers included), applies quiet/verbose per client without mutating shared state, and computes per-request FORCE_COLOR/COLUMNS for child spawns (non-TTY children get neither; ambient values are stripped first, so no cross-client contamination).
  • rush-lib (patch, additive): an optional internal IOperationGraphEventSink on OperationGraph - every operation registration, status transition, ==[ x of y ]== header, and Colorize.* status line dual-emits as a structured event (operation-scoped and stream-tagged exactly where legacy writes them), each operation's raw stdout/stderr is tapped id-tagged upstream of the quiet-mode discard, and an explicit stream-closed event marks end-of-output. Terminal output is byte-identical with or without a sink.
  • build-tests/rushd-wire-e2e-test - cross-layer conformance: engine dual-emit -> frames -> real socket pair -> renderer, asserting golden byte-parity with the in-process legacy output (normal and quiet, success and failure), exact per-operation raw bytes (unicode + ANSI), and per-client verbosity subsets.

Details

  • Reporter-free by design (for now). The reporter stack is still unmerged, so the 0x05 event frame carries a placeholder envelope that mirrors IReporterEventEnvelope field-for-field, plus the closed core event union and rushd.* namespaced extension validation. A type-level test pins the placeholder key set so the swap to a referenced @rushstack/reporter import is mechanical. The renderer's IDaemonRenderer mirrors IReporter's method signatures so the real default/ai/plaintext reporters drop in unchanged.
  • Stream ordering. Terminal status transitions fire before status-line text in the engine, while legacy closes the collated writer after; an explicit rushd.operation-stream-closed extension event carries the authoritative close, avoiding late-write ordering bugs.
  • Quiet fidelity. Legacy quiet mode discards operation stdout but still prints status lines; the wire models this by routing operation-scoped status lines as required scoped activityChanged events (never filtered) while the client drops raw stdout chunks at display time.
  • Strict codegen lint. The new packages share a strict-codegen ESLint mixin in local-node-rig (complexity 3, max-depth 3, 30-line functions, 100-line files, 4 params, no magic numbers, nullish coalescing, import hygiene, no eval) with noInlineConfig: true so eslint-disable comments are themselves errors, plus a per-package AGENTS.md codifying the no-suppression policy. Only rules already present in the repo's toolchain are enabled.
  • No new external runtime dependencies; new packages carry no rush-lib dependency. API Extractor reports are committed for all three.

Deferred to reporter reconciliation (after #5865-#5868 merge)

  • 0x05 payload becomes a direct IReporterEventEnvelope reference (acceptance item: "referenced, not re-declared").
  • The renderer hosts the reporter package's real default/ai/plaintext reporters (acceptance item: "no re-implementation").

How it was tested

  • New unit suites: protocol 29 tests (round-trip of all five frame types, non-UTF-8 losslessness, split/coalesce, oversized/unknown rejection, handshake match + typed mismatch, verbosity matrix, type-level contract pin), transport 17 tests (key stability/distinctness, POSIX+Windows path derivation, live socket exchange, stale reclaim vs live-daemon refusal, 16 MiB slow-consumer backpressure, negotiation over a real socket), renderer 10 tests (legacy byte-parity incl. quiet, per-client subsets, child-env matrix).
  • rush-lib: new OperationGraphEventSink suite (transitions, headers, activity, quiet tap, byte-identical output with/without sink); full existing suite 736/736 unchanged.
  • rush test --to rushd-wire-e2e-test green end to end; rush test --from @microsoft/rush-lib (147 operations across all dependents) green except pre-existing @rushstack/heft and @rushstack/node-core-library failures that reproduce identically on pristine main in this environment.
  • rush change --verify passes; change files included for all touched packages.

Impacted documentation

N/A

Add the engine-agnostic rushd wire layer as three new 0.x packages plus
additive rush-lib engine instrumentation:

- @rushstack/rush-daemon-protocol: frame taxonomy (0x01 control-json,
  0x02/0x03 log-stdout/stderr, 0x04 stdin, 0x05 event), length-prefixed
  binary codec, DAEMON_PROTOCOL_VERSION, hello/version negotiation with
  typed mismatch errors, and per-subscription verbosity filtering at
  serialization. Event envelope mirrors the @rushstack/reporter contract
  as a placeholder pending its merge (#5858).
- @rushstack/rush-daemon-transport: workspace-key hashing, per-user
  runtime-dir socket/pipe paths, net listener/connector with
  backpressure, and PID/lockfile stale-socket reclaim.
- @rushstack/rush-terminal-renderer: client reporter host with
  StreamCollator-backed per-op collation (byte-parity with legacy),
  per-client verbosity, and FORCE_COLOR/COLUMNS child-env threading.
- rush-lib: optional internal IOperationGraphEventSink dual-emit
  (structured status/activity/header events + per-op raw output tap)
  with byte-identical legacy output.
- build-tests/rushd-wire-e2e-test: cross-layer conformance suite
  (golden parity over a real socket, raw-stream integrity, verbosity
  isolation, failure propagation).
- rigs: shared strict-codegen ESLint mixin for the new packages, with
  inline suppression disabled (noInlineConfig).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Comment thread libraries/rush-lib/src/logic/operations/OperationEventSink.ts
Comment thread libraries/rush-daemon-transport/src/DaemonFrameConnection.ts Outdated
Comment thread libraries/rush-daemon-transport/src/DaemonReclaim.ts
Comment thread libraries/rush-daemon-protocol/src/DaemonEventFrameCodec.ts Outdated
Comment thread libraries/rush-daemon-protocol/src/DaemonFrame.ts Outdated
Comment thread libraries/rush-daemon-protocol/src/DaemonFrame.ts Outdated
Comment thread libraries/rush-daemon-protocol/src/DaemonFrameType.ts Outdated
Comment thread libraries/rush-daemon-protocol/src/DaemonControlMessage.ts
Comment thread libraries/rush-daemon-protocol/src/DaemonEventEnvelope.ts Outdated
Comment thread libraries/rush-daemon-protocol/AGENTS.md
Comment thread libraries/rush-daemon-protocol/eslint.config.js
Comment thread libraries/rush-daemon-protocol/LICENSE Outdated
Comment thread libraries/rush-daemon-protocol/package.json Outdated
Comment thread libraries/rush-daemon-protocol/tsconfig.json Outdated
Review pass over the wire-layer packages (#5922):

Protocol (@rushstack/rush-daemon-protocol):
- kind/payload naming throughout (frames + control messages); control
  messages are a uniform { kind, payload } discriminated union so kind
  reads stay monomorphic.
- Uint8Array wire payloads instead of Buffer, so the protocol is
  platform-agnostic and drops the @types/node peer dependency.
- FrameDecoder uses ECMAScript private fields and accumulates received
  bytes in a SegmentBuffer (no per-push Buffer.concat); payloads copy out
  once per completed frame.
- encodeDaemonFrames returns a Uint8Array[] (no batch concat); the
  transport writes parts sequentially.
- Containment checks use numeric ranges/Sets; the event-type list is
  as-const with the union derived from it (list and type cannot drift).
- Error codes are a plain string union; DaemonProtocolError accepts
  { cause } per the standard Error convention.
- decodeDaemonEventFrame structurally validates envelopes (typed error
  instead of routing malformed input); new isDaemonEventEnvelope guard.
- Envelope optional fields moved to the end of the layout; log chunk
  encoding measures the id once and allocates the payload once; TS target
  ES2022.

Transport (@rushstack/rush-daemon-transport):
- Decoder/handler failures in the socket callback now fail the connection
  closed instead of crashing the daemon (new ConnectionRobustness test).
- Reclaim is serialized through a dedicated .reclaim mutex (wx create,
  dead-PID steal) so a concurrent starter cannot unlink a socket another
  process just bound; the daemon lockfile is written after bind so a
  stale record never reads as a live owner.

Renderer/e2e: track the renames and Uint8Array decode boundary.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
rush retest runs with warnings-as-failures; the reviewer-requested
as-const literal lists tripped the friendly-locals @typescript-eslint/typedef
rule (variableDeclaration). Annotate DAEMON_EVENT_TYPES and
DAEMON_CONTROL_MESSAGE_KINDS with explicit literal-tuple types and derive the
unions from them, preserving the single-source-of-truth invariant (adding a
member requires updating the tuple annotation, which the compiler enforces via
the derived union) with zero lint warnings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
The renderer pipeline normalizes newlines to the OS default (CRLF on
Windows) via colorsNewlinesTransform, but the e2e test sink compared the
raw LF-carrying chunks, so the byte-parity assertions failed only on
Windows. Normalize the captured golden to the OS newline before
comparing so the test is platform-correct.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
The previous OS-newline normalization mapped every LF to CRLF, producing
CRCRLF on Windows for text that already carried CRLF. Normalize only
lone-LF newlines (leave existing CRLF intact) so the golden comparison is
correct on both platforms.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Collapse existing CRLF to LF before re-applying the OS newline, so the
golden comparison is correct on Windows (no CRCRLF) and a no-op on POSIX.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 47388e8f-8d41-4ca4-819b-688ea6c510a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

[rushd][WS1] Wire layer: protocol, transport & presentation

3 participants