feat(plugin)!: Add Switchyard Decision API routing plugin

[bbednarski9]

Overview

This adds the Relay-side P0 integration between NeMo Relay and a Switchyard remote Decision API.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Summary

• Add a standalone nemo-relay-switchyard first-party plugin with no Switchyard Rust dependency.
• Normalize stable Relay identity and build the current switchyard.routing_request.v1 contract.
• Validate switchyard.routing_decision.v1 responses against exact Relay-owned backend bindings.
• Support enforce and observe-only routing, six request materialization modes, bounded retry-aware dispatch, per-protocol trusted fallback, and strict stream commit semantics.
• Translate buffered and incremental SSE traffic across OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages while preserving same-protocol payloads.
• Extend ATOF endpoint secrets, routing marks, structured upstream failures, reserved dispatch-header hardening, CLI validation, doctor checks, and the interactive plugin editor.
• Add a CI-safe process-boundary E2E, cumulative-Switchyard compatibility E2E, and deterministic two-model Hermes/Ollama Cascade trajectory with ATOF, ATIF, and OTLP artifacts.

Details

The Decision API selects only a configured backend ID and exact model/protocol/endpoint tuple. Relay retains ownership of provider URLs, credentials, request/response translation, retries, and final trusted fallbacks. Decision failures, malformed responses, binding drift, unsupported provider extensions, and missing required identity all fail open to the same inbound protocol.

context_mode = "atof_required" requires Relay's existing ATOF HTTP exporter to target Switchyard's /v1/atof/events endpoint with preserved field names and environment-referenced authentication. The plugin emits best-effort canonical switchyard.routing.* marks but does not install a duplicate event subscriber or Relay-side feature accumulator.

The Hermes smoke runs a fixed three-query session through local Ollama and verifies llama3.2:latest → qwen3.6:35b → llama3.2:latest. Because the current Cascade Decision API classifies from accumulated Relay snapshots rather than current_request.body, the smoke inserts clearly labeled canonical tool-result ATOF fixtures between queries: a critical failure drives the strong override and a clean-test result drives the final weak classifier decision. Each fixture and its ingestion report is preserved with the trajectory.

The smoke writes:

• one cumulative ATOF JSONL stream plus per-query ATOF segments and ingestion reports;
• one ATIF trajectory per fixed query;
• OTLP JSON through the OpenTelemetry Collector while forwarding the same spans to Phoenix;
• a machine-readable trajectory summary containing the expected/actual route sequence and decision reasons;
• a compressed shareable bundle with a generated README mapping every file to the behavior it verifies and explicitly labeling the intentional OOM fixture.

Where should the reviewer start?

1. crates/switchyard/src/component.rs — lifecycle, validation, Decision API calls, retries, fallback, and marks.
2. crates/switchyard/src/translation.rs and stream_translation.rs — the buffered and SSE 3×3 protocol matrices, including the standard OpenAI stream_options.include_usage transport hint.
3. crates/cli/src/gateway.rs — reserved routed-dispatch headers and structured retry-aware provider failures.
4. crates/cli/tests/switchyard_process_e2e.rs — CI-safe process-boundary coverage for routing, streaming, fail-open, and reserved-header stripping.
5. examples/switchyard/run-real-e2e.sh and run-hermes-ollama-smoke.sh — cumulative Switchyard compatibility and shareable local trajectories.

The commits separate runtime/ATOF prerequisites, gateway retry infrastructure, plugin implementation, and E2E/trajectory work.

Review guidelines

• Verify the plugin remains contract-only and does not introduce local Switchyard routers, accumulators, or Switchyard crate dependencies.
• Check fail-open boundaries, especially unsupported extensions and the no-retry-after-first-stream-item rule.
• Check that all target fields are validated exactly and internal dispatch headers cannot cross the gateway trust boundary.
• Review environment-secret activation failures and the atof_required cross-component validation together.
• Treat protocol translation and retry behavior as one atomic routing transaction: one Decision API call per provider attempt, followed by at most one trusted fallback.
• For the trajectory smoke, distinguish user-query decisions from extra Hermes tool/summary calls using the recorded per-query ATOF line ranges.

Testing summary

• cargo fmt --all -- --check
• cargo clippy -p nemo-relay-switchyard -p nemo-relay-cli --all-targets -- -D warnings
• cargo test --workspace (all workspace unit, integration, and doc tests passed)
• cargo test -p nemo-relay-switchyard (22 passed)
• cargo test -p nemo-relay-cli --test switchyard_process_e2e (buffered, SSE, Decision API fail-open, and reserved headers)
• examples/switchyard/run-real-e2e.sh (weak → strong → strong, including SSE)
• examples/switchyard/run-hermes-ollama-smoke.sh (weak → strong → weak across two Ollama models)
• Generated trajectory: 4,994 ATOF events, 10 canonical routing marks, three ATIF files, and six OTLP export batches containing 18 spans; the run was also verified in Phoenix 13.22.2.

Previous feedback addressed

• Split the previously combined implementation at the repository boundary and kept this as one atomic Relay-side PR.
• Removed in-process Random, LLM, and Cascade routers and all Switchyard Rust dependencies.
• Removed Relay-side Cascade accumulation and duplicate Switchyard event subscription.
• Replaced the former single global fallback with exact trusted defaults for all three inbound protocols.
• Replaced literal credentials with environment-referenced headers and activation-time validation.
• Updated the integration to the current Decision API schemas and additive retry metadata rather than restoring legacy router fields.
• Added focused mark-shape assertions and moved stable contract/translation behavior into unit tests.
• Added deterministic fake-provider and real cumulative-Switchyard E2Es, plus a real Hermes/Ollama trajectory with shareable ATOF, ATIF, and OTLP output.

Related Issues

• Relates to the Switchyard Decision API integration work split from a previous implementation.
• This Relay PR is paired with the corresponding Switchyard series:
  • NVIDIA-NeMo/Switchyard#8
  • NVIDIA-NeMo/Switchyard#9
  • NVIDIA-NeMo/Switchyard#11

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 58b0641e-4b33-4b3d-9a5f-45dadcf22d28

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-369.docs.buildwithfern.com/nemo/relay (​https://nvidia-preview-pull-request-369.docs.buildwithfern.com/nemo/relay​)

[bbednarski9]

reference arch:

flowchart TD
    S["Relay startup"] --> V["Validate configuration, URLs, bindings, and secret environment variables"]
    V --> C["Build HTTP clients"]
    C --> R["Relay becomes ready"]
    R -. "No startup probe" .-> SY["Switchyard server"]

    Q["LLM request"] --> D["POST Decision API"]
    D -->|Valid decision| P["Dispatch selected provider"]
    D -->|Unavailable, timeout, invalid response, or binding drift| F["Dispatch trusted same-protocol fallback"]

    E["ATOF event"] --> A["Best-effort POST"]
    A -->|Switchyard unavailable| L["Log failure; routing continues"]

Loading

[willkill07]

The structure of this README should be in line with other READMEs.

[willkill07]

Avoid relative links for READMEs which ultimately get published/viewed outside of GitHub

[willkill07]

Should explicitly mention that all steps should be run from the repository root.

(Note: if this is still applicable/included)

[willkill07]

example/switchyard needs to have a comprehensive README. The crate's README is not the correct place.

[willkill07]

This is a bit confusing to me. Is this something that we want to expose to users, just to change it later on to the proper switchyard crate?