feat: export OTLP traces to a local file

[SandyChapman]

Overview

Relay's OpenTelemetry plugin can only ship spans to a collector. ATIF and ATOF both write to disk, so a consumer that treats a trajectory as an artifact rather than as telemetry has no OTLP option at all. This adds [[components.config.opentelemetry.file_sinks]], which writes the same ExportTraceServiceRequest an endpoint would receive to a local file.

The argument for a sink here rather than a collector is that it removes a routable address and an egress rule for isolated sandboxes. Additionally, some users using Fabric+Relay for evaluation, may not have provision the infrastructure required to collect traces.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Note for reviewers: a companion change is needed in NeMo-Fabric

This change is necessary but not sufficient for the evaluation use case.

Callers that reach Relay through Fabric use nemo_fabric.RelayOpenTelemetryConfig, which has two fields (enabled, endpoints) and a hand-written validator in its own models.py:

if self.enabled and not self.endpoints:
    raise ValueError("enabled NeMo Relay OpenTelemetry requires at least one endpoint")

Checked against nemo-fabric 0.3.0b1:

• Multiple endpoints already work and need nothing from anyone.
• A file sink alongside at least one endpoint works with this PR alone: the model sets extra="allow", so file_sinks survives serialization and reaches Relay untouched.
• A file sink as the only destination — the configuration an eval actually wants — is rejected by Fabric before Relay sees the config.

Fabric needs file_sinks on the model and the validator relaxed to accept a file sink as a destination, which is the same shape as the validate_opentelemetry_section change in the second commit here.

Details

Destination, not transport. A file sink has no endpoint, headers, or timeout, and endpoint validation does not apply to it, so it is modelled as a separate destination on OpenTelemetryConfig rather than as a third OtlpTransport variant. Configuration mirrors the existing ATOF file sink (output_directory, filename, mode) rather than overloading endpoint with a path.

Everything above the exporter — projection, id generation, batching, resource attributes, shutdown — is shared, so the two destination kinds cannot drift on what a span means. SpanExporter::export returns impl Future, so the trait is not dyn-compatible and dispatch is a concrete enum rather than a boxed trait object.

Conversion is upstream. group_spans_by_resource_and_scope and the SpanData → protobuf From impls come from opentelemetry-proto, the crate opentelemetry-otlp already uses to build its wire payload. Both opentelemetry-proto and prost were already in crates/core/Cargo.toml under [dev-dependencies]; they move to [dependencies] with the trace and with-serde features. No new crates and no version changes.

Durability and permissions. Each export is flushed before it is reported as delivered, so a run that exits between batches leaves a readable prefix rather than an empty file. Output is created with the same owner-only permissions and directory confinement as the ATOF and ATIF sinks, because a trajectory carries prompt and response content. Two sinks writing one path are rejected at activation.

Surfaces. nemo-relay configure lists file sinks beside trace endpoints; Python, Node, Go, and the FFI each gain the destination.

Where should the reviewer start?

crates/core/src/observability/otel_file.rs — the exporter is ~200 lines and the whole design is visible there. Then crates/core/tests/unit/observability/otel_file_tests.rs: the round-trips decode with prost and serde_json directly rather than through this module's own encoder, so a writer and reader that agree only with each other cannot pass. json_lines_encode_span_identifiers_as_hex is the OTLP/JSON conformance check.

The design decision worth arguing about is file_sinks as a separate array versus a transport = "file" variant on endpoints. I chose the former because endpoint is required and URL-validated, and conditional validation on a field that sometimes holds a path seemed worse than a second array.

Testing

• cargo test --workspace — 1713 core lib tests pass. 39 new tests: exporter round-trip and framing, config validation, editor schema, and per-binding coverage.
  • Pre-existing unrelated failure: native_plugin_integration needs just build-test-plugin-fixtures, which I could not run locally.
• cargo clippy --workspace --all-targets and cargo fmt --all clean.
• Node: node --test crates/node/tests/observability_plugin_tests.mjs — 10/10. One test drives a file-sink-only section through the plugin host and asserts the trace file appears.
• Python: pytest python/tests/test_types.py — 59/59, ruff clean, ty clean (9 pre-existing unused-ignore warnings).
• Go: go test on otel_test.go passes; gofmt and go vet clean.
• Not run locally: cargo deny (not installed) and the fixture-dependent native plugin integration tests.

Breaking changes

None. file_sinks defaults to empty and every existing configuration behaves identically.

Related Issues:

None — raised on Slack rather than as a GitHub issue. Happy to file one if you would like the paper trail before review.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

• const-hex 1.19.1 (Apache-2.0)
• proptest 1.11.0 (MIT OR Apache-2.0)
• rand_xorshift 0.4.0 (MIT OR Apache-2.0)
• unarray 0.1.4 (MIT OR Apache-2.0)

Removed

• None

Updated/Changed

• None

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (463 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (424 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (459 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (424 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-1042.docs.buildwithfern.com/nemo/relay/dev

[willkill07]

I feel like this needs a proper proposal rather than instructing Claude to implement the functionality loosely based on need.

I understand this is in draft form, but it isn't conducive to any significant review time without a corresponding proposal.

Ultimately, we need a configuration shape for the new functionality without breaking existing plugin configuration files.

[willkill07]

This is incompatible with several other options. Given that we permit programmatic modification of plugin configuration through nemo-relay plugins edit this naive extension is not appropriate.

transport and endpoint must be mutually exclusive with file_sink. No silent ignore behavior.

[willkill07]

Inappropriate for implementation

[willkill07]

This doesn't match the OTel alpha proposal for options including rotation/file size limits. If we implement this, I'd like for there to be feature parity with the alpha-level proposal.