feat(data-pipeline): add fork-safe OTLP gRPC trace transport - #2273
Open
bm1549 wants to merge 2 commits into
Open
feat(data-pipeline): add fork-safe OTLP gRPC trace transport#2273bm1549 wants to merge 2 commits into
bm1549 wants to merge 2 commits into
Conversation
Adds a plaintext gRPC-over-HTTP/2 (h2c) trace-export transport for OTLP, implemented as a custom tower::Service plugged into tonic's Grpc<T>. It holds no persistent connection or background task: each send opens a fresh connection driven by ephemeral per-request tasks that are torn down when the send completes, so nothing is orphaned across fork(2). Includes a minimal prost codec (tonic 0.14 moved ProstCodec to a separate crate; hand-rolled here to avoid the extra dependency), endpoint validation (plaintext http:// only; https:// rejected), gRPC-status-to-error mapping that recovers nested IO errors, per-request metadata (validated once at build), and unit plus in-process h2 integration tests. This is the transport primitive; the OTLP export path is wired to it in a follow-up change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 1 job - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: f5ea26d | Docs | Datadog PR Page | Give us feedback! |
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
BenchmarksComparisonBenchmark execution time: 2026-07-27 16:22:29 Comparing candidate commit f5ea26d in PR branch Found 9 performance improvements and 10 performance regressions! Performance is the same for 123 metrics, 0 unstable metrics.
|
bm1549
commented
Jul 24, 2026
|
|
||
| /// Per-request OTLP gRPC trace exporter configuration. | ||
| // Not yet wired to the trace exporter's send loop; exercised by tests only. | ||
| #[allow(dead_code)] |
|
|
||
| /// Validate a gRPC endpoint (plaintext `http://` only) and build the transport. | ||
| // Not yet wired to the trace exporter's send loop; exercised by tests only. | ||
| #[allow(dead_code)] |
|
|
||
| /// Send one OTLP trace export request over gRPC. Bounds connect + RPC with a single timeout. | ||
| // Not yet wired to the trace exporter's send loop; exercised by tests only. | ||
| #[allow(dead_code)] |
yannham
reviewed
Jul 28, 2026
yannham
left a comment
yannham
left a comment
Contributor
There was a problem hiding this comment.
Overall LGTM, nothing major blocking.
Though the same caveat as in the original PR applies about the size issue:
A first naive question: could/should this be feature-gated? I assume you'd want all tracers to support otlp out of the box, so the answers might be no, bu asking just > in case.
A good test for the impact of the binary size increase: would you mind trying to create a PR against dd-trace-py pointing to this PR's libdatadog ref, and see if it triggers the size gates for e.g. serverless?
Comment on lines
+136
to
+148
|
|
||
| #[test] | ||
| fn grpc_config_constructs_and_clones() { | ||
| let cfg = OtlpGrpcTraceConfig { | ||
| headers: vec![("k".to_string(), "v".to_string())], | ||
| timeout: Duration::from_secs(3), | ||
| otel_trace_semantics_enabled: true, | ||
| }; | ||
| let clone = cfg.clone(); | ||
| assert_eq!(clone.headers, cfg.headers); | ||
| assert_eq!(clone.timeout, Duration::from_secs(3)); | ||
| assert!(clone.otel_trace_semantics_enabled); | ||
| } |
Contributor
There was a problem hiding this comment.
We're basically testing the builtin Clone derive implementation.
Suggested change
| #[test] | |
| fn grpc_config_constructs_and_clones() { | |
| let cfg = OtlpGrpcTraceConfig { | |
| headers: vec![("k".to_string(), "v".to_string())], | |
| timeout: Duration::from_secs(3), | |
| otel_trace_semantics_enabled: true, | |
| }; | |
| let clone = cfg.clone(); | |
| assert_eq!(clone.headers, cfg.headers); | |
| assert_eq!(clone.timeout, Duration::from_secs(3)); | |
| assert!(clone.otel_trace_semantics_enabled); | |
| } |
| // private `DecodeBuf` constructor. | ||
| fn decode_from(src: &mut impl bytes::Buf) -> Result<Option<T>, Status> { | ||
| // copy_to_bytes drains the whole buffer even if the backing store is non-contiguous. | ||
| let buf = src.copy_to_bytes(src.remaining()); |
Contributor
There was a problem hiding this comment.
I'm not sure to understand the point of this line. Sure we get a contiguous buffer out of it, but we need to go through the original one anyways (and copy it). So why not read directly read from the original one?
| let resp = sender.send_request(req).await?; | ||
| let (parts, incoming) = resp.into_parts(); | ||
| let collected = incoming.collect().await?; | ||
| driver.abort(); |
Contributor
There was a problem hiding this comment.
It seems driver.abort() kinda ends the connection abruptly, instead of a graceful shutdown. On one hand, we already got our response, so we should be good on our side. I just wonder if this is an issue on the intake side.
One alternative is to await the driver after dropping the sender (drop(sender); let _ = driver.await;). This is a graceful shutdown. However this is now subject to network hangs or whatnot, so it probably needs a timeout.
I'm not 100% sure what's the best solution here. abort is not graceful shutdown, but on the other hand it ensures we don't block the calling task for unimportant reasons.
| pub(crate) config: OtlpGrpcTraceConfig, | ||
| origin: http::Uri, | ||
| service: H2Service, | ||
| /// Custom headers parsed to gRPC metadata once at build time; invalid entries are dropped. |
Contributor
There was a problem hiding this comment.
Nit: "invalid entries are dropped" sounds like it should go on the function that builds metadata_headers, not on the field itself.
Suggested change
| /// Custom headers parsed to gRPC metadata once at build time; invalid entries are dropped. | |
| /// Custom headers parsed to gRPC metadata once at build time. |
| } | ||
|
|
||
| /// A gRPC transport for OTLP trace export: per-request config, request origin, and the dial | ||
| /// service. Holds no live connection (nothing to rebuild across fork). |
Contributor
There was a problem hiding this comment.
Nit: connections in itself is not necessarily the issue, but background async tasks that we don't control are.
Suggested change
| /// service. Holds no live connection (nothing to rebuild across fork). | |
| /// service. Holds no live connection and thus no background task (nothing to rebuild across fork). |
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Aug 5, 2026
## Description When native OTLP trace export fails, `NativeWriter` currently reports its configured Datadog Agent intake URL even though libdatadog attempted the OTLP endpoint. This makes an OTLP protocol or connectivity error look like an Agent fallback. Use the configured OTLP trace endpoint in failure diagnostics whenever OTLP trace export is active. Agent export diagnostics keep their existing behavior. The original investigation also exposed a separate protocol mismatch: released versions send OTLP traces over HTTP/JSON, while the repro targeted the collector's gRPC port. HTTP/protobuf and gRPC support are already tracked by #18609, DataDog/libdatadog#2171, and DataDog/libdatadog#2273; this PR does not duplicate those transport changes. ## Testing - `scripts/run-tests --venv 1ef5a52 -- -- tests/tracer/test_writer.py -k 'native_writer_reports_otlp_intake_endpoint or native_writer_stores_otlp_endpoint'` (Python 3.13; 2 passed) - `scripts/lint checks` - `scripts/lint style -- ddtrace/internal/writer/writer.py tests/tracer/test_writer.py` - `scripts/lint spelling -- releasenotes/notes/fix-otlp-export-error-endpoint-6e7b7c8f756ef3a7.yaml` - `riot run reno lint` ## Risks Low. The change only affects the endpoint displayed in trace-export failure diagnostics when an OTLP endpoint is configured. It does not change routing, serialization, or retry behavior. ## Additional Notes The OTLP/DDOT reference and the original handoff document were corrected separately to record that the old port-4317 reproduction was testing an HTTP exporter against a gRPC receiver. Co-authored-by: munir.abdinur <munir.abdinur@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a plaintext gRPC-over-HTTP/2 (h2c) trace-export transport for OTLP to
libdd-data-pipeline. It's a customtower::Service(H2Service) plugged into tonic's genericGrpc<T>client, so the exporter holds no persistent connection or background task: each send opens a fresh connection driven by ephemeral per-request tasks that are torn down when the send completes.This is the transport primitive only. The OTLP export path is wired to it in a follow-up PR that rebases on top of this one.
Motivation
The earlier OTLP gRPC approach built a tonic
Channel, which spawns a persistent background task. Because libdatadog rebuilds its runtime acrossfork(2), that task dies silently after a fork, which forced aWorkerpluswatch-channel to rebuild it. Making the transport per-request and task-free removes that machinery and is fork-safe by construction.Additional Notes
default-features = falseon tonic (notransport/channel/server); a minimal hand-rolled prost codec replacesProstCodec(which moved to the separatetonic-prostcrate) to avoid the extra dependency.https://endpoints are rejected at build time; terminate TLS in a proxy in front of the endpoint if encryption is needed.How to test the change?
cargo nextest run -p libdd-data-pipelinecovers unit tests plus in-processh2integration tests: a real unary send decoded by a mock gRPC server, and connection-refused, timeout, and post-connect-reset cases that all map toTraceExporterError::Io.cargo clippy -p libdd-data-pipeline --all-targets -- -D warnings,cargo +nightly fmt --check, andcargo check -p libdd-data-pipeline --target wasm32-unknown-unknown --no-default-features(the gRPC path is wasm-gated) pass.