feat(data-pipeline): add agentless stats export - #2309
Open
paullegranddc wants to merge 2 commits into
Open
Conversation
# Motivation Perform stats aggregation and export directly to the public intake when agentless tracing is enabled # Changes * Add paramateres on the TraceExporterBuilder * agentless_stats_endpoint - the fully formed URL. The API key is derived from the agentless export parameter * Refactor the stats exporter to add a path that wraps tje ClientStatsPayload into a top-level StatsPayloas * Hardcode the agent_version to libdd version + a libdatadog suffix to be able to differentiate metrics origin
Contributor
📚 Documentation Check Results📦
|
Contributor
🔒 Cargo Deny Results📦
|
|
BenchmarksComparisonBenchmark execution time: 2026-08-05 15:01:48 Comparing candidate commit 823e943 in PR branch Found 3 performance improvements and 7 performance regressions! Performance is the same for 95 metrics, 10 unstable metrics.
|
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
paullegranddc
marked this pull request as ready for review
August 5, 2026 12:25
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc1ff2cd86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+846
to
+849
| if let (Some(stats_url), Some(bucket_size)) = ( | ||
| self.agentless_stats_endpoint.as_ref(), | ||
| self.stats_bucket_size, | ||
| ) { |
There was a problem hiding this comment.
When this branch is taken, set_agentless_stats_endpoint already requires agentless_config to be present, but send_trace_chunks_inner returns through the agentless path at trace_exporter/mod.rs:786-795 before stats::process_traces_for_stats is called. Therefore spans are never added to the concentrator created here, so the worker flushes an empty concentrator and agentless stats are never exported for the exact configuration this feature enables.
Useful? React with 👍 / 👎.
| // `client_computed` is always set to `true` since the stats were computed by the | ||
| // tracer/client, not the Agent | ||
| client_computed: true, | ||
| split_payload: false, |
There was a problem hiding this comment.
For high-cardinality clients, a single flush can contain more than 4000 grouped stats: the concentrator default whole-key limit is 7000 (span_concentrator/mod.rs:123-145). Always setting split_payload false and wrapping the entire client payload in one StatsPayload sends oversized direct-intake payloads instead of matching the Agent's splitter, so agentless stats can be rejected or dropped for those workloads.
Useful? React with 👍 / 👎.
Comment on lines
+348
to
+350
| StatsDestination::Agentless(target) => { | ||
| build_agentless_request(&self.meta, sequence, buckets, target)? | ||
| } |
There was a problem hiding this comment.
With stats-obfuscation enabled, the agentless builder forces obfuscation on and send() calls send_payload(..., true) for obfuscated buckets, but this match arm discards the obfuscated flag and build_agentless_request never adds datadog-obfuscation-version (the Agent arm does). Agentless intake receives obfuscated resource names without the marker for any obfuscated payload.
Useful? React with 👍 / 👎.
Comment on lines
+948
to
+951
| stats = StatsComputationStatus::Enabled { | ||
| stats_concentrator: concentrator, | ||
| worker_handle, | ||
| }; |
There was a problem hiding this comment.
This puts agentless stats in StatsComputationStatus::Enabled but leaves otlp_stats_enabled false. send_trace_chunks_async still calls check_agent_info() in agentless mode, and that function only skips the agent gate when otlp_stats_enabled is true; if the process has cached AgentInfo from another exporter, handle_stats_enabled can stop this worker when that unrelated agent does not advertise /v0.6/stats. In mixed agent/agentless processes, the first send can therefore shut off the direct-intake stats worker.
Useful? React with 👍 / 👎.
| buckets: Vec<pb::ClientStatsBucket>, | ||
| target: &AgentlessStatsTarget, | ||
| ) -> anyhow::Result<StatsRequest> { | ||
| let mut client_payload = encode_stats_payload(meta, sequence, buckets); |
There was a problem hiding this comment.
This reuses encode_stats_payload, which still clears container_id and tags because the Agent normally enriches those fields later. In the agentless path there is no Agent, and StatsMetadata is built without the builder's container_id, so containerized deployments send direct stats without container identity or resolved tags.
Useful? React with 👍 / 👎.
Comment on lines
+445
to
+448
| #[cfg(feature = "compression")] | ||
| let compression = CompressionStrategy::Zstd { level: 1 }; | ||
| #[cfg(not(feature = "compression"))] | ||
| let compression = CompressionStrategy::None; |
There was a problem hiding this comment.
When compression is enabled this sends Content-Encoding: zstd, and without it sends raw msgpack, but the Agent sender for the same /api/v0.2/stats route always posts gzipped msgpack with Content-Encoding: gzip (see pkg/trace/writer/stats.go in datadog-agent). If the stats intake follows that contract, these agentless stats requests can be rejected despite passing the local mock.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Perform stats aggregation and export directly to the public intake when agentless tracing is enabled
Changes