POC: native pup setup verify command

[tedkahwaji]

Why

Exploring how product-setup capabilities could live in pup. Today an agent that has just instrumented a service has no single pup command to answer "is telemetry arriving yet?". This POC ports that one slice, the post-setup verification check, as a native command, to make the cost of porting setup functionality into pup concrete. It is a draft for discussion, not a merge request.

What

pup setup verify --product <apm|logs|rum> --service <name> [--env <env>] [--from 15m] [--wait 0s]

• Counts matching events over the lookback window through the public analytics aggregate endpoints (/api/v2/spans/analytics/aggregate, /api/v2/logs/analytics/aggregate, /api/v2/rum/analytics/aggregate) using raw_client::raw_post, so auth and site handling are pup's existing ones.
• With --wait, polls every 10s until data appears or the wait elapses.
• Prints one structured result (found, count, query, lookback, elapsed_seconds, explorer url) through formatter::output, so -o, --jq and the agent envelope work as for other commands.
• Exits non-zero when nothing is found, so agents and scripts can branch on it.
• Read-only. Registered in docs/COMMANDS.md.

Testing

• cargo test commands::setup: 8 passed (found, not found, polling until data arrives, API error, empty service and bad duration rejected, query building, response shapes, URL encoding).
• cargo fmt and cargo clippy --all-targets -- -D warnings clean.
• Full cargo test: 2075 passed, 6 failed in unrelated modules (ddsql, debugger, notebook_images, notebooks, security, users) when run from inside an agent session; all 6 pass when rerun without the agent-session env vars, so they are environment-sensitive, not caused by this change.
• Against a real org (service names redacted):

$ pup --no-agent setup verify --product apm --service <busy-service> --env prod -o json
{ "count": <n>, "found": true, "product": "apm", "query": "service:<busy-service> env:prod",
  "url": "https://app.datadoghq.com/apm/traces?query=service%3A<busy-service>%20env%3Aprod", ... }
exit 0

$ pup --no-agent setup verify --product apm --service pup-verify-bogus-service-xyz --wait 15s -o json
{ "count": 0, "found": false, "elapsed_seconds": 10, ... }
Error: no apm data found for service "pup-verify-bogus-service-xyz" in the last 15m
exit 1   (polled at 0s and 10s, 11.2s wall clock)

Logs (found for a busy service) and RUM (not found) behave the same way.

Cost notes

• Size: about 160 lines of implementation plus 157 lines of tests in src/commands/setup.rs, and 67 lines of clap wiring in src/main.rs. This covers three products and only the verification step.
• Hand-maintained, not generated: the generated path (src/generated) covers spec tags only. This command chains three different request and response shapes, polling, and a pass/fail exit, which is the kind of logic pup's generator does not produce. Every future product means editing this file by hand.
• What this approach cannot port: the agent-driven parts of a setup flow (orchestrating an LLM agent, prompts per product and framework, editing customer code, detecting existing setups, recommending products). Those need an agent runtime and a different security review, so they do not fit as native pup commands. Only API-shaped steps like this one port cleanly.

🤖 Generated with Claude Code