Skip to content

feat(agent): per-command agent schema with response shapes - #883

Merged
platinummonkey merged 10 commits into
mainfrom
feat/agent-schema-filter
Oct 2, 2026
Merged

platinummonkey merged 10 commits into
mainfrom
feat/agent-schema-filter

Conversation

@platinummonkey

@platinummonkey platinummonkey commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

pup agent schema returns everything at once (~185 KB) and only describes what each command takes, not what it returns, so agents guess the output format. This adds lookups for a single command or domain that also show what the command returns.

Changes

  • pup agent schema logs aggregate (or logs.aggregate, or a domain like logs) returns just that part, with a returns shape for each command. --search TERM finds commands by name or description.
  • Agent-mode pup <cmd> --help now covers the exact command asked for, not its whole domain.
  • pup security findings schema --search/--section returns only the matching fields instead of the full ~200 KB reference.
  • Output with no arguments is unchanged.

Testing

  • Unit, integration, clippy and fmt all pass.
  • Tests check the size limits, error paths, and that the published returns shapes match real command output.

Follow-ups

  • Making all commands share one output format, and returning errors as structured data (a breaking change, so a separate PR).
  • The --jq note in agent-mode output still conflicts with the published jq paths.

🤖 Generated with Claude Code

platinummonkey and others added 5 commits October 1, 2026 10:00
`pup agent schema` was all-or-nothing (~185 KB compact) and described
inputs only, so agents paid ~50k tokens per lookup and still guessed
response shapes.

- `pup agent schema <path>` accepts `logs aggregate`, `logs.aggregate`,
  or a domain (`logs`); aliases resolve like clap. Unknown paths exit
  non-zero listing valid subcommands. `auth token` stays hidden.
- `--search TERM` lists matching leaf commands (path + short description),
  optionally scoped to a path.
- Filtered output adds a shared `envelope` contract and a `returns`
  entry on every leaf; hand-verified shapes for logs aggregate/search,
  traces aggregate/search, and metrics query, with `jq_root`, notes, and
  a worked example on single-command lookups.
- Filtered output is minified: logs aggregate ~3.8 KB, logs ~16.8 KB.
- No-argument and `--compact` output are unchanged.
- Conformance test feeds recorded API bodies through
  `output::build_agent_envelope` and validates them against `returns`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In agent mode, `pup logs aggregate --help` resolved only the top-level
token and returned the whole logs domain (~32 KB), with every entry's
full_path built from an empty parent (so read_only could be wrong for
nested commands) and no response shape.

- Resolve the deepest command named on the command line (aliases map to
  canonical names; positional values and global flag values are skipped;
  `auth token` is never descended into).
- Build the entry with its real parent path and attach `envelope` and
  per-leaf `returns`, as `pup agent schema <path>` does.
- Keep global_flags, script_authoring and anti_patterns so a lone
  `--help` still carries the --no-agent guidance. Root `--help` is
  unchanged.
- Replace `find_subcommand` with `help_command_path` and port its tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`pup security findings schema` downloads the whole findings reference
(~200 KB, ~50k tokens) on every call with no way to narrow it.

- Parse the reference's attribute tables into {path, type, section,
  description} rows (all 618 rows in the current page).
- `--search TERM` keeps fields whose path or description match;
  `--section NAME` keeps one namespace (nested headings included).
  Filtered output goes through the formatter, so it honors agent mode,
  --jq and --output. `--section advisory` is ~2 KB.
- An unknown section errors, listing the available ones; a page that
  yields no parseable rows errors rather than returning an empty list.
- No flags: output unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without filters the command prints markdown, not an enveloped JSON
value, so the generic contract misdescribed it. Document both modes,
add a conformance fixture, and note in the envelope that some commands
print plain text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@platinummonkey
platinummonkey marked this pull request as ready for review October 1, 2026 20:51
@platinummonkey
platinummonkey requested a review from a team as a code owner October 1, 2026 20:51
@platinummonkey
platinummonkey marked this pull request as draft October 1, 2026 20:52

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Three distinct issues were found: a timestamp unit mismatch in the metrics query contract (milliseconds vs. seconds), a missing blank-input guard in the section filter logic, and incorrect skipping of parent-local option values during help resolution.

Open Bits AI session

🤖 Bits Code Review · Commit d1e2550 · @DataDog review to ask questions

Comment thread src/commands/agent.rs
Comment thread src/commands/security.rs
Comment thread src/main.rs
platinummonkey and others added 5 commits October 2, 2026 08:17
Co-authored-by: datadog-prod-us1-3[bot] <266080212+datadog-prod-us1-3[bot]@users.noreply.github.com>
Co-authored-by: datadog-prod-us1-3[bot] <266080212+datadog-prod-us1-3[bot]@users.noreply.github.com>
`pup profiling --header "x: 1" services list --help` treated the header
value as a subcommand name and fell back to the profiling domain, since
only a hard-coded list of global flags was known to take values. Look
each flag up in clap at the current level (plus root globals) instead,
handling `--flag=value` and `-ovalue` forms.

Also add a test for the blank --section guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit c50e909. The v1 metrics query response documents
from_date/to_date as milliseconds since Unix epoch (datadog-api-client
MetricsQueryResponse), so the contract's original "ms since epoch" was
correct.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@platinummonkey
platinummonkey marked this pull request as ready for review October 2, 2026 14:05

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Targeted schemas overstate the JSON envelope: non-JSON output formats and plain-text leaves such as agent guide emit shapes the published contract says are required.

Open Bits AI session

🤖 Bits Code Review · Commit dc22579 · @DataDog review to ask questions

Comment thread src/commands/agent.rs
Comment thread src/commands/agent.rs
@platinummonkey
platinummonkey merged commit b604636 into main Oct 2, 2026
10 checks passed
@platinummonkey
platinummonkey deleted the feat/agent-schema-filter branch October 2, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants