Skip to content

Add agent setup eval - #13482

Open
sarahchen6 wants to merge 2 commits into
masterfrom
sarahchen6/add-agent-setup-eval
Open

sarahchen6 wants to merge 2 commits into
masterfrom
sarahchen6/add-agent-setup-eval

Conversation

@sarahchen6

@sarahchen6 sarahchen6 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Continue integrating the LLM Validation Platform with another optional CI job that:

  • Compares all repo guidance against a blank state at the same commit.
  • Runs once a month based on a GitLab pipeline schedule (to be added after this PR is merged).

Motivation

Measure the value of repository instructions and skills as a whole, including effects on unrelated control questions. For more details, see https://datadoghq.atlassian.net/browse/APMLP-2032. See the original pilot in DataDog/dd-trace-dotnet#9474.

Additional Notes

These checks are only as valuable as the scenarios that are listed and tested. The idea is to expand upon the .llm-validation/suites/java-agent-instructions.yaml file with scenarios on how we want to evaluate agent instructions. This PR just starts with a few expectations that have already been documented.

Contributor Checklist

Jira ticket: https://datadoghq.atlassian.net/browse/APMLP-2032

@sarahchen6 sarahchen6 added type: feature Enhancements and improvements tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM labels Nov 16, 2026
@datadog-official

datadog-official Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 60.22% (-0.12%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ef19dfa | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.03 s 13.88 s [+0.3%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.88 s 12.96 s [-1.3%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.76 s 17.58 s [+0.1%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent 17.57 s 17.65 s [-1.3%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 17.30 s 17.30 s [-1.0%; +1.0%] (no difference)
startup:petclinic:sca:Agent 17.81 s 17.72 s [-0.4%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 16.77 s 16.33 s [-1.6%; +6.9%] (no difference)

Commit: ef19dfa4 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@sarahchen6
sarahchen6 marked this pull request as ready for review November 16, 2026 22:30
@sarahchen6
sarahchen6 requested review from a team as code owners November 16, 2026 22:30
@sarahchen6
sarahchen6 requested review from bric3, erikayasuda and ygree and removed request for a team November 16, 2026 22:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T22:32:48.083759Z ef19dfa Draft marked ready
🔒 Security Review ✅ Completed 2026-11-16T22:33:53.993992Z ef19dfa Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The setup evaluation remains opt-in and advisory. No actionable defects were reported, though shared-runner integration remains unverified because the external platform CI template was unavailable.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit ef19dfa

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant