Repository navigation
Add agent setup eval - #13482
Add agent setup eval#13482sarahchen6 wants to merge 2 commits into
Conversation
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
More details
The setup evaluation remains opt-in and advisory. No actionable defects were reported, though shared-runner integration remains unverified because the external platform CI template was unavailable.
🤖 Bits Code Review · Commit ef19dfa
What Does This Do
Continue integrating the LLM Validation Platform with another optional CI job that:
Motivation
Measure the value of repository instructions and skills as a whole, including effects on unrelated control questions. For more details, see https://datadoghq.atlassian.net/browse/APMLP-2032. See the original pilot in DataDog/dd-trace-dotnet#9474.
Additional Notes
These checks are only as valuable as the scenarios that are listed and tested. The idea is to expand upon the
.llm-validation/suites/java-agent-instructions.yamlfile with scenarios on how we want to evaluate agent instructions. This PR just starts with a few expectations that have already been documented.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: https://datadoghq.atlassian.net/browse/APMLP-2032