Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1576,7 +1576,7 @@ update_central_configurations_version_range_v2:
MULTIPLE_RELEASE_LINES: "false"
SEND_ALIASES: "true"

# Regular instruction-change validation; setup evaluation remains disabled.
# Regular instruction-change validation.
"llm validation":
# Advisory during calibration: retain job failures without failing the pipeline.
allow_failure: true
Expand Down Expand Up @@ -1606,3 +1606,20 @@ update_central_configurations_version_range_v2:
- results-partial.json
- report-partial.md
- build/llm-validation/context-growth.md

# A GitLab monthly schedule selects setup-full and sets LLMVAL_SETUP_ENABLED=true.
# Other pipelines can opt in with the same enable flag.
"llm validation (setup smoke)":
allow_failure: true
before_script: []
variables:
LLMVAL_SETUP_PRESET: "setup-calibration"
rules:
- if: '$POPULATE_CACHE'
when: never
- if: '$LLMVAL_SETUP_ENABLED == "true"'
when: on_success
- when: never
artifacts:
# Retain the shared setup artifact paths and preserve monthly evidence for a year.
expire_in: 13 mos
100 changes: 32 additions & 68 deletions .llm-validation/README.md
Original file line number Diff line number Diff line change
@@ -1,84 +1,48 @@
# Agent instruction validation

The [LLM Validation Platform](https://github.com/ddoghq/llm-validation-platform) compares agent
answers under baseline and candidate instructions. This directory also reports instruction text
growth and documents how to review guidance for correctness, scope, and placement.
The [LLM Validation Platform](https://github.com/ddoghq/llm-validation-platform) checks whether
repository guidance improves agent answers. Both CI jobs are advisory and do not block merging.

## CI
## PR checks

`llm validation` reports context growth, then evaluates changes to the
`instruction_files` in [config.yaml](config.yaml): `AGENTS.md` and `.claude/CLAUDE.md`.
`llm validation` compares answers before and after changes to `AGENTS.md` and `.claude/CLAUDE.md`.
It skips `master` and skips answer evaluation when those files are unchanged. It also reports
instruction size, including relative imports, in bytes rather than runtime tokens.

The job skips `master`. Context reporting runs on other eligible pipelines even when unchanged
instructions cause the behavioral evaluation to skip. Growth and quality findings are advisory.
Preflight errors, configuration errors, and incomplete evaluations after detecting instruction
changes fail the job. For now, we set `allow_failure: true` to keep these failures visible without
failing the GitLab pipeline.
When reviewing guidance, check correctness, duplication, and whether detailed procedures belong
in a skill or scoped documentation. Read the answers alongside the scores; a PASS does not prove
every instruction is useful.

Read the job artifacts:
## Monthly setup evaluation

- `build/llm-validation/context-growth.md`: committed instruction bytes, including relative imports
counted once. This is a size proxy, not runtime tokens or total context; user settings and
on-demand skill bodies are excluded. Missing imports or unsupported external imports and symlinks
produce an incomplete measurement.
- `report.md`: behavioral comparison summary; `details.json`: answers and judgments.
- `results.json`: scores; `telemetry.json`: outcome delivery status.
`llm validation (setup smoke)` compares the declared repository setup against minimal guidance
at the same commit. The `setup-full` preset runs every setup-eligible case three times.

## Review instruction changes
Setup artifacts are retained for
13 months. A result applies to the evaluated commit and tasks, not to every individual skill.

A PASS does not establish that guidance is necessary or belongs in `AGENTS.md`. Maintainers should check:
The initial cases use read-only answers and fixtures. They do not measure full implementation
workflows, multi-agent reviews, external MCPs/plugins, or alternative model/effort settings.

- Correctness against maintained documentation and code; cite sources and resolve contradictions.
- Whether a concise root rule or pointer is sufficient, with procedures in `.agents/skills/`,
module-scoped guidance, or existing documentation.
- Duplication, unnecessary imports, and requirements that impose unrelated work.
## Where to find results

Use [CONTRIBUTING.md](../CONTRIBUTING.md) and the relevant [developer guides](../AGENTS.md#key-documentation-read-on-demand-dont-load-upfront).
When adding imports or moving guidance into a skill, list every affected file in `instruction_files`
so both instruction versions are compared.
Open the CI job's artifacts:

As instructions evolve, add or update cases to check the intended behavior and catch regressions.
Ground expectations in repository sources, include new cases in the `minimum` preset in `config.yaml`,
and keep a general Java control case. Review the actual answers alongside the scores.
| Check | Report | Answers and judgments |
|---|---|---|
| PR instructions | `report.md` | `details.json` |
| Monthly setup | `setup-report.md` | `setup-details.json` |
| Instruction size | `build/llm-validation/context-growth.md` | — |

## Setup evaluation
## Maintaining the checks

The separate `setup` inventory in [config.yaml](config.yaml) declares root/build-module guidance.
Setup mode compares the declared guidance against a minimal setup at the same commit. It is not
enabled in CI yet. With the pinned platform runner, use `--comparison setup --level minimum`;
add `--dry-run` to inspect the inventory and selection without calling a model. It reuses the
existing suite, so conclusions apply only to the tasks covered.
- Update [config.yaml](config.yaml): `instruction_files` controls PR comparisons; `setup` lists
guidance removed from the minimal variant. Include new guidance and required resources.
- Add cases to [the suite](suites/java-agent-instructions.yaml), verifying expected behavior
against repository code and documentation. Add PR cases to the `minimum` preset; use
`comparisons: [setup]` for setup-only cases. `setup-full` automatically includes eligible cases.
- Keep both platform references in `.gitlab-ci.yml` in sync when upgrading.

## Local checks

From the repository root, check context growth with Bash, Git, awk, and standard shell utilities.
This compares committed revisions, excluding unstaged edits:

```shell
bash .llm-validation/tests/test_context_growth.sh
git fetch --no-tags origin master
instruction_base=$(git merge-base HEAD origin/master)
bash .llm-validation/context_growth.sh --base-ref "$instruction_base" \
--report build/llm-validation/context-growth.md
```

Run a fake evaluation with Docker (requires internal registry access):

```shell
mkdir -p build/llm-validation
llmval_git_dir=$(git rev-parse --path-format=absolute --git-common-dir)
docker run --rm --network none \
-v "$PWD:/repo" -v "$llmval_git_dir:$llmval_git_dir" \
registry.ddbuild.io/ci/llm-validation-platform/llmval:latest \
--repo /repo --base-sha HEAD --level minimum --runs 1 --fake \
--out /repo/build/llm-validation/results.json \
--report /repo/build/llm-validation/report.md \
--details /repo/build/llm-validation/details.json
```

The extra Git mount supports worktrees. Fake runs check configuration and reporting, not answer quality.
The `latest` image can differ from the pinned CI release. See the platform's
For local execution and troubleshooting, see the pinned platform's
[runner guide](https://github.com/ddoghq/llm-validation-platform/blob/v0.1.6/README.md#run-locally)
for live evaluations and [CI guide](https://github.com/ddoghq/llm-validation-platform/blob/v0.1.6/ci/README.md)
for configuration. When upgrading, update both platform references in `.gitlab-ci.yml` together
and preserve the shared template's artifact paths. The coding-agent CLI is pinned separately.
and [CI guide](https://github.com/ddoghq/llm-validation-platform/blob/v0.1.6/ci/README.md).
22 changes: 21 additions & 1 deletion .llm-validation/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ instruction_files:
- AGENTS.md
- .claude/CLAUDE.md

# Inventory for on-demand setup evaluation; this does not enable a CI job.
# Guidance removed from minimal and retained in default at the same commit.
setup:
files:
- AGENTS.md
Expand All @@ -20,6 +20,7 @@ setup:
directories:
- .agents/skills
- .claude/skills
- .claude/agents

defaults:
bad_signals:
Expand All @@ -36,3 +37,22 @@ presets:
- java-ci-download-guidance-001
- java-documentation-only-validation-001
- java-existing-spock-maintenance-001
# Extend as representative workflows are added.
setup-calibration:
runs: 3
cases:
- java-core-validation-001
- java-unit-test-conventions-001
- java-premain-safety-001
- java-ci-download-guidance-001
- java-documentation-only-validation-001
- java-existing-spock-maintenance-001
- java-setup-instrumentation-plan-001
- java-setup-muzzle-triage-001
- java-setup-flaky-reproduction-001
- java-setup-sdk-review-001
- java-setup-control-resource-closing-001
- java-setup-control-map-atomicity-001
# No case list or max_cases: include every setup-eligible case, including future additions.
setup-full:
runs: 3
Loading
Loading