Skip to content

Run the frozen requirement sets in conformance CI - #638

Open
koic wants to merge 1 commit into
modelcontextprotocol:mainfrom
koic:run_frozen_requirements_legs_in_ci
Open

koic wants to merge 1 commit into
modelcontextprotocol:mainfrom
koic:run_frozen_requirements_legs_in_ci

Conversation

@koic

@koic koic commented Oct 29, 2026

Copy link
Copy Markdown
Member

rake conformance:test (and the Conformance Tests workflow) runs the conformance CLI's default active suite, which does not cover the frozen SEP-1730 requirement sets: the scenarios a spec revision required when it shipped, run at that revision's own wire version. The 2026-07-28 set (server-stateless, caching, the input-required-result-* family) therefore had no CI leg, so the Tier 1 evidence of 100% on scored scenarios per revision was only verifiable by hand.

  • rake conformance:test accepts REQUIREMENTS=<revision>, forwarded as --requirements to both legs. On the client leg it replaces the --suite all selection, mirroring the CLI's own rule that a requirement set already fixes which scenarios run; combining it with SCENARIO or SPEC_VERSION aborts for the same reason.
  • --expected-failures stays on both legs: under a requirement set the CLI judges the baseline against scored scenarios only, so not_scored entries (extensions, post-anchor additions) keep not affecting the exit code.
  • The workflow gains a requirements-conformance matrix job over 2025-11-25 and 2026-07-28. Both legs pass today, and the job carries the same continue-on-error as the active-suite job.

Testing: test/conformance/client_runner_test.rb and test/conformance/server_runner_test.rb (command construction); REQUIREMENTS=2025-11-25 and REQUIREMENTS=2026-07-28 bundle exec rake conformance:test both exit 0 with every scored scenario passing on both legs (the failures they report are unscored extension and pending scenarios); REQUIREMENTS=... SCENARIO=ping aborts with the exclusivity message.

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

`rake conformance:test` (and the Conformance Tests workflow) runs the conformance CLI's default active suite,
which does not cover the frozen SEP-1730 requirement sets: the scenarios a spec revision required when
it shipped, run at that revision's own wire version. The 2026-07-28 set (`server-stateless`, `caching`,
the `input-required-result-*` family) therefore had no CI leg, so the Tier 1 evidence of 100% on scored scenarios
per revision was only verifiable by hand.

- `rake conformance:test` accepts `REQUIREMENTS=<revision>`, forwarded as `--requirements` to both legs.
  On the client leg it replaces the `--suite all` selection, mirroring the CLI's own rule that a requirement set
  already fixes which scenarios run; combining it with `SCENARIO` or `SPEC_VERSION` aborts for the same reason.
- `--expected-failures` stays on both legs: under a requirement set the CLI judges the baseline against
  scored scenarios only, so not_scored entries (extensions, post-anchor additions) keep not affecting the exit code.
- The workflow gains a `requirements-conformance` matrix job over 2025-11-25 and 2026-07-28. Both legs pass
  today, and the job carries the same `continue-on-error` as the active-suite job.

Testing: test/conformance/client_runner_test.rb and test/conformance/server_runner_test.rb (command construction);
`REQUIREMENTS=2025-11-25` and `REQUIREMENTS=2026-07-28 bundle exec rake conformance:test` both exit 0 with every
scored scenario passing on both legs (the failures they report are unscored extension and pending scenarios);
`REQUIREMENTS=... SCENARIO=ping` aborts with the exclusivity message.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants