Add conformance-server: host conformance test harness (addresses #674)#693
Add conformance-server: host conformance test harness (addresses #674)#693qchuchu wants to merge 1 commit into
Conversation
A WPT-style host-conformance test runner for the MCP Apps spec (2026-01-26): ships a ui:// page that renders inside the host's sandboxed iframe, drives the postMessage/JSON-RPC bridge, asserts host behaviour against the spec, and reports PASS/FAIL right in the iframe. First step toward modelcontextprotocol#674 (public MCP Apps platform tests). Mirrors the basic-server-react layout/toolchain and is registered alongside the other examples in both E2E specs and both publish workflows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new examples/conformance-server package that serves a ui:// in-iframe conformance runner (WPT-style) for validating MCP Apps host behavior against the 2026-01-26 spec, and wires it into existing E2E/grid and publishing workflows.
Changes:
- Introduces the conformance server example (MCP server + React runner UI + in-view test harness/catalogue).
- Registers the new server in E2E server lists and grid screenshot generation.
- Includes the example in publish workflows and updates the lockfile/workspaces wiring.
Reviewed changes
Copilot reviewed 17 out of 22 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/servers.spec.ts | Adds conformance-server to the set of E2E-tested servers. |
| tests/e2e/generate-grid-screenshots.spec.ts | Includes conformance-server in the grid screenshot generation list. |
| package-lock.json | Adds the new example workspace and updates dependency lock state. |
| examples/conformance-server/vite.config.ts | Vite build config for single-file runner bundling. |
| examples/conformance-server/tsconfig.server.json | Server-side TS config for emitting declaration files. |
| examples/conformance-server/tsconfig.json | Client/runner TS config for React/Vite. |
| examples/conformance-server/src/vite-env.d.ts | Vite client type reference. |
| examples/conformance-server/src/tests.ts | In-view conformance test catalogue (automatic + manual/human-verified cases). |
| examples/conformance-server/src/testharness.ts | WPT-style harness (mcp_test, assertions, timeouts, manual interaction plumbing). |
| examples/conformance-server/src/style.css | Runner UI styling (grid + interaction prompts). |
| examples/conformance-server/src/mcp-app.tsx | React runner UI that executes the suite and renders results/prompts. |
| examples/conformance-server/server.ts | MCP server registering the runner ui:// resource and fixture tools. |
| examples/conformance-server/README.md | Usage docs and host-requirement catalogue. |
| examples/conformance-server/package.json | New workspace package definition and build/dev scripts. |
| examples/conformance-server/mcp-app.html | Runner HTML entrypoint for Vite. |
| examples/conformance-server/main.ts | HTTP/stdio entrypoint for running the server. |
| examples/conformance-server/.gitignore | Ignores node_modules/ and dist/. |
| .github/workflows/publish.yml | Adds the new example to the publish matrix. |
| .github/workflows/npm-publish.yml | Adds conformance-server to the npm publish matrix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| requestInteraction: (req) => | ||
| new Promise<boolean>((resolve) => { | ||
| const settle = (v: boolean) => { | ||
| setInteraction(null); | ||
| resolve(v); | ||
| }; | ||
| // "await" mode: pass automatically the moment the test's signal settles | ||
| // (e.g. the host-context-changed notification arrives). | ||
| if (req.kind === "await" && req.signal) { | ||
| req.signal.then( | ||
| () => settle(true), | ||
| () => settle(false), | ||
| ); | ||
| } | ||
| setInteraction({ req, resolve: settle }); | ||
| }), |
| * Entry point for running the MCP server. | ||
| * Run with: npx @modelcontextprotocol/server-basic-react | ||
| * Or: node dist/index.js [--stdio] |
|
Thanks for putting this together @qchuchu ! I think this is a useful seed, and I’m glad to see the work moving towards a generic solution. That said, I think we should continue exploring the browser-automation direction with the hosts as the official approach before we release a server. That'd give us far greater visibility, and will remove the need for manual work. The current PR looks like a valuable fixture but without the holistic solution it may not serve its purpose. Would you like to start sketching a design around that? For example, given an automation framework (e.g., Playwirght), what adaptor surface we'd need, how to automate the things that require manual work today, what we'd need from hosts, etc. |
|
Hi @idosal, Thanks for the feedback. Indeed, without the "runner" (basically what is going to drive the test - so a browser + CDP (if chromium based - Electron apps like VSCode, Cursor, Goose) or WebDriver (is browser agnostic) + some "host driver interface that the host should expose like @domfarolino mentioned in the issue). I think I'm going to list a first "tier-list" of MCP-Apps compliant on which we should run the test so that will help me scope better the surface & what we require from hosts. |
Summary
Adds
examples/conformance-server, a host-conformance test harness for theMCP Apps spec (
2026-01-26), modeled on web-platform-tests. It ships a singleui://runner that renders inside the host's sandboxed iframe, drives thepostMessage/JSON-RPC bridge, asserts the host's behaviour against the spec, and
reports PASS/FAIL right in the iframe.
This is a concrete first step toward #674 ("Set up public MCP Apps platform
tests"): the in-iframe runner that a public wpt.fyi-style grid would later
aggregate across hosts. Opening per the examples policy; happy to adjust scope.
What's included
examples/conformance-server/: React runner plus MCP server, mirroring thebasic-server-reactlayout and toolchain (vite + bun build, stdio/HTTP entry).normative requirement in the spec, each tagged with its RFC-2119 clause, a
vantage (where it's observable: in-view / host / server), and status.
24 of 45 implemented today (in-view automatic checks plus 6 human-in-the-loop).
debug-servereverywhere it belongs: both E2E specs(
servers.spec.ts,generate-grid-screenshots.spec.ts) and both publishworkflows (
pkg-pr-newplus npm matrix).Testing
npm run buildplusnpm run --workspace examples/conformance-server build: cleannpm run prettier: cleanrendering);
servers.spec.tspasses for the new server.Checklist
npm run prettier)🤖 Generated with Claude Code