[burtenshaw]

Summary

Add tool/task declaration accuracy, rubric introspection and per-step reward attribution on the merged #1321.

declared capabilities + observed discovery/telemetry → four runtime graders

Omitted declarations remain distinct from explicit empty assertions. Discovery failures cannot become empty successes. Rubric configuration comparisons retain exact decimal values; supported stock aggregations are checked against fresh per-step scores and rewards.

Evidence

• Before: the legacy non-MCP reply could fail an empty declaration, and float decoding could hide a precise rubric-configuration change.
• After: discovery accepts explicit UNKNOWN_TYPE/METHOD_NOT_FOUND signals and the narrowly validated legacy -32603 reply with the exact message Environment does not support MCP. Other internal errors, authorization refusals and malformed envelopes remain FAIL; new non-MCP servers emit METHOD_NOT_FOUND.
• Tool/task checks use completed discovery independently of episode startup. Confirmed declaration mismatches survive ordinary episode failure, JSON support limits and cancellation. Completed grader findings are retained incrementally, and recovery cannot bypass subject teardown. Source-verification failure withdraws all affected runtime findings.
• Explicitly unsupported telemetry produces named SKIPs. A generic Rubric introspection unavailable exception marker remains FAIL. Unsupported custom aggregation semantics remain explicit; established mismatches are retained.

Test Plan

Final Linux qualification at 7490ebe276535f47ea0697e20d26ae76703c3931: 1,493 locked fast, 138 protocol and 58 Docker tests passed, with zero failures, errors or skips. All suite manifests are clean, exact-head and require-complete; all artifact hashes, required IDs and empty run-label cleanup records verified. The merged main tree equals this tested tree.

The prior 5fa87990 run completed all 58 Docker cases with 55 passes and three stale blanket-SKIP assertion failures. The correction requires completed tool/task discovery PASS, the other 13 non-startup checks SKIP, and validates the discovery payload. Timeout, signal, trace-prefix and cleanup assertions remain strict. Three retained-report regressions and 18 corruption plus 135 wrong-status controls passed independent review. All 58 cases and production code were preserved.

Discovery scheduling, incremental finding retention, unconditional teardown, exact rubric comparisons and retained evidence received independent source reviews. The lab keeps serial Docker execution with a 30-minute suite budget and 45-minute workflow budget. Linux Docker evidence does not qualify live HF runtime or every RFC/provider seam.

RFC Status

RFC 008 / #778 remains In Review. Builds on merged #1321; adds four runtime implementations, moving 12/16 to 16/16. This counts implemented grader IDs, not qualification of every provider, capability, task or rubric. Provider gaps and unsupported aggregation modes remain explicit.

Merge Danger

Door: Two-way.

Blast Radius: Grading.

Changes declaration applicability, discovery collection and rubric/attribution evidence.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Alignment Review Report

Automated Checks

• Lint: PASS — usort check, ruff format --check, and ruff check all pass clean on every file this PR touches (src/openenv/validation/graders/runtime/discovery.py, manifest.py, runner.py, runtime/{artifacts,collector,contracts,discovery,replay}.py, and all new/changed test files). The two pre-existing usort findings (tests/envs/test_grid_world.py, tests/envs/test_julia_env.py) and the 58 ruff format findings on unrelated markdown/env files are identical on the base commit (88232843) — none of this PR's changes.
• Debug code: CLEAN — no print(), TODO/FIXME/XXX/HACK, or debugger statements in any file this PR touches.
• Tests: tests/test_validation/ — 786 passed, 36 skipped. Full repo suite — 3777 passed, 177 skipped, 2 pre-existing failures in tests/envs/test_websockets.py (test_protocol_client_state, test_protocol_client_multiple_episodes) confirmed identical on base 88232843 — unrelated to this change (echo_env step_count tracking, outside the validation module).

Open RFCs Context

• RFC 008 (Environment Auto-Validation) — In Review, authored by @zkwentz. This PR directly implements RFC 008's "Declaration and rubric evidence (PR6)" section, which the PR itself adds to the RFC text alongside the implementation. The grader semantics (weighted_sum/sequential/gate aggregation, fail-fast on zero, gate threshold direction, weights summing to 1.0 with abs_tol=1e-6) were cross-checked against the production Rubric/Sequential/Gate/WeightedSum implementations in src/openenv/core/rubrics/containers.py and match exactly. The RubricNode wire schema this PR consumes (name, class_name, children, aggregation, config, config_available, dotted child-naming convention) matches src/openenv/core/env_server/session_telemetry.py byte-for-byte, which predates this PR (stacked on #1246/#1247 in this same chain). No conflict.
• RFC 004 (Rubrics) — In Review. Aggregation semantics match as above. No conflict.
• RFC 011 (ARD-backed catalog discovery) — Draft, authored by @thegovind. Different feature despite the shared word "discovery": RFC 011 is about environment-catalog/package discovery for selecting an unfamiliar environment, while this PR's ToolDeclarationAccuracyGrader/TaskDeclarationAccuracyGrader validate an already-selected subject's declared MCP tools/tasks against runtime reality. RFC 011 explicitly notes "Basic discovery does not depend on RFC 008 validation reports; any claim of a validated property does," so the two are anticipated to coexist. No conflict, flagging only for naming-overlap awareness.
• Other open RFCs (000–003, 005, 006, 010, 012) are unrelated to this change's surface area.

Tier 1: Fixes Required

None — lint, formatting, and debug-code checks are clean on every file this PR touches.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. This is validation/test infrastructure under src/openenv/validation/, not the core Environment/EnvClient API:

• Agent isolation / dual API boundary: tool discovery calls tools/list only (never tools/call) over the validator's own measured WebSocket session — it inspects the subject server for conformance, exactly like the existing reset/step/state replay checks already do. It does not expose reset()/step() to an agent and does not change what any agent can call.
• Client-server separation: all changes stay inside src/openenv/validation/; no cross-boundary imports.
• Rewards in environment: RewardAttributionGrader only verifies that the subject's own emitted reward matches its own rubric tree's root score — it computes nothing externally.
• No credential exposure: new code is unusually careful here — contains_credential() checks (including Unicode-escape evasion), the discovery.json artifact redaction path, and _json_float's rejection of non-finite JSON numbers are all covered by dedicated tests (test_discovery_credential_echo_is_discarded_even_with_unicode_escapes, test_discovery_artifact_redacts_tool_task_and_error_credentials, etc.).
• SSRF/network hygiene in runtime/discovery.py: trust_env=False, follow_redirects=False, explicit validation of the /list_environments namespace string (rejects ., .., /, \) before building the next request path, and byte/response-size/time budgets throughout.

RFC Conflicts

None identified beyond the naming-overlap note above (RFC 011, Draft, no implementation yet — no actual collision).

Summary

• 0 mechanical issues to fix
• 0 alignment points for human review
• 0 RFC conflicts to discuss

This is a clean, well-tested, incremental completion of RFC 008's declaration/rubric-attribution checks, consistent with the existing rubric and session-telemetry contracts it depends on.

  

_{Sent by Cursor Automation: Pre-review}

[rycerzes]

Reviewed at cad68dd3 and ran the pinned lab on Docker 29.7.2, linux/amd64. fast, protocol and docker (28/28) pass. Two comments inline.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Unsupported discovery treated as failure
  • Legacy unsupported MCP, telemetry, and rubric-introspection responses now produce named skips while genuine collection errors still fail.

Or push these changes by commenting:

@cursor push 08a327ed87

Preview (08a327ed87)

diff --git a/src/openenv/validation/graders/runtime/discovery.py b/src/openenv/validation/graders/runtime/discovery.py
--- a/src/openenv/validation/graders/runtime/discovery.py
+++ b/src/openenv/validation/graders/runtime/discovery.py
@@ -236,7 +236,7 @@
         if type(payload["schema_version"]) is not int or payload["schema_version"] != 1:
             raise ValueError("unsupported telemetry version")
         if payload.get("rubric_error") is not None:
-            return ["subject reported unavailable rubric introspection"], None
+            return [], "unsupported capability: rubric_introspection"
         nodes = _rubric_nodes(payload["rubric"])
         return (
             ["rubric configuration is unavailable"]
@@ -250,7 +250,9 @@
     depends_on = ("runtime.startup", "runtime.rubric_introspectable")
 
     def check(self, subject, payload):
-        problems, _ = super().check(subject, payload)
+        problems, incomplete = super().check(subject, payload)
+        if incomplete:
+            return problems, incomplete
         if subject.runtime_evidence.failure_reason:
             problems.append("wire replay is incomplete")
         if problems:

diff --git a/src/openenv/validation/runtime/collector.py b/src/openenv/validation/runtime/collector.py
--- a/src/openenv/validation/runtime/collector.py
+++ b/src/openenv/validation/runtime/collector.py
@@ -258,7 +258,10 @@
                         and 16 <= len(data["capability"]) <= 256
                     ):
                         capability = data["capability"]
-                    elif code == WSErrorCode.UNKNOWN_TYPE:
+                    elif code in {
+                        WSErrorCode.UNKNOWN_TYPE,
+                        WSErrorCode.VALIDATION_ERROR,
+                    }:
                         telemetry_unavailable = (
                             "unsupported capability: session_telemetry"
                         )
@@ -313,8 +316,15 @@
                         if (
                             isinstance(error, dict)
                             and type(error.get("code")) is int
-                            and error["code"] == -32601
                             and isinstance(error.get("message"), str)
+                            and (
+                                error["code"] == -32601
+                                or (
+                                    error["code"] == -32603
+                                    and error["message"]
+                                    == "Environment does not support MCP"
+                                )
+                            )
                             and rpc.get("result") is None
                         ):
                             tools_unavailable = "unsupported capability: tool_discovery"

diff --git a/tests/test_validation/test_discovery_collection.py b/tests/test_validation/test_discovery_collection.py
--- a/tests/test_validation/test_discovery_collection.py
+++ b/tests/test_validation/test_discovery_collection.py
@@ -300,7 +300,7 @@
                     },
                 },
             },
-            "fail",
+            "skip",
         ),
         (
             {

diff --git a/tests/test_validation/test_runtime_discovery.py b/tests/test_validation/test_runtime_discovery.py
--- a/tests/test_validation/test_runtime_discovery.py
+++ b/tests/test_validation/test_runtime_discovery.py
@@ -364,7 +364,6 @@
         "nonfinite_score",
         "stale_score",
         "bad_version",
-        "rubric_error",
         "empty_child_segment",
     ],
 )
@@ -387,14 +386,22 @@
     elif change == "empty_child_segment":
         payload["rubric"][0]["children"][0] = "root."
         payload["rubric"][1]["name"] = "root."
-    else:
-        payload["rubric_error"] = "private-error"
     result = RubricIntrospectableGrader().run(subject(tmp_path, snapshot=payload))
     assert result.status is CheckStatus.FAIL
-    assert "private-error" not in result.model_dump_json()
 
 
 @pytest.mark.parametrize(
+    "grader", [RubricIntrospectableGrader, RewardAttributionGrader]
+)
+def test_unavailable_rubric_introspection_is_a_named_skip(tmp_path, grader):
+    payload = telemetry()
+    payload["rubric_error"] = "private-error"
+    result = grader().run(subject(tmp_path, snapshot=payload))
+    assert result.status is CheckStatus.SKIP
+    assert result.evidence == ["unsupported capability: rubric_introspection"]
+
+
+@pytest.mark.parametrize(
     "change",
     [
         "root_reward",

diff --git a/tests/test_validation/test_runtime_telemetry_collector.py b/tests/test_validation/test_runtime_telemetry_collector.py
--- a/tests/test_validation/test_runtime_telemetry_collector.py
+++ b/tests/test_validation/test_runtime_telemetry_collector.py
@@ -196,8 +196,12 @@
     assert TOKEN not in repr(result)
 
 
-@pytest.mark.parametrize("code", ["VALIDATION_ERROR", TOKEN])
-def test_optional_opening_refusal_does_not_fail_the_episode(monkeypatch, code):
+@pytest.mark.parametrize(
+    "code,unsupported", [("VALIDATION_ERROR", True), (TOKEN, False)]
+)
+def test_optional_opening_refusal_does_not_fail_the_episode(
+    monkeypatch, code, unsupported
+):
     result = collect(
         monkeypatch,
         opening_reply=json.dumps(
@@ -206,10 +210,17 @@
     )
     assert result.failure_reason is None
     assert result.telemetry_json is None
-    assert (
-        result.telemetry_error
-        == "session telemetry unavailable or authorization refused"
-    )
+    if unsupported:
+        assert result.telemetry_error is None
+        assert (
+            result.telemetry_unavailable == "unsupported capability: session_telemetry"
+        )
+    else:
+        assert (
+            result.telemetry_error
+            == "session telemetry unavailable or authorization refused"
+        )
+        assert result.telemetry_unavailable is None
     assert [row.operation for row in result.exchanges] == [
         "reset",
         "state",

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Discovery graders skip after episode failure
  • Declaration graders now run independently on preserved evidence, including primary JSON-limit prefixes, so mismatches remain failures.
• ✅ Fixed: Non-MCP tools/call still internal error
  • Non-MCP tools/call now returns METHOD_NOT_FOUND consistently with tools/list.

Preview (4e1f729006)

diff --git a/.github/workflows/validation-runtime.yml b/.github/workflows/validation-runtime.yml
--- a/.github/workflows/validation-runtime.yml
+++ b/.github/workflows/validation-runtime.yml
@@ -15,7 +15,7 @@
       contents: read
     name: Runtime validation / Linux Docker
     runs-on: ubuntu-24.04
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:

diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -69,10 +69,11 @@
 during the fresh-container replay. Both use the same immutable image and are
 removed even if a check fails.
 
-The runtime level implements startup, reward, observation, state, network policy,
-host containment, resource bounds, episode isolation, oracle containment, seed
-forwarding, determinism and emitted trajectory record checks. Other applicable
-Level 2 checks appear explicitly as `SKIP`; they make
+The 16 implemented runtime checks cover startup, rewards, observation schemas,
+state continuity, network policy, host containment, resource bounds, episode isolation,
+oracle containment, seed forwarding, determinism, emitted trajectory records, tool
+and task discovery, rubric introspection and reward attribution. Applicable checks
+with unsupported capabilities or incomplete evidence appear as `SKIP`; they make
 the result `WARN`, which exits zero and does not mean Level 2 is complete.
 `FAIL` exits 1, unsupported package formats exit 2, and internal or policy errors
 exit 3. `--level semantic`
@@ -144,7 +145,9 @@
 full agent boundary.
 `--local` explicitly selects this default package mode and rejects a remote URL.
 Explicitly unsupported session telemetry yields a `SKIP` naming `session_telemetry`;
-authorization refusals and malformed telemetry remain failures. Seed control
+authorization refusals and malformed telemetry remain failures. Explicitly unsupported
+`tools/list` similarly skips with `tool_discovery` named. Internal discovery errors
+and malformed responses remain failures, even for an empty declared tool set. Seed control
 reports observed forwarding to reset, separately from empirical determinism.
 The declared `episode_timeout_s` bounds each episode; a reset or judged step may use
 its remaining budget. Original and replay collection share a 300-second deadline

diff --git a/rfcs/008-environment-auto-validation.md b/rfcs/008-environment-auto-validation.md
--- a/rfcs/008-environment-auto-validation.md
+++ b/rfcs/008-environment-auto-validation.md
@@ -660,6 +660,11 @@
 Stock container aggregation is named explicitly; custom rubrics may supply
 `validation_config()` to expose public JSON configuration. Arbitrary attributes
 and `state_dict()` are never serialized as configuration.
+Attribution compares each numeric step reward with its freshly evaluated root
+score. Null non-terminal rewards have no emitted score to compare; their records
+must still have the correct identity, unchanged configuration, and valid evaluated
+aggregation. An entirely unscored episode is incomplete rather than passing
+attribution. Terminal null rewards remain invalid.
 
 The subject server also emits a bounded record of the reset/step/state request
 and response envelopes it executed. The validator captures the wire separately
@@ -668,6 +673,27 @@
 Missing configuration, attribution or records cannot be inferred from other
 successful operations. This transport introduces no new passing grader by itself.
 
+#### Declaration and rubric evidence (PR6)
+
+The validator discovers MCP tools over the authenticated replay connection before
+reset and compares their names with the declaration, including an explicitly empty
+set. Task discovery queries split names and authoritative per-split counts, then
+reads at most two individual tasks per split. Preview length is never treated as
+the dataset size. Requests share the collection deadline and bounded response
+budgets; unavailable or malformed discovery cannot become an empty passing set.
+The task namespace comes from the server's single `/list_environments` entry,
+independently of the package name. Omitted tool/count declarations stay unknown;
+explicit `[]`/`{}` declarations assert zero tools/splits. A declared task API
+without declared counts leaves count accuracy incomplete.
+
+Rubric checks require the declared tree and explicit public configuration, then
+compare every observed step reward with its fresh root score and named child
+attribution. Built-in weighted, sequential and gated aggregation is checked using
+its published configuration. Unknown custom aggregation remains incomplete.
+Optional task and rubric checks are selected only when declared. `discovery.json`
+retains raw discovery results, availability and sanitized errors alongside session
+telemetry; ordinary discovery responses never enter the episode trajectory.
+
 Applicability predicates must distinguish empty declared sets from absent
 capabilities. Missing subject features, missing provider support and checks whose
 implementation has not shipped are distinct outcomes. Independent graders must

diff --git a/scripts/validation/reproduce.py b/scripts/validation/reproduce.py
--- a/scripts/validation/reproduce.py
+++ b/scripts/validation/reproduce.py
@@ -512,7 +512,7 @@
                 ],
                 cwd=output,
                 env=environment,
-                timeout=1200 if args.suite == "docker" else 600,
+                timeout=1800 if args.suite == "docker" else 600,
                 log=output / "logs/tests.log",
             )
             status = 0

diff --git a/src/openenv/core/env_server/http_server.py b/src/openenv/core/env_server/http_server.py
--- a/src/openenv/core/env_server/http_server.py
+++ b/src/openenv/core/env_server/http_server.py
@@ -1264,7 +1264,7 @@
                                 request_id=request_id,
                             )
                         return JsonRpcResponse.error_response(
-                            JsonRpcErrorCode.INTERNAL_ERROR,
+                            JsonRpcErrorCode.METHOD_NOT_FOUND,
                             "Environment does not support MCP",
                             request_id=request_id,
                         )
@@ -1368,7 +1368,7 @@
                                 request_id=request_id,
                             )
                         return JsonRpcResponse.error_response(
-                            JsonRpcErrorCode.INTERNAL_ERROR,
+                            JsonRpcErrorCode.METHOD_NOT_FOUND,
                             "Environment does not support MCP",
                             request_id=request_id,
                         )

diff --git a/src/openenv/validation/graders/runtime/discovery.py b/src/openenv/validation/graders/runtime/discovery.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/graders/runtime/discovery.py
@@ -1,0 +1,367 @@
+"""Declaration and rubric checks over bounded, independently collected evidence."""
+
+import json
+import math
+import time
+from decimal import Decimal
+
+from ...report import CheckResult
+from ...types import CheckStatus
+from .basic import _RuntimeGrader
+from .repeatability import _difference
+
+
+def _finite(value):
+    try:
+        return type(value) in (int, float, Decimal) and math.isfinite(value)
+    except OverflowError:
+        return False
+
+
+def _json_decimal(value):
+    number = float(value)
+    if not math.isfinite(number):
+        raise ValueError("runtime JSON numbers must be finite")
+    # Retain the original token for configuration equality.
+    return Decimal(value)
+
+
+def _arithmetic_value(value):
+    return float(value) if isinstance(value, Decimal) else value
+
+
+class _EvidenceGrader(_RuntimeGrader):
+    """Discovery can be measured without an episode step; missing evidence cannot pass."""
+
+    unavailable_field = None
+
+    def run(self, subject):
+        started = time.monotonic()
+        problems, incomplete = [], None
+        evidence = subject.runtime_evidence
+        if not self.applies_to(subject.manifest):
+            incomplete = "declared capability does not apply"
+        elif evidence is None:
+            incomplete = "runtime evidence is unavailable"
+        elif getattr(evidence, self.error_field):
+            problems = [f"{self.label} collection failed"]
+        elif self.unavailable_field and getattr(evidence, self.unavailable_field):
+            incomplete = getattr(evidence, self.unavailable_field)
+        elif getattr(evidence, self.field) is None:
+            incomplete = f"missing prerequisite: {self.label} evidence"
+        else:
+            try:
+                payload = json.loads(
+                    getattr(evidence, self.field),
+                    parse_float=_json_decimal,
+                    parse_constant=_json_decimal,
+                )
+                if not isinstance(payload, dict):
+                    raise ValueError("evidence must be an object")
+                problems, incomplete = self.check(subject, payload)
+            except (ValueError, TypeError, KeyError, RecursionError, OverflowError):
+                problems = [f"malformed {self.label} evidence"]
+        status = (
+            CheckStatus.FAIL
+            if problems
+            else CheckStatus.SKIP
+            if incomplete
+            else CheckStatus.PASS
+        )
+        return CheckResult(
+            check_id=self.check_id,
+            status=status,
+            evidence=problems[:20]
+            or [incomplete or f"observed {self.label} contract holds"],
+            duration_s=time.monotonic() - started,
+        )
+
+
+class ToolDeclarationAccuracyGrader(_EvidenceGrader):
+    check_id = "runtime.tool_declaration_accuracy"
+    depends_on = ()
+    field, error_field, label = "tools_json", "tools_error", "tool discovery"
+    unavailable_field = "tools_unavailable"
+
+    def applies_to(self, manifest):
+        return "declared_tools" in manifest.capabilities.model_fields_set
+
+    def check(self, subject, payload):
+        tools = payload["tools"]
+        if not isinstance(tools, list):
+            raise ValueError("tools must be a list")
+        names = [tool["name"] for tool in tools]
+        if any(not isinstance(name, str) or not name for name in names):
+            raise ValueError("invalid tool name")
+        cursor = payload.get("nextCursor")
+        if cursor is not None and not isinstance(cursor, str):
+            raise ValueError("invalid tool pagination cursor")
+        declared = subject.manifest.capabilities.declared_tools
+        problems = []
+        if len(set(names)) != len(names):
+            problems.append("discovery contains duplicate tool names")
+        if len(set(declared)) != len(declared):
+            problems.append("declaration contains duplicate tool names")
+        missing = set(declared) - set(names)
+        extra = set(names) - set(declared)
+        if missing and cursor is None:
+            problems.append(f"{len(missing)} declared tools are missing from discovery")
+        if extra:
+            problems.append(f"{len(extra)} discovered tools are undeclared")
+        return problems, (
+            "tool inventory is incomplete: pagination was not collected"
+            if cursor is not None
+            else None
+        )
+
+
+class TaskDeclarationAccuracyGrader(_EvidenceGrader):
+    check_id = "runtime.task_declaration_accuracy"
+    depends_on = ()
+    field, error_field, label = "tasks_json", "tasks_error", "task discovery"
+
+    def applies_to(self, manifest):
+        return manifest.capabilities.task_api or (
+            "declared_task_count" in manifest.capabilities.model_fields_set
+        )
+
+    def check(self, subject, payload):
+        splits, counts, previews = (
+            payload["splits"],
+            payload["counts"],
+            payload["previews"],
+        )
+        if (
+            not isinstance(splits, list)
+            or not isinstance(counts, dict)
+            or not isinstance(previews, dict)
+        ):
+            raise ValueError("invalid task discovery")
+        names = [split["name"] for split in splits]
+        if any(not isinstance(name, str) or not name for name in names) or len(
+            set(names)
+        ) != len(names):
+            raise ValueError("invalid split names")
+        if set(names) != counts.keys() or set(names) != previews.keys():
+            raise ValueError("incomplete split discovery")
+        problems = []
+        declared = subject.manifest.capabilities.declared_task_count
+        has_counts = (
+            "declared_task_count" in subject.manifest.capabilities.model_fields_set
+        )
+        if has_counts and set(names) != declared.keys():
+            problems.append("discovered splits differ from declared task counts")
+        for name in names:
+            count, preview = counts[name], previews[name]
+            if type(count) is not int or count < 0:
+                raise ValueError("invalid task count")
+            if not isinstance(preview, list):
+                raise ValueError("invalid task preview")
+            if len(preview) > count or (count > 0 and not preview):
+                problems.append(
+                    "task preview is inconsistent with the advertised count"
+                )
+            if name in declared and count != declared[name]:
+                problems.append("advertised task count differs from the declaration")
+        return problems, None if has_counts else "task counts were not declared"
+
+
+def _rubric_nodes(payload):
+    if not isinstance(payload, list) or not payload:
+        raise ValueError("missing rubric tree")
+    nodes = {}
+    for node in payload:
+        name = node["name"]
+        if not isinstance(name, str) or not all(name.split(".")) or name in nodes:
+            raise ValueError("invalid rubric name")
+        if not isinstance(node["class_name"], str) or not node["class_name"]:
+            raise ValueError("missing rubric class")
+        children = node["children"]
+        if not isinstance(children, list) or any(
+            not isinstance(child, str) for child in children
+        ):
+            raise ValueError("invalid rubric children")
+        if len(set(children)) != len(children):
+            raise ValueError("duplicate rubric children")
+        if type(node["config_available"]) is not bool or not isinstance(
+            node["config"], dict
+        ):
+            raise ValueError("invalid rubric configuration")
+        if type(node["evaluated"]) is not bool:
+            raise ValueError("invalid evaluation marker")
+        if node["evaluated"] and not _finite(node["score"]):
+            raise ValueError("evaluated score is not finite")
+        if not node["evaluated"] and node["score"] is not None:
+            raise ValueError("unevaluated score must be absent")
+        aggregation = node["aggregation"]
+        if aggregation not in {"weighted_sum", "sequential", "gate", "leaf", "unknown"}:
+            raise ValueError("invalid aggregation")
+        if aggregation == "leaf" and children:
+            raise ValueError("leaf has children")
+        if aggregation == "weighted_sum":
+            weights = node["config"]["weights"]
+            if (
+                not isinstance(weights, list)
+                or len(weights) != len(children)
+                or not all(_finite(weight) for weight in weights)
+            ):
+                raise ValueError("invalid rubric weights")
+            if not math.isclose(
+                sum(_arithmetic_value(weight) for weight in weights),
+                1,
+                rel_tol=0,
+                abs_tol=1e-6,
+            ):
+                raise ValueError("rubric weights do not sum to one")
+        if aggregation == "gate" and (
+            len(children) != 1 or not _finite(node["config"]["threshold"])
+        ):
+            raise ValueError("invalid rubric gate")
+        nodes[name] = node
+    if "root" not in nodes:
+        raise ValueError("missing rubric root")
+    linked = set()
+    for name, node in nodes.items():
+        for child in node["children"]:
+            if (
+                child not in nodes
+                or child in linked
+                or child.rsplit(".", 1)[0] != name
+                or child == name
+            ):
+                raise ValueError("invalid rubric edge")
+            linked.add(child)
+    if linked != nodes.keys() - {"root"}:
+        raise ValueError("disconnected rubric tree")
+    return nodes
+
+
+class RubricIntrospectableGrader(_EvidenceGrader):
+    check_id = "runtime.rubric_introspectable"
+    requires_capabilities = frozenset({"rubric_tree"})
+    field, error_field, label = "telemetry_json", "telemetry_error", "rubric telemetry"
+    unavailable_field = "telemetry_unavailable"
+
+    def applies_to(self, manifest):
+        return manifest.capabilities.rubric_tree
+
+    def check(self, subject, payload):
+        if type(payload["schema_version"]) is not int or payload["schema_version"] != 1:
+            raise ValueError("unsupported telemetry version")
+        if payload.get("rubric_error") is not None:
+            return ["subject reported unavailable rubric introspection"], None
+        nodes = _rubric_nodes(payload["rubric"])
+        return (
+            ["rubric configuration is unavailable"]
+            if any(not node["config_available"] for node in nodes.values())
+            else []
+        ), None
+
+
+class RewardAttributionGrader(RubricIntrospectableGrader):
+    check_id = "runtime.reward_attribution"
+    depends_on = ("runtime.startup", "runtime.rubric_introspectable")
+
+    def check(self, subject, payload):
+        problems, _ = super().check(subject, payload)
+        if subject.runtime_evidence.failure_reason:
+            problems.append("wire replay is incomplete")
+        if problems:
+            return problems, None
+        baseline = _rubric_nodes(payload["rubric"])
+        steps = [
+            row for row in subject.runtime_evidence.exchanges if row.operation == "step"
+        ]
+        records = payload["attribution"]
+        if not isinstance(records, list):
+            raise ValueError("invalid attribution")
+        if not steps:
+            return problems, "missing prerequisite: an observed step reward"
+        if len(records) != len(steps):
+            return problems + ["attribution does not cover every observed step"], None
+
+        def definition(tree):
+            return {
+                name: {
+                    key: value
+                    for key, value in node.items()
+                    if key not in {"score", "evaluated"}
+                }
+                for name, node in tree.items()
+            }
+
+        unsupported = False
+        scored_steps = 0
+        for index, (record, step) in enumerate(zip(records, steps, strict=True)):
+            if type(record["step_index"]) is not int or record["step_index"] != index:
+                problems.append(
+                    f"step {index}: attribution identity differs from the wire"
+                )
+            nodes = _rubric_nodes(record["rubric"])
+            if _difference(definition(nodes), definition(baseline)):
+                problems.append(f"step {index}: rubric configuration changed")
+            data = json.loads(step.response_json)["data"]
+            reward = data["reward"]
+            root = nodes["root"]
+            unscored = reward is None and data.get("done") is False
+            scored_steps += not unscored
+            if not unscored and (
+                not root["evaluated"]
+                or not _finite(reward)
+                or not math.isclose(root["score"], reward, rel_tol=1e-9, abs_tol=1e-9)
+            ):
+                problems.append(
+                    f"step {index}: fresh root score differs from emitted reward"
+                )
+            for node in nodes.values():
+                if not node["evaluated"]:
+                    continue
+                children = [nodes[name] for name in node["children"]]
+                aggregation = node["aggregation"]
+                if aggregation == "unknown":
+                    unsupported = True
+                    continue
+                if aggregation == "leaf":
+                    continue
+                expected = 1.0 if aggregation == "sequential" else 0.0
+                visited = 0
+                for child in children:
+                    if not child["evaluated"]:
+                        problems.append(
+                            f"step {index}: required child score was not evaluated"
+                        )
+                        break
+                    visited += 1
+                    score = _arithmetic_value(child["score"])
+                    if aggregation == "weighted_sum":
+                        expected += score * _arithmetic_value(
+                            node["config"]["weights"][visited - 1]
+                        )
+                    elif aggregation == "gate":
+                        expected = (
+                            0.0
+                            if score < _arithmetic_value(node["config"]["threshold"])
+                            else score
+                        )
+                    else:
+                        expected = score
+                        if score == 0:
+                            break
+                if aggregation == "sequential" and any(
+                    child["evaluated"] for child in children[visited:]
+                ):
+                    problems.append(
+                        f"step {index}: sequential evaluated children after its gate"
+                    )
+                if not _finite(expected) or not math.isclose(
+                    node["score"], expected, rel_tol=1e-9, abs_tol=1e-9
+                ):
+                    problems.append(
+                        f"step {index}: child attribution does not match the parent score"
+                    )
+        incomplete = (
+            "unsupported custom rubric aggregation semantics" if unsupported else None
+        )
+        if not scored_steps:
+            incomplete = "missing prerequisite: an observed numeric step reward"
+        return problems, incomplete

diff --git a/src/openenv/validation/manifest.py b/src/openenv/validation/manifest.py
--- a/src/openenv/validation/manifest.py
+++ b/src/openenv/validation/manifest.py
@@ -9,7 +9,14 @@
 from pathlib import PurePosixPath
 from typing import Annotated, Any, Literal
 
-from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    field_validator,
+    model_serializer,
+    model_validator,
+)
 
 from .types import SignatureKind
 
@@ -241,6 +248,14 @@
     declared_tools: list[str] = Field(default_factory=list)
     declared_task_count: dict[str, int] = Field(default_factory=dict)
 
+    @model_serializer(mode="wrap")
+    def _preserve_declaration_presence(self, serialize):
+        data = serialize(self)
+        for field in ("declared_tools", "declared_task_count"):
+            if field not in self.model_fields_set:
+                data.pop(field, None)
+        return data
+
     @model_validator(mode="after")
     def _injected_state_requires_set_state(self) -> "CapabilitiesSpec":
         if (

diff --git a/src/openenv/validation/runner.py b/src/openenv/validation/runner.py
--- a/src/openenv/validation/runner.py
+++ b/src/openenv/validation/runner.py
@@ -24,6 +24,12 @@
     RewardWellFormedGrader,
     StateContractGrader,
 )
+from .graders.runtime.discovery import (
+    RewardAttributionGrader,
+    RubricIntrospectableGrader,
+    TaskDeclarationAccuracyGrader,
+    ToolDeclarationAccuracyGrader,
+)
 from .graders.runtime.oracle import oracle_inventory
 from .graders.runtime.repeatability import (
     EpisodeDeterminismGrader,
@@ -128,6 +134,10 @@
     registry.register(SeedControlGrader())
     registry.register(EpisodeDeterminismGrader())
     registry.register(TrajectoryRecordGrader())
+    registry.register(ToolDeclarationAccuracyGrader())
+    registry.register(TaskDeclarationAccuracyGrader())
+    registry.register(RubricIntrospectableGrader())
+    registry.register(RewardAttributionGrader())
 
     registry.register(NetworkPolicyGrader())
     registry.register(HostContainmentGrader())
@@ -256,9 +266,9 @@
     if check_id in {"runtime.rubric_introspectable", "runtime.reward_attribution"}:
         return manifest.capabilities.rubric_tree
     if check_id == "runtime.task_declaration_accuracy":
-        return manifest.capabilities.task_api or bool(
-            manifest.capabilities.declared_task_count
-        )
+        return TaskDeclarationAccuracyGrader().applies_to(manifest)
+    if check_id == "runtime.tool_declaration_accuracy":
+        return ToolDeclarationAccuracyGrader().applies_to(manifest)
     return True
 
 
@@ -334,6 +344,8 @@
                 manifest.resources.episode_timeout_s, REPLAY_BUDGET_SECONDS
             ),
             validation_token=spec.env_vars["OPENENV_VALIDATION_TOKEN"],
+            collect_tools=_applicable("runtime.tool_declaration_accuracy", manifest),
+            collect_tasks=_applicable("runtime.task_declaration_accuracy", manifest),
         )
         # Independent isolation and oracle probes do not spend the replay budget.
         replay_remaining_s = max(0.0, replay_deadline - time.monotonic())
@@ -462,6 +474,13 @@
         result = _outcome(
             "runtime.startup", CheckStatus.SKIP, str(exc), started=started
         )
+        subject = replace(subject, runtime_evidence=evidence)
+        checks = execute_graders(
+            graders,
+            subject,
+            provider_capabilities=provider.capabilities,
+            prior=[result],
+        )
     except UnsupportedCapability as exc:
         result = _outcome(
             "runtime.startup", CheckStatus.SKIP, str(exc), started=started
@@ -684,8 +703,12 @@
                 reason = "grader not implemented in this build"
                 if manifest is None:
                     reason = "unmet dependency: valid manifest"
-                elif grader is not None and grader.depends_on:
-                    reason = "unmet dependency: " + ", ".join(grader.depends_on)
+                elif grader is not None:
+                    dependencies = grader.depends_on or (
+                        ("runtime.startup",) if grader.level is Level.RUNTIME else ()
+                    )
+                    if dependencies:
+                        reason = "unmet dependency: " + ", ".join(dependencies)
                 results.append(_outcome(entry.check_id, CheckStatus.SKIP, reason))
         source_problem = None
         if not digest_before:
@@ -702,7 +725,11 @@
                     else "package source could not be verified after validation"
                 )
         if source_problem is not None:
-            invalidated = {"runtime.startup"}
+            invalidated = {
+                "runtime.startup",
+                "runtime.tool_declaration_accuracy",
+                "runtime.task_declaration_accuracy",
+            }
             for grader in order_graders(list(selected.values())):
                 if invalidated.intersection(grader.depends_on):
                     invalidated.add(grader.check_id)

diff --git a/src/openenv/validation/runtime/artifacts.py b/src/openenv/validation/runtime/artifacts.py
--- a/src/openenv/validation/runtime/artifacts.py
+++ b/src/openenv/validation/runtime/artifacts.py
@@ -243,6 +243,27 @@
             after["redacted"] |= before["body"] != after["body"]
             retained["redacted"] |= after["redacted"]
     files["oracle-http.json"] = retained_probes
+    if evidence:
+        discovery = {}
+        omitted = []
+        for name in ("tools", "tasks"):
+            raw = getattr(evidence, name + "_json")
+            discovery[name + "_available"] = raw is not None
+            discovery[name + "_error"] = getattr(evidence, name + "_error")
+            if name == "tools":
+                discovery["tools_unavailable"] = evidence.tools_unavailable
+            try:
+                discovery[name] = (
+                    loads(raw, object_pairs_hook=_unique_object)
+                    if raw is not None
+                    else None
+                )
+            except (ValueError, ArithmeticError, RecursionError):
+                discovery[name] = "[malformed evidence omitted]"
+                omitted.append(name)
+        discovery["omitted_fields"] = omitted
+        discovery["redacted"] = bool(omitted) or _redact(discovery) != discovery
+        files["discovery.json"] = discovery
     if evidence and (evidence.replays or evidence.replay_failure_reason):
         samples = []
         for replay in evidence.replays:
@@ -298,7 +319,12 @@
             "failure_reason": evidence.replay_failure_reason,
             "samples": samples,
         }
-    for name in ("runtime-plan.json", "session-telemetry.json", "replays.json"):
+    for name in (
+        "runtime-plan.json",
+        "session-telemetry.json",
+        "replays.json",
+        "discovery.json",
+    ):
         if name not in files:
             (directory / name).unlink(missing_ok=True)
     digests = []

diff --git a/src/openenv/validation/runtime/collector.py b/src/openenv/validation/runtime/collector.py
--- a/src/openenv/validation/runtime/collector.py
+++ b/src/openenv/validation/runtime/collector.py
@@ -11,6 +11,7 @@
 
 from ...core.env_server.types import WSErrorCode
 from .contracts import RuntimeEvidence, RuntimePlan, WireExchange
+from .discovery import collect_task_evidence
 from .json_values import dumps, loads
 from .transport import abort_socket, http_deadline
 
@@ -82,6 +83,8 @@
     request_timeout_s: float | None = None,
     episodes: int = 1,
     validation_token: str | None = None,
+    collect_tools: bool = False,
+    collect_tasks: bool = False,
     request_telemetry: bool = True,
 ) -> RuntimeEvidence:
     """
@@ -110,6 +113,10 @@
         validation_token (`str`, *optional*):
             Run-scoped credential to omit from evidence; also authorizes optional
             telemetry when `request_telemetry` is enabled.
+        collect_tools (`bool`, *optional*, defaults to `False`):
+            Discover tools on the measured WebSocket before reset.
+        collect_tasks (`bool`, *optional*, defaults to `False`):
+            Discover the server's environment namespace and sample its task metadata.
         request_telemetry (`bool`, *optional*, defaults to `True`):
             Request session telemetry with the provided authorization. Disable
             for scrub-only collection with the ordinary protocol byte limits.
@@ -124,6 +131,8 @@
     trace_bytes = 0
     telemetry_json = None
     telemetry_error = None
+    tools_json = tools_error = tasks_json = tasks_error = None
+    tools_unavailable = None
     telemetry_unavailable = None
     server_code = None
 
@@ -231,10 +240,10 @@
                     raise
                 return connection.recv(timeout=remaining())
 
-            def telemetry_request(operation, data):
+            def telemetry_request(operation, data, max_bytes=MAX_TRACE_BYTES):
                 request = json.dumps({"type": operation, "data": data})
                 raw = receive_response(request)
-                if not isinstance(raw, str) or len(raw.encode()) > MAX_TRACE_BYTES:
+                if not isinstance(raw, str) or len(raw.encode()) > max_bytes:
                     raise ValueError("invalid telemetry response")
                 response = loads(raw)
                 if not isinstance(response, dict):
@@ -296,6 +305,84 @@
                         pending.extend(child)
                 return False
 
+            if collect_tools:
+                phase = "tools/list"
+                try:
+                    response = telemetry_request(
+                        "mcp",
+                        {
+                            "jsonrpc": "2.0",
+                            "id": 1,
+                            "method": "tools/list",
+                            "params": {},
+                        },
+                        MAX_MESSAGE_BYTES,
+                    )
+                    if contains_credential(response):
+                        raise ValueError("discovery contains validation credentials")
+                    rpc = response.get("data")
+                    if _server_error_code(response) == WSErrorCode.UNKNOWN_TYPE:
+                        tools_unavailable = "unsupported capability: tool_discovery"
+                    else:
+                        if (
+                            response.get("type") != "mcp"
+                            or not isinstance(rpc, dict)
+                            or rpc.get("jsonrpc") != "2.0"
+                            or type(rpc.get("id")) is not int
+                            or rpc["id"] != 1
+                        ):
+                            raise ValueError("invalid tool discovery envelope")
+                        error = rpc.get("error")
+                        if (
+                            isinstance(error, dict)
+                            and type(error.get("code")) is int
+                            and isinstance(error.get("message"), str)
+                            and rpc.get("result") is None
+                            and (
+                                error["code"] == -32601
+                                or (
+                                    # Legacy non-MCP tools/list used INTERNAL_ERROR.
+                                    error["code"] == -32603
+                                    and error["message"]
+                                    == "Environment does not support MCP"
+                                    and error.get("data") is None
+                                )
+                            )
+                        ):
+                            tools_unavailable = "unsupported capability: tool_discovery"
+                        else:
+                            if (
+                                error is not None
+                                or not isinstance(rpc.get("result"), dict)
+                                or not isinstance(rpc["result"].get("tools"), list)
+                            ):
+                                raise ValueError("invalid tool discovery response")
+                            discovered = dumps(rpc["result"], ensure_ascii=False)
+                            if len(discovered.encode()) > MAX_MESSAGE_BYTES:
+                                raise ValueError("discovery exceeds size bound")
+                            tools_json = discovered
+                except Exception as exc:
+                    tools_error = f"tool discovery failed ({type(exc).__name__})"
+
+            if collect_tasks:
+                phase = "tasks"
+                try:
+                    discovered, tasks_error = collect_task_evidence(
+                        base_url,
+                        deadline=deadline,
+                        request_timeout_s=request_timeout_s,
+                    )
+                    if discovered is not None:
+                        if contains_credential(discovered) or contains_credential(
+                            json.loads(discovered)
+                        ):
+                            raise ValueError(
+                                "discovery contains validation credentials"
+                            )
... diff truncated: showing 800 of 3844 lines

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 573cbde. Configure here.}