feat(quality): add local context evidence - #303
Conversation
This reverts commit 27a80f8.
📝 WalkthroughWalkthroughAdded context-evidence generation and validation for behavior ownership. The tooling executes exact pytest nodes with coverage contexts, writes bounded non-authoritative artifacts, validates integrity and completeness, exposes CLI commands, and documents local operation. ChangesContext evidence
Sequence Diagram(s)sequenceDiagram
participant CLI
participant build_context_evidence
participant pytest
participant Coverage
participant Artifact
CLI->>build_context_evidence: Provide target and test module
build_context_evidence->>pytest: Collect and execute exact nodes
pytest->>Coverage: Record per-test coverage contexts
Coverage-->>build_context_evidence: Return covered lines
build_context_evidence->>Artifact: Write digest-bound evidence
Artifact-->>CLI: Return generation result
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
backend/tests/test_behavior_ownership.py (1)
879-950: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd fail-closed cases for the post-execution branches.
test_context_builder_fails_closed_before_executionstops at the collection phase. The followingbuild_context_evidenceguards have no test:
context_test_failurewhen the execution run returns a non-zero code.incomplete_context_executionwhencollectedorcompleteddoes not equal the collected nodes.weakened_context_executionwhen a node is skipped or deselected.missing_context_coveragewhen the coverage file is absent.context_runtime_exceededwhen the execution subprocess raisessubprocess.TimeoutExpired.These branches carry the evidence-integrity guarantees of the tool. The
fake_runhelper at Line 660 already provides the injection point for each case.As per coding guidelines: "New or materially changed backend subsystems must maintain at least 90% test coverage".
Do you want me to generate the additional parameterized cases?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/tests/test_behavior_ownership.py` around lines 879 - 950, Add parameterized fail-closed cases to test_context_builder_fails_closed_before_execution, extending the fake_run injection used by build_context_evidence to cover non-zero execution, incomplete collected/completed nodes, skipped or deselected nodes, missing coverage output, and subprocess.TimeoutExpired. Assert BehaviorOwnershipError with context_test_failure, incomplete_context_execution, weakened_context_execution, missing_context_coverage, and context_runtime_exceeded respectively, while preserving the existing pre-execution cases.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/scripts/behavior_ownership.py`:
- Around line 649-663: The collection and execution phases must share one total
runtime budget, and both timeout paths must raise
BehaviorOwnershipError("context_runtime_exceeded"). In
backend/scripts/behavior_ownership.py lines 649-663, update
build_context_evidence to compute remaining time from started before each
subprocess.run, pass that remaining value as timeout, and catch
subprocess.TimeoutExpired for both collection and execution. In
docs/operations_backend_testing.md lines 236-238, retain the “runs over two
minutes” wording once the shared budget is implemented; otherwise revise it to
describe per-phase limits and the post-execution total check.
- Around line 557-566: The _coverage_lines_by_context flow should record a short
evidence-limitation note that fixture setup and teardown coverage contexts are
intentionally excluded because only |run contexts are processed. Add this note
alongside the existing missing_test_context handling without changing the
current context filtering or result behavior.
---
Nitpick comments:
In `@backend/tests/test_behavior_ownership.py`:
- Around line 879-950: Add parameterized fail-closed cases to
test_context_builder_fails_closed_before_execution, extending the fake_run
injection used by build_context_evidence to cover non-zero execution, incomplete
collected/completed nodes, skipped or deselected nodes, missing coverage output,
and subprocess.TimeoutExpired. Assert BehaviorOwnershipError with
context_test_failure, incomplete_context_execution, weakened_context_execution,
missing_context_coverage, and context_runtime_exceeded respectively, while
preserving the existing pre-execution cases.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b41d300f-16d9-41f0-8c44-16cc994036cc
📒 Files selected for processing (8)
.agent-loop/initiatives/WS-QUAL-002-behavior-ownership-catalogue/CHUNK_MAP.md.agent-loop/initiatives/WS-QUAL-002-behavior-ownership-catalogue/STATUS.md.agent-loop/initiatives/WS-QUAL-002-behavior-ownership-catalogue/chunks/WS-QUAL-002-02-context-evidence.md.agent-loop/initiatives/WS-QUAL-002-behavior-ownership-catalogue/reviews/WS-QUAL-002-02-internal-review-evidence.md.agent-loop/initiatives/WS-QUAL-002-behavior-ownership-catalogue/reviews/WS-QUAL-002-02-pr-trust-bundle.mdbackend/scripts/behavior_ownership.pybackend/tests/test_behavior_ownership.pydocs/operations_backend_testing.md
| for context in sorted(data.measured_contexts()): | ||
| node = context.removesuffix("|run") | ||
| if node not in completed_nodes: | ||
| continue | ||
| data.set_query_contexts([f"^{re.escape(context)}$"]) | ||
| lines = data.lines(str(target_path.resolve())) or [] | ||
| result.setdefault(node, set()).update(lines) | ||
| data.set_query_contexts(None) | ||
| if set(result) != completed_nodes: | ||
| raise BehaviorOwnershipError("missing_test_context") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
pytest-cov cov-context=test coverage context name suffix run setup teardown
💡 Result:
When you use the --cov-context=test option with pytest-cov, the plugin records separate coverage data for individual tests during a single test suite execution [1][2]. This functionality leverages the context-switching capabilities of Coverage.py [1][3]. The context name recorded in the coverage database is constructed by combining the pytest test ID with the specific phase of execution [1][4]. These two components are separated by a pipe symbol (|) [1][5]: Context format: <test_id>| The phase can be one of three values: setup, run, or teardown [1][4]. Examples of these contexts include: - test_functions.py::test_addition|run [1][4] - test_fancy.py::test_parametrized[1-101]|setup [1][4] - test_oldschool.py::RegressionTests::test_error|run [1][4] For parameterized tests, the test ID includes the specific parameter values, and each set of parameters is recorded as a distinct context [1][5]. To visualize these contexts in an HTML coverage report, you must enable the show_contexts option in your.coveragerc file [1][5]: [html] show_contexts = True Once enabled, the HTML report will display an annotation on covered lines indicating the number of contexts that executed that line; clicking the annotation reveals the list of specific contexts [1][5].
Citations:
- 1: https://pytest-cov.readthedocs.io/en/latest/contexts.html
- 2: https://pytest-cov.readthedocs.io/en/stable/readme.html
- 3: https://coverage.readthedocs.io/en/latest/contexts.html
- 4: https://github.com/pytest-dev/pytest-cov/blob/master/docs/contexts.rst
- 5: https://pytest-cov.readthedocs.io/en/stable/contexts.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the dependency specification and the relevant behavior helper.
printf 'Dependency references:\n'
rg -n "pytest-cov|cov-context" -S pyproject.toml backend backend/scripts || true
printf '\nbehavior_ownership relevant section:\n'
sed -n '530,575p' backend/scripts/behavior_ownership.py
printf '\nCall sites / usage of _coverage_lines_by_context:\n'
rg -n "_coverage_lines_by_context|BehaviorOwnershipError|missing_test_context" backend/scripts/behavior_ownership.pyRepository: Flow-Research/workstream
Length of output: 11234
Document fixture coverage as intentionally excluded.
pytest-cov==7.1.0 uses <test_id>|<phase> contexts where phase is setup, run, or teardown. _coverage_lines_by_context only accepts |run, so fixture setup teardown coverage lines are not included. If a node has coverage only in setup or teardown (for example |setup), missing_test_context is raised. Add a short evidence-limitation note recording this fixture-line exclusion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/scripts/behavior_ownership.py` around lines 557 - 566, The
_coverage_lines_by_context flow should record a short evidence-limitation note
that fixture setup and teardown coverage contexts are intentionally excluded
because only |run contexts are processed. Add this note alongside the existing
missing_test_context handling without changing the current context filtering or
result behavior.
Source: Linters/SAST tools
2 participants
Workstream PR Trust Bundle
Chunk
WS-QUAL-002-02- Local Coverage-Context EvidenceGoal
Emit bounded, non-authoritative callable-to-test candidate evidence from exact
local coverage contexts without adding hosted CI infrastructure.
Intent And Planning Context
../INTENT.md../chunks/WS-QUAL-002-02-context-evidence.mdWhat Changed
context-evidencegeneration and validation commands.and non-authoritative-schema controls.
Why It Changed
Imports and coverage totals do not show which exact tests execute a callable.
This bounded evidence makes later ownership review better informed without
claiming that inferred execution proves assertions or reviewed ownership.
Design Chosen
One local-only artifact uses the existing lane plugin for collection and
completion, coverage.py test contexts for line evidence, and an independent
schema that catalogue validation cannot consume.
Alternatives Rejected
cost or required gate.
behavior execution.
not reviewed ownership.
Scope Control
Allowed Files Changed
backend/scripts/behavior_ownership.pyandrun_test_lanes.py.Files Outside Stated Scope
Product Behavior
Evidence
Commands Run
cd backend .venv/bin/ruff check scripts/behavior_ownership.py scripts/run_test_lanes.py \ tests/test_behavior_ownership.py tests/test_ci_test_lanes.py .venv/bin/python -m pytest -q \ tests/test_behavior_ownership.py tests/test_ci_test_lanes.py .venv/bin/coverage erase .venv/bin/python -m pytest -q --cov=scripts.behavior_ownership \ --cov-report=term tests/test_behavior_ownership.py python3 ../scripts/check_markdown_links.py python3 ../scripts/check_stale_workstream_wording.py git diff --check origin/main...HEADResult Summary
The exact implementation-head calibration completed 90 nodes in 18.12 seconds
and emitted 141,701 bytes, below both adoption limits.
Acceptance Criteria Proof
run_test_lanes.py.digest-mismatched, overwritten, or timeout evidence.
Test Delta
Tests Added
collection, completion, skip/deselect, coverage, and privacy behavior.
Tests Modified
Tests Removed Or Skipped
Internal Reviewer Results
Reviewed implementation SHA:
836865b1d84d06f534c0c45f551e72f335d810aaReviewed at: 2026-08-09
Reviewer run IDs:
qual002_02_{arch,qa,security,ci,reuse,test_delta}External Review
CI And Gate Integrity
Remaining Risks
The artifact is candidate evidence only. It cannot establish reviewed ownership
and no authoritative catalogue consumer accepts it.
Follow-Up Work
Use this evidence during later bounded catalogue-population chunks. Promote a
public lane-runner API only if a second consumer needs the same custody path.
Human Review Focus
Please inspect the non-authoritative boundary, shared runtime budget,
secret-free subprocesses, exact-head callable custody, and lack of hosted-CI
changes.
Human Merge Ownership