Skip to content

feat(report): add verified static joint-evidence reports - #5

Open
Cubits11 wants to merge 1 commit into
AmenRa:mainfrom
Cubits11:codex/joint-evidence-reporter
Open

Cubits11 wants to merge 1 commit into
AmenRa:mainfrom
Cubits11:codex/joint-evidence-reporter

Conversation

@Cubits11

Copy link
Copy Markdown

Summary

This adds Report.joint_evidence() for a declared parallel full-shadow OR over
saved per-item score maps. It reports the fixed population/denominator, score
thresholds, caller-declared guard configuration, full-exposure checks,
per-guard detection, union detection, static all-miss, residual coverage,
pairwise intersections, benign union burden when negatives exist, and an
explicit “not reported” uncertainty status for a fixed evaluated population.

The benchmark now writes an additive sidecar manifest with exact dataset and
prediction-file hashes. Joint reports require matching sidecars by default;
legacy maps can be inspected only with require_manifest=False and are labelled
unverified. The benchmark rejects a score-count mismatch, duplicate dataset IDs,
and a source file that changes while it is scored. The report similarly rejects
input changes while it is prepared.

Why this shape

The statistic is deliberately narrow: it makes static all-miss observable only
when all selected guards scored the same labelled items. It does not infer
runtime route risk, adaptive robustness, calibration, causal complementarity,
or deployment safety.

Validation

.venv/bin/black --check guardbench/datasets/__init__.py guardbench/benchmark/effectiveness.py guardbench/report.py guardbench/utils.py tests/unit/guardbench/report_test.py
.venv/bin/isort --check guardbench/datasets/__init__.py guardbench/benchmark/effectiveness.py guardbench/report.py guardbench/utils.py tests/unit/guardbench/report_test.py
TEST_BASE=$(mktemp -d) GUARDBENCH_BASE_PATH="$TEST_BASE" make test
.venv/bin/ruff check tests/unit/guardbench/report_test.py
git diff --check e57e4aaa7d82a8ab0716f7261f816fb0056710c1...be4616b5ab73fa16e820a9f887f97a926d182a6d

Observed locally: formatting/import checks passed; 19 passed; focused new
report tests passed; targeted test lint passed; diff check passed. The upstream
repository's broader Ruff baseline still reports 23 pre-existing diagnostics in
the touched source files, unchanged in kind/count from e57e4aa.

Compatibility and maintenance boundaries

  • Existing benchmark(..., **kwargs) behavior is preserved; arbitrary
    moderator kwargs—including a pre-existing model_configuration kwarg—still
    reach the moderator.
  • Existing reports and result JSON remain unchanged. The only new benchmark
    output is an additive <model>.manifest.json sidecar.
  • The new method refuses legacy maps by default, but callers can explicitly use
    require_manifest=False; that output is marked unverified_legacy_maps.
  • The model_configurations report field is caller-declared at reporting time,
    not inferred from a filename or authenticated by a sidecar.
  • The sidecar detects byte drift in the present local files. It is neither
    signed nor immutable and does not authenticate weights, code, or remote
    services.

Full local diff

The Git commit is the non-duplicated source of the complete patch:

git -C /Users/pranavbhave/Documents/GitHub/guardbench-joint-evidence show --format=fuller --binary be4616b5ab73fa16e820a9f887f97a926d182a6d
git -C /Users/pranavbhave/Documents/GitHub/guardbench-joint-evidence diff --binary e57e4aaa7d82a8ab0716f7261f816fb0056710c1...be4616b5ab73fa16e820a9f887f97a926d182a6d

The contribution check found no upstream CONTRIBUTING, pull-request template,
CLA, DCO, or local submission rule. This payload deliberately avoids broader
claims about runtime route risk, adaptive robustness, calibration, causal
complementarity, or deployment safety.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant