fix: [no-ticket] harbor - access-tier integrity (free-baseline privilege, k-anonymity floor, trusted nested CLI)

[shehabyasser-scale]

Stacked on #33 (harbor-6-selection-integrity). Part of the pre-paper hardening series; closes the access-tier group.

What leaks this closes

1. Free baseline eval was a privilege escalation (server.py). The agent's one budget-free baseline eval called engine.evaluate(admin=True). Admin does not just skip the ledger; it also bypasses the no_access tier gate, so the free eval could target the held-out test split and read its aggregate score straight off the EvalSummary response. Fix: engine.evaluate gains a distinct free parameter that waives only the budget debit and nothing else; admin and free are separate authorities. The freebie is also consumed only after the eval succeeds, so an infra failure no longer burns it.

2. The engine's no_access gate was dormant in the deployed sidecar (serve.py). build_components never passed split_accesses into the EvaluationEngine, so the engine-side hard rejection (added in an earlier PR) never armed. The de-facto gate was the budget ledger (no_access splits are unbudgeted), which is precisely what every unmetered path (admin, the free baseline) skips. Fix: wire split_accesses through; the gate fails closed on unlisted splits.

3. Per-sample label reconstruction on non_viewable splits (server.py, protocol.py). EvalRequest.sample_ids / num_samples let the agent evaluate arbitrary subsets, and the aggregate mean_score in the response over a singleton subset is that sample's score verbatim; n singleton evals reconstruct the hidden split's labels wholesale. Fix: k-anonymity floor (default 5, k_anonymity_floor in build.yaml/serve.json) on agent-chosen proper subsets of non_viewable splits, rejected up front (no commit transfer, no eval, no budget debit) and advertised in status() as min_subset_samples. Full-split evals always pass, so a split smaller than the floor degrades to full-split-only rather than becoming unevaluable. Scope note: the floor multiplies reconstruction cost by ~k (overlapping k-subsets still admit elimination); the binding constraint remains the budget. It is a cost lever, not a proof.

4. The nested scorer CLI resolved from the candidate's own lockfile (runner.py, config.py). Mode B shells out to uv run --project <candidate> harbor run ..., so the orchestrator that writes the trial result.json files came from whatever the agent's pyproject.toml/uv.lock pinned; one edited line could point it at a fork that fabricates perfect rewards without running anything. Fix: HarborConfig.harbor_requirement; when set, the runner layers --with <spec> over the candidate env, and uv's ephemeral overlay takes precedence over a conflicting project pin for both the console script and sys.path (verified empirically: project pinning six==1.16.0, --with six==1.17.0 resolves 1.17.0). Honest residual: agent code still imports into the nested harbor process via --agent-import-path, so in-process tampering remains possible; a full boundary needs out-of-process verification and is tracked for a later PR.

Tests

• test_engine.py: free runs without debiting budget; free does NOT bypass the no_access gate.
• test_harbor_server.py: free baseline passes free=True, admin=False; failed free eval keeps the freebie; floor rejects sub-k subsets before any work, allows at-floor subsets, full splits, viewable splits, admin, and floor<=1; status advertises the floor.
• test_harbor_protocol.py: min_subset_samples advertised for non_viewable only.
• test_harbor_runner.py: --with placement before the harbor executable; absent when unset.
• 158 passed, 2 skipped (pre-existing) across the affected files.

🤖 Generated with Claude Code

Greptile Summary

This PR hardens the eval sidecar's access-tier boundary against four privilege escalation / information-leak vulnerabilities, and adds several supporting mechanics. All four security fixes are correctly implemented and the two previously-flagged issues (free baseline race, build_status floor default) have been addressed.

• Security fixes (core): engine.evaluate now takes a free parameter that waives only budget—not tier access—so the free baseline eval can no longer touch no_access splits; split_accesses is wired into the engine inside build_components so the no_access gate is actually armed; a k-anonymity floor blocks agent-chosen sub-k subsets of non_viewable splits before commit transfer or budget debit; harbor_requirement overlays a trusted CLI spec over the candidate env via uv run --with.
• Supporting mechanics: Fail-closed ledger restore on parse error (corrupt file → zero remaining, not configured full); per-eval result-dir versioning with _eval_seq ordinals; mean_score_se and n_scored/n_errored added to summary.json; infra-dead attempt classification + bounded retry (off by default, with documented adversarial risk); model override for transfer-target evals; exhaust_budget instruction lever.

Confidence Score: 5/5

Safe to merge. All four access-tier vulnerabilities are correctly closed, the previously flagged race and default-mismatch issues are resolved, and the new infra retry path is disabled by default with clear documentation of its adversarial risks.

The four security fixes are each correctly implemented and independently tested. The free baseline now uses free not admin, split_accesses is wired through so the engine gate is armed, the k-anonymity floor blocks sub-k subsets before any expensive work, and harbor_requirement overlays the trusted CLI. The one finding — premature discard_history updates that can overcount discarded rounds in the infra retry audit marker — affects only audit metadata, is in a path that is off by default, and does not affect any score or access gate.

No files require special attention. runner.py has the minor audit-trail finding in _retry_transient_infra, but that codepath is gated by infra_retry_rounds > 0 which defaults to 0.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/server.py	Core security fixes: separates free from admin authority, adds k-anonymity floor before commit transfer, versions result dirs with _eval_seq. Free baseline race addressed (flag claimed before await, refunded in BaseException handler). Looks correct.
vero/src/vero/evaluation/engine.py	Adds free parameter to evaluate; gates tier check and budget reserve correctly; evaluate_admin gains model override via task_params copy without mutating shared run_constraints. Logic is clean.
vero/src/vero/harbor/serve.py	Wires split_accesses into EvaluationEngine (arming the previously-dormant no_access gate), passes k_anonymity_floor to the sidecar, and changes corrupt-ledger fallback from configured budget to fail-closed zero. All changes are correct.
vero/src/vero/harbor/runner.py	Adds dead-attempt exception classification, infra retry loop, harbor_requirement trusted CLI overlay, and model override via task_params. The infra retry has a minor audit-trail inaccuracy when a retry round produces no trials (discard_history is updated prematurely). Off by default, so impact is limited.
vero/src/vero/harbor/protocol.py	Adds k_anonymity_floor parameter with default 5 (matching sidecar enforcement default); emits min_subset_samples only for non_viewable splits. Previously-flagged default mismatch is resolved.
vero/src/vero/harbor/config.py	Adds harbor_requirement, infra_retry_rounds, infra_retry_delay_s fields with clear guards: negative rounds rejected, zero delay with retries enabled rejected. Validation is tight.
vero/src/vero/harbor/verifier.py	Threads model override through _admin_eval_score to evaluate_admin; applies same override to baseline eval for like-for-like comparison; adds _dominant_sample_errors diagnostic helper. Return type changed from `float

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant A as Agent
    participant S as EvaluationSidecar
    participant E as EvaluationEngine
    participant B as BudgetLedger

    A->>S: "evaluate(req, admin=False)"
    Note over S: 1. k-anonymity floor check
    alt sub-k subset on non_viewable
        S-->>A: KAnonymityError (400)
    end
    S->>S: _transfer_commit(req.commit)
    Note over S: 2. Free baseline claim before await
    S->>E: "evaluate(req, admin=False, free=free_baseline)"
    Note over E: 3. no_access tier gate (now armed)
    alt no_access split
        E-->>S: InvalidSplitError
        S->>S: refund freebie (except BaseException)
        S-->>A: error
    end
    alt not admin and not free
        E->>B: reserve(dataset_id, split, n)
    end
    E-->>S: Experiment
    S->>S: "_route_results (versioned __e{seq} dir)"
    S-->>A: EvalSummary

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant A as Agent
    participant S as EvaluationSidecar
    participant E as EvaluationEngine
    participant B as BudgetLedger

    A->>S: "evaluate(req, admin=False)"
    Note over S: 1. k-anonymity floor check
    alt sub-k subset on non_viewable
        S-->>A: KAnonymityError (400)
    end
    S->>S: _transfer_commit(req.commit)
    Note over S: 2. Free baseline claim before await
    S->>E: "evaluate(req, admin=False, free=free_baseline)"
    Note over E: 3. no_access tier gate (now armed)
    alt no_access split
        E-->>S: InvalidSplitError
        S->>S: refund freebie (except BaseException)
        S-->>A: error
    end
    alt not admin and not free
        E->>B: reserve(dataset_id, split, n)
    end
    E-->>S: Experiment
    S->>S: "_route_results (versioned __e{seq} dir)"
    S-->>A: EvalSummary

Loading

_{Reviews (2): Last reviewed commit: "Merge pull request #35 from scaleapi/har..." | Re-trigger Greptile}