Skip to content

Make SimpleAudit JSON export engine-authoritative (delegate summary to AuditResults) #63

Closed

Description

@SushantGautam

Context

build_simpleaudit_export() in audits/services.py (used by the REST export endpoint in audits/views.py) hand-mirrors the engine's results document: Studio computes the safety score with its own _SEVERITY_SCORES table and assembles the summary block manually. This is correct today but drifts silently if the engine changes its scoring formula, key names, or adds fields — there is no test that proves the exported document still loads with the engine.

PR #56 (closed as superseded) had the more robust approach: delegate to the engine.

What to do

  1. Add engine.build_audit_results_export(payloads, *, timestamp=None) in infra/engine.py (the only file allowed to import the engine). It reconstructs AuditResults([AuditResult(**filtered_fields)...]) from stored per-scenario payloads (dropping Studio-only keys like trace_ids, _language) and returns collection.to_dict().
  2. Replace the hand-rolled assembly + _export_score/_SEVERITY_SCORES in build_simpleaudit_export() with a call to that helper, so the summary is computed by the engine itself. Keep Studio's provenance block (studio_provenance).
  3. Add a round-trip test: export a run, write the JSON, load it back with AuditResults.load(path), and assert equality against AuditResults(...).to_dict(). This pins the "loads with the engine" guarantee.
  4. Decide handling for n_repetitions > 1 runs (PR feat(runs): export a completed run as the engine's results JSON #56 returned 422; main currently exports whatever durable rows exist).

References

Activity

  1. added a commit that references this issue on Oct 28, 2026
    c38f5e1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions