Skip to content

[FEAT]: Manually scoring messages and attacks via GUI - #2589

Open
jbolor21 wants to merge 13 commits into
microsoft:mainfrom
jbolor21:jbolor/manual_scores
Open

[FEAT]: Manually scoring messages and attacks via GUI #2589
jbolor21 wants to merge 13 commits into
microsoft:mainfrom
jbolor21:jbolor/manual_scores

Conversation

@jbolor21

@jbolor21 jbolor21 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Description

Separates automated and human scoring on attack results and keeps manual scoring focused on objective true/false verdicts.

  • Adds automated_score and human_score to attack results.
  • Makes last_score a read-only effective-score property with human-score precedence.
  • Migrates existing last_score_id values to automated_score_id and adds human_score_id.
  • Updates attack executors to write objective scorer output to automated_score.
  • Adds boolean manual scoring with an optional rationale and authenticated user identifier metadata.
  • Adds an explicit, enabled-by-default Update attack score and outcome option.
    • When selected, only human_score, outcome, and outcome_reason are updated.
    • When not selected, the score is retained only in message score history.
  • Validates that the scored message belongs to the supplied attack before objective checks or updates.
  • Supports manual scoring for messages in the attack's forked conversations.
  • Extends attack PATCH support to update the shared objective and its hash.
image

Tests

  • Added migration and score-precedence coverage.
  • Added backend coverage for manual scoring, ownership validation, and forked conversations.
  • Added frontend coverage for boolean scoring and attack update behavior.
  • Ran impacted Python tests, focused frontend Jest suites, Ruff, ESLint, TypeScript type-checking, and git diff --check.

@jbolor21
jbolor21 marked this pull request as draft September 7, 2026 00:05
Comment thread pyrit/backend/models/scores.py Outdated
Comment thread pyrit/backend/routes/scores.py Outdated
Comment thread pyrit/backend/models/scores.py Outdated
Comment thread pyrit/score/true_false/manual_scorer.py
@behnam-o Behnam (behnam-o) changed the title [DRAFT]: Add manual scores [DRAFT]: Separate automated and human attack scores Sep 10, 2026
Behnam Ousat and others added 7 commits September 10, 2026 13:42
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 097eaf76-ae49-44f8-b44d-9841fa2cc1cb
Preserve legacy attack scores as automated scores while adding human-score precedence and a boolean manual-scoring workflow. Validate attack/message ownership so scoring also works safely for forked conversations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Permit PATCH requests through CORS and restrict manual scoring to user-visible attack conversations, excluding internal adversarial chats.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Point the attack-score split migration at the current upstream migration head and remove imports made obsolete by upstream test organization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@behnam-o Behnam (behnam-o) changed the title [DRAFT]: Separate automated and human attack scores [FEAT]: Add support for manually scoring messages and attacks via GUI Sep 10, 2026
@behnam-o Behnam (behnam-o) changed the title [FEAT]: Add support for manually scoring messages and attacks via GUI [FEAT]: Manually scoring messages and attacks via GUI Sep 10, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@behnam-o
Behnam (behnam-o) marked this pull request as ready for review September 10, 2026 21:05
Behnam Ousat and others added 5 commits September 10, 2026 14:19
Allow objectives to be set only when absent, while treating the same value as an idempotent update and returning a conflict for replacements.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep objective entry available after messages are sent, reuse history outcome badges, and center empty-objective controls.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants