Skip to content

HTTPAppTarget: send each turn once and no transcript markers (fixes #124) - #125

Merged
kelkalot merged 3 commits into
devfrom
fix/http-target-history
Oct 24, 2026
Merged

kelkalot merged 3 commits into
devfrom
fix/http-target-history

Conversation

@kelkalot

@kelkalot kelkalot commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #124.

run_scenario passes the whole conversation as history, current turn included. On the conversation's message dicts, the engine keeps markers beside role and content:

  • documents with their marks;
  • file_uri;
  • decision.

HTTPAppTarget._build_body forwarded this list unchanged. As a result:

  • Duplicate turn. The current user turn reached the app twice. In messages mode it was appended again; in other modes it went out as message and again at the end of history.
  • Marker leak. The markers reached the app, including the evaluation-only document marks.

examples/audit_openwebui_rag.py uses messages mode, so it was affected.

Changes

  • History. History goes out without the engine's markers and without a trailing entry that is the current user turn. Other fields (tool_calls, tool_call_id, name, …) stay, so direct callers that pass earlier turns get the body they built. In message mode, history is a new list of the earlier turns.
  • Documents.
    • A multi-turn scenario's documents are taken from the history marker and sent through documents_field as text, as in context-grounding runs.
    • They're parsed with parse_documents, like the model and decision targets: a dict without a string text fails, and empty documents keep their position, matching the judge's numbering.
    • An explicit documents=[] is respected.
  • The history contract. Target.send now documents it, and history_documents() in targets/base.py replaces the copies in http.py and decision.py.

Behaviour changes:

  • Documents without a destination. A multi-turn scenario with documents and no documents_field now ends in an ERROR that names documents_field, the same rule context-grounding runs follow.
  • Attached files. A scenario with file_uri now ends in an ERROR; an HTTP app can't receive the file, and it used to be dropped silently.

Tests

Eight new tests in tests/test_targets.py. Through real audits against a recording HTTP client, they cover:

  • each turn sent once in both body modes;
  • documents sent as text with no marks;
  • the missing documents_field and file_uri errors.

Direct _build_body calls cover:

  • marker removal;
  • OpenAI tool messages kept as sent;
  • document parsing and positions.

Live check against a real OpenAI-compatible chat endpoint (Anthropic's, with claude-haiku-5-5), recording every request body:

  • In a three-turn audit, each request carried every turn once and no SimpleAudit keys. On dev, the same run sent the current user turn twice in every request.
  • Scenarios with documents and no documents_field, or with a file, stopped before any request was sent.

Not covered here

CallableTarget also passes history to the wrapped function unchanged, markers included.

kelkalot and others added 3 commits October 24, 2026 18:27
run_scenario passes the whole conversation as history, current turn
included, with SimpleAudit's markers beside the text. _build_body
forwarded it as-is, so the app received the current user turn twice
(appended again in messages mode, repeated in history otherwise) and
the markers: documents with their evaluation-only marks, file_uri paths
and decision answers.

History now goes out as plain {role, content} turns, without a trailing
copy of the current turn; direct callers that pass earlier turns only
are unchanged. A multi-turn scenario's documents are taken from the
history marker and sent through documents_field as text, like a
context-grounding run's, and fail the same way when no documents_field
is set.

Fixes #124

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to the history fix, from review:

- Only the engine's markers (documents, file_uri, decision, tool_events)
  are removed from history; other fields stay, so direct callers sending
  OpenAI tool messages (tool_calls, tool_call_id, name) or turns without
  content get the body they built.
- Documents go through parse_documents like the model and decision
  targets: a dict without a string text fails, empty documents keep their
  position, and an explicit documents=[] is not replaced by the history
  marker. The error no longer assumes a context-grounding run.
- A scenario with file_uri now fails with an error; the app could never
  receive the file, so it was dropped silently.
- Target.send documents the history contract, and history_documents()
  in targets/base.py replaces the copies in http.py and decision.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant