test(model-onboarding): fix false passes in the file-reading assertion

[denispetre]

Follow-up to #1009, addressing review findings 2, 3 and 7. Each was confirmed by executing the merged code rather than reading it.

The problem

The test could pass without the model reading the file. Run against the code now on main:

_matches("dog", "There is no dog in this image; it is a cat.")            # -> True
_matches("dog", "I don't have access to the image, but... probably a dog.")  # -> True

A model that never opened the attachment was graded correct — the one thing this test exists to prevent.

Root cause was the fixtures, not the matcher

Credit to @tudormatei1, whose review identified this: the fixtures were guessable.

• dog.jpg was asked "what animal is in this image?" — "dog" is the most likely answer to that question with no image at all, and the file name was visible in the prompt.
• dummy.pdf's text is the literal "Dummy PDF file", which reads as a placeholder. The model kept commenting on whether the content was real instead of reporting it: flaky in 3 of 6 runs.

Compensating for guessable answers required a refusal-phrase blocklist — which then rejected correct answers that carried commentary (finding 2), because the model says "placeholder" while reporting the text accurately.

The fix

Generated local fixtures whose answers cannot be guessed:

File	Content	Question	Expected
fixtures/document.pdf (608 B)	Verification code: PDF-CODE-74915	"What is the verification code?"	PDF-CODE-74915
fixtures/shape.png (1.9 KB)	purple square on white	"What colour is the shape?"	purple

With no prior to fall back on, the blocklist becomes unnecessary — refusals now fail simply because they don't contain the code. _matches is a plain case-insensitive whole-token match. A 17-case suite covers correct formattings (markdown, parser prefixes, capitalisation), refusals, and near misses (PDF-CODE-11111, magenta).

Also fixes finding 7 — and why it mattered

create_messages interpolated state.model_dump(), rendering the attachment as:

{'id': UUID('8da6…'), 'full_name': 'animal.jpg', 'mime_type': …}

while the Analyze Files tool documents {"ID": "8da6…", "FullName": …}. A model copying UUID('…') gets INVALID_ATTACHMENT_ID; one emitting lowercase id has the item skipped, analyzes nothing, and then passed the permissive matcher.

Finding 3 was masking finding 7 — fixing either alone would have looked worse or falsely better. Now model_dump(by_alias=True, mode="json") (mode="json" is required, or the UUID stays an object and serialization raises downstream).

Notes

• Fixtures are local so the expected answer is a property of bytes in this repo, not a third-party host that could change a file and quietly weaken the test.
• Pinned *.pdf/*.png as -text in .gitattributes — EOL translation on checkout would corrupt them and fail the test for a reason nobody would connect to line endings.
• document.pdf is written uncompressed on purpose, so grep PDF-CODE-74915 fixtures/document.pdf works without a PDF library.

Verification

Ran against alpha: image and pdf pass 3/3 (the PDF was 3/6 before).

openai:responses:
  image: ✓ purple
  pdf: ✓ PDF-CODE-74915

Not included

Remaining review findings, deliberately out of scope:

• Finding 1 — one bad api_flavor aborts the whole run, discarding flavors that already passed. Real, and the next thing I'd fix.
• Findings 4, 5, 6, 8 — a Bedrock flavor advertised without its client installed; text-only models no longer onboardable; unvalidated flavor strings that silently build the wrong client; a README-promised TOOL span assertion that doesn't exist.

On @tudormatei1's f-string/3.11 comment: the mechanism is real (requires-python = ">=3.11" while CI runs the python3.12 image), but it doesn't apply here — I parsed all six testcase files under real Python 3.11 and they're clean, so those line numbers were from an earlier push.

🤖 Generated with Claude Code

Update: findings 1 and 6 also fixed

Finding 1 — a bad flavor no longer aborts the run. PlatformSettings and each get_chat_model call sat outside any try, so one unusable flavor killed the node: uipath run exits non-zero and set -e in run.sh stops the script before validate_output.sh. No summary, no trace assertion, and every flavor that already passed was thrown away — during model onboarding, the most likely condition of all.

Reproduced with ["bogusvendor:whatever", "openai:responses"], bad one first so an abort would hide the good one. Before, the run died. Now:

bogusvendor:whatever:
  build: ✗ AgentStartupError: The model '...' is not available...
openai:responses:
  build: ✓ UiPathAzureChatOpenAI
  image: ✓ purple
  pdf: ✓ PDF-CODE-74915

success=False, exit 0, summary intact — the failure is reported as a finding instead of a crash.

Finding 6 — the built client is now logged. build: ✓ UiPathAzureChatOpenAI names the class actually constructed. Discovery can override the requested api_flavor, and an unrecognized Bedrock flavor falls through to Converse, so echoing the requested string alone would sign off on a surface never exercised.

Also: empty api_flavors now fails with ✗ no api_flavors supplied rather than reporting a vacuous success on an empty summary, and summary lines are logged as they are produced so a run that dies unexpectedly still leaves partial results in the job log.

Happy path unchanged. Full assert.py, including trace assertions, passes.

Still not included

• Finding 4 — awsbedrock:AnthropicMessages is advertised in the workflow input but [bedrock] ships langchain-aws[anthropic], not langchain-anthropic, so that module raises on import. Now surfaces as a build: ✗ cell rather than a whole-run abort, but the extra is still wrong.
• Finding 5 — text-only models can't be onboarded: FILE_REGISTRY is iterated unconditionally and run() raises without a file.
• Finding 8 — the README-promised Analyze Files TOOL span assertion doesn't exist (the span name would be Analyze_Files, sanitized).

These are scope decisions rather than bugs in the assertion, so I've left them for you to prioritize.

[Copilot]

Pull request overview

Tightens the model-onboarding file-processing probe so it can’t “pass by guessing”, by switching to locally committed, unguessable fixtures and simplifying the answer matcher accordingly. It also updates the coded FileProcessingAgent prompt serialization to better align attachment rendering with the Analyze Files tool’s documented schema.

Changes:

• Replaces remote/guessable fixtures (dog.jpg, dummy.pdf) with local, generated fixtures (fixtures/shape.png, fixtures/document.pdf) whose answers are intended to be unguessable.
• Simplifies _matches() to a case-insensitive whole-token (token-sequence) matcher, removing refusal-phrase heuristics.
• Updates the file-processing coded agent to serialize state with model_dump(by_alias=True, mode="json") and to load fixture bytes from disk (or HTTP) before uploading as a platform attachment.
• Pins *.pdf/*.png as -text in .gitattributes to prevent EOL translation from corrupting binary fixtures.

Reviewed changes

Copilot reviewed 4 out of 6 changed files in this pull request and generated 1 comment.

File	Description
testcases/model-onboarding/src/main.py	Updates file registry to use local, unguessable fixtures and replaces the matcher with a token-sequence match.
testcases/model-onboarding/src/agents/file_processing/agent.py	Adjusts prompt interpolation inputs for attachment IDs (aliases + JSON mode) and adds local-path fixture reading for upload.
testcases/model-onboarding/fixtures/README.md	Documents the new generated fixtures, rationale, and regeneration notes.
.gitattributes	Disables text normalization for PDF/PNG to avoid fixture corruption on checkout.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud