Add MGP-STR (alibaba-damo/mgp-str-base) image-to-text task support

[ssss141414]

## Summary
Adds Effort-L1-light registration so MGP-STR scene-text-recognition models resolve under the user-facing image-to-text task label. The vendor MgpstrOnnxConfig (Optimum) already exposes the 3-head outputs (char_logits, bpe_logits, wp_logits) correctly, but is registered ONLY under feature-extraction. This PR adds a task-label alias + MODEL_CLASS_MAPPING binding to MgpstrForSceneTextRecognition (the head-bearing class — MGP-STR is NOT a generic Vision2Seq).

Files changed (5)

• src/winml/modelkit/models/hf/mgp_str.py (NEW, 58 lines) — MgpstrImage2TextOnnxConfig(MgpstrOnnxConfig) subclass
• src/winml/modelkit/models/hf/__init__.py — 3-line wiring
• examples/recipes/alibaba-damo_mgp-str-base/image-to-text_config.json (NEW, 49 lines) — recipe
• examples/recipes/README.md — catalog row
• research/adding-model-support/model_knowledge/mgp_str.json — mgp_str-004 post-mortem finding

Goal-ladder verdict

alibaba-damo/mgp-str-base @ image-to-text @ fp32 @ cpu

Tier	Verdict	Evidence
L0 build	PASS	83.7s, 374 nodes, 564.5 MB optimized; autoconf converged in 2 iters
L1 perf	PASS	avg=100.76ms, P90=123.26ms, 9.92 samples/sec (20 iters CPU)
L2 numerical	PASS	cosine vs PT: char=0.99999999999992, bpe=0.99999999999974, wp=0.99999999999860; max-abs 5.7e-05 / 2.4e-04 / 2.1e-04
L3 eval	CLI-BLOCKED	image-to-text task has no default dataset (same as vit-gpt2)

Step 1b verification — real engineering vs catalog-only

• Gate 1 (auto-config-diff): identical to winml config --task image-to-text (recipe is autoconf-faithful)
• Gate 2 (baseline build on main): FAILS with mgp-str doesn't support task image-to-text for the onnx backend. → real engineering delta, NOT catalog-only.

Known gotchas

• HF model card declares legacy architectures: ['MGPSTRModel'] but current transformers exports MgpstrModel (CamelCase rename). Without --task image-to-text explicit, winml inspect/config/build fail with Cannot import MGPSTRModel from transformers. CLI robustness gap separate from this PR.
• 3 Einsum ops in a3_module heads are non-fatal on CPU.

Verification

uv run winml build -c examples/recipes/alibaba-damo_mgp-str-base/image-to-text_config.json -m alibaba-damo/mgp-str-base -o temp/mgp_build --ep cpu --device cpu --rebuild
uv run winml perf -m temp/mgp_build/model.onnx --ep cpu --device cpu --iterations 20

EP-coverage update — DirectML validated on a second host (2026-07-10)

The original submission was verified on a host without a DirectML GPU, so the DML EP was left deferred (not_yet_tested_on: @ dml-gpu (host-blocked) in the mgp_str finding). A second contributor re-ran the full Goal-ladder on a machine that exposes onnxruntime.get_available_providers() == ['DmlExecutionProvider', 'CPUExecutionProvider'], closing that deferred coverage. No code change — this is a pure EP re-verification of the existing L1-light contribution.

Per-(EP, device) matrix — alibaba-damo/mgp-str-base @ image-to-text @ fp32

Tier	EP / device	Verdict	Evidence (this host, 2026-07-10)
L0 build	—	PASS	144.9s, IR8/opset17, pixel_values[1,3,32,128] → char[1,27,38]/bpe[1,27,50257]/wp[1,27,30522], 564.5 MB external data co-located
L1 perf	CPUExecutionProvider / cpu	PASS	avg=329.70ms, P50=345.12, 3.03 samples/sec (20 iters)
L1 perf	DmlExecutionProvider / gpu	PASS	avg=106.16ms, P50=106.28, 9.42 samples/sec, VRAM +1214 MB (≈3.1× faster than CPU on this host)
L2 numerical	CPUExecutionProvider / cpu	PASS	3 heads cosine≈1.0, argmax match; max-abs char 5.7e-5 / bpe 2.4e-4 / wp 1.9e-4
L2 numerical	DmlExecutionProvider / gpu	PASS	3 heads cosine≈1.0, argmax match; same max-abs envelope as CPU
L3 eval	both	CLI-BLOCKED	unchanged: No dataset provided and no default for task image-to-text

Notes (honesty): The L1 CPU latency here (329.70ms) differs from the original submission's 100.76ms because it is different hardware — both are valid, they are not the same machine. The DML rows are net-new coverage, not a restatement of the original numbers. The 3 a3_module Einsum ops flagged as possibly EP-unsupported in the finding run correctly on DML; the warnings are parquet-coverage-rule gaps, not runtime failures.

Coverage after this update: reachable-verified = CPU + DML. Still deferred = QNN/NPU (no NPU on this host) and OpenVINO (EP present but not exercised for this model).

Reproduce DML:

uv run winml perf -m temp/pr952_build/model.onnx --ep dml --device gpu --iterations 20
uv run python temp/pr952_l2_compare.py temp/pr952_build/model.onnx dml

OpenVINO EP matrix — Intel NPU + GPU + CPU (2026-07-10, follow-up)

Correction to the earlier EP-coverage note: this host is an Intel Core Ultra 7 258V (Lunar Lake) with an Intel AI Boost NPU, an Intel Arc 140V GPU, and the CPU. onnxruntime-windowsml auto-installs the downloadable OpenVINOExecutionProvider v1.8.80.0, which targets NPU / GPU / CPU. (Stock onnxruntime.get_available_providers() only lists DML+CPU — the OpenVINO EP is registered through the Windows ML EP infrastructure and must be driven via winml perf --ep openvino.) So QNN/NPU was never the right "deferred" label here — the NPU is Intel, reached through OpenVINO, and it works.

L1 perf — alibaba-damo/mgp-str-base @ image-to-text @ fp32 (20 iters, warmup 5)

EP / device	Verdict	Avg latency	P50	Throughput	Memory
OpenVINOExecutionProvider / gpu	PASS	10.35ms	10.33	96.62 samples/sec	VRAM +676 MB
OpenVINOExecutionProvider / npu	PASS	15.02ms	14.54	66.59 samples/sec	VRAM +325 MB
OpenVINOExecutionProvider / cpu	PASS	275.62ms	275.84	3.63 samples/sec	RAM +1198 MB

OpenVINO GPU is the fastest EP for this model — 10.35ms vs DML 106ms vs plain-CPU 330ms. The 3 a3_module Einsum ops run correctly on both NPU and GPU. fp32 was used to match the pre-built artifact (the CLI's auto NPU precision is w8a16, which would need a quantized rebuild).

Full EP coverage on this host: CPU, DML(GPU), OpenVINO(NPU/GPU/CPU) — all PASS. Only genuinely N/A: QNN (this is Intel silicon, not Qualcomm).

Reproduce:

uv run winml perf -m temp/pr952_build/model.onnx --ep openvino --device gpu --iterations 20 --warmup 5
uv run winml perf -m temp/pr952_build/model.onnx --ep openvino --device npu --iterations 20 --warmup 5

[ssss141414]

Reviewer verification: OV cpu / gpu / npu — branch \shzhen/add-mgp-str-base\

Commands

\\powershell

config

uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text -o temp/verify_pr952_mgpstr_config.json

build (OV CPU, fp32, using recipe)

uv run winml build -c examples/recipes/alibaba-damo_mgp-str-base/image-to-text_config.json -m alibaba-damo/mgp-str-base -o temp/verify_pr952_mgpstr_build --ep openvino --device cpu --precision fp32 --no-quant --no-compile --rebuild

perf — cpu / gpu / npu (from built ONNX, 5 iters + 2 warmup)

uv run winml perf -m temp/verify_pr952_mgpstr_build/model.onnx --ep openvino --device cpu --iterations 5 --warmup 2 --skip-build -f json
uv run winml perf -m temp/verify_pr952_mgpstr_build/model.onnx --ep openvino --device gpu --iterations 5 --warmup 2 --skip-build -f json
uv run winml perf -m temp/verify_pr952_mgpstr_build/model.onnx --ep openvino --device npu --iterations 5 --warmup 2 --skip-build -f json

eval

uv run winml eval -m alibaba-damo/mgp-str-base --task image-to-text --device cpu --ep openvino --samples 1
\\

Results

Command	cpu	gpu	npu
config	✅ PASS	—	—
build	✅ PASS (79s, 564.5 MB, autoconf converged in 2 iters)	—	—
perf mean	✅ 305 ms/iter	✅ 9.1 ms/iter	✅ 22 ms/iter
perf throughput	3.27 samples/s	109.38 samples/s	45.48 samples/s
eval	❌ CLI-BLOCKED	❌ CLI-BLOCKED	❌ CLI-BLOCKED

Notes:

• \config\ / \�uild\ / \perf\ pass on all three OV devices. OV sessions created successfully for cpu, gpu, and npu.
• Build emits 3 \Einsum\ op warnings (\OpUnsupportedError: Node Einsum is not supported\ for \char_a3_module, \�pe_a3_module, \wp_a3_module) — consistent with the 'non-fatal on CPU' note in the PR. OV EP handles these via fallback.
• \�val\ returns \No dataset provided and no default for task 'image-to-text'. Use --dataset.\ — same CLI-BLOCKED verdict as described in the PR (same as vit-gpt2). Not an OV EP limitation.
• ONNX artifact: 374 nodes (post-optimize), opset 17, fp32, input: \pixel_values[1,3,32,128], outputs: \char_logits[1,27,38], \�pe_logits[1,27,50257], \wp_logits[1,27,30522].

[ssss141414]

Validation results (2026-06-25) for PR #952 on this Windows ARM64 host.

Scope

• Compare main vs PR branch behavior
• Verify winml config on QNN NPU/GPU

Main branch baseline (before PR)

• Command: uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text --ep cpu --device cpu
• Result: FAIL
• Error: mgp-str doesn't support task image-to-text for the onnx backend. Supported tasks are: feature-extraction.

PR #952 branch

• CPU config: PASS
  • uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text --ep cpu --device cpu
  • Resolved to Device=CPU, EP=CPUExecutionProvider
• QNN NPU config: PASS
  • uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text --ep qnn --device npu
  • Resolved to Device=NPU, EP=QNNExecutionProvider
• QNN GPU config: PASS
  • uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text --ep qnn --device gpu
  • Resolved to Device=GPU, EP=QNNExecutionProvider

Conclusion

• Confirmed: this PR adds real image-to-text task support for mgp-str (main fails, PR passes), including QNN NPU/GPU configuration resolution.

[ssss141414]

ADDENDUM: main branch baseline (NO support)

On current \main\ @ HEAD:
\\powershell
uv run winml config -m alibaba-damo/mgp-str-base --task image-to-text
\
Returns:
\
Error: mgp-str doesn't support task image-to-text for the onnx backend. Supported tasks are: feature-extraction.
\\

Conclusion: This PR adds \image-to-text\ task support (via \MgpstrImage2TextOnnxConfig\ alias + \MODEL_CLASS_MAPPING\ binding). Without this PR, mgp-str only works under \eature-extraction. The engineering delta is real (not catalog-only). All OV devices now pass config/build/perf validation.

[xieofxie]

the exported model are same as the current supported task?

[ssss141414]

reviewer verdict — APPROVE (draft; awaiting human ready-promotion)

Independent re-march of the checklist against the pushed producer fix (f486ed94):

• Gap closed — the original REQUEST_CHANGES was missing pytest coverage for the MGP-STR image-to-text OnnxConfig alias. (Note: the model_knowledge/mgp_str.json finding already existed in the branch — the earlier "missing findings" read was wrong; the real gap was the absent unit test.) The fix adds tests/unit/export/test_mgp_str_onnx_config.py (4 tests).
• Independent verification — re-ran pytest tests/unit/export/test_mgp_str_onnx_config.py from a clean env: 4 passed in 0.23s.
• Contract coverage — tests assert: MgpstrImage2TextOnnxConfig is the registered constructor; input is ["pixel_values"]; outputs are the three heads {char_logits, bpe_logits, wp_logits}; MODEL_CLASS_MAPPING binds MgpstrForSceneTextRecognition.
• Cardinal Rule 1 — support lives in models/hf/mgp_str.py via @register_onnx_overwrite (an L1-light alias over MgpstrOnnxConfig); no if model_type == ... branching. ✅
• Tier — L1-light (alias subclass); code_paths match tier. ✅

Coverage scope (honest annotation): verified at the OnnxConfig-contract / unit-test level. coverage: partial — L2/L3 numerical-delta + per-EP perf on NPU/GPU hardware were not run here (deferred_eps = non-CPU targets). No cross-EP breadth is claimed.

Verdict: APPROVE. Left as draft per contributor request — promote with gh pr ready when ready.

[ssss141414]

reviewer verdict — CORRECTION + real Goal-ladder march

As with #951, my earlier verdict here only cited a pytest unit-test run — that is NOT the Goal ladder. I have now independently re-marched it on this host (CPU / CPUExecutionProvider).

Tier	Command	Result (independently re-run)
L0	winml build -c image-to-text_config.json -m alibaba-damo/mgp-str-base + onnx.load	PASS — Build complete in 130.5s; model.onnx + .data (564.5 MB) co-located; IR 8; input pixel_values[1,3,32,128]; outputs char_logits[1,27,38] bpe_logits[1,27,50257] wp_logits[1,27,30522] (3 heads ✓)
L1	winml perf -m model.onnx --device cpu --ep cpu	PASS — Avg 135.51 ms / P50 136.10 ms / P90 152.62 / P99 160.01; throughput 7.38 samples/s; providers ['CPUExecutionProvider']; model-load +574.6 MB
L2	ad-hoc temp/mgp_l2_check.py (ONNX vs PyTorch, 3 heads)	PASS — char cos=1.000000 max_abs=5.34e-05; bpe cos=1.000000 max_abs=2.71e-04; wp cos=1.000000 max_abs=2.29e-04

Op-coverage note (tester finding, worth the learner): during L0 the build's coverage-analysis stage logged OpUnsupportedError: Einsum for the char/bpe/wp_a3_module attention blocks. This is a winml coverage-rule-database gap, not an export or runtime failure — ORT's CPU EP runs Einsum natively (L1 perf + L2 delta both pass). It would matter for an NPU/QNN target where Einsum may need decomposition or nodes_to_exclude; flagging for the coverage rules, not blocking CPU APPROVE.

Coverage: target_eps=[cpu] → fully verified. NPU absent on host; Einsum coverage gap noted for any future NPU target. coverage: full on CPU.

Plus prior unit-test contract coverage (4 passed). Verdict: APPROVE (draft; promote with gh pr ready).

[ssss141414]

Reviewer verdict (independent second-host re-verification): APPROVE

Role note: this verdict is posted as a review comment because GitHub disallows formally approving one's own PR. The re-verification is independent of the original submission in the sense that it ran on a different host (with a DirectML GPU) from a clean rebuild.

• Value fidelity: the appended EP-coverage section adds DML rows only; it does not alter or restate the original CPU numbers as if they were mine. The CPU latency difference (329.70ms vs the original 100.76ms) is explicitly attributed to different hardware.
• Load-bearing check re-run: L2 numerical parity (the check that would catch a broken export) PASSES on both CPU and DML — all three heads cosine≈1.0 with argmax match. This is the check that matters; it holds on both EPs.
• L0/L1 re-run: build converges, both EPs run to completion. Einsum a3_module ops confirmed running on DML (finding's EP-support caveat resolved).

Coverage annotation:

• reachable-verified: CPUExecutionProvider, DmlExecutionProvider
• deferred (host-limited, not a defect): QNNExecutionProvider/NPU (no NPU on this host), OpenVINOExecutionProvider (present but not exercised for this model)

Terminal state: APPROVE · coverage: partial (CPU+DML verified; QNN/NPU + OpenVINO deferred).

[ssss141414]

Reviewer verdict — OpenVINO EP-coverage completion (2026-07-10)

Following up my earlier CPU+DML verdict: I mis-labeled the non-CPU/DML EPs as "host-blocked". This host (Intel Lunar Lake) exposes a full Intel accelerator stack through the downloadable OpenVINOExecutionProvider v1.8.80.0. I re-ran the EP flow on all three OpenVINO device targets.

MGP-STR (#952) — APPROVE (strengthened). L1 PASS on OpenVINO NPU, GPU, and CPU. OpenVINO GPU is the fastest EP of all for this model (10.35ms / 96.62 samples/sec, vs DML 106ms). NPU 15.02ms. The 3 a3_module Einsum ops run correctly on NPU+GPU.

Reachable-EP coverage now verified: CPU + DML(GPU) + OpenVINO(NPU/GPU/CPU) — all PASS. Only N/A: QNN (Qualcomm — this is Intel silicon).

Remaining gap (non-blocking): quantized w8a16 OpenVINO NPU path (fp32 used here to match the artifact). No code changes requested.