feat(genai): honor --ep/--device as an EP override for GenaiSession (#1025)#1046
Merged
DingmaomaoBJTU merged 9 commits intoJul 10, 2026
Merged
Conversation
…1025) Previously GenaiSession's `ep` arg was informational only: EP registration and pre-compilation were decided purely from the bundle's genai_config.json per-stage session_options. This adds an explicit override path with precedence explicit arg > bundle config. GenaiSession(ep=None) (new default, replacing `cpu` and retiring the `mixed` sentinel) respects the config unchanged. A concrete ep forces the whole decoder pipeline onto that EP by rewriting every stage's provider_options: `cpu` strips hardware providers (no WinML EP registration/compile), a hardware EP routes every stage to it (registering EPs, compiling when requested). Registration and compile gates now derive from the effective post-override config. Because og.Config cannot override per-stage session_options, an override loads og.Model from a derived _compiled/ bundle. perf --runtime winml-genai now applies precedence explicit --ep > concrete --device > respect config; device_to_genai_ep maps auto/unknown to None; reporting shows the resolved override alias or `config`. The rewrite/validation is generic (shared EP union, no hardcoded EP short-name behavior map). Adds 40+ tests (137 pass across the two affected modules).
xieofxie
reviewed
Jul 6, 2026
The derived-bundle EPContext cache was keyed only on the pipeline stage
name ({stage}_ctx.onnx) and freshness compared provider_options alone.
The new ep override can route the same stage onto different
EPContext-capable providers (QNN/OpenVINO/VitisAI) across runs; when a
non-native EP is forced both runs present empty provider_options, so a
binary compiled for EP-A could be reused for EP-B (wrong-accelerator
load or silent misroute).
Encode the EP in the artifact name ({stage}_{ep}_ctx.onnx) so each EP
keeps a distinct cache, and additionally require the recorded EP to
match in the freshness check. Add regression tests for per-EP artifact
naming and cross-EP cache non-reuse.
…session-ep-device-override # Conflicts: # src/winml/modelkit/session/genai_session.py
Replace the hardcoded _DEVICE_TO_GENAI_EP map (npu->qnn, gpu->dml) with resolve_genai_ep(), which reuses the same resolve_device/resolve_eps path the WinML ONNX runtime uses so a concrete --device picks the best EP actually available for that device on this machine (e.g. an NPU that is VitisAI/OpenVINO rather than QNN) instead of a static short-name guess. Add a 'config' sentinel (winml-genai default) meaning 'respect the bundle's genai_config.json per-stage routing': omitting --device maps to it, --device config selects it explicitly, and the winml runtime rejects it. --device auto now matches ONNX (best device -> best EP, whole pipeline). Precedence unchanged: --ep > --device > config.
…session-ep-device-override # Conflicts: # tests/unit/session/test_genai_session.py
Add two e2e tiers to test_perf_e2e.py for the genai EP-override path: - TestPerfGenaiContract (no bundle): asserts perf --help advertises the 'config' device sentinel and that --device config is rejected on the ONNX runtime with a genai-only usage error (exit 2). - TestPerfGenai (slow, network): builds a tiny Qwen3-0.6B int4 CPU genai bundle once per class via the onnxruntime-genai model builder, then benchmarks the four device/ep resolutions -- default and --device config respect the bundle (device=ep=config), --ep cpu overrides the EP only, and --device cpu overrides both device and EP. Skips gracefully when the LLM stack is absent or the model build fails.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Bind proc = None in the TimeoutExpired handler so proc is definitely initialized on every path, then skip on None (timeout) vs non-zero returncode (build failure). Resolves the code-scanning 'potentially uninitialized local variable' alert; happy path is unchanged.
xieofxie
reviewed
Jul 7, 2026
xieofxie
reviewed
Jul 7, 2026
xieofxie
reviewed
Jul 7, 2026
Address PR review comments on the GenaiSession --ep/--device override:
- Device-aware provider_options: when re-routing a stage to a
device-parameterized EP (OpenVINO/VitisAI) that the bundle defines no
options for, synthesize {device_type: <DEVICE>} from --device (falling
back to the EP's primary supported device). QNN reuses the bundle's own
backend_path/soc_model when present, else warns and writes empty options
instead of hardcoding a backend.
- Add a device param to GenaiSession so the forced EP can target a concrete
device; perf threads --device through via _session_device.
- Report 'config' (not the requested EP) when an override took no effect
(flat/all-CPU pipeline). effective_ep is computed from the post-override
config, and load() warns when a requested override is a no-op.
- Only re-route pipeline stages already on a hardware EP; leave CPU-intended
stages (embeddings/lm_head) on CPU. Forcing large CPU graphs onto QNN HTP
triggered a multi-minute on-the-fly compile that looked like a hang on
--device npu; skipping them keeps the forced-EP path as fast as config.
Verified end-to-end on a QNN bundle: --device npu reports ep=qnn/device=npu
and completes without the prior hang.
xieofxie
approved these changes
Jul 7, 2026
DingmaomaoBJTU
deleted the
dingmaomaobjtu-genai-session-ep-device-override
branch
July 10, 2026 03:43
DingmaomaoBJTU
added a commit
that referenced
this pull request
Jul 20, 2026
On OpenVINO hosts, resolve_genai_ep('cpu') incorrectly resolved to
'openvino' because OpenVINO advertises (npu, gpu, cpu) support and
appears before CPUExecutionProvider in the priority list.
Generic fix: prefer EPs whose *primary* device (first entry in
EP_SUPPORTED_DEVICES) matches the resolved device. Multi-device EPs
like OpenVINO target npu primarily — when the user says --device cpu
or --device gpu they expect the native EP, not a cross-device
accelerator that also happens to support it.
Examples:
--device cpu → CPUExecutionProvider (not OpenVINO)
--device gpu → DmlExecutionProvider (not OpenVINO/QNN)
--device npu → QNN/OpenVINO/VitisAI (unchanged, all primary=npu)
Falls back to eps[0] when no native EP is available.
Fixes the born-broken test_device_cpu_overrides_device_and_ep on OV
hosts (PR #1046).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DingmaomaoBJTU
added a commit
that referenced
this pull request
Jul 20, 2026
…1138) ## Summary On OpenVINO hosts, `resolve_genai_ep("cpu")` incorrectly resolved to `"openvino"` instead of `"cpu"`, causing `test_device_cpu_overrides_device_and_ep` to fail (born-broken since #1046). ## Root Cause `resolve_eps("cpu")` returns EPs in `_DEVICE_EP_MAP` priority order derived from `EP_SUPPORTED_DEVICES` declaration order. Since `OpenVINOExecutionProvider` declares `("npu", "gpu", "cpu")` and appears before `CPUExecutionProvider`, the old code blindly took `eps[0]` → `"openvino"`. ## Fix (generic) Prefer EPs whose **primary device** (first entry in `EP_SUPPORTED_DEVICES`) matches the resolved device. Multi-device EPs like OpenVINO target `npu` primarily — when the user says `--device cpu` or `--device gpu` they expect the native EP, not a cross-device accelerator. | `--device` | Before (OV host) | After | |---|---|---| | `cpu` | `openvino` ❌ | `cpu` ✅ | | `gpu` | `openvino` (if OV first) | `dml`/`cuda` ✅ | | `npu` | `qnn`/`openvino` | unchanged (all primary=npu) | Falls back to `eps[0]` when no device-native EP is available. ## Verification - Unit tests updated (11 pass, including new GPU cross-device case) - E2E `test_device_cpu_overrides_device_and_ep` passes on OV host - Ruff clean Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1025.
Previously
GenaiSession'separgument (andwinml perf --runtime winml-genai's--ep/--device) was informational only: EP registration and pre-compilation were decided purely from the bundle'sgenai_config.jsonper-stagesession_options. This PR adds an explicit override path so callers can force EP routing that differs from the bundle config, with precedence explicit arg > bundle config.What changed
GenaiSessionepdefault is nowNone(was"cpu"), and the"mixed"sentinel is retired.ep=None— respect the bundle config unchanged (registration + compile derive fromgenai_config.jsonexactly as before).ep=<concrete EP>— force the whole decoder pipeline onto that EP by rewriting every stage'sprovider_options:cpustrips hardware providers (CPU fallback; no WinML EP registration/compile).compile=True). A stage's existing options are carried over only when it already targeted the same canonical EP._bundle_uses_hardware_ep— no hardcoded EP short-name → behavior map.og.Config.clear_providers/append_providercan only touch the top-level provider (not per-stagesession_options), an override loadsog.Modelfrom a derived_compiled/bundle._prepare_compiled_bundlenow writes that derived bundle for override-only passes too (not just compilation).epvalues raiseValueError(validated against the shared EP union), so an override can never silently become a no-op.perf --runtime winml-genai--ep> concrete--device> respect config. Removedepfrom_GENAI_IGNORED_FLAGS(it is now honored).device_to_genai_epreturnsEPNameOrAlias | None—auto(and any unrecognized device) maps toNone(respect config), replacing the oldauto → "mixed".benchmark_info.ep/ device line) shows the resolved override alias, or"config"when respecting the bundle.Design notes / compatibility
normalize_ep_name/EP_NAME_TO_ALIAS/EP_NAMES); the only literal provider name is the universalcpufallback.cpu. Forcing a hardware EP onto stages the bundle did not build for it (e.g.embeddings/lm_head) may fall back to JIT/CPU inside onnxruntime-genai, which is documented in theepdocstring.