Skip to content

Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend - #24377

Open
pytorchbot wants to merge 8 commits into
release/1.6from
cherry-pick-24294-by-pytorch_bot_bot_
Open

pytorchbot wants to merge 8 commits into
release/1.6from
cherry-pick-24294-by-pytorch_bot_bot_

Conversation

@pytorchbot

@pytorchbot pytorchbot commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

What is wrong today

The OpenVINO backend has these problems:

  1. A 0-d (scalar) input or output fails on every run, because OpenVINO's ov_shape_create refuses rank 0.
  2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it.
  3. An output whose size depends on an int input keeps its largest size. x[:n].view(n, -1) with x of shape (4, 8) and n = 3 returns (4, 8).
  4. cumsum and prod of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions.

What this change does

  • 0-d tensors: a rank-0 tensor keeps its empty shape, which OpenVINO accepts.

  • Precision: when the device is CPU, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision:

    OpenvinoPartitioner(
        [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")]
    )

    The value is f32, f16 or bf16, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too.

  • Dynamic output sizes: an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly.

  • Integer reductions: the input of cumsum and prod is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over ones_like or zeros_like of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning.

Trade-offs:

  • float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with inference_precision set to bf16.
  • OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to cumsum and prod over smaller integer types.

Tests

Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with bf16 checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from ones_like; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer sum, which OpenVINO already computes in int64, and a static output after a dynamic view.

The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod.

Follow-up fix from #24305

This pick also carries the test runner fix from #24305. When a test failed, the OpenVINO test runner printed "completed with failures" but still exited with code 0, so the CI job stayed green. It now exits with code 1 when a test fails, so the job fails.

…and in exir (#24222)

Fixes found while exporting models to ExecuTorch backends.

## Fixes

- **exir:** `constant_prop_pass` keeps tensors that in-place ops write,
also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps
the output node last. `to_edge` lowers `torch._check_tensor_all` and
`torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that
depend on each other, unless caller node groups already do. Tensor
constants get static shapes; deduplicated constants keep their data.
- **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches
scatter and index_put fallbacks. `detach` lowers, also inside
`torch.cond`. Strided outputs are copied. Transposed convolutions (1-D,
2-D, no dilation) and `torch._check_tensor_all` lower.
- **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects
scoped configs. ELU is delegated when XNNPACK can run it.
- **Qualcomm:** autocast regions are inlined correctly. Unquantized
bfloat16 nodes, data-dependent shapes and HTP partitions without runtime
inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64
constants.
- **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it.
Partitions are converted with their input types.
- **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout
chains, stay on the CPU.
- **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX
tensor lists, one constant slot per weight, slice updates on a constant
base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes
a constant operand; Vulkan constants above 4 dims; non-strict programs
in the NXP quantizer; `cummin.out`.

## Behavior changes

- A CUDA `.pte` loads only the weights it was exported with. A `.ptd`
with other weights, for example after fine-tuning, needs a newly
exported `.pte`.
- CUDA graphs turn off when outputs do not match the ETensor layout. If
only the capture call sees such a layout, that call returns an error.
- `GroupBasedPartitioner` is about 2x slower on large graphs, and some
graphs get a different number of partitions. `to_backend` keeps the
original order of top-level nodes.
- XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears
(8-bit dynamic activations, 4-bit or 8-bit weights) lowered with
`to_backend`, fail at export instead of giving wrong results. Delegated
ELU flushes subnormal values to zero. Cats outside a scoped quantizer
config are no longer quantized.
- Ethos-U raises an error for a delegate with unused inputs instead of
returning wrong values.

We would like to cherry-pick this PR to release/1.6 after it merges. It
needs #23915 there first.

## Testing

Each fix has a test that fails without it. Run on an A100 (CUDA,
Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA
reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP).
Not run: Metal, ROCm, Windows CUDA.

(cherry picked from commit 054de1f)
This branch was picked before #24263 and #24295 reached release/1.6, so
three files carried their content only in part. Take release/1.6 and use
the versions a clean pick of 054de1f gives for
backends/qualcomm/partition/qnn_partitioner.py,
backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.
release/1.6 regenerated aoti_cuda_shims.lib to add
aoti_torch_dtype_element_size, and this branch regenerated it to add
aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both
names, so it lists exactly the shims this tree exports.
…ault to float32 on CPU, in the OpenVINO backend (#24294)

## What is wrong today

The OpenVINO backend has these problems:

1. A 0-d (scalar) input or output fails on every run, because OpenVINO's
`ov_shape_create` refuses rank 0.
2. A float32 program does not compute in float32 on many CPUs. OpenVINO
picks bfloat16 (two to three significant digits) on x86 CPUs with
bfloat16 hardware and float16 on ARM CPUs, and no compile spec can
change it.
3. An output whose size depends on an int input keeps its largest size.
`x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8).
4. `cumsum` and `prod` of a bool or integer tensor are not computed in
int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so
pi0 and pi0.5 get wrong position ids and wrong actions.

## What this change does

- **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO
accepts.
- **Precision:** when the device is `CPU`, a program computes in float32
by default, both when it is lowered and when it is loaded. A program can
ask for a faster precision:

  ```python
  OpenvinoPartitioner(
[CompileSpec("device", b"CPU"), CompileSpec("inference_precision",
b"bf16")]
  )
  ```

The value is `f32`, `f16` or `bf16`, checked before conversion starts.
Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the
precision when the runtime loads the program, so the runtime passes it
on too.
- **Dynamic output sizes:** an output whose size can change goes to a
buffer that OpenVINO owns, reused by every call. It starts empty and
OpenVINO grows it to the largest size a call writes, so a large upper
bound costs nothing until a call needs it. After inference the backend
resizes the ExecuTorch output to OpenVINO's shape and copies the data
in, so one delegate can also serve several calls, as programs from
ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch
buffer directly.
- **Integer reductions:** the input of `cumsum` and `prod` is cast to
int64 before conversion. An integer cumsum that reads only constants is
computed during lowering, because OpenVINO's CPU plugin gives it wrong
results from the second inference on. This includes a cumsum over
`ones_like` or `zeros_like` of an input with a static shape, which
OpenVINO folds into a constant too. Each one is folded on its own, and
one that cannot be folded is named in a warning.

Trade-offs:
- float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP
takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies
to programs exported before this change, because OpenVINO applies the
precision when a program loads. To get the old speed, export again with
`inference_precision` set to `bf16`.
- OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result
past the int32 range wraps without an error. That was already true for
int64 programs, and now also applies to `cumsum` and `prod` over smaller
integer types.

## Tests

Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256
addmm checked to 1e-4, and the same addmm lowered with `bf16` checked
not to reach 1e-4, so the runtime must apply an explicit precision (both
skipped where OpenVINO already computes in float32); an unknown
precision refused before conversion; dynamic output sizes, on one
program and on one delegate shared by two calls; bool and int32 cumsum
and prod; pi0-style masks with int64 and int32 cumsums over two calls,
built from a constant and from `ones_like`; and folding when one cumsum
cannot be evaluated. Two guards pass either way: integer `sum`, which
OpenVINO already computes in int64, and a static output after a dynamic
view.

The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16
units) with OpenVINO 2026.4.1, through both pytest and the CI test
runner. With OpenVINO 2025.1.0 it also passes, except that the prod test
is skipped because that release does not delegate prod.

(cherry picked from commit 7b14148)
@pytorch-bot

pytorch-bot Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24377

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit 333bc8f with merge base 4db778e (image):

CANCELLED JOB - The following job was cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 31, 2026
@JacobSzwejbka JacobSzwejbka reopened this Oct 31, 2026
…ault to float32 on CPU, in the OpenVINO backend (#24294)

## What is wrong today

The OpenVINO backend has these problems:

1. A 0-d (scalar) input or output fails on every run, because OpenVINO's
`ov_shape_create` refuses rank 0.
2. A float32 program does not compute in float32 on many CPUs. OpenVINO
picks bfloat16 (two to three significant digits) on x86 CPUs with
bfloat16 hardware and float16 on ARM CPUs, and no compile spec can
change it.
3. An output whose size depends on an int input keeps its largest size.
`x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8).
4. `cumsum` and `prod` of a bool or integer tensor are not computed in
int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so
pi0 and pi0.5 get wrong position ids and wrong actions.

## What this change does

- **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO
accepts.
- **Precision:** when the device is `CPU`, a program computes in float32
by default, both when it is lowered and when it is loaded. A program can
ask for a faster precision:

  ```python
  OpenvinoPartitioner(
[CompileSpec("device", b"CPU"), CompileSpec("inference_precision",
b"bf16")]
  )
  ```

The value is `f32`, `f16` or `bf16`, checked before conversion starts.
Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the
precision when the runtime loads the program, so the runtime passes it
on too.
- **Dynamic output sizes:** an output whose size can change goes to a
buffer that OpenVINO owns, reused by every call. It starts empty and
OpenVINO grows it to the largest size a call writes, so a large upper
bound costs nothing until a call needs it. After inference the backend
resizes the ExecuTorch output to OpenVINO's shape and copies the data
in, so one delegate can also serve several calls, as programs from
ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch
buffer directly.
- **Integer reductions:** the input of `cumsum` and `prod` is cast to
int64 before conversion. An integer cumsum that reads only constants is
computed during lowering, because OpenVINO's CPU plugin gives it wrong
results from the second inference on. This includes a cumsum over
`ones_like` or `zeros_like` of an input with a static shape, which
OpenVINO folds into a constant too. Each one is folded on its own, and
one that cannot be folded is named in a warning.

Trade-offs:
- float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP
takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies
to programs exported before this change, because OpenVINO applies the
precision when a program loads. To get the old speed, export again with
`inference_precision` set to `bf16`.
- OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result
past the int32 range wraps without an error. That was already true for
int64 programs, and now also applies to `cumsum` and `prod` over smaller
integer types.

## Tests

Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256
addmm checked to 1e-4, and the same addmm lowered with `bf16` checked
not to reach 1e-4, so the runtime must apply an explicit precision (both
skipped where OpenVINO already computes in float32); an unknown
precision refused before conversion; dynamic output sizes, on one
program and on one delegate shared by two calls; bool and int32 cumsum
and prod; pi0-style masks with int64 and int32 cumsums over two calls,
built from a constant and from `ones_like`; and folding when one cumsum
cannot be evaluated. Two guards pass either way: integer `sum`, which
OpenVINO already computes in int64, and a static output after a dynamic
view.

The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16
units) with OpenVINO 2026.4.1, through both pytest and the CI test
runner. With OpenVINO 2025.1.0 it also passes, except that the prod test
is skipped because that release does not delegate prod.

(cherry picked from commit 7b14148)
shoumikhin and others added 2 commits October 31, 2026 01:01
The pick was made on release/1.6, which does not have #24298 (the #24222
pick) yet. #24294 builds on #24222, so the pick depends on code that
release/1.6 lacks, such as _nodes_written_in_place in exir's
constant_prop_pass, and its conflict resolution brought in some of
#24222's OpenVINO changes but not others. This merges a clean pick of
7b14148 onto #24298's branch and takes the clean pick's side in the
OpenVINO files, so that, on top of #24298, the change is exactly #24294
(7b14148). A plain merge of the two would keep two copies of
_PRECISIONS in preprocess.py.
The OpenVINO CI job runs backends/openvino/tests/test_runner.py. When a test
failed, the runner printed "completed with failures" but still exited with
code 0, so the CI job stayed green and the failure went unnoticed.

Now the runner exits with code 1 when any test fails, so CI turns red. This
is the test runner fix from #24305.

Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@shoumikhin

Copy link
Copy Markdown
Contributor

@claude /pr-review

@claude

claude Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 36m 2s —— View job


PR Review: #24377

This cherry-pick can't land as is. It also brings in the unmerged #24222 pick, and its first commit, d1174e4, breaks OpenVINO lowering without that pick. Its new float32 default on CPU also slows programs already exported for x86 CPUs with bfloat16 units, and there is no way to opt out from export_llm or at runtime.

Detailed review

Summary

This PR cherry-picks two changes into release/1.6:

It also carries the unmerged #24222 pick, and that blocks it. The precision default needs a release decision. The rest is minor.

Not reviewed:

Fix the code points below on main first, then pick them, so release/1.6 doesn't diverge.

Code Quality

Testing

  • Integer results are checked with float tolerances. backends/openvino/tests/ops/base_openvino_op_test.py:91-100 compares every output with torch.allclose(atol=1e-3, rtol=1e-3). An integer result is therefore checked exactly only while it is below about 1000. test_prod (test_reductions.py:48-63) multiplies ten values between 1 and 3, so the product can reach 3^10 = 59049. A prod.default result that is off by up to 59 still passes. Fix this →

Backward Compatibility

  • Programs exported before this change slow down on x86 after upgrading only the runtime.

    • When a program has no inference_precision, OpenvinoBackend.cpp:351-355 sets INFERENCE_PRECISION_HINT=f32 at load. On x86, OpenVINO applies that hint at load.
    • So every existing CPU .pte on a CPU with bfloat16 units moves from bf16 to f32, which is slower and uses more memory. The CI runner is one of these CPUs: test_addmm_computes_in_float32 skips wherever the default is already f32, and it ran there.
    • README.md:83-89 documents this, but re-exporting is the only remedy; no runtime option keeps the earlier precision.
    • Choosing accuracy over speed for programs that have already shipped is the release manager's call for a cherry-pick, and it needs a release note.
    • One way to leave old programs alone: have OpenvinoPartitioner write inference_precision=f32 into the compile specs of new programs. The runtime then applies only what a program asks for.
  • export_llm and the OpenVINO examples can't opt out of the new default.

    • get_openvino_partitioner builds only CompileSpec("device", …) (extension/llm/export/partitioner_lib.py:68), and OpenvinoConfig has no precision field (extension/llm/export/config/llm_config.py:599-608).
    • The examples do the same: examples/openvino/aot_optimize_and_infer.py:289 and examples/openvino/stable_diffusion/export_lcm.py:220,267,314.
    • So models exported through these tools always compute in float32 on CPU. README.md:68-69 puts that at about 1.8x slower than float16 on ARM, and the README's remedy, inference_precision, needs code changes to use.

    Fix this →

Performance

  • Every non-static output now costs a copy and a second buffer.
    • Outputs that aren't STATIC now go to an OpenVINO-owned tensor (OpenvinoBackend.cpp:544-557). After every call it is copied into the ExecuTorch buffer (:660), and it stays at its largest size until destroy (:611-616).
    • Before, OpenVINO wrote these outputs in place. That includes outputs that always run at their upper bound, which needed no copy then.
    • For large dynamic outputs, such as full-sequence LLM logits, this adds a memcpy on every call and a second buffer of the same size.
    • The PR doesn't mention this cost. Please measure it on such a model and note it in the description or the release notes.

Recommendation

Request Changes

The blocker is merge order: as is, this PR puts the #24222 pick, and a commit that breaks lowering, on release/1.6. The release manager needs to decide on the f32 default for older programs and on the missing export_llm option before 1.6. Everything else is minor, and like any code fix here it should land on main first and then be picked.

  • Gather context (diff vs origin/release/1.6, commits, CI log)
  • Run three independent reviews
  • Consolidate findings
  • Fact-check findings
  • Post the final review

This branch was successfully deployed

2 active deployments
upload-benchmark-results — 333bc8f1 Deployed Oct 31, 2026 by shoumikhin via upload-benchmark-results #21667
cadence — 333bc8f1 Deployed Oct 31, 2026 by shoumikhin via hifi-op-test / hifi4 #35725
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants