Repository navigation
Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend - #24377
pytorchbot wants to merge 8 commits into
Conversation
…and in exir (#24222) Fixes found while exporting models to ExecuTorch backends. ## Fixes - **exir:** `constant_prop_pass` keeps tensors that in-place ops write, also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps the output node last. `to_edge` lowers `torch._check_tensor_all` and `torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data. - **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches scatter and index_put fallbacks. `detach` lowers, also inside `torch.cond`. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) and `torch._check_tensor_all` lower. - **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects scoped configs. ELU is delegated when XNNPACK can run it. - **Qualcomm:** autocast regions are inlined correctly. Unquantized bfloat16 nodes, data-dependent shapes and HTP partitions without runtime inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64 constants. - **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it. Partitions are converted with their input types. - **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout chains, stay on the CPU. - **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX tensor lists, one constant slot per weight, slice updates on a constant base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer; `cummin.out`. ## Behavior changes - A CUDA `.pte` loads only the weights it was exported with. A `.ptd` with other weights, for example after fine-tuning, needs a newly exported `.pte`. - CUDA graphs turn off when outputs do not match the ETensor layout. If only the capture call sees such a layout, that call returns an error. - `GroupBasedPartitioner` is about 2x slower on large graphs, and some graphs get a different number of partitions. `to_backend` keeps the original order of top-level nodes. - XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears (8-bit dynamic activations, 4-bit or 8-bit weights) lowered with `to_backend`, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized. - Ethos-U raises an error for a delegate with unused inputs instead of returning wrong values. We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first. ## Testing Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA. (cherry picked from commit 054de1f)
This branch was picked before #24263 and #24295 reached release/1.6, so three files carried their content only in part. Take release/1.6 and use the versions a clean pick of 054de1f gives for backends/qualcomm/partition/qnn_partitioner.py, backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.
release/1.6 regenerated aoti_cuda_shims.lib to add aoti_torch_dtype_element_size, and this branch regenerated it to add aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both names, so it lists exactly the shims this tree exports.
…ault to float32 on CPU, in the OpenVINO backend (#24294) ## What is wrong today The OpenVINO backend has these problems: 1. A 0-d (scalar) input or output fails on every run, because OpenVINO's `ov_shape_create` refuses rank 0. 2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it. 3. An output whose size depends on an int input keeps its largest size. `x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8). 4. `cumsum` and `prod` of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions. ## What this change does - **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO accepts. - **Precision:** when the device is `CPU`, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision: ```python OpenvinoPartitioner( [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")] ) ``` The value is `f32`, `f16` or `bf16`, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too. - **Dynamic output sizes:** an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly. - **Integer reductions:** the input of `cumsum` and `prod` is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over `ones_like` or `zeros_like` of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning. Trade-offs: - float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with `inference_precision` set to `bf16`. - OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to `cumsum` and `prod` over smaller integer types. ## Tests Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with `bf16` checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from `ones_like`; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer `sum`, which OpenVINO already computes in int64, and a static output after a dynamic view. The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod. (cherry picked from commit 7b14148)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24377
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 Cancelled JobAs of commit 333bc8f with merge base 4db778e ( CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…ault to float32 on CPU, in the OpenVINO backend (#24294) ## What is wrong today The OpenVINO backend has these problems: 1. A 0-d (scalar) input or output fails on every run, because OpenVINO's `ov_shape_create` refuses rank 0. 2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it. 3. An output whose size depends on an int input keeps its largest size. `x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8). 4. `cumsum` and `prod` of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions. ## What this change does - **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO accepts. - **Precision:** when the device is `CPU`, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision: ```python OpenvinoPartitioner( [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")] ) ``` The value is `f32`, `f16` or `bf16`, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too. - **Dynamic output sizes:** an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly. - **Integer reductions:** the input of `cumsum` and `prod` is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over `ones_like` or `zeros_like` of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning. Trade-offs: - float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with `inference_precision` set to `bf16`. - OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to `cumsum` and `prod` over smaller integer types. ## Tests Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with `bf16` checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from `ones_like`; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer `sum`, which OpenVINO already computes in int64, and a static output after a dynamic view. The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod. (cherry picked from commit 7b14148)
The pick was made on release/1.6, which does not have #24298 (the #24222 pick) yet. #24294 builds on #24222, so the pick depends on code that release/1.6 lacks, such as _nodes_written_in_place in exir's constant_prop_pass, and its conflict resolution brought in some of #24222's OpenVINO changes but not others. This merges a clean pick of 7b14148 onto #24298's branch and takes the clean pick's side in the OpenVINO files, so that, on top of #24298, the change is exactly #24294 (7b14148). A plain merge of the two would keep two copies of _PRECISIONS in preprocess.py.
The OpenVINO CI job runs backends/openvino/tests/test_runner.py. When a test failed, the runner printed "completed with failures" but still exited with code 0, so the CI job stayed green and the failure went unnoticed. Now the runner exits with code 1 when any test fails, so CI turns red. This is the test runner fix from #24305. Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 36m 2s —— View job PR Review: #24377This cherry-pick can't land as is. It also brings in the unmerged #24222 pick, and its first commit, Detailed reviewSummaryThis PR cherry-picks two changes into release/1.6:
It also carries the unmerged #24222 pick, and that blocks it. The precision default needs a release decision. The rest is minor. Not reviewed:
Fix the code points below on main first, then pick them, so release/1.6 doesn't diverge. Code Quality
Testing
Backward Compatibility
Performance
RecommendationRequest Changes The blocker is merge order: as is, this PR puts the #24222 pick, and a commit that breaks lowering, on release/1.6. The release manager needs to decide on the f32 default for older programs and on the missing
|
What is wrong today
The OpenVINO backend has these problems:
ov_shape_createrefuses rank 0.x[:n].view(n, -1)with x of shape (4, 8) and n = 3 returns (4, 8).cumsumandprodof a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions.What this change does
0-d tensors: a rank-0 tensor keeps its empty shape, which OpenVINO accepts.
Precision: when the device is
CPU, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision:The value is
f32,f16orbf16, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too.Dynamic output sizes: an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly.
Integer reductions: the input of
cumsumandprodis cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum overones_likeorzeros_likeof an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning.Trade-offs:
inference_precisionset tobf16.cumsumandprodover smaller integer types.Tests
Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with
bf16checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and fromones_like; and folding when one cumsum cannot be evaluated. Two guards pass either way: integersum, which OpenVINO already computes in int64, and a static output after a dynamic view.The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod.
Follow-up fix from #24305
This pick also carries the test runner fix from #24305. When a test failed, the OpenVINO test runner printed "completed with failures" but still exited with code 0, so the CI job stayed green. It now exits with code 1 when a test fails, so the job fails.