Stop shipping vendored third-party sources in the wheel - #22584
Open
shoumikhin wants to merge 2 commits into
Open
Stop shipping vendored third-party sources in the wheel#22584shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22584
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
shoumikhin
force-pushed
the
slim-wheel-tests-thirdparty
branch
from
September 6, 2026 01:24
360f24a to
4711a4e
Compare
shoumikhin
changed the title
shoumikhin
force-pushed
the
slim-wheel-tests-thirdparty
branch
from
September 6, 2026 07:33
4711a4e to
ab12a82
Compare
shoumikhin
changed the title
The NXP backend keeps a table of short names for edge operators, for example `AddTensor = exir_ops.edge.aten.add.Tensor`. It lives in the backend's test package, but three modules that ship in the wheel read it at import time: backend/edge_helper.py backend/node_format_inference.py edge_passes/move_auxiliary_operator_into_separate_qdq_cluster_pass.py Two dozen production files import those modules, so the backend cannot load at all without a test package. The dependency points the wrong way: the tests should depend on the backend, not the backend on the tests. The table has no test logic in it, so this moves it to backend/ops_aliases.py, next to the code that reads it. That follows the shape the Arm backend already uses for shared operator constants in backends/arm/constants.py. No behavior changes. The names, values, and every importer stay the same, including the roughly fifty tests that use the table. The Buck target that published the table from the test package is removed, since the backend library already globs the directory it moves into. Test plan: imported the moved module and every updated importer from an installed wheel, and confirmed the table exposes the same names from the same values. Checked that every NXP test target can still reach the table through the backend package. Ran the formatter and linter over the changed files.
A pip install of ExecuTorch carries files copied out of the vendored
third-party checkouts: most of the MLX Python source tree, the XNNPACK codegen
scripts and their operator yaml, the Vulkan header registry, and two git
submodules checked out under ordinary names. Those exist to build the C++
targets. Once the libraries are built nothing in an installed wheel imports
them, and the nested submodule copies cannot satisfy the imports the code does
use: the Cadence helper imports facto.specdb from the top level, and the
tokenizers are already declared as pytorch-tokenizers.
The cause is one missing setting. setup.py passes a `packages` list only for
the minimal build, so the full build falls back to setuptools auto discovery
and picks up everything under src/executorch.
This adds the list for the full build and closes the two ways those files
arrive:
- `packages` excludes directories named third-party or third_party, plus the
submodule paths read from .gitmodules. A submodule under an ordinary name
cannot be recognized by its name, so it is read from the file that defines
it rather than hardcoded here.
- The data-file manifest is filtered with the same rule. A directory left out
of `packages` is not simply skipped: setuptools walks up to the nearest
listed package and records the file as that package's data, so a vendored
yaml still arrives under its parent without this.
One rule feeds both, so they cannot drift apart.
Test packages deliberately stay. The suites here import each other through the
installed name, for example `from executorch.backends.arm.test import common`,
so dropping them stops a non-editable install from collecting most tests.
Pruning them needs that cross-import removed first, and the note in
pyproject.toml now says so.
Test plan: built wheels for macOS arm64, Linux x86_64, and Linux x86_64 with
CUDA. Confirmed no vendored directory or submodule remains, the test packages
are still present, and the shipped headers, cmake files and flatbuffer schemas
are unchanged in number. Installed into a clean environment and, from outside
the checkout, exported, loaded and ran a model through the portable kernels and
the XNNPACK delegate, checking outputs against eager.
Added a unit test beside the existing wheel checks. It fails when the package
list stops excluding the vendored directories, when the submodule rule is
broken, and when the line that hands the list to setuptools is removed, each
checked by deleting that piece and watching the suite go red.
shoumikhin
force-pushed
the
slim-wheel-tests-thirdparty
branch
from
September 6, 2026 21:33
ab12a82 to
a358865
Compare
shoumikhin
requested a deployment
to
cadence
September 6, 2026 21:33 — with 
GitHub Actions
In progress
GitHub Actions
In progress
shoumikhin
requested a deployment
to
cadence
September 6, 2026 21:33 — with
GitHub Actions
In progress
GitHub Actions
In progress
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pip install of ExecuTorch carries 348 files copied out of the vendored third-party checkouts: most of the MLX Python source tree, the XNNPACK codegen scripts and their operator yaml, the Vulkan header registry. Those exist to build the C++ targets. Nothing in an installed wheel imports them.
The cause is one missing setting.
setup.pypasses apackageslist only for the minimal build. The full build has no list, so setuptools discovers packages by itself and picks up everything undersrc/executorch.First commit: move the NXP op alias table out of the test package
The NXP backend keeps a table of short names for edge operators in its test package, and three shipped modules read it at import time. Two dozen production files import those modules, so the backend cannot load without a test package.
The table has no test logic in it, so it moves next to the code that reads it, the way
backends/arm/constants.pyalready does. The calibration dataset classes move too. No behavior changes.Second commit: stop shipping vendored third-party sources
This adds the
packageslist for the full build, excluding the vendored directories, and filters the data-file manifest with the same list. The manifest is built from the source tree, not from that list, so a*.yamlpattern still matched 60 files inside them. One list feeds both, so they cannot drift apart.Test packages deliberately stay. The suites here import each other through the installed name, for example
from executorch.backends.arm.test import common, about 1800 times. Dropping them stops a non-editable install from collecting most tests. An editable install still works, because it exposes the whole source tree. Pruning the test packages needs that cross-import removed first, so it is not done here. The note inpyproject.tomlnow says why.Result: 348 files per wheel. The macOS wheel goes from 18.8 MB to 17.9 MB, and unpacked from 65.9 MB to 59.4 MB.
Test plan
Built wheels for macOS arm64, Linux x86_64, and Linux x86_64 with CUDA. Confirmed no vendored directory remains, the test packages are still there, and the shipped headers, cmake files and schemas are unchanged.
Installed into a clean environment and, from outside the checkout, exported, loaded and ran a model through the portable kernels and the XNNPACK delegate, checking outputs against eager. Added a unit test beside the existing wheel checks, and confirmed it fails when the package list stops excluding the vendored directories.