fix(download-libs): rebuild against refreshed artifacts instead of reusing stale objects - #1407
Merged
Merged
Conversation
`tar -xzf` keeps the mtime recorded in the archive, so extracted headers carry
the timestamp of the release cut rather than of the extraction. Upgrading to a
newer artifacts release can therefore install headers that look OLDER than
object files from an earlier build. Ninja compares those mtimes, decides the
objects are up to date, and never recompiles them -- then links them against
the new libraries.
The result is an undefined symbol for whichever API changed between the two
releases, reported against whatever happens to call it. Hitting it with the
1.4.1 artifacts, `make_tensor_ptr` gained a trailing `Device` parameter, so a
stale object kept referencing the old 7-argument symbol that the new
libexecutorch.so no longer exports:
ld.lld: error: undefined symbol: executorch::extension::make_tensor_ptr(
std::vector<int>, void*, std::vector<unsigned char>, std::vector<int>,
ScalarType, TensorShapeDynamism, std::function<void (void*)>)
>>> referenced by tensor_ptr.h:91
>>> neural_phonemizer.cpp.o in archive phonemis/libphonemis.a
`-m` stamps extracted files with the extraction time, so refreshed headers
always look newer than existing objects and dependents rebuild.
`isCacheValid` compared the cached tarball against the CACHED checksum, so a stale cache validated itself. Since the cache directory is keyed only on the libs version, a release re-cut at the same version was never picked up: every artifact reported a cache hit and the old files were reused indefinitely, while a fresh clone got the new ones. Two machines on the same pin could hold different artifacts with no way to tell them apart. Fetch the checksum from the release before trusting the cache, falling back to the cached copy when it cannot be reached so an already-populated cache still builds offline.
barhanc
approved these changes
Sep 2, 2026
The `v0.10.0-libs` artifacts now carry ExecuTorch 1.4.1 headers, and the pin
above `EXECUTORCH_VERSION` requires it to track the release that
`third-party/include` is vendored from. Left at v1.3.1 the host tests compiled
1.4.1 headers against a 1.3.1 build, and `make_tensor_ptr` gained a trailing
`Device` parameter in between:
undefined reference to `executorch::extension::make_tensor_ptr(
std::vector<int>, void*, std::vector<unsigned char>, std::vector<int>,
ScalarType, TensorShapeDynamism, std::function<void (void*)>,
etensor::Device)'
The tokenizers pin moves with it. headers.tar.gz ships the fork's tokenizers
headers, and the released copies match a03231a2 rather than 56a30afb, verified
by hashing hf_tokenizer.h against both commits. That one matters more than a
link error: as the comment notes, a class layout mismatch here crashes inside
setup_pretokenizer at runtime instead of failing to link.
Editing this file also changes the dependency cache key, so the deps rebuild
rather than restoring the 1.3.1 tree.
third-party/include comes from the artifacts release while the host tests build ExecuTorch from source at a pin in scripts/build-native-test-deps.sh. Nothing checked that the two matched, so re-cutting the artifacts without moving the pin compiled 1.4.1 headers against a 1.3.1 build. That surfaced nine minutes into the dependency build as an undefined reference to a mangled symbol, attributed to whichever code happened to call the changed API rather than to the drift. Compare the two before the build and name both values. Scope is release to release: ET_VERSION is generated from the source tree's version.txt, so it cannot distinguish the fork's patches from upstream and says nothing about TOKENIZERS_COMMIT.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two fixes to
download-libs.jsthat let a native artifacts upgrade actually reach a machine that has built before.Upgrading to a newer artifacts release could leave a build with new libraries but stale object files, surfacing as an undefined symbol far from the cause:
make_tensor_ptrgained a trailingDeviceparameter in ExecuTorch 1.4.1, so an object compiled before the upgrade kept referencing the old 7 argument symbol that the newlibexecutorch.sono longer exports. Phonemis was incidental, it just happens to call the changed API.extract()passedtar -xzf, which keeps the mtime recorded in the archive. Extracted headers therefore carried the timestamp of the release cut, so newer headers could look older than existing objects. Ninja compared those mtimes, treated the objects as up to date, and never recompiled them.-mstamps files with the extraction time instead.isCacheValid()compared the cached tarball against the cached checksum, so a stale cache validated itself. The cache directory is keyed only on the libs version, so a release re-cut at the same version was never picked up: every artifact reported a cache hit while a fresh clone got the new files. It now fetches the checksum from the release, falling back to the cached copy when unreachable so a populated cache still builds offline.Introduces a breaking change?
Type of change
Tested on
Testing instructions
With a cache already populated from an earlier release:
node scripts/download-libs.jsand confirm the artifacts whose checksum changed report a cache miss and re-download, rather thanCache hit, skipping download.ls -la third-party/include/executorch/extension/tensor/tensor_ptr.hand confirm the mtime is the extraction time, not the release date..cxxand confirm objects depending on changed headers are recompiled and the link succeeds.Verified against the live
v0.10.0-libsrelease: a warm cache holding the previous tarball validated as a hit under the old check and correctly misses under the new one. The mtime behaviour was reproduced in a minimal CMake and Ninja project, where a header with changed content but an older mtime producedninja: no work to do.Related issues
Checklist
Additional notes
The cache directory is still keyed only on
nativeLibsVersion, soRNET_BASE_URLoverrides pointing at a different release at the same version share a cache directory. Fetching the remote checksum makes that safe now, since the contents are compared rather than assumed, but keying the cache on the resolved base URL would be a further improvement.