Skip to content

fix(download-libs): rebuild against refreshed artifacts instead of reusing stale objects - #1407

Merged
msluszniak merged 4 commits into
rne-rewritefrom
@ms/download-libs-extract-mtime
Sep 2, 2026
Merged

fix(download-libs): rebuild against refreshed artifacts instead of reusing stale objects#1407
msluszniak merged 4 commits into
rne-rewritefrom
@ms/download-libs-extract-mtime

Conversation

@msluszniak

Copy link
Copy Markdown
Member

Description

Two fixes to download-libs.js that let a native artifacts upgrade actually reach a machine that has built before.

Upgrading to a newer artifacts release could leave a build with new libraries but stale object files, surfacing as an undefined symbol far from the cause:

ld.lld: error: undefined symbol: executorch::extension::make_tensor_ptr(
    std::vector<int>, void*, std::vector<unsigned char>, std::vector<int>,
    ScalarType, TensorShapeDynamism, std::function<void (void*)>)
>>> referenced by tensor_ptr.h:91
>>> neural_phonemizer.cpp.o in archive phonemis/libphonemis.a

make_tensor_ptr gained a trailing Device parameter in ExecuTorch 1.4.1, so an object compiled before the upgrade kept referencing the old 7 argument symbol that the new libexecutorch.so no longer exports. Phonemis was incidental, it just happens to call the changed API.

  1. extract() passed tar -xzf, which keeps the mtime recorded in the archive. Extracted headers therefore carried the timestamp of the release cut, so newer headers could look older than existing objects. Ninja compared those mtimes, treated the objects as up to date, and never recompiled them. -m stamps files with the extraction time instead.

  2. isCacheValid() compared the cached tarball against the cached checksum, so a stale cache validated itself. The cache directory is keyed only on the libs version, so a release re-cut at the same version was never picked up: every artifact reported a cache hit while a fresh clone got the new files. It now fetches the checksum from the release, falling back to the cached copy when unreachable so a populated cache still builds offline.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

With a cache already populated from an earlier release:

  1. node scripts/download-libs.js and confirm the artifacts whose checksum changed report a cache miss and re-download, rather than Cache hit, skipping download.
  2. ls -la third-party/include/executorch/extension/tensor/tensor_ptr.h and confirm the mtime is the extraction time, not the release date.
  3. Build the Android app without wiping .cxx and confirm objects depending on changed headers are recompiled and the link succeeds.

Verified against the live v0.10.0-libs release: a warm cache holding the previous tarball validated as a hit under the old check and correctly misses under the new one. The mtime behaviour was reproduced in a minimal CMake and Ninja project, where a header with changed content but an older mtime produced ninja: no work to do.

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

The cache directory is still keyed only on nativeLibsVersion, so RNET_BASE_URL overrides pointing at a different release at the same version share a cache directory. Fetching the remote checksum makes that safe now, since the contents are compared rather than assumed, but keying the cache on the resolved base URL would be a further improvement.

`tar -xzf` keeps the mtime recorded in the archive, so extracted headers carry
the timestamp of the release cut rather than of the extraction. Upgrading to a
newer artifacts release can therefore install headers that look OLDER than
object files from an earlier build. Ninja compares those mtimes, decides the
objects are up to date, and never recompiles them -- then links them against
the new libraries.

The result is an undefined symbol for whichever API changed between the two
releases, reported against whatever happens to call it. Hitting it with the
1.4.1 artifacts, `make_tensor_ptr` gained a trailing `Device` parameter, so a
stale object kept referencing the old 7-argument symbol that the new
libexecutorch.so no longer exports:

  ld.lld: error: undefined symbol: executorch::extension::make_tensor_ptr(
      std::vector<int>, void*, std::vector<unsigned char>, std::vector<int>,
      ScalarType, TensorShapeDynamism, std::function<void (void*)>)
  >>> referenced by tensor_ptr.h:91
  >>> neural_phonemizer.cpp.o in archive phonemis/libphonemis.a

`-m` stamps extracted files with the extraction time, so refreshed headers
always look newer than existing objects and dependents rebuild.
`isCacheValid` compared the cached tarball against the CACHED checksum, so a
stale cache validated itself. Since the cache directory is keyed only on the
libs version, a release re-cut at the same version was never picked up: every
artifact reported a cache hit and the old files were reused indefinitely, while
a fresh clone got the new ones. Two machines on the same pin could hold
different artifacts with no way to tell them apart.

Fetch the checksum from the release before trusting the cache, falling back to
the cached copy when it cannot be reached so an already-populated cache still
builds offline.
@msluszniak msluszniak self-assigned this Sep 2, 2026
@msluszniak msluszniak added the bug fix PRs that are fixing bugs label Sep 2, 2026
@msluszniak
msluszniak requested a review from barhanc September 2, 2026 15:38
The `v0.10.0-libs` artifacts now carry ExecuTorch 1.4.1 headers, and the pin
above `EXECUTORCH_VERSION` requires it to track the release that
`third-party/include` is vendored from. Left at v1.3.1 the host tests compiled
1.4.1 headers against a 1.3.1 build, and `make_tensor_ptr` gained a trailing
`Device` parameter in between:

  undefined reference to `executorch::extension::make_tensor_ptr(
      std::vector<int>, void*, std::vector<unsigned char>, std::vector<int>,
      ScalarType, TensorShapeDynamism, std::function<void (void*)>,
      etensor::Device)'

The tokenizers pin moves with it. headers.tar.gz ships the fork's tokenizers
headers, and the released copies match a03231a2 rather than 56a30afb, verified
by hashing hf_tokenizer.h against both commits. That one matters more than a
link error: as the comment notes, a class layout mismatch here crashes inside
setup_pretokenizer at runtime instead of failing to link.

Editing this file also changes the dependency cache key, so the deps rebuild
rather than restoring the 1.3.1 tree.
third-party/include comes from the artifacts release while the host tests build
ExecuTorch from source at a pin in scripts/build-native-test-deps.sh. Nothing
checked that the two matched, so re-cutting the artifacts without moving the pin
compiled 1.4.1 headers against a 1.3.1 build. That surfaced nine minutes into
the dependency build as an undefined reference to a mangled symbol, attributed
to whichever code happened to call the changed API rather than to the drift.

Compare the two before the build and name both values. Scope is release to
release: ET_VERSION is generated from the source tree's version.txt, so it
cannot distinguish the fork's patches from upstream and says nothing about
TOKENIZERS_COMMIT.
@msluszniak
msluszniak merged commit 8265a69 into rne-rewrite Sep 2, 2026
4 checks passed
@msluszniak
msluszniak deleted the @ms/download-libs-extract-mtime branch September 2, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix PRs that are fixing bugs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants