[RNE Rewrite] feat(models)!: resolve the DEFAULT model variant per platform - #1392
Open
msluszniak wants to merge 24 commits into
Open
[RNE Rewrite] feat(models)!: resolve the DEFAULT model variant per platform#1392msluszniak wants to merge 24 commits into
msluszniak wants to merge 24 commits into
Conversation
`DEFAULT` pointed at a fixed export, almost always the universal XNNPACK one, so an iOS app that did nothing special ran on the CPU while the Neural Engine sat idle. It now resolves when the library loads, from the platform, the backends the binary was actually linked with, and the order the variants are declared in. - iOS device: the Core ML export where one exists. - Android and the iOS simulator: XNNPACK. - Narrowed by the app's `react-native-executorch` block, so opting out of a backend moves `DEFAULT` back rather than failing to load. - MLX and Vulkan stay an explicit opt-in. Also fixes `instanceSegmentation.RFDETR_NANO`, which defaulted to a Core ML file on Android, and refreshes the `features` -> backend map, which omitted coreml for semanticSegmentation, imageEmbeddings and textToImage, and mlx for speechToText and textToSpeech. A backend a feature does not provision is one `DEFAULT` can never resolve to, so a test now holds the map in sync with the registry. Closes #1389
An accelerated export only gets published once it beats the CPU one, so a published variant is itself the signal to prefer it. MLX now sits above XNNPACK on iOS and Vulkan above XNNPACK on Android, rather than both being an explicit opt-in. Core ML still leads on iOS, but only to give the pair a deterministic order. The 8 groups publishing both Core ML and MLX -- 6 Whisper sizes, RF-DETR keypoint and the CLIP vision encoder -- now pin their winner at the call site with the reason, and a test fails until a new such group does the same. On Android this moves both LFM2.5-VL sizes and PP-OCRv6 to Vulkan; on iOS it moves 4 LLMs, 2 text embedders, both privacy filters and Supertonic TTS to MLX.
msluszniak
marked this pull request as draft
August 27, 2026 15:28
RF-DETR keypoint was the registry's only fp32 Core ML model, and the Neural Engine is fp16-only, so it could never reach the ANE whatever compute unit it asked for. Measured on an iPhone 16, raw execute, 20 iterations: coreml fp16 122.7 ms 75 MB coreml fp32 159.9 ms 148 MB (published) mlx fp32 270.3 ms 145 MB A compute-unit sweep confirms the cause: restricting fp32 to CPU_AND_NE costs 2x (337.5 ms) and matches CPU_ONLY, so the ANE contributes nothing at fp32; the same restriction on fp16 costs 2 ms. ALL stays the best compute unit at both precisions, which is the ExecuTorch default the export scripts already rely on. The build is published alongside the fp32 one, same graph and same I/O ([1,3,576,576] in, [100,4]/[100]/[100,17,3] out), and iOS now pins to it.
This reverts commit 923bd38. The fp16 build does not work. Run on device against a real photo, it returns zero detections where both fp32 builds return the same person at 0.863 confidence with matching boxes and landmarks: coreml-fp32 1 detection, conf 0.863, box [280.5, 43.1, 496.8, 391.2] xnnpack-fp32 1 detection, identical to within 0.01 px coreml-fp16 0 detections The benchmark that justified the pin ran on a synthetic scene that detects nothing, so it measured speed on a model that produces no output. Reverting the pin until the fp16 export is fixed and checked against real input.
All three rationales were written before the measurements that settled them, and two of them state a mechanism that turned out to be wrong. Whisper was pinned to Core ML on memory grounds alone, because at the time it was losing the pipeline benchmark at tiny and base. It no longer is: the republished builds run `decode` on the CPU rather than the Neural Engine (export-scripts MR !18) and are 2.5-3.1x faster end to end on real speech, so the pin now rests on speed, memory and accuracy rather than memory alone. RF-DETR keypoint credited its win to the Neural Engine. It cannot be: the ANE is fp16-only and this is the registry's only fp32 Core ML model, which a compute-unit sweep confirms — `cpu_and_ne` matches `cpu_only`, and `all` matches `cpu_and_gpu`. Records the fp16 attempt too, which is faster and returns zero detections. CLIP cited an issue number instead of its numbers.
msluszniak
added
performance
… precision The earlier note said the score head does not survive fp16. That compared a local fp16 build against the published fp32 one, which confounds precision with build provenance. Built from the same trace, fp16 and fp32 agree to 0.0008 confidence on device while both score 3x below the published fp32, so the gap is the export environment.
…on iOS An fp16 Core ML build is now published for this model, and it is faster and smaller than the fp32 one it replaces as the iOS default: 144.0 ms against 165.1, a 263 MB peak against 389 MB, and half the download, matching fp32 to 0.0008 confidence and 0.23 px on an iPhone 16. fp32 stays available as COREML_FP32. The comment on the pin records why this build is shaped the way it is. fp16 is correct on device only with the CPU and the Neural Engine both excluded, and only at an iOS17 deployment target, and neither constraint is visible from a host-side check. Those live in export-scripts MR !18; the note here exists so nobody "simplifies" the pin without knowing what it is holding.
…lly are The previous commit added COREML_FP16 next to COREML_FP32 and inherited that line's VERSION_TAG (v0.9.0). The fp16 build was published to v0.10.0, and v0.9.0 holds only the three original fp32 files, so the URL 404s -- confirmed against the Hub, 404 at v0.9.0 and 206 at v0.10.0. Since the same commit made COREML_FP16 the iOS default, every iOS app resolving RFDETR_KEYPOINT.DEFAULT would have failed to download it. Moves all four keypoint URLs to NEXT_VERSION_TAG, which is the tag this branch targets and which carries xnnpack, coreml fp16, coreml fp32 and mlx. All four verified to resolve. rfdetr-nano-detector is the only entry still on VERSION_TAG; it is untouched here because nothing in this change republished it.
…ano-detector fp32 was kept as a safety net for the fp16 build, which is correct on device only under CPU_AND_GPU. Having pinned and documented that, the net was worth what it cost only if fp16 were unproven, so it was checked properly: 13 real photos, 40 detections, fp16 against fp32 on each. Visible landmarks agree to 1.04 px, boxes to 0.653 px, scores to 0.072, and no detection crossed the confidence threshold. Landmarks the model marks invisible drift up to 15.7 px, but their coordinates are undefined when `vis` is 0. That leaves no argument for shipping 148 MB nobody downloads, so COREML_FP32 is gone and fp16 is the only Core ML build. The file is removed from the Hub as well; v0.9.0 keeps its own copy, so released versions are unaffected. Also repoints rfdetr-nano-detector's xnnpack URL at NEXT_VERSION_TAG. That entry was split across tags, xnnpack on v0.9.0 and coreml on v0.10.0, for no reason; the file exists at both, so this only makes it consistent.
…2 build Missed in the previous commit: the demo's model picker referenced RFDETR_KEYPOINT.COREML_FP32, which no longer exists. Points it at COREML_FP16.
An audit of the 48 HF repos the registry references found 41 published .pte files no app could reach: the entry simply was not there. That matters most for the accelerated ones, because `DEFAULT` prefers Core ML on iOS and Vulkan on Android, so a missing key silently pins those models to XNNPACK forever. 38 of the 41 are now wired: - Core ML for all-MiniLM-L6-v2, multi-qa-MiniLM, paraphrase-multilingual, distiluse and CLIP text - Core ML for YOLO26-seg (5 scales x 3 input sizes) and YOLO26-pose - Core ML for Kokoro, all three model sets, reached by all 9 languages - MLX int4/int8 for LFM2.5-VL 1.6B, Vulkan for Gemma 4 E2B - XNNPACK fp32 for paraphrase, distiluse and PP-OCRv6, XNNPACK int8 for the three English Whisper sizes Left out: `all-MiniLM-L6-v2_xnnpack.pte`, a pre-rename duplicate of the file the registry already uses, and the two Llama 3.2 QAT+LoRA builds, which the rewrite does not support. New XNNPACK variants are declared after the incumbent so no CPU default moves. Accelerated ones take the platform default, per the rule that a published accelerated export is the signal to prefer it: 32 groups move on iOS, 1 on Android. Two calls that rule does not make on its own: Kokoro's Core ML files are fp32, the registry's only Core ML fp32 now that RF-DETR keypoint's is gone, and fp32 never reaches the Neural Engine. They still default because the duration predictor runs 14-21x faster warm on an iPhone 16; the first-use compile cost is in the comment. distiluse is the only Core ML/MLX pair that does not go Core ML's way, so it is pinned to MLX. Measured on an iPhone 16 over 150 warm embeds of five sentences in five languages: MLX int8 3.46 ms, Core ML fp16 3.96 ms, XNNPACK 8da4w 5.86 ms, holding when the arms are reversed. MLX is half the download and skips a 786 ms first-use compile, and fidelity does not break the tie (both 0.973 worst-case cosine against XNNPACK). That pin is why `defaults to Core ML on iOS` now exempts groups carrying an explicit iOS pin: Core ML leads the order, but the order was never a benchmark result. Groups with both backends still have to pin one. `download-libs.js` gains coreml for textEmbeddings and textToSpeech and vulkan for llm, without which those defaults could never resolve.
Android GPU build of the registry's most downloaded model. Under the platform rule it becomes the Android default, which is what the benchmark supports: on a Mali-G76 (Galaxy S10+) at the published 254 token shape it runs 169.8 ms against the XNNPACK build's 408.4 ms median, a 2.4x speedup, bit-identical across 10 executions and cosine 0.9999924 to the CPU reference. textEmbeddings gains vulkan in the download-libs feature map, without which DEFAULT would resolve to a backend the binary was never linked with. The sync test catches exactly this.
12 tasks
tiny.en, base, base.en, small and small.en, matching the multilingual tiny entries already added. Encode on Vulkan is 1.63x to 1.94x the XNNPACK build across the six sizes on an Adreno 840, with transcripts identical to the fp32 reference, so each takes the Android default under the platform rule. The demo gains the three Vulkan entries that are useful to compare against their CPU counterparts; the rest are reachable by name.
Adds Vulkan fp16 variants for CLIP ViT-B/32 vision and text and for
all-mpnet-base-v2, and moves all-MiniLM-L6-v2 from fp32 to fp16. Vulkan leads
BACKEND_ORDER.android, so each of these becomes the Android default.
Measured against XNNPACK, medians of 5 interleaved rounds of 50 iterations, on
an S26 Ultra (Adreno 840) and an S10+ (Mali-G76):
Adreno Mali
MiniLM 2.05x 2.85x
CLIP vision tie 2.06x
CLIP text 1.10x 2.00x
mpnet 1.35x 1.90x
fp16 rather than fp32 throughout. fp32 is the faster arm on Mali, but it loses
to XNNPACK on Adreno for CLIP text and mpnet, so fp16 is the only precision
that wins on both parts. It also halves every download: MiniLM 90 to 45 MB,
CLIP 352 to 176 and 254 to 127, mpnet 435 to 218.
CLIP vision is a tie on Adreno rather than a win. It is defaulted anyway: it
costs that device nothing, doubles throughput on the older part, and halves
the download everywhere.
imageEmbeddings now provisions vulkan in download-libs. Without it the backend
is never downloaded and the model silently falls back to XNNPACK.
Outputs match the CPU reference at cosine >= 0.99995 on both GPUs, and both
CLIP encoders are deterministic over 60 consecutive runs on each.
…embedders
Wires multi-qa-MiniLM-L6-cos-v1, multi-qa-mpnet-base-dot-v1 and
paraphrase-multilingual-MiniLM-L12-v2. Vulkan leads BACKEND_ORDER.android, so
each becomes the Android default.
Against XNNPACK, medians of 5 interleaved rounds of 50 iterations, on an
S26 Ultra (Adreno 840) and an S10+ (Mali-G76):
Adreno Mali
multi_qa_minilm 1.63x 2.45x
paraphrase_ml 1.40x 1.50x
multi_qa_mpnet 1.14x (noisy) 1.60x
multi_qa_mpnet is the weakest of the three: both arms swing widely on Adreno
(vk 123-306 ms against xnn 204-319), so that figure is indicative only. On Mali
its worst sample still beats XNNPACK's best, so the direction holds.
Each halves its download, paraphrase-multilingual most of all at 470 to 235 MB.
All three verified against the CPU reference at cosine >= 0.99997 on both GPUs.
textEmbeddings already provisions vulkan in download-libs, so no change there.
Completes the sentence-transformer family on Vulkan. distiluse is the one embedder whose XNNPACK Android default is quantized (8da4w) rather than fp32, so this is not the speedup the others were. Medians at 126 tokens, 5 interleaved rounds, against that 8da4w default: Adreno 840 13.72 vs 13.60 ms (a tie), Mali-G76 155.75 vs 190.77 ms (1.22x). What it does buy everywhere is accuracy and size. Cosine against the fp32 CPU reference goes 0.962244 -> 0.999995, and the download drops 393 -> 270 MB. 60/60 runs bit-identical on both GPUs.
All four sub-models lower to Vulkan. On a Galaxy S26 Ultra (Adreno 840),
medians over interleaved rounds, at 512 text tokens and 1000 latent frames:
vulkan xnnpack
duration_predictor 18.7 33.2 1.78x
text_encoder 72.3 140.9 1.95x
vector_estimator 682.5 1253.7 1.84x
vocoder 865.8 1284.3 1.48x
2.12x end to end at the default 8 flow-matching steps, where vector_estimator
is 84% of the total. Outputs match the fp32 CPU references at cosine 1.000000,
0.999414, 0.999994 and 0.999977.
Vulkan leads BACKEND_ORDER.android, so this becomes the Android default.
textToSpeech now provisions vulkan in download-libs; without it the backend is
never downloaded and the model silently falls back to XNNPACK.
Needs the ExecuTorch fixes in pytorch/executorch#22399, #22401, #22402, #22403
and #22406, all cherry-picked into the labs fork and built into the 1.4.1
native libs.
Adds the Vulkan entry to the model picker and makes it the initial selection on Android, which is also what SUPERTONIC.DEFAULT resolves to there.
12 tasks
Kokoro Core ML landed upstream in #1384 while this branch was adding the same variants; kept this branch's `kokoroCoreMl` helper, which produces identical URLs. SmolLM2 was re-published as 8da8w in #1385, so the three `variants({ ... })` groups now hold a single `XNNPACK_8DA8W` entry and the 8da4w/bf16 presets are gone.
msluszniak
marked this pull request as ready for review
September 1, 2026 14:54
barhanc
reviewed
Sep 2, 2026
barhanc
left a comment
barhanc
left a comment
Member
There was a problem hiding this comment.
I will test the models later today.
Member
There was a problem hiding this comment.
No need for additional file in root for this imo. This can be compressed to around 80 lines and put directly inside model.ts as local helpers. Something like
// =============================================================================
// MODEL REGISTRY
// =============================================================================
/** All supported ExecuTorch backend identifiers. */
const ALL_BACKENDS = ['xnnpack', 'coreml', 'mlx', 'vulkan'] as const;
/** Canonical tag representing an ExecuTorch backend. */
type BackendTag = (typeof ALL_BACKENDS)[number];
/** Platform-specific backend priority (accelerators first with CPU fallback). */
const BACKENDS_PREFERENCE: Record<'ios' | 'android', readonly BackendTag[]> = {
ios: rnexecutorchJsi.isEmulator ? ['xnnpack'] : ['coreml', 'mlx', 'xnnpack'],
android: ['vulkan', 'xnnpack'],
};
function getCandidateBackends(): readonly BackendTag[] {
if (Platform.OS !== 'ios' && Platform.OS !== 'android') {
return ALL_BACKENDS;
}
let registered: string[] = [];
try {
registered = getRegisteredBackends().map((name) => name.toLowerCase());
} catch {
registered = [];
}
if (registered.length === 0) registered = [...ALL_BACKENDS];
return BACKENDS_PREFERENCE[Platform.OS].filter((backend) =>
registered.some((name) => name.includes(backend))
);
}
/**
* Candidate backends for default resolution on this platform, ordered best-first
* (accelerators before CPU fallback). Falls back to all preferred backends if
* the native runtime cannot be queried.
*/
const CANDIDATE_BACKENDS = getCandidateBackends();
/**
* Resolves the platform-optimal `DEFAULT` variant for a group of model exports.
* @param options Map of named model variants.
* @param pinned Optional per-platform override for models where a specific
* export is preferred.
* @returns The variant map augmented with a resolved `DEFAULT` alias.
*/
function variants<T extends Record<string, unknown>>(
options: T,
pinned?: Partial<Record<'ios' | 'android', keyof T>>
): T & { readonly DEFAULT: T[keyof T] } {
if (Platform.OS !== 'ios' && Platform.OS !== 'android') {
return { ...options, DEFAULT: options[Object.keys(options)[0] as keyof T] };
}
const backendOf = (k: string) => k.toLowerCase().split('_')[0] as BackendTag;
const pin = pinned?.[Platform.OS];
if (pin !== undefined && CANDIDATE_BACKENDS.includes(backendOf(pin as string))) {
return { ...options, DEFAULT: options[pin] };
}
for (const backend of CANDIDATE_BACKENDS) {
const match = Object.keys(options).find((k) => backendOf(k) === backend);
if (match !== undefined) {
return { ...options, DEFAULT: options[match as keyof T] };
}
}
return { ...options, DEFAULT: options[Object.keys(options)[0] as keyof T] };
}
Member
Author
There was a problem hiding this comment.
Done, folded into models.ts as local helpers and deleted the file. The variants() unit tests now assert the same four rules against the real registry instead of a synthetic map.
…shape, default whisper base.en/small.en to int8 - fold modelVariants.ts into models.ts as local helpers and reframe its unit tests as registry-level assertions - restore kokoroModelPaths(backend, variant, dir) and the literal Core ML presets from the kokoro review - move the kokoro Core ML rationale out of the public TSDoc block - trim the RF-DETR, whisper and distiluse notes to their gist - put XNNPACK_INT8 ahead of XNNPACK_FP32 for whisper base.en and small.en: over 250 LibriSpeech test-clean clips int8 moves base.en 4.84%% -> 5.20%% WER and small.en 3.42%% -> 3.38%%, too little to outweigh halving the download. tiny.en keeps fp32 first, where int8 costs 6.08%% -> 7.77%%
Both mpnet encoders now ship an int8 weight-only Vulkan export alongside the fp16 one: two thirds the download (218 -> 133 MB) and faster on the Adreno 840, at cosine 0.998 against the fp32 reference instead of 0.99997. all-mpnet-base-v2 defaults to it on Android, where the accuracy is flat across the length range (0.9989 at 1 token, 0.9983 at the 382-token bound) and the download saving is worth 0.002 cosine. multi-qa-mpnet-base-dot-v1 keeps fp16 as its default: its int8 error grows with length, 0.9985 down to 0.9952 at the 510-token bound, and it is a dot-product retrieval model, so a systematic similarity shift matters more there. Both variants remain reachable by name; the declaration order within a backend picks the default. The int8 files need the runtime fix in v0.10.0-libs-1.4.1 (pytorch/executorch#22430); an older Vulkan lib aborts on a missing linear_qcs8w_* shader.
The int8 export is 6 to 8 times slower than fp16 on a Mali-G76: 25.4 s against 3.0 s per execution at 382 tokens, 5.4 s against 1.17 s at 128. It wins 1.34x on an Adreno 840, so this is not a bad export, it is a kernel that only suits one GPU family. linear_q8csw_tiled unpacks the int8 weights with bitfieldExtract inside the accumulation loop and ships with TILE_M4 = TILE_N4 = TILE_K4 = 1, so each unpacked weight texel feeds only four FMAs. Adreno issues integer and float at comparable rates and still comes out ahead on memory traffic; Bifrost dual-issues fp16 on the FMA pipe but not the integer path, so the fp16 model collects a 2x bonus the int8 model cannot. The registry resolves by backend, not by GPU vendor, so one Android default has to serve both. fp16 is the safe one. VULKAN_INT8 stays reachable by name and is still the better choice on Adreno.
14 tasks
barhanc
reviewed
Sep 2, 2026
Comment on lines
+69
to
+74
| type BackendTag = 'XNNPACK' | 'COREML' | 'MLX' | 'VULKAN'; | ||
|
|
||
| /** The platforms the registry resolves defaults for. */ | ||
| type TargetPlatform = 'ios' | 'android'; | ||
|
|
||
| const ALL_BACKENDS: readonly BackendTag[] = ['XNNPACK', 'COREML', 'MLX', 'VULKAN']; |
Member
There was a problem hiding this comment.
Better to write
const ALL_BACKENDS = [ ... ] as const;
type BackendTag = (typeof ALL_BACKENDS)[number]so there is no duplication.
| function usableBackends(): ReadonlySet<BackendTag> { | ||
| let registered: readonly string[] = []; | ||
| try { | ||
| registered = rnexecutorchJsi.getExecuTorchRegisteredBackends(); |
Member
There was a problem hiding this comment.
Use utils/getRegisteredBackends not the untyped version attached to jsi.
Comment on lines
+109
to
+110
| const onSimulator = PLATFORM === 'ios' && rnexecutorchJsi.isEmulator === true; | ||
| return new Set(onSimulator ? usable.filter((tag) => tag !== 'COREML' && tag !== 'MLX') : usable); |
Member
There was a problem hiding this comment.
Maybe cleaner to wire this directly into BACKEND_ORDER so it's immediately visible what backends are available on what platform / emulator in one place, like in the example snippet.
| * @returns The usable tags — every one of them when the native runtime cannot | ||
| * be asked, so that a missing answer widens the choice rather than emptying it. | ||
| */ | ||
| function usableBackends(): ReadonlySet<BackendTag> { |
Member
There was a problem hiding this comment.
Name is misleading because if registered fails or is empty then we don't really know what is usable. getCandidateBackends is imo more clear.
|
|
||
| const names = registered.map((name) => name.toLowerCase()); | ||
| const usable = ALL_BACKENDS.filter((tag) => | ||
| names.some((name) => name.startsWith(tag.toLowerCase())) |
Member
There was a problem hiding this comment.
.toLowerCase wouldn't be needed if backends were lowercase already.
Comment on lines
+115
to
+121
| /** | ||
| * Reads the backend out of a variant key. | ||
| * @param key The variant key, e.g. `COREML_FP16`. | ||
| * @returns The backend the key names, or `undefined` when it names none. | ||
| */ | ||
| const backendOf = (key: string): BackendTag | undefined => | ||
| ALL_BACKENDS.find((tag) => key === tag || key.startsWith(`${tag}_`)); |
Member
There was a problem hiding this comment.
Imo can be just defined inside function where it is used, it's a simple one liner. Also wouldn't just key.includes(tag) be enough here?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
models.<task>.<MODEL>.DEFAULTpointed at one fixed export, almost always XNNPACK, so an app that did nothing special ran on the CPU while the accelerator sat idle.DEFAULTis now resolved at load time from the platform (iOS: Core ML, MLX, XNNPACK; Android: Vulkan, XNNPACK; iOS simulator: XNNPACK only), the backends the binary was linked with, and declaration order as a tie-break. Naming a variant directly still overrides it. 96 of 117 groups change on an iOS device; on Android the embedders, whisper-tiny, Supertonic, PP-OCRv6, LFM2.5-VL and Gemma 4 E2B move to Vulkan.Where a model ships both Core ML and MLX the winner is a benchmark result, not something ordering can state, so those 9 groups pin their choice at the call site with the reason and a test fails until a new such group does the same. Core ML wins every measured pair (1.5x to 4.1x) except distiluse, which goes to MLX.
Auditing the 48 referenced HF repos turned up 41 published
.ptefiles no entry pointed at; 38 are now wired. Left out: a pre-rename duplicate and the two Llama 3.2 QAT+LoRA builds, which the rewrite does not support.Vulkan margins over XNNPACK, per execution on device:
all-MiniLM-L6-v22.4x, whisper encode 1.6x to 1.9x across all six sizes, Supertonic 3.1x on a Mali-G76 and 1.0x to 1.5x on an Adreno 840. Every Vulkan export was checked against the fp32 CPU reference and against run-to-run drift. Vulkan pays a one-time shader compile of roughly 300 ms (Adreno) to 700 ms (Mali) on first load.Both mpnet embedders also ship an int8 weight-only Vulkan export, 133 MB against 218 MB, but fp16 stays the Android default for both. int8 runs 1.34x faster than fp16 on an Adreno 840 (59 ms against 79 ms at the 382-token bound) and 6 to 8 times slower on a Mali-G76 (25.4 s against 3.0 s at 382 tokens, 5.4 s against 1.17 s at 128).
linear_q8csw_tiledunpacks the weights withbitfieldExtractinside the accumulation loop and ships withTILE_M4 = TILE_N4 = TILE_K4 = 1, so each unpacked texel feeds four FMAs; Adreno issues integer and float at comparable rates and still wins on memory traffic, Bifrost dual-issues fp16 on the FMA pipe but not the integer path. The registry resolves by backend rather than by GPU vendor, so one Android default has to serve both and fp16 is the safe one.VULKAN_INT8is reachable by name and is the better choice on Adreno.Accuracy is not the reason: int8 holds cosine 0.9983 to 0.9994 against the fp32 reference for
all-mpnet-base-v2and 0.9952 to 0.9993 formulti-qa-mpnet-base-dot-v1, measured on a Mali-G76 at 24 token counts from 1 to each model's bound, chosen to exercise every tile size the int8 kernel selects.Two fixes the resolution depends on:
download-libs.jsomitted backendsDEFAULTcould then never resolve to (now held in sync by a test), and the demo apps did not declare features they use. Diff bulk is registry entries trading a hand writtenDEFAULTline for avariants({ ... })/family({ ... })wrapper.Introduces a breaking change?
Type of change
Tested on
Testing instructions
yarn workspace react-native-executorch testandyarn lint..DEFAULT, and confirm the accelerated file is the one downloaded.Related issues
Closes #1389
Checklist
Additional notes
The Vulkan exports need the runtime fixes on the labs fork branch
@ms/separate-backends-1.4.1, published inv0.10.0-libs-1.4.1. Without them the file loses the Vulkan device, aborts on a missing operator, or returns NaNs. That release also carries six upstream fixes (pytorch/executorch#22348, #22349, #22409, #22410, #22411, #22430). The last one is what the int8 exports need:aten._weight_int8pack_mmdispatched tolinear_qcs8w_*shaders that no longer exist, so before it every weight-only int8 model aborted on its first execute. Only thevulkan-android-*artifacts changed; core and xnnpack are untouched.