feat(speech): add Vulkan whisper-tiny variants - #1398
Closed
msluszniak wants to merge 1 commit into
Closed
Conversation
Registers the fp16 and int8 Vulkan exports of multilingual whisper-tiny and lets the speech demo pick them on Android. The Vulkan encode needs the ExecuTorch 1.4.1 runtime from v0.10.0-libs-1.4.1: earlier Vulkan libs lack the gelu tanh clamp, the reduce-shader barrier fix and the Adreno local-work-group fix.
Member
Author
|
Folded into #1392, which already carries the registry variant wiring. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Registers the fp16 and int8 Vulkan exports of multilingual whisper-tiny as
WHISPER.TINY.VULKAN_FP16/WHISPER.TINY.VULKAN_INT8, addsvulkanto thespeechToTextentry in thedownload-libs.jsfeature map so the backend lib is fetched, and offers both in the speech demo on Android.Encode runs on Vulkan (mel preprocessor falls back to XNNPACK); decode stays on XNNPACK, where it measured faster. On an Adreno 840 the encode is about 1.6x the XNNPACK time and decode is 4.08 ms, down from 11.96 ms.
The exports require the ExecuTorch 1.4.1 Vulkan runtime published in
v0.10.0-libs-1.4.1. Older Vulkan libs lack the gelu tanh clamp, the reduce-shader barrier fix and the Adreno local-work-group fix, and produce NaNs or a device loss.Introduces a breaking change?
Type of change
Tested on
Testing instructions
apps/speechon an Android device.<|0.00|>timestamp tokens.Screenshots
Related issues
Checklist
Additional notes
Verified on a Galaxy S26 Ultra against the exact
libvulkan_executorch_backend.sopublished inv0.10.0-libs-1.4.1(md53aba5c82..., matched between the installed APK and the release tarball).The
speechToTextfeature map listsxnnpack,coremland nowvulkan, but notmlx, even though the registry exposesMLX_BF16/MLX_INT8whisper variants. That looks like a pre-existing gap and is left alone here.