[RNE Rewrite] test: add an on-device performance regression harness

[msluszniak]

Description

Adds apps/benchmarks, a headless Expo app that runs the task pipelines against deterministic synthetic inputs and reports model load time, inference latency and peak memory as JSON, plus a driver that collects a run and a comparator that diffs two runs and fails on regressions.

Built to bracket the ExecuTorch 1.3.1 to 1.4.1 bump: run the suite on 1.3.1, bump, run it again on the same device, compare.

Three design points worth a look:

• A raw-execute pass isolates ExecuTorch from the pipeline. A task timing folds model.execute together with preprocessing and post-processing, which are TypeScript and unaffected by a bump. The pass loads the .pte on its own and sizes its tensors from model.schema, so it covers every method a program exports and needs no per-model wiring. Methods whose schema cannot be pinned to concrete shapes are reported as skipped, with the reason.
• Memory is sampled in a pass of its own. Reading total PSS on Android walks /proc/self/smaps and costs milliseconds, which would otherwise land in the inference numbers. modules/bench-probe is a local Expo module reading task_vm_info.phys_footprint on iOS and total PSS on Android, both of which count the resident pages of a memory mapped .pte that the native heap counters miss.
• Inputs are pure functions of their parameters, so two runs feed byte identical data. Post-processing cost is input dependent, so a harness reading a photo off the device would move for reasons unrelated to the change under test.

The comparator refuses to diff runs from different devices, and reports a metric whose workload changed as INCOMPARABLE rather than as a delta. A delta inside the run's own interquartile range is reported but not failed.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

cd apps/benchmarks
yarn bench --platform android --label et-1.3.1     # builds, launches, collects
yarn bench --platform android --label et-1.4.1     # after the bump
yarn bench:compare results/et-1.3.1-*.json results/et-1.4.1-*.json

yarn bench starts a collector, sets the app's EXPO_PUBLIC_BENCH_* variables, builds and launches the app, and writes results/<label>-<platform>-<device>.json. It sets up adb reverse on Android and binds to the host LAN address for an iOS device. --suite full adds the larger models, --only <id> runs a single case, --no-launch waits for an app started by hand.

The full quick suite was run on a Galaxy S26 Ultra (SM-S948B, Android 16) with all six cases reporting. A full suite run covering Whisper and Supertonic was in progress at the time of writing.

Screenshots

Related issues

Closes #1078

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

Not yet run on iOS. The CoreML classification case is in the suite but unexercised, since CoreML programs fail to encode on the Simulator and need a physical device.

Two things the device runs surfaced that are worth knowing when reading a report:

• execute.<method> and pipeline.median are not comparable to each other. The raw pass takes dynamic dimensions at the top of their declared domain, so on all-MiniLM-L6-v2 it runs a 254 token forward while the pipeline runs a 20 token one, and the raw number comes out several times larger. Each is comparable against itself across runs, which is all the comparator asks. The resolved shapes are recorded per method.
• Style transfer peaks at roughly 1257 MB against a 372 MB loaded footprint on the S26 Ultra. That is the first measurement rather than a regression against anything, but it may be worth a look independent of the bump.

The suite is triggered by hand rather than run in CI. A host side benchmark on a CI runner would exercise none of the backends a bump actually changes, and its noise floor is wider than most regressions.

Rebased onto rne-rewrite now that the feature work has landed there. Two things came out of that:

• The harness had drifted off the library's public surface. The model schema contract moved behind a schema namespace and its specs gained a dimension type parameter ([RNE Rewrite] refactor!: add better model schema contract and validation logic #1327), and the task constants are re-exported flat rather than under a constants object. Both are tracked now, and apps/benchmarks typechecks against the current surface again.
• Three task families that landed after this branch was cut had no coverage, so the suite now benchmarks FastSAM instance segmentation, PP-OCRv6 OCR and Kokoro text-to-speech. All three sit in the full tier. FastSAM and PP-OCRv6 time their worklet entry point; Kokoro streams chunks from the RN thread and carries the same caveat as Supertonic.

That brings the suite to 16 cases, 7 quick and 9 full.

LLM and SDXS text-to-image are still uncovered. Both pull multi-gigabyte artifacts, which materially changes what a full run costs on a device, so they seemed worth a deliberate decision rather than being folded into a rebase.

Added a mechanism to hold the clock still, since the clock rather than the code is the largest source of noise here. Android exposes PowerManager's fixed-performance mode over cmd power, and vendors implement it as a hard frequency cap: on the S26 Ultra it takes every cluster from 3.19/3.40 GHz down to about 1.98 GHz, measured by reading scaling_max_freq back. Absolute numbers come out lower than a boosted run, which is the trade being made deliberately, because the same clock in every run is worth more than a fast one.

--pin-clocks defaults to auto, pinning where the device supports it and carrying on where it does not; on fails the run rather than measuring unpinned, off opts out. The driver verifies by reading the frequency back rather than trusting the call, because the command exists on devices whose HAL ignores it. Normal clocks are restored on exit, Ctrl-C and crash alike, so a run can never leave a phone capped at 2 GHz and quietly poison every later measurement taken on it. The run records whether it was pinned, and the comparator refuses to diff a pinned run against an unpinned one for the same reason it refuses across throttling.

There is no iOS equivalent. Nothing in the public API pins or caps the clock, so runs there still depend on the thermal gate and on cooling between suites.

The cooldown is adaptive too. --cooldown auto polls dumpsys until the framework reports no throttling and the battery temperature has stopped falling, rather than sleeping a fixed guess that is too long on a cold phone and too short after a heavy suite. It waits on a plateau instead of an absolute threshold, since what counts as cool differs per device while "no longer dropping" does not, and it requires two consecutive settled samples so a flat reading mid-fall does not release it early. There is a 30s floor so the heat of building and installing dissipates first, and a --cooldown-max ceiling, 900s by default, so a warm room or a charging phone cannot stall a run indefinitely. Charging is called out when detected because it keeps the device warm. On an idle S26 Ultra it releases after 30s where the fixed wait took 420s.

This is Android-only as well: iOS exposes no thermal readout to the host, so auto falls back to a fixed sleep there rather than pretending to measure something.