Wait for first video frame to be received before responding to waitForDimension

[1egoman]

Overview

Fixes #2099

For background context, an ios specific delay was added to the LocalTrack.waitForDimensions method in #898 seemingly to work around some sort of platform specific issue. It's unclear exactly why this was added or why 10ms was chosen, however.

In the linked #2099 issue, the reporter mentions that this previously added 10ms delay isn't always long enough on ios - they report that for the first few hundred milliseconds, the dimensions of the front camera come back in landscape orientation even when the phone is in portrait. I like the reporter's suggested fix - instead of waiting for a fixed 10ms duration, wait for the first frame to be received on the video stream and THEN report back the dimensions of the video track. I took a crack at implementing this suggested fix here.

The biggest con to the reporter's approach is that this waitForDimensions method can now take longer - on a older iPhone 11 I had handy to test with, it took ~400ms. However, this seems to me like the right tradeoff given the result of the current version on main was just flat out incorrect.

Test plan

I had a LLM put together a small app that would let me test both sides of this - the old waitForDimensions implementation, and the new function implementation side by side.

What this test app was doing

track = await createLocalVideoTrack({ facingMode: 'user', resolution: VideoPresets.h1080.resolution });
line('track created', fmt(track.dimensions));

let answer;
if (mode === 'old') {
  // verbatim pre-fix algorithm
  await sleep(10);
  const started = Date.now();
  while (Date.now() - started < 1000) {
    const d = track.dimensions;
    if (d) { answer = d; break; }
    await sleep(50);
  }
} else {
  // New implementation of algorithm on this branch
  answer = await track.waitForDimensions();
}
line('answer', fmt(answer));

const truth = await groundTruth(track.mediaStreamTrack);
line('ground truth after frame', fmt(truth));

const ok = answer && answer.width === truth.width && answer.height === truth.height;
verdict(ok, ok ? 'answer matches the real frame' : `answered ${fmt(answer)} but the real frame is ${fmt(truth)}`);

Below is a screenshot of my results on an iPhone 11 - each is after a fresh page reload:

[changeset-bot]

🦋 Changeset detected

Latest commit: c05a42a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
livekit-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[1egoman]

(cc @davidzhao , tagging you since you originally reviewed #898 and might have some context on why this change was made given lukas is out this week)

[github-actions]

size-limit report 📦

Path	Size
dist/livekit-client.esm.mjs	111.45 KB (+0.13% 🔺)
dist/livekit-client.umd.js	120.59 KB (+0.18% 🔺)

[devin-ai-integration]

Devin Review found 2 potential issues.

[davidzhao]

lgtm

[davidzhao]

thank you for fixing this hack

[lagudafuadtosin]

Thanks for the fast turnaround, and for tracing the 10ms back to #898.

Read the diff against the measurements in the issue and it matches. I then ran three checks on the iPhone 15, iOS 26.6, in normal power and in Low Power Mode, so these are answers rather than questions.

• track.muted is false, enabled is true and readyState is live at the instant getUserMedia resolves, and no mute or unmute event fires in the first second. Your early return is safe on this device.
• A detached element with play() called gets its first requestVideoFrameCallback at +145ms in normal power and +144ms in Low Power Mode, and getSettings() reads 1080x1920 at exactly that callback. So on this phone your wait costs about 145ms, next to your ~400ms on the iPhone 11.
• With play() never called, the callback still fires on the paused detached element, +143ms normal and +127ms Low Power Mode, so the comment in the code holds. play() was not rejected in Low Power Mode either, on a muted playsInline element.

On why 10ms never worked: it was a guess at this window, and the window is about 145ms here and about 400ms on your iPhone 11, so it only passed when the read happened to land late.

One thing I could not test, no iPad here. getBrowser() only calls Safari iOS when the UA contains mobile/ (browserParser.ts:66), and iPadOS sends a desktop UA by default, so iPads skip the wait. The sensor is mounted the same way, so I would expect the same window. navigator.maxTouchPoints > 1 on a Macintosh UA is the usual tell, or drop the OS gate, since the wait is cheap wherever frames arrive fast.