diff --git a/.gitignore b/.gitignore index e95e0003..e8a7a934 100644 --- a/.gitignore +++ b/.gitignore @@ -39,8 +39,13 @@ packages/nightwatch-devtools/nightwatch-video-*.webm trace-*.zip examples/**/trace-*/ +# test results +examples/**/test-results/ + # vitest --coverage output coverage/ # pnpm state, cache, logs, and debug files /packages/**/*.mjs + +.claude diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6d4fecc0..1324f454 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -144,7 +144,7 @@ The execution environment is the browser, not Node, so this package cannot impor Per-framework demo projects used for manual verification. -- `examples/wdio/` — WebdriverIO with Mocha (default). Run via `pnpm demo:wdio`. +- `examples/wdio/` — WebdriverIO, split into `cucumber/` and `mocha/` (shared page objects in `pageobjects/`). Run via `pnpm demo:wdio` (Cucumber) or `pnpm demo:wdio:mocha`. - `examples/nightwatch/` — Nightwatch (both vanilla and Cucumber). Run via `pnpm demo:nightwatch`. - `examples/selenium/` — Selenium with subdirs for `mocha-test/`, `jest-test/`, `cucumber-test/`, `jasmine-test/`, `vitest-test/`. `pnpm demo:selenium` runs mocha; `pnpm --filter @wdio/selenium-devtools example:` runs the others. diff --git a/CLAUDE.md b/CLAUDE.md index 0416c6ae..14fff1a0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -233,12 +233,19 @@ Documented divergences from the conventions above. They exist today as debt to b ### Architecture - `replaceCommand` has two semantics — Selenium mutates in place (preserves `_id`/`id` for chained calls); Nightwatch splices and reissues. Both call the same `core/suite-helpers` factories; the storage strategy stays adapter-specific because runner integrations differ. Could be unified by parameterizing the policy if the divergence ever causes a real problem. -- `patchNodeAssert` is wired only in `selenium-devtools`. The shared helper lives in `core/assert-patcher`; Service and Nightwatch can opt in via a one-line call when ready. Not auto-enabled — both communities lean on chai/expect. +- `patchNodeAssert` (via `core/assert-patcher`) is now wired in all three adapters, default-on behind each adapter's `captureAssertions` option (opt out with `captureAssertions: false`). Framework matcher libraries differ: Service additionally taps expect-webdriverio's `afterAssertion` hook so passing+failing matchers render as `expect.*` actions; Selenium (chai/jest) and Nightwatch (`browser.assert`/`verify`) don't yet surface passing framework-matcher assertions as trace actions. - BiDi is auto-attached in Service and Selenium; Nightwatch is opt-in via `bidi: true` and requires `webSocketUrl: true` in capabilities. +- Retry-aware trace policies aren't uniform across adapters. Selenium emits one metadata entry per attempt (counter-based `generateStableUid` uid + retry-stable tracker key), so all six policies — including `retain-on-first-failure` (needs the first attempt's outcome) — evaluate correctly. Service and Nightwatch key metadata on the retry-stable uid and overwrite/mutate per attempt, collapsing to the *final* attempt's outcome; the retry-count policies (`retain-on-failure`, `on-first-retry`, `on-all-retries`, `retain-on-failure-and-retries`) are still correct, but `retain-on-first-failure` can't see a failed-then-passed first attempt and under-retains. Proper fix rides with B5 (per-test granularity preserves per-attempt slices); until then all three set `attemptInfoAvailable: true`. +- Nightwatch negated native asserts (`browser.assert.not.*` / `verify.not.*`) produce no trace row: the `not` namespace is a nested Proxy that `wrapAssertionNamespaces` passes through untouched, and the internal command it issues is dropped by the `hasUserSource` gate in `browserProxy`. Positive `assert.*`/`verify.*` still record. +- Eager per-test trace slice (Nightwatch + Selenium) can drop an action snapshot whose fire-and-forget capture hasn't resolved by `afterEach` / scenario end — the slice is written from whatever snapshots exist at flush time. The WDIO service is immune because it awaits each snapshot inline before flushing. +- Nightwatch native asserts populate all at once in live mode and finalize pass/fail in one test-end batch: Nightwatch exposes no per-assertion execution/outcome hook reachable from the plugin (`client.queue.tree` / `client.reporter` aren't on `browser`), so call-time capture streams neutral pending rows and `afterEach` reconciles outcomes. Trace-timeline positions are corrected from `results.commands`. +- Service assertion-suppression self-heal is command-triggered: a swallowed soft-assert (`expect.soft` / user `try/catch`) whose matcher hard-throws leaves `#assertionDepth` stuck until the next top-level user command's stack shows no `expect-webdriverio` frame (heal) or the next `beforeAssertion` (reset). A stuck depth with no following command or assertion before test end persists until the next test's `resetStack()` — benign (no later command in that test to suppress). ### File-size (raw line counts; soft cap is 500 logic lines) -None of the entries below trigger the `max-lines` lint rule after `skipBlankLines`/`skipComments`. They're documented because their raw line count is over 500, and the next substantive change to any of them should still look for an extraction opportunity. +Most entries below don't trigger the `max-lines` lint rule after `skipBlankLines`/`skipComments`; they're documented because their raw line count is over 500, and the next substantive change to any of them should still look for an extraction opportunity. The service plugin is the exception — it's now over the *logic*-line cap. + +- `packages/service/src/index.ts` — over the 500-**logic**-line soft cap (the WDIO plugin god-file: session/screencast/command/assertion/trace-slice wiring in one class). Split candidate; the trace-slice, assertion-capture, and screencast concerns are the natural extraction seams. - `packages/nightwatch-devtools/src/index.ts` (~536 raw). Cucumber/test/run-lifecycle, session-init, event-hub modules already extracted; remainder is the `PluginInternals` accessor bag plus per-method delegators plus the factory. Accept-as-is. - `packages/selenium-devtools/src/index.ts` (~560 raw). Session/test-lifecycle extracted; remainder is the `PluginInternals` accessor bag plus onCommand/onDriverCreated wiring. Same situation as nightwatch. diff --git a/README.md b/README.md index 96e7adf8..09f93c2e 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ When BiDi is active in Selenium or Nightwatch, the per-command Chrome performanc ### 📦 Trace mode (trace.zip) -Headless capture path — no DevTools UI window opens. At session end the adapter writes a trace artifact next to the user's output directory, suitable for offline replay, AI-agent diffing, or any consumer that prefers a portable artifact over a live UI. +Headless capture path — no DevTools UI window opens. At session end the adapter writes trace artifacts into a `test-results/` folder (created next to the resolved spec/config directory), suitable for offline replay, AI-agent diffing, or any consumer that prefers a portable artifact over a live UI. | Adapter | How to enable | |---|---| @@ -79,7 +79,7 @@ Headless capture path — no DevTools UI window opens. At session end the adapte | **Nightwatch** | `globals: nightwatchDevtools({ mode: 'trace' })` | The trace artifact contains: -- `trace.trace` — NDJSON `context-options` + `before`/`after` action events. When test hooks are available (Mocha's `it()` / Cucumber's `Scenario()`), each test becomes a [`Tracing.tracingGroup`](https://github.com/VibiumDev/vibium/blob/main/docs/explanation/recording-format.md#action-groups-user-defined) span — an open/close `before`/`after` pair with `method: "tracingGroup"` and `params.name` set to the test title. Child actions inside the group carry `parentId` pointing back to the group's `callId`, so timeline viewers render tests as labelled spans wrapping their commands. +- `trace.trace` — NDJSON `context-options` + `before`/`after` action events. When test hooks are available (Mocha's `it()` / Cucumber's `Scenario()`), each test becomes a `Tracing.tracingGroup` span — an open/close `before`/`after` pair with `method: "tracingGroup"` and `params.name` set to the test title. Child actions inside the group carry `parentId` pointing back to the group's `callId`, so timeline viewers render tests as labelled spans wrapping their commands. - `trace.network` — HAR-style network entries derived from the existing capture - `resources/page@-.jpeg` — screenshot per user-facing action - `resources/elements-page@-.json` — flat interactable element list extracted by the page-injected scripts in `@wdio/devtools-core/element-scripts` @@ -103,7 +103,7 @@ npx show-trace path/to/trace.zip # in a project that installs an adapter The `show-trace` bin is exposed by each adapter (`@wdio/devtools-service`, `@wdio/nightwatch-devtools`, `@wdio/selenium-devtools`), so `pnpm show-trace ` / `npx show-trace ` work in any project that installs one — no extra dependency. -**Other viewers.** Because the format is unchanged, the same `.zip` also drops into [player.vibium.dev](https://player.vibium.dev) or `npx playwright show-trace `. The format follows the [Vibium recording format](https://github.com/VibiumDev/vibium/blob/main/docs/explanation/recording-format.md) spec — a Playwright-compatible NDJSON schema that the ecosystem already renders. This is the same format [`@wdio/mcp`](https://webdriver.io/docs/mcp) uses for AI-driven session recording. +**Other viewers.** The trace uses a portable NDJSON schema, so the same `.zip` also opens in other compatible trace viewers that read the format. This is the same format [`@wdio/mcp`](https://webdriver.io/docs/mcp) uses for AI-driven session recording. #### Options @@ -111,7 +111,9 @@ The `show-trace` bin is exposed by each adapter (`@wdio/devtools-service`, `@wdi |--------|--------|---------|-------------| | `mode` | `'live'` \| `'trace'` | `'live'` | `'live'` launches the DevTools UI; `'trace'` writes an offline artifact. | | `traceFormat` | `'zip'` \| `'ndjson-directory'` | `'zip'` | Output layout. `'zip'` writes a single archive; `'ndjson-directory'` unpacks into `trace-/`. | -| `traceGranularity` | `'session'` \| `'spec'` | `'session'` | `'session'` writes one trace per worker; `'spec'` writes one trace per spec file — smaller artifacts, easier to navigate. | +| `traceGranularity` | `'session'` \| `'spec'` \| `'test'` | `'session'` | `'session'` writes one trace per worker; `'spec'` one per spec file; `'test'` one per test into its own `--<browser>[-retryN]/trace.zip` folder — the smallest, most navigable artifacts, and the best pairing for a retention policy. | +| `tracePolicy` | `'on'` \| `'retain-on-failure'` \| `'retain-on-first-failure'` \| `'on-first-retry'` \| `'on-all-retries'` \| `'retain-on-failure-and-retries'` | `'on'` | Which traces to keep. `'on'` keeps every trace; the rest keep only failing/retried tests — pairs well with `traceGranularity: 'test'`. | +| `captureAssertions` | `boolean` | `true` | Capture assertions as action rows: `node:assert` (all adapters), WebdriverIO `expect(...)` matchers, and Nightwatch `browser.assert`/`verify`. Set `false` to opt out. | WDIO config example: @@ -129,7 +131,7 @@ services: [[DevToolsHookService, { Adapters detect mobile sessions via `platformName: 'android' | 'ios'` (case-insensitive) and adjust the per-action snapshot to extract elements from the mobile XML tree instead of the DOM. The trace's `context-options` records `title: 'android' — <deviceName>` / `'ios' — <deviceName>` so the viewer labels frames correctly. -A reference WDIO config is at [examples/wdio/wdio.mobile.conf.ts](examples/wdio/wdio.mobile.conf.ts). Prereqs to run it end-to-end with a local emulator: +A reference WDIO config is at [examples/wdio/cucumber/wdio.mobile.conf.ts](examples/wdio/cucumber/wdio.mobile.conf.ts). Prereqs to run it end-to-end with a local emulator: 1. **Java JDK** — `brew install --cask temurin` 2. **Android SDK** — `brew install --cask android-commandlinetools` then `yes | sdkmanager --licenses && sdkmanager "platform-tools" "emulator" "system-images;android-34;google_apis_playstore;arm64-v8a"`. The brew cask installs sdkmanager under `/opt/homebrew/share/android-commandlinetools/`, and sdkmanager downloads other SDK pieces alongside it — set `ANDROID_HOME` to that path (not `~/Library/Android/sdk/`). diff --git a/examples/nightwatch/tests/assert-capture-check.js b/examples/nightwatch/tests/assert-capture-check.js new file mode 100644 index 00000000..6f8511a3 --- /dev/null +++ b/examples/nightwatch/tests/assert-capture-check.js @@ -0,0 +1,18 @@ +// Verification harness for native-assert trace capture (browser.assert / verify). +// Run `pnpm demo:nightwatch` and inspect the dashboard Actions: the PASSING +// asserts must render green and the FAILING ones RED. If a failing assert shows +// green, the classic-chained capture is mis-reporting failures (the known risk +// in nativeAssertCapture — the wrapper sees the enqueue, not the queued result). +describe('Native assert capture check', function () { + it('renders passing and failing native asserts', async function (browser) { + await browser.url('https://example.com').waitForElementVisible('body', 5000) + + // Soft verify.* first — never aborts the test, so all four always run. + browser.verify.titleContains('Example') // PASS → expect green + browser.verify.titleContains('SOFT_FAIL_ME') // FAIL → expect RED + + // Hard assert.* — the classic-chained/queued path under test. + browser.assert.titleContains('Example') // PASS → expect green + browser.assert.titleContains('HARD_FAIL_ME') // FAIL → expect RED + }) +}) diff --git a/examples/nightwatch/tests/login.js b/examples/nightwatch/tests/login.js deleted file mode 100644 index d8964d8d..00000000 --- a/examples/nightwatch/tests/login.js +++ /dev/null @@ -1,47 +0,0 @@ -describe('The Internet Guinea Pig Website', function () { - afterEach(async function (browser) { - await browser.end() - }) - - it('should log into the secure area with valid credentials', async function (browser) { - console.log('[TEST] Navigating to login page') - await browser - .url('https://the-internet.herokuapp.com/login') - .waitForElementVisible('body') - - console.log('[TEST] Attempting login with username: tomsmith') - await browser - .setValue('#username', 'tomsmith') - .setValue('#password', 'SuperSecretPassword!') - .click('button[type="submit"]') - - console.log( - '[TEST] Verifying flash message: You logged into a secure area!' - ) - await browser - .waitForElementVisible('#flash', 10000) - .assert.textContains('#flash', 'You logged into a secure area!') - - console.log('[TEST] Flash message verified successfully') - }) - - it('should show error with invalid credentials', async function (browser) { - console.log('[TEST] Navigating to login page') - await browser - .url('https://the-internet.herokuapp.com/login') - .waitForElementVisible('body') - - console.log('[TEST] Attempting login with username: foobar') - await browser - .setValue('#username', 'foobar') - .setValue('#password', 'barfoo') - .click('button[type="submit"]') - - console.log('[TEST] Verifying flash message: Your username is invalid!') - await browser - .waitForElementVisible('#flash', 10000) - .assert.textContains('#flash', 'Your username is invalid!') - - console.log('[TEST] Flash message verified successfully') - }) -}) diff --git a/examples/nightwatch/tests/sample.js b/examples/nightwatch/tests/sample.js deleted file mode 100644 index ff2e4b2a..00000000 --- a/examples/nightwatch/tests/sample.js +++ /dev/null @@ -1,21 +0,0 @@ -describe('Sample Nightwatch Test with DevTools', function () { - it('should navigate to example.com and check title', async function (browser) { - await browser - .url('https://example.com') - .waitForElementVisible('body', 5000) - .assert.titleContains('Example') - .assert.visible('h1') - - const result = await browser.getText('h1') - browser.assert.ok(result.includes('Example'), 'H1 contains "Example"') - }) - - it('should perform basic interactions', async function (browser) { - await browser - .url('https://www.google.com') - .waitForElementVisible('body', 5000) - .assert.visible('textarea[name="q"]') - .setValue('textarea[name="q"]', 'WebdriverIO DevTools') - .pause(1000) - }) -}) diff --git a/examples/wdio/cucumber/features/login-fail.feature b/examples/wdio/cucumber/features/login-fail.feature new file mode 100644 index 00000000..c8d325b8 --- /dev/null +++ b/examples/wdio/cucumber/features/login-fail.feature @@ -0,0 +1,9 @@ +Feature: Retention check — deliberately failing login + + # Logs in with bad credentials but asserts the SUCCESS message, so the Then + # step fails. Used to verify retain-on-failure keeps this spec's trace while + # dropping the passing login.feature. Reuses the existing step definitions. + Scenario: Failing assertion exercises retain-on-failure + Given I am on the login page + When I login with foobar and barfoo + Then I should see a flash message saying You logged into a secure area! diff --git a/examples/wdio/features/login.feature b/examples/wdio/cucumber/features/login.feature similarity index 84% rename from examples/wdio/features/login.feature rename to examples/wdio/cucumber/features/login.feature index e6dd0f87..8b9bcf9a 100644 --- a/examples/wdio/features/login.feature +++ b/examples/wdio/cucumber/features/login.feature @@ -8,5 +8,5 @@ Feature: The Internet Guinea Pig Website Examples: | username | password | message | - | tomsmith | SuperSecretPassword! | You logged into a secure area! | + | tomsmith | SuperSecretPassword1! | You logged into a secure area! | | foobar | barfoo | Your username is invalid! | diff --git a/examples/wdio/features/step-definitions/steps.ts b/examples/wdio/cucumber/features/step-definitions/steps.ts similarity index 75% rename from examples/wdio/features/step-definitions/steps.ts rename to examples/wdio/cucumber/features/step-definitions/steps.ts index 32522982..5d02133e 100644 --- a/examples/wdio/features/step-definitions/steps.ts +++ b/examples/wdio/cucumber/features/step-definitions/steps.ts @@ -1,8 +1,8 @@ import { Given, When, Then, After } from '@wdio/cucumber-framework' import { browser, expect } from '@wdio/globals' -import LoginPage from '../pageobjects/login.page.js' -import SecurePage from '../pageobjects/secure.page.js' +import LoginPage from '../../../pageobjects/login.page.js' +import SecurePage from '../../../pageobjects/secure.page.js' const pages = { login: LoginPage @@ -26,9 +26,9 @@ When(/^I login with (\w+) and (.+)$/, async (username, password) => { Then(/^I should see a flash message saying (.*)$/, async (message) => { console.log(`[TEST] Verifying flash message: ${message}`) - const el = await SecurePage.flashAlert - await expect(el).toBeExisting() - await expect(el).toHaveText(expect.stringContaining(message)) + await expect(SecurePage.flashAlert).toBeExisting() + await expect(SecurePage.flashAlert).toHaveText( + expect.stringContaining(message) + ) console.log('[TEST] Flash message verified successfully') - // await browser.pause(15000) }) diff --git a/examples/wdio/wdio.conf.ts b/examples/wdio/cucumber/wdio.conf.ts similarity index 98% rename from examples/wdio/wdio.conf.ts rename to examples/wdio/cucumber/wdio.conf.ts index 0f7b3470..4c535acc 100644 --- a/examples/wdio/wdio.conf.ts +++ b/examples/wdio/cucumber/wdio.conf.ts @@ -54,7 +54,9 @@ export const config: Options.Testrunner = { // and 30 processes will get spawned. The property handles how many capabilities // from the same test should run tests. // - maxInstances: 10, + // Live mode drives a single-session dashboard; >1 worker streams two feature + // files into it at once and neither renders cleanly. One instance = readable demo. + maxInstances: 1, // // If you have trouble getting all important capabilities together, check out the // Sauce Labs platform configurator - a great tool to configure your capabilities: @@ -87,7 +89,7 @@ export const config: Options.Testrunner = { // Define all options that are relevant for the WebdriverIO instance here // // Level of logging verbosity: trace | debug | info | warn | error | silent - logLevel: 'debug', + logLevel: 'warn', // // Set specific log levels per logger // loggers: diff --git a/examples/wdio/wdio.mobile.conf.ts b/examples/wdio/cucumber/wdio.mobile.conf.ts similarity index 99% rename from examples/wdio/wdio.mobile.conf.ts rename to examples/wdio/cucumber/wdio.mobile.conf.ts index 7f693ab4..13618ffe 100644 --- a/examples/wdio/wdio.mobile.conf.ts +++ b/examples/wdio/cucumber/wdio.mobile.conf.ts @@ -44,7 +44,7 @@ export const config: Options.Testrunner = { // eslint-disable-next-line @typescript-eslint/no-explicit-any ] as any, - logLevel: 'info', + logLevel: 'warn', bail: 0, baseUrl: 'http://localhost', waitforTimeout: 15000, diff --git a/examples/wdio/cucumber/wdio.retention.conf.ts b/examples/wdio/cucumber/wdio.retention.conf.ts new file mode 100644 index 00000000..a71b97bb --- /dev/null +++ b/examples/wdio/cucumber/wdio.retention.conf.ts @@ -0,0 +1,68 @@ +import path from 'node:path' + +// Disposable harness for verifying tracePolicy end-to-end. Runs one passing +// spec (login.feature) and one failing spec (login-fail.feature) at spec +// granularity so retain-on-failure can be seen dropping the passing spec's +// trace while keeping the failing one. Change `tracePolicy` below to try each. + +const __dirname = path.resolve(path.dirname(new URL(import.meta.url).pathname)) + +export const config: WebdriverIO.Config = { + runner: 'local', + tsConfigPath: './tsconfig.json', + + specs: ['./features/login.feature', './features/login-fail.feature'], + + maxInstances: 1, + + capabilities: [ + { + browserName: 'chrome', + browserVersion: '149.0.7827.201', // specify chromium browser version for testing + 'goog:chromeOptions': { + args: [ + '--headless', + '--disable-gpu', + '--remote-allow-origins=*', + '--window-size=1600,1200' + ] + } + } + ], + + logLevel: 'warn', + baseUrl: 'http://localhost', + waitforTimeout: 10000, + connectionRetryTimeout: 120000, + connectionRetryCount: 3, + + services: [ + [ + 'devtools', + { + mode: 'trace' as const, + traceGranularity: 'spec' as const + // tracePolicy: 'retain-on-failure' as const + } + ] + ], + + framework: 'cucumber', + reporters: ['spec'], + + cucumberOpts: { + require: [ + path.resolve(__dirname, 'features', 'step-definitions', 'steps.ts') + ], + backtrace: false, + requireModule: [], + dryRun: false, + failFast: false, + snippets: true, + source: true, + strict: false, + tagExpression: '', + timeout: 60000, + ignoreUndefinedDefinitions: false + } +} diff --git a/examples/wdio/wdio.trace.conf.ts b/examples/wdio/cucumber/wdio.trace.conf.ts similarity index 100% rename from examples/wdio/wdio.trace.conf.ts rename to examples/wdio/cucumber/wdio.trace.conf.ts diff --git a/examples/wdio/mocha/retry/flaky.e2e.ts b/examples/wdio/mocha/retry/flaky.e2e.ts new file mode 100644 index 00000000..0b87bc83 --- /dev/null +++ b/examples/wdio/mocha/retry/flaky.e2e.ts @@ -0,0 +1,19 @@ +import { browser, expect } from '@wdio/globals' + +// Deterministically flaky: throws on the first attempt and passes on the retry. +// The module-level counter survives mocha's in-process retry, so attempt 0 fails +// and attempt 1 succeeds. The test ends PASSED — so retain-on-failure would drop +// its trace, but on-first-retry keeps it, which is exactly what B4 verifies: +// the retry attempt is captured and is distinct from a failure. +let attempts = 0 + +describe('Flaky (passes on retry)', () => { + it('fails the first attempt, then passes', async () => { + await browser.url('https://the-internet.herokuapp.com/login') + attempts += 1 + if (attempts === 1) { + throw new Error('intentional first-attempt failure — should retry') + } + await expect(browser).toHaveTitle('The Internet') + }) +}) diff --git a/examples/wdio/mocha/specs/login-fail.e2e.ts b/examples/wdio/mocha/specs/login-fail.e2e.ts new file mode 100644 index 00000000..b939a28b --- /dev/null +++ b/examples/wdio/mocha/specs/login-fail.e2e.ts @@ -0,0 +1,17 @@ +import { expect } from '@wdio/globals' + +import LoginPage from '../../pageobjects/login.page.js' +import SecurePage from '../../pageobjects/secure.page.js' + +// Deliberately failing — mirrors login-fail.feature, so the Errors tab and +// tracePolicy: 'retain-on-failure' have something to exercise. +describe('Login (failing)', () => { + it('asserts the wrong flash message so the run fails', async () => { + console.log('[TEST] submitting invalid credentials') + await LoginPage.open() + await LoginPage.login('foobar', 'barfoo') + await expect(SecurePage.flashAlert).toHaveText( + expect.stringContaining('You logged into a secure area!') + ) + }) +}) diff --git a/examples/wdio/mocha/specs/login.e2e.ts b/examples/wdio/mocha/specs/login.e2e.ts new file mode 100644 index 00000000..0282a744 --- /dev/null +++ b/examples/wdio/mocha/specs/login.e2e.ts @@ -0,0 +1,26 @@ +import { expect } from '@wdio/globals' + +import LoginPage from '../../pageobjects/login.page.js' +import SecurePage from '../../pageobjects/secure.page.js' + +describe('Login', () => { + it('logs into the secure area with valid credentials', async () => { + console.log('[TEST] logging in with valid credentials') + await LoginPage.open() + await LoginPage.login('tomsmith', 'SuperSecretPassword!') + await expect(SecurePage.flashAlert).toBeExisting() + await expect(SecurePage.flashAlert).toHaveText( + expect.stringContaining('You logged into a secure area!') + ) + console.log('[TEST] secure area reached') + }) + + it('shows an error message for an invalid username', async () => { + console.log('[TEST] logging in with an invalid username') + await LoginPage.open() + await LoginPage.login('foobar', 'barfoo') + await expect(SecurePage.flashAlert).toHaveText( + expect.stringContaining('Your username is invalid!') + ) + }) +}) diff --git a/examples/wdio/mocha/wdio.conf.ts b/examples/wdio/mocha/wdio.conf.ts new file mode 100644 index 00000000..966226f9 --- /dev/null +++ b/examples/wdio/mocha/wdio.conf.ts @@ -0,0 +1,55 @@ +import type { Options } from '@wdio/types' + +// Mocha counterpart to wdio.conf.ts (which runs the Cucumber example). Same +// capabilities and devtools service; only the framework + spec layout differ. +export const config: Options.Testrunner = { + runner: 'local', + autoCompileOpts: { + autoCompile: true, + tsNodeOpts: { + project: './tsconfig.json', + transpileOnly: true + } + }, + specs: ['./specs/**/*.e2e.ts'], + exclude: [], + // Live mode drives a single-session dashboard; >1 worker streams two sessions + // into it at once and neither renders cleanly. One instance = readable demo. + maxInstances: 1, + capabilities: [ + { + browserName: 'chrome', + 'goog:chromeOptions': { + args: [ + '--headless', + '--disable-gpu', + '--remote-allow-origins=*', + '--window-size=1600,900' + ] + } + } + ], + logLevel: 'warn', + bail: 0, + baseUrl: 'http://localhost', + waitforTimeout: 10000, + connectionRetryTimeout: 120000, + connectionRetryCount: 3, + services: [ + [ + 'devtools', + { + mode: 'live' as const, + traceGranularity: 'spec' as const + // tracePolicy: 'retain-on-failure' as const + // screencast: { enabled: true, pollIntervalMs: 200 } + } + ] + ], + framework: 'mocha', + reporters: ['spec'], + mochaOpts: { + ui: 'bdd', + timeout: 60000 + } +} diff --git a/examples/wdio/mocha/wdio.retry.conf.ts b/examples/wdio/mocha/wdio.retry.conf.ts new file mode 100644 index 00000000..d9c873bb --- /dev/null +++ b/examples/wdio/mocha/wdio.retry.conf.ts @@ -0,0 +1,58 @@ +import type { Options } from '@wdio/types' + +// Disposable harness for verifying B4 (retry-aware trace policies). Runs the +// deterministically-flaky spec (fails once, passes on retry) alongside a clean +// passing spec at spec granularity. With tracePolicy 'on-first-retry' only the +// flaky spec's trace is retained — proving the retry attempt is captured and is +// distinct from failure (the flaky test ends PASSED, so retain-on-failure would +// drop it). Flip tracePolicy below to exercise the other retry-aware policies. +export const config: Options.Testrunner = { + runner: 'local', + autoCompileOpts: { + autoCompile: true, + tsNodeOpts: { + project: './tsconfig.json', + transpileOnly: true + } + }, + specs: ['./retry/flaky.e2e.ts', './specs/login.e2e.ts'], + exclude: [], + maxInstances: 1, + capabilities: [ + { + browserName: 'chrome', + 'goog:chromeOptions': { + args: [ + '--headless', + '--disable-gpu', + '--remote-allow-origins=*', + '--window-size=1600,900' + ] + } + } + ], + logLevel: 'warn', + bail: 0, + baseUrl: 'http://localhost', + waitforTimeout: 10000, + connectionRetryTimeout: 120000, + connectionRetryCount: 3, + services: [ + [ + 'devtools', + { + mode: 'trace' as const, + traceGranularity: 'test' as const, + tracePolicy: 'retain-on-failure' as const + } + ] + ], + framework: 'mocha', + reporters: ['spec'], + mochaOpts: { + ui: 'bdd', + timeout: 60000, + // 1 retry = 2 total attempts; the flaky spec fails attempt 0, passes attempt 1. + retries: 1 + } +} diff --git a/examples/wdio/package.json b/examples/wdio/package.json index c783a577..4bbb0957 100644 --- a/examples/wdio/package.json +++ b/examples/wdio/package.json @@ -6,6 +6,7 @@ "devDependencies": { "@wdio/cli": "9.28.0", "@wdio/cucumber-framework": "9.28.0", + "@wdio/mocha-framework": "9.28.0", "@wdio/devtools-service": "workspace:*", "@wdio/globals": "9.28.0", "@wdio/local-runner": "9.28.0", @@ -18,8 +19,10 @@ "typescript": "^6.0.3" }, "scripts": { - "wdio": "wdio run ./wdio.conf.ts", - "mobile": "wdio run ./wdio.mobile.conf.ts", - "trace": "wdio run ./wdio.trace.conf.ts" + "cucumber": "wdio run ./cucumber/wdio.conf.ts", + "mocha": "wdio run ./mocha/wdio.conf.ts", + "mobile": "wdio run ./cucumber/wdio.mobile.conf.ts", + "trace": "wdio run ./cucumber/wdio.trace.conf.ts", + "retention": "wdio run ./cucumber/wdio.retention.conf.ts" } } diff --git a/examples/wdio/features/pageobjects/login.page.ts b/examples/wdio/pageobjects/login.page.ts similarity index 100% rename from examples/wdio/features/pageobjects/login.page.ts rename to examples/wdio/pageobjects/login.page.ts diff --git a/examples/wdio/features/pageobjects/page.ts b/examples/wdio/pageobjects/page.ts similarity index 100% rename from examples/wdio/features/pageobjects/page.ts rename to examples/wdio/pageobjects/page.ts diff --git a/examples/wdio/features/pageobjects/secure.page.ts b/examples/wdio/pageobjects/secure.page.ts similarity index 100% rename from examples/wdio/features/pageobjects/secure.page.ts rename to examples/wdio/pageobjects/secure.page.ts diff --git a/examples/wdio/tsconfig.json b/examples/wdio/tsconfig.json index 8acd81f7..8b8a6786 100644 --- a/examples/wdio/tsconfig.json +++ b/examples/wdio/tsconfig.json @@ -11,7 +11,8 @@ "node", "@wdio/globals/types", "expect-webdriverio", - "@wdio/cucumber-framework" + "@wdio/cucumber-framework", + "@wdio/mocha-framework" ], "skipLibCheck": true, "noEmit": true, @@ -24,7 +25,8 @@ "noFallthroughCasesInSwitch": true }, "include": [ - "test", - "wdio.conf.ts" + "cucumber", + "mocha", + "pageobjects" ] } diff --git a/package.json b/package.json index e4037c25..45fba2f4 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,9 @@ "type": "module", "scripts": { "build": "pnpm -r build", - "demo:wdio": "wdio run ./examples/wdio/wdio.conf.ts", + "demo:wdio": "wdio run ./examples/wdio/cucumber/wdio.conf.ts", + "demo:wdio:mocha": "wdio run ./examples/wdio/mocha/wdio.conf.ts", + "demo:wdio:retry": "wdio run ./examples/wdio/mocha/wdio.retry.conf.ts", "demo:nightwatch": "pnpm --filter @wdio/nightwatch-devtools example", "demo:selenium": "pnpm --filter @wdio/selenium-devtools example", "dev": "pnpm --parallel dev", diff --git a/packages/app/src/app.ts b/packages/app/src/app.ts index d9a89e5d..d8076a4b 100644 --- a/packages/app/src/app.ts +++ b/packages/app/src/app.ts @@ -232,9 +232,10 @@ export class WebdriverIODevtoolsApplication extends Element { class="flex h-[calc(100%-40px)] w-full relative" > ${ - // Only render the test-suite sidebar (and its resize slider) when the - // trace came from a testrunner — the player (standalone) has no tree, - // so the slider would otherwise show a stray dragger on hover. + // Only render the test-suite sidebar (and its resize slider) for a + // live testrunner session. The player has no run/rerun affordances, + // so the tree is dead weight even for testrunner-captured zips. + !this.dataManager.playerMode && this.dataManager.traceType === TraceType.Testrunner ? html`<wdio-devtools-sidebar style="${this.#drag?.getPosition()}" diff --git a/packages/app/src/components/browser/snapshot-styles.ts b/packages/app/src/components/browser/snapshot-styles.ts index 70de8b15..72699132 100644 --- a/packages/app/src/components/browser/snapshot-styles.ts +++ b/packages/app/src/components/browser/snapshot-styles.ts @@ -1,4 +1,6 @@ -import { css } from 'lit' +import { css, unsafeCSS } from 'lit' + +import { BROWSER_BACKDROP_GRADIENT } from '../../controller/constants.js' /** Component styles for `<wdio-devtools-snapshot>`. Pulled out of snapshot.ts * so the main component file stays focused on the iframe/screencast logic. */ @@ -11,11 +13,7 @@ export const snapshotStyles = css` align-items: center; justify-content: center; box-sizing: border-box !important; - background: radial-gradient( - 120% 120% at 50% 0%, - var(--vscode-editorWidget-background), - var(--vscode-editor-background) - ); + background: ${unsafeCSS(BROWSER_BACKDROP_GRADIENT)}; } section { diff --git a/packages/app/src/components/browser/trace-player-controls.ts b/packages/app/src/components/browser/trace-player-controls.ts new file mode 100644 index 00000000..a882364d --- /dev/null +++ b/packages/app/src/components/browser/trace-player-controls.ts @@ -0,0 +1,137 @@ +import { Element } from '@core/element' +import { html, css, type TemplateResult } from 'lit' +import { customElement, state } from 'lit/decorators.js' + +import { emit, KBD } from '../../controller/keyboard.js' +import { + PLAYER_RESTART_EVENT, + PLAYER_SPEED_EVENT, + PLAYER_STATE_EVENT, + SPEEDS, + type PlayerState +} from './trace-timeline-constants.js' +import { formatTimecode } from './trace-timeline-utils.js' + +import '~icons/mdi/play.js' +import '~icons/mdi/pause.js' +import '~icons/mdi/skip-previous.js' +import '~icons/mdi/skip-next.js' +import '~icons/mdi/restart.js' + +const COMPONENT = 'wdio-devtools-trace-player-controls' + +/** Playback controls bar; drives the timeline via window events and mirrors its broadcast state. */ +@customElement(COMPONENT) +export class TracePlayerControls extends Element { + @state() playerState: PlayerState = { + currentMs: 0, + duration: 0, + playing: false, + speed: 1 + } + + static styles = [ + ...Element.styles, + css` + :host { + display: flex; + align-items: center; + background-color: var(--vscode-editor-background); + color: var(--vscode-foreground); + } + ` + ] + + connectedCallback(): void { + super.connectedCallback() + window.addEventListener(PLAYER_STATE_EVENT, this.#onState) + } + + disconnectedCallback(): void { + super.disconnectedCallback() + window.removeEventListener(PLAYER_STATE_EVENT, this.#onState) + } + + #onState = (event: Event): void => { + this.playerState = (event as CustomEvent<PlayerState>).detail + } + + #button( + title: string, + icon: TemplateResult, + onClick: () => void, + extra = '' + ): TemplateResult { + return html`<button + class="p-1 hover:bg-toolbarHoverBackground rounded ${extra}" + title="${title}" + @click="${onClick}" + > + ${icon} + </button>` + } + + #renderSpeedSelect(speed: number): TemplateResult { + return html` + <select + class="ml-1 bg-sideBarBackground border border-panelBorder rounded px-1 py-0.5" + title="Playback speed" + @change="${(event: Event) => + emit(PLAYER_SPEED_EVENT, { + value: Number((event.target as HTMLSelectElement).value) + })}" + > + ${SPEEDS.map( + (value) => + html`<option value="${value}" ?selected="${value === speed}"> + ${value}× + </option>` + )} + </select> + ` + } + + render() { + const { currentMs, duration, playing, speed } = this.playerState + return html` + <div class="flex items-center gap-1 px-2 w-full text-[12px]"> + <code class="tabular-nums text-chartsYellow" + >${formatTimecode(currentMs)}</code + > + <span class="opacity-60">/</span> + <code class="tabular-nums opacity-80">${formatTimecode(duration)}</code> + <span class="ml-auto"></span> + ${this.#button( + 'Restart', + html`<icon-mdi-restart></icon-mdi-restart>`, + () => emit(PLAYER_RESTART_EVENT) + )} + ${this.#button( + 'Previous action', + html`<icon-mdi-skip-previous></icon-mdi-skip-previous>`, + () => emit(KBD.step, { dir: -1 }) + )} + ${this.#button( + playing ? 'Pause' : 'Play', + playing + ? html`<icon-mdi-pause></icon-mdi-pause>` + : html`<icon-mdi-play></icon-mdi-play>`, + () => emit(KBD.togglePlay), + 'text-chartsBlue' + )} + ${this.#button( + 'Next action', + html`<icon-mdi-skip-next></icon-mdi-skip-next>`, + () => emit(KBD.step, { dir: 1 }) + )} + ${this.#renderSpeedSelect(speed)} + </div> + ` + } +} + +declare global { + interface HTMLElementTagNameMap { + [COMPONENT]: TracePlayerControls + } +} diff --git a/packages/app/src/components/browser/trace-timeline-constants.ts b/packages/app/src/components/browser/trace-timeline-constants.ts index d88f1c61..9f057994 100644 --- a/packages/app/src/components/browser/trace-timeline-constants.ts +++ b/packages/app/src/components/browser/trace-timeline-constants.ts @@ -1,19 +1,23 @@ -import type { ActionCategory } from '../workbench/actionItems/category.js' - /** Playback speed multipliers offered in the timeline controls. */ export const SPEEDS = [0.5, 1, 2, 3, 5] -/** Width of the track-label gutter (px) — lanes start after it. */ -export const GUTTER = 80 +/** Candidate ruler intervals (ms); tickStep picks the smallest fitting one. */ +export const TICK_STEPS = [ + 100, 250, 500, 1_000, 2_000, 5_000, 10_000, 15_000, 30_000, 60_000, 120_000, + 300_000, 600_000 +] + +/** Ruler divisions to aim for — keeps labels readable at any duration. */ +export const TICK_TARGET_DIVISIONS = 14 -/** Right breathing room (px) so end-of-timeline markers don't hug the edge. */ -export const INSET = 14 +/** Window events linking the controls bar and the timeline strip (KBD-style). */ +export const PLAYER_STATE_EVENT = 'trace-player:state' +export const PLAYER_RESTART_EVENT = 'trace-player:restart' +export const PLAYER_SPEED_EVENT = 'trace-player:speed' -/** Tailwind background class per action category, for the timeline chips. */ -export const CATEGORY_BG: Record<ActionCategory, string> = { - navigation: 'bg-chartsBlue', - input: 'bg-chartsPurple', - assertion: 'bg-chartsGreen', - query: 'bg-chartsYellow', - other: 'bg-gray-500' +export interface PlayerState { + currentMs: number + duration: number + playing: boolean + speed: number } diff --git a/packages/app/src/components/browser/trace-timeline-styles.ts b/packages/app/src/components/browser/trace-timeline-styles.ts index f2e276ad..f1482adc 100644 --- a/packages/app/src/components/browser/trace-timeline-styles.ts +++ b/packages/app/src/components/browser/trace-timeline-styles.ts @@ -1,7 +1,6 @@ import { css } from 'lit' -/** Styles for the trace-player timeline: host layout, hidden scrollbars, and - * the network-detail drawer. Detail-block styles come from networkStyles. */ +/** Host layout for the trace-player timeline strip. */ export const timelineStyles = css` :host { position: relative; @@ -12,53 +11,4 @@ export const timelineStyles = css` background-color: var(--vscode-editor-background); color: var(--vscode-foreground); } - .no-scrollbar { - scrollbar-width: none; - -ms-overflow-style: none; - } - .no-scrollbar::-webkit-scrollbar { - display: none; - } - .net-drawer { - position: absolute; - left: 0; - right: 0; - bottom: 0; - max-height: 62%; - display: flex; - flex-direction: column; - background: var(--vscode-sideBar-background); - border-top: 1px solid var(--accent, #ff7a3c); - box-shadow: 0 -16px 40px -24px #000; - z-index: 30; - } - .net-drawer-head { - display: flex; - align-items: center; - gap: 8px; - padding: 6px 10px; - border-bottom: 1px solid var(--vscode-panel-border); - font-size: 12px; - } - .net-drawer-head .url { - font-family: monospace; - font-size: 11.5px; - overflow: hidden; - text-overflow: ellipsis; - white-space: nowrap; - opacity: 0.85; - } - .net-drawer-head .close { - margin-left: auto; - cursor: pointer; - padding: 2px 6px; - border-radius: 4px; - } - .net-drawer-head .close:hover { - background: var(--vscode-toolbar-hoverBackground); - } - .net-drawer-body { - overflow: auto; - padding: 4px 0; - } ` diff --git a/packages/app/src/components/browser/trace-timeline-utils.ts b/packages/app/src/components/browser/trace-timeline-utils.ts index 1083ee65..ef99a363 100644 --- a/packages/app/src/components/browser/trace-timeline-utils.ts +++ b/packages/app/src/components/browser/trace-timeline-utils.ts @@ -1,9 +1,37 @@ +import { + TICK_STEPS, + TICK_TARGET_DIVISIONS +} from './trace-timeline-constants.js' + /** Detect image mime from a base64 string's magic bytes — trace screenshots * may be PNG (polling capture) or JPEG (CDP), and the zip names both `.jpeg`. */ export function imageMime(base64: string): string { return base64.startsWith('/9j/') ? 'image/jpeg' : 'image/png' } +export function tickStep( + durationMs: number, + targetTicks = TICK_TARGET_DIVISIONS +): number { + const raw = durationMs / targetTicks + return ( + TICK_STEPS.find((step) => step >= raw) ?? TICK_STEPS[TICK_STEPS.length - 1] + ) +} + +/** Ruler tick label: `500ms`, `3.5s`, `1:15`. */ +export function formatTickLabel(ms: number): string { + if (ms < 1_000) { + return `${ms}ms` + } + if (ms < 60_000) { + return `${(ms / 1_000).toFixed(1)}s` + } + const minutes = Math.floor(ms / 60_000) + const seconds = Math.round((ms % 60_000) / 1_000) + return `${minutes}:${String(seconds).padStart(2, '0')}` +} + /** `m:ss.cc` timecode (e.g. 32_270ms → `0:32.27`). */ export function formatTimecode(ms: number): string { const safe = Number.isFinite(ms) && ms > 0 ? ms : 0 diff --git a/packages/app/src/components/browser/trace-timeline.ts b/packages/app/src/components/browser/trace-timeline.ts index c2a62d52..07cb81a1 100644 --- a/packages/app/src/components/browser/trace-timeline.ts +++ b/packages/app/src/components/browser/trace-timeline.ts @@ -1,42 +1,30 @@ import { Element } from '@core/element' -import { html, nothing, type TemplateResult } from 'lit' +import { html, type TemplateResult } from 'lit' import { customElement, state, query } from 'lit/decorators.js' import { consume } from '@lit/context' import type { CommandLog, TracePlayerFrame } from '@wdio/devtools-shared' -import { - commandContext, - framesContext, - networkRequestContext -} from '../../controller/context.js' -import { commandCategory } from '../workbench/actionItems/category.js' -import { activeTimestampAt } from '../workbench/active-entry.js' -import { networkStyles } from '../workbench/network/styles.js' -import { renderNetworkRequestDetail } from '../workbench/network/request-detail.js' +import { commandContext, framesContext } from '../../controller/context.js' +import { activeSpanAt } from '../workbench/active-entry.js' import { KBD } from '../../controller/keyboard.js' import { - CATEGORY_BG, - GUTTER, - INSET, - SPEEDS + PLAYER_RESTART_EVENT, + PLAYER_SPEED_EVENT, + PLAYER_STATE_EVENT, + SPEEDS, + type PlayerState } from './trace-timeline-constants.js' -import { formatTimecode, imageMime } from './trace-timeline-utils.js' +import { + formatTickLabel, + formatTimecode, + imageMime, + tickStep +} from './trace-timeline-utils.js' import { timelineStyles } from './trace-timeline-styles.js' -import '~icons/mdi/play.js' -import '~icons/mdi/pause.js' -import '~icons/mdi/skip-previous.js' -import '~icons/mdi/skip-next.js' -import '~icons/mdi/restart.js' - const COMPONENT = 'wdio-devtools-trace-timeline' -/** - * Trace-player timeline (replaces the workbench dock in `pnpm show-trace` - * mode). Owns the playback clock, the screenshot filmstrip, the per-track - * timeline (actions / network / console), and the playhead. Advancing the - * clock dispatches `show-command` so the reused browser pane swaps screenshots. - */ +/** Player timeline strip: owns the playback clock, filmstrip, and playhead; wired to the controls bar and keyboard via window events, and drives the workbench via `show-command`. */ @customElement(COMPONENT) export class TraceTimeline extends Element { @consume({ context: commandContext, subscribe: true }) @@ -47,10 +35,6 @@ export class TraceTimeline extends Element { @state() frames: TracePlayerFrame[] = [] - @consume({ context: networkRequestContext, subscribe: true }) - @state() - networkRequests: NetworkRequest[] = [] - /** Playback position in ms relative to the recording start. */ @state() currentMs = 0 @state() playing = false @@ -58,17 +42,14 @@ export class TraceTimeline extends Element { #rafId?: number #rafLast = 0 - #activeTimestamp?: number + #activeCommand?: CommandLog #started = false - @query('[data-lanes]') lanesEl?: HTMLElement + @query('[data-scrub]') scrubEl?: HTMLElement #dragging = false - /** Network request whose detail drawer is open, or undefined. */ - @state() selectedRequest?: NetworkRequest - - static styles = [...Element.styles, networkStyles, timelineStyles] + static styles = [...Element.styles, timelineStyles] connectedCallback(): void { super.connectedCallback() @@ -76,6 +57,8 @@ export class TraceTimeline extends Element { window.addEventListener(KBD.step, this.#onKbdStep) window.addEventListener(KBD.jump, this.#onKbdJump) window.addEventListener(KBD.speed, this.#onKbdSpeed) + window.addEventListener(PLAYER_RESTART_EVENT, this.#onRestartEvent) + window.addEventListener(PLAYER_SPEED_EVENT, this.#onSpeedEvent) } disconnectedCallback(): void { @@ -87,6 +70,13 @@ export class TraceTimeline extends Element { window.removeEventListener(KBD.step, this.#onKbdStep) window.removeEventListener(KBD.jump, this.#onKbdJump) window.removeEventListener(KBD.speed, this.#onKbdSpeed) + window.removeEventListener(PLAYER_RESTART_EVENT, this.#onRestartEvent) + window.removeEventListener(PLAYER_SPEED_EVENT, this.#onSpeedEvent) + } + + #onRestartEvent = (): void => this.#restart() + #onSpeedEvent = (event: Event): void => { + this.speed = (event as CustomEvent<{ value: number }>).detail.value } #onKbdTogglePlay = (): void => this.#togglePlay() @@ -152,6 +142,17 @@ export class TraceTimeline extends Element { if (!this.#started && this.commands.length) { this.#syncActiveCommand() } + // Mirror playback state to the controls bar on the tab-header line. + window.dispatchEvent( + new CustomEvent<PlayerState>(PLAYER_STATE_EVENT, { + detail: { + currentMs: this.currentMs, + duration: this.#duration, + playing: this.playing, + speed: this.speed + } + }) + ) } #stopRaf(): void { @@ -228,38 +229,25 @@ export class TraceTimeline extends Element { return } const clock = this.#start + this.currentMs - const timestamps = sorted.map((c) => c.timestamp ?? 0) - const activeTs = activeTimestampAt(timestamps, clock) ?? timestamps[0] - if (this.#started && activeTs === this.#activeTimestamp) { + const command = activeSpanAt(sorted, clock) ?? sorted[0] + if (this.#started && command === this.#activeCommand) { return } this.#started = true - this.#activeTimestamp = activeTs - const command = sorted.find((c) => (c.timestamp ?? 0) === activeTs) - if (command) { - window.dispatchEvent( - new CustomEvent('show-command', { detail: { command } }) - ) - } - } - - #onSpeedChange(event: Event): void { - this.speed = Number((event.target as HTMLSelectElement).value) + this.#activeCommand = command + window.dispatchEvent( + new CustomEvent('show-command', { detail: { command } }) + ) } - // ─── scrubbing (free-flow playhead drag) ─────────────────────────────────── + // ─── scrubbing (drag anywhere on the strip) ─────────────────────────────── #fractionFromClientX(clientX: number): number { - const rect = this.lanesEl?.getBoundingClientRect() - if (!rect) { - return 0 - } - const laneStart = rect.left + GUTTER - const laneWidth = rect.width - GUTTER - INSET - if (laneWidth <= 0) { + const rect = this.scrubEl?.getBoundingClientRect() + if (!rect || rect.width <= 0) { return 0 } - return Math.min(1, Math.max(0, (clientX - laneStart) / laneWidth)) + return Math.min(1, Math.max(0, (clientX - rect.left) / rect.width)) } #onPointerDown = (event: PointerEvent): void => { @@ -290,72 +278,6 @@ export class TraceTimeline extends Element { // ─── render ─────────────────────────────────────────────────────────────── - #ctrlButton( - title: string, - icon: TemplateResult, - onClick: () => void, - extra = '' - ): TemplateResult { - return html`<button - class="p-1 hover:bg-toolbarHoverBackground rounded ${extra}" - title="${title}" - @click="${onClick}" - > - ${icon} - </button>` - } - - #renderControls(): TemplateResult { - return html` - <div - class="flex items-center gap-1 px-2 h-9 border-b border-panelBorder flex-none text-[12px]" - > - ${this.#ctrlButton( - 'Restart', - html`<icon-mdi-restart></icon-mdi-restart>`, - () => this.#restart() - )} - ${this.#ctrlButton( - 'Previous action', - html`<icon-mdi-skip-previous></icon-mdi-skip-previous>`, - () => this.#step(-1) - )} - ${this.#ctrlButton( - this.playing ? 'Pause' : 'Play', - this.playing - ? html`<icon-mdi-pause></icon-mdi-pause>` - : html`<icon-mdi-play></icon-mdi-play>`, - () => this.#togglePlay(), - 'text-chartsBlue' - )} - ${this.#ctrlButton( - 'Next action', - html`<icon-mdi-skip-next></icon-mdi-skip-next>`, - () => this.#step(1) - )} - <code class="ml-2 tabular-nums text-chartsYellow" - >${formatTimecode(this.currentMs)}</code - > - <span class="opacity-60">/</span> - <code class="tabular-nums opacity-80" - >${formatTimecode(this.#duration)}</code - > - <select - class="ml-auto bg-sideBarBackground border border-panelBorder rounded px-1 py-0.5" - title="Playback speed" - @change="${this.#onSpeedChange}" - > - ${SPEEDS.map( - (speed) => - html`<option value="${speed}" ?selected="${speed === this.speed}"> - ${speed}× - </option>` - )} - </select> - </div> - ` - } - /** Timestamp of the frame nearest the playhead — drives filmstrip highlight. */ get #activeFrameTimestamp(): number | undefined { const clock = this.#start + this.currentMs @@ -371,167 +293,113 @@ export class TraceTimeline extends Element { return best } - // CSS left for a marker inside a track body (which starts after the gutter), - // leaving INSET of right margin so end-of-timeline markers don't hug the edge. - #laneLeft(fraction: number): string { - return `calc(${fraction} * (100% - ${INSET}px))` + get #ticks(): number[] { + const step = tickStep(this.#duration) + const out: number[] = [] + for (let t = step; t < this.#duration; t += step) { + out.push(t) + } + return out } - #renderFilmstrip(): TemplateResult { - if (!this.frames.length) { - return html`<div - class="flex-none h-16 border-b border-panelBorder flex items-center justify-center text-[11px] opacity-50" - > - No frames captured - </div>` - } - const activeFrame = this.#activeFrameTimestamp - return html` - <div - class="flex-none h-16 border-b border-panelBorder flex items-stretch" - > - <div class="flex-none w-20 border-r border-panelBorder"></div> - <div - class="no-scrollbar flex-1 min-w-0 flex items-stretch gap-1 px-1 py-1 overflow-x-auto" - > - ${this.frames.map( - (frame) => - html`<button - class="h-full aspect-video flex-none border rounded overflow-hidden hover:border-chartsBlue ${frame.timestamp === - activeFrame - ? 'border-chartsBlue ring-1 ring-chartsBlue' - : 'border-panelBorder'}" - title="${formatTimecode(frame.timestamp - this.#start)}" - @click="${() => this.#seekToTimestamp(frame.timestamp)}" - > - <img - class="h-full w-full object-cover" - src="data:${imageMime( - frame.screenshot - )};base64,${frame.screenshot}" - /> - </button>` - )} - </div> - </div> - ` + // Faint vertical gridlines at each ruler tick, spanning the whole strip. + #renderGridlines(): TemplateResult { + return html`${this.#ticks.map( + (tick) => + html`<div + class="absolute top-0 bottom-0 w-px bg-panelBorder/60 pointer-events-none" + style="left:${(tick / this.#duration) * 100}%;" + ></div>` + )}` } - #renderTrack( - label: string, - body: TemplateResult | typeof nothing - ): TemplateResult { + // Ruler labels stay inside the strip via the bounded translateX trick. + #renderRulerLabels(): TemplateResult { return html` - <div class="flex items-stretch h-7 border-b border-panelBorder/50"> - <div - class="flex-none w-20 px-2 flex items-center text-[11px] opacity-60 border-r border-panelBorder" - > - ${label} - </div> - <div class="relative flex-1 overflow-hidden">${body}</div> + <div class="relative h-5 flex-none text-[10px] opacity-70"> + ${this.#ticks.map((tick) => { + const fraction = tick / this.#duration + return html`<span + class="absolute top-0.5 whitespace-nowrap" + style="left:${fraction * 100}%; transform:translateX(-${fraction * + 100}%);" + >${formatTickLabel(tick)}</span + >` + })} </div> ` } - #renderActionsTrack(): TemplateResult { - const body = html`${this.#sortedCommands.map((command) => { - const ts = command.timestamp ?? 0 - const fraction = this.#fraction(ts) - const active = ts === this.#activeTimestamp - const color = CATEGORY_BG[commandCategory(command.command)] - // Track chips stay compact with the short command name; the full - // Playwright label is the hover tooltip (and the left Actions list). - return html`<button - class="absolute top-1 bottom-1 ${color} rounded-sm px-1 text-[10px] leading-none text-black/80 whitespace-nowrap max-w-[140px] overflow-hidden text-ellipsis ${active - ? 'ring-1 ring-white' - : ''}" - style="left:${this.#laneLeft( - fraction - )}; transform:translateX(-${fraction * 100}%);" - title="${command.title ?? command.command}" - @click="${(event: MouseEvent) => { - event.stopPropagation() - this.#seekToTimestamp(ts) - }}" - > - ${command.command} - </button>` - })}` - return this.#renderTrack('Actions', body) - } - - #renderNetworkTrack(): TemplateResult { - if (!this.networkRequests.length) { - return this.#renderTrack('Network', nothing) - } - const body = html`${this.networkRequests.map((request) => { - const leftFr = this.#fraction(request.startTime) - const rawFr = Math.max(0.004, (request.time ?? 0) / this.#duration) - const widthFr = Math.min(rawFr, 1 - leftFr) - const selected = this.selectedRequest?.id === request.id - // stopPropagation so a click selects the request rather than scrubbing the - // playhead (the lanes container owns the pointerdown drag handler). + // Thumbnails sit at their wall-clock position along the axis. + #renderThumbTrack(): TemplateResult { + if (!this.frames.length) { return html`<div - class="absolute top-2 bottom-2 rounded-sm cursor-pointer ${selected - ? 'bg-chartsBlue ring-1 ring-white' - : 'bg-chartsBlue/60 hover:bg-chartsBlue'}" - style="left:${this.#laneLeft(leftFr)}; width:${this.#laneLeft( - widthFr - )}; min-width:3px;" - title="${request.method} ${request.url}" - @pointerdown="${(e: PointerEvent) => e.stopPropagation()}" - @click="${(e: MouseEvent) => { - e.stopPropagation() - this.selectedRequest = selected ? undefined : request - }}" - ></div>` - })}` - return this.#renderTrack('Network', body) - } - - #renderNetworkDrawer(): TemplateResult | typeof nothing { - const req = this.selectedRequest - if (!req) { - return nothing + class="flex-1 min-h-0 flex items-center justify-center text-[11px] opacity-50" + > + No frames captured + </div>` } + const activeFrame = this.#activeFrameTimestamp return html` - <div class="net-drawer"> - <div class="net-drawer-head"> - <span class="url" title="${req.url}">${req.method} ${req.url}</span> - <span - class="close" - title="Close" - @click="${() => (this.selectedRequest = undefined)}" - >✕</span + <div class="relative flex-1 min-h-0"> + ${this.frames.map((frame) => { + const fraction = this.#fraction(frame.timestamp) + const active = frame.timestamp === activeFrame + return html`<button + class="absolute top-0.5 bottom-0.5 aspect-video border rounded overflow-hidden hover:border-chartsBlue hover:z-10 ${active + ? 'border-chartsBlue ring-1 ring-chartsBlue z-10' + : 'border-panelBorder'}" + style="left:${fraction * 100}%; transform:translateX(-${fraction * + 100}%);" + title="${formatTimecode(frame.timestamp - this.#start)}" + @click="${() => this.#seekToTimestamp(frame.timestamp)}" > - </div> - <div class="net-drawer-body">${renderNetworkRequestDetail(req)}</div> + <img + class="h-full w-full object-cover" + src="data:${imageMime( + frame.screenshot + )};base64,${frame.screenshot}" + /> + </button>` + })} </div> ` } - #renderPlayhead(): TemplateResult { + // Bottom scrub bar: full-width line, action tick marks, draggable knob. + #renderScrubBar(): TemplateResult { const fraction = Math.min(1, Math.max(0, this.currentMs / this.#duration)) - // Anchored at the gutter and inset on the right so it tracks the same lane - // coordinates as the action/network markers. - return html`<div - class="absolute top-0 bottom-0 w-0.5 bg-chartsRed z-20 pointer-events-none" - style="left:calc(${GUTTER}px + ${fraction} * (100% - ${GUTTER}px - ${INSET}px));" - ></div>` + return html` + <div class="relative h-6 flex-none"> + <div + class="absolute left-0 right-0 top-1/2 -translate-y-1/2 h-0.5 bg-chartsBlue/50 rounded" + ></div> + ${this.#sortedCommands.map((command) => { + const tickFraction = this.#fraction(command.timestamp ?? 0) + return html`<div + class="absolute top-1/2 -translate-y-1/2 h-2.5 w-px bg-white/80" + style="left:${tickFraction * 100}%;" + title="${command.title ?? command.command}" + ></div>` + })} + <div + class="absolute top-1/2 -translate-y-1/2 w-3 h-3 rounded-full bg-chartsBlue ring-1 ring-white/70 pointer-events-none" + style="left:calc(${fraction * 100}% - 6px);" + ></div> + </div> + ` } render() { return html` - ${this.#renderControls()} ${this.#renderFilmstrip()} <div - data-lanes - class="relative flex-1 min-h-0 overflow-y-auto overflow-x-hidden cursor-ew-resize select-none" + data-scrub + class="relative flex-1 min-h-0 flex flex-col cursor-ew-resize select-none" @pointerdown="${this.#onPointerDown}" > - ${this.#renderActionsTrack()} ${this.#renderNetworkTrack()} - ${this.#renderTrack('Console', nothing)} ${this.#renderPlayhead()} + ${this.#renderGridlines()} ${this.#renderRulerLabels()} + ${this.#renderThumbTrack()} ${this.#renderScrubBar()} </div> - ${this.#renderNetworkDrawer()} ` } } diff --git a/packages/app/src/components/sidebar/test-suite.ts b/packages/app/src/components/sidebar/test-suite.ts index cf0d43f3..9cc7e0ee 100644 --- a/packages/app/src/components/sidebar/test-suite.ts +++ b/packages/app/src/components/sidebar/test-suite.ts @@ -287,10 +287,11 @@ export class ExplorerTestEntry extends CollapseableEntry { return this.state === TestState.RUNNING } get testStateIcon() { - // Fixed-height box (= the label's line-height) centred so the icon aligns - // with the first line of the title whether it wraps or not — no margin hacks. + // Vertically centred in the row so the status icon lines up with the + // row's action buttons (run/stop), which are also centre-aligned. const box = (inner: unknown) => - html`<span class="w-4 h-[18px] shrink-0 flex items-center justify-center" + html`<span + class="w-4 h-[18px] shrink-0 self-center grid place-items-center" >${inner}</span >` if (this.isRunning) { @@ -407,7 +408,9 @@ export class ExplorerTestEntry extends CollapseableEntry { class="row flex w-full items-start text-sm group/sidebar rounded-md my-0.5 px-1 py-1 cursor-pointer hover:bg-toolbarHoverBackground" > <button - class="flex-none pointer px-2 h-8 ${hasNoChildren ? 'hidden' : ''}" + class="flex-none pointer px-2 h-[18px] flex items-center justify-center ${hasNoChildren + ? 'hidden' + : ''}" @click="${() => this.#toggleEntry()}" > <icon-mdi-menu-down diff --git a/packages/app/src/components/tabs.ts b/packages/app/src/components/tabs.ts index ccda011a..316c6bb8 100644 --- a/packages/app/src/components/tabs.ts +++ b/packages/app/src/components/tabs.ts @@ -2,6 +2,9 @@ import { Element } from '@core/element' import { html, css, nothing } from 'lit' import { customElement, property } from 'lit/decorators.js' +/** Badge colour variant; `danger` tints the count red (e.g. the Errors tab). */ +type BadgeTone = 'default' | 'danger' + const TABS_COMPONENT = 'wdio-devtools-tabs' @customElement(TABS_COMPONENT) export class DevtoolsTabs extends Element { @@ -50,15 +53,27 @@ export class DevtoolsTabs extends Element { ); color: var(--vscode-descriptionForeground); } + .tab-badge--danger { + background: color-mix( + in srgb, + var(--vscode-charts-red) 18%, + transparent + ); + color: var(--vscode-charts-red); + } ` ] #getTabButton(tabId: string) { const tabElement = this.tabs.find( (el) => el.getAttribute('label') === tabId - ) - const badge = (tabElement as { badge?: number } | undefined)?.badge + ) as { badge?: number; badgeTone?: BadgeTone } | undefined + const badge = tabElement?.badge const showBadge = badge && badge > 0 + const badgeClass = + tabElement?.badgeTone === 'danger' + ? 'tab-badge tab-badge--danger' + : 'tab-badge' return html` <button @@ -69,7 +84,9 @@ export class DevtoolsTabs extends Element { : 'border-transparent'}" > <span>${tabId}</span> - ${showBadge ? html`<span class="tab-badge">${badge}</span>` : nothing} + ${showBadge + ? html`<span class="${badgeClass}">${badge}</span>` + : nothing} </button> ` } @@ -197,6 +214,9 @@ export class DevtoolsTab extends Element { @property({ type: Number }) badge?: number + @property({ type: String }) + badgeTone?: BadgeTone + static styles = [ ...Element.styles, css` diff --git a/packages/app/src/components/workbench.ts b/packages/app/src/components/workbench.ts index 5b87cd9d..a97b20c9 100644 --- a/packages/app/src/components/workbench.ts +++ b/packages/app/src/components/workbench.ts @@ -6,10 +6,19 @@ import { consume } from '@lit/context' import { DragController, Direction } from '../utils/DragController.js' import { consoleLogContext, + metadataContext, networkRequestContext, - baselineContext + baselineContext, + commandContext, + suiteContext } from '../controller/context.js' -import type { PreservedAttempt } from '@wdio/devtools-shared' +import type { + CommandLog, + Metadata, + PreservedAttempt +} from '@wdio/devtools-shared' +import type { SuiteStatsFragment } from '../controller/types.js' +import { collectErrors } from './workbench/errors/collect.js' import '~icons/mdi/arrow-collapse-down.js' import '~icons/mdi/arrow-collapse-up.js' @@ -23,26 +32,41 @@ import './workbench/logs.js' import './workbench/console.js' import './workbench/metadata.js' import './workbench/network.js' +import './workbench/errors.js' import './workbench/compare.js' import './browser/snapshot.js' import './browser/trace-timeline.js' +import './browser/trace-player-controls.js' import { + BROWSER_BACKDROP_GRADIENT, + HEADER_HEIGHT, MIN_WORKBENCH_HEIGHT, MIN_METATAB_WIDTH, ACTIONS_DEFAULT_WIDTH, BROWSER_HEIGHT_RATIO, - RERENDER_TIMEOUT + PLAYER_CONTROLS_HEIGHT, + PLAYER_DOCK_DEFAULT_HEIGHT, + PLAYER_DOCK_MIN_HEIGHT, + PLAYER_SNAPSHOT_WIDTH_RATIO, + RERENDER_TIMEOUT, + TRACE_TIMELINE_MIN_HEIGHT, + TRACE_TIMELINE_DEFAULT_HEIGHT } from '../controller/constants.js' const COMPONENT = 'wdio-devtools-workbench' + +/** Pixel value from a DragController position string (`flex-basis: 123px`). */ +function basisPx(position: string): number | undefined { + const value = parseFloat(position.split(':')[1] ?? '') + return Number.isFinite(value) ? value : undefined +} @customElement(COMPONENT) export class DevtoolsWorkbench extends Element { #toolbarCollapsed = localStorage.getItem('toolbar') === 'true' #workbenchSidebarCollapsed = localStorage.getItem('workbenchSidebar') === 'true' - // Trace-player mode (`pnpm show-trace`): hide the Metadata tab and swap the - // workbench tabs for the timeline player. + // Trace-player mode: full workbench plus the timeline strip and controls bar. @property({ type: Boolean }) playerMode = false @@ -58,6 +82,18 @@ export class DevtoolsWorkbench extends Element { @state() baselines: Map<string, PreservedAttempt> | undefined = undefined + @consume({ context: commandContext, subscribe: true }) + @state() + commands: CommandLog[] | undefined = undefined + + @consume({ context: suiteContext, subscribe: true }) + @state() + suites: Record<string, SuiteStatsFragment>[] | undefined = undefined + + @consume({ context: metadataContext, subscribe: true }) + @state() + metadata: Metadata | undefined = undefined + static styles = [ ...Element.styles, css` @@ -99,6 +135,54 @@ export class DevtoolsWorkbench extends Element { direction: Direction.horizontal }) + #dragTimeline = new DragController(this, { + localStorageKey: 'traceTimelineHeight', + minPosition: TRACE_TIMELINE_MIN_HEIGHT, + maxPosition: () => window.innerHeight * 0.4, + initialPosition: TRACE_TIMELINE_DEFAULT_HEIGHT, + getContainerEl: () => this.#getVerticalWindow(), + direction: Direction.vertical + }) + + // Player-mode pane height; own storage key so it never disturbs the live split. + // The live max bound keeps the handle (and pane) inside the current budget. + #dragVerticalPlayer = new DragController(this, { + localStorageKey: 'playerPaneHeight', + minPosition: MIN_WORKBENCH_HEIGHT, + maxPosition: () => this.#playerPaneBudget(), + initialPosition: Math.max( + MIN_WORKBENCH_HEIGHT, + window.innerHeight - + HEADER_HEIGHT - + PLAYER_CONTROLS_HEIGHT - + TRACE_TIMELINE_DEFAULT_HEIGHT - + PLAYER_DOCK_DEFAULT_HEIGHT + ), + getContainerEl: () => this.#getVerticalWindow(), + direction: Direction.vertical + }) + + // Player snapshot keeps the recorded viewport's shape, slightly narrowed. + #playerAspectRatio(): string { + const viewport = this.metadata?.viewport + const width = Math.round( + (viewport?.width || 1280) * PLAYER_SNAPSHOT_WIDTH_RATIO + ) + return `${width} / ${viewport?.height || 800}` + } + + // Space left for the snapshot pane once the fixed rows and dock minimum eat theirs. + #playerPaneBudget(): number { + return Math.max( + MIN_WORKBENCH_HEIGHT, + window.innerHeight - + HEADER_HEIGHT - + PLAYER_CONTROLS_HEIGHT - + PLAYER_DOCK_MIN_HEIGHT - + this.#timelinePaneHeight() + ) + } + async #getHorizontalWindow() { await this.updateComplete return this.horizontalResizerWindow as Element @@ -144,13 +228,34 @@ export class DevtoolsWorkbench extends Element { if (this.#toolbarCollapsed) { return '' } - const m = this.#dragVertical.getPosition().match(/(\d+(?:\.\d+)?)px/) - const raw = m ? parseFloat(m[1]) : window.innerHeight * BROWSER_HEIGHT_RATIO + if (this.playerMode) { + // Snapshot pane dominates; the CSS clamp keeps the dock minimum in view. + // Literal getPosition() basis lets adjustPosition sync slider ↔ clamped height. + const maxHeight = `calc(100vh - ${ + HEADER_HEIGHT + PLAYER_CONTROLS_HEIGHT + PLAYER_DOCK_MIN_HEIGHT + }px - ${this.#timelinePaneHeight()}px)` + return `flex-grow:0; flex-shrink:0; ${this.#dragVerticalPlayer.getPosition()}; max-height:${maxHeight}; min-height:${MIN_WORKBENCH_HEIGHT}px;` + } + const raw = + basisPx(this.#dragVertical.getPosition()) ?? + window.innerHeight * BROWSER_HEIGHT_RATIO const capped = Math.min(raw, window.innerHeight * 0.7) const paneHeight = Math.max(MIN_WORKBENCH_HEIGHT, capped) return `flex:0 0 ${paneHeight}px; height:${paneHeight}px; max-height:70vh; min-height:0;` } + #timelinePaneHeight(): number { + const raw = + basisPx(this.#dragTimeline.getPosition()) ?? TRACE_TIMELINE_DEFAULT_HEIGHT + const capped = Math.min(raw, window.innerHeight * 0.4) + return Math.max(TRACE_TIMELINE_MIN_HEIGHT, capped) + } + + #computeTimelinePaneStyle(): string { + const paneHeight = this.#timelinePaneHeight() + return `flex:0 0 ${paneHeight}px; height:${paneHeight}px; max-height:40vh; min-height:0;` + } + #computeSidebarStyle(): string { if (this.#workbenchSidebarCollapsed) { return 'width:0; flex:0 0 0; overflow:hidden;' @@ -173,11 +278,9 @@ export class DevtoolsWorkbench extends Element { <wdio-devtools-tab label="Actions"> <wdio-devtools-actions></wdio-devtools-actions> </wdio-devtools-tab> - ${this.playerMode - ? nothing - : html`<wdio-devtools-tab label="Metadata"> - <wdio-devtools-metadata></wdio-devtools-metadata> - </wdio-devtools-tab>`} + <wdio-devtools-tab label="Metadata"> + <wdio-devtools-metadata></wdio-devtools-metadata> + </wdio-devtools-tab> <nav class="ml-auto" slot="actions"> <button @click="${() => this.#toggle('workbenchSidebar')}" @@ -231,6 +334,44 @@ export class DevtoolsWorkbench extends Element { ` } + #errorCount(): number { + return collectErrors(this.commands, this.suites).length + } + + // Dock tab list — extracted so #renderWorkbenchTabs stays under the size cap. + #renderDockTabItems() { + return html` + <wdio-devtools-tab label="Source"> + <wdio-devtools-source></wdio-devtools-source> + </wdio-devtools-tab> + <wdio-devtools-tab label="Log"> + <wdio-devtools-logs></wdio-devtools-logs> + </wdio-devtools-tab> + <wdio-devtools-tab + label="Console" + .badge="${this.consoleLogs?.length || 0}" + > + <wdio-devtools-console-logs + id="console-logs-tab" + ></wdio-devtools-console-logs> + </wdio-devtools-tab> + <wdio-devtools-tab + label="Network" + .badge="${this.networkRequests?.length || 0}" + > + <wdio-devtools-network></wdio-devtools-network> + </wdio-devtools-tab> + <wdio-devtools-tab + label="Errors" + badgeTone="danger" + .badge="${this.#errorCount()}" + > + <wdio-devtools-errors></wdio-devtools-errors> + </wdio-devtools-tab> + ${this.#renderCompareTabIfAvailable()} + ` + } + #renderWorkbenchTabs() { return html` <wdio-devtools-tabs @@ -240,27 +381,7 @@ export class DevtoolsWorkbench extends Element { ? 'hidden' : ''} flex-1 min-h-0" > - <wdio-devtools-tab label="Source"> - <wdio-devtools-source></wdio-devtools-source> - </wdio-devtools-tab> - <wdio-devtools-tab label="Log"> - <wdio-devtools-logs></wdio-devtools-logs> - </wdio-devtools-tab> - <wdio-devtools-tab - label="Console" - .badge="${this.consoleLogs?.length || 0}" - > - <wdio-devtools-console-logs - id="console-logs-tab" - ></wdio-devtools-console-logs> - </wdio-devtools-tab> - <wdio-devtools-tab - label="Network" - .badge="${this.networkRequests?.length || 0}" - > - <wdio-devtools-network></wdio-devtools-network> - </wdio-devtools-tab> - ${this.#renderCompareTabIfAvailable()} + ${this.#renderDockTabItems()} <nav class="ml-auto" slot="actions"> <button @click="${() => this.#toggle('toolbar')}" @@ -276,11 +397,51 @@ export class DevtoolsWorkbench extends Element { ` } + #renderBrowserPane() { + // Player: the boxed host goes transparent and the pane carries the shared + // backdrop, so the aspect box blends instead of showing a gradient seam. + const playerPaneExtra = this.playerMode + ? ` background:${BROWSER_BACKDROP_GRADIENT};` + : '' + return html` + <section + class="basis-auto text-gray-500 flex items-center justify-center flex-1 min-h-0" + style="${this.#computeBrowserPaneStyle()}${playerPaneExtra}" + > + ${this.playerMode + ? html`<div + class="h-full max-w-full mx-auto" + style="aspect-ratio:${this.#playerAspectRatio()};" + > + <wdio-devtools-browser + style="background:transparent" + ></wdio-devtools-browser> + </div>` + : html`<wdio-devtools-browser></wdio-devtools-browser>`} + </section> + ` + } + + // Full-width playback strip above the workbench row — player mode only. + #renderTimelineStrip() { + if (!this.playerMode) { + return nothing + } + return html` + <wdio-devtools-trace-timeline + class="relative z-10 flex-none border-b-[1px] border-b-panelBorder" + style="${this.#computeTimelinePaneStyle()}" + ></wdio-devtools-trace-timeline> + ${this.#dragTimeline.getSlider('z-[999] pointer-events-auto')} + ` + } + render() { return html` + ${this.#renderTimelineStrip()} <section data-horizontal-resizer-window - class="flex relative w-full h-full min-h-0 overflow-hidden" + class="flex relative w-full flex-1 min-h-0 overflow-hidden" > <section data-sidebar @@ -297,20 +458,23 @@ export class DevtoolsWorkbench extends Element { data-vertical-resizer-window class="relative flex flex-col flex-grow min-w-0 min-h-0 overflow-hidden" > + ${this.playerMode + ? html`<wdio-devtools-trace-player-controls + class="flex-none h-10 border-b-[1px] border-b-panelBorder" + ></wdio-devtools-trace-player-controls>` + : nothing} <section - class="basis-auto text-gray-500 flex items-center justify-center flex-1 min-h-0" - style="${this.#computeBrowserPaneStyle()}" + class="relative flex flex-col flex-1 min-w-0 min-h-0 overflow-hidden" > - <wdio-devtools-browser></wdio-devtools-browser> + ${this.#renderBrowserPane()} + ${!this.#toolbarCollapsed + ? (this.playerMode + ? this.#dragVerticalPlayer + : this.#dragVertical + ).getSlider('z-[999] pointer-events-auto') + : nothing} + ${this.#renderWorkbenchTabs()} </section> - ${!this.#toolbarCollapsed - ? this.#dragVertical.getSlider('z-[999] pointer-events-auto') - : nothing} - ${this.playerMode - ? html`<wdio-devtools-trace-timeline - class="relative z-10 border-t-[1px] border-t-panelBorder flex-1 min-h-0" - ></wdio-devtools-trace-timeline>` - : this.#renderWorkbenchTabs()} </section> </section> ` diff --git a/packages/app/src/components/workbench/action-tree.ts b/packages/app/src/components/workbench/action-tree.ts new file mode 100644 index 00000000..723f83af --- /dev/null +++ b/packages/app/src/components/workbench/action-tree.ts @@ -0,0 +1,72 @@ +// Pure helpers behind the player's collapsible action tree: flattening the +// group tree into render rows and deciding which groups start expanded. + +import type { + TraceActionChild, + TraceActionGroupNode +} from '@wdio/devtools-shared' + +export interface GroupRow { + kind: 'group' + group: TraceActionGroupNode + depth: number + expanded: boolean +} + +export interface CommandRow { + kind: 'command' + commandIndex: number + depth: number +} + +export type ActionTreeRow = GroupRow | CommandRow + +/** Command indices anywhere under a group, nested groups included. */ +export function collectCommandIndices(group: TraceActionGroupNode): number[] { + const indices: number[] = [] + for (const child of group.children) { + if ('group' in child) { + indices.push(...collectCommandIndices(child.group)) + } else { + indices.push(child.commandIndex) + } + } + return indices +} + +/** Groups open by default when failed or when holding the active command. */ +export function defaultExpanded( + group: TraceActionGroupNode, + activeCommandIndex?: number +): boolean { + if (group.failed) { + return true + } + return ( + activeCommandIndex !== undefined && + collectCommandIndices(group).includes(activeCommandIndex) + ) +} + +/** Flatten the tree into render rows, descending only into expanded groups. */ +export function flattenActionTree( + children: TraceActionChild[], + isExpanded: (group: TraceActionGroupNode) => boolean, + depth = 0 +): ActionTreeRow[] { + const rows: ActionTreeRow[] = [] + for (const child of children) { + if ('group' in child) { + const expanded = isExpanded(child.group) + rows.push({ kind: 'group', group: child.group, depth, expanded }) + if (expanded) { + rows.push( + ...flattenActionTree(child.group.children, isExpanded, depth + 1) + ) + } + } else { + rows.push({ kind: 'command', commandIndex: child.commandIndex, depth }) + } + } + return rows +} diff --git a/packages/app/src/components/workbench/actionItems/command.ts b/packages/app/src/components/workbench/actionItems/command.ts index a587369e..1a596f56 100644 --- a/packages/app/src/components/workbench/actionItems/command.ts +++ b/packages/app/src/components/workbench/actionItems/command.ts @@ -33,6 +33,10 @@ export class CommandItem extends ActionItem { @property({ type: Object, attribute: true }) entry?: CommandLog + willUpdate(): void { + this.failed = Boolean(this.entry?.error) + } + #highlightLine() { const event = new CustomEvent('show-command', { detail: { @@ -85,7 +89,10 @@ export class CommandItem extends ActionItem { @click="${() => this.#highlightLine()}" > ${this.iconChip(this.#renderIcon(entry.command))} - <code class="text-[12.5px] flex-wrap text-left break-all" + <code + class="text-[12.5px] flex-wrap text-left break-all ${this.failed + ? 'text-chartsRed' + : ''}" >${entry.title ?? entry.command}</code > ${this.renderTime()} diff --git a/packages/app/src/components/workbench/actionItems/duration.ts b/packages/app/src/components/workbench/actionItems/duration.ts index db4b7fbc..f0dd5d63 100644 --- a/packages/app/src/components/workbench/actionItems/duration.ts +++ b/packages/app/src/components/workbench/actionItems/duration.ts @@ -1,19 +1,23 @@ +import type { CommandLog, TraceMutation } from '@wdio/devtools-shared' + export type DurationHeat = 'fast' | 'mid' | 'slow' const ONE_SECOND = 1000 const ONE_MINUTE = ONE_SECOND * 60 -/** Human-readable duration: `ms` under a second, `s` under a minute, `m s` above. */ +/** Human-readable duration: `ms` under a second, `s` under a minute, `m s` + * above. Rounds first — reconstructed traces carry fractional-ms clocks. */ export function formatDuration(ms: number): string { - if (ms > ONE_MINUTE) { - const minutes = Math.floor(ms / ONE_MINUTE) - const seconds = Math.floor((ms - minutes * ONE_MINUTE) / ONE_SECOND) + const rounded = Math.round(ms) + if (rounded > ONE_MINUTE) { + const minutes = Math.floor(rounded / ONE_MINUTE) + const seconds = Math.floor((rounded - minutes * ONE_MINUTE) / ONE_SECOND) return `${minutes}m ${seconds}s` } - if (ms > ONE_SECOND) { - return `${(ms / ONE_SECOND).toFixed(2)}s` + if (rounded > ONE_SECOND) { + return `${(rounded / ONE_SECOND).toFixed(2)}s` } - return `${ms}ms` + return `${rounded}ms` } /** Bucket a step duration so slow steps stand out: fast < 500ms ≤ mid < 2s ≤ slow. */ @@ -27,6 +31,24 @@ export function durationHeat(ms: number): DurationHeat { return 'fast' } +/** + * True per-action duration: a command's own execution span (`timestamp − + * startTime`) when it has one, else the inter-action `gapFallback`. Prefer the + * span — the gap over-counts idle time before an action, so e.g. an assertion + * whose internal polling commands are suppressed would otherwise report the + * navigation gap that preceded it rather than its own runtime. Used by both the + * flat (live) and grouped (trace-player) action views so they agree. + */ +export function entryDuration( + entry: CommandLog | TraceMutation, + gapFallback: number | undefined +): number | undefined { + if ('command' in entry && entry.startTime !== undefined) { + return entry.timestamp - entry.startTime + } + return gapFallback +} + /** * Per-step duration for each timeline entry: the gap to the next entry. The * final entry has no next, so it falls back to the gap from the previous entry diff --git a/packages/app/src/components/workbench/actionItems/group.ts b/packages/app/src/components/workbench/actionItems/group.ts new file mode 100644 index 00000000..10780076 --- /dev/null +++ b/packages/app/src/components/workbench/actionItems/group.ts @@ -0,0 +1,69 @@ +import { html } from 'lit' +import { customElement, property } from 'lit/decorators.js' + +import type { TraceActionGroupNode } from '@wdio/devtools-shared' + +import { ActionItem } from './item.js' +import '~icons/mdi/chevron-right.js' + +const SOURCE_COMPONENT = 'wdio-devtools-group-item' + +/** Collapsible step/group row of the trace player's action tree. */ +@customElement(SOURCE_COMPONENT) +export class GroupItem extends ActionItem { + @property({ type: Object }) + group?: TraceActionGroupNode + + /** Whether the group's children are currently rendered below it. */ + @property({ type: Boolean, reflect: true }) + expanded = false + + willUpdate(): void { + this.failed = Boolean(this.group?.failed) + this.duration = this.group + ? this.group.endTime - this.group.startTime + : undefined + } + + #toggle() { + this.dispatchEvent( + new CustomEvent('group-toggle', { + detail: { callId: this.group?.callId, expanded: this.expanded }, + bubbles: true, + composed: true + }) + ) + } + + render() { + if (!this.group) { + return + } + return html` + <button + class="flex px-1 py-0.5 w-full items-center hover:bg-toolbarHoverBackground" + @click="${() => this.#toggle()}" + > + <icon-mdi-chevron-right + class="w-[14px] h-[14px] block shrink-0 mx-1 transition-transform ${this + .expanded + ? 'rotate-90' + : ''}" + ></icon-mdi-chevron-right> + <span + class="text-[12.5px] font-medium text-left break-all ${this.failed + ? 'text-chartsRed' + : ''}" + >${this.group.title}</span + > + ${this.renderTime()} + </button> + ` + } +} + +declare global { + interface HTMLElementTagNameMap { + [SOURCE_COMPONENT]: GroupItem + } +} diff --git a/packages/app/src/components/workbench/actionItems/item.ts b/packages/app/src/components/workbench/actionItems/item.ts index 931a2299..945a79db 100644 --- a/packages/app/src/components/workbench/actionItems/item.ts +++ b/packages/app/src/components/workbench/actionItems/item.ts @@ -29,6 +29,10 @@ export class ActionItem extends Element { @property({ type: Boolean, reflect: true }) active = false + /** Whether this row's action errored — drives the red row treatment. */ + @property({ type: Boolean, reflect: true }) + failed = false + static styles = [ ...Element.styles, css` @@ -67,6 +71,19 @@ export class ActionItem extends Element { :host([active]) .ic { border-color: var(--accent); } + :host([failed]) button { + background: color-mix( + in srgb, + var(--vscode-charts-red) 8%, + transparent + ); + box-shadow: inset 2px 0 0 var(--vscode-charts-red); + } + :host([failed][active]) button { + box-shadow: + inset 2px 0 0 var(--vscode-charts-red), + inset 0 0 0 1px var(--vscode-panel-border); + } ` ] diff --git a/packages/app/src/components/workbench/actions.ts b/packages/app/src/components/workbench/actions.ts index 86f4e856..5e630c8e 100644 --- a/packages/app/src/components/workbench/actions.ts +++ b/packages/app/src/components/workbench/actions.ts @@ -1,21 +1,38 @@ import { Element } from '@core/element' -import { html, css } from 'lit' +import { html, css, nothing } from 'lit' import { customElement, state } from 'lit/decorators.js' import { consume } from '@lit/context' -import type { CommandLog } from '@wdio/devtools-shared' -import { mutationContext, commandContext } from '../../controller/context.js' +import type { + CommandLog, + TraceActionChild, + TraceActionGroupNode +} from '@wdio/devtools-shared' +import { + mutationContext, + commandContext, + actionGroupsContext +} from '../../controller/context.js' import '../placeholder.js' import './actionItems/command.js' +import './actionItems/group.js' import './actionItems/mutation.js' -import { stepDurations } from './actionItems/duration.js' -import { activeTimestampAt } from './active-entry.js' +import { entryDuration, stepDurations } from './actionItems/duration.js' +import { activeSpanAt } from './active-entry.js' +import { + defaultExpanded, + flattenActionTree, + type ActionTreeRow +} from './action-tree.js' type TimelineEntry = TraceMutation | CommandLog const SOURCE_COMPONENT = 'wdio-devtools-actions' +/** Horizontal shift per tree depth level in the player's action tree. */ +const TREE_INDENT_PX = 14 + @customElement(SOURCE_COMPONENT) export class DevtoolsActions extends Element { static styles = [ @@ -47,6 +64,11 @@ export class DevtoolsActions extends Element { background: var(--vscode-panel-border); pointer-events: none; } + + /* Tree mode indents rows, so the straight rail no longer lines up. */ + .timeline.tree::before { + display: none; + } ` ] @@ -56,12 +78,32 @@ export class DevtoolsActions extends Element { @consume({ context: commandContext, subscribe: true }) commands: CommandLog[] = [] + @consume({ context: actionGroupsContext, subscribe: true }) + groups?: TraceActionChild[] + // The selected timeline row, tracked by object reference — timestamps aren't // unique (commands logged in the same millisecond would all match), so // reference identity is what highlights exactly one row. @state() private activeEntry?: TimelineEntry + // User chevron toggles, by group callId; unset groups follow the default + // (failed or containing the active command → open). + @state() + private expandOverrides: ReadonlyMap<string, boolean> = new Map() + + #onGroupToggle = (event: Event) => { + const { callId, expanded } = ( + event as CustomEvent<{ callId?: string; expanded: boolean }> + ).detail + if (!callId) { + return + } + const next = new Map(this.expandOverrides) + next.set(callId, !expanded) + this.expandOverrides = next + } + #onShowCommand = (event: Event) => { const command = (event as CustomEvent<{ command?: CommandLog }>).detail ?.command @@ -86,12 +128,7 @@ export class DevtoolsActions extends Element { // every timeupdate tick. #onScreencastProgress = (event: Event) => { const { time } = (event as CustomEvent<{ time: number }>).detail - const entries = this.#sortedEntries() - const timestamp = activeTimestampAt( - entries.map((entry) => entry.timestamp), - time - ) - const active = entries.find((entry) => entry.timestamp === timestamp) + const active = activeSpanAt(this.#sortedEntries(), time) if (active === this.activeEntry) { return } @@ -146,7 +183,62 @@ export class DevtoolsActions extends Element { } } + // Player tree mode: group rows expand/collapse; leaf rows are the same + // command items as the flat list, indented under their group. + #renderTree(rootChildren: TraceActionChild[]) { + const commands = this.commands || [] + const activeIndex = + this.activeEntry && 'command' in this.activeEntry + ? commands.indexOf(this.activeEntry) + : -1 + const isExpanded = (group: TraceActionGroupNode) => + this.expandOverrides.get(group.callId) ?? + defaultExpanded(group, activeIndex >= 0 ? activeIndex : undefined) + const rows = flattenActionTree(rootChildren, isExpanded) + const baseline = commands[0]?.timestamp ?? 0 + const gaps = stepDurations(commands.map((command) => command.timestamp)) + return html`<div class="timeline tree" @group-toggle=${this.#onGroupToggle}> + ${rows.map((row) => this.#renderTreeRow(row, commands, baseline, gaps))} + </div>` + } + + #renderTreeRow( + row: ActionTreeRow, + commands: CommandLog[], + baseline: number, + gaps: Array<number | undefined> + ) { + const indent = `padding-left: ${row.depth * TREE_INDENT_PX}px` + if (row.kind === 'group') { + return html` + <wdio-devtools-group-item + style=${indent} + .group=${row.group} + ?expanded=${row.expanded} + ></wdio-devtools-group-item> + ` + } + const entry = commands[row.commandIndex] + if (!entry) { + return nothing + } + // Reconstructed zips carry the real invocation span; gap is the fallback. + const duration = entryDuration(entry, gaps[row.commandIndex]) + return html` + <wdio-devtools-command-item + style=${indent} + elapsedTime=${entry.timestamp - baseline} + .duration=${duration} + .entry=${entry} + ?active=${entry === this.activeEntry} + ></wdio-devtools-command-item> + ` + } + render() { + if (this.groups?.length) { + return this.#renderTree(this.groups) + } const entries = this.#sortedEntries() if (!entries.length) { @@ -157,7 +249,7 @@ export class DevtoolsActions extends Element { const rows = entries.map((entry, index) => { const elapsedTime = entry.timestamp - baselineTimestamp - const duration = durations[index] + const duration = entryDuration(entry, durations[index]) const active = entry === this.activeEntry if ('command' in entry) { diff --git a/packages/app/src/components/workbench/active-entry.ts b/packages/app/src/components/workbench/active-entry.ts index 3cdd2ed2..edbe3eb9 100644 --- a/packages/app/src/components/workbench/active-entry.ts +++ b/packages/app/src/components/workbench/active-entry.ts @@ -1,20 +1,51 @@ +/** Timeline item occupying a span from `startTime` (inclusive) to `timestamp` + * (inclusive). Items without a `startTime` occupy the single point `timestamp`. */ +export interface TimeSpanned { + timestamp: number + startTime?: number +} + /** - * Given action timestamps sorted ascending and a playback time (wall-clock ms), - * return the timestamp of the action that is "current" — the latest one at or - * before `time`. Returns undefined when `time` precedes the first action, so - * nothing is highlighted before the first command runs. + * Given timeline items and a playback time (wall-clock ms), return the item + * that is "current" at `time`. An item's span is `[startTime ?? timestamp, + * timestamp]` and the item is current when `time` falls inside it — so a + * long-running command (e.g. a polling `expect.*` matcher) stays selected for + * its whole duration, not just at completion. When several spans contain `time` + * (nested/overlapping commands), the one that started most recently wins, with + * ties broken toward the tighter span — the most specific action in progress. + * When none contain `time` (a point-like command, or a gap between actions), + * fall back to the latest item that has already ended at or before `time`. + * Returns undefined when `time` precedes every item, so nothing is highlighted + * before the first action runs. */ -export function activeTimestampAt( - sortedTimestamps: number[], +export function activeSpanAt<T extends TimeSpanned>( + items: readonly T[], time: number -): number | undefined { - let active: number | undefined - for (const ts of sortedTimestamps) { - if (ts <= time) { - active = ts - } else { - break +): T | undefined { + let containing: T | undefined + let containingStart = -Infinity + let containingEnd = Infinity + let ended: T | undefined + let endedAt = -Infinity + + for (const item of items) { + const end = item.timestamp + const start = item.startTime ?? end + if ( + start <= time && + time <= end && + (start > containingStart || + (start === containingStart && end < containingEnd)) + ) { + containing = item + containingStart = start + containingEnd = end + } + if (end <= time && end >= endedAt) { + ended = item + endedAt = end } } - return active + + return containing ?? ended } diff --git a/packages/app/src/components/workbench/call-source.ts b/packages/app/src/components/workbench/call-source.ts index 9349b184..3fa2d5ca 100644 --- a/packages/app/src/components/workbench/call-source.ts +++ b/packages/app/src/components/workbench/call-source.ts @@ -23,6 +23,51 @@ export function parseCallSource( } } +/** Path with any trailing `:line` / `:line:column` suffix stripped — some + * recorded traces glue the line onto the file, so lookups compare clean paths. */ +export function normalizeSourcePath(path: string): string { + const match = path.match(/:\d+:\d+$/) || path.match(/:\d+$/) + return match && match.index ? path.slice(0, match.index) : path +} + +/** Key in `sources` holding `file`'s content — exact match first, then a + * normalized-path match so suffixed keys and clean queries still pair up. */ +export function resolveSourceFile( + sources: Record<string, string>, + file: string +): string | undefined { + if (file in sources) { + return file + } + const target = normalizeSourcePath(file) + return Object.keys(sources).find((key) => normalizeSourcePath(key) === target) +} + +/** Normalized display list: every captured source file plus any file referenced + * by a command call source, deduplicated by clean path. */ +export function listSourceFiles( + sources: Record<string, string>, + callSources: (string | undefined)[] +): string[] { + const files: string[] = [] + const seen = new Set<string>() + const add = (path: string) => { + const normalized = normalizeSourcePath(path) + if (!seen.has(normalized)) { + seen.add(normalized) + files.push(normalized) + } + } + Object.keys(sources).forEach(add) + for (const callSource of callSources) { + const parsed = callSource ? parseCallSource(callSource) : null + if (parsed) { + add(parsed.file) + } + } + return files +} + /** Last path segment, handling both POSIX (`/`) and Windows (`\`) separators. */ export function fileBasename(path: string): string { const segments = path.split(/[/\\]/) diff --git a/packages/app/src/components/workbench/errors.ts b/packages/app/src/components/workbench/errors.ts new file mode 100644 index 00000000..f4470005 --- /dev/null +++ b/packages/app/src/components/workbench/errors.ts @@ -0,0 +1,211 @@ +import { Element } from '@core/element' +import { html, css, nothing, type TemplateResult } from 'lit' +import { customElement, state } from 'lit/decorators.js' +import { consume } from '@lit/context' + +import type { CommandLog } from '@wdio/devtools-shared' +import { commandContext, suiteContext } from '../../controller/context.js' +import type { SuiteStatsFragment } from '../../controller/types.js' +import { collectErrors, type CollectedError } from './errors/collect.js' + +const COMPONENT = 'wdio-devtools-errors' + +/** Last three path segments of a `file:line:col` source, so the label stays + * short (`step-definitions/steps.ts:31:3`) instead of an absolute path. */ +function shortSource(callSource: string): string { + return callSource.split(/[\\/]/).slice(-3).join('/') +} + +@customElement(COMPONENT) +export class DevtoolsErrors extends Element { + @consume({ context: commandContext, subscribe: true }) + @state() + commands: CommandLog[] | undefined = undefined + + @consume({ context: suiteContext, subscribe: true }) + @state() + suites: Record<string, SuiteStatsFragment>[] | undefined = undefined + + static styles = [ + ...Element.styles, + css` + :host { + display: flex; + flex: 1; + flex-direction: column; + min-height: 0; + overflow-y: auto; + background-color: var(--vscode-editor-background); + color: var(--vscode-foreground); + } + + .error-entry { + border-bottom: 1px solid var(--vscode-panel-border); + border-left: 2px solid var(--vscode-charts-red); + padding: 12px 16px; + display: flex; + flex-direction: column; + gap: 8px; + } + + /* Clickable source anchor (@ path:line) — the affordance that opens the + Source tab at the exact line. Styled as a link, not selected text. */ + .error-loc { + align-self: flex-start; + font-family: var(--vscode-editor-font-family); + font-size: 12px; + font-weight: 600; + color: var(--accent); + background: none; + border: none; + padding: 0; + cursor: pointer; + text-decoration: underline; + text-decoration-style: dotted; + text-underline-offset: 3px; + } + .error-loc:hover { + text-decoration-style: solid; + } + .error-loc:focus-visible { + outline: 1px solid var(--accent); + outline-offset: 2px; + border-radius: 2px; + } + + .error-title { + font-size: 12.5px; + font-weight: 600; + color: var(--vscode-charts-red); + word-break: break-word; + white-space: pre-wrap; + } + + /* Aligned key/value block, like the reference viewer's Expected/Received. */ + .error-diff { + font-family: var(--vscode-editor-font-family); + font-size: 11.5px; + display: grid; + grid-template-columns: max-content 1fr; + gap: 3px 12px; + white-space: pre-wrap; + word-break: break-word; + } + .error-diff .label { + color: var(--vscode-descriptionForeground); + } + .error-diff .expected { + color: var(--vscode-charts-green, #3fb950); + } + .error-diff .received { + color: var(--vscode-charts-red); + } + + .error-stack { + margin: 2px 0 0; + } + .error-stack summary { + font-size: 11px; + color: var(--vscode-descriptionForeground); + cursor: pointer; + user-select: none; + } + .error-stack pre { + font-family: var(--vscode-editor-font-family); + font-size: 11px; + white-space: pre-wrap; + word-break: break-word; + color: var(--vscode-descriptionForeground); + max-height: 200px; + overflow: auto; + margin: 6px 0 0; + } + + .empty-state { + flex: 1; + display: flex; + flex-direction: column; + align-items: center; + justify-content: center; + gap: 10px; + color: var(--vscode-descriptionForeground); + } + .empty-state-icon { + font-size: 40px; + opacity: 0.3; + } + .empty-state-text { + font-size: 14px; + opacity: 0.6; + } + ` + ] + + // Only the source anchor is interactive — it opens the Source tab at the exact + // line (app-source-highlight activates the tab + scrolls). The entry itself has + // no click action. + #openSource(callSource: string) { + window.dispatchEvent( + new CustomEvent('app-source-highlight', { detail: callSource }) + ) + } + + #renderDiff(error: CollectedError): TemplateResult | typeof nothing { + if (error.expected === undefined && error.actual === undefined) { + return nothing + } + return html`<div class="error-diff"> + ${error.expected !== undefined + ? html`<span class="label">Expected</span + ><span class="expected">${error.expected}</span>` + : nothing} + ${error.actual !== undefined + ? html`<span class="label">Received</span + ><span class="received">${error.actual}</span>` + : nothing} + </div>` + } + + #renderEntry(error: CollectedError): TemplateResult { + return html` + <div class="error-entry"> + ${error.callSource + ? html`<button + class="error-loc" + title="Open source at this line" + @click="${() => this.#openSource(error.callSource!)}" + > + @${shortSource(error.callSource)} + </button>` + : nothing} + <div class="error-title">${error.message}</div> + ${this.#renderDiff(error)} + ${error.stack + ? html`<details class="error-stack"> + <summary>Stack</summary> + <pre>${error.stack}</pre> + </details>` + : nothing} + </div> + ` + } + + render() { + const errors = collectErrors(this.commands, this.suites) + if (!errors.length) { + return html` + <div class="empty-state"> + <div class="empty-state-icon">✓</div> + <div class="empty-state-text">No errors</div> + </div> + ` + } + return html`${errors.map((error) => this.#renderEntry(error))}` + } +} + +declare global { + interface HTMLElementTagNameMap { + [COMPONENT]: DevtoolsErrors + } +} diff --git a/packages/app/src/components/workbench/errors/collect.ts b/packages/app/src/components/workbench/errors/collect.ts new file mode 100644 index 00000000..ca7429e1 --- /dev/null +++ b/packages/app/src/components/workbench/errors/collect.ts @@ -0,0 +1,232 @@ +/** + * Pure error-collection for the workbench Errors tab. Merges failed commands + * (from `commandContext`) with failed tests (from `suiteContext`) into a single + * ordered, de-duplicated list. Kept framework-free and side-effect-free so the + * tab component only has to render what this returns. + */ + +import type { CommandLog } from '@wdio/devtools-shared' +import type { + SuiteStatsFragment, + TestStatsFragment +} from '../../../controller/types.js' +import { stripAnsi } from '../console-filter.js' + +/** One row in the Errors tab. */ +export interface CollectedError { + /** Failing action/step or test title — the row heading. */ + title: string + /** Error message shown message-first, monospace. */ + message: string + /** Optional stack, rendered under the message when present. */ + stack?: string + /** `file:line:col` source anchor for the "open source" link. */ + callSource?: string + /** The failing command, when the error came from one — lets the tab dispatch + * `show-command` to select and scroll to that action. */ + command?: CommandLog + /** Command timestamp; drives ordering and the `show-command` elapsed time. */ + timestamp?: number + /** Assertion expected value, rendered as a labelled row when present. */ + expected?: string + /** Assertion received/actual value, rendered as a labelled row when present. */ + actual?: string +} + +const ASSERTION_COMMAND_RE = /^(expect|assert|verify)\./ + +/** Display string for an expected/actual value that may already be serialized. */ +function displayValue(value: unknown): string | undefined { + if (value === undefined || value === null) { + return undefined + } + if (typeof value === 'string') { + return value + } + try { + return JSON.stringify(value) + } catch { + return String(value) + } +} + +/** Assertion commands carry `[actual, expected]` in args (see the exporter's + * Assert params + synthesizeExpectFailure). */ +function assertionValues(command: CommandLog): { + actual?: string + expected?: string +} { + if (!ASSERTION_COMMAND_RE.test(command.command) || command.args.length < 2) { + return {} + } + return { + actual: displayValue(command.args[0]), + expected: displayValue(command.args[1]) + } +} + +interface ReadableError { + message?: string + name?: string + stack?: string + expected?: unknown + actual?: unknown +} + +/** Split trailing `at …` stack-frame lines off the message body. */ +function splitStack(clean: string): { body: string; stack?: string } { + const lines = clean.split('\n') + const idx = lines.findIndex((line) => /^\s*at\s/.test(line)) + if (idx === -1) { + return { body: clean.trimEnd() } + } + return { + body: lines.slice(0, idx).join('\n').trimEnd(), + stack: lines.slice(idx).join('\n').trim() + } +} + +/** Trim each line and drop blanks — assertion libraries indent continuation + * lines, which would otherwise show as ragged whitespace. */ +function dedent(text: string): string { + return text + .split('\n') + .map((line) => line.trim()) + .filter(Boolean) + .join('\n') +} + +/** Pull `Expected:` / `Received:` values out of a matcher body and return the + * remaining headline. The labels may be indented (expect-webdriverio pads + * them), and `Received:` can span several lines up to the end of the body. */ +function extractDiff(body: string): { + headline: string + expected?: string + actual?: string +} { + const expected = body.match(/^[ \t]*Expected:[ \t]*(.*)$/m)?.[1]?.trim() + const receivedAt = body.search(/^[ \t]*Received:/m) + let actual: string | undefined + let headline = body + if (receivedAt !== -1) { + actual = dedent( + body.slice(receivedAt).replace(/^[ \t]*Received:[ \t]*/, '') + ) + headline = body.slice(0, receivedAt) + } + if (expected !== undefined) { + headline = headline.replace(/^[ \t]*Expected:[ \t]*.*$/m, '') + } + return { headline: dedent(headline), expected, actual } +} + +/** Clean, structured view of any error-ish value: ANSI stripped, stack split + * off the message, and assertion Expected/Received pulled into fields. */ +function readError(error: unknown): + | { + message: string + stack?: string + expected?: string + actual?: string + } + | undefined { + if (!error || typeof error !== 'object') { + return undefined + } + const e = error as ReadableError + const raw = e.message?.trim() || e.name?.trim() || '' + if (!raw && !e.stack) { + return undefined + } + const { body, stack } = splitStack(stripAnsi(raw)) + const diff = extractDiff(body) + return { + message: diff.headline || 'Error', + stack: e.stack ? stripAnsi(e.stack) : stack, + expected: diff.expected ?? displayValue(e.expected), + actual: diff.actual ?? displayValue(e.actual) + } +} + +/** Failed leaf tests across every suite map, deduped by uid (last wins, matching + * the sidebar's root-suite dedup so we read the freshest fragment). */ +function collectFailedTests( + suites: Record<string, SuiteStatsFragment>[] | undefined +): TestStatsFragment[] { + const byUid = new Map<string, TestStatsFragment>() + const visit = (suite: SuiteStatsFragment) => { + for (const test of suite.tests ?? []) { + if (test.state === 'failed') { + byUid.set(test.uid, test) + } + } + for (const child of suite.suites ?? []) { + visit(child) + } + } + for (const map of suites ?? []) { + for (const suite of Object.values(map)) { + visit(suite) + } + } + return [...byUid.values()] +} + +function commandErrors(commands: CommandLog[] | undefined): CollectedError[] { + return (commands ?? []) + .flatMap((command) => { + const read = readError(command.error) + if (!read) { + return [] + } + const values = assertionValues(command) + return [ + { + title: command.title ?? command.command, + message: read.message, + stack: read.stack, + callSource: command.callSource, + command, + timestamp: command.timestamp, + expected: values.expected ?? read.expected, + actual: values.actual ?? read.actual + } + ] + }) + .sort((a, b) => (a.timestamp ?? 0) - (b.timestamp ?? 0)) +} + +/** + * Build the Errors-tab list from the live/player contexts. + * + * Command failures come first (time-ordered) because they carry the clickable + * action; a failed test that only echoes a command's message is dropped so the + * same failure isn't listed twice (e.g. a Cucumber `Then` fails as both the + * assertion command and the scenario). + */ +export function collectErrors( + commands: CommandLog[] | undefined, + suites: Record<string, SuiteStatsFragment>[] | undefined +): CollectedError[] { + const fromCommands = commandErrors(commands) + const seenMessages = new Set(fromCommands.map((e) => e.message)) + + const fromTests = collectFailedTests(suites).flatMap((test) => { + const read = readError(test.error ?? test.errors?.[0]) + if (!read || seenMessages.has(read.message)) { + return [] + } + return [ + { + title: test.fullTitle || test.title || test.uid, + message: read.message, + stack: read.stack, + callSource: test.callSource, + expected: read.expected, + actual: read.actual + } + ] + }) + + return [...fromCommands, ...fromTests] +} diff --git a/packages/app/src/components/workbench/source.ts b/packages/app/src/components/workbench/source.ts index 81a12d0b..5b336bda 100644 --- a/packages/app/src/components/workbench/source.ts +++ b/packages/app/src/components/workbench/source.ts @@ -13,7 +13,14 @@ import type { CommandLog } from '@wdio/devtools-shared' import { sourceContext, commandContext } from '../../controller/context.js' import { commandCategory, type ActionCategory } from './actionItems/category.js' -import { parseCallSource, fileBasename, pathSegments } from './call-source.js' +import { + parseCallSource, + fileBasename, + pathSegments, + normalizeSourcePath, + resolveSourceFile, + listSourceFiles +} from './call-source.js' import { sourceStyles } from './source/styles.js' import '../placeholder.js' @@ -95,12 +102,24 @@ export class DevtoolsSource extends Element { return document.body.classList.contains('dark') } - /** File to show: an explicit selection/call-site, else the first available. */ + /** Captured files plus files referenced by command call sources (clean paths). */ + get #fileList(): string[] { + return listSourceFiles( + this.sources || {}, + (this.commands || []).map((c) => c.callSource) + ) + } + + /** File to show: an explicit selection/call-site wins even when it wasn't + * captured as a source (the toolbar then shows a not-captured state instead + * of silently falling back to a different file), else the first available. */ get #effectiveFile(): string | undefined { - if (this.activeFile && this.sources?.[this.activeFile]) { - return this.activeFile - } - return Object.keys(this.sources || {})[0] + return this.activeFile ?? this.#fileList[0] + } + + #contentFor(file: string): string | undefined { + const key = resolveSourceFile(this.sources || {}, file) + return key !== undefined ? this.sources[key] : undefined } connectedCallback(): void { @@ -145,8 +164,7 @@ export class DevtoolsSource extends Element { super.disconnectedCallback() window.removeEventListener('app-source-highlight', this.#onHighlight) window.removeEventListener('app-source-track', this.#onTrack) - this.#editorView?.destroy() - this.#editorView = undefined + this.#unmountEditor() this.#tabObserver?.disconnect() this.#tabObserver = undefined this.#themeObserver?.disconnect() @@ -158,6 +176,10 @@ export class DevtoolsSource extends Element { if (!target) { return } + if (this.#contentFor(target) === undefined) { + this.#unmountEditor() + return + } this.#mountEditor(target) this.#refreshCallSite() } @@ -167,10 +189,11 @@ export class DevtoolsSource extends Element { if (!parsed) { return } - this.activeFile = parsed.file - this.callSiteFile = parsed.file + const file = normalizeSourcePath(parsed.file) + this.activeFile = file + this.callSiteFile = file this.callSiteLine = parsed.line - const cmd = this.#commandAt(parsed.file, parsed.line) + const cmd = this.#commandAt(file, parsed.line) this.callSiteCommand = cmd?.command this.callSiteCategory = cmd ? commandCategory(cmd.command) : 'other' if (activateTab) { @@ -184,7 +207,11 @@ export class DevtoolsSource extends Element { return false } const parsed = parseCallSource(c.callSource) - return parsed?.file === file && parsed.line === line + return ( + !!parsed && + normalizeSourcePath(parsed.file) === file && + parsed.line === line + ) }) } @@ -192,9 +219,15 @@ export class DevtoolsSource extends Element { this.activeFile = file } + #unmountEditor() { + this.#editorView?.destroy() + this.#editorView = undefined + this.#mountedFile = undefined + } + #mountEditor(filePath: string) { - const source = this.sources?.[filePath] - if (!source) { + const source = this.#contentFor(filePath) + if (source === undefined) { return } const container = @@ -256,7 +289,7 @@ export class DevtoolsSource extends Element { } #renderFileTabs(active: string) { - return Object.keys(this.sources || {}).map( + return this.#fileList.map( (file) => html`<button class="src-file ${file === active ? 'active' : ''}" @@ -331,9 +364,14 @@ export class DevtoolsSource extends Element { if (!active) { return html`<wdio-devtools-placeholder></wdio-devtools-placeholder>` } + const hasContent = this.#contentFor(active) !== undefined return html`<div class="source-root"> ${this.#renderToolbar(active)} - <div class="source-container"></div> + ${hasContent + ? html`<div class="source-container"></div>` + : html`<div class="src-empty"> + Source for ${fileBasename(active)} was not captured in this trace. + </div>`} </div>` } } diff --git a/packages/app/src/components/workbench/source/styles.ts b/packages/app/src/components/workbench/source/styles.ts index 9375400a..e40439d9 100644 --- a/packages/app/src/components/workbench/source/styles.ts +++ b/packages/app/src/components/workbench/source/styles.ts @@ -186,6 +186,19 @@ export const sourceStyles = css` overflow: hidden; } + .src-empty { + flex: 1; + display: flex; + align-items: center; + justify-content: center; + padding: 16px; + text-align: center; + font-family: var(--vscode-font-family, sans-serif); + font-size: 12px; + color: var(--vscode-descriptionForeground); + background: var(--vscode-editor-background); + } + .cm-editor { width: 100%; height: 100%; diff --git a/packages/app/src/controller/DataManager.ts b/packages/app/src/controller/DataManager.ts index a9a87e79..2af04a25 100644 --- a/packages/app/src/controller/DataManager.ts +++ b/packages/app/src/controller/DataManager.ts @@ -1,4 +1,4 @@ -import { ContextProvider } from '@lit/context' +import { ContextProvider, type Context, type ContextType } from '@lit/context' import type { ReactiveController, ReactiveControllerHost } from 'lit' import type { Metadata, @@ -21,7 +21,8 @@ import { hasConnectionContext, baselineContext, selectedTestUidContext, - framesContext + framesContext, + actionGroupsContext } from './context.js' import { BASELINE_WS_SCOPE, TRACE_API, WS_SCOPE } from '@wdio/devtools-shared' import { CACHE_ID } from './constants.js' @@ -64,60 +65,48 @@ export class DataManagerController implements ReactiveController { baselineContextProvider: ContextProvider<typeof baselineContext> selectedTestUidContextProvider: ContextProvider<typeof selectedTestUidContext> framesContextProvider: ContextProvider<typeof framesContext> + actionGroupsContextProvider: ContextProvider<typeof actionGroupsContext> #playerMode = false constructor(host: ReactiveControllerHost & HTMLElement) { ;(this.#host = host).addController(this) - this.mutationsContextProvider = new ContextProvider(this.#host, { - context: mutationContext, - initialValue: [] - }) - this.logsContextProvider = new ContextProvider(this.#host, { - context: logContext, - initialValue: [] - }) - this.consoleLogsContextProvider = new ContextProvider(this.#host, { - context: consoleLogContext, - initialValue: [] - }) - this.networkRequestsContextProvider = new ContextProvider(this.#host, { - context: networkRequestContext, - initialValue: [] - }) - this.metadataContextProvider = new ContextProvider(this.#host, { - context: metadataContext - }) - this.metadataBySessionContextProvider = new ContextProvider(this.#host, { - context: metadataBySessionContext, - initialValue: {} - }) - this.commandsContextProvider = new ContextProvider(this.#host, { - context: commandContext, - initialValue: [] - }) - this.sourcesContextProvider = new ContextProvider(this.#host, { - context: sourceContext - }) - this.suitesContextProvider = new ContextProvider(this.#host, { - context: suiteContext - }) - this.hasConnectionProvider = new ContextProvider(this.#host, { - context: hasConnectionContext, - initialValue: false - }) - this.baselineContextProvider = new ContextProvider(this.#host, { - context: baselineContext, - initialValue: new Map<string, PreservedAttempt>() - }) - this.selectedTestUidContextProvider = new ContextProvider(this.#host, { - context: selectedTestUidContext, - initialValue: undefined - }) - this.framesContextProvider = new ContextProvider(this.#host, { - context: framesContext, - initialValue: [] - }) + this.mutationsContextProvider = this.#provide(mutationContext, []) + this.logsContextProvider = this.#provide(logContext, []) + this.consoleLogsContextProvider = this.#provide(consoleLogContext, []) + this.networkRequestsContextProvider = this.#provide( + networkRequestContext, + [] + ) + this.metadataContextProvider = this.#provide(metadataContext) + this.metadataBySessionContextProvider = this.#provide( + metadataBySessionContext, + {} + ) + this.commandsContextProvider = this.#provide(commandContext, []) + this.sourcesContextProvider = this.#provide(sourceContext) + this.suitesContextProvider = this.#provide(suiteContext) + this.hasConnectionProvider = this.#provide(hasConnectionContext, false) + this.baselineContextProvider = this.#provide( + baselineContext, + new Map<string, PreservedAttempt>() + ) + this.selectedTestUidContextProvider = this.#provide( + selectedTestUidContext, + undefined + ) + this.framesContextProvider = this.#provide(framesContext, []) + this.actionGroupsContextProvider = this.#provide( + actionGroupsContext, + undefined + ) + } + + #provide<T extends Context<unknown, unknown>>( + context: T, + initialValue?: ContextType<T> + ): ContextProvider<T> { + return new ContextProvider(this.#host, { context, initialValue }) } get playerMode() { @@ -226,6 +215,7 @@ export class DataManagerController implements ReactiveController { loadPlayerData(data: TracePlayerData) { this.#playerMode = true this.framesContextProvider.setValue(data.frames) + this.actionGroupsContextProvider.setValue(data.groups) this.loadTraceFile(data.trace) this.hasConnectionProvider.setValue(true) this.#host.requestUpdate() diff --git a/packages/app/src/controller/constants.ts b/packages/app/src/controller/constants.ts index 05168031..6992ebf4 100644 --- a/packages/app/src/controller/constants.ts +++ b/packages/app/src/controller/constants.ts @@ -7,6 +7,20 @@ export const RERENDER_TIMEOUT = 10 export const SIDEBAR_DEFAULT_WIDTH = 350 export const ACTIONS_DEFAULT_WIDTH = 360 export const BROWSER_HEIGHT_RATIO = 1.4 / 2.4 +export const TRACE_TIMELINE_MIN_HEIGHT = 80 +export const TRACE_TIMELINE_DEFAULT_HEIGHT = 100 +/** Player-mode dock sizing — the browser pane takes the remaining space. */ +export const PLAYER_DOCK_MIN_HEIGHT = 140 +export const PLAYER_DOCK_DEFAULT_HEIGHT = 220 +/** Controls bar on the tab-header line in player mode (matches h-10 headers). */ +export const PLAYER_CONTROLS_HEIGHT = 40 +/** Width factor on the player snapshot's aspect box — trims width, keeps height. */ +export const PLAYER_SNAPSHOT_WIDTH_RATIO = 0.9 +/** Backdrop behind the browser chrome — shared with the snapshot component styles. */ +export const BROWSER_BACKDROP_GRADIENT = + 'radial-gradient(120% 120% at 50% 0%, var(--vscode-editorWidget-background), var(--vscode-editor-background))' +/** Fixed app-header height (see header.ts / app.ts `h-[calc(100%-40px)]`). */ +export const HEADER_HEIGHT = 40 export const LOG_ICONS: Record<string, string> = { log: '›', info: 'ⓘ', diff --git a/packages/app/src/controller/context.ts b/packages/app/src/controller/context.ts index 0bed9d26..151822ea 100644 --- a/packages/app/src/controller/context.ts +++ b/packages/app/src/controller/context.ts @@ -4,6 +4,7 @@ import type { MetadataBySession, CommandLog, PreservedAttempt, + TraceActionChild, TracePlayerFrame } from '@wdio/devtools-shared' import type { SuiteStatsFragment } from './types.js' @@ -51,3 +52,9 @@ export const selectedTestUidContext = createContext<string | undefined>( export const framesContext = createContext<TracePlayerFrame[]>( Symbol('framesContext') ) + +/** Root children of the trace player's action tree — populated only in player + * mode when the zip carried structural steps; absent means flat list. */ +export const actionGroupsContext = createContext< + TraceActionChild[] | undefined +>(Symbol('actionGroupsContext')) diff --git a/packages/app/src/controller/keyboard.ts b/packages/app/src/controller/keyboard.ts index 87559253..f94ce52c 100644 --- a/packages/app/src/controller/keyboard.ts +++ b/packages/app/src/controller/keyboard.ts @@ -32,7 +32,7 @@ function isTyping(event: KeyboardEvent): boolean { ) } -function emit(name: string, detail?: unknown): void { +export function emit(name: string, detail?: unknown): void { window.dispatchEvent(new CustomEvent(name, { detail })) } diff --git a/packages/app/src/utils/DragController.ts b/packages/app/src/utils/DragController.ts index 593d452d..e74d8d6e 100644 --- a/packages/app/src/utils/DragController.ts +++ b/packages/app/src/utils/DragController.ts @@ -14,15 +14,22 @@ export enum Direction { type DragControllerHost = HTMLElement & ReactiveControllerHost type AsyncGetElFn = () => Element | Promise<Element | null> +/** Bounds accept getters so panes with a layout-dependent budget clamp live. */ +type Bound = number | (() => number) + interface DragControllerOptions { initialPosition: number direction: Direction localStorageKey?: string - minPosition?: number - maxPosition?: number + minPosition?: Bound + maxPosition?: Bound getContainerEl: AsyncGetElFn } +function resolveBound(bound: Bound | undefined): number | undefined { + return typeof bound === 'function' ? bound() : bound +} + type State = 'dragging' | 'idle' const defaultOptions = { @@ -107,16 +114,18 @@ export class DragController implements ReactiveController { } #setPosition(x: number, y: number) { + const min = resolveBound(this.#options.minPosition) ?? 0 + const max = resolveBound(this.#options.maxPosition) if (this.#options.direction === Direction.horizontal) { - let nx = Math.max(x, this.#options.minPosition || 0) - if (this.#options.maxPosition !== undefined) { - nx = Math.min(nx, this.#options.maxPosition) + let nx = Math.max(x, min) + if (max !== undefined) { + nx = Math.min(nx, max) } this.#x = nx } else { - let ny = Math.max(y, this.#options.minPosition || 0) - if (this.#options.maxPosition !== undefined) { - ny = Math.min(ny, this.#options.maxPosition) + let ny = Math.max(y, min) + if (max !== undefined) { + ny = Math.min(ny, max) } this.#y = ny } @@ -247,6 +256,17 @@ export class DragController implements ReactiveController { // if slider was removed (collapsed) and re-added, re-init pointer tracker if (this.#draggableEl !== draggableEl) { this.#draggableEl = draggableEl as HTMLElement + // The container often doesn't exist at construction (the sidebar renders + // only after a connection), so it must be resolved here too — otherwise + // the tracker attaches but #handleWindowMove bails on a null container + // and the drag silently no-ops. + if (!this.#containerEl) { + const containerEl = await this.#options.getContainerEl() + if (containerEl) { + this.#containerEl = containerEl as HTMLElement + window.onresize = () => this.#adjustPosition() + } + } this.#init() } } diff --git a/packages/app/tests/action-tree.test.ts b/packages/app/tests/action-tree.test.ts new file mode 100644 index 00000000..b7ea66e9 --- /dev/null +++ b/packages/app/tests/action-tree.test.ts @@ -0,0 +1,95 @@ +import { describe, it, expect } from 'vitest' +import type { + TraceActionChild, + TraceActionGroupNode +} from '@wdio/devtools-shared' + +import { + collectCommandIndices, + defaultExpanded, + flattenActionTree +} from '../src/components/workbench/action-tree.js' + +function group( + callId: string, + children: TraceActionChild[], + failed = false +): TraceActionGroupNode { + return { + callId, + title: callId, + startTime: 0, + endTime: 10, + children, + ...(failed ? { failed } : {}) + } +} + +const NESTED = group('hook', [ + { group: group('fixture', [{ commandIndex: 0 }, { commandIndex: 1 }]) }, + { commandIndex: 2 } +]) + +describe('collectCommandIndices', () => { + it('gathers indices across nested groups', () => { + expect(collectCommandIndices(NESTED)).toEqual([0, 1, 2]) + }) + + it('is empty for a group with no command descendants', () => { + expect(collectCommandIndices(group('empty', []))).toEqual([]) + }) +}) + +describe('defaultExpanded', () => { + it('opens failed groups', () => { + expect(defaultExpanded(group('g', [], true))).toBe(true) + }) + + it('opens the group holding the active command', () => { + expect(defaultExpanded(NESTED, 1)).toBe(true) + }) + + it('keeps other groups collapsed', () => { + expect(defaultExpanded(NESTED, 7)).toBe(false) + expect(defaultExpanded(NESTED)).toBe(false) + }) +}) + +describe('flattenActionTree', () => { + const root: TraceActionChild[] = [{ group: NESTED }, { commandIndex: 3 }] + + it('hides children of collapsed groups', () => { + const rows = flattenActionTree(root, () => false) + expect(rows).toEqual([ + { kind: 'group', group: NESTED, depth: 0, expanded: false }, + { kind: 'command', commandIndex: 3, depth: 0 } + ]) + }) + + it('descends into expanded groups with increasing depth', () => { + const rows = flattenActionTree(root, () => true) + expect( + rows.map((row) => + row.kind === 'group' + ? [row.group.callId, row.depth] + : [row.commandIndex, row.depth] + ) + ).toEqual([ + ['hook', 0], + ['fixture', 1], + [0, 2], + [1, 2], + [2, 1], + [3, 0] + ]) + }) + + it('expands only the groups the predicate opens', () => { + const rows = flattenActionTree(root, (g) => g.callId === 'hook') + expect( + rows.map((row) => + row.kind === 'group' ? row.group.callId : row.commandIndex + ) + ).toEqual(['hook', 'fixture', 2, 3]) + }) +}) diff --git a/packages/app/tests/active-entry.test.ts b/packages/app/tests/active-entry.test.ts index 9dcf416f..c894f456 100644 --- a/packages/app/tests/active-entry.test.ts +++ b/packages/app/tests/active-entry.test.ts @@ -1,28 +1,91 @@ import { describe, it, expect } from 'vitest' -import { activeTimestampAt } from '../src/components/workbench/active-entry.js' +import { + activeSpanAt, + type TimeSpanned +} from '../src/components/workbench/active-entry.js' -describe('activeTimestampAt', () => { - const stamps = [100, 200, 300, 400] +describe('activeSpanAt', () => { + // Point-like commands (no startTime) — the pre-span behaviour must be intact. + describe('point-like items (no startTime)', () => { + const points: TimeSpanned[] = [ + { timestamp: 100 }, + { timestamp: 200 }, + { timestamp: 300 }, + { timestamp: 400 } + ] - it('returns undefined before the first action', () => { - expect(activeTimestampAt(stamps, 50)).toBeUndefined() - }) + it('returns undefined before the first item', () => { + expect(activeSpanAt(points, 50)).toBeUndefined() + }) - it('returns the action exactly at the playback time', () => { - expect(activeTimestampAt(stamps, 200)).toBe(200) - }) + it('returns the item exactly at the playback time', () => { + expect(activeSpanAt(points, 200)).toBe(points[1]) + }) - it('returns the latest action at or before the playback time', () => { - expect(activeTimestampAt(stamps, 250)).toBe(200) - expect(activeTimestampAt(stamps, 399)).toBe(300) - }) + it('returns the latest item at or before the playback time', () => { + expect(activeSpanAt(points, 250)).toBe(points[1]) + expect(activeSpanAt(points, 399)).toBe(points[2]) + }) - it('clamps to the last action once playback passes it', () => { - expect(activeTimestampAt(stamps, 999)).toBe(400) + it('clamps to the last item once playback passes it', () => { + expect(activeSpanAt(points, 999)).toBe(points[3]) + }) + + it('returns undefined for an empty timeline', () => { + expect(activeSpanAt([], 100)).toBeUndefined() + }) }) - it('returns undefined for an empty timeline', () => { - expect(activeTimestampAt([], 100)).toBeUndefined() + describe('spanned items (startTime → timestamp)', () => { + it('selects a long-running command while the clock is inside its span', () => { + const before = { timestamp: 1000 } + const poll = { startTime: 1000, timestamp: 11000 } + const items: TimeSpanned[] = [before, poll] + // Mid-poll: the span contains the clock, so the poll stays selected even + // though the preceding command ended earlier. + expect(activeSpanAt(items, 5000)).toBe(poll) + expect(activeSpanAt(items, 1500)).toBe(poll) + // At the span end the poll is still selected. + expect(activeSpanAt(items, 11000)).toBe(poll) + }) + + it('picks the most recently started span when several contain the clock', () => { + const outer = { startTime: 100, timestamp: 10000 } + const inner = { startTime: 4000, timestamp: 6000 } + const items: TimeSpanned[] = [outer, inner] + expect(activeSpanAt(items, 5000)).toBe(inner) + // Outside the inner span but still inside the outer one → outer. + expect(activeSpanAt(items, 2000)).toBe(outer) + expect(activeSpanAt(items, 8000)).toBe(outer) + }) + + it('breaks a start-time tie toward the tighter span', () => { + const wide = { startTime: 100, timestamp: 9000 } + const tight = { startTime: 100, timestamp: 3000 } + const items: TimeSpanned[] = [wide, tight] + expect(activeSpanAt(items, 2000)).toBe(tight) + }) + + it('falls back to the last-ended item when no span contains the clock', () => { + const first = { startTime: 100, timestamp: 2000 } + const second = { startTime: 5000, timestamp: 7000 } + const items: TimeSpanned[] = [first, second] + // Clock sits in the gap between the two spans → the last one that ended. + expect(activeSpanAt(items, 3500)).toBe(first) + }) + + it('returns undefined when the clock precedes the first span', () => { + const items: TimeSpanned[] = [{ startTime: 1000, timestamp: 5000 }] + expect(activeSpanAt(items, 500)).toBeUndefined() + }) + + it('prefers a containing span over an already-ended earlier item', () => { + const ended = { timestamp: 1000 } + const running = { startTime: 900, timestamp: 8000 } + const items: TimeSpanned[] = [ended, running] + // Clock is past the point command but inside the running span → running. + expect(activeSpanAt(items, 2000)).toBe(running) + }) }) }) diff --git a/packages/app/tests/call-source.test.ts b/packages/app/tests/call-source.test.ts index e1a6cb7a..fb354b38 100644 --- a/packages/app/tests/call-source.test.ts +++ b/packages/app/tests/call-source.test.ts @@ -2,7 +2,10 @@ import { describe, expect, it } from 'vitest' import { parseCallSource, fileBasename, - pathSegments + pathSegments, + normalizeSourcePath, + resolveSourceFile, + listSourceFiles } from '../src/components/workbench/call-source.js' describe('parseCallSource', () => { @@ -56,3 +59,91 @@ describe('pathSegments', () => { expect(pathSegments('C:\\a\\b.ts')).toEqual(['C:', 'a', 'b.ts']) }) }) + +describe('normalizeSourcePath', () => { + it('returns a clean path unchanged', () => { + expect(normalizeSourcePath('/a/b/steps.ts')).toBe('/a/b/steps.ts') + }) + + it('strips a glued :line suffix', () => { + expect(normalizeSourcePath('/a/b/steps.ts:17')).toBe('/a/b/steps.ts') + }) + + it('strips a glued :line:column suffix', () => { + expect(normalizeSourcePath('/a/b/steps.ts:17:21')).toBe('/a/b/steps.ts') + }) + + it('keeps Windows drive separators intact', () => { + expect(normalizeSourcePath('C:\\tests\\login.ts:42:5')).toBe( + 'C:\\tests\\login.ts' + ) + expect(normalizeSourcePath('C:\\tests\\login.ts')).toBe( + 'C:\\tests\\login.ts' + ) + }) + + it('leaves a leading-colon-only string unchanged', () => { + expect(normalizeSourcePath(':12')).toBe(':12') + }) +}) + +describe('resolveSourceFile', () => { + const content = 'export {}' + + it('returns an exact key match', () => { + expect(resolveSourceFile({ '/a/steps.ts': content }, '/a/steps.ts')).toBe( + '/a/steps.ts' + ) + }) + + it('matches a suffixed key from a clean query', () => { + expect( + resolveSourceFile({ '/a/steps.ts:17': content }, '/a/steps.ts') + ).toBe('/a/steps.ts:17') + }) + + it('matches a clean key from a suffixed query', () => { + expect( + resolveSourceFile({ '/a/steps.ts': content }, '/a/steps.ts:17:21') + ).toBe('/a/steps.ts') + }) + + it('returns undefined when nothing matches', () => { + expect(resolveSourceFile({ '/a/steps.ts': content }, '/a/other.ts')).toBe( + undefined + ) + expect(resolveSourceFile({}, '/a/steps.ts')).toBe(undefined) + }) +}) + +describe('listSourceFiles', () => { + const content = 'export {}' + + it('lists normalized source keys', () => { + expect(listSourceFiles({ '/a/steps.ts:17': content }, [])).toEqual([ + '/a/steps.ts' + ]) + }) + + it('adds files referenced only by call sources', () => { + expect( + listSourceFiles({ '/a/steps.ts': content }, [ + '/a/steps.ts:17:21', + '/a/other.ts:3', + undefined + ]) + ).toEqual(['/a/steps.ts', '/a/other.ts']) + }) + + it('deduplicates by normalized path', () => { + expect( + listSourceFiles({ '/a/steps.ts:17': content, '/a/steps.ts:23': 'x' }, [ + '/a/steps.ts:23:3' + ]) + ).toEqual(['/a/steps.ts']) + }) + + it('ignores call sources without a line number', () => { + expect(listSourceFiles({}, ['/a/steps.ts'])).toEqual([]) + }) +}) diff --git a/packages/app/tests/duration.test.ts b/packages/app/tests/duration.test.ts index c32f3bc1..92ab1c87 100644 --- a/packages/app/tests/duration.test.ts +++ b/packages/app/tests/duration.test.ts @@ -3,8 +3,10 @@ import { describe, it, expect } from 'vitest' import { formatDuration, durationHeat, + entryDuration, stepDurations } from '../src/components/workbench/actionItems/duration.js' +import type { CommandLog, TraceMutation } from '@wdio/devtools-shared' describe('formatDuration', () => { it('shows milliseconds under a second', () => { @@ -22,6 +24,15 @@ describe('formatDuration', () => { expect(formatDuration(60001)).toBe('1m 0s') expect(formatDuration(150000)).toBe('2m 30s') }) + + it('rounds fractional milliseconds from reconstructed traces', () => { + expect(formatDuration(1.02978515625)).toBe('1ms') + expect(formatDuration(22.4)).toBe('22ms') + expect(formatDuration(0.4)).toBe('0ms') + expect(formatDuration(999.7)).toBe('1000ms') + expect(formatDuration(5049.9)).toBe('5.05s') + expect(formatDuration(60000.6)).toBe('1m 0s') + }) }) describe('durationHeat', () => { @@ -64,3 +75,34 @@ describe('stepDurations', () => { expect(stepDurations([0, 0, 0, 500])).toEqual([0, 0, 500, 500]) }) }) + +describe('entryDuration', () => { + const cmd = (over: Partial<CommandLog> = {}): CommandLog => ({ + command: 'expect.toExist', + args: [], + timestamp: 10603, + ...over + }) + + it('uses the command execution span (timestamp − startTime) when present', () => { + // Regression: an assertion whose internal polling is suppressed sits after a + // long navigation gap; the row must show its own 603ms runtime, not the gap. + expect(entryDuration(cmd({ startTime: 10000 }), 10650)).toBe(603) + }) + + it('falls back to the inter-action gap when the command has no startTime', () => { + expect(entryDuration(cmd(), 420)).toBe(420) + }) + + it('falls back to the gap for a mutation entry (no startTime field)', () => { + const mutation = { + type: 'childList', + timestamp: 5 + } as unknown as TraceMutation + expect(entryDuration(mutation, 42)).toBe(42) + }) + + it('returns undefined when neither a span nor a gap is available', () => { + expect(entryDuration(cmd(), undefined)).toBeUndefined() + }) +}) diff --git a/packages/app/tests/errors-collect.test.ts b/packages/app/tests/errors-collect.test.ts new file mode 100644 index 00000000..c92e3680 --- /dev/null +++ b/packages/app/tests/errors-collect.test.ts @@ -0,0 +1,337 @@ +import { describe, it, expect } from 'vitest' +import type { CommandLog } from '@wdio/devtools-shared' + +import { collectErrors } from '../src/components/workbench/errors/collect.js' +import type { + SuiteStatsFragment, + TestStatsFragment +} from '../src/controller/types.js' + +function command(overrides: Partial<CommandLog>): CommandLog { + return { + command: 'click', + args: [], + timestamp: 0, + ...overrides + } +} + +function suiteMap( + ...suites: SuiteStatsFragment[] +): Record<string, SuiteStatsFragment>[] { + return [Object.fromEntries(suites.map((s) => [s.uid, s]))] +} + +function failedTest(overrides: Partial<TestStatsFragment>): TestStatsFragment { + return { + uid: 't1', + state: 'failed', + ...overrides + } +} + +describe('collectErrors', () => { + it('returns an empty list when nothing failed', () => { + expect(collectErrors([], [])).toEqual([]) + expect(collectErrors(undefined, undefined)).toEqual([]) + expect( + collectErrors( + [command({ command: 'click' })], + suiteMap({ + uid: 's1', + state: 'passed', + tests: [failedTest({ state: 'passed' })] + }) + ) + ).toEqual([]) + }) + + it('collects failed commands with title, message, stack and source', () => { + const errors = collectErrors( + [ + command({ + command: 'expect', + title: 'expect(el).toHaveText', + callSource: 'file:///spec.ts:12:3', + error: { + name: 'Error', + message: 'expected foo, got bar', + stack: 'at spec.ts:12' + }, + timestamp: 100 + }) + ], + [] + ) + expect(errors).toHaveLength(1) + expect(errors[0]).toMatchObject({ + title: 'expect(el).toHaveText', + message: 'expected foo, got bar', + stack: 'at spec.ts:12', + callSource: 'file:///spec.ts:12:3', + timestamp: 100 + }) + expect(errors[0].command?.command).toBe('expect') + }) + + it('falls back to the command name when there is no title', () => { + const [error] = collectErrors( + [ + command({ + command: 'navigateTo', + error: { name: 'Error', message: 'boom' } + }) + ], + [] + ) + expect(error.title).toBe('navigateTo') + }) + + it('orders command errors by timestamp', () => { + const errors = collectErrors( + [ + command({ + command: 'second', + error: { name: 'Error', message: 'b' }, + timestamp: 200 + }), + command({ + command: 'first', + error: { name: 'Error', message: 'a' }, + timestamp: 100 + }) + ], + [] + ) + expect(errors.map((e) => e.message)).toEqual(['a', 'b']) + }) + + it('collects failed tests from nested suites', () => { + const errors = collectErrors( + [], + suiteMap({ + uid: 'feature', + state: 'failed', + suites: [ + { + uid: 'scenario', + state: 'failed', + tests: [ + failedTest({ + uid: 'step-1', + title: 'Then it should pass', + fullTitle: 'Scenario > Then it should pass', + callSource: 'steps.ts:8:1', + error: { name: 'AssertionError', message: 'nope' } + }) + ] + } + ] + }) + ) + expect(errors).toHaveLength(1) + expect(errors[0]).toMatchObject({ + title: 'Scenario > Then it should pass', + message: 'nope', + callSource: 'steps.ts:8:1' + }) + expect(errors[0].command).toBeUndefined() + }) + + it('reads the first entry of errors[] when error is absent', () => { + const [error] = collectErrors( + [], + suiteMap({ + uid: 's', + state: 'failed', + tests: [ + failedTest({ + uid: 'a', + errors: [{ name: 'Error', message: 'from errors array' } as Error] + }) + ] + }) + ) + expect(error.message).toBe('from errors array') + }) + + it('ignores failed tests that carry no error payload', () => { + const errors = collectErrors( + [], + suiteMap({ + uid: 's', + state: 'failed', + tests: [failedTest({ uid: 'a' })] + }) + ) + expect(errors).toEqual([]) + }) + + it('dedupes a test failure that only echoes a command failure', () => { + const shared = 'expect(locator).toHaveText failed' + const errors = collectErrors( + [ + command({ + command: 'expect', + error: { name: 'Error', message: shared }, + timestamp: 5 + }) + ], + suiteMap({ + uid: 's', + state: 'failed', + tests: [ + failedTest({ + uid: 'a', + title: 'the scenario', + error: { name: 'Error', message: shared } + }) + ] + }) + ) + expect(errors).toHaveLength(1) + expect(errors[0].command?.command).toBe('expect') + }) + + it('keeps a distinct test failure alongside command failures', () => { + const errors = collectErrors( + [ + command({ + command: 'click', + error: { name: 'Error', message: 'click failed' }, + timestamp: 1 + }) + ], + suiteMap({ + uid: 's', + state: 'failed', + tests: [ + failedTest({ + uid: 'a', + title: 'setup', + error: { name: 'Error', message: 'hook failed' } + }) + ] + }) + ) + expect(errors.map((e) => e.message)).toEqual([ + 'click failed', + 'hook failed' + ]) + }) + + it('dedupes failed tests by uid across repeated suite maps', () => { + const test = failedTest({ + uid: 'dup', + title: 'flaky', + error: { name: 'Error', message: 'x' } + }) + const errors = collectErrors( + [], + [ + { s: { uid: 's', state: 'failed', tests: [test] } }, + { s: { uid: 's', state: 'failed', tests: [test] } } + ] + ) + expect(errors).toHaveLength(1) + }) + + it('strips ANSI, splits the stack, and pulls Expected/Received from a cucumber message', () => { + const raw = + 'Expect $(`#flash`) to have text\n\n' + + 'Expected: StringContaining "secure area"\n' + + 'Received: "invalid"\n' + + ' at World.<anonymous> (/specs/steps.ts:31:20)\n' + + ' at process.processTicksAndRejections (node:internal:104:5)' + const [error] = collectErrors( + [ + command({ + command: 'expect.assertion', + error: { name: 'Error', message: raw } + }) + ], + [] + ) + expect(error.message).toBe('Expect $(`#flash`) to have text') + expect(error.expected).toBe('StringContaining "secure area"') + expect(error.actual).toBe('"invalid"') + expect(error.stack).toContain('at World.<anonymous>') + expect(error.message).not.toContain('') + expect(error.message).not.toContain('at World') + }) + + it('extracts indented Expected/Received and dedents the value', () => { + const raw = + 'Expect $(`#flash`) to have text\n\n' + + ' Expected: StringContaining "secure area"\n' + + ' Received: "invalid!\n×"\n' + + ' at World.<anonymous> (/specs/steps.ts:31:20)' + const [error] = collectErrors( + [command({ command: 'getText', error: { name: 'Error', message: raw } })], + [] + ) + expect(error.message).toBe('Expect $(`#flash`) to have text') + expect(error.expected).toBe('StringContaining "secure area"') + expect(error.actual).toBe('"invalid!\n×"') + expect(error.stack).toContain('at World.<anonymous>') + }) + + it('extracts expected/received from an assertion command', () => { + const [error] = collectErrors( + [ + command({ + command: 'expect.toHaveText', + args: ['Your username is invalid!', 'You logged into a secure area!'], + error: { name: 'Error', message: 'text mismatch' } + }) + ], + [] + ) + expect(error.actual).toBe('Your username is invalid!') + expect(error.expected).toBe('You logged into a secure area!') + }) + + it('extracts expected/received from a failed-test matcher error', () => { + const [error] = collectErrors( + [], + suiteMap({ + uid: 's', + state: 'failed', + tests: [ + failedTest({ + uid: 'a', + title: 'the scenario', + error: { + name: 'Error', + message: 'mismatch', + expected: 42, + actual: 7 + } as unknown as Error + }) + ] + }) + ) + expect(error.expected).toBe('42') + expect(error.actual).toBe('7') + }) + + it('places command errors before test errors', () => { + const errors = collectErrors( + [ + command({ + command: 'click', + error: { name: 'Error', message: 'cmd' }, + timestamp: 999 + }) + ], + suiteMap({ + uid: 's', + state: 'failed', + tests: [ + failedTest({ uid: 'a', error: { name: 'Error', message: 'test' } }) + ] + }) + ) + expect(errors.map((e) => e.message)).toEqual(['cmd', 'test']) + }) +}) diff --git a/packages/app/tests/trace-timeline-utils.test.ts b/packages/app/tests/trace-timeline-utils.test.ts new file mode 100644 index 00000000..e3197c7b --- /dev/null +++ b/packages/app/tests/trace-timeline-utils.test.ts @@ -0,0 +1,33 @@ +import { describe, it, expect } from 'vitest' +import { + formatTickLabel, + tickStep +} from '../src/components/browser/trace-timeline-utils.js' + +describe('tickStep', () => { + it('picks the smallest step yielding at most the target tick count', () => { + expect(tickStep(10_000)).toBe(1_000) + expect(tickStep(78_460)).toBe(10_000) + expect(tickStep(1_200)).toBe(100) + }) + + it('caps at the largest step for very long traces', () => { + expect(tickStep(3 * 60 * 60 * 1000)).toBe(600_000) + }) +}) + +describe('formatTickLabel', () => { + it('formats sub-second ticks as milliseconds', () => { + expect(formatTickLabel(500)).toBe('500ms') + }) + + it('formats seconds with one decimal', () => { + expect(formatTickLabel(1_000)).toBe('1.0s') + expect(formatTickLabel(3_500)).toBe('3.5s') + }) + + it('formats minutes as m:ss', () => { + expect(formatTickLabel(75_000)).toBe('1:15') + expect(formatTickLabel(60_000)).toBe('1:00') + }) +}) diff --git a/packages/backend/src/trace-reader-constants.ts b/packages/backend/src/trace-reader-constants.ts index 6e738464..3cd37bd4 100644 --- a/packages/backend/src/trace-reader-constants.ts +++ b/packages/backend/src/trace-reader-constants.ts @@ -1,15 +1,46 @@ -import { ACTION_MAP } from '@wdio/devtools-shared' +import { + ACTION_MAP, + ASSERT_ACTION_CLASS, + LOG_LEVELS, + TRACKED_ASSERT_METHODS +} from '@wdio/devtools-shared' + +/** Runtime lookup for narrowing foreign trace levels to the shared union. */ +export const LOG_LEVEL_SET: ReadonlySet<string> = new Set(LOG_LEVELS) + +/** Every zip entry ending in this suffix is an NDJSON action-event stream. */ +export const TRACE_STREAM_SUFFIX = '.trace' + +/** Every zip entry ending in this suffix is an NDJSON HAR-snapshot stream. */ +export const NETWORK_STREAM_SUFFIX = '.network' + +/** Sidecar entries holding call stacks keyed by numeric call id. */ +export const STACKS_STREAM_SUFFIX = '.stacks' + +/** Foreign screencast refs may be a bare sha1; probe image extensions too. */ +export const FRAME_RESOURCE_SUFFIXES = ['', '.jpeg', '.png'] as const // Inverse of ACTION_MAP, derived so it can never drift from the forward map. // The forward map is many-to-one (url/navigateTo/get all → Page.navigate); the // first runner command listed for each trace action wins, matching the command // name live mode shows so the UI colours/labels the row identically. -export const REVERSE_ACTION_MAP: Record<string, string> = Object.entries( - ACTION_MAP -).reduce<Record<string, string>>((acc, [command, action]) => { - const key = `${action.class}.${action.method}` - if (!(key in acc)) { - acc[key] = command - } - return acc -}, {}) +// Assert entries come from the same TRACKED_ASSERT_METHODS list the core +// patcher wraps, so `Assert.<m>` rows read back as `assert.<m>` commands. +export const REVERSE_ACTION_MAP: Record<string, string> = { + ...Object.entries(ACTION_MAP).reduce<Record<string, string>>( + (acc, [command, action]) => { + const key = `${action.class}.${action.method}` + if (!(key in acc)) { + acc[key] = command + } + return acc + }, + {} + ), + ...Object.fromEntries( + TRACKED_ASSERT_METHODS.map((method) => [ + `${ASSERT_ACTION_CLASS}.${method}`, + `assert.${method}` + ]) + ) +} diff --git a/packages/backend/src/trace-reader-groups.ts b/packages/backend/src/trace-reader-groups.ts new file mode 100644 index 00000000..600fd6ef --- /dev/null +++ b/packages/backend/src/trace-reader-groups.ts @@ -0,0 +1,165 @@ +// Rebuilds the collapsible action tree from the structural before events that +// buildCommands drops from the flat list: foreign runner steps (self-referencing +// stepId, nested via parentId) and our Tracing.tracingGroup markers (children +// linked via parentId). Library-call leaves attach to the group their stepId +// points at; unreferenced commands surface at the root in chronological order. + +import type { + CommandLog, + TraceActionChild, + TraceActionGroupNode +} from '@wdio/devtools-shared' + +import type { AfterEvent, BeforeEvent } from './trace-reader-types.js' + +/** Runner steps mirrored by a library call (stepId) and parents of nested + * steps are structure — as command rows they duplicate or envelop actions. */ +export function collectStructuralIds( + befores: Map<string, BeforeEvent> +): Set<string> { + const structural = new Set<string>() + for (const before of befores.values()) { + if (before.stepId && before.stepId !== before.callId) { + structural.add(before.stepId) + } + if (before.parentId) { + structural.add(before.parentId) + } + } + return structural +} + +export function isStructuralBefore( + before: BeforeEvent, + structural: Set<string> +): boolean { + return before.class === 'Tracing' || structural.has(before.callId) +} + +function groupTitle(before: BeforeEvent): string { + if (before.title) { + return before.title + } + const name = before.params?.name + return typeof name === 'string' ? name : before.method +} + +// A before nests under its stepId wrapper (foreign library calls) or its +// parentId container (runner steps and our group markers), whichever exists. +function parentGroupOf( + before: BeforeEvent, + nodes: Map<string, TraceActionGroupNode> +): TraceActionGroupNode | undefined { + if (before.stepId && before.stepId !== before.callId) { + const wrapper = nodes.get(before.stepId) + if (wrapper) { + return wrapper + } + } + return before.parentId ? nodes.get(before.parentId) : undefined +} + +function childStartTime(child: TraceActionChild, commands: CommandLog[]) { + if ('group' in child) { + return child.group.startTime + } + const command = commands[child.commandIndex] + return command.startTime ?? command.timestamp +} + +function sortTreeChronologically( + children: TraceActionChild[], + commands: CommandLog[] +): void { + children.sort( + (a, b) => childStartTime(a, commands) - childStartTime(b, commands) + ) + for (const child of children) { + if ('group' in child) { + sortTreeChronologically(child.group.children, commands) + } + } +} + +// Post-order rollup: a group fails when its own after errored or any +// descendant group/command did. +function rollupFailed( + child: TraceActionChild, + commands: CommandLog[] +): boolean { + if (!('group' in child)) { + return Boolean(commands[child.commandIndex].error) + } + let failed = Boolean(child.group.failed) + for (const nested of child.group.children) { + failed = rollupFailed(nested, commands) || failed + } + if (failed) { + child.group.failed = true + } + return failed +} + +function buildGroupNodes( + befores: Map<string, BeforeEvent>, + afters: Map<string, AfterEvent> +): Map<string, TraceActionGroupNode> { + const structural = collectStructuralIds(befores) + const nodes = new Map<string, TraceActionGroupNode>() + for (const before of befores.values()) { + if (!isStructuralBefore(before, structural)) { + continue + } + const after = afters.get(before.callId) + const node: TraceActionGroupNode = { + callId: before.callId, + title: groupTitle(before), + startTime: before.startTime, + endTime: after?.endTime ?? before.startTime, + children: [] + } + if (after?.error) { + node.failed = true + } + nodes.set(before.callId, node) + } + return nodes +} + +/** Build the action tree, or undefined when the zip has no structural steps. */ +export function buildActionTree( + befores: Map<string, BeforeEvent>, + afters: Map<string, AfterEvent>, + commands: CommandLog[], + indexByCallId: Map<string, number> +): TraceActionChild[] | undefined { + const nodes = buildGroupNodes(befores, afters) + if (nodes.size === 0) { + return undefined + } + const root: TraceActionChild[] = [] + for (const before of befores.values()) { + const node = nodes.get(before.callId) + const commandIndex = indexByCallId.get(before.callId) + let child: TraceActionChild + if (node) { + child = { group: node } + } else if (commandIndex !== undefined) { + child = { commandIndex } + } else { + continue + } + const parent = parentGroupOf(before, nodes) + // A malformed self-parent would recurse forever in the rollup; root it. + if (parent && parent !== node) { + parent.children.push(child) + } else { + root.push(child) + } + } + sortTreeChronologically(root, commands) + for (const child of root) { + rollupFailed(child, commands) + } + return root +} diff --git a/packages/backend/src/trace-reader-types.ts b/packages/backend/src/trace-reader-types.ts index 63bc3139..0afc3a9b 100644 --- a/packages/backend/src/trace-reader-types.ts +++ b/packages/backend/src/trace-reader-types.ts @@ -9,6 +9,12 @@ export interface BeforeEvent { class: string method: string params?: Record<string, unknown> + stack?: { file: string; line?: number; column?: number }[] + title?: string + /** Foreign runner streams: the runner step a library call renders under. */ + stepId?: string + /** Group/container step this event nests under. */ + parentId?: string } export interface AfterEvent { @@ -16,6 +22,8 @@ export interface AfterEvent { callId: string endTime: number error?: { message: string } + /** Command return value, restored onto CommandLog.result. */ + result?: unknown } export interface ScreencastFrameEvent { @@ -24,14 +32,49 @@ export interface ScreencastFrameEvent { timestamp: number } +export interface ConsoleEvent { + type: 'console' + time: number + messageType: string + text: string + args?: { preview: string; value: unknown }[] +} + +export interface StdioEvent { + type: 'stdout' | 'stderr' + timestamp: number + text?: string + /** Extension field restoring the test-vs-terminal origin; absent in foreign zips. */ + source?: 'test' | 'terminal' +} + export interface ContextOptionsEvent { type: 'context-options' wallTime: number + /** Monotonic clock reading taken at `wallTime`; anchors monotonic event times. */ + monotonicTime?: number browserName?: string contextId?: string options?: { viewport?: { width: number; height: number } } } +/** Sidecar `.stacks` shape: file table + per-call [fileIndex, line, column, function] frames. */ +export interface SidecarStacks { + files: string[] + stacks: [number, [number, number, number, string][]][] +} + +export interface HarContent { + size: number + mimeType: string + /** Inline body; absent when the body lives in a `resources/` entry. */ + text?: string + /** HAR body encoding — `base64` marks binary inline text. */ + encoding?: string + /** Body resource name under `resources/` — sha1 hex, in foreign zips suffixed with a mime extension. */ + _sha1?: string +} + export interface HarSnapshot { startedDateTime: string time: number @@ -44,14 +87,20 @@ export interface HarSnapshot { status: number statusText: string headers?: { name: string; value: string }[] - content?: { size: number; mimeType: string } + content?: HarContent } } -/** Trace events grouped by kind for the reconstruction pipeline. */ +/** One stream's trace events grouped by kind for the reconstruction pipeline. */ export interface CategorizedEvents { ctx?: ContextOptionsEvent befores: Map<string, BeforeEvent> afters: Map<string, AfterEvent> frameEvents: ScreencastFrameEvent[] + consoleEvents: (ConsoleEvent | StdioEvent)[] +} + +/** All streams' events combined; a zip may carry one context per stream. */ +export interface MergedEvents extends Omit<CategorizedEvents, 'ctx'> { + ctxs: ContextOptionsEvent[] } diff --git a/packages/backend/src/trace-reader-utils.ts b/packages/backend/src/trace-reader-utils.ts index 16a0972e..d821986e 100644 --- a/packages/backend/src/trace-reader-utils.ts +++ b/packages/backend/src/trace-reader-utils.ts @@ -1,15 +1,28 @@ // Pure helpers for reconstructing a player payload from trace.zip events. // No I/O — the reader pipeline (trace-reader.ts) composes these. +import { createHash } from 'node:crypto' +import { strFromU8 } from 'fflate' import { TraceType, + type ConsoleLog, + type LogLevel, type Metadata, type NetworkRequest, type TracePlayerFrame, type Viewport } from '@wdio/devtools-shared' -import type { ContextOptionsEvent, HarSnapshot } from './trace-reader-types.js' +import { LOG_LEVEL_SET } from './trace-reader-constants.js' +import type { + BeforeEvent, + ConsoleEvent, + ContextOptionsEvent, + HarContent, + HarSnapshot, + SidecarStacks, + StdioEvent +} from './trace-reader-types.js' export function parseNdjson(text: string): Record<string, unknown>[] { return text @@ -44,9 +57,9 @@ export function paramsToArgs( return indexKeys.map((key) => params[key]) } -// Playwright/vibium-style action label, built from class.method + the most -// meaningful param (value for fill/type, url for navigate, nothing for click) — -// matching what `playwright show-trace` renders for the same trace. +// Trace-viewer action label, built from class.method + the most meaningful +// param (value for fill/type, url for navigate, nothing for click) — +// matching what standard trace viewers render for the same trace. export function actionLabel( cls: string, method: string, @@ -103,17 +116,42 @@ function mimeToType(mimeType: string): string { return 'other' } +const TEXTUAL_MIME_RE = /json|xml|javascript|ecmascript|^text\// + +// Restore the body from inline `text`, else the content-addressed +// `resources/<_sha1>` entry (the `_sha1` value is the resource filename, +// extension included when the writer added one). Textual mimes only — +// binary bodies stay out of the string field. +function restoreResponseBody( + content: HarContent | undefined, + files: Record<string, Uint8Array> | undefined +): string | undefined { + if (!content || !TEXTUAL_MIME_RE.test(content.mimeType)) { + return undefined + } + if (typeof content.text === 'string' && content.encoding !== 'base64') { + return content.text + } + if (!content._sha1 || !files) { + return undefined + } + const data = files[`resources/${content._sha1}`] + return data ? strFromU8(data) : undefined +} + export function harToNetworkRequest( snapshot: HarSnapshot, - index: number + index: number, + files?: Record<string, Uint8Array> ): NetworkRequest { const started = Date.parse(snapshot.startedDateTime) const startTime = Number.isFinite(started) ? started : 0 - // A foreign trace.zip (the reader accepts any Vibium/Playwright zip) can carry + // A foreign trace.zip (the reader accepts any standard-format zip) can carry // a pending or failed request with no response or content; default those. const response = snapshot.response const content = response?.content const responseHeaders = headerArrayToRecord(response?.headers) + const responseBody = restoreResponseBody(content, files) return { id: String(index), url: snapshot.request.url, @@ -128,6 +166,7 @@ export function harToNetworkRequest( requestHeaders: headerArrayToRecord(snapshot.request.headers), responseHeaders, size: content?.size ?? 0, + ...(responseBody !== undefined ? { responseBody } : {}), response: { fromCache: false, headers: responseHeaders, @@ -137,6 +176,122 @@ export function harToNetworkRequest( } } +// Reverse level mapping; foreign levels outside our union default to 'log'. +function fromTraceLevel(messageType: string): LogLevel { + if (messageType === 'warning') { + return 'warn' + } + return LOG_LEVEL_SET.has(messageType) ? (messageType as LogLevel) : 'log' +} + +/** Map console/stdio events (already carrying absolute times) to console logs. */ +export function buildConsoleLogs( + consoleEvents: (ConsoleEvent | StdioEvent)[] +): ConsoleLog[] { + const logs: ConsoleLog[] = consoleEvents.map((event) => { + if (event.type === 'console') { + return { + type: fromTraceLevel(event.messageType), + args: event.args?.map((arg) => arg.value) ?? [event.text], + timestamp: event.time, + source: 'browser' as const + } + } + return { + type: event.type === 'stderr' ? ('error' as const) : ('log' as const), + args: [event.text ?? ''], + timestamp: event.timestamp, + // Our zips carry the origin; foreign stdio events default to terminal. + source: event.source ?? ('terminal' as const) + } + }) + return logs.sort((a, b) => a.timestamp - b.timestamp) +} + +// Local copy of core's sha1 helper — the backend only imports from shared. +function sha1Hex(data: string): string { + return createHash('sha1').update(data).digest('hex') +} + +// Older zips glued ':<line>[:<column>]' onto the frame's file (and shifted +// line/column); peel up to two numeric suffixes — the innermost is the real +// line. `at < 2` keeps bare Windows drive specs (`C:...`) intact. +function splitGluedLineSuffix(file: string): { + file: string + line?: number +} { + let cleaned = file + let line: number | undefined + for (let pass = 0; pass < 2; pass++) { + const at = cleaned.lastIndexOf(':') + if (at < 2 || !/^\d+$/.test(cleaned.slice(at + 1))) { + break + } + line = Number(cleaned.slice(at + 1)) + cleaned = cleaned.slice(0, at) + } + return { file: cleaned, line } +} + +/** Rebuild the `<file>:<line>` callSource from an event's first stack frame. */ +export function stackToCallSource( + stack: BeforeEvent['stack'] +): string | undefined { + const frame = stack?.[0] + if (!frame) { + return undefined + } + const { file, line } = splitGluedLineSuffix(frame.file) + return `${file}:${line ?? frame.line ?? 0}` +} + +/** Fill in stacks from a sidecar `.stacks` entry (foreign zips store them there, keyed `call@<id>`). */ +export function attachSidecarStacks( + befores: Map<string, BeforeEvent>, + stacksJson: string +): void { + // Cast at the zip boundary: the sidecar is a single JSON document, not NDJSON. + const parsed = JSON.parse(stacksJson) as Partial<SidecarStacks> + if (!Array.isArray(parsed.files) || !Array.isArray(parsed.stacks)) { + return + } + const paths = parsed.files + for (const [id, frames] of parsed.stacks) { + const before = befores.get(`call@${id}`) + if (!before || before.stack?.length) { + continue + } + before.stack = frames.flatMap(([fileIndex, line, column]) => { + const file = paths[fileIndex] + return file ? [{ file, line, column }] : [] + }) + } +} + +/** Recover `sources` by matching stack-frame paths to `src@<sha1(path)>.txt`. */ +export function buildSources( + befores: Iterable<BeforeEvent>, + files: Record<string, Uint8Array> +): Record<string, string> { + const sources: Record<string, string> = {} + for (const before of befores) { + const rawFile = before.stack?.[0]?.file + if (!rawFile) { + continue + } + // Resources were written under the clean path's sha1; unglue before lookup. + const { file } = splitGluedLineSuffix(rawFile) + if (file in sources) { + continue + } + const data = files[`resources/src@${sha1Hex(file)}.txt`] + if (data) { + sources[file] = strFromU8(data) + } + } + return sources +} + export function nearestFrame( frames: TracePlayerFrame[], timestamp: number diff --git a/packages/backend/src/trace-reader.ts b/packages/backend/src/trace-reader.ts index d8643f39..204fd464 100644 --- a/packages/backend/src/trace-reader.ts +++ b/packages/backend/src/trace-reader.ts @@ -1,36 +1,58 @@ -// Reads a trace.zip produced by core/trace-exporter.ts back into a player -// payload. The writer is the inverse: this reconstructs commands from -// before/after events, the frame filmstrip from screencast-frame events + -// resources/*.jpeg, and network requests from the HAR resource-snapshot -// entries. Fields the zip never carried (console logs, mutations, sources, -// suites) come back empty. Constants, event types, and pure helpers live in the +// Reads a standard-format trace.zip back into a player payload. Accepts our +// own exporter's output (core/trace-exporter.ts) and foreign zips: every +// entry ending in `.trace` is an action-event stream (foreign tools write +// `test.trace` plus per-context `0-trace.trace`, ...), every `.network` entry +// is a HAR stream, and `.stacks` sidecars carry call stacks. Commands come +// from before/after events, the filmstrip from screencast-frame events + +// resources/*, console logs from console/stdio events, sources from stack +// frames + src@ resources. Fields the zip never carried (mutations, suites) +// come back empty. Constants, event types, and pure helpers live in the // sibling trace-reader-{constants,types,utils}.ts files. import fs from 'node:fs/promises' import { unzipSync, strFromU8 } from 'fflate' import { type CommandLog, + type NetworkRequest, type TraceLog, type TracePlayerData, type TracePlayerFrame } from '@wdio/devtools-shared' -import { REVERSE_ACTION_MAP } from './trace-reader-constants.js' +import { + FRAME_RESOURCE_SUFFIXES, + NETWORK_STREAM_SUFFIX, + REVERSE_ACTION_MAP, + STACKS_STREAM_SUFFIX, + TRACE_STREAM_SUFFIX +} from './trace-reader-constants.js' import type { AfterEvent, BeforeEvent, CategorizedEvents, + ConsoleEvent, ContextOptionsEvent, HarSnapshot, - ScreencastFrameEvent + MergedEvents, + ScreencastFrameEvent, + StdioEvent } from './trace-reader-types.js' +import { + buildActionTree, + collectStructuralIds, + isStructuralBefore +} from './trace-reader-groups.js' import { actionLabel, + attachSidecarStacks, + buildConsoleLogs, buildMetadata, + buildSources, harToNetworkRequest, nearestFrame, paramsToArgs, - parseNdjson + parseNdjson, + stackToCallSource } from './trace-reader-utils.js' function categorizeEvents( @@ -39,6 +61,7 @@ function categorizeEvents( const befores = new Map<string, BeforeEvent>() const afters = new Map<string, AfterEvent>() const frameEvents: ScreencastFrameEvent[] = [] + const consoleEvents: (ConsoleEvent | StdioEvent)[] = [] let ctx: ContextOptionsEvent | undefined for (const event of events) { switch (event.type) { @@ -58,103 +81,251 @@ function categorizeEvents( case 'screencast-frame': frameEvents.push(event as unknown as ScreencastFrameEvent) break + case 'console': + case 'stdout': + case 'stderr': + consoleEvents.push(event as unknown as ConsoleEvent | StdioEvent) + break + } + } + return { ctx, befores, afters, frameEvents, consoleEvents } +} + +// Anchors both encodings: our offsets (monotonicTime 0 → anchor = wallTime) +// and foreign monotonic readings (anchor = wallTime - monotonicTime). +function rebaseToEpoch(stream: CategorizedEvents): void { + const ctx = stream.ctx + const anchor = ctx ? ctx.wallTime - (ctx.monotonicTime ?? 0) : 0 + if (anchor === 0) { + return + } + for (const before of stream.befores.values()) { + before.startTime += anchor + } + for (const after of stream.afters.values()) { + after.endTime += anchor + } + for (const frame of stream.frameEvents) { + frame.timestamp += anchor + } + for (const event of stream.consoleEvents) { + if (event.type === 'console') { + event.time += anchor + } else { + event.timestamp += anchor + } + } +} + +function mergeStreams(streams: CategorizedEvents[]): MergedEvents { + const merged: MergedEvents = { + ctxs: [], + befores: new Map(), + afters: new Map(), + frameEvents: [], + consoleEvents: [] + } + for (const stream of streams) { + if (stream.ctx) { + merged.ctxs.push(stream.ctx) + } + for (const [callId, before] of stream.befores) { + merged.befores.set(callId, before) } + for (const [callId, after] of stream.afters) { + merged.afters.set(callId, after) + } + merged.frameEvents.push(...stream.frameEvents) + merged.consoleEvents.push(...stream.consoleEvents) } - return { ctx, befores, afters, frameEvents } + return merged +} + +function frameResource( + files: Record<string, Uint8Array>, + sha1: string +): Uint8Array | undefined { + for (const suffix of FRAME_RESOURCE_SUFFIXES) { + const data = files[`resources/${sha1}${suffix}`] + if (data) { + return data + } + } + return undefined } function buildFrames( files: Record<string, Uint8Array>, - frameEvents: ScreencastFrameEvent[], - wallTime: number -): { frames: TracePlayerFrame[]; maxOffset: number } { + frameEvents: ScreencastFrameEvent[] +): { frames: TracePlayerFrame[]; maxTime: number } { const frames: TracePlayerFrame[] = [] - let maxOffset = 0 + let maxTime = 0 for (const event of frameEvents) { - const data = files[`resources/${event.sha1}`] + const data = frameResource(files, event.sha1) if (!data) { continue } frames.push({ - timestamp: wallTime + event.timestamp, + timestamp: event.timestamp, screenshot: Buffer.from(data).toString('base64') }) - maxOffset = Math.max(maxOffset, event.timestamp) + maxTime = Math.max(maxTime, event.timestamp) } frames.sort((a, b) => a.timestamp - b.timestamp) - return { frames, maxOffset } + return { frames, maxTime } +} + +// Foreign runner streams may record a failure only on the wrapper step a +// library call renders under; surface it on the visible command row. +function commandError( + before: BeforeEvent, + after: AfterEvent | undefined, + afters: Map<string, AfterEvent> +): { message: string } | undefined { + if (after?.error) { + return after.error + } + if (before.stepId && before.stepId !== before.callId) { + return afters.get(before.stepId)?.error + } + return undefined +} + +/** Reconstruct one CommandLog from its before/after events + nearest frame. */ +function reconstructCommand( + before: BeforeEvent, + after: AfterEvent | undefined, + afters: Map<string, AfterEvent>, + frames: TracePlayerFrame[] +): CommandLog { + const endTime = after?.endTime ?? before.startTime + const command: CommandLog = { + command: + REVERSE_ACTION_MAP[`${before.class}.${before.method}`] ?? before.method, + args: paramsToArgs(before.params), + // Show the trace-style label (`Element.fill("x")`); the command name above + // still drives the UI's category colour and icon. + title: + before.title ?? actionLabel(before.class, before.method, before.params), + startTime: before.startTime, + timestamp: endTime + } + const error = commandError(before, after, afters) + if (error) { + command.error = { name: 'Error', message: error.message } + } + if (after?.result !== undefined) { + command.result = after.result + } + const callSource = stackToCallSource(before.stack) + if (callSource) { + command.callSource = callSource + } + const frame = nearestFrame(frames, command.timestamp) + if (frame) { + command.screenshot = frame.screenshot + } + return command } function buildCommands( - events: CategorizedEvents, - frames: TracePlayerFrame[], - wallTime: number -): { commands: CommandLog[]; maxOffset: number } { - const commands: CommandLog[] = [] - let maxOffset = 0 + events: MergedEvents, + frames: TracePlayerFrame[] +): { + commands: CommandLog[] + maxTime: number + indexByCallId: Map<string, number> +} { + const entries: { callId: string; command: CommandLog }[] = [] + const structural = collectStructuralIds(events.befores) + let maxTime = 0 for (const [callId, before] of events.befores) { - const after = events.afters.get(callId) - const endOffset = after?.endTime ?? before.startTime - maxOffset = Math.max(maxOffset, endOffset) - const command: CommandLog = { - command: - REVERSE_ACTION_MAP[`${before.class}.${before.method}`] ?? before.method, - args: paramsToArgs(before.params), - // Show the Playwright/vibium label (`Element.fill("x")`); the command - // name above still drives the UI's category colour and icon. - title: actionLabel(before.class, before.method, before.params), - startTime: wallTime + before.startTime, - timestamp: wallTime + endOffset - } - if (after?.error) { - command.error = { name: 'Error', message: after.error.message } - } - const frame = nearestFrame(frames, command.timestamp) - if (frame) { - command.screenshot = frame.screenshot + // Group markers are structure, not actions — as command rows their end + // timestamp ties with the last action and steals the active highlight. + if (isStructuralBefore(before, structural)) { + continue } - commands.push(command) + const after = events.afters.get(callId) + const command = reconstructCommand(before, after, events.afters, frames) + maxTime = Math.max(maxTime, command.timestamp) + entries.push({ callId, command }) } - commands.sort((a, b) => a.timestamp - b.timestamp) - return { commands, maxOffset } + entries.sort((a, b) => a.command.timestamp - b.command.timestamp) + return { + commands: entries.map((entry) => entry.command), + maxTime, + indexByCallId: new Map(entries.map((entry, index) => [entry.callId, index])) + } +} + +function parseNetworkStreams( + files: Record<string, Uint8Array>, + names: string[] +): NetworkRequest[] { + return names + .flatMap((name) => parseNdjson(strFromU8(files[name]))) + .filter((entry) => typeof entry.snapshot === 'object' && entry.snapshot) + .map((entry, index) => + harToNetworkRequest(entry.snapshot as HarSnapshot, index, files) + ) +} + +function earliestWallTime(ctxs: ContextOptionsEvent[]): number { + const wallTimes = ctxs + .map((ctx) => ctx.wallTime) + .filter((time) => Number.isFinite(time)) + return wallTimes.length ? Math.min(...wallTimes) : 0 } /** Parse an in-memory trace.zip buffer into a player payload. Pure (no I/O). */ export function parseTraceZip(zip: Uint8Array): TracePlayerData { const files = unzipSync(zip) - const readEntry = (name: string) => - files[name] ? strFromU8(files[name]) : '' - const categorized = categorizeEvents(parseNdjson(readEntry('trace.trace'))) - const wallTime = categorized.ctx?.wallTime ?? 0 - const { frames, maxOffset: frameMax } = buildFrames( - files, - categorized.frameEvents, - wallTime - ) - const { commands, maxOffset: cmdMax } = buildCommands( - categorized, - frames, - wallTime - ) - const networkRequests = parseNdjson(readEntry('trace.network')).map( - (entry, index) => - harToNetworkRequest((entry as { snapshot: HarSnapshot }).snapshot, index) + const names = Object.keys(files).sort() + const entriesWith = (suffix: string) => + names.filter((name) => name.endsWith(suffix)) + const streams = entriesWith(TRACE_STREAM_SUFFIX).map((name) => { + const stream = categorizeEvents(parseNdjson(strFromU8(files[name]))) + rebaseToEpoch(stream) + return stream + }) + const merged = mergeStreams(streams) + for (const name of entriesWith(STACKS_STREAM_SUFFIX)) { + attachSidecarStacks(merged.befores, strFromU8(files[name])) + } + const { frames, maxTime: frameMax } = buildFrames(files, merged.frameEvents) + const { + commands, + maxTime: cmdMax, + indexByCallId + } = buildCommands(merged, frames) + const groups = buildActionTree( + merged.befores, + merged.afters, + commands, + indexByCallId ) + const startTime = earliestWallTime(merged.ctxs) const trace: TraceLog = { mutations: [], logs: [], - consoleLogs: [], - networkRequests, - metadata: buildMetadata(categorized.ctx), + consoleLogs: buildConsoleLogs(merged.consoleEvents), + networkRequests: parseNetworkStreams( + files, + entriesWith(NETWORK_STREAM_SUFFIX) + ), + metadata: buildMetadata( + merged.ctxs.find((ctx) => ctx.browserName) ?? merged.ctxs[0] + ), commands, - sources: {}, + sources: buildSources(merged.befores.values(), files), suites: [] } return { trace, frames, - startTime: wallTime, - duration: Math.max(frameMax, cmdMax) + startTime, + duration: Math.max(0, Math.max(frameMax, cmdMax) - startTime), + ...(groups ? { groups } : {}) } } diff --git a/packages/backend/tests/trace-reader-asserts.test.ts b/packages/backend/tests/trace-reader-asserts.test.ts new file mode 100644 index 00000000..a32a17f3 --- /dev/null +++ b/packages/backend/tests/trace-reader-asserts.test.ts @@ -0,0 +1,107 @@ +import { describe, it, expect } from 'vitest' +import { zipSync, strToU8 } from 'fflate' +import { parseTraceZip } from '../src/trace-reader.js' +import { REVERSE_ACTION_MAP } from '../src/trace-reader-constants.js' + +const WALL_TIME = 1_000_000 + +const toNdjson = (events: object[]): Uint8Array => + strToU8(events.map((event) => JSON.stringify(event)).join('\n') + '\n') + +// Assert events exactly as core's exporter writes them: semantic +// actual/expected/message params plus the numeric echo of the raw args. +function assertFixtureZip(): Uint8Array { + const events = [ + { + type: 'context-options', + wallTime: WALL_TIME, + browserName: 'chrome', + contextId: 'context@ab12cd34', + options: {} + }, + { + type: 'before', + callId: 'call@1', + startTime: 100, + class: 'Assert', + method: 'strictEqual', + params: { '0': 'a', '1': 'a', actual: 'a', expected: 'a' }, + title: 'assert.strictEqual("a", "a")', + apiName: 'assert.strictEqual' + }, + { type: 'after', callId: 'call@1', endTime: 110 }, + { + type: 'before', + callId: 'call@2', + startTime: 200, + class: 'Assert', + method: 'strictEqual', + params: { '0': 'a', '1': 'b', actual: 'a', expected: 'b' }, + title: 'assert.strictEqual("a", "b")', + apiName: 'assert.strictEqual' + }, + { + type: 'after', + callId: 'call@2', + endTime: 210, + error: { message: 'a !== b' } + }, + { + type: 'before', + callId: 'call@3', + startTime: 300, + class: 'Assert', + method: 'toBe', + params: { '0': 1, '1': 2, actual: 1, expected: 2 }, + title: 'expect.toBe(1, 2)', + apiName: 'assert.toBe' + }, + { + type: 'after', + callId: 'call@3', + endTime: 310, + error: { message: '1 !== 2' } + } + ] + return zipSync({ 'trace.trace': toNdjson(events) }) +} + +describe('REVERSE_ACTION_MAP assert entries', () => { + it('maps every tracked Assert.<m> action back to assert.<m>', () => { + expect(REVERSE_ACTION_MAP['Assert.strictEqual']).toBe('assert.strictEqual') + expect(REVERSE_ACTION_MAP['Assert.match']).toBe('assert.match') + expect(REVERSE_ACTION_MAP['Assert.doesNotMatch']).toBe( + 'assert.doesNotMatch' + ) + // Runner entries stay intact. + expect(REVERSE_ACTION_MAP['Page.navigate']).toBe('url') + }) +}) + +describe('parseTraceZip with assert actions', () => { + it('reads Assert rows back as assert.<m> commands with args round-tripped', () => { + const { trace } = parseTraceZip(assertFixtureZip()) + expect(trace.commands.map((c) => c.command)).toEqual([ + 'assert.strictEqual', + 'assert.strictEqual', + // Untracked assert methods (synthesized expect matchers) fall back to + // the bare method name. + 'toBe' + ]) + expect(trace.commands[0].args).toEqual(['a', 'a']) + expect(trace.commands[1].args).toEqual(['a', 'b']) + }) + + it('keeps the pass/fail split: error only on the failing assert', () => { + const { trace } = parseTraceZip(assertFixtureZip()) + expect(trace.commands[0].error).toBeUndefined() + expect(trace.commands[1].error?.message).toBe('a !== b') + expect(trace.commands[2].error?.message).toBe('1 !== 2') + }) + + it('preserves the exporter-written assert titles', () => { + const { trace } = parseTraceZip(assertFixtureZip()) + expect(trace.commands[0].title).toBe('assert.strictEqual("a", "a")') + expect(trace.commands[2].title).toBe('expect.toBe(1, 2)') + }) +}) diff --git a/packages/backend/tests/trace-reader-bodies.test.ts b/packages/backend/tests/trace-reader-bodies.test.ts new file mode 100644 index 00000000..62f2cec3 --- /dev/null +++ b/packages/backend/tests/trace-reader-bodies.test.ts @@ -0,0 +1,143 @@ +import { createHash } from 'node:crypto' +import { describe, it, expect } from 'vitest' +import { zipSync, strToU8 } from 'fflate' +import { parseTraceZip } from '../src/trace-reader.js' + +const sha1 = (data: string): string => + createHash('sha1').update(data).digest('hex') + +const JSON_BODY = '{"answer":42}' +const HTML_BODY = '<p>hello</p>' +const PNG_BODY = 'not-really-a-png' + +const toNdjson = (events: object[]): Uint8Array => + strToU8(events.map((event) => JSON.stringify(event)).join('\n') + '\n') + +function networkEntry(url: string, content: Record<string, unknown>): object { + return { + type: 'resource-snapshot', + snapshot: { + startedDateTime: '2026-01-01T00:00:00.000Z', + time: 50, + request: { + method: 'GET', + url, + httpVersion: 'HTTP/1.1', + cookies: [], + headers: [], + queryString: [], + headersSize: -1, + bodySize: -1 + }, + response: { + status: 200, + statusText: 'OK', + httpVersion: 'HTTP/1.1', + cookies: [], + headers: [], + content, + redirectURL: '', + headersSize: -1, + bodySize: -1 + }, + cache: {}, + timings: { send: 0, wait: 50, receive: 0 } + } + } +} + +function fixtureZip(): Uint8Array { + const events = [ + { + type: 'context-options', + wallTime: 1_000_000, + browserName: 'chrome', + contextId: 'context@abcd1234', + options: { viewport: { width: 800, height: 600 } } + } + ] + const network = [ + networkEntry('https://api.example.com/data', { + size: JSON_BODY.length, + mimeType: 'application/json', + _sha1: sha1(JSON_BODY) + }), + networkEntry('https://example.com/inline', { + size: 2, + mimeType: 'text/plain', + text: 'hi' + }), + networkEntry('https://example.com/logo.png', { + size: PNG_BODY.length, + mimeType: 'image/png', + _sha1: sha1(PNG_BODY) + }), + // Foreign writers suffix the resource name with a mime extension. + networkEntry('https://example.com/page', { + size: HTML_BODY.length, + mimeType: 'text/html; charset=utf-8', + _sha1: `${sha1(HTML_BODY)}.html` + }), + networkEntry('https://example.com/missing', { + size: 3, + mimeType: 'application/json', + _sha1: sha1('never-stored') + }), + networkEntry('https://example.com/prefers-text', { + size: 4, + mimeType: 'application/json', + text: '"inline-wins"', + _sha1: sha1(JSON_BODY) + }), + networkEntry('https://example.com/binary-inline', { + size: 4, + mimeType: 'application/octet-stream', + text: 'AAAA', + encoding: 'base64' + }) + ] + return zipSync({ + 'trace.trace': toNdjson(events), + 'trace.network': toNdjson(network), + [`resources/${sha1(JSON_BODY)}`]: strToU8(JSON_BODY), + [`resources/${sha1(PNG_BODY)}`]: strToU8(PNG_BODY), + [`resources/${sha1(HTML_BODY)}.html`]: strToU8(HTML_BODY) + }) +} + +function bodyByUrl(url: string): string | undefined { + const data = parseTraceZip(fixtureZip()) + const request = data.trace.networkRequests.find((r) => r.url === url) + expect(request).toBeDefined() + return request?.responseBody +} + +describe('trace-reader response bodies', () => { + it('restores the body from a content-addressed resource', () => { + expect(bodyByUrl('https://api.example.com/data')).toBe(JSON_BODY) + }) + + it('restores the body from inline text', () => { + expect(bodyByUrl('https://example.com/inline')).toBe('hi') + }) + + it('prefers inline text over the _sha1 resource', () => { + expect(bodyByUrl('https://example.com/prefers-text')).toBe('"inline-wins"') + }) + + it('restores extension-suffixed _sha1 resource names', () => { + expect(bodyByUrl('https://example.com/page')).toBe(HTML_BODY) + }) + + it('skips binary mime types', () => { + expect(bodyByUrl('https://example.com/logo.png')).toBeUndefined() + }) + + it('skips base64-encoded inline text', () => { + expect(bodyByUrl('https://example.com/binary-inline')).toBeUndefined() + }) + + it('leaves the body undefined when the resource is missing', () => { + expect(bodyByUrl('https://example.com/missing')).toBeUndefined() + }) +}) diff --git a/packages/backend/tests/trace-reader.test.ts b/packages/backend/tests/trace-reader.test.ts index f2aa880e..25f9a2f9 100644 --- a/packages/backend/tests/trace-reader.test.ts +++ b/packages/backend/tests/trace-reader.test.ts @@ -1,12 +1,39 @@ +import { createHash } from 'node:crypto' import { describe, it, expect } from 'vitest' import { zipSync, strToU8 } from 'fflate' +import type { + TraceActionChild, + TraceActionGroupNode +} from '@wdio/devtools-shared' import { parseTraceZip } from '../src/trace-reader.js' +import { buildSources, stackToCallSource } from '../src/trace-reader-utils.js' +import type { BeforeEvent } from '../src/trace-reader-types.js' + +function allGroups(children: TraceActionChild[]): TraceActionGroupNode[] { + return children.flatMap((child) => + 'group' in child ? [child.group, ...allGroups(child.group.children)] : [] + ) +} + +function commandIndices(children: TraceActionChild[]): number[] { + return children.flatMap((child) => + 'group' in child + ? commandIndices(child.group.children) + : [child.commandIndex] + ) +} + +const SPEC_PATH = '/specs/login.ts' +const SPEC_SHA1 = createHash('sha1').update(SPEC_PATH).digest('hex') const WALL_TIME = 1_000_000 const IMG1 = Buffer.from('frame-one').toString('base64') const IMG2 = Buffer.from('frame-two').toString('base64') const IMG3 = Buffer.from('frame-three').toString('base64') +const toNdjson = (events: object[]): Uint8Array => + strToU8(events.map((event) => JSON.stringify(event)).join('\n') + '\n') + // Build a trace.zip matching the writer's format (core/trace-exporter.ts) // directly, so the reader is exercised without importing core (CLAUDE.md §2.2). function fixtureZip(): Uint8Array { @@ -26,23 +53,34 @@ function fixtureZip(): Uint8Array { method: 'navigate', params: { url: 'https://example.com' } }, - { type: 'after', callId: 'call@1', endTime: 50 }, + { type: 'after', callId: 'call@1', endTime: 50, result: 'example.com' }, { type: 'before', callId: 'call@2', startTime: 100, class: 'Element', method: 'fill', - params: { selector: '#name', value: 'vishnu' } + params: { selector: '#name', value: 'vishnu' }, + parentId: 'call@0' }, { type: 'after', callId: 'call@2', endTime: 160 }, + { + type: 'before', + callId: 'call@0', + startTime: 90, + class: 'Tracing', + method: 'tracingGroup', + params: { name: 'my test' } + }, { type: 'before', callId: 'call@3', startTime: 200, class: 'Element', method: 'click', - params: { selector: '#submit' } + params: { selector: '#submit' }, + stack: [{ file: SPEC_PATH, line: 42, column: 0 }], + parentId: 'call@0' }, { type: 'after', @@ -50,6 +88,7 @@ function fixtureZip(): Uint8Array { endTime: 260, error: { message: 'boom' } }, + { type: 'after', callId: 'call@0', endTime: 260 }, { type: 'screencast-frame', pageId: 'page@abcd1234', @@ -73,7 +112,18 @@ function fixtureZip(): Uint8Array { width: 1024, height: 768, timestamp: 260 - } + }, + { + type: 'console', + time: 120, + pageId: 'page@abcd1234', + messageType: 'warning', + text: 'low disk', + args: [{ preview: 'low disk', value: 'low disk' }], + location: { url: '', lineNumber: 0, columnNumber: 0 } + }, + { type: 'stdout', timestamp: 130, text: 'spec started', source: 'test' }, + { type: 'stderr', timestamp: 140, text: 'worker warning' } ] const networkEntry = { type: 'resource-snapshot', @@ -94,9 +144,7 @@ function fixtureZip(): Uint8Array { } } return zipSync({ - 'trace.trace': strToU8( - events.map((e) => JSON.stringify(e)).join('\n') + '\n' - ), + 'trace.trace': toNdjson(events), 'trace.network': strToU8(JSON.stringify(networkEntry)), 'resources/page@abcd1234-0.jpeg': new Uint8Array(Buffer.from('frame-one')), 'resources/page@abcd1234-160.jpeg': new Uint8Array( @@ -104,7 +152,8 @@ function fixtureZip(): Uint8Array { ), 'resources/page@abcd1234-260.jpeg': new Uint8Array( Buffer.from('frame-three') - ) + ), + [`resources/src@${SPEC_SHA1}.txt`]: strToU8('it("logs in", () => {})') }) } @@ -159,9 +208,392 @@ describe('parseTraceZip', () => { (trace.metadata.capabilities as { browserName: string }).browserName ).toBe('chrome') expect(trace.metadata.sessionId).toBe('abcd1234') - expect(trace.consoleLogs).toEqual([]) expect(trace.mutations).toEqual([]) expect(trace.suites).toEqual([]) - expect(trace.sources).toEqual({}) + }) + + it('restores callSource and sources from stack frames + src resources', () => { + const { trace } = parseTraceZip(fixtureZip()) + const click = trace.commands.find((c) => c.command === 'click') + expect(click?.callSource).toBe(`${SPEC_PATH}:42`) + expect(trace.commands.find((c) => c.command === 'url')?.callSource).toBe( + undefined + ) + expect(trace.sources).toEqual({ + [SPEC_PATH]: 'it("logs in", () => {})' + }) + }) + + it('restores the command result from the after event', () => { + const { trace } = parseTraceZip(fixtureZip()) + expect(trace.commands.find((c) => c.command === 'url')?.result).toBe( + 'example.com' + ) + }) + + it('skips tracing group markers so the last command stays the last action', () => { + const { trace } = parseTraceZip(fixtureZip()) + expect(trace.commands.some((c) => c.command === 'tracingGroup')).toBe(false) + expect(trace.commands[trace.commands.length - 1].command).toBe('click') + }) + + it('nests parentId-linked commands under the tracing group node', () => { + const { trace, groups } = parseTraceZip(fixtureZip()) + expect(groups).toBeDefined() + // Root is chronological: url (start 0) before the group (start 90). + expect(groups).toHaveLength(2) + expect(groups?.[0]).toEqual({ commandIndex: 0 }) + expect(trace.commands[0].command).toBe('url') + const [group] = allGroups(groups ?? []) + expect(group.title).toBe('my test') + expect(group.startTime).toBe(WALL_TIME + 90) + expect(group.endTime).toBe(WALL_TIME + 260) + const children = commandIndices(group.children) + expect(children.map((i) => trace.commands[i].command)).toEqual([ + 'setValue', + 'click' + ]) + }) + + it('marks the group failed when a child command errored', () => { + const { groups } = parseTraceZip(fixtureZip()) + const [group] = allGroups(groups ?? []) + expect(group.failed).toBe(true) + }) + + it('reconstructs console logs from console and stdio events', () => { + const { trace } = parseTraceZip(fixtureZip()) + expect(trace.consoleLogs).toEqual([ + { + type: 'warn', + args: ['low disk'], + timestamp: WALL_TIME + 120, + source: 'browser' + }, + { + type: 'log', + args: ['spec started'], + timestamp: WALL_TIME + 130, + source: 'test' + }, + { + type: 'error', + args: ['worker warning'], + timestamp: WALL_TIME + 140, + source: 'terminal' + } + ]) + }) +}) + +const FOREIGN_SPEC = '/specs/foreign.spec.ts' +const FOREIGN_SPEC_SHA1 = createHash('sha1').update(FOREIGN_SPEC).digest('hex') +const RUNNER_WALL = 2_000_000 +const LIB_WALL = 2_000_400 +// Both streams share one monotonic clock: wallTime - monotonicTime is equal. +const EPOCH_ANCHOR = RUNNER_WALL - 500 +const FOREIGN_IMG_A = Buffer.from('foreign-frame-a').toString('base64') +const FOREIGN_IMG_B = Buffer.from('foreign-frame-b').toString('base64') + +// Emulates a foreign tool's zip: a runner stream (`test.trace`) wrapping +// library calls, a per-context stream, monotonic timestamps, bare-sha1 +// screencast refs, and a `.stacks` sidecar. +function foreignFixtureZip(): Uint8Array { + const runnerEvents = [ + { + type: 'context-options', + origin: 'testRunner', + wallTime: RUNNER_WALL, + monotonicTime: 500, + browserName: '', + options: {} + }, + { + type: 'before', + callId: 'hook@1', + stepId: 'hook@1', + startTime: 600, + class: 'Test', + method: 'hook', + title: 'Before Hooks', + params: {} + }, + { + type: 'before', + callId: 'fixture@2', + stepId: 'fixture@2', + parentId: 'hook@1', + startTime: 610, + class: 'Test', + method: 'fixture', + title: 'Fixture "browser"', + params: {} + }, + { + type: 'before', + callId: 'step@3', + stepId: 'step@3', + parentId: 'fixture@2', + startTime: 700, + class: 'Test', + method: 'step', + title: 'Click "go"', + params: {}, + stack: [{ file: FOREIGN_SPEC, line: 10, column: 5 }] + }, + { + type: 'before', + callId: 'expect@5', + stepId: 'expect@5', + startTime: 950, + class: 'Test', + method: 'expect', + title: 'Expect "toBeVisible"', + params: {} + }, + { + type: 'after', + callId: 'step@3', + endTime: 760, + error: { message: 'wrapped failure' } + }, + { type: 'after', callId: 'fixture@2', endTime: 780 }, + { type: 'after', callId: 'hook@1', endTime: 800 }, + { type: 'after', callId: 'expect@5', endTime: 980 }, + { type: 'error', message: 'ignored non-action event' } + ] + const libraryEvents = [ + { + type: 'context-options', + origin: 'library', + wallTime: LIB_WALL, + monotonicTime: 900, + browserName: 'chromium', + contextId: 'browser-context@f00d', + options: { viewport: { width: 800, height: 600 } } + }, + { + type: 'before', + callId: 'call@10', + stepId: 'step@3', + startTime: 710, + class: 'Frame', + method: 'click', + params: { selector: '#go' } + }, + { type: 'after', callId: 'call@10', endTime: 750 }, + { + type: 'before', + callId: 'call@12', + stepId: 'step@99', + startTime: 800, + class: 'Frame', + method: 'fill', + params: { selector: '#q', value: 'hi' } + }, + { + type: 'after', + callId: 'call@12', + endTime: 900, + error: { message: 'nope' } + }, + { + type: 'screencast-frame', + pageId: 'page@f00d', + sha1: 'aaa111', + width: 800, + height: 600, + timestamp: 720, + frameSwapWallTime: EPOCH_ANCHOR + 720 + }, + { + type: 'screencast-frame', + pageId: 'page@f00d', + sha1: 'bbb222', + width: 800, + height: 600, + timestamp: 880, + frameSwapWallTime: EPOCH_ANCHOR + 880 + }, + { + type: 'console', + messageType: 'warning', + text: 'careful', + args: [{ preview: 'careful', value: 'careful' }], + time: 820, + pageId: 'page@f00d' + }, + { type: 'log', time: 830, message: 'ignored log event' }, + { type: 'frame-snapshot', snapshot: {} }, + { type: 'input', inputSnapshot: {} } + ] + const networkEntry = { + type: 'resource-snapshot', + snapshot: { + startedDateTime: new Date(EPOCH_ANCHOR + 715).toISOString(), + time: 30, + request: { + method: 'GET', + url: 'https://x.dev/api', + headers: [] + }, + response: { + status: 200, + statusText: 'OK', + headers: [], + content: { size: 10, mimeType: 'text/html' } + } + } + } + const stacks = { + files: [FOREIGN_SPEC], + stacks: [ + [10, [[0, 10, 5, '']]], + [12, [[0, 33, 7, '']]] + ] + } + return zipSync({ + 'test.trace': toNdjson(runnerEvents), + '0-trace.trace': toNdjson(libraryEvents), + '0-trace.network': strToU8(JSON.stringify(networkEntry)), + '0-trace.stacks': strToU8(JSON.stringify(stacks)), + 'resources/aaa111.jpeg': new Uint8Array(Buffer.from('foreign-frame-a')), + 'resources/bbb222.png': new Uint8Array(Buffer.from('foreign-frame-b')), + [`resources/src@${FOREIGN_SPEC_SHA1}.txt`]: strToU8('test("foreign")') + }) +} + +describe('parseTraceZip with a foreign multi-stream zip', () => { + it('merges streams, drops runner wrappers and container steps', () => { + const { trace } = parseTraceZip(foreignFixtureZip()) + expect(trace.commands.map((c) => c.command)).toEqual([ + 'click', + 'fill', + 'expect' + ]) + expect(trace.commands[0].args).toEqual(['#go']) + expect(trace.commands[0].title).toBe('Frame.click("#go")') + expect(trace.commands[1].error?.message).toBe('nope') + expect(trace.commands[2].title).toBe('Expect "toBeVisible"') + }) + + it('rebases monotonic timestamps onto the wall clock', () => { + const { trace, frames, startTime, duration } = + parseTraceZip(foreignFixtureZip()) + expect(startTime).toBe(RUNNER_WALL) + expect(trace.commands[0].startTime).toBe(EPOCH_ANCHOR + 710) + expect(trace.commands[0].timestamp).toBe(EPOCH_ANCHOR + 750) + expect(frames.map((f) => f.timestamp)).toEqual([ + EPOCH_ANCHOR + 720, + EPOCH_ANCHOR + 880 + ]) + expect(duration).toBe(EPOCH_ANCHOR + 980 - RUNNER_WALL) + }) + + it('resolves bare-sha1 screencast refs via image extension fallbacks', () => { + const { frames, trace } = parseTraceZip(foreignFixtureZip()) + expect(frames.map((f) => f.screenshot)).toEqual([ + FOREIGN_IMG_A, + FOREIGN_IMG_B + ]) + expect(trace.commands[0].screenshot).toBe(FOREIGN_IMG_A) + }) + + it('merges network streams and console events, picks the browser context', () => { + const { trace } = parseTraceZip(foreignFixtureZip()) + expect(trace.networkRequests).toHaveLength(1) + expect(trace.networkRequests[0].url).toBe('https://x.dev/api') + expect(trace.consoleLogs).toEqual([ + { + type: 'warn', + args: ['careful'], + timestamp: EPOCH_ANCHOR + 820, + source: 'browser' + } + ]) + expect(trace.metadata.viewport?.width).toBe(800) + expect( + (trace.metadata.capabilities as { browserName: string }).browserName + ).toBe('chromium') + }) + + it('restores callSource and sources from the sidecar stacks entry', () => { + const { trace } = parseTraceZip(foreignFixtureZip()) + expect(trace.commands[0].callSource).toBe(`${FOREIGN_SPEC}:10`) + expect(trace.commands[1].callSource).toBe(`${FOREIGN_SPEC}:33`) + expect(trace.sources).toEqual({ [FOREIGN_SPEC]: 'test("foreign")' }) + }) + + it('nests runner steps by parentId and stepId-linked calls under them', () => { + const { trace, groups } = parseTraceZip(foreignFixtureZip()) + // Root chronological: hook (600) < orphan fill (800) < runner expect (950). + expect(groups).toHaveLength(3) + const [hookChild, fillChild, expectChild] = groups ?? [] + expect('group' in hookChild! && hookChild.group.title).toBe('Before Hooks') + expect(fillChild).toEqual({ commandIndex: 1 }) + expect(trace.commands[1].command).toBe('fill') + expect(expectChild).toEqual({ commandIndex: 2 }) + const [hook, fixture, step] = allGroups(groups ?? []) + expect(fixture.title).toBe('Fixture "browser"') + expect(step.title).toBe('Click "go"') + expect(hook.startTime).toBe(EPOCH_ANCHOR + 600) + expect(hook.endTime).toBe(EPOCH_ANCHOR + 800) + expect(commandIndices(step.children)).toEqual([0]) + expect(trace.commands[0].command).toBe('click') + }) + + it('propagates a wrapper-held error to the leaf and rolls failure up', () => { + const { trace, groups } = parseTraceZip(foreignFixtureZip()) + expect(trace.commands[0].error?.message).toBe('wrapped failure') + const [hook, fixture, step] = allGroups(groups ?? []) + expect(step.failed).toBe(true) + expect(fixture.failed).toBe(true) + expect(hook.failed).toBe(true) + }) +}) + +describe('glued callSource recovery from older zips', () => { + it('recovers the line glued onto the file path', () => { + expect( + stackToCallSource([{ file: '/x/steps.ts:17', line: 21, column: 0 }]) + ).toBe('/x/steps.ts:17') + }) + + it('recovers from a doubly glued file:line:column path', () => { + expect( + stackToCallSource([{ file: '/x/steps.ts:17:21', line: 0, column: 0 }]) + ).toBe('/x/steps.ts:17') + }) + + it('keeps Windows drive specs intact', () => { + expect( + stackToCallSource([{ file: 'C:\\proj\\steps.ts', line: 5, column: 1 }]) + ).toBe('C:\\proj\\steps.ts:5') + expect( + stackToCallSource([{ file: 'C:\\proj\\steps.ts:17', line: 21 }]) + ).toBe('C:\\proj\\steps.ts:17') + }) + + it('leaves clean frames unchanged', () => { + expect( + stackToCallSource([{ file: '/x/steps.ts', line: 42, column: 0 }]) + ).toBe('/x/steps.ts:42') + }) + + it('looks up sources under the unglued path sha1', () => { + const clean = '/specs/glued.ts' + const sha1 = createHash('sha1').update(clean).digest('hex') + const before: BeforeEvent = { + type: 'before', + callId: 'call@1', + startTime: 0, + class: 'Element', + method: 'click', + stack: [{ file: `${clean}:17`, line: 21, column: 0 }] + } + const sources = buildSources([before], { + [`resources/src@${sha1}.txt`]: strToU8('glued source') + }) + expect(sources).toEqual({ [clean]: 'glued source' }) }) }) diff --git a/packages/core/src/action-mapping.ts b/packages/core/src/action-mapping.ts index 6a9711cf..368ca8e3 100644 --- a/packages/core/src/action-mapping.ts +++ b/packages/core/src/action-mapping.ts @@ -4,25 +4,76 @@ // existing devtools UI uses its own denylist (`INTERNAL_COMMANDS`) — this map // is for the trace.zip exporter to filter + rename in one step. -import { ACTION_MAP, type TraceAction } from '@wdio/devtools-shared' +import { + ACTION_MAP, + ASSERT_ACTION_CLASS, + mapAssertCommand, + type TraceAction +} from '@wdio/devtools-shared' export type { TraceAction } +export { ASSERT_ACTION_CLASS, mapAssertCommand } // Excluded by design: // clearValue / addValue — WDIO fires these inside setValue (duplicate events). // executeScript — Selenium's `until` polling fires it ~50ms; also recurses // because @wdio/elements uses executeScript inside captureActionSnapshot. // WDIO's user-facing `execute`/`executeAsync` are still captured. +// $ / $$ / findElement(s) / getElement(s) — locator resolution fires on every +// element access; high-frequency internal machinery, not a timeline step. +// Passing expect-webdriverio matchers — never reach the command log (only +// failures do, via the reporter); surfacing them is a per-adapter change. export function mapCommandToAction(command: string): TraceAction | null { - return ACTION_MAP[command] ?? null + return ACTION_MAP[command] ?? mapAssertCommand(command) +} + +const ASSERT_TITLE_VALUE_MAX = 40 + +function formatAssertValue(value: unknown): string { + let text: string + if (typeof value === 'object' && value !== null) { + try { + text = JSON.stringify(value) + } catch { + text = String(value) + } + } else { + text = typeof value === 'string' ? JSON.stringify(value) : String(value) + } + return text.length > ASSERT_TITLE_VALUE_MAX + ? `${text.slice(0, ASSERT_TITLE_VALUE_MAX - 1)}…` + : text +} + +// Prefer normalized actual/expected params (nightwatch collapses them into +// the result); fall back to the first two positional args (node:assert order). +function formatAssertTitle( + action: TraceAction, + args: unknown[], + params?: Record<string, unknown>, + command?: string +): string { + const values = + params && ('actual' in params || 'expected' in params) + ? [params.actual, params.expected] + : args.slice(0, 2) + const label = values + .filter((value) => value !== undefined) + .map(formatAssertValue) + .join(', ') + return `${command ?? `assert.${action.method}`}(${label})` } export function formatActionTitle( action: TraceAction, args: unknown[], - params?: Record<string, unknown> + params?: Record<string, unknown>, + command?: string ): string { + if (action.class === ASSERT_ACTION_CLASS) { + return formatAssertTitle(action, args, params, command) + } const firstArg = args[0] ?? params?.selector if (firstArg === undefined) { return `${action.class}.${action.method}()` diff --git a/packages/core/src/assert-patcher.ts b/packages/core/src/assert-patcher.ts index 10e18a4b..c6b91bf2 100644 --- a/packages/core/src/assert-patcher.ts +++ b/packages/core/src/assert-patcher.ts @@ -1,6 +1,10 @@ import { createRequire } from 'node:module' +import { TRACKED_ASSERT_METHODS, type CommandLog } from '@wdio/devtools-shared' import { getCallSourceFromStack } from './stack.js' import { toError } from './error.js' +import { stripAnsi } from './console.js' + +export { TRACKED_ASSERT_METHODS } const require = createRequire(import.meta.url) @@ -9,26 +13,6 @@ export const ASSERT_PATCHED_SYMBOL = Symbol.for( '@wdio/devtools-core/assert-patched' ) -/** node:assert methods the patcher wraps. */ -export const TRACKED_ASSERT_METHODS = [ - 'equal', - 'strictEqual', - 'deepEqual', - 'deepStrictEqual', - 'notEqual', - 'notStrictEqual', - 'notDeepEqual', - 'notDeepStrictEqual', - 'ok', - 'fail', - 'throws', - 'doesNotThrow', - 'rejects', - 'doesNotReject', - 'match', - 'doesNotMatch' -] as const - /** * Minimum shape `patchNodeAssert` emits. Adapters that need extra bookkeeping * (selenium adds `fromElement` and `rawResult`) wrap the callback to extend @@ -68,57 +52,152 @@ export function safeSerializeAssertArg(value: unknown): unknown { return value } +function makeAssertEmitters( + methodName: string, + args: unknown[], + onCommand: (cmd: CapturedAssert) => void +): { passed: () => void; failed: (err: unknown) => void } { + const callInfo = getCallSourceFromStack() + // No user-code frame means the assert came from a dependency or framework + // internal, not the user's test — drop it so it never reaches the trace. + if (callInfo.filePath === undefined) { + return { passed: () => {}, failed: () => {} } + } + const startedAt = Date.now() + const sanitizedArgs = args.map(safeSerializeAssertArg) + const emit = (result: 'passed' | undefined, error: Error | undefined) => + onCommand({ + command: `assert.${methodName}`, + args: sanitizedArgs, + result, + error, + callSource: callInfo.callSource, + timestamp: startedAt + }) + return { + passed: () => emit('passed', undefined), + failed: (err: unknown) => emit(undefined, toError(err)) + } +} + function makePatchedAssertMethod( methodName: string, + assertObj: Record<string | symbol, unknown>, original: (...a: unknown[]) => unknown, onCommand: (cmd: CapturedAssert) => void ): (...args: unknown[]) => unknown { return function patchedAssert(this: unknown, ...args: unknown[]) { - const callInfo = getCallSourceFromStack() - const startedAt = Date.now() - const sanitizedArgs = args.map(safeSerializeAssertArg) - const passed = () => - onCommand({ - command: `assert.${methodName}`, - args: sanitizedArgs, - result: 'passed', - error: undefined, - callSource: callInfo.callSource, - timestamp: startedAt - }) - const failed = (err: unknown) => - onCommand({ - command: `assert.${methodName}`, - args: sanitizedArgs, - result: undefined, - error: toError(err), - callSource: callInfo.callSource, - timestamp: startedAt - }) - + const { passed, failed } = makeAssertEmitters(methodName, args, onCommand) + let result: unknown + // Node's internalMatch dispatches on `fn === assert.match` (Node ≤20), so + // a wrapper installed on that property silently inverts `match` into + // `doesNotMatch`. Restore the original binding for the call so identity + // checks inside node:assert see the real method. + assertObj[methodName] = original try { - const result = original.apply(this, args) - // Async assert methods (rejects/doesNotReject) return a Promise. - const maybe = result as { then?: unknown } | null | undefined - if (maybe && typeof maybe.then === 'function') { - return (result as Promise<unknown>).then( - (v) => { - passed() - return v - }, - (err) => { - failed(err) - throw err - } - ) - } - passed() - return result + result = original.apply(this, args) } catch (err) { failed(err) throw err + } finally { + assertObj[methodName] = patchedAssert + } + // Async assert methods (rejects/doesNotReject) return a Promise. + const maybe = result as { then?: unknown } | null | undefined + if (maybe && typeof maybe.then === 'function') { + return (result as Promise<unknown>).then( + (v) => { + passed() + return v + }, + (err) => { + failed(err) + throw err + } + ) + } + passed() + return result + } +} + +/** + * Convert a `CapturedAssert` into the shared `CommandLog` shape adapters push + * into their session capturer. Asserts are effectively instantaneous, so the + * capture timestamp doubles as `startTime`. + */ +export function capturedAssertToCommandLog( + cmd: CapturedAssert, + testUid?: string +): CommandLog { + const entry: CommandLog = { + command: cmd.command, + args: cmd.args, + result: cmd.result, + timestamp: cmd.timestamp, + startTime: cmd.timestamp + } + if (cmd.error) { + entry.error = { + name: cmd.error.name, + message: cmd.error.message, + stack: cmd.error.stack } } + if (cmd.callSource) { + entry.callSource = cmd.callSource + } + if (testUid) { + entry.testUid = testUid + } + return entry +} + +/** + * Params any adapter's matcher tap produces. `prefix` selects the command + * namespace (`expect` / `assert` / `verify`), all of which map to an `Assert` + * action via the shared action map. Adapters do the thin framework-specific + * extraction (matcher name, args, pass flag, message); this conversion is the + * single shared path — the generic counterpart to `capturedAssertToCommandLog` + * for libraries that aren't node:assert (expect-webdriverio, chai, Nightwatch). + */ +export interface MatcherAssertion { + prefix?: string + method: string + args?: unknown[] + passed: boolean + message?: string | (() => string) + callSource?: string + /** Explicit display label for the action row. Falls back to `command` when + * absent — set it when the framework carries a richer human message than + * `prefix.method` (e.g. Nightwatch's "Testing if the page title contains …"). */ + title?: string +} + +export function matcherAssertionToCommandLog( + input: MatcherAssertion, + testUid?: string +): CommandLog { + const command = `${input.prefix ?? 'expect'}.${input.method}` + const message = + typeof input.message === 'function' ? input.message() : input.message + const entry = capturedAssertToCommandLog( + { + command, + args: (input.args ?? []).map(safeSerializeAssertArg), + result: input.passed ? 'passed' : undefined, + error: input.passed + ? undefined + : new Error(stripAnsi(message ?? `${command} failed`)), + callSource: input.callSource, + timestamp: Date.now() + }, + testUid + ) + if (input.title) { + entry.title = input.title + } + return entry } /** @@ -167,6 +246,7 @@ export function patchNodeAssert( } assertObj[methodName] = makePatchedAssertMethod( methodName, + assertObj, original as (...a: unknown[]) => unknown, onCommand ) diff --git a/packages/core/src/attempt-tracker.ts b/packages/core/src/attempt-tracker.ts new file mode 100644 index 00000000..e4915584 --- /dev/null +++ b/packages/core/src/attempt-tracker.ts @@ -0,0 +1,39 @@ +/** + * Framework-agnostic per-test attempt counting. Every supported runner + * re-enters its per-test start hook when a test is retried, so recording the + * same uid again yields an incremented attempt number (0-based: the first run + * is attempt 0, the first retry is attempt 1). This is the primary, + * runner-independent retry signal feeding `TestOutcome.attempt` for the + * retry-aware trace policies (see trace-retention.ts). + */ +export class TestAttemptTracker { + #attempts = new Map<string, number>() + #sawRetry = false + + /** Record a starting test; returns its attempt number (0 first, +1 per rerun). */ + recordStart(uid: string): number { + const prior = this.#attempts.get(uid) + const attempt = prior === undefined ? 0 : prior + 1 + this.#attempts.set(uid, attempt) + if (attempt > 0) { + this.#sawRetry = true + } + return attempt + } + + /** Latest attempt recorded for `uid`, or undefined if it never started. */ + attemptFor(uid: string): number | undefined { + return this.#attempts.get(uid) + } + + /** True once any test has started more than once (a retry occurred). */ + get sawRetry(): boolean { + return this.#sawRetry + } + + /** Clear all state — used at session boundaries and reuse-mode reconnects. */ + reset(): void { + this.#attempts.clear() + this.#sawRetry = false + } +} diff --git a/packages/core/src/finalize-screencast.ts b/packages/core/src/finalize-screencast.ts index e8c6b00c..4e1b0b9c 100644 --- a/packages/core/src/finalize-screencast.ts +++ b/packages/core/src/finalize-screencast.ts @@ -62,6 +62,9 @@ export async function finalizeScreencast({ const candidate = outputDir || process.cwd() let videoPath = path.join(candidate, fileName) try { + // Create the (test-results) dir if absent, then confirm it's writable; + // fall back to tmpdir on any failure so a bad path never aborts the run. + fs.mkdirSync(candidate, { recursive: true }) fs.accessSync(candidate, fs.constants.W_OK) } catch { videoPath = path.join(os.tmpdir(), fileName) diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 21fcdf08..89c6c301 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -3,19 +3,29 @@ export * from './action-mapping.js' export * from './action-snapshot.js' +export * from './attempt-tracker.js' export * from './with-timeout.js' export * from './assert-patcher.js' export * from './element-snapshot.js' export * from './element-scripts.js' export * from './element-types.js' +export * from './sha1.js' +export * from './trace-action-events.js' +export * from './trace-console.js' export * from './trace-exporter.js' +export * from './trace-finalizer.js' +export * from './trace-frame-snapshots.js' +export * from './trace-retention.js' +export * from './trace-sources.js' export * from './trace-har.js' +export * from './trace-snapshots.js' export * from './trace-zip-writer.js' export * from './bidi.js' export * from './console.js' export * from './uid.js' export * from './net.js' export * from './stack.js' +export * from './terminal-throttle.js' export * from './error.js' export * from './finalize-screencast.js' export * from './output-dir.js' diff --git a/packages/core/src/output-dir.ts b/packages/core/src/output-dir.ts index fa659ace..463c5960 100644 --- a/packages/core/src/output-dir.ts +++ b/packages/core/src/output-dir.ts @@ -25,6 +25,9 @@ export interface ResolveAdapterOutputDirInput { const NODE_MODULES_SEGMENT = `${path.sep}node_modules${path.sep}` +/** All run output is grouped under this subfolder. */ +const OUTPUT_SUBDIR = 'test-results' + function isWritable(dir: string): boolean { try { fs.accessSync(dir, fs.constants.W_OK) @@ -36,9 +39,11 @@ function isWritable(dir: string): boolean { /** * Resolve the directory where an adapter should write output files - * (screencast .webm, trace JSON, etc.). + * (screencast .webm, trace JSON, etc.). Every artifact is grouped under a + * single `test-results/` subfolder so a run's output is + * self-contained regardless of where the base directory resolves to. * - * Priority: + * The base directory is resolved by priority: * 1. `userConfiguredDir` — explicit opt-in, honored as-is. * 2. `dirname(testFilePath)` — same folder as the spec that just ran. * 3. `dirname(configPath)` — fallback to the framework config dir. @@ -56,6 +61,10 @@ function isWritable(dir: string): boolean { export function resolveAdapterOutputDir( input: ResolveAdapterOutputDirInput = {} ): string { + return path.join(resolveBaseDir(input), OUTPUT_SUBDIR) +} + +function resolveBaseDir(input: ResolveAdapterOutputDirInput): string { const fallback = input.fallbackDir ?? process.cwd() // userConfiguredDir bypasses the node_modules and writability filters // because the user opted into it explicitly — surprising overrides are diff --git a/packages/core/src/session-capturer.ts b/packages/core/src/session-capturer.ts index cf970599..f5b90f4e 100644 --- a/packages/core/src/session-capturer.ts +++ b/packages/core/src/session-capturer.ts @@ -7,9 +7,11 @@ import type { LogSource, Metadata, NetworkRequest, + SerializedError, TraceMutation } from '@wdio/devtools-shared' import { WS_PATHS, WS_SCOPE } from '@wdio/devtools-shared' +import { mapCommandToAction } from './action-mapping.js' import { CONSOLE_METHODS, LOG_SOURCES, @@ -19,6 +21,7 @@ import { isInternalStreamLine, stripAnsi } from './console.js' +import { TerminalLineThrottle } from './terminal-throttle.js' /** * Foundation class for adapter SessionCapturers. Owns the cross-framework @@ -60,6 +63,11 @@ export abstract class SessionCapturerBase { // stdout, OR when stream forwarding wants to log via console. #isCapturingConsole = false #isCapturingStream = false + // Collapses high-frequency identical terminal lines (e.g. WDIO's per-command + // COMMAND/RESULT logger frames reprinted every ~100ms during an expect poll) + // so they don't flood the console lane. Terminal-source only — user + // console.* is source='test' and never throttled. + #terminalThrottle = new TerminalLineThrottle() // Command bookkeeping — used by adapters that emit commands themselves // (nightwatch, selenium). The WDIO service adapter doesn't call sendCommand @@ -206,6 +214,35 @@ export abstract class SessionCapturerBase { }) } + /** + * Mark the most recent user action of a test as failed — for framework + * failures that aren't captured as their own command. Broadcasts the swap so + * live mode highlights it too. Returns false when no eligible action is found, + * OR when that most-recent action already carries an error: the failure is + * then already represented (e.g. an expect matcher captured as its own row via + * afterAssertion), so it must not bleed onto an earlier *passing* action. + */ + failLastAction(testUid: string | undefined, error: SerializedError): boolean { + for (let i = this.commandsLog.length - 1; i >= 0; i--) { + const command = this.commandsLog[i] + if (!mapCommandToAction(command.command)) { + continue + } + if (testUid && command.testUid !== testUid) { + continue + } + // First (most-recent) action for this test. If it's already failed, the + // failure is captured — stop rather than marking an earlier passing one. + if (command.error) { + return false + } + command.error = error + this.sendReplaceCommand(command.timestamp, command) + return true + } + return false + } + /** * Read a file from disk, store in `sources`, and broadcast to the UI via * `sendUpstream('sources', { [path]: text })`. Idempotent — a cached path is @@ -408,7 +445,8 @@ export abstract class SessionCapturerBase { if ( !clean || this.isInternalStreamLine(clean) || - SPINNER_RE.test(clean) + SPINNER_RE.test(clean) || + !this.#terminalThrottle.shouldEmit(clean) ) { continue } diff --git a/packages/core/src/sha1.ts b/packages/core/src/sha1.ts new file mode 100644 index 00000000..68b56e40 --- /dev/null +++ b/packages/core/src/sha1.ts @@ -0,0 +1,6 @@ +import { createHash } from 'node:crypto' + +/** Hex SHA-1 used for content-addressed trace resources. */ +export function sha1Hex(data: Buffer | string): string { + return createHash('sha1').update(data).digest('hex') +} diff --git a/packages/core/src/spec-trace-helpers.ts b/packages/core/src/spec-trace-helpers.ts index e49f2bd6..2eb69298 100644 --- a/packages/core/src/spec-trace-helpers.ts +++ b/packages/core/src/spec-trace-helpers.ts @@ -6,6 +6,7 @@ * is the single source of truth — adapters import from here. */ +import path from 'node:path' import type { ActionSnapshot, TestMetadataMap, @@ -18,9 +19,15 @@ import { deterministicUid } from './uid.js' // ─── SpecRange ──────────────────────────────────────────────────────────────── -/** Index ranges into a SessionCapturer's flat arrays for a single spec file. */ +/** Index ranges into a SessionCapturer's flat arrays for one trace slice + * (a spec file, or a single test under `test` granularity). */ export interface SpecRange { specFile: string + /** Dedupe/identity key: spec path for spec slices; testUid for test slices, + * or `${testUid}-retry${n}` so each retried attempt is its own slice. */ + key: string + /** Present only for test-granularity slices; the base (non-retry) testUid. */ + testUid?: string commandStartIdx: number consoleStartIdx: number networkStartIdx: number @@ -63,6 +70,63 @@ export function buildSpecSessionId( return `${base}-${hash}-${sessionId.slice(0, 8)}` } +// ─── Test slice session ID ────────────────────────────────────────────────── + +/** + * Build a collision-safe test-level session ID from the slice's spec file, its + * identity `key` (testUid, or `${testUid}-retry${n}` for retries), and the + * parent session ID. Hashing the key keeps retries and sibling tests in the + * same spec from colliding on filename, while the spec basename keeps the name + * human-readable. + */ +export function buildTestSliceSessionId( + specFile: string, + key: string, + sessionId: string +): string { + const base = sanitizeSpecName(specFile) + const hash = deterministicUid(key).split('-').pop()!.slice(0, 8) + return `${base}-${hash}-${sessionId.slice(0, 8)}` +} + +// ─── Test slice output folder ──────────────────────────────────────────────── + +/** Max slug length so a title doesn't blow past filesystem path limits. */ +const MAX_SLUG_LENGTH = 60 + +/** Lowercase, collapse runs of non-alphanumerics to `-`, trim edge dashes, + * and cap length (trimming a dash the cut may leave behind). */ +function slugify(text: string): string { + return text + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, MAX_SLUG_LENGTH) + .replace(/-+$/g, '') +} + +/** + * Build the per-test output folder name: `<spec>-<title>-<browser>[-retryN]`. + * The spec basename is sanitized via {@link sanitizeSpecName}; title and browser + * are slugified. An empty title falls back to a short hash of `key`, and a + * `${uid}-retry${n}` key appends a `-retry<N>` suffix so retries don't collide. + */ +export function buildTestSliceFolder( + specFile: string, + testTitle: string | undefined, + browser: string | undefined, + key: string +): string { + const specBase = sanitizeSpecName(path.basename(specFile)) + const titleSlug = + slugify(testTitle ?? '') || + deterministicUid(key).split('-').pop()!.slice(0, 8) + const browserSlug = slugify(browser ?? '') || 'browser' + const retryMatch = key.match(/-retry(\d+)$/) + const retrySuffix = retryMatch ? `-retry${retryMatch[1]}` : '' + return `${specBase}-${titleSlug}-${browserSlug}${retrySuffix}` +} + // ─── TraceCapturer slice ───────────────────────────────────────────────────── /** @@ -127,11 +191,29 @@ export function filterTestMetadataBySpec( return filtered } +/** + * Filter a full `testUid → metadata` map down to a single test's entry. The + * per-test analog of {@link filterTestMetadataBySpec}: a test slice's metadata + * is just that one test's entry, attached as its tracingGroup name. + */ +export function filterTestMetadataByUid( + allMetadata: TestMetadataMap, + testUid: string +): TestMetadataMap { + const filtered: TestMetadataMap = new Map() + const entry = allMetadata.get(testUid) + if (entry) { + filtered.set(testUid, entry) + } + return filtered +} + // ─── Spec boundary recording ────────────────────────────────────────────────── /** - * Minimal context needed by `recordSpecBoundary` to detect spec-file - * transitions and capture array index ranges. + * Minimal context needed by `recordSliceBoundary` to detect spec-file / test + * transitions and capture array index ranges. `flushedSpecs` holds already- + * flushed slice keys (spec paths or test keys), shared with the finalizer. */ export interface SpecBoundaryContext { specRanges: SpecRange[] @@ -146,32 +228,28 @@ export interface SpecBoundaryContext { actionSnapshots: ArrayLike<unknown> } -/** - * Record a spec-file boundary. When `traceGranularity` is `'spec'` and the - * spec file has changed, this pushes a new `SpecRange` and returns the - * previous range so the caller can flush its trace artifact. - * - * Returns `null` when no flush is needed (same spec, or granularity isn't - * `'spec'`, or no capturer). - */ -export function recordSpecBoundary( +/** Push a new slice range and return the previous (unflushed) range to flush. + * `suppressSameKey` skips recording when the incoming key matches the last + * range's — used for spec granularity so consecutive tests in one file share + * a slice; test granularity records every attempt (retries included). */ +function pushSliceRange( ctx: SpecBoundaryContext, specFile: string, - traceGranularity: TraceGranularity | undefined + key: string, + testUid: string | undefined, + suppressSameKey: boolean ): SpecRange | null { - if (traceGranularity !== 'spec') { - return null - } const lastRange = ctx.specRanges[ctx.specRanges.length - 1] - if (lastRange && lastRange.specFile === specFile) { + if (suppressSameKey && lastRange && lastRange.key === key) { return null } - const prevRange = - lastRange && !ctx.flushedSpecs.has(lastRange.specFile) ? lastRange : null + lastRange && !ctx.flushedSpecs.has(lastRange.key) ? lastRange : null ctx.specRanges.push({ specFile, + key, + testUid, commandStartIdx: ctx.capturer.commandsLog.length, consoleStartIdx: ctx.capturer.consoleLogs.length, networkStartIdx: ctx.capturer.networkRequests.length, @@ -183,6 +261,48 @@ export function recordSpecBoundary( return prevRange } +/** + * Record a trace-slice boundary. For `spec` granularity, a new slice starts + * when the spec file changes (existing behavior). For `test` granularity, a + * new slice starts on every recorded test — including retries: a repeated + * `testUid` is keyed `${testUid}-retry${n}` so each attempt is its own slice. + * Returns the previous, not-yet-flushed range so the caller can flush it, or + * `null` when nothing needs flushing (same spec, missing testUid, or a + * non-sliced granularity). + */ +export function recordSliceBoundary( + ctx: SpecBoundaryContext, + granularity: TraceGranularity | undefined, + specFile: string, + testUid?: string +): SpecRange | null { + if (granularity === 'spec') { + return pushSliceRange(ctx, specFile, specFile, undefined, true) + } + if (granularity === 'test' && testUid !== undefined) { + const priorAttempts = ctx.specRanges.filter( + (r) => r.testUid === testUid + ).length + const key = + priorAttempts === 0 ? testUid : `${testUid}-retry${priorAttempts}` + return pushSliceRange(ctx, specFile, key, testUid, false) + } + return null +} + +/** + * Record a spec-file boundary. Thin back-compat wrapper over + * {@link recordSliceBoundary}; behavior is unchanged for `spec` granularity + * and returns `null` for every other granularity. + */ +export function recordSpecBoundary( + ctx: SpecBoundaryContext, + specFile: string, + traceGranularity: TraceGranularity | undefined +): SpecRange | null { + return recordSliceBoundary(ctx, traceGranularity, specFile) +} + // ─── Spec trace I/O ──────────────────────────────────────────────────────────── /** @@ -205,41 +325,83 @@ export interface WriteSpecTraceInput { capabilities?: unknown } -/** - * Write a standalone trace artifact (zip or ndjson-directory) for a single - * spec file. This is the shared I/O path — all three adapters delegate to it - * from their own `flushSpecTrace` wrappers. - */ -export async function writeSpecTrace( - input: WriteSpecTraceInput +/** Slice the parent capturer/snapshots for one range and write the artifact + * under `sliceSessionId` with the pre-filtered `testMetadata`. Shared by the + * spec and test write paths so both slice identically. `overrides` lets the + * test path redirect into a named folder with a fixed `trace` file stem. */ +async function writeSliceTrace( + input: WriteSpecTraceInput, + sliceSessionId: string, + testMetadata: TestMetadataMap, + overrides: { outputDir?: string; fileStem?: string } = {} ): Promise<string> { - const specCapturer = buildSpecCapturer( + const sliceCapturer = buildSpecCapturer( input.capturer, input.range, input.nextRange ) - const specSnapshots = input.actionSnapshots.slice( + const sliceSnapshots = input.actionSnapshots.slice( input.range.snapshotCount, input.nextRange?.snapshotCount ?? input.actionSnapshots.length ) - const specSessionId = buildSpecSessionId( - input.range.specFile, - input.sessionId - ) - - const testMetadata = filterTestMetadataBySpec( - input.testMetadata, - input.range.specFile - ) - - return writeTraceZip(specCapturer, { - outputDir: input.outputDir, - sessionId: specSessionId, + return writeTraceZip(sliceCapturer, { + outputDir: overrides.outputDir ?? input.outputDir, + sessionId: sliceSessionId, + fileStem: overrides.fileStem, capabilities: input.capabilities, - actionSnapshots: specSnapshots.length > 0 ? specSnapshots : undefined, + actionSnapshots: sliceSnapshots.length > 0 ? sliceSnapshots : undefined, format: input.format, testMetadata }) } + +/** + * Write a standalone trace artifact (zip or ndjson-directory) for a single + * spec file. This is the shared I/O path — all three adapters delegate to it + * from their own `flushSpecTrace` wrappers. + */ +export async function writeSpecTrace( + input: WriteSpecTraceInput +): Promise<string> { + return writeSliceTrace( + input, + buildSpecSessionId(input.range.specFile, input.sessionId), + filterTestMetadataBySpec(input.testMetadata, input.range.specFile) + ) +} + +/** + * Write a standalone trace artifact for a single test slice into its own + * folder: `<outputDir>/<specBasename>-<titleSlug>-<browserSlug>[-retryN]/trace.zip`. + * Reuses {@link WriteSpecTraceInput}; the folder is the slice's external + * identity (title/browser/retry), while {@link buildTestSliceSessionId} names + * the sessionId embedded inside the archive. + */ +export async function writeTestSliceTrace( + input: WriteSpecTraceInput +): Promise<string> { + const testUid = input.range.testUid ?? input.range.key + const title = input.testMetadata.get(testUid)?.title + // capabilities is framework-typed unknown; read only browserName here. + const browserName = ( + input.capabilities as { browserName?: string } | undefined + )?.browserName + const folder = buildTestSliceFolder( + input.range.specFile, + title, + browserName, + input.range.key + ) + return writeSliceTrace( + input, + buildTestSliceSessionId( + input.range.specFile, + input.range.key, + input.sessionId + ), + filterTestMetadataByUid(input.testMetadata, testUid), + { outputDir: path.join(input.outputDir, folder), fileStem: 'trace' } + ) +} diff --git a/packages/core/src/suite-helpers.ts b/packages/core/src/suite-helpers.ts index a69569ba..35d0a760 100644 --- a/packages/core/src/suite-helpers.ts +++ b/packages/core/src/suite-helpers.ts @@ -1,4 +1,10 @@ -import type { SuiteStats, TestStats, TestStatus } from '@wdio/devtools-shared' +import type { + SuiteStats, + TestAncestor, + TestMetadataMap, + TestStats, + TestStatus +} from '@wdio/devtools-shared' import { TEST_STATE } from '@wdio/devtools-shared' /** @@ -162,3 +168,64 @@ export function stampTestEnd(test: TestStats, end: Date = new Date()): void { test.end = end test._duration = end.getTime() - (test.start?.getTime() ?? end.getTime()) } + +function deriveAncestorKind( + suite: SuiteStats, + parentKind: TestAncestor['kind'] | undefined +): TestAncestor['kind'] { + if (parentKind === 'feature') { + return 'scenario' + } + // Cucumber scenario suites carry the same .feature file as their parent — + // only a suite not already under a feature/scenario counts as the feature. + if (suite.file?.endsWith('.feature') && parentKind !== 'scenario') { + return 'feature' + } + return 'suite' +} + +/** + * Recursive walk over suite trees collecting every test's metadata entry — + * uid → title/specFile/state/attempt plus the ancestor chain (outermost + * first, the test's own node excluded). Consolidates the per-adapter + * collectTestMetadata walks. + */ +export function collectSuiteTestMetadata( + suites: Iterable<SuiteStats> +): TestMetadataMap { + const metadata: TestMetadataMap = new Map() + const walk = (suite: SuiteStats, ancestors: TestAncestor[]): void => { + const parentKind = ancestors[ancestors.length - 1]?.kind + const chain: TestAncestor[] = [ + ...ancestors, + { + uid: suite.uid, + title: suite.title, + kind: deriveAncestorKind(suite, parentKind) + } + ] + for (const entry of suite.tests ?? []) { + // Trees can contain string placeholders for tests not yet reconciled. + if (typeof entry !== 'object' || entry === null) { + continue + } + metadata.set(entry.uid, { + title: entry.fullTitle || entry.title, + specFile: entry.file ?? suite.file, + state: entry.state, + attempt: entry.retries ?? 0, + ancestry: chain + }) + } + for (const child of suite.suites ?? []) { + walk(child, chain) + } + } + for (const suite of suites) { + if (typeof suite !== 'object' || suite === null) { + continue + } + walk(suite, []) + } + return metadata +} diff --git a/packages/core/src/terminal-throttle.ts b/packages/core/src/terminal-throttle.ts new file mode 100644 index 00000000..017cbf00 --- /dev/null +++ b/packages/core/src/terminal-throttle.ts @@ -0,0 +1,70 @@ +/** + * Rate-limits high-frequency identical terminal (stdout/stderr) lines so a + * polling framework that reprints the same line every ~100ms doesn't flood the + * trace's console lane. The motivating case: WDIO's logger writes a + * `COMMAND`/`RESULT` frame for every WebDriver command, so an `expect` that + * polls for its full 10s timeout emits hundreds of near-identical lines. + * + * Only terminal-source capture goes through this — user `console.*` is + * captured as `source: 'test'` and never throttled, so real user output is + * untouched. Distinct lines always pass immediately; a repeat is emitted at + * most once per window. + */ + +/** + * A repeated identical terminal line is emitted at most once per this window + * (ms). Sized so a 100ms poll collapses ~10:1 while a human-paced reprint of + * the same line (>1s apart) still shows every time. + */ +export const TERMINAL_REPEAT_WINDOW_MS = 1000 + +/** + * Leading ISO-8601 timestamp emitted by most structured loggers (`@wdio/logger`'s + * `%t %l %n:` template, pino, winston, …). It's the only volatile part of + * otherwise-identical successive log frames, so it's stripped from the throttle + * key — never from the emitted text. + */ +const LEADING_ISO_TIMESTAMP_RE = + /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d\d\d)?Z {1,4}/ + +/** Key a terminal line for repeat-detection: drop a leading ISO timestamp so + * successive log frames of the same message collapse. */ +export function terminalRepeatKey(line: string): string { + return line.replace(LEADING_ISO_TIMESTAMP_RE, '') +} + +export class TerminalLineThrottle { + #lastEmitted = new Map<string, number>() + readonly #windowMs: number + + constructor(windowMs: number = TERMINAL_REPEAT_WINDOW_MS) { + this.#windowMs = windowMs + } + + /** + * True if the line should be emitted, false if it's a within-window repeat of + * a line already emitted. The window is anchored to the last *emit*, not the + * last occurrence, so a sustained stream of one line still emits once per + * window instead of going silent after the first. Expired keys are pruned on + * each emit, so state stays bounded by the number of distinct lines seen + * within the window. + */ + shouldEmit(line: string, now: number = Date.now()): boolean { + const key = terminalRepeatKey(line) + const last = this.#lastEmitted.get(key) + if (last !== undefined && now - last < this.#windowMs) { + return false + } + this.#lastEmitted.set(key, now) + this.#prune(now) + return true + } + + #prune(now: number): void { + for (const [key, ts] of this.#lastEmitted) { + if (now - ts >= this.#windowMs) { + this.#lastEmitted.delete(key) + } + } + } +} diff --git a/packages/core/src/trace-action-events.ts b/packages/core/src/trace-action-events.ts new file mode 100644 index 00000000..7cd44eab --- /dev/null +++ b/packages/core/src/trace-action-events.ts @@ -0,0 +1,310 @@ +// Builds the before/after action events of the exported trace stream, +// including tracingGroup test boundaries and frame-snapshot ref stamping. + +import type { CommandLog, TestMetadataMap } from '@wdio/devtools-shared' +import { + ASSERT_ACTION_CLASS, + formatActionTitle, + mapCommandToAction, + FILL_METHODS, + type TraceAction +} from './action-mapping.js' +import { callSourceToStack, type StackFrame } from './trace-sources.js' +import type { FrameSnapshotIndex } from './trace-frame-snapshots.js' + +export interface BeforeEvent { + type: 'before' + callId: string + startTime: number + class: string + method: string + pageId: string + params: Record<string, unknown> + title: string + /** Trace-viewer API name (e.g. 'page.goBack', 'element.click'). */ + apiName: string + /** CallId of the Tracing.tracingGroup that wraps this action (if any). */ + parentId?: string + /** User-code frame the command was issued from, when captured. */ + stack?: StackFrame[] + /** Frame-snapshot name rendered as the action's before state. */ + beforeSnapshot?: string +} + +export interface AfterEvent { + type: 'after' + callId: string + endTime: number + error?: { message: string } + /** Command return value (e.g. the text getText resolved to). */ + result?: unknown + /** CallId of the Tracing.tracingGroup that wraps this action (if any). */ + parentId?: string + /** Frame-snapshot name rendered as the action's after state. */ + afterSnapshot?: string +} + +// Serialized command results over this size are dropped from the trace — a +// huge execute() return shouldn't bloat every action line. +const MAX_RESULT_BYTES = 64 * 1024 + +/** JSON-safe command result within the size cap; undefined when absent, + * oversized, or not serializable. */ +function serializableResult(result: unknown): unknown { + if (result === undefined) { + return undefined + } + try { + const json = JSON.stringify(result) + if (json === undefined || json.length > MAX_RESULT_BYTES) { + return undefined + } + return JSON.parse(json) + } catch { + return undefined + } +} + +export type ActionEvent = BeforeEvent | AfterEvent + +interface ActionStream { + events: ActionEvent[] + prevEndMs: number + callCounter: number + lastTestUid?: string + groupCallId?: string +} + +// Nightwatch built-in assertions collapse {passed, actual, expected, message} +// into the command result on failure — surface those over positional args. +interface CollapsedAssertResult { + passed?: unknown + actual?: unknown + expected?: unknown + message?: unknown +} + +function collapsedAssertResult( + result: unknown +): CollapsedAssertResult | undefined { + if (typeof result === 'object' && result !== null && 'passed' in result) { + return result as CollapsedAssertResult + } + return undefined +} + +// Assert params: node:assert positional order (actual, expected, message?), +// plus a numeric echo of the raw args so the reader's paramsToArgs inverse +// reconstructs the original arg list without assert-specific knowledge. +function buildAssertParams(cmd: CommandLog): Record<string, unknown> { + const params: Record<string, unknown> = Object.fromEntries( + cmd.args.map((arg, index) => [String(index), arg]) + ) + const [actual, expected, message] = cmd.args + const collapsed = collapsedAssertResult(cmd.result) + const semantic = { + actual: collapsed?.actual ?? actual, + expected: collapsed?.expected ?? expected, + message: collapsed?.message ?? message + } + for (const [key, value] of Object.entries(semantic)) { + if (value !== undefined) { + params[key] = value + } + } + return params +} + +// Semantic params from positional args (selector/value/url), falling back to +// index keys; the reader's paramsToArgs is the inverse. +function buildActionParams( + action: TraceAction, + rawArgs: unknown[] +): Record<string, unknown> { + const isValueMethod = FILL_METHODS.has(action.method) + if (action.class === 'Element' && isValueMethod && rawArgs.length >= 2) { + return { selector: rawArgs[0], value: rawArgs[1] } + } + if (action.class === 'Element' && isValueMethod && rawArgs.length === 1) { + return { value: rawArgs[0] } + } + if ( + action.class === 'Element' && + rawArgs.length === 1 && + typeof rawArgs[0] === 'string' + ) { + return { selector: rawArgs[0] } + } + if (rawArgs.length === 1 && typeof rawArgs[0] === 'string') { + return { url: rawArgs[0] } + } + return Object.fromEntries(rawArgs.map((a, i) => [String(i), a])) +} + +function closeGroup(stream: ActionStream): void { + if (!stream.lastTestUid || !stream.groupCallId) { + return + } + stream.callCounter++ + stream.events.push({ + type: 'after', + callId: stream.groupCallId, + endTime: stream.prevEndMs + }) +} + +// When the testUid changes, close the previous Tracing.tracingGroup and open +// a new one; child actions reference it via parentId to render as spans. +function handleTestBoundary( + stream: ActionStream, + cmd: CommandLog, + pageId: string, + wallTime: number, + testMetadata?: TestMetadataMap +): void { + if (!cmd.testUid || cmd.testUid === stream.lastTestUid) { + return + } + closeGroup(stream) + stream.callCounter++ + stream.groupCallId = `call@${stream.callCounter}` + const groupName = testMetadata?.get(cmd.testUid)?.title ?? cmd.testUid + stream.events.push({ + type: 'before', + callId: stream.groupCallId, + startTime: Math.max( + stream.prevEndMs, + (cmd.startTime ?? cmd.timestamp) - wallTime + ), + class: 'Tracing', + method: 'tracingGroup', + pageId, + params: { name: groupName }, + title: groupName, + apiName: 'tracing.tracingGroup' + }) + stream.lastTestUid = cmd.testUid +} + +function buildParamsAndTitle( + action: TraceAction, + cmd: CommandLog +): { params: Record<string, unknown>; title: string } { + const isAssert = action.class === ASSERT_ACTION_CLASS + const params = isAssert + ? buildAssertParams(cmd) + : buildActionParams(action, cmd.args) + return { + params, + title: formatActionTitle( + action, + cmd.args, + params, + isAssert ? cmd.command : undefined + ) + } +} + +function actionError( + cmd: CommandLog, + isAssert: boolean +): { message: string } | undefined { + if (cmd.error) { + const err = cmd.error as { message?: string } + return { message: err.message ?? String(cmd.error) } + } + if (isAssert) { + // Nightwatch assert failures carry no Error — only the collapsed result. + const collapsed = collapsedAssertResult(cmd.result) + if (collapsed && collapsed.passed === false) { + return { message: String(collapsed.message ?? 'Assertion failed') } + } + } + return undefined +} + +function buildAfterEvent( + cmd: CommandLog, + action: TraceAction, + callId: string, + endMs: number, + snapshotIndex?: FrameSnapshotIndex +): AfterEvent { + const afterEvent: AfterEvent = { type: 'after', callId, endTime: endMs } + const error = actionError(cmd, action.class === ASSERT_ACTION_CLASS) + if (error) { + afterEvent.error = error + } + const result = serializableResult(cmd.result) + if (result !== undefined) { + afterEvent.result = result + } + const afterName = snapshotIndex?.claimAfter(cmd.timestamp, callId) + if (afterName) { + afterEvent.afterSnapshot = afterName + } + return afterEvent +} + +function pushActionPair( + stream: ActionStream, + cmd: CommandLog, + action: TraceAction, + pageId: string, + wallTime: number, + snapshotIndex?: FrameSnapshotIndex +): void { + stream.callCounter++ + const callId = `call@${stream.callCounter}` + // Command invocation timestamp, falling back to completion when absent. + const rawStartMs = (cmd.startTime ?? cmd.timestamp) - wallTime + const rawEndMs = cmd.timestamp - wallTime + // Floor at prevEndMs to prevent visual overlap with the previous action. + const startMs = Math.max(stream.prevEndMs, rawStartMs) + // +1ms minimum duration so an `after` never precedes its parsed `before`. + const endMs = Math.max(startMs + 1, rawEndMs) + const { params, title } = buildParamsAndTitle(action, cmd) + const beforeEvent: BeforeEvent = { + type: 'before', + callId, + startTime: startMs, + class: action.class, + method: action.method, + pageId, + params, + title, + apiName: `${action.class.toLowerCase()}.${action.method}`, + parentId: stream.groupCallId + } + const stack = callSourceToStack(cmd.callSource) + if (stack) { + beforeEvent.stack = stack + } + const beforeName = snapshotIndex?.beforeName() + if (beforeName) { + beforeEvent.beforeSnapshot = beforeName + } + stream.events.push(beforeEvent) + stream.events.push(buildAfterEvent(cmd, action, callId, endMs, snapshotIndex)) + stream.prevEndMs = endMs +} + +export function buildActionEvents( + commands: CommandLog[], + pageId: string, + wallTime: number, + testMetadata?: TestMetadataMap, + snapshotIndex?: FrameSnapshotIndex +): ActionEvent[] { + const stream: ActionStream = { events: [], prevEndMs: 0, callCounter: 0 } + for (const cmd of commands) { + const action = mapCommandToAction(cmd.command) + if (!action) { + continue + } + handleTestBoundary(stream, cmd, pageId, wallTime, testMetadata) + pushActionPair(stream, cmd, action, pageId, wallTime, snapshotIndex) + } + closeGroup(stream) + return stream.events +} diff --git a/packages/core/src/trace-console.ts b/packages/core/src/trace-console.ts new file mode 100644 index 00000000..78deb7aa --- /dev/null +++ b/packages/core/src/trace-console.ts @@ -0,0 +1,89 @@ +// Maps captured ConsoleLog entries into trace-event vocabulary: browser +// console entries become `console` events; test/terminal output becomes +// `stdout`/`stderr` events (which carry no location semantics — matching +// what we capture for Node-side lines). + +import type { ConsoleLog, LogSource } from '@wdio/devtools-shared' + +export interface ConsoleEvent { + type: 'console' + time: number + pageId?: string + messageType: string + text: string + args?: { preview: string; value: unknown }[] + location: { url: string; lineNumber: number; columnNumber: number } +} + +export interface StdioEvent { + type: 'stdout' | 'stderr' + timestamp: number + text?: string + /** Extension field: test-vs-terminal origin; foreign viewers ignore it. */ + source?: Extract<LogSource, 'test' | 'terminal'> +} + +// Caps pathological runs (console.log in a loop) so the trace stays openable. +const MAX_CONSOLE_EVENTS = 10_000 + +/** Trace vocabulary uses 'warning'; 'trace' maps to the nearest severity, 'debug'. */ +function toTraceLevel(level: ConsoleLog['type']): string { + if (level === 'warn') { + return 'warning' + } + if (level === 'trace') { + return 'debug' + } + return level +} + +function previewArg(arg: unknown): string { + if (typeof arg === 'string') { + return arg + } + try { + return JSON.stringify(arg) ?? String(arg) + } catch { + return String(arg) + } +} + +export function buildConsoleEvents( + logs: ConsoleLog[], + pageId: string, + wallTime: number +): (ConsoleEvent | StdioEvent)[] { + const capped = logs.slice(0, MAX_CONSOLE_EVENTS) + const events: (ConsoleEvent | StdioEvent)[] = capped.map((log) => { + const time = Math.max(0, log.timestamp - wallTime) + const text = log.args.map(previewArg).join(' ') + // Untagged entries predate source tagging; they came from the page. + if (log.source === 'browser' || log.source === undefined) { + return { + type: 'console', + time, + pageId, + messageType: toTraceLevel(log.type), + text, + args: log.args.map((arg) => ({ preview: previewArg(arg), value: arg })), + // Location isn't captured at the patch site; the required field ships zeroed. + location: { url: '', lineNumber: 0, columnNumber: 0 } + } satisfies ConsoleEvent + } + return { + type: log.type === 'error' || log.type === 'warn' ? 'stderr' : 'stdout', + timestamp: time, + text, + source: log.source + } satisfies StdioEvent + }) + if (logs.length > MAX_CONSOLE_EVENTS) { + const last = capped[capped.length - 1] + events.push({ + type: 'stderr', + timestamp: last ? Math.max(0, last.timestamp - wallTime) : 0, + text: `[devtools] console truncated: dropped ${logs.length - MAX_CONSOLE_EVENTS} entries` + }) + } + return events +} diff --git a/packages/core/src/trace-exporter.ts b/packages/core/src/trace-exporter.ts index 24488442..40079b9a 100644 --- a/packages/core/src/trace-exporter.ts +++ b/packages/core/src/trace-exporter.ts @@ -20,13 +20,83 @@ import { FILL_METHODS, type TraceAction } from './action-mapping.js' +import { + buildConsoleEvents, + type ConsoleEvent, + type StdioEvent +} from './trace-console.js' +import { + buildFilmstripEvents, + buildSnapshotResources, + type ScreencastFrameEvent +} from './trace-snapshots.js' +import { + buildImageFrameSnapshots, + FrameSnapshotIndex, + type FrameSnapshotEvent +} from './trace-frame-snapshots.js' +import { buildActionEvents, type ActionEvent } from './trace-action-events.js' +import { buildSourceResources } from './trace-sources.js' import { networkRequestToHar } from './trace-har.js' import { buildTraceZip, type TraceZipResource } from './trace-zip-writer.js' +import { sha1Hex } from './sha1.js' const TRACE_VERSION = 8 const LIBRARY_NAME = '@wdio/devtools-core' const LIBRARY_VERSION = '1.0.0' +/** Response bodies above this size are not embedded in the trace. */ +const MAX_BODY_RESOURCE_BYTES = 1024 * 1024 +/** Per-trace ceiling on total embedded response-body bytes. */ +const MAX_TOTAL_BODY_RESOURCE_BYTES = 20 * 1024 * 1024 + +export interface NetworkBodyCaps { + maxBodyBytes: number + maxTotalBytes: number +} + +export interface NetworkBodyResources { + resources: TraceZipResource[] + sha1ByRequestId: Map<string, string> +} + +/** Content-addressed `resources/<sha1>` entries for captured response bodies. */ +export function buildNetworkBodyResources( + requests: NetworkRequest[], + caps: NetworkBodyCaps = { + maxBodyBytes: MAX_BODY_RESOURCE_BYTES, + maxTotalBytes: MAX_TOTAL_BODY_RESOURCE_BYTES + } +): NetworkBodyResources { + const resources: TraceZipResource[] = [] + const sha1ByRequestId = new Map<string, string>() + const stored = new Set<string>() + let totalBytes = 0 + // Cap skips are silent by design; a warn hook would slot into these branches. + for (const request of requests) { + if (request.responseBody === undefined) { + continue + } + const data = Buffer.from(request.responseBody, 'utf8') + if (data.byteLength > caps.maxBodyBytes) { + continue + } + const sha1 = sha1Hex(data) + if (stored.has(sha1)) { + sha1ByRequestId.set(request.id, sha1) + continue + } + if (totalBytes + data.byteLength > caps.maxTotalBytes) { + continue + } + stored.add(sha1) + totalBytes += data.byteLength + resources.push({ resourceName: sha1, data }) + sha1ByRequestId.set(request.id, sha1) + } + return { resources, sha1ByRequestId } +} + interface ContextOptionsEvent { version: number type: 'context-options' @@ -43,46 +113,13 @@ interface ContextOptionsEvent { options: { viewport: { width: number; height: number } } } -interface BeforeEvent { - type: 'before' - callId: string - startTime: number - class: string - method: string - pageId: string - params: Record<string, unknown> - title: string - /** Playwright-compatible API name (e.g. 'page.goBack', 'element.click'). */ - apiName: string - /** CallId of the Tracing.tracingGroup that wraps this action (if any). */ - parentId?: string -} - -interface AfterEvent { - type: 'after' - callId: string - endTime: number - error?: { message: string } - /** CallId of the Tracing.tracingGroup that wraps this action (if any). */ - parentId?: string -} - -interface ScreencastFrameEvent { - type: 'screencast-frame' - pageId: string - sha1: string - elements?: string - snapshot?: string - width: number - height: number - timestamp: number -} - type TraceEvent = | ContextOptionsEvent - | BeforeEvent - | AfterEvent + | ActionEvent | ScreencastFrameEvent + | ConsoleEvent + | StdioEvent + | FrameSnapshotEvent function shortId(sessionId?: string): string { return (sessionId ?? Math.random().toString(36).slice(2, 10)).slice(0, 8) @@ -145,143 +182,19 @@ function buildContextOptions( } } -export function buildActionEvents( - commands: CommandLog[], - pageId: string, - wallTime: number, - testMetadata?: TestMetadataMap -): TraceEvent[] { - const events: TraceEvent[] = [] - let prevEndMs = 0 - let callCounter = 0 - let lastTestUid: string | undefined - let groupCallId: string | undefined - - for (const cmd of commands) { - const action = mapCommandToAction(cmd.command) - if (!action) { - continue - } - - // ── Test boundary detection ── - // When the testUid changes, close the previous Tracing.tracingGroup - // and open a new one. Child actions inside the group reference it via - // parentId so trace viewers render them as labelled spans. - if (cmd.testUid && cmd.testUid !== lastTestUid) { - // Close the previous group. - if (lastTestUid && groupCallId) { - callCounter++ - events.push({ - type: 'after', - callId: groupCallId, - endTime: prevEndMs - } satisfies AfterEvent) - } - - // Open a new group for this test. - callCounter++ - groupCallId = `call@${callCounter}` - const meta = testMetadata?.get(cmd.testUid) - const groupName = meta?.title ?? cmd.testUid - events.push({ - type: 'before', - callId: groupCallId, - startTime: Math.max( - prevEndMs, - (cmd.startTime ?? cmd.timestamp) - wallTime - ), - class: 'Tracing', - method: 'tracingGroup', - pageId, - params: { name: groupName }, - title: groupName, - apiName: 'tracing.tracingGroup' - } satisfies BeforeEvent) - lastTestUid = cmd.testUid - } - - // ── Regular action (child of the current group) ── - callCounter++ - const callId = `call@${callCounter}` - // Use the command's actual invocation timestamp for the start, falling - // back to the completion timestamp when startTime isn't recorded. - const rawStartMs = (cmd.startTime ?? cmd.timestamp) - wallTime - const rawEndMs = cmd.timestamp - wallTime - // Floor at prevEndMs to prevent visual overlap with previous action. - const startMs = Math.max(prevEndMs, rawStartMs) - // +1ms minimum duration so the viewer never sees an `after` whose - // matching `before` hasn't been parsed yet. - const endMs = Math.max(startMs + 1, rawEndMs) - const rawArgs = cmd.args as unknown[] - let params: Record<string, unknown> - const isValueMethod = FILL_METHODS.has(action.method) - if (action.class === 'Element' && isValueMethod && rawArgs.length >= 2) { - params = { selector: rawArgs[0], value: rawArgs[1] } - } else if ( - action.class === 'Element' && - isValueMethod && - rawArgs.length === 1 - ) { - params = { value: rawArgs[0] } - } else if ( - action.class === 'Element' && - rawArgs.length === 1 && - typeof rawArgs[0] === 'string' - ) { - params = { selector: rawArgs[0] } - } else if (rawArgs.length === 1 && typeof rawArgs[0] === 'string') { - params = { url: rawArgs[0] } - } else { - params = Object.fromEntries(rawArgs.map((a, i) => [String(i), a])) - } - events.push({ - type: 'before', - callId, - startTime: startMs, - class: action.class, - method: action.method, - pageId, - params, - title: formatActionTitle(action, cmd.args, params), - apiName: `${action.class.toLowerCase()}.${action.method}`, - parentId: groupCallId - }) - const afterEvent: AfterEvent = { - type: 'after', - callId, - endTime: endMs - } - if (cmd.error) { - const err = cmd.error as { message?: string } - afterEvent.error = { message: err.message ?? String(cmd.error) } - } - events.push(afterEvent) - prevEndMs = endMs - } - - // Close the final group after the last action. - if (lastTestUid && groupCallId) { - callCounter++ - events.push({ - type: 'after', - callId: groupCallId, - endTime: prevEndMs - } satisfies AfterEvent) - } - - return events -} - function buildNetworkNdjson( requests: NetworkRequest[], wallTime: number, - pageId: string + pageId: string, + sha1ByRequestId: Map<string, string> ): Buffer { if (!requests.length) { return Buffer.alloc(0) } const lines = requests.map((r) => { - const entry = networkRequestToHar(r) as unknown as Record<string, unknown> + const entry = networkRequestToHar(r, { + bodySha1: sha1ByRequestId.get(r.id) + }) as unknown as Record<string, unknown> entry.snapshot = { ...(entry.snapshot as Record<string, unknown>), // Monotonic offset so the viewer positions bars on the timeline. @@ -294,63 +207,6 @@ function buildNetworkNdjson( return Buffer.from(lines.join('\n'), 'utf8') } -function buildSnapshotResources( - snapshots: ActionSnapshot[], - pageId: string -): TraceZipResource[] { - const out: TraceZipResource[] = [] - for (const snap of snapshots) { - const base = `${pageId}-${snap.timestamp}` - if (snap.screenshot) { - out.push({ - resourceName: `${base}.jpeg`, - data: Buffer.from(snap.screenshot, 'base64') - }) - } - if (snap.elements && snap.elements.length) { - out.push({ - resourceName: `${base}-elements.json`, - data: Buffer.from(JSON.stringify(snap.elements), 'utf8') - }) - } - if (snap.snapshotText) { - out.push({ - resourceName: `${base}-snapshot.txt`, - data: Buffer.from(snap.snapshotText, 'utf8') - }) - } - } - return out -} - -function buildScreencastFrames( - snapshots: ActionSnapshot[], - pageId: string, - wallTime: number, - viewport: { width: number; height: number } -): ScreencastFrameEvent[] { - return snapshots - .filter((s) => s.screenshot) - .map((s) => { - const base = `${pageId}-${s.timestamp}` - const frame: ScreencastFrameEvent = { - type: 'screencast-frame', - pageId, - sha1: `${base}.jpeg`, - width: viewport.width, - height: viewport.height, - timestamp: Math.max(0, s.timestamp - wallTime) - } - if (s.elements && s.elements.length) { - frame.elements = `${base}-elements.json` - } - if (s.snapshotText) { - frame.snapshot = `${base}-snapshot.txt` - } - return frame - }) -} - /** * Build a trace.zip buffer from the captured TraceLog. * Filters commands through ACTION_MAP and renames to trace vocabulary; @@ -368,13 +224,21 @@ function eventTime(e: TraceEvent): number { return e.endTime case 'screencast-frame': return e.timestamp + case 'frame-snapshot': + return e.snapshot.timestamp + case 'console': + return e.time + case 'stdout': + case 'stderr': + return e.timestamp } } /** At the same timestamp T: an action's `after` ends first, then the - * snapshot captured at the action boundary, then the next action's `before`. - * Matches the viewer's expectation that the screencast frame shows the - * state between the previous action's completion and the next one's start. */ + * snapshot captured at the action boundary, then console output observed + * at the boundary, then the next action's `before`. Matches the viewer's + * expectation that the screencast frame shows the state between the + * previous action's completion and the next one's start. */ function eventOrder(e: TraceEvent): number { switch (e.type) { case 'context-options': @@ -382,9 +246,14 @@ function eventOrder(e: TraceEvent): number { case 'after': return 1 case 'screencast-frame': + case 'frame-snapshot': return 2 - case 'before': + case 'console': + case 'stdout': + case 'stderr': return 3 + case 'before': + return 4 } } @@ -446,6 +315,38 @@ interface TraceBundle { resources: TraceZipResource[] } +function buildEventStream( + trace: TraceLog, + ctxOptions: ContextOptionsEvent, + pageId: string, + wallTime: number, + testMetadata?: TestMetadataMap +): TraceEvent[] { + const viewport = trace.metadata.viewport ?? { width: 1280, height: 720 } + const snapshots = trace.actionSnapshots ?? [] + const snapshotIndex = new FrameSnapshotIndex(snapshots) + const events: TraceEvent[] = [ + ctxOptions, + ...buildFilmstripEvents(snapshots, pageId, wallTime, viewport), + ...buildActionEvents( + trace.commands, + pageId, + wallTime, + testMetadata, + snapshotIndex + ), + ...buildImageFrameSnapshots( + snapshotIndex.refs(), + pageId, + wallTime, + viewport + ), + ...buildConsoleEvents(trace.consoleLogs, pageId, wallTime) + ] + events.sort(compareEvents) + return events +} + function buildTraceBundle( trace: TraceLog, opts: { @@ -461,51 +362,33 @@ function buildTraceBundle( const idPrefix = shortId(opts.sessionId) const contextId = `context@${idPrefix}` const pageId = `page@${idPrefix}` - const viewport = trace.metadata.viewport ?? { width: 1280, height: 720 } - const snapshots = trace.actionSnapshots ?? [] const ctxOptions = buildContextOptions(trace, contextId, wallTime) - const events: TraceEvent[] = [ctxOptions] - - // Emit initial screencast-frame (timestamp=0) using the first snapshot's - // resources so trace viewers show the page state before any interaction. - const firstSnap = snapshots.find((s) => s.screenshot) - if (firstSnap) { - const base = `${pageId}-${firstSnap.timestamp}` - const initFrame: ScreencastFrameEvent = { - type: 'screencast-frame', - pageId, - sha1: `${base}.jpeg`, - width: viewport.width, - height: viewport.height, - timestamp: 0 - } - if (firstSnap.elements && firstSnap.elements.length) { - initFrame.elements = `${base}-elements.json` - } - if (firstSnap.snapshotText) { - initFrame.snapshot = `${base}-snapshot.txt` - } - events.push(initFrame) - } - - events.push( - // Skip the first snapshot in buildScreencastFrames — it was already emitted - // as the initial t=0 frame above. - ...buildScreencastFrames( - firstSnap ? snapshots.filter((s) => s !== firstSnap) : snapshots, - pageId, - wallTime, - viewport - ), - ...buildActionEvents(trace.commands, pageId, wallTime, opts.testMetadata) + const events = buildEventStream( + trace, + ctxOptions, + pageId, + wallTime, + opts.testMetadata ) - events.sort(compareEvents) - const ctxBName = ctxOptions.title + const networkBodies = buildNetworkBodyResources(trace.networkRequests) return { traceNdjson: events.map((e) => JSON.stringify(e)).join('\n') + '\n', - networkNdjson: buildNetworkNdjson(trace.networkRequests, wallTime, pageId), - transcriptMd: generateTranscript(trace.commands, wallTime, ctxBName), - resources: buildSnapshotResources(snapshots, pageId) + networkNdjson: buildNetworkNdjson( + trace.networkRequests, + wallTime, + pageId, + networkBodies.sha1ByRequestId + ), + transcriptMd: generateTranscript( + trace.commands, + wallTime, + ctxOptions.title + ), + resources: [ + ...buildSnapshotResources(trace.actionSnapshots ?? [], pageId), + ...buildSourceResources(trace.sources), + ...networkBodies.resources + ] } } @@ -583,6 +466,9 @@ export interface WriteTraceZipOptions { format?: TraceFormat /** Test metadata keyed by testUid for Tracing.tracingGroup events. */ testMetadata?: TestMetadataMap + /** Base name for the artifact (zip file stem / directory name). Defaults to + * `trace-<sessionId>`; per-test slices pass `'trace'` inside a named folder. */ + fileStem?: string } /** @@ -618,14 +504,15 @@ export async function writeTraceZip( wallTimeOverride: capturer.startWallTime, testMetadata: opts.testMetadata } + const stem = opts.fileStem ?? `trace-${opts.sessionId}` if (opts.format === 'ndjson-directory') { - const dir = path.join(opts.outputDir, `trace-${opts.sessionId}`) + const dir = path.join(opts.outputDir, stem) await fs.mkdir(dir, { recursive: true }) await exportTraceDirectory(traceLog, dir, exportOpts) return dir } const zip = await exportTraceZip(traceLog, exportOpts) - const zipPath = path.join(opts.outputDir, `trace-${opts.sessionId}.zip`) + const zipPath = path.join(opts.outputDir, `${stem}.zip`) await fs.writeFile(zipPath, zip) return zipPath } diff --git a/packages/core/src/trace-finalizer.ts b/packages/core/src/trace-finalizer.ts new file mode 100644 index 00000000..415b6289 --- /dev/null +++ b/packages/core/src/trace-finalizer.ts @@ -0,0 +1,289 @@ +/** + * Framework-agnostic orchestration of the trace-mode export at end-of-run: + * session vs per-spec fan-out, the spec-granularity-without-boundaries + * fallback, retention gating, and per-write error isolation. The three + * adapters assemble a TraceExportContext from their own state and call + * finalizeTraceExport — all sequencing lives here. + */ + +import type { + ActionSnapshot, + DevToolsMode, + TestMetadataMap, + TraceFormat, + TraceGranularity, + TraceRetentionPolicy +} from '@wdio/devtools-shared' +import { errorMessage } from './error.js' +import { + filterTestMetadataBySpec, + filterTestMetadataByUid, + writeSpecTrace, + writeTestSliceTrace, + type SpecRange +} from './spec-trace-helpers.js' +import { shouldRetainTrace, type TestOutcome } from './trace-retention.js' +import { writeTraceZip, type TraceCapturer } from './trace-exporter.js' + +/** One artifact produced (or, when `retained` is false, decided-against) by a + * trace-mode finalize pass. */ +export interface TraceArtifact { + kind: 'trace' | 'video' + path: string + scope: 'session' | 'spec' | 'test' + /** specFile for spec scope, testUid for test scope. */ + key?: string + testUids: string[] + /** false = decided-not-to-write (reported, not written). Always true today. */ + retained: boolean +} + +export interface TraceExportContext { + mode?: DevToolsMode + /** undefined → treated as `on` (always retain) by the retention evaluator. */ + policy?: TraceRetentionPolicy + /** True when the adapter fed real per-test attempt numbers (B4); retry-aware + * policies degrade to retain-on-failure when this is false. */ + attemptInfoAvailable?: boolean + granularity?: TraceGranularity + format?: TraceFormat + capturer: TraceCapturer + actionSnapshots?: ActionSnapshot[] + sessionId: string + capabilities?: unknown + testMetadata: TestMetadataMap + /** Recorded spec boundaries; empty for session granularity. */ + ranges: SpecRange[] + /** Spec-file dedupe set shared with the adapter's boundary flushes. */ + flushed: Set<string> + /** Adapters keep their differing dir logic; range is set for spec writes. */ + resolveOutputDir: (range?: SpecRange) => string + /** Service dedupes same-timestamp snapshots; others pass identity. */ + prepareSnapshots?: (snaps: ActionSnapshot[]) => ActionSnapshot[] + /** Pending snapshot captures to settle before writing (selenium/nightwatch). */ + awaitPending?: Promise<unknown>[] + log?: (level: 'info' | 'warn', msg: string) => void + onArtifact?: (a: TraceArtifact) => void +} + +const SPEC_WITHOUT_BOUNDARIES_WARNING = + 'traceGranularity is "spec" but no spec boundaries were detected ' + + '(the runner may not expose per-test hooks). Falling back to ' + + 'session-level trace.' + +const TEST_WITHOUT_BOUNDARIES_WARNING = + 'traceGranularity is "test" but no test boundaries were detected ' + + '(the runner may not expose per-test hooks). Falling back to ' + + 'session-level trace.' + +/** Above this many slices, warn to pair granularity with a retention policy. */ +const SLICE_COUNT_WARN_THRESHOLD = 200 + +function sliceCountWarning(count: number): string { + return ( + `traceGranularity produced ${count} trace slices. Consider pairing it ` + + 'with a retention policy (e.g. tracePolicy: "retain-on-failure") to ' + + 'avoid writing hundreds of trace archives.' + ) +} + +/** Project a metadata slice onto the retention evaluator's outcome shape. */ +function toOutcomes(metadata: TestMetadataMap): TestOutcome[] { + return Array.from(metadata.values(), (m) => ({ + state: m.state, + attempt: m.attempt + })) +} + +/** + * Evaluate the retention policy for one trace slice. Adapters that feed real + * per-test attempt numbers also set `attemptInfoAvailable`; when they don't, + * retry-aware policies degrade to retain-on-failure (see trace-retention.ts). + */ +function shouldRetain( + ctx: TraceExportContext, + metadata: TestMetadataMap +): boolean { + return shouldRetainTrace(ctx.policy, { + outcomes: toOutcomes(metadata), + attemptInfoAvailable: ctx.attemptInfoAvailable ?? false + }).retain +} + +/** + * Policy-aware single-range flush: dedupes via `ctx.flushed` on the slice + * `key`, applies the retention decision, and delegates the byte-level + * slicing/naming to `writeSpecTrace` (spec slices) or `writeTestSliceTrace` + * (test slices, distinguished by `range.testUid`). Returns the artifact, or + * undefined when the range was already flushed. + */ +export async function flushRangeTrace( + ctx: TraceExportContext, + range: SpecRange, + nextRange?: SpecRange +): Promise<TraceArtifact | undefined> { + if (ctx.flushed.has(range.key)) { + return undefined + } + ctx.flushed.add(range.key) + + const isTestSlice = range.testUid !== undefined + const sliceMetadata = isTestSlice + ? filterTestMetadataByUid(ctx.testMetadata, range.testUid!) + : filterTestMetadataBySpec(ctx.testMetadata, range.specFile) + const artifact: TraceArtifact = { + kind: 'trace', + path: '', + scope: isTestSlice ? 'test' : 'spec', + key: range.key, + testUids: Array.from(sliceMetadata.keys()), + retained: shouldRetain(ctx, sliceMetadata) + } + if (!artifact.retained) { + ctx.onArtifact?.(artifact) + return artifact + } + + const writeSlice = isTestSlice ? writeTestSliceTrace : writeSpecTrace + artifact.path = await writeSlice({ + range, + nextRange, + capturer: ctx.capturer, + actionSnapshots: ctx.actionSnapshots ?? [], + sessionId: ctx.sessionId, + outputDir: ctx.resolveOutputDir(range), + format: ctx.format, + testMetadata: ctx.testMetadata, + capabilities: ctx.capabilities + }) + ctx.log?.( + 'info', + `Trace for ${isTestSlice ? 'test' : 'spec'} "${range.key}" saved to ${artifact.path}` + ) + ctx.onArtifact?.(artifact) + return artifact +} + +/** + * Flush one slice via {@link flushRangeTrace}, logging the shared spec/test + * error string on failure so a failed boundary flush can't abort the next test. + * All three adapters wrapped this identically; the label + identity are derived + * from `range.testUid` (test slice → `test "<key>"`, else `spec "<specFile>"`), + * so each call site keeps its exact message. Errors are logged and swallowed + * (resolves `undefined`), so callers `await` it when the write must land before + * a retry overwrites metadata, or fire-and-forget (`void`, or tracked in an + * in-flight list) otherwise. Callers keep their own find-current-range strategy + * and pass the resolved range in. + */ +export async function flushRangeLogged( + ctx: TraceExportContext, + range: SpecRange +): Promise<TraceArtifact | undefined> { + try { + return await flushRangeTrace(ctx, range) + } catch (err) { + const label = + range.testUid !== undefined + ? `test "${range.key}"` + : `spec "${range.specFile}"` + ctx.log?.( + 'warn', + `Failed to flush trace for ${label}: ${errorMessage(err)}` + ) + return undefined + } +} + +async function writeSessionTrace( + ctx: TraceExportContext +): Promise<TraceArtifact | undefined> { + const prepare = ctx.prepareSnapshots ?? ((s) => s) + const snapshots = prepare(ctx.actionSnapshots ?? []) + const artifact: TraceArtifact = { + kind: 'trace', + path: '', + scope: 'session', + testUids: Array.from(ctx.testMetadata.keys()), + retained: shouldRetain(ctx, ctx.testMetadata) + } + if (!artifact.retained) { + ctx.onArtifact?.(artifact) + return artifact + } + + artifact.path = await writeTraceZip(ctx.capturer, { + outputDir: ctx.resolveOutputDir(), + sessionId: ctx.sessionId, + capabilities: ctx.capabilities, + actionSnapshots: snapshots.length ? snapshots : undefined, + format: ctx.format, + testMetadata: ctx.testMetadata + }) + ctx.log?.('info', `Trace saved to ${artifact.path}`) + ctx.onArtifact?.(artifact) + return artifact +} + +/** Run one write, logging and swallowing its error so siblings still write. */ +async function safely( + ctx: TraceExportContext, + write: () => Promise<TraceArtifact | undefined> +): Promise<TraceArtifact | undefined> { + try { + return await write() + } catch (err) { + ctx.log?.('warn', `trace write failed: ${errorMessage(err)}`) + return undefined + } +} + +async function flushAllRanges( + ctx: TraceExportContext +): Promise<TraceArtifact[]> { + const artifacts: TraceArtifact[] = [] + // Bound each slice by the next range's start indices; the final range (no + // nextRange) runs to the end of the arrays. Without this, every slice would + // run to the end and each test slice would swallow all later tests. + for (let i = 0; i < ctx.ranges.length; i++) { + const range = ctx.ranges[i]! + const nextRange = ctx.ranges[i + 1] + const artifact = await safely(ctx, () => + flushRangeTrace(ctx, range, nextRange) + ) + if (artifact) { + artifacts.push(artifact) + } + } + return artifacts +} + +/** + * Entry point for the after/end-of-run hook. No-op outside trace mode. Awaits + * any pending snapshot captures, then fans out to per-spec, per-test, or + * session writes. `spec`/`test` granularity with no recorded boundaries warns + * and falls back to a single session-level trace. + */ +export async function finalizeTraceExport( + ctx: TraceExportContext +): Promise<TraceArtifact[]> { + if (ctx.mode !== 'trace') { + return [] + } + if (ctx.awaitPending?.length) { + await Promise.allSettled(ctx.awaitPending) + } + const sliced = ctx.granularity === 'spec' || ctx.granularity === 'test' + if (sliced && ctx.ranges.length > 0) { + if (ctx.ranges.length > SLICE_COUNT_WARN_THRESHOLD) { + ctx.log?.('warn', sliceCountWarning(ctx.ranges.length)) + } + return flushAllRanges(ctx) + } + if (ctx.granularity === 'spec') { + ctx.log?.('warn', SPEC_WITHOUT_BOUNDARIES_WARNING) + } else if (ctx.granularity === 'test') { + ctx.log?.('warn', TEST_WITHOUT_BOUNDARIES_WARNING) + } + const artifact = await safely(ctx, () => writeSessionTrace(ctx)) + return artifact ? [artifact] : [] +} diff --git a/packages/core/src/trace-frame-snapshots.ts b/packages/core/src/trace-frame-snapshots.ts new file mode 100644 index 00000000..618b612a --- /dev/null +++ b/packages/core/src/trace-frame-snapshots.ts @@ -0,0 +1,154 @@ +// Image-backed frame-snapshot synthesis: each action screenshot becomes a +// minimal DOM document so standard trace viewers render the action pane. +// Compatibility shim until real DOM snapshots are captured. + +import type { ActionSnapshot } from '@wdio/devtools-shared' + +const SNAPSHOT_DOCTYPE = 'html' +const FALLBACK_FRAME_URL = 'about:blank' +const BODY_STYLE = 'margin:0' +const IMAGE_STYLE = 'display:block;width:100vw;height:100vh;object-fit:contain' + +/** Serialized DOM node: text, or a [TAG, attributes, ...children] tuple. */ +export type FrameSnapshotNode = + | string + | [string, Record<string, string>, ...FrameSnapshotNode[]] + +export interface FrameSnapshotResourceOverride { + url: string + sha1?: string + ref?: number +} + +export interface FrameSnapshot { + callId: string + snapshotName: string + pageId: string + frameId: string + frameUrl: string + doctype?: string + html: FrameSnapshotNode + viewport: { width: number; height: number } + timestamp: number + wallTime: number + collectionTime: number + resourceOverrides: FrameSnapshotResourceOverride[] + isMainFrame: boolean +} + +export interface FrameSnapshotEvent { + type: 'frame-snapshot' + snapshot: FrameSnapshot +} + +export interface FrameSnapshotRef { + callId: string + snapshotName: string + snapshot: ActionSnapshot +} + +/** Correlates captured screenshots to callIds as the exporter assigns them. */ +export class FrameSnapshotIndex { + #byTimestamp = new Map<number, ActionSnapshot>() + #refs: FrameSnapshotRef[] = [] + #lastName?: string + + constructor(snapshots: ActionSnapshot[]) { + for (const snap of snapshots) { + if (!snap.screenshot) { + continue + } + const current = this.#byTimestamp.get(snap.timestamp) + // Same-timestamp duplicates keep the richest capture (dedupe parity). + if ( + !current || + snap.screenshot.length > (current.screenshot?.length ?? 0) + ) { + this.#byTimestamp.set(snap.timestamp, snap) + } + } + } + + /** Snapshot name representing the page state before the next action. */ + beforeName(): string | undefined { + return this.#lastName + } + + /** Claims the screenshot captured at the command's completion, if any. */ + claimAfter(timestamp: number, callId: string): string | undefined { + const snap = this.#byTimestamp.get(timestamp) + if (!snap) { + return undefined + } + this.#byTimestamp.delete(timestamp) + const snapshotName = `after@${callId}` + this.#refs.push({ callId, snapshotName, snapshot: snap }) + this.#lastName = snapshotName + return snapshotName + } + + refs(): FrameSnapshotRef[] { + return this.#refs + } +} + +function frameIdForPage(pageId: string): string { + const suffix = pageId.startsWith('page@') + ? pageId.slice('page@'.length) + : pageId + return `frame@${suffix}` +} + +// Captures come from WebDriver screenshots (PNG) or CDP screencasts (JPEG), +// so the mime is sniffed from the base64 magic rather than assumed. +function imageMimeType(base64: string): string { + return base64.startsWith('iVBOR') ? 'image/png' : 'image/jpeg' +} + +function imageDocument(snap: ActionSnapshot): FrameSnapshotNode { + const screenshot = snap.screenshot ?? '' + return [ + 'HTML', + {}, + ['HEAD', {}, ['BASE', { href: snap.url ?? FALLBACK_FRAME_URL }]], + [ + 'BODY', + { style: BODY_STYLE }, + [ + 'IMG', + { + src: `data:${imageMimeType(screenshot)};base64,${screenshot}`, + style: IMAGE_STYLE + } + ] + ] + ] +} + +/** One frame-snapshot event per claimed screenshot, viewer-shape exact. */ +export function buildImageFrameSnapshots( + refs: FrameSnapshotRef[], + pageId: string, + wallTime: number, + viewport: { width: number; height: number } +): FrameSnapshotEvent[] { + const frameId = frameIdForPage(pageId) + return refs.map((ref) => ({ + type: 'frame-snapshot' as const, + snapshot: { + callId: ref.callId, + snapshotName: ref.snapshotName, + pageId, + frameId, + frameUrl: ref.snapshot.url ?? FALLBACK_FRAME_URL, + doctype: SNAPSHOT_DOCTYPE, + html: imageDocument(ref.snapshot), + viewport: { width: viewport.width, height: viewport.height }, + timestamp: Math.max(0, ref.snapshot.timestamp - wallTime), + wallTime: ref.snapshot.timestamp, + collectionTime: 0, + resourceOverrides: [], + isMainFrame: true + } + })) +} diff --git a/packages/core/src/trace-har.ts b/packages/core/src/trace-har.ts index daf7f911..e26ddfb9 100644 --- a/packages/core/src/trace-har.ts +++ b/packages/core/src/trace-har.ts @@ -3,6 +3,18 @@ import type { NetworkRequest } from '@wdio/devtools-shared' +/** Bodies below this inline as plain `text`; larger ones rely on `_sha1` only. */ +const INLINE_BODY_TEXT_MAX_BYTES = 8 * 1024 + +export interface HarResponseContent { + size: number + mimeType: string + /** Inline body fallback for small textual bodies. */ + text?: string + /** Content-addressed body resource name under `resources/`. */ + _sha1?: string +} + export interface ResourceSnapshotEntry { type: 'resource-snapshot' snapshot: { @@ -24,7 +36,7 @@ export interface ResourceSnapshotEntry { httpVersion: string cookies: unknown[] headers: { name: string; value: string }[] - content: { size: number; mimeType: string } + content: HarResponseContent redirectURL: string headersSize: number bodySize: number @@ -54,8 +66,28 @@ function toQueryString(url: string): { name: string; value: string }[] { } } +function toHarContent( + entry: NetworkRequest, + mimeType: string, + bodySha1: string | undefined +): HarResponseContent { + const content: HarResponseContent = { size: entry.size ?? 0, mimeType } + const body = entry.responseBody + if (body === undefined) { + return content + } + if (Buffer.byteLength(body, 'utf8') < INLINE_BODY_TEXT_MAX_BYTES) { + content.text = body + } + if (bodySha1) { + content._sha1 = bodySha1 + } + return content +} + export function networkRequestToHar( - entry: NetworkRequest + entry: NetworkRequest, + opts: { bodySha1?: string } = {} ): ResourceSnapshotEntry { const startedDateTime = new Date(entry.timestamp).toISOString() const duration = @@ -84,7 +116,7 @@ export function networkRequestToHar( httpVersion: 'HTTP/1.1', cookies: [], headers: toHeaderArray(responseHeaders), - content: { size: entry.size ?? 0, mimeType }, + content: toHarContent(entry, mimeType, opts.bodySha1), redirectURL: '', headersSize: -1, bodySize: entry.size ?? -1 diff --git a/packages/core/src/trace-retention.ts b/packages/core/src/trace-retention.ts new file mode 100644 index 00000000..6688dfcb --- /dev/null +++ b/packages/core/src/trace-retention.ts @@ -0,0 +1,92 @@ +import type { + DevToolsMode, + TestStatus, + TraceRetentionPolicy +} from '@wdio/devtools-shared' + +/** + * Pure retention policy evaluation for trace mode. Adapters collect the + * observed test outcomes for a trace scope (session/spec/test) and ask + * whether the written trace should be kept. + */ + +/** Every policy the evaluator recognizes. An unknown string (from a JS config + * that slipped past the type) is treated as `on` — fail open, never silently + * drop a trace the user might need. */ +const KNOWN_POLICIES = new Set<TraceRetentionPolicy>([ + 'on', + 'retain-on-failure', + 'retain-on-first-failure', + 'on-first-retry', + 'on-all-retries', + 'retain-on-failure-and-retries' +]) + +export interface TestOutcome { + state?: TestStatus + attempt?: number +} + +export interface RetentionInput { + outcomes: Iterable<TestOutcome> + /** Whether the adapter can distinguish retries (per-outcome `attempt`). */ + attemptInfoAvailable: boolean +} + +export interface RetentionDecision { + retain: boolean + /** Set when a retry-aware policy fell back to `retain-on-failure` because attempt info was unavailable. */ + degradedToFailure?: boolean + /** Set when no outcomes were observed (standalone scripts) — retained rather than risk losing a needed trace. */ + failOpen?: boolean +} + +export function shouldRetainTrace( + policy: TraceRetentionPolicy | undefined, + input: RetentionInput +): RetentionDecision { + if (policy === undefined || policy === 'on' || !KNOWN_POLICIES.has(policy)) { + return { retain: true } + } + const outcomes = Array.from(input.outcomes) + if (outcomes.length === 0) { + return { retain: true, failOpen: true } + } + const anyFailed = outcomes.some((o) => o.state === 'failed') + if (!input.attemptInfoAvailable && policy !== 'retain-on-failure') { + return { retain: anyFailed, degradedToFailure: true } + } + switch (policy) { + case 'retain-on-failure': + return { retain: anyFailed } + case 'retain-on-first-failure': + return { + retain: outcomes.some( + (o) => o.state === 'failed' && (o.attempt ?? 0) === 0 + ) + } + case 'on-first-retry': + return { retain: outcomes.some((o) => o.attempt === 1) } + case 'on-all-retries': + return { retain: outcomes.some((o) => (o.attempt ?? 0) >= 1) } + case 'retain-on-failure-and-retries': + return { + retain: anyFailed || outcomes.some((o) => (o.attempt ?? 0) >= 1) + } + } +} + +/** + * Warning text when a retention policy is configured outside trace mode, where + * it has no effect (the finalizer no-ops in live mode). Returns undefined when + * there is nothing to warn about, so each adapter can `if (msg) log.warn(msg)`. + */ +export function tracePolicyModeWarning( + policy: TraceRetentionPolicy | undefined, + mode: DevToolsMode | undefined +): string | undefined { + if (policy === undefined || mode === 'trace') { + return undefined + } + return 'tracePolicy only applies in trace mode; ignoring it because mode is not "trace".' +} diff --git a/packages/core/src/trace-snapshots.ts b/packages/core/src/trace-snapshots.ts new file mode 100644 index 00000000..ab7ed3f5 --- /dev/null +++ b/packages/core/src/trace-snapshots.ts @@ -0,0 +1,128 @@ +// Per-action snapshot resources and the screencast-frame filmstrip derived +// from them. Split from trace-exporter.ts; the exporter composes these into +// the trace event stream. + +import type { ActionSnapshot } from '@wdio/devtools-shared' +import type { TraceZipResource } from './trace-zip-writer.js' + +export interface ScreencastFrameEvent { + type: 'screencast-frame' + pageId: string + sha1: string + elements?: string + snapshot?: string + width: number + height: number + timestamp: number +} + +// Two snapshots at the same timestamp (a real post-action capture and a blank +// end-of-scenario one) map to the same `${pageId}-${ts}` resource name, so a +// last-wins write lets a blank frame clobber the real one — and the real +// screenshot and real elements can land on different captures. Collapse to one +// per timestamp: largest screenshot, richest metadata, merged across duplicates. +function collapseByTimestamp(snapshots: ActionSnapshot[]): ActionSnapshot[] { + const byTs = new Map<number, ActionSnapshot>() + const order: number[] = [] + for (const snap of snapshots) { + const existing = byTs.get(snap.timestamp) + if (!existing) { + byTs.set(snap.timestamp, { ...snap }) + order.push(snap.timestamp) + continue + } + if ((snap.screenshot?.length ?? 0) > (existing.screenshot?.length ?? 0)) { + existing.screenshot = snap.screenshot + } + if (!existing.elements?.length && snap.elements?.length) { + existing.elements = snap.elements + } + if (!existing.snapshotText && snap.snapshotText) { + existing.snapshotText = snap.snapshotText + } + } + return order.map((ts) => byTs.get(ts)!) +} + +export function buildSnapshotResources( + rawSnapshots: ActionSnapshot[], + pageId: string +): TraceZipResource[] { + const snapshots = collapseByTimestamp(rawSnapshots) + const out: TraceZipResource[] = [] + for (const snap of snapshots) { + const base = `${pageId}-${snap.timestamp}` + if (snap.screenshot) { + out.push({ + resourceName: `${base}.jpeg`, + data: Buffer.from(snap.screenshot, 'base64') + }) + } + if (snap.elements && snap.elements.length) { + out.push({ + resourceName: `${base}-elements.json`, + data: Buffer.from(JSON.stringify(snap.elements), 'utf8') + }) + } + if (snap.snapshotText) { + out.push({ + resourceName: `${base}-snapshot.txt`, + data: Buffer.from(snap.snapshotText, 'utf8') + }) + } + } + return out +} + +function frameForSnapshot( + snap: ActionSnapshot, + pageId: string, + timestamp: number, + viewport: { width: number; height: number } +): ScreencastFrameEvent { + const base = `${pageId}-${snap.timestamp}` + const frame: ScreencastFrameEvent = { + type: 'screencast-frame', + pageId, + sha1: `${base}.jpeg`, + width: viewport.width, + height: viewport.height, + timestamp + } + if (snap.elements && snap.elements.length) { + frame.elements = `${base}-elements.json` + } + if (snap.snapshotText) { + frame.snapshot = `${base}-snapshot.txt` + } + return frame +} + +/** Filmstrip events; the first snapshot is re-anchored to t=0 for pre-interaction state. */ +export function buildFilmstripEvents( + rawSnapshots: ActionSnapshot[], + pageId: string, + wallTime: number, + viewport: { width: number; height: number } +): ScreencastFrameEvent[] { + const snapshots = collapseByTimestamp(rawSnapshots) + const firstSnap = snapshots.find((s) => s.screenshot) + const events: ScreencastFrameEvent[] = [] + if (firstSnap) { + events.push(frameForSnapshot(firstSnap, pageId, 0, viewport)) + } + for (const snap of snapshots) { + if (snap === firstSnap || !snap.screenshot) { + continue + } + events.push( + frameForSnapshot( + snap, + pageId, + Math.max(0, snap.timestamp - wallTime), + viewport + ) + ) + } + return events +} diff --git a/packages/core/src/trace-sources.ts b/packages/core/src/trace-sources.ts new file mode 100644 index 00000000..9e708de7 --- /dev/null +++ b/packages/core/src/trace-sources.ts @@ -0,0 +1,65 @@ +// Source files and per-action call stacks for the trace: sources are written +// as path-addressed `src@<sha1(path)>.txt` resources, and each action's +// captured `<file>:<line>[:<column>]` callSource becomes an inline stack frame. + +import { sha1Hex } from './sha1.js' +import type { TraceZipResource } from './trace-zip-writer.js' + +export interface StackFrame { + file: string + line: number + column: number +} + +// Test files are small; anything bigger is generated/bundled and not worth shipping. +const MAX_SOURCE_BYTES = 2 * 1024 * 1024 + +export function sourceResourceName(filePath: string): string { + return `src@${sha1Hex(filePath)}.txt` +} + +// Splits one trailing `:<digits>` segment; `sep <= 0` keeps Windows drive +// letters (`C:...`) and colon-less paths intact. +function splitNumericSuffix(value: string): { rest: string; num?: number } { + const sep = value.lastIndexOf(':') + if (sep <= 0) { + return { rest: value } + } + const digits = value.slice(sep + 1) + if (!/^\d+$/.test(digits)) { + return { rest: value } + } + return { rest: value.slice(0, sep), num: Number(digits) } +} + +/** Inline stack from a captured `<file>:<line>[:<column>]` callSource. */ +export function callSourceToStack( + callSource?: string +): StackFrame[] | undefined { + if (!callSource || callSource === 'unknown:0') { + return undefined + } + const last = splitNumericSuffix(callSource) + if (last.num === undefined) { + return [{ file: callSource, line: 0, column: 0 }] + } + const prev = splitNumericSuffix(last.rest) + if (prev.num === undefined) { + return [{ file: last.rest, line: last.num, column: 0 }] + } + return [{ file: prev.rest, line: prev.num, column: last.num }] +} + +export function buildSourceResources( + sources: Record<string, string> +): TraceZipResource[] { + const out: TraceZipResource[] = [] + for (const [filePath, text] of Object.entries(sources)) { + const data = Buffer.from(text, 'utf8') + if (data.byteLength > MAX_SOURCE_BYTES) { + continue + } + out.push({ resourceName: sourceResourceName(filePath), data }) + } + return out +} diff --git a/packages/core/tests/action-mapping.test.ts b/packages/core/tests/action-mapping.test.ts new file mode 100644 index 00000000..ca9ebd00 --- /dev/null +++ b/packages/core/tests/action-mapping.test.ts @@ -0,0 +1,163 @@ +import { describe, it, expect } from 'vitest' +import { ACTION_MAP } from '@wdio/devtools-shared' +import { formatActionTitle, mapCommandToAction } from '../src/action-mapping.js' + +describe('mapCommandToAction for read/query commands', () => { + const elementReads: [string, string][] = [ + ['getText', 'getText'], + ['getValue', 'getValue'], + ['getAttribute', 'getAttribute'], + ['getProperty', 'getProperty'], + ['getCSSProperty', 'getCSSProperty'], + ['getTagName', 'getTagName'], + ['getLocation', 'getLocation'], + ['getSize', 'getSize'], + ['isDisplayed', 'isDisplayed'], + ['isExisting', 'isExisting'], + ['isEnabled', 'isEnabled'], + ['isSelected', 'isSelected'], + ['isClickable', 'isClickable'], + ['isFocused', 'isFocused'], + ['waitForDisplayed', 'waitForDisplayed'], + ['waitForExist', 'waitForExist'], + ['waitForEnabled', 'waitForEnabled'], + ['waitForClickable', 'waitForClickable'] + ] + + it.each(elementReads)('maps %s to Element.%s', (command, method) => { + expect(mapCommandToAction(command)).toEqual({ class: 'Element', method }) + }) + + it('maps waitUntil to Browser.waitForFunction', () => { + expect(mapCommandToAction('waitUntil')).toEqual({ + class: 'Browser', + method: 'waitForFunction' + }) + }) + + it('maps page/browser reads to the Page class', () => { + expect(mapCommandToAction('getTitle')).toEqual({ + class: 'Page', + method: 'getTitle' + }) + expect(mapCommandToAction('getUrl')).toEqual({ + class: 'Page', + method: 'getUrl' + }) + expect(mapCommandToAction('getPageSource')).toEqual({ + class: 'Page', + method: 'getPageSource' + }) + }) + + it('normalizes Selenium read aliases onto the WDIO names', () => { + expect(mapCommandToAction('getCssValue')).toEqual({ + class: 'Element', + method: 'getCSSProperty' + }) + expect(mapCommandToAction('getCurrentUrl')).toEqual({ + class: 'Page', + method: 'getUrl' + }) + expect(mapCommandToAction('getRect')).toEqual({ + class: 'Element', + method: 'getRect' + }) + }) +}) + +describe('mapCommandToAction still excludes noisy/internal commands', () => { + it.each([ + 'clearValue', + 'addValue', + 'executeScript', + '$', + '$$', + 'findElement', + 'findElements', + 'getElement', + 'getElements' + ])('returns null for %s', (command) => { + expect(mapCommandToAction(command)).toBeNull() + }) +}) + +describe('formatActionTitle for read/query commands', () => { + it('renders a selector-first title when the selector is the first arg', () => { + expect( + formatActionTitle({ class: 'Element', method: 'getText' }, ['#sel']) + ).toBe('Element.getText("#sel")') + expect( + formatActionTitle({ class: 'Element', method: 'isDisplayed' }, ['#sel']) + ).toBe('Element.isDisplayed("#sel")') + }) + + it('uses params.selector when there is no positional arg', () => { + expect( + formatActionTitle({ class: 'Element', method: 'waitForExist' }, [], { + selector: '#sel' + }) + ).toBe('Element.waitForExist("#sel")') + }) + + it('falls back to an argument-free title when nothing identifies the target', () => { + expect(formatActionTitle({ class: 'Page', method: 'getTitle' }, [])).toBe( + 'Page.getTitle()' + ) + }) +}) + +describe('ACTION_MAP forward/reverse integrity', () => { + it('never maps two commands to the same class.method with conflicting intent', () => { + // The reader derives REVERSE_ACTION_MAP by keeping the first command per + // class.method. Duplicated targets are only safe when the runner commands + // are true synonyms (e.g. url/navigateTo/get all → Page.navigate). + const targetToCommands = new Map<string, string[]>() + for (const [command, action] of Object.entries(ACTION_MAP)) { + const key = `${action.class}.${action.method}` + targetToCommands.set(key, [...(targetToCommands.get(key) ?? []), command]) + } + + // The new read methods must not collide with an existing interaction target. + const interactionTargets = new Set([ + 'Element.click', + 'Element.fill', + 'Element.clear', + 'Element.submit', + 'Element.hover', + 'Element.tap', + 'Page.navigate', + 'Keyboard.press' + ]) + for (const method of [ + 'getText', + 'getValue', + 'getAttribute', + 'getProperty', + 'getCSSProperty', + 'getTagName', + 'getLocation', + 'getSize', + 'getRect', + 'isDisplayed', + 'isExisting', + 'isEnabled', + 'isSelected', + 'isClickable', + 'isFocused', + 'waitForDisplayed', + 'waitForExist', + 'waitForEnabled', + 'waitForClickable' + ]) { + expect(interactionTargets.has(`Element.${method}`)).toBe(false) + } + + // getCssValue normalizes onto getCSSProperty, and getCurrentUrl onto getUrl; + // the WDIO name is listed first so the reverse map keeps it. + const cssCommands = targetToCommands.get('Element.getCSSProperty') ?? [] + expect(cssCommands[0]).toBe('getCSSProperty') + const urlCommands = targetToCommands.get('Page.getUrl') ?? [] + expect(urlCommands[0]).toBe('getUrl') + }) +}) diff --git a/packages/core/tests/assert-patcher.test.ts b/packages/core/tests/assert-patcher.test.ts index feaf746b..38bea721 100644 --- a/packages/core/tests/assert-patcher.test.ts +++ b/packages/core/tests/assert-patcher.test.ts @@ -1,12 +1,27 @@ -import { describe, it, expect, beforeEach } from 'vitest' +import { describe, it, expect, beforeEach, vi } from 'vitest' import assert from 'node:assert' import { ASSERT_PATCHED_SYMBOL, TRACKED_ASSERT_METHODS, + capturedAssertToCommandLog, + matcherAssertionToCommandLog, patchNodeAssert, safeSerializeAssertArg, type CapturedAssert } from '../src/assert-patcher.js' +import { getCallSourceFromStack } from '../src/stack.js' + +// Stub the call-source resolver so tests choose whether an assert looks like +// it originated in user code (a frame) or a dependency/internal (no frame). +vi.mock('../src/stack.js', () => ({ + getCallSourceFromStack: vi.fn() +})) + +const USER_FRAME = { + filePath: '/specs/login.ts', + callSource: '/specs/login.ts:12:3' +} +const INTERNAL_FRAME = { filePath: undefined, callSource: 'unknown:0' } // Snapshot real methods once so each test starts from a fresh assert. const ASSERT_MUT = assert as unknown as Record<string | symbol, unknown> @@ -20,6 +35,8 @@ beforeEach(() => { for (const m of TRACKED_ASSERT_METHODS) { ASSERT_MUT[m] = originals[m] } + // Default: every assert looks like it came from user code so captures fire. + vi.mocked(getCallSourceFromStack).mockReturnValue(USER_FRAME) }) describe('safeSerializeAssertArg', () => { @@ -67,6 +84,51 @@ describe('patchNodeAssert', () => { expect(results).toEqual(['passed', undefined]) // first resolved, second rejected }) + // Only assertions that originate in user test code should reach the trace. + // Dependency/framework-internal asserts have no user-code frame on the + // stack (getCallSourceFromStack yields 'unknown:0') and must be dropped. + describe('user-origin filtering', () => { + it('drops a passing assert that has no user-code frame', () => { + vi.mocked(getCallSourceFromStack).mockReturnValue(INTERNAL_FRAME) + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + assert.equal(1, 1) + expect(captured).toHaveLength(0) + }) + + it('drops a failing internal assert but still re-throws', () => { + vi.mocked(getCallSourceFromStack).mockReturnValue(INTERNAL_FRAME) + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + expect(() => assert.equal(1, 2)).toThrow() + expect(captured).toHaveLength(0) + }) + + it('drops an internal async assert (rejects/doesNotReject)', async () => { + vi.mocked(getCallSourceFromStack).mockReturnValue(INTERNAL_FRAME) + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + await assert.doesNotReject(async () => 1) + await expect(assert.rejects(async () => 1)).rejects.toThrow() + expect(captured).toHaveLength(0) + }) + + it('keeps a user-origin assert and carries its callSource through', () => { + vi.mocked(getCallSourceFromStack).mockReturnValue(USER_FRAME) + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + assert.equal(1, 1) + expect(() => assert.equal(1, 2)).toThrow() + expect(captured).toHaveLength(2) + expect(captured[0]).toMatchObject({ + result: 'passed', + callSource: USER_FRAME.callSource + }) + expect(captured[1].result).toBeUndefined() + expect(captured[1].error).toBeInstanceOf(Error) + }) + }) + it('is idempotent — second patch is a no-op and sets the guard symbol', () => { const a: CapturedAssert[] = [] patchNodeAssert((c) => a.push(c)) @@ -86,4 +148,140 @@ describe('patchNodeAssert', () => { assert.match('hello', /hello/) expect(captured[0].args).toEqual(['hello', '/hello/']) }) + + // Regression: node:assert dispatches match/doesNotMatch on a function + // identity check against the module binding (`fn === assert.match` on + // Node ≤20). A wrapper left installed during the call inverted `match` + // into `doesNotMatch` — passing regexes threw and failing ones passed. + describe('match/doesNotMatch inversion regression', () => { + it('records a failed capture when match does not match', () => { + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + expect(() => assert.match('a', /b/)).toThrow() + expect(captured[0].command).toBe('assert.match') + expect(captured[0].result).toBeUndefined() + expect(captured[0].error).toBeInstanceOf(Error) + }) + + it('records a passed capture when match matches', () => { + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + expect(() => assert.match('a', /a/)).not.toThrow() + expect(captured[0].result).toBe('passed') + expect(captured[0].error).toBeUndefined() + }) + + it('records a failed capture when doesNotMatch matches', () => { + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + expect(() => assert.doesNotMatch('a', /a/)).toThrow() + expect(captured[0].command).toBe('assert.doesNotMatch') + expect(captured[0].result).toBeUndefined() + expect(captured[0].error).toBeInstanceOf(Error) + }) + + it('records a passed capture when doesNotMatch does not match', () => { + const captured: CapturedAssert[] = [] + patchNodeAssert((c) => captured.push(c)) + expect(() => assert.doesNotMatch('a', /b/)).not.toThrow() + expect(captured[0].result).toBe('passed') + expect(captured[0].error).toBeUndefined() + }) + + it('restores the original binding during the call and the wrapper after', () => { + patchNodeAssert(() => {}) + const wrapped = ASSERT_MUT['throws'] + let duringCall: unknown + assert.throws(() => { + duringCall = ASSERT_MUT['throws'] + throw new Error('boom') + }) + expect(duringCall).toBe(originals['throws']) + expect(ASSERT_MUT['throws']).toBe(wrapped) + }) + + it('re-installs the wrapper after a failing call', () => { + patchNodeAssert(() => {}) + const wrapped = ASSERT_MUT['equal'] + expect(() => assert.equal(1, 2)).toThrow() + expect(ASSERT_MUT['equal']).toBe(wrapped) + }) + }) +}) + +describe('capturedAssertToCommandLog', () => { + const base: CapturedAssert = { + command: 'assert.strictEqual', + args: ['a', 'b'], + result: undefined, + error: undefined, + callSource: '/specs/login.ts:12:3', + timestamp: 1234 + } + + it('maps the captured shape onto CommandLog with startTime = timestamp', () => { + const entry = capturedAssertToCommandLog( + { ...base, result: 'passed', error: undefined }, + 'test-1' + ) + expect(entry).toEqual({ + command: 'assert.strictEqual', + args: ['a', 'b'], + result: 'passed', + timestamp: 1234, + startTime: 1234, + callSource: '/specs/login.ts:12:3', + testUid: 'test-1' + }) + }) + + it('serializes the error to a plain {name, message, stack} object', () => { + const error = new Error('expected a to be b') + const entry = capturedAssertToCommandLog({ ...base, error }) + expect(entry.error).toEqual({ + name: 'Error', + message: 'expected a to be b', + stack: error.stack + }) + expect(entry.testUid).toBeUndefined() + }) +}) + +describe('matcherAssertionToCommandLog', () => { + it('builds a passing expect.<method> command (default prefix)', () => { + const entry = matcherAssertionToCommandLog( + { method: 'toHaveTitle', args: ['The Internet'], passed: true }, + 'uid-1' + ) + expect(entry).toMatchObject({ + command: 'expect.toHaveTitle', + args: ['The Internet'], + result: 'passed', + testUid: 'uid-1' + }) + expect(entry.error).toBeUndefined() + }) + + it('carries the ANSI-stripped message as the error when failed', () => { + const entry = matcherAssertionToCommandLog({ + method: 'toHaveText', + args: ['foo'], + passed: false, + message: () => 'expected foo' + }) + expect(entry.result).toBeUndefined() + expect(entry.error).toMatchObject({ message: 'expected foo' }) + }) + + it('honors a non-default prefix and sanitizes args', () => { + const entry = matcherAssertionToCommandLog({ + prefix: 'verify', + method: 'equal', + args: [/re/, () => 1], + passed: true + }) + expect(entry.command).toBe('verify.equal') + // RegExp → string, function → '[Function]' (via safeSerializeAssertArg). + expect(entry.args).toEqual(['/re/', '[Function]']) + }) }) diff --git a/packages/core/tests/attempt-tracker.test.ts b/packages/core/tests/attempt-tracker.test.ts new file mode 100644 index 00000000..5d9a44b4 --- /dev/null +++ b/packages/core/tests/attempt-tracker.test.ts @@ -0,0 +1,46 @@ +import { describe, it, expect } from 'vitest' +import { TestAttemptTracker } from '../src/attempt-tracker.js' + +describe('TestAttemptTracker', () => { + it('numbers the first start 0 and each rerun +1', () => { + const t = new TestAttemptTracker() + expect(t.recordStart('a')).toBe(0) + expect(t.recordStart('a')).toBe(1) + expect(t.recordStart('a')).toBe(2) + }) + + it('tracks attempts per uid independently', () => { + const t = new TestAttemptTracker() + expect(t.recordStart('a')).toBe(0) + expect(t.recordStart('b')).toBe(0) + expect(t.recordStart('a')).toBe(1) + expect(t.attemptFor('a')).toBe(1) + expect(t.attemptFor('b')).toBe(0) + }) + + it('attemptFor is undefined for an unseen uid', () => { + const t = new TestAttemptTracker() + expect(t.attemptFor('missing')).toBeUndefined() + }) + + it('sawRetry flips only after a second start of some test', () => { + const t = new TestAttemptTracker() + expect(t.sawRetry).toBe(false) + t.recordStart('a') + t.recordStart('b') + expect(t.sawRetry).toBe(false) + t.recordStart('a') + expect(t.sawRetry).toBe(true) + }) + + it('reset clears attempts and the retry flag', () => { + const t = new TestAttemptTracker() + t.recordStart('a') + t.recordStart('a') + expect(t.sawRetry).toBe(true) + t.reset() + expect(t.sawRetry).toBe(false) + expect(t.attemptFor('a')).toBeUndefined() + expect(t.recordStart('a')).toBe(0) + }) +}) diff --git a/packages/core/tests/output-dir.test.ts b/packages/core/tests/output-dir.test.ts index 12d80315..f44ba3ef 100644 --- a/packages/core/tests/output-dir.test.ts +++ b/packages/core/tests/output-dir.test.ts @@ -4,6 +4,9 @@ import path from 'node:path' import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { resolveAdapterOutputDir } from '../src/output-dir.js' +/** Every resolved dir is grouped under this subfolder (Playwright-style). */ +const grouped = (base: string) => path.join(base, 'test-results') + describe('resolveAdapterOutputDir', () => { let tmpDir: string @@ -15,10 +18,16 @@ describe('resolveAdapterOutputDir', () => { fs.rmSync(tmpDir, { recursive: true, force: true }) }) + it('groups the resolved dir under a test-results/ subfolder', () => { + expect(resolveAdapterOutputDir({ fallbackDir: tmpDir })).toBe( + path.join(tmpDir, 'test-results') + ) + }) + it('returns userConfiguredDir verbatim when set, even if non-existent', () => { expect( resolveAdapterOutputDir({ userConfiguredDir: '/whatever/path' }) - ).toBe('/whatever/path') + ).toBe(grouped('/whatever/path')) }) it('prefers testFilePath dir over configPath dir over fallback', () => { @@ -35,14 +44,14 @@ describe('resolveAdapterOutputDir', () => { configPath, fallbackDir: tmpDir }) - ).toBe(path.dirname(testFile)) + ).toBe(grouped(path.dirname(testFile))) }) it('falls back to configPath dir when testFilePath is missing', () => { const configPath = path.join(tmpDir, 'wdio.conf.ts') fs.writeFileSync(configPath, '') expect(resolveAdapterOutputDir({ configPath, fallbackDir: tmpDir })).toBe( - tmpDir + grouped(tmpDir) ) }) @@ -59,11 +68,11 @@ describe('resolveAdapterOutputDir', () => { testFilePath: testFile, configPath }) - ).toBe(tmpDir) + ).toBe(grouped(tmpDir)) }) it('falls back to process.cwd() when no inputs are given', () => { - expect(resolveAdapterOutputDir()).toBe(process.cwd()) + expect(resolveAdapterOutputDir()).toBe(grouped(process.cwd())) }) it('falls back to fallbackDir when given and none of the candidates are writable', () => { @@ -72,12 +81,12 @@ describe('resolveAdapterOutputDir', () => { testFilePath: '/definitely/missing/a.test.ts', fallbackDir: tmpDir }) - ).toBe(tmpDir) + ).toBe(grouped(tmpDir)) }) it('userConfiguredDir bypasses node_modules skip (explicit opt-in)', () => { const nm = '/some/node_modules/pkg/dir' - expect(resolveAdapterOutputDir({ userConfiguredDir: nm })).toBe(nm) + expect(resolveAdapterOutputDir({ userConfiguredDir: nm })).toBe(grouped(nm)) }) it('returns fallback (cwd) when all candidate dirs are missing', () => { @@ -86,6 +95,6 @@ describe('resolveAdapterOutputDir', () => { testFilePath: '/missing/x.test.ts', configPath: '/missing/wdio.conf.ts' }) - ).toBe(process.cwd()) + ).toBe(grouped(process.cwd())) }) }) diff --git a/packages/core/tests/session-capturer-base.test.ts b/packages/core/tests/session-capturer-base.test.ts index d89455ae..92ee7722 100644 --- a/packages/core/tests/session-capturer-base.test.ts +++ b/packages/core/tests/session-capturer-base.test.ts @@ -152,3 +152,49 @@ describe('sendCommand', () => { expect(cap.upstream.filter((u) => u.scope === 'commands')).toHaveLength(1) }) }) + +describe('failLastAction', () => { + const boom = { name: 'Error', message: 'boom' } + + it('marks the most-recent action of the test when it has no error', () => { + cap.commandsLog.push({ + command: 'expect.toBeExisting', + args: [], + timestamp: 1, + testUid: 't1' + }) + expect(cap.failLastAction('t1', boom)).toBe(true) + expect(cap.commandsLog[0]!.error).toEqual(boom) + }) + + it('does not bleed onto an earlier passing action when the latest one already failed', () => { + // Regression: a failing expect matcher is captured as its own row via + // afterAssertion, so failLastAction must stop at it — NOT stamp the error + // onto the preceding passing assertion (the toBeExisting-shown-red bug). + const matcherErr = { name: 'Error', message: 'to have text' } + cap.commandsLog.push( + { command: 'expect.toBeExisting', args: [], timestamp: 1, testUid: 't1' }, + { + command: 'expect.toHaveText', + args: [], + timestamp: 2, + testUid: 't1', + error: matcherErr + } + ) + expect(cap.failLastAction('t1', boom)).toBe(false) + expect(cap.commandsLog[0]!.error).toBeUndefined() + expect(cap.commandsLog[1]!.error).toEqual(matcherErr) + }) + + it('ignores actions belonging to a different test', () => { + cap.commandsLog.push({ + command: 'expect.toBeExisting', + args: [], + timestamp: 1, + testUid: 'other' + }) + expect(cap.failLastAction('t1', boom)).toBe(false) + expect(cap.commandsLog[0]!.error).toBeUndefined() + }) +}) diff --git a/packages/core/tests/spec-trace-helpers.test.ts b/packages/core/tests/spec-trace-helpers.test.ts new file mode 100644 index 00000000..95257afe --- /dev/null +++ b/packages/core/tests/spec-trace-helpers.test.ts @@ -0,0 +1,337 @@ +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, it, expect } from 'vitest' +import { + buildSpecCapturer, + buildSpecSessionId, + buildTestSliceFolder, + buildTestSliceSessionId, + filterTestMetadataBySpec, + filterTestMetadataByUid, + recordSliceBoundary, + recordSpecBoundary, + sanitizeSpecName, + writeSpecTrace, + writeTestSliceTrace, + type SpecBoundaryContext, + type SpecRange, + type TraceCapturer, + type WriteSpecTraceInput +} from '@wdio/devtools-core' +import { TraceType, type TestMetadataMap } from '@wdio/devtools-shared' + +function capturer(): TraceCapturer { + const cmd = (i: number) => ({ + command: 'url', + args: [String(i)], + timestamp: i, + startTime: i + }) + return { + mutations: [{ m: 0 }, { m: 1 }, { m: 2 }] as never, + traceLogs: ['t0', 't1', 't2'], + consoleLogs: [{ c: 0 }, { c: 1 }, { c: 2 }] as never, + networkRequests: [{ n: 0 }, { n: 1 }, { n: 2 }] as never, + commandsLog: [cmd(0), cmd(1), cmd(2), cmd(3)], + sources: new Map([['/a.js', 'source']]), + metadata: { type: TraceType.Standalone }, + startWallTime: 0 + } +} + +const range = (over: Partial<SpecRange> = {}): SpecRange => ({ + specFile: '/a.js', + key: over.key ?? over.specFile ?? '/a.js', + commandStartIdx: 0, + consoleStartIdx: 0, + networkStartIdx: 0, + mutationStartIdx: 0, + traceLogStartIdx: 0, + snapshotCount: 0, + ...over +}) + +function boundaryCtx( + over: Partial<SpecBoundaryContext> = {} +): SpecBoundaryContext { + return { + specRanges: [], + flushedSpecs: new Set(), + capturer: { + commandsLog: [], + consoleLogs: [], + networkRequests: [], + mutations: [], + traceLogs: [] + }, + actionSnapshots: [], + ...over + } +} + +describe('buildSpecCapturer', () => { + it('slices from the range start to the end when no nextRange is given', () => { + const sliced = buildSpecCapturer(capturer(), range({ commandStartIdx: 2 })) + expect(sliced.commandsLog).toHaveLength(2) + expect(sliced.commandsLog.map((c) => c.args[0])).toEqual(['2', '3']) + }) + + it('slices to nextRange start indices when provided', () => { + const sliced = buildSpecCapturer( + capturer(), + range({ commandStartIdx: 1, consoleStartIdx: 1 }), + range({ commandStartIdx: 3, consoleStartIdx: 2 }) + ) + expect(sliced.commandsLog.map((c) => c.args[0])).toEqual(['1', '2']) + expect(sliced.consoleLogs).toHaveLength(1) + }) + + it('clones the source map so later parent mutations do not leak in', () => { + const parent = capturer() + const sliced = buildSpecCapturer(parent, range()) + parent.sources.set('/b.js', 'added-later') + expect(sliced.sources.has('/b.js')).toBe(false) + }) +}) + +describe('filterTestMetadataBySpec', () => { + it('keeps only entries whose specFile matches', () => { + const all: TestMetadataMap = new Map([ + ['u1', { title: 'A', specFile: '/a.js' }], + ['u2', { title: 'B', specFile: '/b.js' }], + ['u3', { title: 'C', specFile: '/a.js' }] + ]) + const filtered = filterTestMetadataBySpec(all, '/a.js') + expect([...filtered.keys()]).toEqual(['u1', 'u3']) + }) +}) + +describe('spec name / session id', () => { + it('sanitizes unsafe characters and falls back to unknown-spec', () => { + expect(sanitizeSpecName('/tests/login flow.ts')).toBe('login_flow') + expect(sanitizeSpecName('/specs/login.spec.ts')).toBe('login_spec') + expect(sanitizeSpecName('....')).toBe('unknown-spec') + }) + + it('derives a stable, collision-resistant spec session id', () => { + const a = buildSpecSessionId('/dir1/login.js', 'session-xyz') + const b = buildSpecSessionId('/dir2/login.js', 'session-xyz') + expect(a).not.toBe(b) + expect(a).toBe(buildSpecSessionId('/dir1/login.js', 'session-xyz')) + expect(a.startsWith('login-')).toBe(true) + }) +}) + +describe('filterTestMetadataByUid', () => { + it('keeps only the entry for the given uid', () => { + const all: TestMetadataMap = new Map([ + ['u1', { title: 'A', specFile: '/a.js' }], + ['u2', { title: 'B', specFile: '/b.js' }] + ]) + expect([...filterTestMetadataByUid(all, 'u1').keys()]).toEqual(['u1']) + expect(filterTestMetadataByUid(all, 'missing').size).toBe(0) + }) +}) + +describe('buildTestSliceSessionId', () => { + it('derives distinct, stable ids per key and stays readable', () => { + const a = buildTestSliceSessionId('/dir/login.js', 'u1', 'session-xyz') + const b = buildTestSliceSessionId('/dir/login.js', 'u2', 'session-xyz') + const retry = buildTestSliceSessionId( + '/dir/login.js', + 'u1-retry1', + 'session-xyz' + ) + expect(a).not.toBe(b) + expect(a).not.toBe(retry) + expect(a).toBe( + buildTestSliceSessionId('/dir/login.js', 'u1', 'session-xyz') + ) + expect(a.startsWith('login-')).toBe(true) + }) +}) + +describe('recordSliceBoundary (test granularity)', () => { + it('opens a new slice per test uid and returns the previous range', () => { + const ctx = boundaryCtx({ + capturer: { + commandsLog: [1, 2], + consoleLogs: [1], + networkRequests: [], + mutations: [1, 2, 3], + traceLogs: [] + }, + actionSnapshots: [1, 1] + }) + expect(recordSliceBoundary(ctx, 'test', '/a.js', 'u1')).toBeNull() + expect(ctx.specRanges).toHaveLength(1) + expect(ctx.specRanges[0]!.key).toBe('u1') + expect(ctx.specRanges[0]!.testUid).toBe('u1') + expect(ctx.specRanges[0]!.commandStartIdx).toBe(2) + expect(ctx.specRanges[0]!.snapshotCount).toBe(2) + + const prev = recordSliceBoundary(ctx, 'test', '/a.js', 'u2') + expect(prev?.key).toBe('u1') + expect(ctx.specRanges).toHaveLength(2) + expect(ctx.specRanges[1]!.key).toBe('u2') + }) + + it('keys each retry of the same uid as its own slice', () => { + const ctx = boundaryCtx() + recordSliceBoundary(ctx, 'test', '/a.js', 'u1') + recordSliceBoundary(ctx, 'test', '/a.js', 'u1') + recordSliceBoundary(ctx, 'test', '/a.js', 'u1') + expect(ctx.specRanges.map((r) => r.key)).toEqual([ + 'u1', + 'u1-retry1', + 'u1-retry2' + ]) + expect(ctx.specRanges.every((r) => r.testUid === 'u1')).toBe(true) + }) + + it('returns null when no testUid is provided', () => { + const ctx = boundaryCtx() + expect(recordSliceBoundary(ctx, 'test', '/a.js')).toBeNull() + expect(ctx.specRanges).toHaveLength(0) + }) + + it('does not return a previous range already in the flushed set', () => { + const ctx = boundaryCtx() + recordSliceBoundary(ctx, 'test', '/a.js', 'u1') + ctx.flushedSpecs.add('u1') + expect(recordSliceBoundary(ctx, 'test', '/a.js', 'u2')).toBeNull() + }) +}) + +describe('buildTestSliceFolder', () => { + it('combines sanitized spec, title slug, and browser slug', () => { + expect( + buildTestSliceFolder( + '/tests/login.e2e.js', + 'shows an error message for an invalid username', + 'chrome', + 'u1' + ) + ).toBe('login_e2e-shows-an-error-message-for-an-invalid-username-chrome') + }) + + it('appends a -retry<N> suffix when the key is a retry key', () => { + expect( + buildTestSliceFolder('/a.js', 'My Test', 'chrome', 'u1-retry2') + ).toBe('a-my-test-chrome-retry2') + }) + + it('defaults the browser slug to "browser" when the browser is absent', () => { + expect(buildTestSliceFolder('/a.js', 'My Test', undefined, 'u1')).toBe( + 'a-my-test-browser' + ) + }) + + it('falls back to a stable short hash of the key when the title is empty', () => { + const folder = buildTestSliceFolder('/a.js', '', 'chrome', 'u1') + expect(folder).toMatch(/^a-[a-z0-9]+-chrome$/) + expect(buildTestSliceFolder('/a.js', undefined, 'chrome', 'u1')).toBe( + folder + ) + }) + + it('lowercases, collapses non-alphanumerics, and caps the slug length', () => { + const folder = buildTestSliceFolder( + '/a.js', + `${'A'.repeat(80)} !!! End`, + 'Chrome', + 'u1' + ) + const titleSlug = folder.slice('a-'.length, folder.lastIndexOf('-chrome')) + expect(titleSlug.length).toBeLessThanOrEqual(60) + expect(titleSlug).toMatch(/^[a-z0-9-]+$/) + expect(titleSlug.endsWith('-')).toBe(false) + }) +}) + +describe('writeTestSliceTrace / writeSpecTrace output layout', () => { + let outputDir: string + beforeEach(async () => { + outputDir = await fs.mkdtemp(path.join(os.tmpdir(), 'slice-layout-')) + }) + afterEach(async () => { + await fs.rm(outputDir, { recursive: true, force: true }) + }) + + const writableCapturer = (): TraceCapturer => ({ + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + commandsLog: [ + { command: 'url', args: ['https://example.test'], timestamp: 1000 } + ], + sources: new Map(), + metadata: { type: TraceType.Standalone }, + startWallTime: 1000 + }) + + const input = ( + over: Partial<WriteSpecTraceInput> = {} + ): WriteSpecTraceInput => ({ + range: range({ specFile: '/tests/login.js', key: 'u1', testUid: 'u1' }), + capturer: writableCapturer(), + actionSnapshots: [], + sessionId: 'sess1234', + outputDir, + testMetadata: new Map([ + ['u1', { title: 'My Test', specFile: '/tests/login.js' }] + ]), + capabilities: { browserName: 'firefox' }, + ...over + }) + + it('writes a test slice into <folder>/trace.zip', async () => { + const written = await writeTestSliceTrace(input()) + const folder = buildTestSliceFolder( + '/tests/login.js', + 'My Test', + 'firefox', + 'u1' + ) + expect(folder).toBe('login-my-test-firefox') + expect(written).toBe(path.join(outputDir, folder, 'trace.zip')) + await expect(fs.access(written)).resolves.toBeUndefined() + }) + + it('keeps the spec write flat as trace-<id>.zip (unchanged layout)', async () => { + const written = await writeSpecTrace( + input({ + range: range({ specFile: '/tests/login.js', key: '/tests/login.js' }) + }) + ) + const name = buildSpecSessionId('/tests/login.js', 'sess1234') + expect(written).toBe(path.join(outputDir, `trace-${name}.zip`)) + await expect(fs.access(written)).resolves.toBeUndefined() + }) +}) + +describe('recordSliceBoundary / recordSpecBoundary (spec granularity)', () => { + it('keeps the spec-file behavior: one slice per spec, key equals specFile', () => { + const ctx = boundaryCtx() + expect(recordSliceBoundary(ctx, 'spec', '/a.js')).toBeNull() + expect(recordSliceBoundary(ctx, 'spec', '/a.js')).toBeNull() + expect(ctx.specRanges).toHaveLength(1) + expect(ctx.specRanges[0]!.key).toBe('/a.js') + expect(ctx.specRanges[0]!.testUid).toBeUndefined() + + const prev = recordSliceBoundary(ctx, 'spec', '/b.js') + expect(prev?.specFile).toBe('/a.js') + expect(ctx.specRanges).toHaveLength(2) + }) + + it('recordSpecBoundary returns null for session and test granularities', () => { + expect(recordSpecBoundary(boundaryCtx(), '/a.js', 'session')).toBeNull() + expect(recordSpecBoundary(boundaryCtx(), '/a.js', 'test')).toBeNull() + const ctx = boundaryCtx() + recordSpecBoundary(ctx, '/a.js', 'spec') + expect(ctx.specRanges).toHaveLength(1) + expect(ctx.specRanges[0]!.key).toBe('/a.js') + }) +}) diff --git a/packages/core/tests/suite-helpers.test.ts b/packages/core/tests/suite-helpers.test.ts index b1114b56..af49d5eb 100644 --- a/packages/core/tests/suite-helpers.test.ts +++ b/packages/core/tests/suite-helpers.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from 'vitest' import { + collectSuiteTestMetadata, computeSuiteFinalStatePermissive, computeSuiteFinalStateStrict, computeSuiteRunningState, @@ -197,3 +198,145 @@ describe('stampTestEnd', () => { expect(t._duration).toBe(300) }) }) + +describe('collectSuiteTestMetadata', () => { + function namedTest( + uid: string, + parent: string, + overrides: Partial<TestStats> = {} + ): TestStats { + const t = createTestStats({ + uid, + title: uid, + fullTitle: `full ${uid}`, + file: `/spec/${uid}.ts`, + parent + }) + return Object.assign(t, overrides) + } + + it('walks nested suites and collects every test entry', () => { + const root = createSuiteStats({ + uid: 'root', + title: 'Root', + file: '/spec/a.ts' + }) + const child = createSuiteStats({ + uid: 'child', + title: 'Child', + file: '/spec/a.ts' + }) + root.tests = [namedTest('t1', 'root')] + child.tests = [namedTest('t2', 'child')] + root.suites = [child] + + const map = collectSuiteTestMetadata([root]) + expect([...map.keys()]).toEqual(['t1', 't2']) + expect(map.get('t1')?.title).toBe('full t1') + expect(map.get('t2')?.specFile).toBe('/spec/t2.ts') + }) + + it('walks multiple root suites from any iterable', () => { + const a = createSuiteStats({ uid: 'a', title: 'A', file: '/a.ts' }) + const b = createSuiteStats({ uid: 'b', title: 'B', file: '/b.ts' }) + a.tests = [namedTest('t1', 'a')] + b.tests = [namedTest('t2', 'b')] + const map = collectSuiteTestMetadata(new Set([a, b])) + expect(map.size).toBe(2) + }) + + it('carries state and attempt (from retries), defaulting attempt to 0', () => { + const root = createSuiteStats({ uid: 'r', title: 'R', file: '/a.ts' }) + root.tests = [ + namedTest('flaky', 'r', { state: TEST_STATE.FAILED, retries: 2 }), + namedTest('fresh', 'r', { state: TEST_STATE.PASSED }) + ] + const map = collectSuiteTestMetadata([root]) + expect(map.get('flaky')).toMatchObject({ state: 'failed', attempt: 2 }) + expect(map.get('fresh')).toMatchObject({ state: 'passed', attempt: 0 }) + }) + + it('prefers fullTitle, falling back to title, and suite file when entry file is missing', () => { + const root = createSuiteStats({ uid: 'r', title: 'R', file: '/suite.ts' }) + const bare = namedTest('bare', 'r', { fullTitle: '' }) + // Simulate a partially-populated tree entry lacking `file`. + delete (bare as { file?: string }).file + root.tests = [bare] + const map = collectSuiteTestMetadata([root]) + expect(map.get('bare')).toMatchObject({ + title: 'bare', + specFile: '/suite.ts' + }) + }) + + it('records the ancestor chain outermost-first, excluding the test itself', () => { + const outer = createSuiteStats({ + uid: 'outer', + title: 'Outer', + file: '/a.ts' + }) + const mid = createSuiteStats({ uid: 'mid', title: 'Mid', file: '/a.ts' }) + const inner = createSuiteStats({ + uid: 'inner', + title: 'Inner', + file: '/a.ts' + }) + inner.tests = [namedTest('t', 'inner')] + mid.suites = [inner] + outer.suites = [mid] + + const ancestry = collectSuiteTestMetadata([outer]).get('t')?.ancestry + expect(ancestry).toEqual([ + { uid: 'outer', title: 'Outer', kind: 'suite' }, + { uid: 'mid', title: 'Mid', kind: 'suite' }, + { uid: 'inner', title: 'Inner', kind: 'suite' } + ]) + }) + + // Heuristic: a suite whose file ends with '.feature' (and isn't already + // under a feature/scenario) is the feature; its direct child suites are + // scenarios — matching the cucumber tree shape where scenario sub-suites + // carry the same .feature file as their parent feature suite. + it('derives feature/scenario kinds for a .feature-file tree', () => { + const root = createSuiteStats({ uid: 'root', title: 'Root', file: '/cwd' }) + const feature = createSuiteStats({ + uid: 'feat', + title: 'Login', + file: '/features/login.feature' + }) + const scenario = createSuiteStats({ + uid: 'scen', + title: 'Valid creds', + file: '/features/login.feature' + }) + scenario.tests = [namedTest('step1', 'scen')] + feature.suites = [scenario] + root.suites = [feature] + + const ancestry = collectSuiteTestMetadata([root]).get('step1')?.ancestry + expect(ancestry?.map((a) => a.kind)).toEqual([ + 'suite', + 'feature', + 'scenario' + ]) + }) + + it('treats a top-level .feature suite as the feature', () => { + const feature = createSuiteStats({ + uid: 'feat', + title: 'Login', + file: '/features/login.feature' + }) + feature.tests = [namedTest('t', 'feat')] + const ancestry = collectSuiteTestMetadata([feature]).get('t')?.ancestry + expect(ancestry?.map((a) => a.kind)).toEqual(['feature']) + }) + + it('skips string placeholder entries', () => { + const root = createSuiteStats({ uid: 'r', title: 'R', file: '/a.ts' }) + root.tests = ['placeholder-uid', namedTest('real', 'r')] + const map = collectSuiteTestMetadata([root]) + expect(map.size).toBe(1) + expect(map.has('real')).toBe(true) + }) +}) diff --git a/packages/core/tests/terminal-throttle.test.ts b/packages/core/tests/terminal-throttle.test.ts new file mode 100644 index 00000000..4e26f11b --- /dev/null +++ b/packages/core/tests/terminal-throttle.test.ts @@ -0,0 +1,74 @@ +import { describe, it, expect } from 'vitest' +import { + TERMINAL_REPEAT_WINDOW_MS, + TerminalLineThrottle, + terminalRepeatKey +} from '../src/terminal-throttle.js' + +describe('terminalRepeatKey', () => { + it('strips a leading ISO-8601 timestamp so log frames of the same message match', () => { + const a = '2026-07-09T22:12:38.497Z INFO webdriver: COMMAND getText("#x")' + const b = '2026-07-09T22:12:38.597Z INFO webdriver: COMMAND getText("#x")' + expect(terminalRepeatKey(a)).toBe(terminalRepeatKey(b)) + expect(terminalRepeatKey(a)).toBe('INFO webdriver: COMMAND getText("#x")') + }) + + it('leaves lines without a leading timestamp untouched', () => { + expect(terminalRepeatKey('[TEST] hello')).toBe('[TEST] hello') + }) +}) + +describe('TerminalLineThrottle', () => { + it('emits distinct lines immediately', () => { + const t = new TerminalLineThrottle() + expect(t.shouldEmit('one', 0)).toBe(true) + expect(t.shouldEmit('two', 0)).toBe(true) + expect(t.shouldEmit('three', 0)).toBe(true) + }) + + it('suppresses a within-window repeat but re-emits once the window passes', () => { + const t = new TerminalLineThrottle(1000) + expect(t.shouldEmit('poll', 0)).toBe(true) // first + expect(t.shouldEmit('poll', 100)).toBe(false) // repeat within window + expect(t.shouldEmit('poll', 900)).toBe(false) + expect(t.shouldEmit('poll', 1000)).toBe(true) // window elapsed → re-emit + expect(t.shouldEmit('poll', 1100)).toBe(false) + }) + + it('anchors the window to the last emit, not the last occurrence (a sustained stream still emits ~1/window)', () => { + const t = new TerminalLineThrottle(1000) + let emitted = 0 + // 100ms poll for 10s = 101 occurrences of the same line. + for (let ms = 0; ms <= 10000; ms += 100) { + if (t.shouldEmit('COMMAND getText', ms)) { + emitted++ + } + } + // ~1 per second rather than ~100 total — flood collapsed ~10:1. + expect(emitted).toBe(11) + }) + + it('collapses successive WDIO logger frames of the same command despite changing timestamps', () => { + const t = new TerminalLineThrottle(1000) + const line = (ms: number) => + `2026-07-09T22:12:${String(30 + Math.floor(ms / 1000)).padStart(2, '0')}.${String(ms % 1000).padStart(3, '0')}Z INFO webdriver: COMMAND getText("#x")` + expect(t.shouldEmit(line(0), 0)).toBe(true) + expect(t.shouldEmit(line(100), 100)).toBe(false) + expect(t.shouldEmit(line(200), 200)).toBe(false) + }) + + it('treats different messages independently', () => { + const t = new TerminalLineThrottle(1000) + expect(t.shouldEmit('COMMAND getText', 0)).toBe(true) + expect(t.shouldEmit('RESULT ""', 10)).toBe(true) // interleaved, distinct + expect(t.shouldEmit('COMMAND getText', 100)).toBe(false) // repeat + expect(t.shouldEmit('RESULT ""', 110)).toBe(false) // repeat + }) + + it('defaults to the exported window constant', () => { + const t = new TerminalLineThrottle() + expect(t.shouldEmit('x', 0)).toBe(true) + expect(t.shouldEmit('x', TERMINAL_REPEAT_WINDOW_MS - 1)).toBe(false) + expect(t.shouldEmit('x', TERMINAL_REPEAT_WINDOW_MS)).toBe(true) + }) +}) diff --git a/packages/core/tests/trace-assertions.test.ts b/packages/core/tests/trace-assertions.test.ts new file mode 100644 index 00000000..0d3d4ccd --- /dev/null +++ b/packages/core/tests/trace-assertions.test.ts @@ -0,0 +1,198 @@ +import { describe, it, expect } from 'vitest' +import { + ASSERT_ACTION_CLASS, + mapAssertCommand, + type CommandLog +} from '@wdio/devtools-shared' +import { formatActionTitle, mapCommandToAction } from '../src/action-mapping.js' +import { + buildActionEvents, + type AfterEvent, + type BeforeEvent +} from '../src/trace-action-events.js' + +describe('mapAssertCommand', () => { + it('maps assert./verify./expect. prefixed commands to the Assert class', () => { + expect(mapAssertCommand('assert.strictEqual')).toEqual({ + class: ASSERT_ACTION_CLASS, + method: 'strictEqual' + }) + expect(mapAssertCommand('verify.visible')).toEqual({ + class: ASSERT_ACTION_CLASS, + method: 'visible' + }) + expect(mapAssertCommand('expect.toBe')).toEqual({ + class: ASSERT_ACTION_CLASS, + method: 'toBe' + }) + }) + + it('returns null for anything else', () => { + expect(mapAssertCommand('click')).toBeNull() + expect(mapAssertCommand('url')).toBeNull() + expect(mapAssertCommand('assertx.foo')).toBeNull() + expect(mapAssertCommand('assert.')).toBeNull() + expect(mapAssertCommand('assert.deep.equal')).toBeNull() + }) +}) + +describe('mapCommandToAction assert fallthrough', () => { + it('keeps the ACTION_MAP lookup for runner commands', () => { + expect(mapCommandToAction('url')).toEqual({ + class: 'Page', + method: 'navigate' + }) + }) + + it('falls through to the assert mapping instead of filtering', () => { + expect(mapCommandToAction('assert.ok')).toEqual({ + class: ASSERT_ACTION_CLASS, + method: 'ok' + }) + expect(mapCommandToAction('notACommand')).toBeNull() + }) +}) + +describe('formatActionTitle for asserts', () => { + const action = { class: ASSERT_ACTION_CLASS, method: 'strictEqual' } + + it('renders the original command with quoted actual/expected', () => { + expect( + formatActionTitle( + action, + ['a', 'b', 'msg'], + undefined, + 'assert.strictEqual' + ) + ).toBe('assert.strictEqual("a", "b")') + }) + + it('prefers normalized actual/expected params over positional args', () => { + expect( + formatActionTitle( + action, + ['#el', 'foo'], + { actual: 'bar', expected: 'foo' }, + 'verify.textContains' + ) + ).toBe('verify.textContains("bar", "foo")') + }) + + it('falls back to assert.<method> when no command is supplied', () => { + expect(formatActionTitle(action, [1, 2])).toBe('assert.strictEqual(1, 2)') + }) + + it('truncates long values', () => { + const long = 'x'.repeat(100) + const title = formatActionTitle(action, [long], undefined, 'assert.ok') + expect(title.length).toBeLessThan(60) + expect(title.endsWith('…)')).toBe(true) + }) + + it('leaves non-assert titles unchanged', () => { + expect( + formatActionTitle({ class: 'Element', method: 'click' }, ['#go']) + ).toBe('Element.click("#go")') + }) +}) + +describe('buildActionEvents with assert commands', () => { + const WALL = 1_000_000 + + const befores = (events: (BeforeEvent | AfterEvent)[]) => + events.filter((e): e is BeforeEvent => e.type === 'before') + const afterOf = (events: (BeforeEvent | AfterEvent)[], callId: string) => + events.find( + (e): e is AfterEvent => e.type === 'after' && e.callId === callId + ) + + it('emits an action pair with assert params, apiName and title', () => { + const commands: CommandLog[] = [ + { + command: 'assert.strictEqual', + args: ['a', 'b', 'values differ'], + timestamp: WALL + 200, + startTime: WALL + 200, + error: { name: 'AssertionError', message: 'a !== b' } + }, + { + command: 'assert.ok', + args: [true], + result: 'passed', + timestamp: WALL + 300 + } + ] + const events = buildActionEvents(commands, 'page@1', WALL) + const [failed, passed] = befores(events) + + expect(failed.class).toBe(ASSERT_ACTION_CLASS) + expect(failed.method).toBe('strictEqual') + expect(failed.apiName).toBe('assert.strictEqual') + expect(failed.title).toBe('assert.strictEqual("a", "b")') + expect(failed.params).toEqual({ + '0': 'a', + '1': 'b', + '2': 'values differ', + actual: 'a', + expected: 'b', + message: 'values differ' + }) + expect(afterOf(events, failed.callId)?.error).toEqual({ + message: 'a !== b' + }) + + expect(passed.apiName).toBe('assert.ok') + expect(passed.params).toEqual({ '0': true, actual: true }) + expect(afterOf(events, passed.callId)?.error).toBeUndefined() + }) + + it('normalizes nightwatch collapsed assertion results', () => { + const commands: CommandLog[] = [ + { + command: 'verify.textContains', + args: ['#el', 'foo'], + result: { + passed: false, + actual: 'bar', + expected: 'foo', + message: 'nope' + }, + timestamp: WALL + 100 + } + ] + const events = buildActionEvents(commands, 'page@1', WALL) + const [before] = befores(events) + expect(before.params).toMatchObject({ + '0': '#el', + '1': 'foo', + actual: 'bar', + expected: 'foo', + message: 'nope' + }) + expect(before.title).toBe('verify.textContains("bar", "foo")') + // No Error instance on the entry — the failure comes from the collapsed result. + expect(afterOf(events, before.callId)?.error).toEqual({ message: 'nope' }) + }) + + it('keeps passing nightwatch asserts error-free and groups by testUid', () => { + const commands: CommandLog[] = [ + { + command: 'assert.visible', + args: ['#el'], + result: true, + timestamp: WALL + 100, + testUid: 't1' + } + ] + const metadata = new Map([ + ['t1', { title: 'logs in', specFile: '/specs/login.ts' }] + ]) + const events = buildActionEvents(commands, 'page@1', WALL, metadata) + const [group, action] = befores(events) + expect(group.method).toBe('tracingGroup') + expect(group.title).toBe('logs in') + expect(action.parentId).toBe(group.callId) + expect(action.params).toEqual({ '0': '#el', actual: '#el' }) + expect(afterOf(events, action.callId)?.error).toBeUndefined() + }) +}) diff --git a/packages/core/tests/trace-console.test.ts b/packages/core/tests/trace-console.test.ts new file mode 100644 index 00000000..5bb6ee59 --- /dev/null +++ b/packages/core/tests/trace-console.test.ts @@ -0,0 +1,111 @@ +import { describe, it, expect } from 'vitest' +import { buildConsoleEvents } from '@wdio/devtools-core' +import type { ConsoleLog } from '@wdio/devtools-shared' + +const WALL_TIME = 1000 +const PAGE_ID = 'page@abc123' + +function log(overrides: Partial<ConsoleLog> = {}): ConsoleLog { + return { + type: 'log', + args: ['hello'], + timestamp: WALL_TIME + 50, + source: 'browser', + ...overrides + } +} + +describe('buildConsoleEvents', () => { + it('returns empty array for no logs', () => { + expect(buildConsoleEvents([], PAGE_ID, WALL_TIME)).toEqual([]) + }) + + it('maps browser logs to console events with monotonic offsets', () => { + const events = buildConsoleEvents([log()], PAGE_ID, WALL_TIME) + expect(events).toEqual([ + { + type: 'console', + time: 50, + pageId: PAGE_ID, + messageType: 'log', + text: 'hello', + args: [{ preview: 'hello', value: 'hello' }], + location: { url: '', lineNumber: 0, columnNumber: 0 } + } + ]) + }) + + it('treats untagged logs as browser console', () => { + const events = buildConsoleEvents( + [log({ source: undefined })], + PAGE_ID, + WALL_TIME + ) + expect(events[0]!.type).toBe('console') + }) + + it("maps 'warn' to 'warning' and 'trace' to 'debug'", () => { + const events = buildConsoleEvents( + [log({ type: 'warn' }), log({ type: 'trace' })], + PAGE_ID, + WALL_TIME + ) + expect( + events.map((e) => (e.type === 'console' ? e.messageType : '')) + ).toEqual(['warning', 'debug']) + }) + + it('previews non-string args as JSON and joins into text', () => { + const events = buildConsoleEvents( + [log({ args: ['count', { a: 1 }, 2] })], + PAGE_ID, + WALL_TIME + ) + const event = events[0]! + expect(event.type).toBe('console') + if (event.type === 'console') { + expect(event.text).toBe('count {"a":1} 2') + expect(event.args?.[1]).toEqual({ preview: '{"a":1}', value: { a: 1 } }) + } + }) + + it('routes test/terminal logs to stdout/stderr by level with source kept', () => { + const events = buildConsoleEvents( + [ + log({ source: 'test', type: 'log', args: ['out'] }), + log({ source: 'test', type: 'error', args: ['bad'] }), + log({ source: 'terminal', type: 'warn', args: ['careful'] }) + ], + PAGE_ID, + WALL_TIME + ) + expect(events).toEqual([ + { type: 'stdout', timestamp: 50, text: 'out', source: 'test' }, + { type: 'stderr', timestamp: 50, text: 'bad', source: 'test' }, + { type: 'stderr', timestamp: 50, text: 'careful', source: 'terminal' } + ]) + }) + + it('floors offsets at zero for logs before wallTime', () => { + const events = buildConsoleEvents( + [log({ timestamp: WALL_TIME - 500 })], + PAGE_ID, + WALL_TIME + ) + const event = events[0]! + expect(event.type === 'console' ? event.time : -1).toBe(0) + }) + + it('caps output and appends a truncation marker', () => { + const logs = Array.from({ length: 10_001 }, (_, i) => + log({ timestamp: WALL_TIME + i }) + ) + const events = buildConsoleEvents(logs, PAGE_ID, WALL_TIME) + expect(events).toHaveLength(10_001) + const last = events[events.length - 1]! + expect(last.type).toBe('stderr') + if (last.type === 'stderr') { + expect(last.text).toContain('dropped 1 entries') + } + }) +}) diff --git a/packages/core/tests/trace-exporter.test.ts b/packages/core/tests/trace-exporter.test.ts index aef79694..3dd16c49 100644 --- a/packages/core/tests/trace-exporter.test.ts +++ b/packages/core/tests/trace-exporter.test.ts @@ -1,6 +1,14 @@ +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' import { describe, it, expect } from 'vitest' -import { buildActionEvents } from '@wdio/devtools-core' -import type { CommandLog } from '@wdio/devtools-shared' +import { + buildActionEvents, + writeTraceZip, + FrameSnapshotIndex, + type TraceCapturer +} from '@wdio/devtools-core' +import { TraceType, type CommandLog } from '@wdio/devtools-shared' function cmd(command: string, overrides: Partial<CommandLog> = {}): CommandLog { const base = (overrides.timestamp ?? 1000) + 100 @@ -17,12 +25,42 @@ describe('buildActionEvents', () => { const pageId = 'page@abc123' const wallTime = 1000 + it('stamps a stack frame from the captured callSource', () => { + const commands = [ + cmd('click', { callSource: '/specs/login.ts:42' }), + cmd('url', { timestamp: 1400, startTime: 1350 }) + ] + const befores = buildActionEvents(commands, pageId, wallTime).filter( + (e) => e.type === 'before' + ) + expect(befores[0]!.stack).toEqual([ + { file: '/specs/login.ts', line: 42, column: 0 } + ]) + expect(befores[1]!.stack).toBeUndefined() + }) + + it('stamps the command result on the after event', () => { + const commands = [cmd('execute', { result: 'flash text' })] + const after = buildActionEvents(commands, pageId, wallTime).find( + (e) => e.type === 'after' + )! + expect(after.result).toBe('flash text') + }) + + it('drops oversized command results from the after event', () => { + const commands = [cmd('execute', { result: 'x'.repeat(70 * 1024) })] + const after = buildActionEvents(commands, pageId, wallTime).find( + (e) => e.type === 'after' + )! + expect(after.result).toBeUndefined() + }) + it('returns empty array for no commands', () => { expect(buildActionEvents([], pageId, wallTime)).toEqual([]) }) it('returns empty array when no commands map to actions', () => { - const commands = [cmd('getTitle'), cmd('executeScript')] + const commands = [cmd('clearValue'), cmd('executeScript')] expect(buildActionEvents(commands, pageId, wallTime)).toEqual([]) }) @@ -193,9 +231,37 @@ describe('buildActionEvents — tracingGroup (testUid boundaries)', () => { expect(groupBefore.params).toEqual({ name: 'unknown-uid' }) }) + it('does not stamp snapshot refs on tracingGroup events', () => { + const commands = [ + cmd('url', { testUid: 'uid-1', timestamp: 1200, startTime: 1150 }), + cmd('click', { testUid: 'uid-2', timestamp: 1400, startTime: 1350 }) + ] + const index = new FrameSnapshotIndex([ + { timestamp: 1200, command: 'url', screenshot: 'AAAA' }, + { timestamp: 1400, command: 'click', screenshot: 'BBBB' } + ]) + const events = buildActionEvents( + commands, + pageId, + wallTime, + testMeta, + index + ) + const groups = events.filter( + (e) => e.type === 'before' && e.method === 'tracingGroup' + ) + for (const group of groups) { + expect(group).not.toHaveProperty('beforeSnapshot') + const groupAfter = events.find( + (e) => e.type === 'after' && e.callId === group.callId + )! + expect(groupAfter.afterSnapshot).toBeUndefined() + } + }) + it('skips non-action commands but still handles group boundaries', () => { const commands = [ - cmd('getTitle', { testUid: 'uid-1' }), // non-action — skipped + cmd('clearValue', { testUid: 'uid-1' }), // non-action — skipped cmd('url', { testUid: 'uid-2', timestamp: 1200, startTime: 1150 }) ] const events = buildActionEvents(commands, pageId, wallTime, testMeta) @@ -211,3 +277,130 @@ describe('buildActionEvents — tracingGroup (testUid boundaries)', () => { expect(groups[0]!.params).toEqual({ name: 'test B' }) }) }) + +describe('buildActionEvents — frame-snapshot refs', () => { + const pageId = 'page@abc123' + const wallTime = 1000 + + it('stamps afterSnapshot when a screenshot matches the command timestamp', () => { + const commands = [cmd('url', { timestamp: 1200, startTime: 1150 })] + const index = new FrameSnapshotIndex([ + { timestamp: 1200, command: 'url', screenshot: 'AAAA' } + ]) + const events = buildActionEvents( + commands, + pageId, + wallTime, + undefined, + index + ) + const after = events.find((e) => e.type === 'after')! + expect(after.afterSnapshot).toBe('after@call@1') + expect(index.refs()).toHaveLength(1) + }) + + it('uses the previous action snapshot as the next action before ref', () => { + const commands = [ + cmd('url', { timestamp: 1200, startTime: 1150 }), + cmd('click', { timestamp: 1400, startTime: 1350 }) + ] + const index = new FrameSnapshotIndex([ + { timestamp: 1200, command: 'url', screenshot: 'AAAA' }, + { timestamp: 1400, command: 'click', screenshot: 'BBBB' } + ]) + const events = buildActionEvents( + commands, + pageId, + wallTime, + undefined, + index + ) + const befores = events.filter((e) => e.type === 'before') + expect(befores[0]!.beforeSnapshot).toBeUndefined() + expect(befores[1]!.beforeSnapshot).toBe('after@call@1') + const afters = events.filter((e) => e.type === 'after') + expect(afters[1]!.afterSnapshot).toBe('after@call@2') + }) + + it('leaves refs unset for commands without matching screenshots', () => { + const commands = [cmd('url', { timestamp: 1200, startTime: 1150 })] + const index = new FrameSnapshotIndex([]) + const events = buildActionEvents( + commands, + pageId, + wallTime, + undefined, + index + ) + expect( + events.find((e) => e.type === 'before')!.beforeSnapshot + ).toBeUndefined() + expect( + events.find((e) => e.type === 'after')!.afterSnapshot + ).toBeUndefined() + }) +}) + +describe('exported trace stream — frame-snapshot events', () => { + it('emits an image frame-snapshot sorted after its action', async () => { + const outputDir = await fs.mkdtemp( + path.join(os.tmpdir(), 'trace-frame-snapshots-') + ) + const capturer: TraceCapturer = { + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + commandsLog: [ + { + command: 'url', + args: ['https://example.test'], + timestamp: 1200, + startTime: 1150, + screenshot: 'AAAA' + } + ], + sources: new Map(), + metadata: { + type: TraceType.Standalone, + viewport: { + width: 800, + height: 600, + offsetLeft: 0, + offsetTop: 0, + scale: 1 + } + }, + startWallTime: 1000 + } + const dir = await writeTraceZip(capturer, { + outputDir, + sessionId: 'abc12345', + format: 'ndjson-directory' + }) + const raw = await fs.readFile(path.join(dir, 'trace.trace'), 'utf8') + const lines = raw + .trim() + .split('\n') + .map((l) => JSON.parse(l) as Record<string, unknown>) + + const afterIdx = lines.findIndex((l) => l.type === 'after') + const snapIdx = lines.findIndex((l) => l.type === 'frame-snapshot') + expect(afterIdx).toBeGreaterThan(-1) + expect(snapIdx).toBeGreaterThan(afterIdx) + + const after = lines[afterIdx] as { afterSnapshot?: string } + expect(after.afterSnapshot).toBe('after@call@1') + + const snapshot = (lines[snapIdx] as { snapshot: Record<string, unknown> }) + .snapshot + expect(snapshot.snapshotName).toBe('after@call@1') + expect(snapshot.callId).toBe('call@1') + expect(snapshot.pageId).toBe('page@abc12345') + expect(snapshot.frameId).toBe('frame@abc12345') + expect(snapshot.isMainFrame).toBe(true) + expect(snapshot.timestamp).toBe(200) + expect(snapshot.viewport).toEqual({ width: 800, height: 600 }) + await fs.rm(outputDir, { recursive: true, force: true }) + }) +}) diff --git a/packages/core/tests/trace-finalizer.test.ts b/packages/core/tests/trace-finalizer.test.ts new file mode 100644 index 00000000..6d2b5013 --- /dev/null +++ b/packages/core/tests/trace-finalizer.test.ts @@ -0,0 +1,493 @@ +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + buildSpecSessionId, + buildTestSliceFolder, + finalizeTraceExport, + flushRangeTrace, + type SpecRange, + type TraceArtifact, + type TraceCapturer, + type TraceExportContext +} from '@wdio/devtools-core' +import { + TraceType, + type TestMetadataEntry, + type TestMetadataMap +} from '@wdio/devtools-shared' + +function makeCapturer(commandCount = 4): TraceCapturer { + return { + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + commandsLog: Array.from({ length: commandCount }, (_, i) => ({ + command: 'url', + args: [`https://example.test/${i}`], + timestamp: 1000 + i * 100, + startTime: 1000 + i * 100 - 50 + })), + sources: new Map(), + metadata: { + type: TraceType.Standalone, + viewport: { + width: 800, + height: 600, + offsetLeft: 0, + offsetTop: 0, + scale: 1 + } + }, + startWallTime: 1000 + } +} + +function meta(entries: Array<[string, TestMetadataEntry]>): TestMetadataMap { + return new Map(entries) +} + +function range(specFile: string, startIdx: number): SpecRange { + return { + specFile, + key: specFile, + commandStartIdx: startIdx, + consoleStartIdx: 0, + networkStartIdx: 0, + mutationStartIdx: 0, + traceLogStartIdx: 0, + snapshotCount: 0 + } +} + +function testRange( + specFile: string, + testUid: string, + startIdx: number, + key = testUid +): SpecRange { + return { ...range(specFile, startIdx), key, testUid } +} + +describe('finalizeTraceExport', () => { + let outputDir: string + let artifacts: TraceArtifact[] + let logs: Array<[string, string]> + + beforeEach(async () => { + outputDir = await fs.mkdtemp(path.join(os.tmpdir(), 'trace-finalizer-')) + artifacts = [] + logs = [] + }) + afterEach(async () => { + await fs.rm(outputDir, { recursive: true, force: true }) + }) + + function baseCtx( + overrides: Partial<TraceExportContext> = {} + ): TraceExportContext { + return { + mode: 'trace', + granularity: 'session', + format: 'zip', + capturer: makeCapturer(), + actionSnapshots: [], + sessionId: 'abcd1234', + testMetadata: new Map(), + ranges: [], + flushed: new Set(), + resolveOutputDir: () => outputDir, + log: (level, msg) => logs.push([level, msg]), + onArtifact: (a) => artifacts.push(a), + ...overrides + } + } + + const exists = (p: string) => + fs.access(p).then( + () => true, + () => false + ) + + it('is a no-op outside trace mode', async () => { + const result = await finalizeTraceExport(baseCtx({ mode: 'live' })) + expect(result).toEqual([]) + expect(await fs.readdir(outputDir)).toEqual([]) + }) + + it('writes one session-level trace for session granularity', async () => { + const result = await finalizeTraceExport( + baseCtx({ + testMetadata: meta([['u1', { title: 'T1', specFile: '/a.js' }]]) + }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.scope).toBe('session') + expect(result[0]!.retained).toBe(true) + expect(result[0]!.testUids).toEqual(['u1']) + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe(true) + expect(artifacts).toHaveLength(1) + }) + + it('fans out one trace per recorded spec range', async () => { + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'spec', + ranges: [range('/a.js', 0), range('/b.js', 2)], + testMetadata: meta([ + ['a1', { title: 'A1', specFile: '/a.js' }], + ['b1', { title: 'B1', specFile: '/b.js' }] + ]) + }) + ) + expect(result).toHaveLength(2) + expect(result.map((a) => a.scope)).toEqual(['spec', 'spec']) + expect(result.map((a) => a.key)).toEqual(['/a.js', '/b.js']) + // testUids filtered to each spec's own tests. + expect(result[0]!.testUids).toEqual(['a1']) + expect(result[1]!.testUids).toEqual(['b1']) + const nameA = buildSpecSessionId('/a.js', 'abcd1234') + const nameB = buildSpecSessionId('/b.js', 'abcd1234') + expect(await exists(path.join(outputDir, `trace-${nameA}.zip`))).toBe(true) + expect(await exists(path.join(outputDir, `trace-${nameB}.zip`))).toBe(true) + }) + + it('warns and falls back to a session trace when spec has no boundaries', async () => { + const result = await finalizeTraceExport( + baseCtx({ granularity: 'spec', ranges: [] }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.scope).toBe('session') + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe(true) + expect( + logs.some( + ([level, msg]) => + level === 'warn' && msg.includes('no spec boundaries were detected') + ) + ).toBe(true) + }) + + it('fans out one trace per recorded test range', async () => { + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'test', + ranges: [testRange('/a.js', 'a1', 0), testRange('/a.js', 'a2', 2)], + testMetadata: meta([ + ['a1', { title: 'A1', specFile: '/a.js' }], + ['a2', { title: 'A2', specFile: '/a.js' }] + ]) + }) + ) + expect(result).toHaveLength(2) + expect(result.map((a) => a.scope)).toEqual(['test', 'test']) + expect(result.map((a) => a.key)).toEqual(['a1', 'a2']) + // Each test slice carries only its own test's metadata. + expect(result[0]!.testUids).toEqual(['a1']) + expect(result[1]!.testUids).toEqual(['a2']) + // Each slice lands in its own <spec>--<title>-<browser>/trace.zip folder. + const folderA1 = buildTestSliceFolder('/a.js', 'A1', undefined, 'a1') + const folderA2 = buildTestSliceFolder('/a.js', 'A2', undefined, 'a2') + expect(folderA1).not.toBe(folderA2) + expect(result[0]!.path).toBe(path.join(outputDir, folderA1, 'trace.zip')) + expect(await exists(path.join(outputDir, folderA1, 'trace.zip'))).toBe(true) + expect(await exists(path.join(outputDir, folderA2, 'trace.zip'))).toBe(true) + }) + + it('treats a retry-keyed range as its own test slice', async () => { + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'test', + ranges: [ + testRange('/a.js', 'a1', 0), + testRange('/a.js', 'a1', 2, 'a1-retry1') + ], + testMetadata: meta([['a1', { title: 'A1', specFile: '/a.js' }]]) + }) + ) + expect(result).toHaveLength(2) + expect(result.map((a) => a.key)).toEqual(['a1', 'a1-retry1']) + // The retry attempt gets a distinct folder via the -retry<N> suffix. + const first = buildTestSliceFolder('/a.js', 'A1', undefined, 'a1') + const retry = buildTestSliceFolder('/a.js', 'A1', undefined, 'a1-retry1') + expect(first).not.toBe(retry) + expect(retry.endsWith('-retry1')).toBe(true) + expect(await exists(path.join(outputDir, first, 'trace.zip'))).toBe(true) + expect(await exists(path.join(outputDir, retry, 'trace.zip'))).toBe(true) + }) + + it('warns and falls back to a session trace when test has no boundaries', async () => { + const result = await finalizeTraceExport( + baseCtx({ granularity: 'test', ranges: [] }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.scope).toBe('session') + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe(true) + expect( + logs.some( + ([level, msg]) => + level === 'warn' && msg.includes('no test boundaries were detected') + ) + ).toBe(true) + }) + + it('warns to pair with a retention policy above the slice-count threshold', async () => { + const ranges = Array.from({ length: 201 }, (_, i) => + testRange('/a.js', `u${i}`, i) + ) + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'test', + // retain-on-failure + all-passing declines every write, so the guard + // is exercised without emitting 201 archives. + policy: 'retain-on-failure', + ranges, + testMetadata: meta( + ranges.map((r) => [ + r.testUid!, + { + title: r.testUid!, + specFile: '/a.js', + state: 'passed', + attempt: 0 + } + ]) + ) + }) + ) + expect(result).toHaveLength(201) + expect(result.every((a) => !a.retained)).toBe(true) + expect(await fs.readdir(outputDir)).toEqual([]) + expect( + logs.some( + ([level, msg]) => level === 'warn' && msg.includes('retention policy') + ) + ).toBe(true) + }) + + it('does not rewrite a range already in the flushed set', async () => { + const flushed = new Set<string>(['/a.js']) + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'spec', + ranges: [range('/a.js', 0), range('/b.js', 2)], + flushed, + testMetadata: meta([['b1', { title: 'B1', specFile: '/b.js' }]]) + }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.key).toBe('/b.js') + const nameA = buildSpecSessionId('/a.js', 'abcd1234') + expect(await exists(path.join(outputDir, `trace-${nameA}.zip`))).toBe(false) + }) + + it('catches a failing write and still writes the remaining ranges', async () => { + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'spec', + ranges: [range('/a.js', 0), range('/b.js', 2)], + resolveOutputDir: (r) => { + if (r?.specFile === '/a.js') { + throw new Error('boom') + } + return outputDir + } + }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.key).toBe('/b.js') + expect( + logs.some(([lvl, msg]) => lvl === 'warn' && msg.includes('boom')) + ).toBe(true) + const nameB = buildSpecSessionId('/b.js', 'abcd1234') + expect(await exists(path.join(outputDir, `trace-${nameB}.zip`))).toBe(true) + }) + + it('settles pending captures before writing', async () => { + let settled = false + const pending = Promise.reject(new Error('ignored')).catch(() => { + settled = true + }) + await finalizeTraceExport(baseCtx({ awaitPending: [pending] })) + expect(settled).toBe(true) + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe(true) + }) + + describe('retention wiring', () => { + it("writes everything under the default 'on' policy", async () => { + const result = await finalizeTraceExport( + baseCtx({ + policy: 'on', + testMetadata: meta([ + [ + 'u1', + { title: 'T', specFile: '/a.js', state: 'passed', attempt: 0 } + ] + ]) + }) + ) + expect(result[0]!.retained).toBe(true) + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe( + true + ) + }) + + // Forward-looking: the retention decision is consulted now (B3 flips the + // option). A non-default policy with all-passing outcomes must NOT write, + // proving shouldRetainTrace gates the write rather than being dead-wired. + it('reports retained:false and writes nothing when the policy declines', async () => { + const result = await finalizeTraceExport( + baseCtx({ + policy: 'retain-on-failure', + testMetadata: meta([ + [ + 'u1', + { title: 'T', specFile: '/a.js', state: 'passed', attempt: 0 } + ] + ]) + }) + ) + expect(result).toHaveLength(1) + expect(result[0]!.retained).toBe(false) + expect(result[0]!.path).toBe('') + expect(await fs.readdir(outputDir)).toEqual([]) + expect(artifacts).toHaveLength(1) + expect(artifacts[0]!.retained).toBe(false) + }) + + it('writes when a retain-on-failure policy sees a failure', async () => { + const result = await finalizeTraceExport( + baseCtx({ + policy: 'retain-on-failure', + testMetadata: meta([ + [ + 'u1', + { title: 'T', specFile: '/a.js', state: 'failed', attempt: 0 } + ] + ]) + }) + ) + expect(result[0]!.retained).toBe(true) + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe( + true + ) + }) + + // Session slice = OR over every test: one failure keeps the whole trace. + it('session slice retains when ANY test failed under retain-on-failure', async () => { + const result = await finalizeTraceExport( + baseCtx({ + policy: 'retain-on-failure', + testMetadata: meta([ + [ + 'u1', + { title: 'T1', specFile: '/a.js', state: 'passed', attempt: 0 } + ], + [ + 'u2', + { title: 'T2', specFile: '/a.js', state: 'failed', attempt: 0 } + ] + ]) + }) + ) + expect(result[0]!.retained).toBe(true) + expect(await exists(path.join(outputDir, 'trace-abcd1234.zip'))).toBe( + true + ) + }) + + // Spec slice = per-spec decision: the failing spec writes, the clean one + // does not — proving the retention gate runs against each spec's own tests. + it('spec slices retain per-spec failure under retain-on-failure', async () => { + const result = await finalizeTraceExport( + baseCtx({ + granularity: 'spec', + policy: 'retain-on-failure', + ranges: [range('/a.js', 0), range('/b.js', 2)], + testMetadata: meta([ + [ + 'a1', + { title: 'A1', specFile: '/a.js', state: 'failed', attempt: 0 } + ], + [ + 'b1', + { title: 'B1', specFile: '/b.js', state: 'passed', attempt: 0 } + ] + ]) + }) + ) + expect(result).toHaveLength(2) + const a = result.find((r) => r.key === '/a.js')! + const b = result.find((r) => r.key === '/b.js')! + expect(a.retained).toBe(true) + expect(b.retained).toBe(false) + const nameA = buildSpecSessionId('/a.js', 'abcd1234') + const nameB = buildSpecSessionId('/b.js', 'abcd1234') + expect(await exists(path.join(outputDir, `trace-${nameA}.zip`))).toBe( + true + ) + expect(await exists(path.join(outputDir, `trace-${nameB}.zip`))).toBe( + false + ) + }) + }) + + it('applies prepareSnapshots only to the session write', async () => { + const prepare = vi.fn((s: unknown[]) => s) + await finalizeTraceExport(baseCtx({ prepareSnapshots: prepare as never })) + expect(prepare).toHaveBeenCalledOnce() + }) +}) + +describe('flushRangeTrace', () => { + let outputDir: string + beforeEach(async () => { + outputDir = await fs.mkdtemp(path.join(os.tmpdir(), 'flush-range-')) + }) + afterEach(async () => { + await fs.rm(outputDir, { recursive: true, force: true }) + }) + + function ctx( + overrides: Partial<TraceExportContext> = {} + ): TraceExportContext { + return { + mode: 'trace', + granularity: 'spec', + format: 'zip', + capturer: makeCapturer(), + actionSnapshots: [], + sessionId: 'sess0001', + testMetadata: new Map(), + ranges: [], + flushed: new Set(), + resolveOutputDir: () => outputDir, + ...overrides + } + } + + it('adds the spec to the flushed set and returns the written artifact', async () => { + const flushed = new Set<string>() + const artifact = await flushRangeTrace( + ctx({ flushed }), + range('/spec.js', 0) + ) + expect(artifact?.scope).toBe('spec') + expect(artifact?.key).toBe('/spec.js') + expect(flushed.has('/spec.js')).toBe(true) + const name = buildSpecSessionId('/spec.js', 'sess0001') + await expect( + fs.access(path.join(outputDir, `trace-${name}.zip`)) + ).resolves.toBeUndefined() + }) + + it('returns undefined for an already-flushed spec', async () => { + const flushed = new Set<string>(['/spec.js']) + const artifact = await flushRangeTrace( + ctx({ flushed }), + range('/spec.js', 0) + ) + expect(artifact).toBeUndefined() + }) +}) diff --git a/packages/core/tests/trace-frame-snapshots.test.ts b/packages/core/tests/trace-frame-snapshots.test.ts new file mode 100644 index 00000000..81bdfebd --- /dev/null +++ b/packages/core/tests/trace-frame-snapshots.test.ts @@ -0,0 +1,180 @@ +import { describe, it, expect } from 'vitest' +import { + buildImageFrameSnapshots, + FrameSnapshotIndex, + type FrameSnapshotRef +} from '@wdio/devtools-core' +import type { ActionSnapshot } from '@wdio/devtools-shared' + +function snap(overrides: Partial<ActionSnapshot> = {}): ActionSnapshot { + return { timestamp: 2000, command: 'click', screenshot: 'AAAA', ...overrides } +} + +describe('FrameSnapshotIndex', () => { + it('claims a snapshot by exact command timestamp and names it after@<callId>', () => { + const index = new FrameSnapshotIndex([snap()]) + expect(index.claimAfter(2000, 'call@2')).toBe('after@call@2') + expect(index.refs()).toEqual([ + { callId: 'call@2', snapshotName: 'after@call@2', snapshot: snap() } + ]) + }) + + it('returns undefined for unmatched timestamps', () => { + const index = new FrameSnapshotIndex([snap()]) + expect(index.claimAfter(1234, 'call@2')).toBeUndefined() + expect(index.refs()).toEqual([]) + }) + + it('consumes the snapshot on claim', () => { + const index = new FrameSnapshotIndex([snap()]) + index.claimAfter(2000, 'call@2') + expect(index.claimAfter(2000, 'call@3')).toBeUndefined() + }) + + it('ignores snapshots without a screenshot', () => { + const index = new FrameSnapshotIndex([snap({ screenshot: undefined })]) + expect(index.claimAfter(2000, 'call@2')).toBeUndefined() + }) + + it('keeps the richest capture for duplicate timestamps', () => { + const index = new FrameSnapshotIndex([ + snap({ screenshot: 'AA' }), + snap({ screenshot: 'AAAAAAAA' }), + snap({ screenshot: 'AAAA' }) + ]) + index.claimAfter(2000, 'call@2') + expect(index.refs()[0]!.snapshot.screenshot).toBe('AAAAAAAA') + }) + + it('tracks the last claimed name as the next action before state', () => { + const index = new FrameSnapshotIndex([ + snap(), + snap({ timestamp: 3000, command: 'setValue' }) + ]) + expect(index.beforeName()).toBeUndefined() + index.claimAfter(2000, 'call@2') + expect(index.beforeName()).toBe('after@call@2') + index.claimAfter(3000, 'call@3') + expect(index.beforeName()).toBe('after@call@3') + }) +}) + +describe('buildImageFrameSnapshots', () => { + const pageId = 'page@abc123' + const wallTime = 1000 + const viewport = { width: 800, height: 600 } + + function ref(overrides: Partial<ActionSnapshot> = {}): FrameSnapshotRef { + return { + callId: 'call@2', + snapshotName: 'after@call@2', + snapshot: snap(overrides) + } + } + + it('emits events with exactly the reference field set', () => { + const [event] = buildImageFrameSnapshots( + [ref()], + pageId, + wallTime, + viewport + ) + expect(Object.keys(event!)).toEqual(['type', 'snapshot']) + expect(event!.type).toBe('frame-snapshot') + expect(Object.keys(event!.snapshot).sort()).toEqual( + [ + 'callId', + 'collectionTime', + 'doctype', + 'frameId', + 'frameUrl', + 'html', + 'isMainFrame', + 'pageId', + 'resourceOverrides', + 'snapshotName', + 'timestamp', + 'viewport', + 'wallTime' + ].sort() + ) + }) + + it('encodes the screenshot as an image document in node-array format', () => { + const [event] = buildImageFrameSnapshots( + [ref({ url: 'https://example.test/login' })], + pageId, + wallTime, + viewport + ) + expect(event!.snapshot.html).toEqual([ + 'HTML', + {}, + ['HEAD', {}, ['BASE', { href: 'https://example.test/login' }]], + [ + 'BODY', + { style: 'margin:0' }, + [ + 'IMG', + { + src: 'data:image/jpeg;base64,AAAA', + style: 'display:block;width:100vw;height:100vh;object-fit:contain' + } + ] + ] + ]) + }) + + it('stamps identity, timing, and frame fields from the ref', () => { + const [event] = buildImageFrameSnapshots( + [ref()], + pageId, + wallTime, + viewport + ) + const s = event!.snapshot + expect(s.callId).toBe('call@2') + expect(s.snapshotName).toBe('after@call@2') + expect(s.pageId).toBe(pageId) + expect(s.frameId).toBe('frame@abc123') + expect(s.doctype).toBe('html') + expect(s.viewport).toEqual(viewport) + expect(s.timestamp).toBe(1000) + expect(s.wallTime).toBe(2000) + expect(s.collectionTime).toBe(0) + expect(s.resourceOverrides).toEqual([]) + expect(s.isMainFrame).toBe(true) + }) + + it('sniffs a png data uri from the base64 magic', () => { + const [event] = buildImageFrameSnapshots( + [ref({ screenshot: 'iVBORw0KGgo' })], + pageId, + wallTime, + viewport + ) + const html = JSON.stringify(event!.snapshot.html) + expect(html).toContain('data:image/png;base64,iVBORw0KGgo') + }) + + it('falls back to about:blank when the snapshot has no url', () => { + const [event] = buildImageFrameSnapshots( + [ref()], + pageId, + wallTime, + viewport + ) + expect(event!.snapshot.frameUrl).toBe('about:blank') + }) + + it('clamps timestamps captured before wallTime to zero', () => { + const [event] = buildImageFrameSnapshots( + [ref({ timestamp: 500 })], + pageId, + wallTime, + viewport + ) + expect(event!.snapshot.timestamp).toBe(0) + expect(event!.snapshot.wallTime).toBe(500) + }) +}) diff --git a/packages/core/tests/trace-network-bodies.test.ts b/packages/core/tests/trace-network-bodies.test.ts new file mode 100644 index 00000000..60bc955d --- /dev/null +++ b/packages/core/tests/trace-network-bodies.test.ts @@ -0,0 +1,197 @@ +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' +import { createHash } from 'node:crypto' +import { describe, it, expect } from 'vitest' +import type { NetworkRequest } from '@wdio/devtools-shared' +import { networkRequestToHar } from '../src/trace-har.js' +import { + buildNetworkBodyResources, + writeTraceZip, + type TraceCapturer +} from '../src/trace-exporter.js' + +const sha1 = (data: string): string => + createHash('sha1').update(data).digest('hex') + +function req( + id: string, + overrides: Partial<NetworkRequest> = {} +): NetworkRequest { + return { + id, + url: `https://api.example.com/items/${id}`, + method: 'GET', + status: 200, + statusText: 'OK', + timestamp: 1_000, + startTime: 1_000, + endTime: 1_050, + time: 50, + type: 'fetch', + size: 7, + response: { + fromCache: false, + headers: {}, + mimeType: 'application/json', + status: 200 + }, + ...overrides + } +} + +describe('buildNetworkBodyResources', () => { + it('writes a bare content-addressed resource per body', () => { + const body = '{"a":1}' + const { resources, sha1ByRequestId } = buildNetworkBodyResources([ + req('r1', { responseBody: body }) + ]) + expect(resources).toHaveLength(1) + expect(resources[0]!.resourceName).toBe(sha1(body)) + expect(resources[0]!.data.toString('utf8')).toBe(body) + expect(sha1ByRequestId.get('r1')).toBe(sha1(body)) + }) + + it('skips requests without a responseBody', () => { + const { resources, sha1ByRequestId } = buildNetworkBodyResources([ + req('r1'), + req('r2', { responseBody: '{"b":2}' }) + ]) + expect(resources).toHaveLength(1) + expect(sha1ByRequestId.has('r1')).toBe(false) + expect(sha1ByRequestId.get('r2')).toBe(sha1('{"b":2}')) + }) + + it('dedupes identical bodies into a single resource', () => { + const body = '{"shared":true}' + const { resources, sha1ByRequestId } = buildNetworkBodyResources([ + req('r1', { responseBody: body }), + req('r2', { responseBody: body }) + ]) + expect(resources).toHaveLength(1) + expect(sha1ByRequestId.get('r1')).toBe(sha1(body)) + expect(sha1ByRequestId.get('r2')).toBe(sha1(body)) + }) + + it('skips bodies above the per-body cap', () => { + const { resources, sha1ByRequestId } = buildNetworkBodyResources( + [req('r1', { responseBody: 'x'.repeat(11) })], + { maxBodyBytes: 10, maxTotalBytes: 100 } + ) + expect(resources).toHaveLength(0) + expect(sha1ByRequestId.size).toBe(0) + }) + + it('stops storing new bodies past the total cap but keeps dedupe refs', () => { + const first = 'aaaaaaaa' + const second = 'bbbbbbbb' + const { resources, sha1ByRequestId } = buildNetworkBodyResources( + [ + req('r1', { responseBody: first }), + req('r2', { responseBody: second }), + req('r3', { responseBody: first }) + ], + { maxBodyBytes: 10, maxTotalBytes: 12 } + ) + expect(resources.map((r) => r.resourceName)).toEqual([sha1(first)]) + expect(sha1ByRequestId.get('r1')).toBe(sha1(first)) + expect(sha1ByRequestId.has('r2')).toBe(false) + expect(sha1ByRequestId.get('r3')).toBe(sha1(first)) + }) + + it('measures caps in utf8 bytes, not string length', () => { + // Three-byte characters: 4 chars = 12 bytes, over a 10-byte cap. + const multibyte = '€€€€' + const { resources } = buildNetworkBodyResources( + [req('r1', { responseBody: multibyte })], + { maxBodyBytes: 10, maxTotalBytes: 100 } + ) + expect(resources).toHaveLength(0) + }) +}) + +describe('networkRequestToHar response bodies', () => { + it('emits plain content when no body was captured', () => { + const { snapshot } = networkRequestToHar(req('r1')) + expect(snapshot.response.content).toEqual({ + size: 7, + mimeType: 'application/json' + }) + }) + + it('inlines small bodies as text and stamps _sha1 when provided', () => { + const body = '{"a":1}' + const { snapshot } = networkRequestToHar( + req('r1', { responseBody: body }), + { + bodySha1: sha1(body) + } + ) + expect(snapshot.response.content.text).toBe(body) + expect(snapshot.response.content._sha1).toBe(sha1(body)) + }) + + it('omits inline text at and above the 8 KiB threshold', () => { + const body = 'x'.repeat(8 * 1024) + const { snapshot } = networkRequestToHar( + req('r1', { responseBody: body }), + { + bodySha1: sha1(body) + } + ) + expect(snapshot.response.content.text).toBeUndefined() + expect(snapshot.response.content._sha1).toBe(sha1(body)) + }) + + it('inlines text even without a bodySha1 ref', () => { + const body = 'plain' + const { snapshot } = networkRequestToHar(req('r1', { responseBody: body })) + expect(snapshot.response.content.text).toBe(body) + expect(snapshot.response.content._sha1).toBeUndefined() + }) +}) + +describe('writeTraceZip body wiring (ndjson-directory)', () => { + it('writes body resources and _sha1 refs into the trace output', async () => { + const body = '{"answer":42}' + const capturer: TraceCapturer = { + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [ + req('n1', { responseBody: body }), + req('n2', { responseBody: body }) + ], + commandsLog: [], + sources: new Map(), + startWallTime: 900 + } + const outputDir = await fs.mkdtemp(path.join(os.tmpdir(), 'trace-bodies-')) + const dir = await writeTraceZip(capturer, { + outputDir, + sessionId: 'abc12345', + format: 'ndjson-directory' + }) + const network = await fs.readFile(path.join(dir, 'trace.network'), 'utf8') + const entries = network + .split('\n') + .filter((line) => line.trim()) + .map( + (line) => + JSON.parse(line) as { + snapshot: { response: { content: Record<string, unknown> } } + } + ) + expect(entries).toHaveLength(2) + for (const entry of entries) { + expect(entry.snapshot.response.content._sha1).toBe(sha1(body)) + expect(entry.snapshot.response.content.text).toBe(body) + } + const resource = await fs.readFile( + path.join(dir, 'resources', sha1(body)), + 'utf8' + ) + expect(resource).toBe(body) + await fs.rm(outputDir, { recursive: true, force: true }) + }) +}) diff --git a/packages/core/tests/trace-retention.test.ts b/packages/core/tests/trace-retention.test.ts new file mode 100644 index 00000000..f2e34f82 --- /dev/null +++ b/packages/core/tests/trace-retention.test.ts @@ -0,0 +1,233 @@ +import { describe, it, expect } from 'vitest' +import { + shouldRetainTrace, + tracePolicyModeWarning +} from '../src/trace-retention.js' +import type { TestOutcome } from '../src/trace-retention.js' +import type { TraceRetentionPolicy } from '@wdio/devtools-shared' + +const allPass: TestOutcome[] = [ + { state: 'passed', attempt: 0 }, + { state: 'passed', attempt: 0 } +] +const oneFail: TestOutcome[] = [ + { state: 'passed', attempt: 0 }, + { state: 'failed', attempt: 0 } +] +const failOnRetry: TestOutcome[] = [ + { state: 'passed', attempt: 0 }, + { state: 'failed', attempt: 1 } +] +const passAfterRetry: TestOutcome[] = [ + { state: 'passed', attempt: 0 }, + { state: 'passed', attempt: 1 } +] +const secondRetryOnly: TestOutcome[] = [ + { state: 'passed', attempt: 0 }, + { state: 'passed', attempt: 2 } +] +const skippedOnly: TestOutcome[] = [{ state: 'skipped', attempt: 0 }] +const failNoAttempt: TestOutcome[] = [{ state: 'failed' }] + +function retain( + policy: TraceRetentionPolicy | undefined, + outcomes: TestOutcome[], + attemptInfoAvailable = true +): boolean { + return shouldRetainTrace(policy, { outcomes, attemptInfoAvailable }).retain +} + +describe('shouldRetainTrace policy matrix', () => { + const matrix: Array< + [TraceRetentionPolicy | undefined, TestOutcome[], string, boolean] + > = [ + [undefined, allPass, 'all-pass', true], + [undefined, oneFail, 'one-fail', true], + ['on', allPass, 'all-pass', true], + ['on', oneFail, 'one-fail', true], + ['on', skippedOnly, 'skipped-only', true], + ['retain-on-failure', allPass, 'all-pass', false], + ['retain-on-failure', oneFail, 'one-fail', true], + ['retain-on-failure', failOnRetry, 'fail-on-retry', true], + ['retain-on-failure', passAfterRetry, 'pass-after-retry', false], + ['retain-on-failure', skippedOnly, 'skipped-only', false], + ['retain-on-first-failure', allPass, 'all-pass', false], + ['retain-on-first-failure', oneFail, 'first-attempt-fail', true], + ['retain-on-first-failure', failOnRetry, 'fail-only-on-retry', false], + ['retain-on-first-failure', failNoAttempt, 'fail-without-attempt', true], + ['retain-on-first-failure', passAfterRetry, 'pass-after-retry', false], + ['on-first-retry', allPass, 'all-pass', false], + ['on-first-retry', oneFail, 'fail-without-retry', false], + ['on-first-retry', failOnRetry, 'fail-on-retry', true], + ['on-first-retry', passAfterRetry, 'pass-after-retry', true], + ['on-first-retry', secondRetryOnly, 'second-retry-only', false], + ['on-all-retries', allPass, 'all-pass', false], + ['on-all-retries', oneFail, 'fail-without-retry', false], + ['on-all-retries', failOnRetry, 'fail-on-retry', true], + ['on-all-retries', passAfterRetry, 'pass-after-retry', true], + ['on-all-retries', secondRetryOnly, 'second-retry-only', true], + ['retain-on-failure-and-retries', allPass, 'all-pass', false], + ['retain-on-failure-and-retries', oneFail, 'one-fail', true], + ['retain-on-failure-and-retries', passAfterRetry, 'retry-no-fail', true], + ['retain-on-failure-and-retries', skippedOnly, 'skipped-only', false] + ] + + it.each(matrix)( + '%s + %j (%s) → retain %s', + (policy, outcomes, _label, expected) => { + expect(retain(policy, outcomes)).toBe(expected) + } + ) + + it('sets no flags on a plain decision', () => { + expect( + shouldRetainTrace('retain-on-failure', { + outcomes: oneFail, + attemptInfoAvailable: true + }) + ).toEqual({ retain: true }) + }) + + it('accepts any iterable of outcomes', () => { + function* gen(): Generator<TestOutcome> { + yield { state: 'failed', attempt: 0 } + } + expect( + shouldRetainTrace('retain-on-failure', { + outcomes: gen(), + attemptInfoAvailable: true + }).retain + ).toBe(true) + }) +}) + +describe('shouldRetainTrace empty outcomes (fail-open)', () => { + const conditionalPolicies: TraceRetentionPolicy[] = [ + 'retain-on-failure', + 'retain-on-first-failure', + 'on-first-retry', + 'on-all-retries', + 'retain-on-failure-and-retries' + ] + + it.each(conditionalPolicies)('%s retains with failOpen', (policy) => { + expect( + shouldRetainTrace(policy, { outcomes: [], attemptInfoAvailable: true }) + ).toEqual({ retain: true, failOpen: true }) + }) + + it('fail-open wins over degradation when attempt info is also missing', () => { + expect( + shouldRetainTrace('on-first-retry', { + outcomes: [], + attemptInfoAvailable: false + }) + ).toEqual({ retain: true, failOpen: true }) + }) + + it('undefined and "on" retain without the failOpen flag', () => { + expect( + shouldRetainTrace(undefined, { outcomes: [], attemptInfoAvailable: true }) + ).toEqual({ retain: true }) + expect( + shouldRetainTrace('on', { outcomes: [], attemptInfoAvailable: true }) + ).toEqual({ retain: true }) + }) +}) + +describe('shouldRetainTrace degradation without attempt info', () => { + const retryPolicies: TraceRetentionPolicy[] = [ + 'retain-on-first-failure', + 'on-first-retry', + 'on-all-retries', + 'retain-on-failure-and-retries' + ] + + it.each(retryPolicies)('%s degrades to retain-on-failure', (policy) => { + expect( + shouldRetainTrace(policy, { + outcomes: oneFail, + attemptInfoAvailable: false + }) + ).toEqual({ retain: true, degradedToFailure: true }) + expect( + shouldRetainTrace(policy, { + outcomes: allPass, + attemptInfoAvailable: false + }) + ).toEqual({ retain: false, degradedToFailure: true }) + }) + + it.each(retryPolicies)( + '%s ignores untrustworthy attempt values when degraded', + (policy) => { + expect( + shouldRetainTrace(policy, { + outcomes: passAfterRetry, + attemptInfoAvailable: false + }) + ).toEqual({ retain: false, degradedToFailure: true }) + } + ) + + it('retain-on-failure never degrades — it needs no attempt info', () => { + expect( + shouldRetainTrace('retain-on-failure', { + outcomes: oneFail, + attemptInfoAvailable: false + }) + ).toEqual({ retain: true }) + }) + + it('"on" and undefined never degrade', () => { + expect( + shouldRetainTrace('on', { + outcomes: allPass, + attemptInfoAvailable: false + }) + ).toEqual({ retain: true }) + expect( + shouldRetainTrace(undefined, { + outcomes: allPass, + attemptInfoAvailable: false + }) + ).toEqual({ retain: true }) + }) +}) + +describe('shouldRetainTrace unknown policy (fail open)', () => { + // A JS config can pass a string TS never validated. It must retain (treated + // as `on`) rather than silently drop traces the user might need. + const unknown = 'retain-on-tuesdays' as TraceRetentionPolicy + + it('retains all-passing outcomes', () => { + expect(retain(unknown, allPass)).toBe(true) + expect(retain(unknown, allPass, false)).toBe(true) + }) + + it('retains failing outcomes', () => { + expect(retain(unknown, oneFail)).toBe(true) + }) + + it('retains with no outcomes', () => { + expect( + shouldRetainTrace(unknown, { outcomes: [], attemptInfoAvailable: false }) + ).toEqual({ retain: true }) + }) +}) + +describe('tracePolicyModeWarning', () => { + it('warns when a policy is set outside trace mode', () => { + expect(tracePolicyModeWarning('retain-on-failure', 'live')).toMatch( + /trace mode/ + ) + expect(tracePolicyModeWarning('retain-on-failure', undefined)).toMatch( + /trace mode/ + ) + }) + + it('stays silent in trace mode or when no policy is set', () => { + expect(tracePolicyModeWarning('retain-on-failure', 'trace')).toBeUndefined() + expect(tracePolicyModeWarning(undefined, 'live')).toBeUndefined() + }) +}) diff --git a/packages/core/tests/trace-sources.test.ts b/packages/core/tests/trace-sources.test.ts new file mode 100644 index 00000000..90b6640d --- /dev/null +++ b/packages/core/tests/trace-sources.test.ts @@ -0,0 +1,71 @@ +import { describe, it, expect } from 'vitest' +import { + buildSourceResources, + callSourceToStack, + sha1Hex, + sourceResourceName +} from '@wdio/devtools-core' + +describe('callSourceToStack', () => { + it('parses <file>:<line> into a single frame', () => { + expect(callSourceToStack('/specs/login.ts:42')).toEqual([ + { file: '/specs/login.ts', line: 42, column: 0 } + ]) + }) + + it('parses <file>:<line>:<column> into a single frame', () => { + expect(callSourceToStack('/specs/steps.ts:17:21')).toEqual([ + { file: '/specs/steps.ts', line: 17, column: 21 } + ]) + }) + + it('keeps windows-style drive paths intact', () => { + expect(callSourceToStack('C:\\specs\\login.ts:7')).toEqual([ + { file: 'C:\\specs\\login.ts', line: 7, column: 0 } + ]) + }) + + it('parses windows-style paths with line and column', () => { + expect(callSourceToStack('C:\\specs\\login.ts:7:5')).toEqual([ + { file: 'C:\\specs\\login.ts', line: 7, column: 5 } + ]) + }) + + it('returns undefined for missing or unknown call sources', () => { + expect(callSourceToStack(undefined)).toBeUndefined() + expect(callSourceToStack('unknown:0')).toBeUndefined() + }) + + it('falls back to line 0 when no numeric suffix exists', () => { + expect(callSourceToStack('plainfile')).toEqual([ + { file: 'plainfile', line: 0, column: 0 } + ]) + }) +}) + +describe('buildSourceResources', () => { + it('writes each source under its path-addressed resource name', () => { + const resources = buildSourceResources({ + '/specs/login.ts': 'it("logs in")' + }) + expect(resources).toEqual([ + { + resourceName: `src@${sha1Hex('/specs/login.ts')}.txt`, + data: Buffer.from('it("logs in")', 'utf8') + } + ]) + expect(resources[0]!.resourceName).toBe( + sourceResourceName('/specs/login.ts') + ) + }) + + it('skips sources above the size cap', () => { + const resources = buildSourceResources({ + '/big.js': 'x'.repeat(2 * 1024 * 1024 + 1), + '/small.ts': 'ok' + }) + expect(resources.map((r) => r.resourceName)).toEqual([ + sourceResourceName('/small.ts') + ]) + }) +}) diff --git a/packages/nightwatch-devtools/package.json b/packages/nightwatch-devtools/package.json index 39ce8788..60f418ca 100644 --- a/packages/nightwatch-devtools/package.json +++ b/packages/nightwatch-devtools/package.json @@ -62,7 +62,7 @@ "@types/ws": "^8.18.1", "@wdio/devtools-core": "workspace:^", "@wdio/devtools-shared": "workspace:^", - "chromedriver": "^148.0.4", + "chromedriver": "^150.0.0", "nightwatch": "^3.16.0", "tsup": "^8.5.1", "typescript": "^6.0.3" diff --git a/packages/nightwatch-devtools/src/cucumber-lifecycle.ts b/packages/nightwatch-devtools/src/cucumber-lifecycle.ts index afb8a832..44197c0e 100644 --- a/packages/nightwatch-devtools/src/cucumber-lifecycle.ts +++ b/packages/nightwatch-devtools/src/cucumber-lifecycle.ts @@ -13,9 +13,12 @@ import logger from '@wdio/logger' import { errorMessage } from '@wdio/devtools-core' -import { WS_SCOPE } from '@wdio/devtools-shared' +import { + WS_SCOPE, + type CucumberPickle, + type CucumberPickleStep +} from '@wdio/devtools-shared' -import type { SessionCapturer } from './session.js' import type { TestReporter } from './reporter.js' import type { TestManager } from './helpers/testManager.js' import type { SuiteManager } from './helpers/suiteManager.js' @@ -29,29 +32,21 @@ import { import { buildCucumberScenarioSuite } from './helpers/cucumberScenarioBuilder.js' import { scanFeatureFile } from './helpers/featureFileScan.js' import { parseCucumberScenario } from './helpers/utils.js' +import { + recordTestSliceBoundary, + flushTestSlice, + type TestSliceCtx +} from './trace-slices.js' const log = logger('@wdio/nightwatch-devtools:cucumber') -/** Minimal shapes for the Cucumber objects we touch. Cucumber's own types - * vary across major versions; we pin only fields we read. */ -export interface CucumberPickleStep { - text?: string - astNodeIds?: string[] - location?: { line?: number } -} -export interface CucumberPickle { - uri?: string - name?: string - location?: { line?: number } - astNodeIds?: string[] - steps?: CucumberPickleStep[] -} +/** Cucumber's result shape varies across major versions; we pin only the + * status field we read. Pickle shapes come from shared. */ export interface CucumberResult { status?: string } -export interface CucumberLifecycleCtx { - readonly sessionCapturer: SessionCapturer +export interface CucumberLifecycleCtx extends TestSliceCtx { readonly testReporter: TestReporter readonly testManager: TestManager readonly suiteManager: SuiteManager @@ -66,6 +61,7 @@ export interface CucumberLifecycleCtx { setCurrentStep(s: unknown): void getCurrentStep(): unknown setCurrentTest(t: unknown): void + recordAttempt(uid: string): number } type MutStep = { @@ -139,6 +135,15 @@ function normalizeSteps( return (pickleSteps ?? []).map((s) => ({ text: s.text ?? '' })) } +function captureFeatureSources( + ctx: CucumberLifecycleCtx, + paths: string[] +): void { + for (const p of paths) { + ctx.sessionCapturer.captureSource(p).catch(() => {}) + } +} + export async function initCucumberScenario( ctx: CucumberLifecycleCtx, browser: NightwatchBrowser, @@ -155,9 +160,7 @@ export async function initCucumberScenario( stepDefFiles, capturedPaths } = scanFeatureFile(featureUri) - for (const p of capturedPaths) { - ctx.sessionCapturer.captureSource(p).catch(() => {}) - } + captureFeatureSources(ctx, capturedPaths) const { featureSuite, scenarioLine, stepLines, stepKeywords } = createFeatureSuite( ctx, @@ -178,9 +181,12 @@ export async function initCucumberScenario( stepLines, stepKeywords, scenarioLine, - parentFeatureSuiteUid: featureSuite.uid + parentFeatureSuiteUid: featureSuite.uid, + recordAttempt: (uid) => ctx.recordAttempt(uid) }) attachScenarioToFeature(ctx, featureSuite, scenarioSuite) + // The scenario is the `test` unit; its steps are the leaf metadata entries. + recordTestSliceBoundary(ctx, featureUri, scenarioSuite.uid) ctx.setCurrentScenarioSuite(scenarioSuite) ctx.setCurrentStep(null) ctx.setCurrentTest(null) @@ -228,6 +234,9 @@ export async function finalizeCucumberScenario( ctx.setCurrentTest(null) } await ctx.sessionCapturer.captureTrace(browser) + // Flush before the next attempt's attachScenarioToFeature overwrites this + // scenario's suite (and thus its outcome) in the tree. + flushTestSlice(ctx) } catch (err) { log.error(`Failed to finalize Cucumber scenario: ${errorMessage(err)}`) } diff --git a/packages/nightwatch-devtools/src/helpers/assertCapture.ts b/packages/nightwatch-devtools/src/helpers/assertCapture.ts new file mode 100644 index 00000000..a38f025b --- /dev/null +++ b/packages/nightwatch-devtools/src/helpers/assertCapture.ts @@ -0,0 +1,55 @@ +// node:assert capture wiring: routes patched assertions into the session +// capturer through the same captureCommand path driver commands use, so +// retry bookkeeping and trace-mode action snapshots behave identically. + +import logger from '@wdio/logger' +import { + errorMessage, + patchNodeAssert, + type CapturedAssert +} from '@wdio/devtools-core' +import type { SessionCapturer } from '../session.js' + +const log = logger('@wdio/nightwatch-devtools:assertCapture') + +/** + * Patch node:assert once per process. Getters are read at capture time — the + * capturer is created lazily in session init and the test UID changes per test. + */ +export function wireAssertCapture( + getCapturer: () => SessionCapturer | undefined, + getTestUid: () => string | undefined +): void { + patchNodeAssert( + (cmd) => captureAssert(getCapturer(), getTestUid(), cmd), + (level, message) => log[level](message) + ) +} + +function captureAssert( + capturer: SessionCapturer | undefined, + testUid: string | undefined, + cmd: CapturedAssert +): void { + if (!capturer) { + return + } + capturer + .captureCommand( + cmd.command, + cmd.args, + cmd.result, + cmd.error, + testUid, + cmd.callSource, + cmd.timestamp + ) + .catch((err) => + log.warn(`Failed to capture ${cmd.command}: ${errorMessage(err)}`) + ) + // captureCommand pushes synchronously; mirror captureCommandError's send. + const last = capturer.commandsLog[capturer.commandsLog.length - 1] + if (last) { + capturer.sendCommand(last) + } +} diff --git a/packages/nightwatch-devtools/src/helpers/browserProxy.ts b/packages/nightwatch-devtools/src/helpers/browserProxy.ts index 019ebfb2..dff01a1f 100644 --- a/packages/nightwatch-devtools/src/helpers/browserProxy.ts +++ b/packages/nightwatch-devtools/src/helpers/browserProxy.ts @@ -10,11 +10,16 @@ import { } from '../constants.js' import { getCallSourceFromStack } from './utils.js' import { serializeCommandResult } from './serializeCommandResult.js' +import { + latestResolvedScreenshot, + pendingAssertionCommand +} from './nativeAssertions.js' import { RetryTracker, toError } from '@wdio/devtools-core' import type { SessionCapturer } from '../session.js' import type { TestManager } from './testManager.js' import type { CommandLog, + NativeAssertCall, NightwatchBrowser, NightwatchCurrentTest, CommandStackFrame @@ -42,10 +47,20 @@ export class BrowserProxy { */ private retryTracker = new RetryTracker() + /** + * Per-test buffer of explicit `browser.assert.*` / `browser.verify.*` calls, + * recorded synchronously at call time by {@link wrapAssertionNamespaces}. + * Nightwatch exposes no per-assertion hook and its test-end results carry no + * source location for passing assertions, so call-time capture is the only + * way to get real args + callSource. Drained (and cleared) each `afterEach`. + */ + private nativeAssertCalls: NativeAssertCall[] = [] + constructor( private sessionCapturer: SessionCapturer, private testManager: TestManager, - private getCurrentTest: () => { uid?: string } | null + private getCurrentTest: () => { uid?: string } | null, + private captureAssertions = true ) {} /** @@ -60,6 +75,78 @@ export class BrowserProxy { this.commandStack = [] this.lastCommandSig = null this.retryTracker.reset() + this.nativeAssertCalls = [] + } + + /** Hand off this test's recorded native-assertion calls and clear the + * buffer so the next test starts fresh. */ + drainNativeAssertCalls(): NativeAssertCall[] { + const calls = this.nativeAssertCalls + this.nativeAssertCalls = [] + return calls + } + + /** + * Replace `browser.assert` / `browser.verify` (both are Nightwatch Proxies + * whose `get` returns a fresh function per access) with a recording Proxy: + * on each method call it captures `{prefix, method, args, callSource}` from a + * user-code frame, streams a neutral pending row immediately (live), then + * delegates to the ORIGINAL namespace method so Nightwatch's queue, chaining, + * and abortOnFailure semantics (assert aborts, verify does not) are + * byte-for-byte unchanged. Called once per browser. + */ + private wrapAssertionNamespaces(browser: NightwatchBrowser): void { + // captureAssertions:false → leave assert/verify original so no neutral + // pending rows stream (the afterEach finalize path is gated to match). + if (!this.captureAssertions) { + return + } + // Cast once: the assert/verify namespaces aren't on the public type; each + // is a dynamic method bag reached by property name. + const b = browser as unknown as Record<string, unknown> + ;(['assert', 'verify'] as const).forEach((prefix) => { + const original = b[prefix] + if (!original || typeof original !== 'object') { + return + } + b[prefix] = new Proxy(original as object, { + get: (target, name, receiver) => { + const orig = Reflect.get(target, name, receiver) + // `not` (negation Proxy) and non-method props pass through untouched. + if (typeof orig !== 'function' || typeof name !== 'string') { + return orig + } + return (...args: unknown[]) => { + const callInfo = getCallSourceFromStack() + if (callInfo.filePath !== undefined) { + this.emitPendingAssertion({ + prefix, + method: name, + args, + callSource: callInfo.callSource, + timestamp: Date.now() + }) + } + return (orig as (...a: unknown[]) => unknown)(...args) + } + } + }) + }) + } + + /** Stream a neutral pending row for an explicit assert/verify call the moment + * it's invoked, and buffer the call (with its emitted row) for `afterEach` + * to finalize pass/fail in place. */ + private emitPendingAssertion(call: NativeAssertCall): void { + const testUid = this.getCurrentTest()?.uid + const entry = pendingAssertionCommand( + call, + testUid, + latestResolvedScreenshot(this.sessionCapturer) + ) + this.sessionCapturer.captureAssertCommand(entry) + call.entry = entry + this.nativeAssertCalls.push(call) } getCurrentTestFullPath(): string | null { @@ -189,6 +276,7 @@ export class BrowserProxy { wrappedMethods.push(methodName) }) + this.wrapAssertionNamespaces(browser) this.proxiedBrowsers.add(browser as object) log.info(`✓ Wrapped ${wrappedMethods.length} browser methods`) } @@ -307,6 +395,7 @@ export class BrowserProxy { logArgs: unknown[], cmdSig: string, callSource: string | undefined, + hasUserSource: boolean, commandTimestamp: number, testUid: string | undefined, userCallback: Function | null @@ -318,7 +407,12 @@ export class BrowserProxy { methodName ) const effectiveUid = this.getCurrentTest()?.uid ?? testUid - if (effectiveUid) { + // Only surface commands that originate from a user-code frame. Commands + // Nightwatch issues from inside its own queue (e.g. the getTitle a + // `browser.assert.titleContains` runs) execute in a detached tick with no + // user frame on the stack, so they'd otherwise leak as top-level actions + // with an "unknown" source. Mirrors the service's user-spec-source guard. + if (effectiveUid && hasUserSource) { if (this.retryTracker.isRetry(cmdSig)) { this.handleRetryReplacement( browser, @@ -402,6 +496,7 @@ export class BrowserProxy { logArgs, cmdSig, callSource, + callInfo.filePath !== undefined, commandTimestamp, this.getCurrentTest()?.uid, userCallback diff --git a/packages/nightwatch-devtools/src/helpers/cucumberScenarioBuilder.ts b/packages/nightwatch-devtools/src/helpers/cucumberScenarioBuilder.ts index 9ea426e9..b60d5ad4 100644 --- a/packages/nightwatch-devtools/src/helpers/cucumberScenarioBuilder.ts +++ b/packages/nightwatch-devtools/src/helpers/cucumberScenarioBuilder.ts @@ -21,6 +21,9 @@ export interface CucumberScenarioBuildInput { scenarioLine: number /** Parent feature-suite uid — scenarios nest under this. */ parentFeatureSuiteUid: string + /** Records the scenario start under its (retry-stable) uid and returns the + * 0-based attempt number stamped on every step. Omitted → attempt 0. */ + recordAttempt?: (scenarioUid: string) => number } /** @@ -36,6 +39,7 @@ export interface CucumberScenarioBuildInput { function buildScenarioStepTest( input: CucumberScenarioBuildInput, scenarioUid: string, + attempt: number, i: number ): SuiteStats['tests'][number] { const { @@ -57,7 +61,9 @@ function buildScenarioStepTest( ? `${featureAbsPath}:${stepLines[i]}` : undefined return { - uid: deterministicUid(featureUri, `step:${scenarioName}:${step.text}`), + // Scope by the scenario uid (which carries the scenario line) so identical + // step text in sibling scenarios and outline example rows stays distinct. + uid: deterministicUid(featureUri, `step:${scenarioUid}:${step.text}`), cid: DEFAULTS.CID, title: stepLabel, fullTitle: `${scenarioName} ${stepLabel}`, @@ -67,7 +73,9 @@ function buildScenarioStepTest( end: null, type: 'test' as const, file: featureUri, - retries: 0, + // Scenario-level attempt from the tracker (0 first run, +1 per retry); + // flows to TestOutcome.attempt via collectSuiteTestMetadata. + retries: attempt, _duration: 0, hooks: [], callSource @@ -87,8 +95,13 @@ export function buildCucumberScenarioSuite( parentFeatureSuiteUid } = input // deterministicUid (no counter) so the SAME scenario gets the SAME uid - // across retries — that's what makes retry-coalescing work upstream. - const scenarioUid = deterministicUid(featureUri, `scenario:${scenarioName}`) + // across retries — that's what makes retry-coalescing work upstream. The + // scenario line disambiguates outline example rows that share a name. + const scenarioUid = deterministicUid( + featureUri, + `scenario:${scenarioName}:${scenarioLine}` + ) + const attempt = input.recordAttempt?.(scenarioUid) ?? 0 const scenarioSuite: SuiteStats = { uid: scenarioUid, cid: DEFAULTS.CID, @@ -110,7 +123,9 @@ export function buildCucumberScenarioSuite( : undefined } for (let i = 0; i < steps.length; i++) { - scenarioSuite.tests.push(buildScenarioStepTest(input, scenarioUid, i)) + scenarioSuite.tests.push( + buildScenarioStepTest(input, scenarioUid, attempt, i) + ) } return scenarioSuite } diff --git a/packages/nightwatch-devtools/src/helpers/nativeAssertions.ts b/packages/nightwatch-devtools/src/helpers/nativeAssertions.ts new file mode 100644 index 00000000..16dd3bd2 --- /dev/null +++ b/packages/nightwatch-devtools/src/helpers/nativeAssertions.ts @@ -0,0 +1,299 @@ +// Native-assertion capture: turns explicit `browser.assert.*` / +// `browser.verify.*` calls into concise trace/UI action rows with real args, +// a clickable source location, and pass/fail colour — streamed LIVE. +// +// Nightwatch exposes no per-assertion hook, and its test-end +// `currentTest.results` carries no source location for PASSING assertions +// (results.assertions[i].stackTrace is '' on pass) and only stringified args. +// So each explicit call is intercepted at CALL TIME +// (BrowserProxy.wrapAssertionNamespaces): a neutral "pending" row is emitted +// immediately (concise title, real args, callSource) so rows stream in one by +// one like normal commands. At TEST END the pass/fail truth + verbose failure +// message are read from results.assertions and the already-emitted row is +// UPDATED IN PLACE (same stable id) — never re-created, so no duplicates. +// Iterating the recorded calls (not results.assertions) also excludes +// Nightwatch's implicit command-generated assertions (e.g. +// waitForElementVisible's "element was visible" entry). + +import logger from '@wdio/logger' +import { + matcherAssertionToCommandLog, + safeSerializeAssertArg, + stripAnsi +} from '@wdio/devtools-core' +import type { SessionCapturer } from '../session.js' +import type { + CommandLog, + NativeAssertCall, + NightwatchBrowser, + NightwatchCurrentTest +} from '../types.js' + +const log = logger('@wdio/nightwatch-devtools:nativeAssertions') + +/** + * One entry Nightwatch pushes to `results.assertions` (from + * `NightwatchAssertion.getAssertResult`, lib/assertion/assertion.js). Carries + * only a human message — no method/args. `failure === false` is the reliable + * pass signal; any truthy `failure` (message string, or `true`) means failed. + */ +interface NwAssertionEntry { + message?: string + fullMsg?: string + failure?: string | boolean +} + +/** One entry Nightwatch pushes to `results.commands` (lib/reporter/index.js + * logCommandResult). For an assert/verify `name` is the namespaced method and + * `startTime`/`endTime` are the real queue-execution window in ms + * (treenode.js). Read only to position the finalized row on the timeline. */ +interface NwCommandEntry { + name?: string + startTime?: number + endTime?: number +} + +const ASSERT_CMD_RE = /^(assert|verify)\.\w+$/ + +/** Real per-assertion execution windows, in call order, aligned to `count` + * recorded calls. Nightwatch enqueues assertions synchronously (all at once) + * but runs them later one at a time, so the enqueue timestamp clusters the + * rows; this recovers each row's true execution time so the trace timeline + * spreads them out. Returns `null` per slot when the executed-command count + * doesn't line up (e.g. retries) — the enqueue timestamp is kept then. */ +function assertCommandTimings( + commands: NwCommandEntry[], + count: number +): Array<{ startTime: number; endTime: number } | null> { + const executed = commands.filter( + (c) => typeof c?.name === 'string' && ASSERT_CMD_RE.test(c.name) + ) + if (executed.length !== count) { + return new Array(count).fill(null) + } + return executed.map((c) => + typeof c.startTime === 'number' && typeof c.endTime === 'number' + ? { startTime: c.startTime, endTime: c.endTime } + : null + ) +} + +/** Nightwatch embeds the assertion arguments in the message text + * (AssertionInstance.initialize → Logger.formatMessage), so a passing entry + * for `titleContains('Example')` reads "Testing if the page title contains + * 'Example'". Match a call to its result entry when every string/number arg + * appears in the message. */ +function messageMatchesArgs(entry: NwAssertionEntry, args: unknown[]): boolean { + const text = String(entry.fullMsg ?? entry.message ?? '') + const literals = args.filter( + (a): a is string | number => typeof a === 'string' || typeof a === 'number' + ) + return literals.length > 0 && literals.every((a) => text.includes(String(a))) +} + +interface Outcome { + passed: boolean + message: string +} + +/** + * Pair each recorded call with its `results.assertions` entry for pass/fail + + * verbose message. Both lists are in call/execution order and each explicit + * call produces exactly one assertion entry, so: match by args-in-message + * first (most specific, skips interleaved implicit entries), then fall back to + * the next unconsumed entry positionally. A call with no matching entry left + * (never happens for a real assertion) is dropped (`null`). + */ +function correlate( + calls: NativeAssertCall[], + assertions: NwAssertionEntry[] +): Array<Outcome | null> { + const consumed = new Array(assertions.length).fill(false) + const toOutcome = (idx: number): Outcome => { + consumed[idx] = true + const entry = assertions[idx] + return { + passed: !entry.failure, + message: stripAnsi(String(entry.fullMsg ?? entry.message ?? '')).trim() + } + } + const matched = calls.map((call) => { + const idx = assertions.findIndex( + (entry, i) => !consumed[i] && messageMatchesArgs(entry, call.args) + ) + return idx === -1 ? null : toOutcome(idx) + }) + return matched.map((outcome) => { + if (outcome) { + return outcome + } + const idx = consumed.findIndex((used) => !used) + return idx === -1 ? null : toOutcome(idx) + }) +} + +/** Last already-resolved screenshot in the command log — the DOM the assertion + * evaluated against (title/most asserts don't mutate it). Synchronous, so it's + * usable from the call-time wrapper. */ +export function latestResolvedScreenshot( + capturer: SessionCapturer +): string | null { + for (let i = capturer.commandsLog.length - 1; i >= 0; i--) { + const shot = capturer.commandsLog[i]?.screenshot + if (shot) { + return shot + } + } + return null +} + +/** Reuse the nearest preceding command's screenshot; if the fire-and-forget + * capture hasn't resolved yet (race), fall back to a fresh end-of-test one. */ +async function resolveAssertionScreenshot( + capturer: SessionCapturer, + browser: NightwatchBrowser +): Promise<string | null> { + return ( + latestResolvedScreenshot(capturer) ?? + (await capturer.takeScreenshotViaHttp(browser)) + ) +} + +/** `assert.titleContains('SOFT_FAIL_ME')` — the concise row label. Strings are + * quoted, objects elided; never the verbose failure message. */ +function conciseTitle(call: NativeAssertCall): string { + const preview = call.args + .map((a) => + typeof a === 'string' + ? `'${a}'` + : a !== null && typeof a === 'object' + ? '…' + : String(a) + ) + .join(', ') + return `${call.prefix}.${call.method}(${preview})` +} + +/** + * Build the neutral "pending" row emitted the moment an assert/verify is + * called — everything known at call time (concise title, real args, callSource, + * screenshot) but NO result/error yet, so it renders neutral (not red/green) + * and streams in like a normal in-flight command. `startTime`/`timestamp` are + * the call time; the row is finalized in place later by + * {@link captureNativeAssertions}. + */ +export function pendingAssertionCommand( + call: NativeAssertCall, + testUid: string | undefined, + screenshot: string | null +): CommandLog { + const entry: CommandLog = { + command: `${call.prefix}.${call.method}`, + args: call.args.map(safeSerializeAssertArg), + title: conciseTitle(call), + timestamp: call.timestamp, + startTime: call.timestamp + } + if (call.callSource) { + entry.callSource = call.callSource + } + if (testUid) { + entry.testUid = testUid + } + if (screenshot) { + entry.screenshot = screenshot + } + return entry +} + +/** Update one streamed pending row in place: apply pass/fail + verbose error, + * a screenshot, and its real execution window, then re-broadcast by stable id. */ +function finalizeAssertionRow( + capturer: SessionCapturer, + call: NativeAssertCall, + outcome: Outcome, + timing: { startTime: number; endTime: number } | null, + screenshot: string | null, + testUid: string | undefined +): void { + const entry = call.entry! + const finalized = matcherAssertionToCommandLog( + { + prefix: call.prefix, + method: call.method, + args: call.args, + passed: outcome.passed, + message: outcome.message || `${call.prefix}.${call.method} failed`, + callSource: call.callSource, + title: entry.title + }, + testUid + ) + entry.result = finalized.result + entry.error = finalized.error + if (!entry.screenshot && screenshot) { + entry.screenshot = screenshot + } + // Reposition the row on its REAL execution window (Nightwatch enqueues all + // asserts at once, so the emit/enqueue timestamp clustered them). The row is + // matched for replacement by its stable id, so the old enqueue timestamp is + // what the UI still keys on until this swap lands. + const oldTimestamp = entry.timestamp + if (timing) { + entry.startTime = timing.startTime + entry.timestamp = + timing.endTime > timing.startTime ? timing.endTime : timing.startTime + } + capturer.sendReplaceCommand(oldTimestamp, entry) + log.info(`[assert] ${entry.title} → ${outcome.passed ? 'pass' : 'fail'}`) +} + +/** + * Finalize the streamed pending rows at test-end: correlate the recorded calls + * with `results.assertions`, then UPDATE each row's `entry` in place with + * pass/fail (`result`) + the verbose failure message (`error`, failures only) + * + a screenshot + its real execution window, and re-broadcast via + * `sendReplaceCommand` keyed on the row's stable id. No new rows are created + * (no duplicates); a call with no matching result is left pending (defensive). + */ +export async function captureNativeAssertions( + capturer: SessionCapturer, + browser: NightwatchBrowser, + currentTest: NightwatchCurrentTest | undefined, + testUid: string | undefined, + calls: NativeAssertCall[] +): Promise<void> { + if (calls.length === 0) { + return + } + // Boundary cast: `results` is Nightwatch's loosely-typed per-test bag; we read + // only the assertions + commands arrays whose shapes are documented above. + const results = currentTest?.results as + | { assertions?: NwAssertionEntry[]; commands?: NwCommandEntry[] } + | undefined + const assertions = Array.isArray(results?.assertions) + ? results.assertions + : [] + const outcomes = correlate(calls, assertions) + const timings = assertCommandTimings( + Array.isArray(results?.commands) ? results.commands : [], + calls.length + ) + // One shared screenshot for all rows — same DOM, ran consecutively. + const screenshot = await resolveAssertionScreenshot(capturer, browser) + + calls.forEach((call, index) => { + const outcome = outcomes[index] + // Leave an unmatched (or never-emitted) row in its last state. + if (call.entry && outcome) { + finalizeAssertionRow( + capturer, + call, + outcome, + timings[index], + screenshot, + testUid + ) + } + }) +} diff --git a/packages/nightwatch-devtools/src/index.ts b/packages/nightwatch-devtools/src/index.ts index a1a96e4c..db5cefc0 100644 --- a/packages/nightwatch-devtools/src/index.ts +++ b/packages/nightwatch-devtools/src/index.ts @@ -8,14 +8,23 @@ import { fileURLToPath } from 'node:url' import { errorMessage, - recordSpecBoundary, - resolveAdapterOutputDir, - writeSpecTrace, - writeTraceZip, - type SpecRange + finalizeTraceExport, + flushRangeLogged, + TestAttemptTracker, + tracePolicyModeWarning, + type SpecRange, + type TraceArtifact, + type TraceExportContext } from '@wdio/devtools-core' +import { buildTraceContext } from './trace-context.js' +import { wireAssertCapture } from './helpers/assertCapture.js' import { stop as stopBackend } from '@wdio/devtools-backend' -import { REUSE_ENV, SCREENCAST_DEFAULTS } from '@wdio/devtools-shared' +import { + REUSE_ENV, + SCREENCAST_DEFAULTS, + type CucumberPickle, + type CucumberPickleStep +} from '@wdio/devtools-shared' import logger from '@wdio/logger' import { handleReuseMode, @@ -40,8 +49,7 @@ import type { NightwatchEventHub, ScreencastOptions, SuiteStats, - TestStats, - TestMetadataMap + TestStats } from './types.js' import { registerEventHandlers as registerEventHandlersImpl } from './event-hub.js' import { @@ -49,8 +57,6 @@ import { cucumberAfter as cucumberLifecycleAfter, cucumberBeforeStep as cucumberLifecycleBeforeStep, cucumberAfterStep as cucumberLifecycleAfterStep, - type CucumberPickle, - type CucumberPickleStep, type CucumberResult } from './cucumber-lifecycle.js' import { @@ -65,6 +71,8 @@ import { ensureSessionInitialized, finalizeCurrentScreencast } from './session-init.js' +import { captureNativeAssertions } from './helpers/nativeAssertions.js' +import { flushTestSlice, recordSpecSliceBoundary } from './trace-slices.js' import { getTestIcon, incrementCounters, @@ -112,6 +120,10 @@ class NightwatchDevToolsPlugin { #bidiEnabled = false #bidiAttachAttempted = false + // Nightwatch `--retries` and cross-worker reruns may reset this in-process + // tracker; only retries that re-enter this process's start hook are counted. + #attemptTracker = new TestAttemptTracker() + constructor(options: DevToolsOptions = {}) { const mode = options.mode ?? 'live' const ignore = mode === 'trace' && options.screencast?.enabled === true @@ -124,9 +136,15 @@ class NightwatchDevToolsPlugin { hostname: options.hostname ?? 'localhost', screencast, bidi: options.bidi ?? false, + captureAssertions: options.captureAssertions ?? true, mode, traceFormat: options.traceFormat ?? 'zip', - traceGranularity: options.traceGranularity ?? 'session' + traceGranularity: options.traceGranularity ?? 'session', + tracePolicy: options.tracePolicy ?? 'on' + } + const policyWarning = tracePolicyModeWarning(options.tracePolicy, mode) + if (policyWarning) { + log.warn(policyWarning) } this.#screencastOptions = { ...SCREENCAST_DEFAULTS, ...screencast } this.#bidiEnabled = options.bidi === true @@ -162,6 +180,9 @@ class NightwatchDevToolsPlugin { get bidiEnabled() { return self.#bidiEnabled }, + get captureAssertions() { + return self.options.captureAssertions + }, get sessionCapturer() { return self.sessionCapturer }, @@ -276,7 +297,10 @@ class NightwatchDevToolsPlugin { clearExecutionData: () => { self.testReporter.clearExecutionData() self.suiteManager.clearExecutionData() + self.#attemptTracker.reset() }, + recordAttempt: (uid) => self.#attemptTracker.recordStart(uid), + attemptFor: (uid) => self.#attemptTracker.attemptFor(uid), buildMetadataOptions: () => self.#buildMetadataOptions(), ensureSessionInitialized: (b) => self.#ensureSessionInitialized(b), wrapBrowserOnce: (b) => self.#wrapBrowserOnce(b), @@ -285,11 +309,29 @@ class NightwatchDevToolsPlugin { setCucumberRunner: (v) => { self.#isCucumberRunner = v }, - getRerunLabel: () => self.#getRerunLabel() + getRerunLabel: () => self.#getRerunLabel(), + get traceMode() { + return self.options.mode === 'trace' + }, + get traceGranularity() { + return self.options.traceGranularity + }, + get specRanges() { + return self.#specRanges + }, + get flushedSpecs() { + return self.#flushedSpecs + }, + flushTraceRange: (range) => self.#flushSpecTrace(range) } return this.#internals } + /** Boundary cast: currentTest is Nightwatch's loose bag; only uid is read. */ + #currentTestUid(): string | undefined { + return (this.#currentTest as { uid?: string } | null)?.uid + } + #handleReuseMode(): void { handleReuseMode(this.#getInternals()) } @@ -307,6 +349,12 @@ class NightwatchDevToolsPlugin { internals.handleReuse = () => this.#handleReuseMode() internals.plugin = this await runPluginBefore(internals) + if (this.options.captureAssertions) { + wireAssertCapture( + () => this.sessionCapturer, + () => this.#currentTestUid() + ) + } } async #ensureSessionInitialized(browser: NightwatchBrowser) { @@ -374,13 +422,15 @@ class NightwatchDevToolsPlugin { async #startNextTest( currentSuite: SuiteStats, currentTestName: string, - processedTests: Set<string> + processedTests: Set<string>, + specFile: string | null ): Promise<void> { await startNextTest( this.#getInternals(), currentSuite, currentTestName, - processedTests + processedTests, + specFile ) } @@ -426,25 +476,8 @@ class NightwatchDevToolsPlugin { this.sessionCapturer.captureSource(fullPath).catch(() => {}) } - // ── Per-spec boundary detection ── if (fullPath) { - const prevRange = recordSpecBoundary( - { - specRanges: this.#specRanges, - flushedSpecs: this.#flushedSpecs, - capturer: this.sessionCapturer, - actionSnapshots: this.sessionCapturer.actionSnapshots - }, - fullPath, - this.options.traceGranularity - ) - if (prevRange) { - void this.#flushSpecTrace(prevRange).catch((err) => - log.warn( - `Failed to flush trace for spec "${prevRange.specFile}": ${errorMessage(err)}` - ) - ) - } + recordSpecSliceBoundary(this.#getInternals(), fullPath) } await this.#closePreviousRunningTest(currentSuite, testFile, currentTest) @@ -456,7 +489,12 @@ class NightwatchDevToolsPlugin { processedTests ) if (currentTestName) { - await this.#startNextTest(currentSuite, currentTestName, processedTests) + await this.#startNextTest( + currentSuite, + currentTestName, + processedTests, + fullPath + ) } this.#wrapBrowserOnce(browser) } @@ -470,7 +508,18 @@ class NightwatchDevToolsPlugin { if (browser && this.sessionCapturer) { try { await this.#closeOutTestcases(browser) + if (this.options.captureAssertions) { + await captureNativeAssertions( + this.sessionCapturer, + browser, + browser.currentTest as NightwatchCurrentTest | undefined, + this.#currentTestUid(), + this.browserProxy.drainNativeAssertCalls() + ) + } await this.sessionCapturer.captureTrace(browser) + // Flush this test's slice before the next test overwrites its outcome. + flushTestSlice(this.#getInternals()) } catch (err) { log.error(`Failed to capture trace: ${errorMessage(err)}`) } @@ -487,7 +536,7 @@ class NightwatchDevToolsPlugin { await this.#finalizeAllSuites(browser) this.#logRunSummary() if (this.options.mode === 'trace') { - await this.#writeTraceZipIfNeeded() + await this.#writeTraceIfNeeded() await this.sessionCapturer?.closeWebSocket() await stopBackend() return @@ -515,108 +564,43 @@ class NightwatchDevToolsPlugin { logRunSummary(this.#getInternals()) } - async #flushSpecTrace( - range: SpecRange, - nextRange?: SpecRange - ): Promise<string | undefined> { - if (this.#flushedSpecs.has(range.specFile)) { - return undefined - } - this.#flushedSpecs.add(range.specFile) - + /** Thin wrapper so boundary flushes and the final flush share one path. + * flushRangeLogged logs+swallows a failed flush (shared spec/test string) so + * the fire-and-forget boundary callers don't each re-implement the guard. */ + #flushSpecTrace(range: SpecRange): Promise<TraceArtifact | undefined> { const sessionId = this.sessionCapturer.metadata?.sessionId if (!sessionId) { - return undefined - } - - // Collect test metadata from the suite tree. Nightwatch stores one suite - // per spec file (flat data model) — a flat iteration is correct. If - // child suites are ever introduced, this must switch to a recursive walk - // matching the selenium adapter. - const allMetadata: TestMetadataMap = new Map() - if (this.suiteManager) { - for (const suite of this.suiteManager.getAllSuites().values()) { - for (const entry of suite.tests) { - if (typeof entry === 'string') { - continue - } - allMetadata.set(entry.uid, { - title: entry.fullTitle, - specFile: entry.file - }) - } - } + return Promise.resolve(undefined) } + return flushRangeLogged(this.#traceContext(sessionId), range) + } - const tracePath = await writeSpecTrace({ - range, - nextRange, - capturer: this.sessionCapturer, - actionSnapshots: this.sessionCapturer.actionSnapshots, - sessionId, - outputDir: resolveAdapterOutputDir({ configPath: this.#configPath }), - format: this.options.traceFormat, - testMetadata: allMetadata - }) - log.info(`Trace for spec "${range.specFile}" saved to ${tracePath}`) - return tracePath + /** Assemble the framework-agnostic trace-export context from plugin state. + * Output dir ignores the spec range — nightwatch writes next to config. */ + #traceContext(sessionId: string): TraceExportContext { + return buildTraceContext( + { + mode: this.options.mode, + policy: this.options.tracePolicy, + granularity: this.options.traceGranularity, + format: this.options.traceFormat, + capturer: this.sessionCapturer, + suites: this.suiteManager.getAllSuites().values(), + ranges: this.#specRanges, + flushed: this.#flushedSpecs, + configPath: this.#configPath, + log: (level, msg) => log[level](msg) + }, + sessionId + ) } - async #writeTraceZipIfNeeded(): Promise<void> { - if (this.options.mode !== 'trace' || !this.sessionCapturer) { - return - } - const sessionId = this.sessionCapturer.metadata?.sessionId - if (!sessionId) { + async #writeTraceIfNeeded(): Promise<void> { + const sessionId = this.sessionCapturer?.metadata?.sessionId + if (this.options.mode !== 'trace' || !this.sessionCapturer || !sessionId) { return } - try { - if (this.sessionCapturer.snapshotCaptures.length) { - await Promise.allSettled(this.sessionCapturer.snapshotCaptures) - } - - if (this.options.traceGranularity === 'spec') { - // Per-spec traces — flush any remaining ranges that weren't - // flushed at spec boundaries (the last spec in the run). - for (const range of this.#specRanges) { - if (!this.#flushedSpecs.has(range.specFile)) { - await this.#flushSpecTrace(range) - } - } - return - } - - // Session-level trace (default) — single artifact for the - // entire worker session. - const testMetadata: TestMetadataMap = new Map() - if (this.suiteManager) { - for (const suite of this.suiteManager.getAllSuites().values()) { - for (const entry of suite.tests) { - if (typeof entry === 'string') { - continue - } - testMetadata.set(entry.uid, { - title: entry.fullTitle, - specFile: entry.file - }) - } - } - } - - const snapshots = this.sessionCapturer.actionSnapshots - const tracePath = await writeTraceZip(this.sessionCapturer, { - outputDir: resolveAdapterOutputDir({ - configPath: this.#configPath - }), - sessionId, - actionSnapshots: snapshots.length ? snapshots : undefined, - format: this.options.traceFormat, - testMetadata - }) - log.info(`Trace saved to ${tracePath}`) - } catch (err) { - log.warn(`trace write failed: ${errorMessage(err)}`) - } + await finalizeTraceExport(this.#traceContext(sessionId)) } async #waitForDevtoolsBrowserClose(): Promise<void> { diff --git a/packages/nightwatch-devtools/src/plugin-internals.ts b/packages/nightwatch-devtools/src/plugin-internals.ts index afce58b4..1753d09c 100644 --- a/packages/nightwatch-devtools/src/plugin-internals.ts +++ b/packages/nightwatch-devtools/src/plugin-internals.ts @@ -7,6 +7,7 @@ * four) while still letting each lifecycle module narrow its dependencies. */ +import type { SpecRange, TraceArtifact } from '@wdio/devtools-core' import type { SessionCapturer } from './session.js' import type { TestReporter } from './reporter.js' import type { ScreencastRecorder } from './screencast.js' @@ -18,7 +19,8 @@ import type { NightwatchBrowser, ScreencastOptions, SuiteStats, - TestStats + TestStats, + TraceGranularity } from './types.js' export interface PluginInternals { @@ -29,6 +31,7 @@ export interface PluginInternals { readonly mode: DevToolsMode readonly screencastOptions: ScreencastOptions readonly bidiEnabled: boolean + readonly captureAssertions: boolean // Runtime instances (mutable — bringup/session-change replaces them) sessionCapturer: SessionCapturer @@ -73,4 +76,18 @@ export interface PluginInternals { testIcon(state: TestStats['state']): string setCucumberRunner(v: boolean): void getRerunLabel(): string | undefined + + /** Records a test/scenario start under its retry-stable uid; returns the + * 0-based attempt number (0 first run, +1 per rerun). */ + recordAttempt(uid: string): number + /** Latest attempt recorded for `uid`, or undefined if it never started. */ + attemptFor(uid: string): number | undefined + + // Per-test trace slicing (`test` granularity). Boundary state is shared with + // the finalizer; flushTraceRange writes one slice via the plugin's context. + readonly traceMode: boolean + readonly traceGranularity: TraceGranularity + readonly specRanges: SpecRange[] + readonly flushedSpecs: Set<string> + flushTraceRange(range: SpecRange): Promise<TraceArtifact | undefined> } diff --git a/packages/nightwatch-devtools/src/reporter.ts b/packages/nightwatch-devtools/src/reporter.ts index a6e8282e..63626d27 100644 --- a/packages/nightwatch-devtools/src/reporter.ts +++ b/packages/nightwatch-devtools/src/reporter.ts @@ -1,5 +1,5 @@ import logger from '@wdio/logger' -import { TestReporterBase } from '@wdio/devtools-core' +import { TestReporterBase, type ReporterUpstream } from '@wdio/devtools-core' import { REUSE_ENV } from '@wdio/devtools-shared' import { DEFAULTS } from './constants.js' import { extractTestMetadata, generateStableUid } from './helpers/utils.js' @@ -11,6 +11,15 @@ export class TestReporter extends TestReporterBase { #currentSpecFile?: string #testNamesCache = new Map<string, string[]>() #currentSuite?: SuiteStats + #attemptFor: (uid: string) => number | undefined + + constructor( + report: ReporterUpstream, + attemptFor: (uid: string) => number | undefined + ) { + super(report) + this.#attemptFor = attemptFor + } onSuiteStart(suiteStats: SuiteStats) { this.#currentSpecFile = suiteStats.file @@ -92,11 +101,12 @@ export class TestReporter extends TestReporterBase { cachedNames.forEach((testName) => { if (!processedTestNames.has(testName)) { + const uid = generateStableUid({ + file: suiteStats.file, + fullTitle: `${suiteStats.title} ${testName}` + } as TestStats) const skippedTest: TestStats = { - uid: generateStableUid({ - file: suiteStats.file, - fullTitle: `${suiteStats.title} ${testName}` - } as TestStats), + uid, cid: DEFAULTS.CID, title: testName, fullTitle: `${suiteStats.title} ${testName}`, @@ -106,7 +116,7 @@ export class TestReporter extends TestReporterBase { end: new Date(), type: 'test', file: suiteStats.file, - retries: DEFAULTS.RETRIES, + retries: this.#attemptFor(uid) ?? DEFAULTS.RETRIES, _duration: DEFAULTS.DURATION, hooks: [] } diff --git a/packages/nightwatch-devtools/src/session-init.ts b/packages/nightwatch-devtools/src/session-init.ts index 71a1c21b..e9d1004a 100644 --- a/packages/nightwatch-devtools/src/session-init.ts +++ b/packages/nightwatch-devtools/src/session-init.ts @@ -39,6 +39,7 @@ export interface SessionInitCtx { readonly screencastOptions: ScreencastOptions readonly bidiEnabled: boolean readonly mode: DevToolsMode + readonly captureAssertions: boolean sessionCapturer: SessionCapturer testReporter: TestReporter @@ -57,6 +58,7 @@ export interface SessionInitCtx { getCurrentTest(): unknown getCurrentScenarioSuite(): SuiteStats | null buildMetadataOptions(): unknown + attemptFor(uid: string): number | undefined } async function handleSessionChange( @@ -84,17 +86,21 @@ function initReporterChain(ctx: SessionInitCtx): void { // These must NOT be recreated on session change — doing so generates a // new feature suite with a fresh start timestamp, which DataManager sees // as a new run and wipes all accumulated commands. - ctx.testReporter = new TestReporter((suitesData) => { - if (ctx.sessionCapturer) { - ctx.sessionCapturer.sendUpstream('suites', suitesData) - } - }) + ctx.testReporter = new TestReporter( + (suitesData) => { + if (ctx.sessionCapturer) { + ctx.sessionCapturer.sendUpstream('suites', suitesData) + } + }, + (uid) => ctx.attemptFor(uid) + ) ctx.testManager = new TestManager(ctx.testReporter) ctx.suiteManager = new SuiteManager(ctx.testReporter) ctx.browserProxy = new BrowserProxy( ctx.sessionCapturer, ctx.testManager, - () => ctx.getCurrentTest() ?? ctx.getCurrentScenarioSuite() + () => ctx.getCurrentTest() ?? ctx.getCurrentScenarioSuite(), + ctx.captureAssertions ) } diff --git a/packages/nightwatch-devtools/src/session.ts b/packages/nightwatch-devtools/src/session.ts index ae830af7..ed9c5457 100644 --- a/packages/nightwatch-devtools/src/session.ts +++ b/packages/nightwatch-devtools/src/session.ts @@ -213,6 +213,41 @@ export class SessionCapturer extends SessionCapturerBase { ) } + /** + * Ingest a pre-built assertion entry (native `browser.assert`/`verify` + * synthesized from Nightwatch's results, or a node:assert capture) into the + * same command stream driver commands use. Unlike {@link captureCommand} this + * preserves the entry's `title` (the human assertion message) and never + * dedups — each call is a distinct action row. Assigns a fresh `_id` and a + * matching stable public `id` (so a later `sendReplaceCommand` can update + * this exact row in place — native asserts stream a pending row at call time, + * then finalize their pass/fail), pushes, and broadcasts. + */ + captureAssertCommand(entry: CommandLog): void { + const withId = entry as CommandLog & { _id?: number } + withId._id = this.commandCounter++ + withId.id = withId._id + this.commandsLog.push(withId) + if ( + this.traceMode === 'trace' && + !entry.error && + this.#browser && + mapCommandToAction(entry.command) + ) { + const browser = this.#browser + this.snapshotCaptures.push( + captureActionSnapshot(browser, entry.command, () => + this.takeScreenshotViaHttp(browser) + ).then((snap) => { + if (snap) { + this.actionSnapshots.push(snap) + } + }) + ) + } + this.sendCommand(withId) + } + /** * Replace an already-captured command entry (used for retried commands so * only the final execution result is shown in the UI). diff --git a/packages/nightwatch-devtools/src/test-lifecycle.ts b/packages/nightwatch-devtools/src/test-lifecycle.ts index 4bbd52e6..7192c12a 100644 --- a/packages/nightwatch-devtools/src/test-lifecycle.ts +++ b/packages/nightwatch-devtools/src/test-lifecycle.ts @@ -9,7 +9,6 @@ */ import logger from '@wdio/logger' -import type { SessionCapturer } from './session.js' import type { TestReporter } from './reporter.js' import type { TestManager } from './helpers/testManager.js' import type { SuiteManager } from './helpers/suiteManager.js' @@ -26,11 +25,11 @@ import { DEFAULTS, TIMING, TEST_STATE } from './constants.js' import { resolveSpecFilePath } from './helpers/specFileResolver.js' import { closePreviousTest } from './helpers/closePreviousTest.js' import { extractTestMetadata, determineTestState } from './helpers/utils.js' +import { recordTestSliceBoundary, type TestSliceCtx } from './trace-slices.js' const log = logger('@wdio/nightwatch-devtools:test-lifecycle') -export interface TestLifecycleCtx { - readonly sessionCapturer: SessionCapturer +export interface TestLifecycleCtx extends TestSliceCtx { readonly testReporter: TestReporter readonly testManager: TestManager readonly suiteManager: SuiteManager @@ -41,6 +40,7 @@ export interface TestLifecycleCtx { incrementCount(state: TestStats['state']): void testIcon(state: TestStats['state']): string setCurrentTest(t: unknown): void + recordAttempt(uid: string): number } interface SuiteMetadata { @@ -117,13 +117,19 @@ export async function startNextTest( ctx: TestLifecycleCtx, currentSuite: SuiteStats, currentTestName: string, - processedTests: Set<string> + processedTests: Set<string>, + specFile: string | null ): Promise<void> { if (processedTests.size === 0) { ctx.suiteManager.markSuiteAsRunning(currentSuite) } const test = ctx.testManager.findTestInSuite(currentSuite, currentTestName) if (test) { + // Nightwatch has no per-test retry index; the tracker is the retry signal. + test.retries = ctx.recordAttempt(test.uid) + if (specFile) { + recordTestSliceBoundary(ctx, specFile, test.uid) + } test.state = TEST_STATE.RUNNING as TestStats['state'] test.start = new Date() test.end = null diff --git a/packages/nightwatch-devtools/src/trace-context.ts b/packages/nightwatch-devtools/src/trace-context.ts new file mode 100644 index 00000000..1e451ac7 --- /dev/null +++ b/packages/nightwatch-devtools/src/trace-context.ts @@ -0,0 +1,60 @@ +/** + * Assembles the framework-agnostic trace-export context from plugin state. + * + * Extracted from `NightwatchDevToolsPlugin.#traceContext` so the assembly is + * unit-testable and to keep `index.ts` under the file-size cap. Both the + * per-spec boundary flush and the final trace write share this one builder. + */ + +import { + collectSuiteTestMetadata, + resolveAdapterOutputDir, + type SpecRange, + type TraceExportContext +} from '@wdio/devtools-core' +import type { SessionCapturer } from './session.js' +import type { + DevToolsMode, + SuiteStats, + TraceFormat, + TraceGranularity, + TraceRetentionPolicy +} from './types.js' + +export interface TraceContextInput { + mode: DevToolsMode + policy: TraceRetentionPolicy + granularity: TraceGranularity + format: TraceFormat + capturer: SessionCapturer + suites: Iterable<SuiteStats> + ranges: SpecRange[] + flushed: Set<string> + configPath: string | undefined + log: (level: 'info' | 'warn', msg: string) => void +} + +export function buildTraceContext( + input: TraceContextInput, + sessionId: string +): TraceExportContext { + return { + mode: input.mode, + policy: input.policy, + granularity: input.granularity, + format: input.format, + capturer: input.capturer, + actionSnapshots: input.capturer.actionSnapshots, + sessionId, + testMetadata: collectSuiteTestMetadata(input.suites), + ranges: input.ranges, + flushed: input.flushed, + resolveOutputDir: () => + resolveAdapterOutputDir({ configPath: input.configPath }), + awaitPending: input.capturer.snapshotCaptures, + // Nightwatch feeds real per-test attempt numbers via TestAttemptTracker + // (B4), so retry-aware policies use per-test attempts, not the fallback. + attemptInfoAvailable: true, + log: input.log + } +} diff --git a/packages/nightwatch-devtools/src/trace-slices.ts b/packages/nightwatch-devtools/src/trace-slices.ts new file mode 100644 index 00000000..bdd0cf0d --- /dev/null +++ b/packages/nightwatch-devtools/src/trace-slices.ts @@ -0,0 +1,102 @@ +/** + * Trace-slice boundary recording for `spec` and `test` granularity. + * + * `spec` slices are recorded/flushed at each spec-file transition (unchanged + * from the original inline `beforeEach` logic). For `test`, Nightwatch builds + * test outcomes into the suite tree in place — a retry overwrites the previous + * attempt's state (regular tests reuse the same test object; cucumber replaces + * the scenario suite). So each attempt's slice is flushed at its own + * test/scenario END, before the next attempt can overwrite the outcome the + * flush reads. Regular tests drive this from test-lifecycle, cucumber scenarios + * from cucumber-lifecycle; both share this module. + */ + +import { + recordSliceBoundary, + recordSpecBoundary, + type SpecBoundaryContext, + type SpecRange, + type TraceArtifact +} from '@wdio/devtools-core' +import type { SessionCapturer } from './session.js' +import type { TraceGranularity } from './types.js' + +export interface TestSliceCtx { + readonly sessionCapturer: SessionCapturer + readonly traceMode: boolean + readonly traceGranularity: TraceGranularity + readonly specRanges: SpecRange[] + readonly flushedSpecs: Set<string> + flushTraceRange(range: SpecRange): Promise<TraceArtifact | undefined> +} + +function sliceActive(ctx: TestSliceCtx): boolean { + return ctx.traceMode && ctx.traceGranularity === 'test' +} + +function boundaryContext(ctx: TestSliceCtx): SpecBoundaryContext { + return { + specRanges: ctx.specRanges, + flushedSpecs: ctx.flushedSpecs, + capturer: ctx.sessionCapturer, + actionSnapshots: ctx.sessionCapturer.actionSnapshots + } +} + +function flushPrevious(ctx: TestSliceCtx, prevRange: SpecRange | null): void { + if (!prevRange) { + return + } + // flushTraceRange (→ core flushRangeLogged) logs+swallows on failure. + void ctx.flushTraceRange(prevRange) +} + +/** + * Record a spec-file boundary at test/scenario start and eagerly flush the + * previous spec's slice. No-op for `session`/`test` granularity (core returns + * null). Preserves the original `beforeEach` behavior verbatim. + */ +export function recordSpecSliceBoundary( + ctx: TestSliceCtx, + specFile: string +): void { + const prevRange = recordSpecBoundary( + boundaryContext(ctx), + specFile, + ctx.traceGranularity + ) + flushPrevious(ctx, prevRange) +} + +/** + * Record a per-test slice boundary at test/scenario start. No-op outside trace + * mode + `test` granularity. Retries push a distinct range (core keys a repeat + * `${testUid}-retry${n}`), so each attempt becomes its own artifact. + */ +export function recordTestSliceBoundary( + ctx: TestSliceCtx, + specFile: string, + testUid: string +): void { + if (!sliceActive(ctx)) { + return + } + recordSliceBoundary(boundaryContext(ctx), 'test', specFile, testUid) +} + +/** + * Flush the current test's slice at test/scenario end — before a retry can + * overwrite its outcome in the suite tree. Fire-and-forget; the finalize pass + * is the safety net for any range this misses. No-op outside trace + `test`. + */ +export function flushTestSlice(ctx: TestSliceCtx): void { + if (!sliceActive(ctx)) { + return + } + const range = ctx.specRanges[ctx.specRanges.length - 1] + if (!range) { + return + } + // flushTraceRange (→ core flushRangeLogged) logs+swallows on failure. + void ctx.flushTraceRange(range) +} diff --git a/packages/nightwatch-devtools/src/types.ts b/packages/nightwatch-devtools/src/types.ts index 6a927d98..203dbbdb 100644 --- a/packages/nightwatch-devtools/src/types.ts +++ b/packages/nightwatch-devtools/src/types.ts @@ -19,15 +19,11 @@ export { type TraceFormat, type TestMetadataMap, type TraceGranularity, + type TraceRetentionPolicy, type TraceLog } from '@wdio/devtools-shared' -import type { - DevToolsMode, - ScreencastOptions, - TraceFormat, - TraceGranularity -} from '@wdio/devtools-shared' +import type { BaseDevToolsOptions, CommandLog } from '@wdio/devtools-shared' export interface CommandStackFrame { command: string @@ -35,6 +31,20 @@ export interface CommandStackFrame { signature: string } +/** One explicit `browser.assert.*` / `browser.verify.*` call, recorded at call + * time by BrowserProxy. A neutral "pending" row (`entry`) is streamed live at + * call time; `captureNativeAssertions` finalizes its pass/fail at test-end. */ +export interface NativeAssertCall { + prefix: 'assert' | 'verify' + method: string + args: unknown[] + callSource?: string + /** Wall-clock at call time; also the streamed row's timestamp/startTime. */ + timestamp: number + /** The pending row emitted at call time, updated in place when finalized. */ + entry?: CommandLog +} + export interface NightwatchTestCase { passed: number failed: number @@ -85,16 +95,7 @@ export interface StepLocation { line: number } -export interface DevToolsOptions { - port?: number - hostname?: string - /** - * Screencast recording options. When enabled, a continuous video of the - * browser session is recorded and saved as a .webm file at the end of the - * test run. Polling mode only on Nightwatch (no CDP push); works on every - * browser Nightwatch supports. - */ - screencast?: ScreencastOptions +export interface DevToolsOptions extends BaseDevToolsOptions { /** * Enable WebDriver BiDi capture (browser console + JS exceptions + network * via `selenium-webdriver/bidi`). Requires `webSocketUrl: true` in your @@ -103,15 +104,6 @@ export interface DevToolsOptions { * entries. Defaults to `false` — opt-in. */ bidi?: boolean - /** `live` (default) launches the DevTools UI; `trace` skips it. */ - mode?: DevToolsMode - /** Trace output layout — `zip` (default) writes a single archive, - * `ndjson-directory` unpacks into `trace-<id>/`. Only applies in trace mode. */ - traceFormat?: TraceFormat - /** Trace output granularity — `session` (default) writes one trace per - * worker session; `spec` writes one trace per spec file. Only applies in - * trace mode. */ - traceGranularity?: TraceGranularity } export interface NightwatchBrowser { @@ -136,12 +128,7 @@ export interface NightwatchBrowser { webdriver?: { host?: string } [key: string]: unknown } - currentTest?: { - name?: string - module?: string - group?: string - [key: string]: unknown - } + currentTest?: NightwatchCurrentTest results?: unknown queue?: unknown } diff --git a/packages/nightwatch-devtools/tests/assertCapture.test.ts b/packages/nightwatch-devtools/tests/assertCapture.test.ts new file mode 100644 index 00000000..61602964 --- /dev/null +++ b/packages/nightwatch-devtools/tests/assertCapture.test.ts @@ -0,0 +1,94 @@ +import { describe, it, expect, vi, afterAll } from 'vitest' +import assert from 'node:assert' +import { + ASSERT_PATCHED_SYMBOL, + TRACKED_ASSERT_METHODS +} from '@wdio/devtools-core' +import { wireAssertCapture } from '../src/helpers/assertCapture.js' +import type { SessionCapturer } from '../src/session.js' +import type { CommandLog } from '../src/types.js' + +// Snapshot real methods so the process-wide patch is undone after this file. +const ASSERT_MUT = assert as unknown as Record<string | symbol, unknown> +const originals: Record<string, unknown> = {} +for (const method of TRACKED_ASSERT_METHODS) { + originals[method] = ASSERT_MUT[method] +} +afterAll(() => { + delete ASSERT_MUT[ASSERT_PATCHED_SYMBOL] + for (const method of TRACKED_ASSERT_METHODS) { + ASSERT_MUT[method] = originals[method] + } +}) + +function makeFakeCapturer() { + const commandsLog: CommandLog[] = [] + const captureCommand = vi.fn( + ( + command: string, + args: unknown[], + result: unknown, + error: Error | undefined, + testUid?: string, + callSource?: string, + timestamp?: number + ) => { + commandsLog.push({ + command, + args, + result, + error, + testUid, + callSource, + timestamp: timestamp ?? Date.now() + }) + return Promise.resolve(true) + } + ) + const sendCommand = vi.fn() + // Fake narrowed to the three members the wiring touches. + const capturer = { + commandsLog, + captureCommand, + sendCommand + } as unknown as SessionCapturer + return { capturer, commandsLog, captureCommand, sendCommand } +} + +describe('wireAssertCapture', () => { + it('routes node:assert calls through captureCommand and sends the entry', () => { + const live: { + fake?: ReturnType<typeof makeFakeCapturer> + uid?: string + } = {} + wireAssertCapture( + () => live.fake?.capturer, + () => live.uid + ) + + // No capturer yet — asserts must not throw from the capture path. + expect(() => assert.ok(true)).not.toThrow() + + const fake = makeFakeCapturer() + live.fake = fake + live.uid = 'test-uid' + assert.equal(2, 2) + expect(fake.captureCommand).toHaveBeenCalledWith( + 'assert.equal', + [2, 2], + 'passed', + undefined, + 'test-uid', + expect.any(String), + expect.any(Number) + ) + expect(fake.sendCommand).toHaveBeenCalledWith(fake.commandsLog[0]) + + expect(() => assert.strictEqual('a', 'b')).toThrow() + const failed = fake.commandsLog[1] + expect(failed.command).toBe('assert.strictEqual') + expect(failed.result).toBeUndefined() + expect(failed.error).toBeInstanceOf(Error) + expect(fake.sendCommand).toHaveBeenCalledTimes(2) + }) +}) diff --git a/packages/nightwatch-devtools/tests/attemptTracking.test.ts b/packages/nightwatch-devtools/tests/attemptTracking.test.ts new file mode 100644 index 00000000..30df38e2 --- /dev/null +++ b/packages/nightwatch-devtools/tests/attemptTracking.test.ts @@ -0,0 +1,134 @@ +import { describe, it, expect, vi } from 'vitest' +import { + TestAttemptTracker, + collectSuiteTestMetadata +} from '@wdio/devtools-core' +import { buildCucumberScenarioSuite } from '../src/helpers/cucumberScenarioBuilder.js' +import { buildTraceContext } from '../src/trace-context.js' +import { startNextTest, type TestLifecycleCtx } from '../src/test-lifecycle.js' +import { TEST_STATE } from '../src/constants.js' +import type { SessionCapturer } from '../src/session.js' +import type { SuiteStats, TestStats } from '../src/types.js' + +function scenarioInput(recordAttempt: (uid: string) => number) { + return { + featureUri: 'login.feature', + scenarioName: 'Valid login', + featureName: 'Login', + stepDefFiles: [], + steps: [{ text: 'I open the page' }, { text: 'I log in' }], + stepLines: [0, 0], + stepKeywords: ['Given', 'When'], + scenarioLine: 3, + parentFeatureSuiteUid: 'feature-uid', + recordAttempt + } +} + +function makeTest(uid: string, title: string): TestStats { + return { + uid, + cid: '0-0', + title, + fullTitle: title, + parent: 'suite-uid', + state: TEST_STATE.PENDING, + start: new Date(), + end: null, + type: 'test', + file: '/spec.ts', + retries: 0, + _duration: 0, + hooks: [] + } +} + +describe('cucumber scenario retry attempt', () => { + it('stamps the tracker attempt on every step and flows it to metadata.attempt', () => { + const tracker = new TestAttemptTracker() + const record = (uid: string) => tracker.recordStart(uid) + + const first = buildCucumberScenarioSuite(scenarioInput(record)) + expect((first.tests as TestStats[]).map((t) => t.retries)).toEqual([0, 0]) + + // Same scenario name + line → same (retry-stable) uid → attempt increments. + const retried = buildCucumberScenarioSuite(scenarioInput(record)) + expect((retried.tests as TestStats[]).map((t) => t.retries)).toEqual([1, 1]) + expect(tracker.sawRetry).toBe(true) + + const metadata = collectSuiteTestMetadata([retried]) + for (const step of retried.tests as TestStats[]) { + expect(metadata.get(step.uid)?.attempt).toBe(1) + } + }) + + it('defaults step attempt to 0 when no tracker is wired', () => { + const suite = buildCucumberScenarioSuite({ + ...scenarioInput(() => 0), + recordAttempt: undefined + }) + expect((suite.tests as TestStats[]).every((t) => t.retries === 0)).toBe( + true + ) + }) +}) + +describe('regular test retry attempt via startNextTest', () => { + it('records the attempt under the test uid and increments on rerun', async () => { + const tracker = new TestAttemptTracker() + const test = makeTest('test-uid', 'my test') + const suite = { + uid: 'suite-uid', + tests: [test] + } as unknown as SuiteStats + const ctx = { + suiteManager: { markSuiteAsRunning: vi.fn() }, + testManager: { findTestInSuite: () => test }, + testReporter: { onTestStart: vi.fn() }, + setCurrentTest: vi.fn(), + recordAttempt: (uid: string) => tracker.recordStart(uid) + } as unknown as TestLifecycleCtx + + await startNextTest(ctx, suite, 'my test', new Set(), null) + expect(test.retries).toBe(0) + + await startNextTest(ctx, suite, 'my test', new Set(['my test']), null) + expect(test.retries).toBe(1) + }) +}) + +describe('buildTraceContext', () => { + it('sets attemptInfoAvailable and carries retries through to attempt', () => { + const capturer = { + actionSnapshots: [], + snapshotCaptures: [] + } as unknown as SessionCapturer + const test = makeTest('t1', 'retried test') + test.retries = 2 + const suite = { + uid: 'suite-uid', + tests: [test], + suites: [] + } as unknown as SuiteStats + + const ctx = buildTraceContext( + { + mode: 'trace', + policy: 'on', + granularity: 'session', + format: 'zip', + capturer, + suites: [suite], + ranges: [], + flushed: new Set(), + configPath: undefined, + log: () => {} + }, + 'session-1' + ) + + expect(ctx.attemptInfoAvailable).toBe(true) + expect(ctx.sessionId).toBe('session-1') + expect(ctx.testMetadata.get('t1')?.attempt).toBe(2) + }) +}) diff --git a/packages/nightwatch-devtools/tests/browserProxy.test.ts b/packages/nightwatch-devtools/tests/browserProxy.test.ts new file mode 100644 index 00000000..d69ffcf3 --- /dev/null +++ b/packages/nightwatch-devtools/tests/browserProxy.test.ts @@ -0,0 +1,168 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import type { CommandLog, NightwatchBrowser } from '../src/types.js' + +// browserProxy resolves the caller's source via this helper; stub it so each +// test can decide whether the command looks user-issued or framework-internal. +// vi.hoisted so the stub exists when the hoisted vi.mock factory runs. +const { getCallSourceFromStack } = vi.hoisted(() => ({ + getCallSourceFromStack: vi.fn() +})) +vi.mock('../src/helpers/utils.js', () => ({ getCallSourceFromStack })) + +import { BrowserProxy } from '../src/helpers/browserProxy.js' +import type { SessionCapturer } from '../src/session.js' +import type { TestManager } from '../src/helpers/testManager.js' + +function makeCapturer() { + const commandsLog: (CommandLog & { _id?: number })[] = [] + let counter = 0 + const captureCommand = vi.fn( + async ( + command: string, + args: unknown[], + result: unknown, + error: Error | undefined, + testUid?: string, + callSource?: string, + timestamp?: number + ) => { + commandsLog.push({ + _id: counter++, + command, + args, + result, + error, + testUid, + callSource, + timestamp + }) + return true + } + ) + const capturer = { + commandsLog, + captureCommand, + replaceCommand: vi.fn(), + sendCommand: vi.fn(), + sendReplaceCommand: vi.fn(), + takeScreenshotViaHttp: vi.fn(async () => null), + captureTrace: vi.fn(async () => {}) + } as unknown as SessionCapturer + return { capturer, commandsLog, captureCommand } +} + +function makeTestManager() { + return { + detectTestBoundary: vi.fn(() => ''), + startTestIfPending: vi.fn() + } as unknown as TestManager +} + +/** A browser whose single command echoes its result into the capture callback + * Nightwatch appends as the last argument. */ +function makeBrowser() { + return { + titleContains: (_arg: unknown, cb: (result: unknown) => void) => { + cb('Example Domain') + return undefined + } + } as unknown as NightwatchBrowser +} + +describe('BrowserProxy internal-command suppression', () => { + beforeEach(() => { + getCallSourceFromStack.mockReset() + }) + + it('captures a command issued from a user-code frame', () => { + const { capturer, commandsLog, captureCommand } = makeCapturer() + getCallSourceFromStack.mockReturnValue({ + filePath: '/tests/spec.js', + callSource: '/tests/spec.js:5' + }) + const proxy = new BrowserProxy(capturer, makeTestManager(), () => ({ + uid: 'test-1' + })) + const browser = makeBrowser() + proxy.wrapBrowserCommands(browser) + ;( + browser as unknown as Record<string, (a: unknown) => unknown> + ).titleContains('Example') + + expect(captureCommand).toHaveBeenCalledTimes(1) + expect(commandsLog).toHaveLength(1) + expect(commandsLog[0].command).toBe('titleContains') + expect(commandsLog[0].callSource).toBe('/tests/spec.js:5') + }) + + it('suppresses a framework-internal command with no user-code frame', () => { + const { capturer, commandsLog, captureCommand } = makeCapturer() + // Mirrors the getTitle a `browser.assert.titleContains` issues from inside + // Nightwatch's queue: no user frame, so getCallSourceFromStack returns none. + getCallSourceFromStack.mockReturnValue({ + filePath: undefined, + callSource: 'unknown:0' + }) + const proxy = new BrowserProxy(capturer, makeTestManager(), () => ({ + uid: 'test-1' + })) + const browser = makeBrowser() + proxy.wrapBrowserCommands(browser) + ;( + browser as unknown as Record<string, (a: unknown) => unknown> + ).titleContains('Example') + + expect(captureCommand).not.toHaveBeenCalled() + expect(commandsLog).toHaveLength(0) + }) +}) + +describe('BrowserProxy captureAssertions gating', () => { + beforeEach(() => { + getCallSourceFromStack.mockReset() + }) + + /** A browser exposing `assert`/`verify` namespace objects (as Nightwatch + * does), so the test can check whether wrapAssertionNamespaces replaced them + * with a recording Proxy. */ + function makeAssertBrowser() { + return { + assert: { titleContains: vi.fn() }, + verify: { titleContains: vi.fn() } + } as unknown as NightwatchBrowser & { + assert: object + verify: object + } + } + + it('leaves assert/verify namespaces original when captureAssertions is false', () => { + const { capturer } = makeCapturer() + const browser = makeAssertBrowser() + const originalAssert = browser.assert + const originalVerify = browser.verify + const proxy = new BrowserProxy( + capturer, + makeTestManager(), + () => ({ uid: 't1' }), + false + ) + proxy.wrapBrowserCommands(browser) + // No wrapping → the namespaces are untouched, so no pending rows can stream. + expect(browser.assert).toBe(originalAssert) + expect(browser.verify).toBe(originalVerify) + }) + + it('wraps assert/verify namespaces by default (captureAssertions true)', () => { + const { capturer } = makeCapturer() + const browser = makeAssertBrowser() + const originalAssert = browser.assert + const originalVerify = browser.verify + const proxy = new BrowserProxy(capturer, makeTestManager(), () => ({ + uid: 't1' + })) + proxy.wrapBrowserCommands(browser) + // Replaced with a recording Proxy → no longer the original object. + expect(browser.assert).not.toBe(originalAssert) + expect(browser.verify).not.toBe(originalVerify) + }) +}) diff --git a/packages/nightwatch-devtools/tests/nativeAssertions.test.ts b/packages/nightwatch-devtools/tests/nativeAssertions.test.ts new file mode 100644 index 00000000..d904dbde --- /dev/null +++ b/packages/nightwatch-devtools/tests/nativeAssertions.test.ts @@ -0,0 +1,340 @@ +import { describe, it, expect, vi } from 'vitest' +import { + captureNativeAssertions, + latestResolvedScreenshot, + pendingAssertionCommand +} from '../src/helpers/nativeAssertions.js' +import type { SessionCapturer } from '../src/session.js' +import type { + CommandLog, + NativeAssertCall, + NightwatchBrowser, + NightwatchCurrentTest +} from '../src/types.js' + +/** Fake capturer mimicking the live stream: `captureAssertCommand` assigns a + * stable public `id` and appends to `commandsLog` (like the real one); + * `sendReplaceCommand` records in-place updates; `takeScreenshotViaHttp` is + * the fresh-fallback. */ +function makeFakeCapturer( + commandsLog: CommandLog[] = [], + freshScreenshot: string | null = 'FRESH_SHOT' +) { + const sent: CommandLog[] = [] + const replaced: Array<{ oldTimestamp: number; command: CommandLog }> = [] + let counter = 100 + const captureAssertCommand = vi.fn( + (entry: CommandLog & { _id?: number; id?: number }) => { + entry._id = counter++ + entry.id = entry._id + commandsLog.push(entry) + sent.push(entry) + } + ) + const sendReplaceCommand = vi.fn( + (oldTimestamp: number, command: CommandLog) => { + replaced.push({ oldTimestamp, command }) + } + ) + const takeScreenshotViaHttp = vi.fn(() => Promise.resolve(freshScreenshot)) + const capturer = { + commandsLog, + captureAssertCommand, + sendReplaceCommand, + takeScreenshotViaHttp + } as unknown as SessionCapturer + return { + capturer, + commandsLog, + sent, + replaced, + captureAssertCommand, + sendReplaceCommand, + takeScreenshotViaHttp + } +} + +const browser = {} as unknown as NightwatchBrowser + +/** Mimic BrowserProxy.emitPendingAssertion: stream a neutral pending row at + * call time and attach it to the call for later finalization. */ +function emitPending( + capturer: SessionCapturer, + calls: NativeAssertCall[], + testUid: string | undefined +) { + for (const call of calls) { + const entry = pendingAssertionCommand( + call, + testUid, + latestResolvedScreenshot(capturer) + ) + capturer.captureAssertCommand(entry) + call.entry = entry + } +} + +/** Nightwatch's `getAssertResult` entries: pass → `failure: false`; fail → + * `failure` is a message string. The message embeds the assertion args. */ +function passing(message: string) { + return { message, fullMsg: message, failure: false as const } +} +function failing(message: string) { + return { message, fullMsg: message, failure: `${message} — failed` } +} + +function currentTestWith( + assertions: unknown[], + commands: unknown[] = [] +): NightwatchCurrentTest { + return { + name: 'renders asserts', + module: 'assert-check', + results: { assertions, commands } + } as unknown as NightwatchCurrentTest +} + +let clock = 1000 +function call( + prefix: 'assert' | 'verify', + method: string, + args: unknown[], + callSource = 'spec.js:8' +): NativeAssertCall { + return { prefix, method, args, callSource, timestamp: clock++ } +} + +describe('captureNativeAssertions (live pending → finalize)', () => { + it('streams neutral pending rows at call time, then finalizes pass/fail in place with a stable id and no duplicates', async () => { + // Preceding real commands: the last with a resolved screenshot is the DOM + // the assertions evaluated against — reused for the pending rows. + const precedingCommands: CommandLog[] = [ + { command: 'url', args: [], timestamp: 1, screenshot: 'URL_SHOT' }, + { + command: 'waitForElementVisible', + args: ['body'], + timestamp: 2, + screenshot: 'BODY_SHOT' + } + ] + const { + capturer, + sent, + replaced, + commandsLog, + captureAssertCommand, + sendReplaceCommand, + takeScreenshotViaHttp + } = makeFakeCapturer(precedingCommands) + const calls = [ + call('verify', 'titleContains', ['Example'], 'spec.js:11'), + call('verify', 'titleContains', ['SOFT_FAIL_ME'], 'spec.js:12'), + call('assert', 'titleContains', ['Example'], 'spec.js:15'), + call('assert', 'titleContains', ['HARD_FAIL_ME'], 'spec.js:16') + ] + + // 1) CALL TIME: each call streams one neutral pending row. + emitPending(capturer, calls, 'test-uid') + expect(captureAssertCommand).toHaveBeenCalledTimes(4) + expect(sent).toHaveLength(4) + for (const row of sent) { + expect(row.result).toBeUndefined() // neutral — not green + expect(row.error).toBeUndefined() // neutral — not red + expect(row.testUid).toBe('test-uid') + expect(row.screenshot).toBe('BODY_SHOT') // reused preceding DOM + expect(typeof (row as { id?: number }).id).toBe('number') + } + expect(sent[0].title).toBe("verify.titleContains('Example')") + expect(sent[0].command).toBe('verify.titleContains') + expect(sent[0].args).toEqual(['Example']) + expect(sent[0].callSource).toBe('spec.js:11') + expect(sent[0].timestamp).toBe(sent[0].startTime) + const pendingIds = sent.map((r) => (r as { id?: number }).id) + + // 2) TEST END: results.assertions includes the implicit waitForElementVisible + // assertion (first entry) which must NOT create/finalize a row. + const assertions = [ + passing('Element <body> was visible after 16 milliseconds'), + passing("Testing if the page title contains 'Example'"), + failing("Testing if the page title contains 'SOFT_FAIL_ME'"), + passing("Testing if the page title contains 'Example'"), + failing("Testing if the page title contains 'HARD_FAIL_ME'") + ] + await captureNativeAssertions( + capturer, + browser, + currentTestWith(assertions), + 'test-uid', + calls + ) + + // Each row updated exactly once, in place — no new rows, no duplicates. + expect(captureAssertCommand).toHaveBeenCalledTimes(4) + expect(sendReplaceCommand).toHaveBeenCalledTimes(4) + expect(commandsLog).toHaveLength(2 + 4) + expect(takeScreenshotViaHttp).not.toHaveBeenCalled() + + // Same row objects, same stable ids (updated, not recreated). + const finalized = calls.map((c) => c.entry!) + expect(finalized.map((r) => (r as { id?: number }).id)).toEqual(pendingIds) + + // Pass/fail applied; failures carry the verbose message as error. + expect(finalized[0].result).toBe('passed') + expect(finalized[0].error).toBeUndefined() + expect(finalized[1].result).toBeUndefined() + expect((finalized[1].error as { message: string }).message).toContain( + 'SOFT_FAIL_ME' + ) + expect(finalized[2].result).toBe('passed') // duplicate 'Example' → 2nd entry + expect(finalized[3].error).toBeDefined() + + // Labels/args/callSource preserved from the pending row. + expect(finalized[3].title).toBe("assert.titleContains('HARD_FAIL_ME')") + expect(finalized[3].callSource).toBe('spec.js:16') + + // With no results.commands timing, rows keep their enqueue timestamp, so + // the replace is keyed on that same timestamp. + replaced.forEach(({ oldTimestamp, command }) => { + expect(oldTimestamp).toBe(command.timestamp) + }) + }) + + it('repositions each row on its real execution window from results.commands (not enqueue time)', async () => { + const { capturer, sent, replaced } = makeFakeCapturer() + const calls = [ + call('verify', 'titleContains', ['Example']), + call('assert', 'titleContains', ['SOFT_FAIL_ME']) + ] + emitPending(capturer, calls, 'uid') + // Both enqueued ~together (clock++), clustered. + const enqueue = sent.map((r) => r.timestamp) + + // Nightwatch ran them ~29ms apart; results.commands carries the real window. + const assertions = [ + passing("Testing if the page title contains 'Example'"), + failing("Testing if the page title contains 'SOFT_FAIL_ME'") + ] + const commands = [ + { name: 'url', startTime: 5000, endTime: 5100 }, + { name: 'verify.titleContains', startTime: 6000, endTime: 6029 }, + { name: 'assert.titleContains', startTime: 6100, endTime: 6132 } + ] + await captureNativeAssertions( + capturer, + browser, + currentTestWith(assertions, commands), + 'uid', + calls + ) + + const finalized = calls.map((c) => c.entry!) + // startTime/timestamp now reflect the real execution window, spread apart. + expect(finalized[0].startTime).toBe(6000) + expect(finalized[0].timestamp).toBe(6029) + expect(finalized[1].startTime).toBe(6100) + expect(finalized[1].timestamp).toBe(6132) + expect(finalized[1].timestamp - finalized[0].timestamp).toBeGreaterThan(50) + // The replace is keyed on the ORIGINAL enqueue timestamp (stable id also + // matches), while the row now carries its real execution timestamp. + expect(replaced[0].oldTimestamp).toBe(enqueue[0]) + expect(replaced[0].command.timestamp).toBe(6029) + }) + + it('finalizes with a fresh screenshot when no preceding one has resolved yet', async () => { + // Race: preceding capture is fire-and-forget and unresolved → pending rows + // stream without a screenshot; finalize takes a fresh end-of-test one. + const pending: CommandLog[] = [ + { command: 'waitForElementVisible', args: ['body'], timestamp: 2 } + ] + const { capturer, sent, takeScreenshotViaHttp } = makeFakeCapturer( + pending, + 'FRESH_SHOT' + ) + const calls = [call('verify', 'titleContains', ['Example'])] + emitPending(capturer, calls, 'uid') + expect(sent[0].screenshot).toBeUndefined() + + await captureNativeAssertions( + capturer, + browser, + currentTestWith([ + passing("Testing if the page title contains 'Example'") + ]), + 'uid', + calls + ) + expect(takeScreenshotViaHttp).toHaveBeenCalledTimes(1) + expect(calls[0].entry!.screenshot).toBe('FRESH_SHOT') + expect(calls[0].entry!.result).toBe('passed') + }) + + it('preserves real multi-arg assertions (no faked args)', async () => { + const { capturer, sent, replaced } = makeFakeCapturer() + const calls = [call('assert', 'containsText', ['#btn', 'Save'])] + emitPending(capturer, calls, 'uid') + expect(sent[0].args).toEqual(['#btn', 'Save']) + expect(sent[0].title).toBe("assert.containsText('#btn', 'Save')") + + await captureNativeAssertions( + capturer, + browser, + currentTestWith([ + failing("Testing if element <#btn> contains text 'Save'") + ]), + 'uid', + calls + ) + expect(replaced).toHaveLength(1) + expect(calls[0].entry!.error).toBeDefined() + }) + + it('falls back to positional correlation when args are not literals', async () => { + const { capturer, sent } = makeFakeCapturer() + // Element-object arg won't substring-match; positional order still pairs + // the single call with the single explicit assertion outcome. + const calls = [call('assert', 'elementPresent', [{ selector: 'body' }])] + emitPending(capturer, calls, 'uid') + expect(sent[0].title).toBe('assert.elementPresent(…)') + + await captureNativeAssertions( + capturer, + browser, + currentTestWith([failing('Testing if element <body> is present')]), + 'uid', + calls + ) + expect(calls[0].entry!.error).toBeDefined() + }) + + it('leaves an uncorrelated pending row in its neutral state (defensive)', async () => { + const { capturer, sent, sendReplaceCommand } = makeFakeCapturer() + const calls = [call('assert', 'ok', [true])] + emitPending(capturer, calls, 'uid') + // No matching results entry — row must not be dropped or mis-coloured. + await captureNativeAssertions( + capturer, + browser, + currentTestWith([]), + 'uid', + calls + ) + expect(sent).toHaveLength(1) + expect(sendReplaceCommand).not.toHaveBeenCalled() + expect(calls[0].entry!.result).toBeUndefined() + expect(calls[0].entry!.error).toBeUndefined() + }) + + it('is a no-op when there are no recorded calls (implicit assertions ignored)', async () => { + const { capturer, sendReplaceCommand, takeScreenshotViaHttp } = + makeFakeCapturer() + await captureNativeAssertions( + capturer, + browser, + currentTestWith([passing('Element <body> was visible')]), + 'uid', + [] + ) + expect(sendReplaceCommand).not.toHaveBeenCalled() + expect(takeScreenshotViaHttp).not.toHaveBeenCalled() + }) +}) diff --git a/packages/nightwatch-devtools/tests/traceSlices.test.ts b/packages/nightwatch-devtools/tests/traceSlices.test.ts new file mode 100644 index 00000000..b42e92d5 --- /dev/null +++ b/packages/nightwatch-devtools/tests/traceSlices.test.ts @@ -0,0 +1,155 @@ +import { describe, it, expect, vi } from 'vitest' +import type { SpecRange, TraceArtifact } from '@wdio/devtools-core' +import { + recordTestSliceBoundary, + recordSpecSliceBoundary, + flushTestSlice, + type TestSliceCtx +} from '../src/trace-slices.js' +import type { SessionCapturer } from '../src/session.js' +import type { TraceGranularity } from '../src/types.js' + +function makeCtx(traceMode: boolean, granularity: TraceGranularity) { + const capturer = { + commandsLog: [], + consoleLogs: [], + networkRequests: [], + mutations: [], + traceLogs: [], + actionSnapshots: [] + } as unknown as SessionCapturer + const artifacts: TraceArtifact[] = [] + const flushTraceRange = vi.fn( + async (range: SpecRange): Promise<TraceArtifact> => { + const artifact: TraceArtifact = { + kind: 'trace', + path: `/out/${range.key}.zip`, + scope: range.testUid ? 'test' : 'spec', + key: range.key, + testUids: range.testUid ? [range.testUid] : [], + retained: true + } + artifacts.push(artifact) + return artifact + } + ) + const ctx: TestSliceCtx = { + sessionCapturer: capturer, + traceMode, + traceGranularity: granularity, + specRanges: [], + flushedSpecs: new Set<string>(), + flushTraceRange + } + return { ctx, artifacts, flushTraceRange } +} + +describe('test granularity — recordTestSliceBoundary', () => { + it('records a per-test slice keyed on the test uid', () => { + const { ctx } = makeCtx(true, 'test') + recordTestSliceBoundary(ctx, '/login.spec.js', 'uid-1') + expect(ctx.specRanges).toHaveLength(1) + expect(ctx.specRanges[0]).toMatchObject({ + key: 'uid-1', + testUid: 'uid-1', + specFile: '/login.spec.js' + }) + }) + + it('keys a retry of the same test as a distinct slice', () => { + const { ctx } = makeCtx(true, 'test') + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') // retry + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-2') // sibling test + expect(ctx.specRanges.map((r) => r.key)).toEqual([ + 'uid-1', + 'uid-1-retry1', + 'uid-2' + ]) + // Every slice stays a test slice (base uid preserved for metadata filter). + expect(ctx.specRanges.map((r) => r.testUid)).toEqual([ + 'uid-1', + 'uid-1', + 'uid-2' + ]) + }) +}) + +describe('test granularity — flushTestSlice', () => { + it('emits one artifact per test slice, flushing the current test at its end', () => { + const { ctx, artifacts, flushTraceRange } = makeCtx(true, 'test') + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + flushTestSlice(ctx) + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-2') + flushTestSlice(ctx) + + expect(flushTraceRange).toHaveBeenCalledTimes(2) + expect(artifacts.map((a) => a.key)).toEqual(['uid-1', 'uid-2']) + expect(artifacts.every((a) => a.scope === 'test')).toBe(true) + }) + + it('flushes each attempt separately so retries become distinct artifacts', () => { + const { ctx, artifacts } = makeCtx(true, 'test') + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + flushTestSlice(ctx) // attempt 0 flushed before the retry can overwrite it + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + flushTestSlice(ctx) // attempt 1 + + expect(artifacts.map((a) => a.key)).toEqual(['uid-1', 'uid-1-retry1']) + }) + + it('does nothing when there is no recorded slice', () => { + const { ctx, flushTraceRange } = makeCtx(true, 'test') + flushTestSlice(ctx) + expect(flushTraceRange).not.toHaveBeenCalled() + }) +}) + +describe('test-slice helpers are inert outside trace + test granularity', () => { + it('no-ops for spec and session granularity', () => { + for (const granularity of ['spec', 'session'] as TraceGranularity[]) { + const { ctx, flushTraceRange } = makeCtx(true, granularity) + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + flushTestSlice(ctx) + expect(ctx.specRanges).toHaveLength(0) + expect(flushTraceRange).not.toHaveBeenCalled() + } + }) + + it('no-ops in live mode even at test granularity', () => { + const { ctx, flushTraceRange } = makeCtx(false, 'test') + recordTestSliceBoundary(ctx, '/a.spec.js', 'uid-1') + flushTestSlice(ctx) + expect(ctx.specRanges).toHaveLength(0) + expect(flushTraceRange).not.toHaveBeenCalled() + }) +}) + +describe('spec granularity is unchanged by the test-slice work', () => { + it('records one slice per spec file and flushes the previous on transition', () => { + const { ctx, artifacts, flushTraceRange } = makeCtx(true, 'spec') + recordSpecSliceBoundary(ctx, '/a.spec.js') + expect(ctx.specRanges.map((r) => r.key)).toEqual(['/a.spec.js']) + expect(flushTraceRange).not.toHaveBeenCalled() // no previous spec yet + + recordSpecSliceBoundary(ctx, '/a.spec.js') // same spec → shared slice + expect(ctx.specRanges).toHaveLength(1) + + recordSpecSliceBoundary(ctx, '/b.spec.js') // new spec → flush previous + expect(ctx.specRanges.map((r) => r.key)).toEqual([ + '/a.spec.js', + '/b.spec.js' + ]) + expect(artifacts.map((a) => a.key)).toEqual(['/a.spec.js']) + expect(artifacts[0].scope).toBe('spec') + }) + + it('records nothing for session or test granularity', () => { + for (const granularity of ['session', 'test'] as TraceGranularity[]) { + const { ctx, flushTraceRange } = makeCtx(true, granularity) + recordSpecSliceBoundary(ctx, '/a.spec.js') + expect(ctx.specRanges).toHaveLength(0) + expect(flushTraceRange).not.toHaveBeenCalled() + } + }) +}) diff --git a/packages/selenium-devtools/src/constants.ts b/packages/selenium-devtools/src/constants.ts index 50457bea..cc39f0c2 100644 --- a/packages/selenium-devtools/src/constants.ts +++ b/packages/selenium-devtools/src/constants.ts @@ -103,28 +103,3 @@ export const BLANK_FRAME_THRESHOLD_BYTES = 4_000 /** Per-prototype "already patched" guard for driverPatcher / assertPatcher. */ export const PATCHED_SYMBOL = Symbol.for('@wdio/selenium-devtools/patched') - -/** Per-prototype guard for the (currently disabled) node:assert patcher. */ -export const ASSERT_PATCHED_SYMBOL = Symbol.for( - '@wdio/selenium-devtools/assert-patched' -) - -/** node:assert methods the (currently disabled) assertPatcher would wrap. */ -export const TRACKED_ASSERT_METHODS = [ - 'equal', - 'strictEqual', - 'deepEqual', - 'deepStrictEqual', - 'notEqual', - 'notStrictEqual', - 'notDeepEqual', - 'notDeepStrictEqual', - 'ok', - 'fail', - 'throws', - 'doesNotThrow', - 'rejects', - 'doesNotReject', - 'match', - 'doesNotMatch' -] as const diff --git a/packages/selenium-devtools/src/helpers/testManager.ts b/packages/selenium-devtools/src/helpers/testManager.ts index fc3bfebf..70fcb14b 100644 --- a/packages/selenium-devtools/src/helpers/testManager.ts +++ b/packages/selenium-devtools/src/helpers/testManager.ts @@ -1,5 +1,9 @@ import logger from '@wdio/logger' -import { createTestStats, stampTestEnd } from '@wdio/devtools-core' +import { + createTestStats, + stampTestEnd, + TestAttemptTracker +} from '@wdio/devtools-core' import { DEFAULTS, TEST_STATE } from '../constants.js' import type { SuiteStats, TestStats } from '../types.js' import type { TestReporter } from '../reporter.js' @@ -19,6 +23,9 @@ export class TestManager { #currentTest: TestStats | null = null #lastMarkedTest: TestStats | null = null #mode: 'session' | 'marked' = 'session' + // Per-test attempt counter. Persists for the whole in-process session so + // same-process retries (Mocha/Jest/etc re-entering the start hook) accumulate. + #attemptTracker = new TestAttemptTracker() /** Set true the first time the user calls startMarkedTest. Once true we * never auto-create the synthetic session test — orphan commands attach * to the most-recently-marked test instead. */ @@ -90,7 +97,7 @@ export class TestManager { */ startMarkedTest( name: string, - opts: { file?: string; callSource?: string } = {} + opts: { file?: string; callSource?: string; attempt?: number } = {} ): TestStats { if (!this.#userTookOver) { this.#userTookOver = true @@ -115,14 +122,22 @@ export class TestManager { const file = opts.file || this.suite.file // Scope by parent so two suites with the same test/step name don't // collide on signatureCounter disambiguation across rerun processes. + const signature = `${this.suite.uid}::${name}` const test = createTestStats({ - uid: generateStableUid(file, `${this.suite.uid}::${name}`), + uid: generateStableUid(file, signature), cid: DEFAULTS.CID, title: name, file, parent: this.suite.uid, callSource: opts.callSource }) + // deterministicUid is retry-stable (no counter), so re-entering with the + // same logical test increments the heuristic. Mocha supplies an + // authoritative per-test retry index via opts.attempt; prefer it. + const heuristicAttempt = this.#attemptTracker.recordStart( + deterministicUid(file, signature) + ) + test.retries = opts.attempt ?? heuristicAttempt log.info( `Started marked test "${name}" (callSource: ${opts.callSource || 'n/a'})` ) diff --git a/packages/selenium-devtools/src/index.ts b/packages/selenium-devtools/src/index.ts index 11bada94..7289cd81 100644 --- a/packages/selenium-devtools/src/index.ts +++ b/packages/selenium-devtools/src/index.ts @@ -21,7 +21,8 @@ import { onDriverEnd as sessionOnDriverEnd, onSessionEnd as sessionOnSessionEnd, setPluginRef, - recordSpecBoundary + recordTraceBoundary, + flushCurrentTestTrace } from './session-lifecycle.js' import type { SpecRange } from '@wdio/devtools-core' import { @@ -30,8 +31,10 @@ import { startScenario as tmStartScenario, endScenario as tmEndScenario, flushPendingTestActions as tmFlushPendingTestActions, + buildStartTestMeta, type StartTestMeta, - type StartScenarioMeta + type StartScenarioMeta, + type PendingTestAction } from './test-management.js' import type { PluginInternals } from './plugin-internals.js' import { @@ -40,7 +43,11 @@ import { detectSeleniumVersion } from './helpers/runtime.js' import { findFreePort } from './helpers/utils.js' -import { RetryTracker, errorMessage } from '@wdio/devtools-core' +import { + RetryTracker, + errorMessage, + tracePolicyModeWarning +} from '@wdio/devtools-core' import { tryRegisterRunnerHooks } from './runnerHooks.js' import { patchNodeAssert } from './assertPatcher.js' import { @@ -100,16 +107,7 @@ class SeleniumDevToolsPlugin { #uiReadyPromise?: Promise<void> // First it() body fires before onDriverCreated's async setup completes — // buffer startTest/endTest until testManager exists. - #pendingTestActions: Array< - | { - kind: 'start' - name: string - meta: { file?: string; callSource?: string } - suiteName?: string - suiteCallSource?: string - } - | { kind: 'end'; state: TestStats['state'] } - > = [] + #pendingTestActions: PendingTestAction[] = [] // Cucumber Before fires before the driver-build Before — stash and replay. #pendingScenario: { name: string @@ -125,17 +123,29 @@ class SeleniumDevToolsPlugin { /** Set of spec files already flushed to disk. */ #flushedSpecs = new Set<string>() + /** In-flight per-test eager flushes (test granularity), awaited at finalize. */ + #traceFlushes: Promise<unknown>[] = [] + constructor(options: DevToolsOptions = {}) { this.#options = { port: options.port ?? 3000, hostname: options.hostname ?? 'localhost', openUi: options.openUi ?? true, captureScreenshots: options.captureScreenshots ?? true, + captureAssertions: options.captureAssertions ?? true, rerunCommand: options.rerunCommand, headless: options.headless ?? false, mode: options.mode ?? 'live', traceFormat: options.traceFormat ?? 'zip', - traceGranularity: options.traceGranularity ?? 'session' + traceGranularity: options.traceGranularity ?? 'session', + tracePolicy: options.tracePolicy ?? 'on' + } + const policyWarning = tracePolicyModeWarning( + options.tracePolicy, + this.#options.mode + ) + if (policyWarning) { + log.warn(policyWarning) } this.#rerunManager = new RerunManager(RUNNER) if (options.rerunCommand) { @@ -263,6 +273,7 @@ class SeleniumDevToolsPlugin { screencast?: ScreencastOptions headless?: boolean openUi?: boolean + captureAssertions?: boolean mode?: DevToolsMode traceFormat?: TraceFormat traceGranularity?: TraceGranularity @@ -275,6 +286,9 @@ class SeleniumDevToolsPlugin { if (typeof opts.headless === 'boolean') { this.#options.headless = opts.headless } + if (typeof opts.captureAssertions === 'boolean') { + this.#options.captureAssertions = opts.captureAssertions + } if (typeof opts.openUi === 'boolean') { this.#options.openUi = opts.openUi } @@ -417,6 +431,9 @@ class SeleniumDevToolsPlugin { get flushedSpecs() { return self.#flushedSpecs }, + get traceFlushes() { + return self.#traceFlushes + }, setFinalized: (v) => { self.#finalized = v }, @@ -441,20 +458,17 @@ class SeleniumDevToolsPlugin { /** Public API: start a marked test. */ startTest(name: string, meta: StartTestMeta = {}) { tmStartTest(this.#getInternals(), name, meta) - if (meta.file) { - recordSpecBoundary(this.#getInternals(), meta.file) - } + recordTraceBoundary(this.#getInternals(), meta.file) } endTest(state: TestStats['state'] = 'passed') { tmEndTest(this.#getInternals(), state) + flushCurrentTestTrace(this.#getInternals()) } startScenario(name: string, meta: StartScenarioMeta = {}) { tmStartScenario(this.#getInternals(), name, meta) - if (meta.file) { - recordSpecBoundary(this.#getInternals(), meta.file) - } + recordTraceBoundary(this.#getInternals(), meta.file) } endScenario(state: TestStats['state'] = 'passed') { @@ -463,6 +477,11 @@ class SeleniumDevToolsPlugin { #flushPendingTestActions() { tmFlushPendingTestActions(this.#getInternals()) + // The first test's startTest fires before the driver/capturer exists, so + // its per-test boundary is recorded here once capture is live. + if (this.#options.traceGranularity === 'test') { + recordTraceBoundary(this.#getInternals(), this.#testFilePath) + } } async onDriverCreated(driver: SeleniumDriverLike) { @@ -568,33 +587,35 @@ if (patched) { log.info('✓ selenium-devtools attached — waiting for driver creation') } -// node:assert wrappers silently invert match/doesNotMatch — kept disabled. -void patchNodeAssert +// Patch eagerly (user specs must see the wrappers before they import assert); +// the gate runs at capture time because DevTools.configure() may arrive later. +patchNodeAssert((cmd) => { + if (plugin.options.captureAssertions) { + void plugin.onCommand(cmd) + } +}) // Runner globals are published after `--require`, so retry briefly. function registerHooks() { return tryRegisterRunnerHooks({ - onTestStart: (name, file, callSource, suiteName, suiteCallSource) => { - const meta: { - file?: string - callSource?: string - suiteName?: string - suiteCallSource?: string - } = {} - if (file) { - meta.file = file - } - if (callSource) { - meta.callSource = callSource - } - if (suiteName) { - meta.suiteName = suiteName - } - if (suiteCallSource) { - meta.suiteCallSource = suiteCallSource - } - plugin.startTest(name, meta) - }, + onTestStart: ( + name, + file, + callSource, + suiteName, + suiteCallSource, + attempt + ) => + plugin.startTest( + name, + buildStartTestMeta( + file, + callSource, + suiteName, + suiteCallSource, + attempt + ) + ), onTestEnd: (state) => { plugin.endTest(state === 'pending' ? 'skipped' : state) }, @@ -638,6 +659,7 @@ export const DevTools = { screencast?: ScreencastOptions headless?: boolean openUi?: boolean + captureAssertions?: boolean mode?: DevToolsMode traceFormat?: TraceFormat traceGranularity?: TraceGranularity diff --git a/packages/selenium-devtools/src/plugin-internals.ts b/packages/selenium-devtools/src/plugin-internals.ts index c769c4ad..a9672e50 100644 --- a/packages/selenium-devtools/src/plugin-internals.ts +++ b/packages/selenium-devtools/src/plugin-internals.ts @@ -67,6 +67,8 @@ export interface PluginInternals { // Per-spec trace tracking (populated at spec file boundaries). readonly specRanges: SpecRange[] readonly flushedSpecs: Set<string> + // In-flight per-test eager flushes (test granularity), awaited at finalize. + readonly traceFlushes: Promise<unknown>[] // Plugin-side delegates setFinalized(v: boolean): void diff --git a/packages/selenium-devtools/src/runnerHooks/cucumber.ts b/packages/selenium-devtools/src/runnerHooks/cucumber.ts index cffc6c76..86f8d3c2 100644 --- a/packages/selenium-devtools/src/runnerHooks/cucumber.ts +++ b/packages/selenium-devtools/src/runnerHooks/cucumber.ts @@ -1,6 +1,11 @@ import { createRequire } from 'node:module' import logger from '@wdio/logger' import { errorMessage } from '@wdio/devtools-core' +import type { + CucumberPickle, + CucumberPickleStep, + TestStatus +} from '@wdio/devtools-shared' import type { RunnerHookCallbacks } from '../types.js' const log = logger('@wdio/selenium-devtools:runnerHooks:cucumber') @@ -178,17 +183,6 @@ interface GherkinFeature { location?: { line?: number } children?: GherkinFeatureChild[] } -interface CucumberPickleStep { - text?: string - astNodeIds?: string[] - location?: { line?: number } -} -interface CucumberPickle { - name?: string - uri?: string - location?: { line?: number } - astNodeIds?: string[] -} interface CucumberTestCase { gherkinDocument?: { feature?: GherkinFeature } pickle?: CucumberPickle @@ -227,7 +221,7 @@ function populateGherkinIndex( } } -type ScenarioState = 'passed' | 'failed' | 'pending' +type ScenarioState = Extract<TestStatus, 'passed' | 'failed' | 'pending'> function mapCucumberStatus(status: string): ScenarioState | 'skipped' { const s = status.toUpperCase() diff --git a/packages/selenium-devtools/src/runnerHooks/mocha.ts b/packages/selenium-devtools/src/runnerHooks/mocha.ts index 5f2ad0f5..e7b8a616 100644 --- a/packages/selenium-devtools/src/runnerHooks/mocha.ts +++ b/packages/selenium-devtools/src/runnerHooks/mocha.ts @@ -87,7 +87,8 @@ function makeMochaBeforeEach( parentTitle, parentTitle ? resolveCallSource(test.file, parentTitle, 'suite') - : undefined + : undefined, + typeof test._currentRetry === 'number' ? test._currentRetry : undefined ) } } diff --git a/packages/selenium-devtools/src/session-lifecycle.ts b/packages/selenium-devtools/src/session-lifecycle.ts index d3551793..856d05ee 100644 --- a/packages/selenium-devtools/src/session-lifecycle.ts +++ b/packages/selenium-devtools/src/session-lifecycle.ts @@ -10,13 +10,17 @@ import logger from '@wdio/logger' import { + collectSuiteTestMetadata, errorMessage, finalizeScreencast, + finalizeTraceExport, + flushRangeLogged, + recordSliceBoundary as coreRecordSliceBoundary, recordSpecBoundary as coreRecordSpecBoundary, resolveAdapterOutputDir, - writeSpecTrace, - writeTraceZip, - type SpecRange + type SpecBoundaryContext, + type SpecRange, + type TraceExportContext } from '@wdio/devtools-core' import { TIMING } from './constants.js' import { SessionCapturer } from './session.js' @@ -32,10 +36,9 @@ import type { Metadata, ScreencastOptions, SeleniumDriverLike, - SuiteStats, - TestMetadataMap, TraceFormat, - TraceGranularity + TraceGranularity, + TraceRetentionPolicy } from './types.js' import type { TestManager } from './helpers/testManager.js' @@ -51,6 +54,7 @@ export interface SessionLifecycleCtx { mode?: DevToolsMode traceFormat?: TraceFormat traceGranularity?: TraceGranularity + tracePolicy?: TraceRetentionPolicy } readonly screencastOptions: ScreencastOptions readonly runner: string @@ -77,6 +81,9 @@ export interface SessionLifecycleCtx { // Per-spec trace tracking (populated at spec file boundaries). readonly specRanges: SpecRange[] readonly flushedSpecs: Set<string> + // In-flight per-test eager flushes (test granularity); awaited at finalize + // so the last test's write completes before the process exits. + readonly traceFlushes: Promise<unknown>[] setFinalized(v: boolean): void ensureBackendStarted(): Promise<void> @@ -247,7 +254,7 @@ export async function onSessionEnd(ctx: SessionLifecycleCtx): Promise<void> { ctx.testManager?.finalizeSession() ctx.testReporter?.updateSuites() - await maybeWriteTrace(ctx, capturerAtStart, testFilePathAtStart) + await writeTraceIfNeeded(ctx, capturerAtStart, testFilePathAtStart) logSessionSummary(ctx) ctx.sessionCapturer?.cleanup() @@ -283,101 +290,158 @@ export async function onSessionEnd(ctx: SessionLifecycleCtx): Promise<void> { } /** - * Record a spec-file boundary in the session lifecycle context. Called from - * the plugin's startTest / startScenario when `traceGranularity` is `'spec'`. - * Flushes the previous spec's trace if it hasn't been written yet. + * Assemble the framework-agnostic trace-export context from the selenium + * session state. `resolveOutputDir` writes per-spec traces next to the spec + * file and the session trace next to the run's first test file. Test metadata + * is recomputed from the suite tree so a boundary flush sees the current tree. */ -export function recordSpecBoundary( +export function buildTraceExportContext( ctx: SessionLifecycleCtx, - specFile: string + capturer: SessionCapturer, + sessionId: string, + testFilePath: string | undefined +): TraceExportContext { + const root = ctx.suiteManager?.getRootSuite() + return { + mode: ctx.options.mode, + policy: ctx.options.tracePolicy, + granularity: ctx.options.traceGranularity, + format: ctx.options.traceFormat, + capturer, + actionSnapshots: ctx.actionSnapshots, + sessionId, + testMetadata: collectSuiteTestMetadata(root ? [root] : []), + // TestStats.retries carries the per-test attempt (Mocha authoritative, + // other runners heuristic), so retry-aware policies can trust it. + attemptInfoAvailable: true, + ranges: ctx.specRanges, + flushed: ctx.flushedSpecs, + resolveOutputDir: (range) => + resolveAdapterOutputDir({ + testFilePath: range ? range.specFile : testFilePath + }), + awaitPending: [...ctx.snapshotCaptures, ...ctx.traceFlushes], + log: (level, msg) => log[level](msg) + } +} + +/** Narrow view of the lifecycle ctx that the core boundary recorder needs. */ +function boundaryContext( + ctx: SessionLifecycleCtx, + capturer: SessionCapturer +): SpecBoundaryContext { + return { + specRanges: ctx.specRanges, + flushedSpecs: ctx.flushedSpecs, + capturer, + actionSnapshots: ctx.actionSnapshots + } +} + +/** + * Record a trace-slice boundary for the active granularity, called from the + * plugin's startTest / startScenario. `spec` keys by spec file (flushing the + * previous spec lazily at the next boundary); `test` keys by the just-started + * marked test's uid so each test — and each retry, which selenium gives a + * distinct uid — becomes its own slice. No-op for `session`. + */ +export function recordTraceBoundary( + ctx: SessionLifecycleCtx, + specFile: string | undefined ): void { + if (ctx.options.traceGranularity === 'test') { + recordTestBoundary(ctx, specFile) + return + } + if (specFile) { + recordSpecBoundary(ctx, specFile) + } +} + +/** + * Record a spec-file boundary and lazily flush the previous spec's trace if it + * hasn't been written yet. `coreRecordSpecBoundary` no-ops for non-spec + * granularities, so this only records under `spec`. + */ +function recordSpecBoundary(ctx: SessionLifecycleCtx, specFile: string): void { if (!ctx.sessionCapturer) { return } const prevRange = coreRecordSpecBoundary( - { - specRanges: ctx.specRanges, - flushedSpecs: ctx.flushedSpecs, - capturer: ctx.sessionCapturer, - actionSnapshots: ctx.actionSnapshots - }, + boundaryContext(ctx, ctx.sessionCapturer), specFile, ctx.options.traceGranularity ) - if (prevRange) { - void flushOneSpecTrace(ctx, prevRange).catch((err) => - log.warn( - `Failed to flush trace for spec "${prevRange.specFile}": ${errorMessage(err)}` - ) - ) + const sessionId = ctx.sessionCapturer.metadata?.sessionId + if (!prevRange || !sessionId) { + return } + void flushRangeLogged( + buildTraceExportContext( + ctx, + ctx.sessionCapturer, + sessionId, + ctx.testFilePath + ), + prevRange + ) } -async function flushOneSpecTrace( +/** + * Record a per-test boundary keyed by the currently-active marked test's uid. + * The first test's startTest fires before the driver exists (no capturer yet), + * so flushPendingTestActions re-invokes this once capture is live; the + * same-uid guard keeps that replay from minting a spurious retry slice. + */ +function recordTestBoundary( ctx: SessionLifecycleCtx, - range: SpecRange, - nextRange?: SpecRange -): Promise<string | undefined> { - const capturer = ctx.sessionCapturer - if (!capturer || ctx.flushedSpecs.has(range.specFile)) { - return undefined - } - ctx.flushedSpecs.add(range.specFile) - - const sessionId = capturer.metadata?.sessionId - if (!sessionId) { - return undefined + specFile: string | undefined +): void { + const testUid = ctx.testManager?.getCurrentTest()?.uid + const file = specFile ?? ctx.testFilePath + if (!ctx.sessionCapturer || !testUid || !file) { + return } - - const tracePath = await writeSpecTrace({ - range, - nextRange, - capturer, - actionSnapshots: ctx.actionSnapshots, - sessionId, - outputDir: resolveAdapterOutputDir({ testFilePath: range.specFile }), - format: ctx.options.traceFormat, - testMetadata: collectTestMetadata(ctx.suiteManager) - }) - log.info(`Trace for spec "${range.specFile}" saved to ${tracePath}`) - return tracePath -} - -async function flushSpecTraces(ctx: SessionLifecycleCtx): Promise<void> { - for (const range of ctx.specRanges) { - if (!ctx.flushedSpecs.has(range.specFile)) { - await flushOneSpecTrace(ctx, range) - } + const lastRange = ctx.specRanges[ctx.specRanges.length - 1] + if (lastRange?.testUid === testUid) { + return } + coreRecordSliceBoundary( + boundaryContext(ctx, ctx.sessionCapturer), + 'test', + file, + testUid + ) } -function collectTestMetadata( - suiteManager: SuiteManager | undefined -): TestMetadataMap { - const metadata: TestMetadataMap = new Map() - const root = suiteManager?.getRootSuite() - if (!root) { - return metadata +/** + * Eager-flush the just-ended test's trace slice (test granularity), after + * endTest has finalized its state so collectSuiteTestMetadata sees the final + * outcome. flushRangeTrace dedupes by key, so finalizeTraceExport won't + * re-write it; the promise is tracked so finalize awaits the last write. + */ +export function flushCurrentTestTrace(ctx: SessionLifecycleCtx): void { + if (ctx.options.traceGranularity !== 'test' || !ctx.sessionCapturer) { + return } - const walk = (suite: SuiteStats) => { - for (const entry of suite.tests) { - if (typeof entry === 'string') { - continue - } - metadata.set(entry.uid, { - title: entry.fullTitle, - specFile: entry.file - }) - } - for (const child of suite.suites) { - walk(child) - } + const sessionId = ctx.sessionCapturer.metadata?.sessionId + const currentRange = ctx.specRanges[ctx.specRanges.length - 1] + if (!sessionId || currentRange?.testUid === undefined) { + return } - walk(root) - return metadata + const flush = flushRangeLogged( + buildTraceExportContext( + ctx, + ctx.sessionCapturer, + sessionId, + ctx.testFilePath + ), + currentRange + ) + ctx.traceFlushes.push(flush) } -async function maybeWriteTrace( +async function writeTraceIfNeeded( ctx: SessionLifecycleCtx, capturer: SessionCapturer | undefined, testFilePath: string | undefined @@ -386,30 +450,9 @@ async function maybeWriteTrace( if (ctx.options.mode !== 'trace' || !capturer || !sessionId) { return } - try { - if (ctx.snapshotCaptures.length) { - await Promise.allSettled(ctx.snapshotCaptures) - } - - if (ctx.options.traceGranularity === 'spec') { - await flushSpecTraces(ctx) - return - } - - const testMetadata = collectTestMetadata(ctx.suiteManager) - const tracePath = await writeTraceZip(capturer, { - outputDir: resolveAdapterOutputDir({ testFilePath }), - sessionId, - actionSnapshots: ctx.actionSnapshots.length - ? ctx.actionSnapshots - : undefined, - format: ctx.options.traceFormat, - testMetadata - }) - log.info(`Trace saved to ${tracePath}`) - } catch (err) { - log.warn(`trace write failed: ${errorMessage(err)}`) - } + await finalizeTraceExport( + buildTraceExportContext(ctx, capturer, sessionId, testFilePath) + ) } function logSessionSummary(ctx: SessionLifecycleCtx): void { diff --git a/packages/selenium-devtools/src/test-management.ts b/packages/selenium-devtools/src/test-management.ts index 842c0654..e62d17de 100644 --- a/packages/selenium-devtools/src/test-management.ts +++ b/packages/selenium-devtools/src/test-management.ts @@ -24,7 +24,7 @@ export type PendingTestAction = | { kind: 'start' name: string - meta: { file?: string; callSource?: string } + meta: { file?: string; callSource?: string; attempt?: number } suiteName?: string suiteCallSource?: string } @@ -54,6 +54,8 @@ export interface StartTestMeta { callSource?: string suiteName?: string suiteCallSource?: string + /** Authoritative per-test retry index when the runner exposes one (Mocha). */ + attempt?: number } export interface StartScenarioMeta { @@ -63,6 +65,53 @@ export interface StartScenarioMeta { featureCallSource?: string } +/** Assemble a StartTestMeta from a runner hook's positional callback args. */ +export function buildStartTestMeta( + file?: string, + callSource?: string, + suiteName?: string, + suiteCallSource?: string, + attempt?: number +): StartTestMeta { + const meta: StartTestMeta = {} + if (file) { + meta.file = file + } + if (callSource) { + meta.callSource = callSource + } + if (suiteName) { + meta.suiteName = suiteName + } + if (suiteCallSource) { + meta.suiteCallSource = suiteCallSource + } + if (typeof attempt === 'number') { + meta.attempt = attempt + } + return meta +} + +type ResolvedTestMeta = { file?: string; callSource?: string; attempt?: number } + +function resolveStartMeta( + meta: StartTestMeta, + file: string | undefined, + callSource: string | undefined +): ResolvedTestMeta { + const resolved: ResolvedTestMeta = {} + if (file) { + resolved.file = file + } + if (callSource && callSource !== 'unknown:0') { + resolved.callSource = callSource + } + if (typeof meta.attempt === 'number') { + resolved.attempt = meta.attempt + } + return resolved +} + export function startTest( ctx: TestManagementCtx, name: string, @@ -74,13 +123,7 @@ export function startTest( const stackInfo = getCallSourceFromStack() const file = meta.file || stackInfo.filePath const callSource = meta.callSource || stackInfo.callSource - const resolvedMeta: { file?: string; callSource?: string } = {} - if (file) { - resolvedMeta.file = file - } - if (callSource && callSource !== 'unknown:0') { - resolvedMeta.callSource = callSource - } + const resolvedMeta = resolveStartMeta(meta, file, callSource) if (!ctx.suiteManager || !ctx.testReporter) { ctx.pendingTestActions.push({ kind: 'start', diff --git a/packages/selenium-devtools/src/types.ts b/packages/selenium-devtools/src/types.ts index e0cbdbe8..af764d69 100644 --- a/packages/selenium-devtools/src/types.ts +++ b/packages/selenium-devtools/src/types.ts @@ -16,29 +16,17 @@ export { type TestStatus, type TestMetadataMap, type TraceFormat, - type TraceGranularity + type TraceGranularity, + type TraceRetentionPolicy } from '@wdio/devtools-shared' -export interface DevToolsOptions { - port?: number - hostname?: string +export interface DevToolsOptions extends BaseDevToolsOptions { /** Open a Chrome window pointing at the UI. Default true. */ openUi?: boolean - /** `live` (default) launches the DevTools UI; `trace` skips it. Overrides `openUi`. */ - mode?: DevToolsMode - /** Trace output layout — `zip` (default) writes a single archive, - * `ndjson-directory` unpacks into `trace-<id>/`. Only applies in trace mode. */ - traceFormat?: TraceFormat - /** Trace output granularity — `session` (default) writes one trace per - * worker session; `spec` writes one trace per spec file. Only applies in - * trace mode. */ - traceGranularity?: TraceGranularity /** Capture screenshots after each command. Default true. */ captureScreenshots?: boolean /** Command template for per-test rerun. {{testName}} is substituted. */ rerunCommand?: string - /** Per-session screencast recording. Disabled by default. */ - screencast?: ScreencastOptions /** * Force the *test* browser into headless mode by injecting --headless=new * into Chrome capabilities. The dashboard window (auto-opened by openUi) @@ -49,12 +37,7 @@ export interface DevToolsOptions { // ScreencastFrame, ScreencastOptions hoisted to @wdio/devtools-shared; re-exported // here for backwards compatibility with existing selenium-internal imports. -import type { - DevToolsMode, - ScreencastOptions, - TraceFormat, - TraceGranularity -} from '@wdio/devtools-shared' +import type { BaseDevToolsOptions, TestStatus } from '@wdio/devtools-shared' export type { ScreencastFrame, ScreencastOptions } from '@wdio/devtools-shared' /** @@ -126,24 +109,16 @@ export interface ElementOriginals { getTagName?: (element: unknown) => Promise<string> } -// ─── bidi ─────────────────────────────────────────────────────────────────── - -import type { ConsoleLog, NetworkRequest } from '@wdio/devtools-shared' - -export interface BidiHandlerSinks { - pushConsoleLog: (entry: ConsoleLog) => void - pushNetworkRequest: (entry: NetworkRequest) => void - replaceNetworkRequest: (id: string, entry: NetworkRequest) => void -} - // ─── runnerHooks ──────────────────────────────────────────────────────────── export interface MochaTestCtx { title?: string file?: string - state?: 'passed' | 'failed' | 'pending' | 'running' | 'skipped' + state?: TestStatus duration?: number parent?: { title?: string } + /** Mocha's authoritative retry index — 0 on the first run, +1 per retry. */ + _currentRetry?: number } export interface RunnerHookCallbacks { @@ -152,9 +127,12 @@ export interface RunnerHookCallbacks { file?: string, callSource?: string, suiteName?: string, - suiteCallSource?: string + suiteCallSource?: string, + // Authoritative per-test retry index when the runner exposes one (Mocha's + // `_currentRetry`); undefined leaves the heuristic tracker to decide. + attempt?: number ) => void - onTestEnd: (state: 'passed' | 'failed' | 'skipped' | 'pending') => void + onTestEnd: (state: Exclude<TestStatus, 'running'>) => void // Cucumber-only: scenario boundary creates a sub-suite under the feature // rootSuite; subsequent onTestStart/onTestEnd attach as Gherkin steps inside. onScenarioStart?: ( @@ -164,7 +142,7 @@ export interface RunnerHookCallbacks { featureName?: string, featureCallSource?: string ) => void - onScenarioEnd?: (state: 'passed' | 'failed' | 'skipped' | 'pending') => void + onScenarioEnd?: (state: Exclude<TestStatus, 'running'>) => void // Fires from the runner's after-all hook so the dashboard suite header // updates without waiting for process exit. onTestRunComplete?: (summary: { diff --git a/packages/selenium-devtools/tests/attempt-capture.test.ts b/packages/selenium-devtools/tests/attempt-capture.test.ts new file mode 100644 index 00000000..2f2840ff --- /dev/null +++ b/packages/selenium-devtools/tests/attempt-capture.test.ts @@ -0,0 +1,96 @@ +import { describe, it, expect, vi } from 'vitest' +import { collectSuiteTestMetadata } from '@wdio/devtools-core' + +import { resetSignatureCounters } from '../src/helpers/utils.js' +import { TestManager } from '../src/helpers/testManager.js' +import { SuiteManager } from '../src/helpers/suiteManager.js' +import { TestReporter } from '../src/reporter.js' +import { SessionCapturer } from '../src/session.js' +import { + buildTraceExportContext, + type SessionLifecycleCtx +} from '../src/session-lifecycle.js' + +function makeManager() { + resetSignatureCounters() + const reporter = new TestReporter(vi.fn()) + const suiteManager = new SuiteManager(reporter) + const rootSuite = suiteManager.getOrCreateRootSuite('/spec.ts', 'Suite') + const mgr = new TestManager(rootSuite, reporter, suiteManager) + return { mgr, suiteManager, rootSuite } +} + +describe('retry/attempt capture', () => { + it('re-entering the start hook for the same test increments retries (heuristic)', () => { + const { mgr } = makeManager() + + const first = mgr.startMarkedTest('flaky') + expect(first.retries).toBe(0) + + mgr.endCurrent('failed') + const retry = mgr.startMarkedTest('flaky') + expect(retry.retries).toBe(1) + + mgr.endCurrent('failed') + const retry2 = mgr.startMarkedTest('flaky') + expect(retry2.retries).toBe(2) + }) + + it('prefers the authoritative Mocha attempt over the heuristic', () => { + const { mgr } = makeManager() + + // First start: heuristic would be 0, authoritative says 2 → 2 wins. + const t = mgr.startMarkedTest('login', { attempt: 2 }) + expect(t.retries).toBe(2) + + mgr.endCurrent('failed') + // Re-entry: heuristic would be 1, authoritative says 5 → 5 wins. + const retry = mgr.startMarkedTest('login', { attempt: 5 }) + expect(retry.retries).toBe(5) + }) + + it('flows retries into metadata attempt and the trace ctx flags attempt info', () => { + const { mgr, suiteManager, rootSuite } = makeManager() + + mgr.startMarkedTest('flaky') + mgr.endCurrent('failed') + const retry = mgr.startMarkedTest('flaky') + mgr.endCurrent('passed') + expect(retry.retries).toBe(1) + + const metadata = collectSuiteTestMetadata([rootSuite]) + expect(metadata.get(retry.uid)?.attempt).toBe(1) + + const capturer = new SessionCapturer() + try { + // Minimal structural ctx: buildTraceExportContext only reads options, + // suiteManager and the trace accumulators, so we cast a partial to the + // full lifecycle interface rather than standing up the whole plugin. + const ctx = { + options: { + mode: 'trace', + tracePolicy: 'retain-on-failure-and-retries', + traceGranularity: 'session', + traceFormat: 'zip' + }, + suiteManager, + actionSnapshots: [], + specRanges: [], + flushedSpecs: new Set<string>(), + traceFlushes: [], + snapshotCaptures: [] + } as unknown as SessionLifecycleCtx + + const traceCtx = buildTraceExportContext( + ctx, + capturer, + 'sess-1', + '/spec.ts' + ) + expect(traceCtx.attemptInfoAvailable).toBe(true) + expect(traceCtx.testMetadata.get(retry.uid)?.attempt).toBe(1) + } finally { + capturer.cleanup() + } + }) +}) diff --git a/packages/selenium-devtools/tests/trace-granularity.test.ts b/packages/selenium-devtools/tests/trace-granularity.test.ts new file mode 100644 index 00000000..8f133a24 --- /dev/null +++ b/packages/selenium-devtools/tests/trace-granularity.test.ts @@ -0,0 +1,191 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' + +import { resetSignatureCounters } from '../src/helpers/utils.js' +import { TestManager } from '../src/helpers/testManager.js' +import { SuiteManager } from '../src/helpers/suiteManager.js' +import { TestReporter } from '../src/reporter.js' +import { SessionCapturer } from '../src/session.js' +import { + flushCurrentTestTrace, + recordTraceBoundary, + type SessionLifecycleCtx +} from '../src/session-lifecycle.js' +import type { TraceGranularity } from '../src/types.js' + +const SESSION_ID = 'sess-abcd1234ef' + +const capturers: SessionCapturer[] = [] +const tmpDirs: string[] = [] + +afterEach(() => { + while (capturers.length) { + capturers.pop()!.cleanup() + } + while (tmpDirs.length) { + fs.rmSync(tmpDirs.pop()!, { recursive: true, force: true }) + } +}) + +function makeTmpSpec(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sel-trace-gran-')) + tmpDirs.push(dir) + return path.join(dir, 'login.spec.ts') +} + +function makeCtx( + granularity: TraceGranularity, + specFile: string, + withSessionId = false +) { + resetSignatureCounters() + const reporter = new TestReporter(vi.fn()) + const suiteManager = new SuiteManager(reporter) + const rootSuite = suiteManager.getOrCreateRootSuite(specFile, 'Suite') + const testManager = new TestManager(rootSuite, reporter, suiteManager) + const capturer = new SessionCapturer() + capturers.push(capturer) + if (withSessionId) { + // Only sessionId matters to the flush path; the rest of Metadata is + // filled with writeTraceZip's defaults, so a partial cast is safe here. + capturer.metadata = { sessionId: SESSION_ID } as SessionCapturer['metadata'] + } + + // Minimal structural ctx: the boundary/flush helpers read only the trace + // accumulators, options, capturer, test/suite managers and testFilePath, so + // we cast a partial to the full lifecycle interface. + const ctx = { + options: { + mode: 'trace', + tracePolicy: 'on', + traceGranularity: granularity, + traceFormat: 'zip' + }, + sessionCapturer: capturer, + testManager, + suiteManager, + testFilePath: specFile, + actionSnapshots: [], + snapshotCaptures: [], + specRanges: [], + flushedSpecs: new Set<string>(), + traceFlushes: [] + } as unknown as SessionLifecycleCtx + + return { ctx, capturer, testManager } +} + +describe('trace granularity: test', () => { + it('records one per-test slice keyed by the started test uid', () => { + const spec = makeTmpSpec() + const { ctx, testManager } = makeCtx('test', spec) + + const test = testManager.startMarkedTest('logs in') + recordTraceBoundary(ctx, spec) + + expect(ctx.specRanges).toHaveLength(1) + expect(ctx.specRanges[0].testUid).toBe(test.uid) + expect(ctx.specRanges[0].key).toBe(test.uid) + expect(ctx.specRanges[0].specFile).toBe(spec) + }) + + it('re-recording the same active test is idempotent (no spurious slice)', () => { + const spec = makeTmpSpec() + const { ctx, testManager } = makeCtx('test', spec) + + testManager.startMarkedTest('logs in') + recordTraceBoundary(ctx, spec) + // Mirrors the buffered-first-test replay: startTest recorded nothing (no + // capturer yet), flushPendingTestActions re-invokes for the same test. + recordTraceBoundary(ctx, spec) + + expect(ctx.specRanges).toHaveLength(1) + }) + + it('a retry gets its own distinct slice (selenium gives each attempt a uid)', () => { + const spec = makeTmpSpec() + const { ctx, testManager } = makeCtx('test', spec) + + const first = testManager.startMarkedTest('flaky') + recordTraceBoundary(ctx, spec) + testManager.endCurrent('failed') + + const retry = testManager.startMarkedTest('flaky') + recordTraceBoundary(ctx, spec) + testManager.endCurrent('passed') + + expect(first.uid).not.toBe(retry.uid) + expect(ctx.specRanges).toHaveLength(2) + expect(ctx.specRanges.map((r) => r.key)).toEqual([first.uid, retry.uid]) + }) + + it('eager-flushes each test to its own artifact at test end', async () => { + const spec = makeTmpSpec() + const outDir = path.dirname(spec) + const { ctx, testManager } = makeCtx('test', spec, true) + + testManager.startMarkedTest('logs in') + recordTraceBoundary(ctx, spec) + testManager.endCurrent('passed') + flushCurrentTestTrace(ctx) + + testManager.startMarkedTest('logs out') + recordTraceBoundary(ctx, spec) + testManager.endCurrent('passed') + flushCurrentTestTrace(ctx) + + expect(ctx.traceFlushes).toHaveLength(2) + // Both slice keys are recorded as flushed synchronously, so end-of-run + // finalizeTraceExport dedupes and won't re-write them. + expect(ctx.flushedSpecs.size).toBe(2) + await Promise.all(ctx.traceFlushes) + + // Per-test slices land under test-results/<spec--title-browser>/trace.zip, + // so recurse to find them rather than scanning the flat dir. + const zips = fs + .readdirSync(outDir, { recursive: true }) + .map(String) + .filter((f) => f.endsWith('.zip')) + expect(zips).toHaveLength(2) + expect(zips.every((f) => path.basename(f) === 'trace.zip')).toBe(true) + expect(zips.every((f) => f.includes('test-results'))).toBe(true) + }) + + it('flushCurrentTestTrace is a no-op with no recorded range', () => { + const spec = makeTmpSpec() + const { ctx } = makeCtx('test', spec, true) + flushCurrentTestTrace(ctx) + expect(ctx.traceFlushes).toHaveLength(0) + expect(ctx.flushedSpecs.size).toBe(0) + }) +}) + +describe('trace granularity: spec/session unchanged', () => { + it('spec granularity still records one slice per spec file, keyed by file', () => { + const specA = makeTmpSpec() + const specB = makeTmpSpec() + const { ctx } = makeCtx('spec', specA) + + recordTraceBoundary(ctx, specA) + recordTraceBoundary(ctx, specA) // same file → shares the slice + recordTraceBoundary(ctx, specB) + + expect(ctx.specRanges.map((r) => r.key)).toEqual([specA, specB]) + expect(ctx.specRanges.every((r) => r.testUid === undefined)).toBe(true) + }) + + it('session granularity records no slices and never eager-flushes', () => { + const spec = makeTmpSpec() + const { ctx, testManager } = makeCtx('session', spec, true) + + testManager.startMarkedTest('logs in') + recordTraceBoundary(ctx, spec) + testManager.endCurrent('passed') + flushCurrentTestTrace(ctx) + + expect(ctx.specRanges).toHaveLength(0) + expect(ctx.traceFlushes).toHaveLength(0) + }) +}) diff --git a/packages/service/src/action-snapshot.ts b/packages/service/src/action-snapshot.ts index 3311587a..f8dc7fe9 100644 --- a/packages/service/src/action-snapshot.ts +++ b/packages/service/src/action-snapshot.ts @@ -8,9 +8,13 @@ // library. No external input reaches new Function() — the lint flag here is // a false positive given the closed input set. -import { captureActionSnapshot as coreCapture } from '@wdio/devtools-core' +import { + captureActionSnapshot as coreCapture, + mapCommandToAction +} from '@wdio/devtools-core' import type { ActionSnapshot } from '@wdio/devtools-shared' import { isNativeMobile, mobilePlatform } from './mobile.js' +import { INTERNAL_COMMANDS } from './constants.js' function reviveScript(src: string): () => unknown { // `src` from core/element-scripts is already a self-invoking IIFE @@ -19,6 +23,82 @@ function reviveScript(src: string): () => unknown { return new Function(`return (${src})`) as () => unknown } +/** + * After a mapped action, wait for the resulting page to settle before the + * post-action screenshot. readyState alone is unreliable — right after a click + * the OLD document still reports 'complete'. beforeCommand tags the document; + * if the tag is gone the action navigated, so we wait for the NEW document to + * finish loading AND render content before the destination is screenshotted. + */ +export async function waitForActionResult( + browser: WebdriverIO.Browser +): Promise<void> { + const navigated = await browser + .execute( + () => !(window as Window & { __wdioSnapMark?: boolean }).__wdioSnapMark + ) + .catch(() => true) + if (!navigated) { + return + } + await browser + .waitUntil( + async () => + (await browser + .execute( + () => + document.readyState === 'complete' && + !!document.body && + document.body.childElementCount > 0 + ) + .catch(() => false)) === true, + { timeout: 8000, interval: 150 } + ) + .catch(() => undefined) + // Headless renderers can return a blank shot right after load; let it paint. + await browser.pause(250).catch(() => undefined) +} + +/** Post-action capture: settle the resulting page, screenshot it, and push the + * snapshot stamped at the latest logged action. No-op for internal/non-mapped + * commands. Skipped by the caller outside trace mode. */ +export async function captureActionResult( + browser: WebdriverIO.Browser, + command: string, + actionSnapshots: ActionSnapshot[], + stampTimestamp: () => number +): Promise<void> { + if (!mapCommandToAction(command) || INTERNAL_COMMANDS.includes(command)) { + return + } + if (!isNativeMobile(browser)) { + await waitForActionResult(browser) + } + const snap = await captureActionSnapshot(browser, command) + if (snap) { + snap.timestamp = stampTimestamp() + actionSnapshots.push(snap) + } +} + +/** Capture a DOM snapshot for a synthesized action row (e.g. an `expect.*` + * assertion) and push it stamped at the row's OWN timestamp — the trace + * player's Snapshot tab claims it by timestamp the same way it claims a + * regular command's post-action snapshot (see FrameSnapshotIndex.claimAfter). + * Mirrors the tail of `captureActionResult` for a command with no page-settle. */ +export async function pushActionSnapshotAt( + browser: WebdriverIO.Browser, + command: string, + timestamp: number, + actionSnapshots: ActionSnapshot[] +): Promise<void> { + const snap = await captureActionSnapshot(browser, command) + if (snap) { + snap.timestamp = timestamp + actionSnapshots.push(snap) + } +} + export function captureActionSnapshot( browser: WebdriverIO.Browser, command: string diff --git a/packages/service/src/assert-capture.ts b/packages/service/src/assert-capture.ts new file mode 100644 index 00000000..49725ec8 --- /dev/null +++ b/packages/service/src/assert-capture.ts @@ -0,0 +1,164 @@ +// Assertion capture wiring for the WDIO worker: node:assert patching plus +// marking the failing action when an expect-webdriverio matcher throws. + +import logger from '@wdio/logger' +import { + capturedAssertToCommandLog, + matcherAssertionToCommandLog, + patchNodeAssert, + stripAnsi +} from '@wdio/devtools-core' +import type { CommandLog, SerializedError } from '@wdio/devtools-shared' +import { parse } from 'stack-trace' +import { + resolveCallSourceFromFrame, + resolveFilePathFromFrame +} from './call-source.js' +import { isUserSpecFile } from './utils.js' +import type { SessionCapturer } from './session.js' + +const log = logger('@wdio/devtools-service:assert-capture') + +/** + * Capture the user's `expect()` call site from the SYNCHRONOUS stack at matcher + * entry (call from `beforeAssertion`). The matcher then runs async, so this is + * the only point a user frame is still on the stack — reading it in + * `afterAssertion` resolves to the service bundle instead. Mirrors + * `beforeCommand`'s resolver (`parse(new Error()).reverse()` → first user-spec + * frame → `resolveCallSourceFromFrame`) so assertion rows share regular + * commands' Source-tab behaviour. Also loads that file's source via + * `captureSource` so the tab renders. Returns `undefined` when no user frame is + * present (row falls back to no callSource, exactly as before this fix). + */ +export function resolveAssertionCallSource( + captureSource: (filePath: string) => void +): string | undefined { + Error.stackTraceLimit = 20 + const frame = parse(new Error('')) + .reverse() + .find((f) => isUserSpecFile(f.getFileName())) + if (!frame) { + return undefined + } + const filePath = resolveFilePathFromFrame(frame) + if (filePath) { + captureSource(filePath) + } + return resolveCallSourceFromFrame(frame) +} + +/** + * Patch node:assert so every tracked assertion lands in the session capturer + * as a command. Getters are read at capture time — the capturer instance is + * replaced in `before()` and the test UID changes per test. + */ +export function wireAssertCapture( + getCapturer: () => SessionCapturer, + getTestUid: () => string | undefined +): void { + patchNodeAssert( + (cmd) => + getCapturer().captureAssertCommand( + capturedAssertToCommandLog(cmd, getTestUid()) + ), + (level, message) => log[level](message) + ) +} + +/** + * Normalize a framework failure into a SerializedError. Cucumber hands a plain + * message string (@wdio/cucumber-framework getResultObject → world.result.message); + * Mocha/Jasmine hand an Error object. Returns undefined when there's nothing to + * show, or when the failure was already captured as its own command: a + * node:assert AssertionError (via the patcher) or an expect-webdriverio matcher + * error (via afterAssertion — it carries `matcherResult`). Skipping those keeps + * `failLastAction` from double-marking a passing command. ANSI is stripped. + */ +export function toCommandError(error: unknown): SerializedError | undefined { + if (!error) { + return undefined + } + if (typeof error === 'string') { + const message = stripAnsi(error).trim() + return message ? { name: 'Error', message } : undefined + } + if (typeof error !== 'object') { + return undefined + } + const err = error as { + name?: string + message?: string + stack?: string + matcherResult?: unknown + } + if (err.name === 'AssertionError' || err.matcherResult !== undefined) { + return undefined + } + return { + name: err.name || 'Error', + message: stripAnsi(err.message || String(error)), + ...(err.stack ? { stack: stripAnsi(err.stack) } : {}) + } +} + +/** + * Mark the action that was running when an expect-webdriverio matcher failed + * (the assertion isn't its own command, so its query carries the error). No-op + * when assertion capture is disabled. Mocha calls from afterTest, Cucumber from + * afterStep. + */ +export function captureExpectFailure( + capturer: SessionCapturer, + testUid: string | undefined, + error: unknown, + enabled: boolean +): void { + if (!enabled) { + return + } + const commandError = toCommandError(error) + if (commandError) { + capturer.failLastAction(testUid, commandError) + } +} + +/** The subset of expect-webdriverio's afterAssertion hook params we read to + * turn a matcher call into a trace command. The matcher passes `{ pass }` at + * runtime, but @wdio/types declares `{ result }`, so we accept and read both. */ +export interface ExpectAssertion { + matcherName: string + expectedValue?: unknown + result: { pass?: boolean; result?: boolean; message?: () => string } +} + +/** + * Adapt expect-webdriverio's afterAssertion params to the shared matcher + * converter. Framework-specific extraction only (matcher name, expectedValue → + * args, the runtime `pass` vs typed `result` flag); the actual CommandLog + * shaping lives once in core's `matcherAssertionToCommandLog`. `callSource` is + * the user's `expect()` call site captured in `beforeAssertion` (the matcher + * runs async, so afterAssertion's own stack no longer holds a user frame) — it + * makes the row's Source tab point at the spec, not the service bundle. + */ +export function expectAssertionToCommandLog( + params: ExpectAssertion, + testUid: string | undefined, + callSource?: string +): CommandLog { + const { matcherName, expectedValue, result } = params + return matcherAssertionToCommandLog( + { + method: matcherName, + args: + expectedValue === undefined + ? [] + : Array.isArray(expectedValue) + ? expectedValue + : [expectedValue], + passed: result.pass ?? result.result ?? false, + message: result.message, + callSource + }, + testUid + ) +} diff --git a/packages/service/src/call-source.ts b/packages/service/src/call-source.ts new file mode 100644 index 00000000..53a6fc80 --- /dev/null +++ b/packages/service/src/call-source.ts @@ -0,0 +1,35 @@ +import type { parse } from 'stack-trace' + +type StackFrame = ReturnType<typeof parse>[number] + +/** Absolute file path from a parsed stack frame; strips file:// and query. */ +export function resolveFilePathFromFrame( + frame: StackFrame +): string | undefined { + const rawFile = frame.getFileName() ?? undefined + let absPath = rawFile + if (rawFile?.startsWith('file://')) { + try { + absPath = decodeURIComponent(new URL(rawFile).pathname) + } catch { + absPath = rawFile + } + } + if (absPath?.includes('?')) { + absPath = absPath.split('?')[0] + } + return absPath +} + +/** `<file>:<line>:<column>` from a parsed stack frame; strips file:// and query. */ +export function resolveCallSourceFromFrame( + frame: StackFrame +): string | undefined { + const absPath = resolveFilePathFromFrame(frame) + if (absPath === undefined) { + return undefined + } + const line = frame.getLineNumber() ?? undefined + const column = frame.getColumnNumber() ?? undefined + return `${absPath}:${line ?? 0}:${column ?? 0}` +} diff --git a/packages/service/src/constants.ts b/packages/service/src/constants.ts index 46e38878..a30f6e3a 100644 --- a/packages/service/src/constants.ts +++ b/packages/service/src/constants.ts @@ -43,7 +43,6 @@ export const INTERNAL_COMMANDS = [ 'emit', 'browsingContextLocateNodes', 'browsingContextNavigate', - 'waitUntil', 'getTitle', 'getUrl', 'getWindowSize', @@ -51,7 +50,6 @@ export const INTERNAL_COMMANDS = [ 'deleteSession', 'findElementFromShadowRoot', 'findElementsFromShadowRoot', - 'waitForExist', 'browsingContextGetTree', 'scriptCallFunction', 'getElement', diff --git a/packages/service/src/index.ts b/packages/service/src/index.ts index 389a9f97..411d75d7 100644 --- a/packages/service/src/index.ts +++ b/packages/service/src/index.ts @@ -1,18 +1,42 @@ /// <reference types="../../script/types.d.ts" /> import logger from '@wdio/logger' import { - deterministicUid, errorMessage, finalizeScreencast, + finalizeTraceExport, mapCommandToAction, - recordSpecBoundary, + recordSliceBoundary, resolveAdapterOutputDir, + TestAttemptTracker, + tracePolicyModeWarning, type SpecRange, - writeSpecTrace, - writeTraceZip + type TraceExportContext } from '@wdio/devtools-core' -import { captureActionSnapshot } from './action-snapshot.js' -import { dedupeSnapshotsByTimestamp } from './snapshot-dedupe.js' +import { + captureExpectFailure, + expectAssertionToCommandLog, + resolveAssertionCallSource, + wireAssertCapture, + type ExpectAssertion +} from './assert-capture.js' +import { + cucumberScenarioUid, + resolveTestAttempt, + stampTestState, + testMetadataUid, + type TestOutcomeResult +} from './test-metadata.js' +import { resolveCallSourceFromFrame } from './call-source.js' +import { flushPrevSlice, flushTestSlice } from './trace-slices.js' +import { + captureActionResult, + captureActionSnapshot, + pushActionSnapshotAt +} from './action-snapshot.js' +import { + dedupeSnapshotsByTimestamp, + upsertRichestSnapshot +} from './snapshot-dedupe.js' import type { ActionSnapshot, TestMetadataMap } from '@wdio/devtools-shared' import { SevereServiceError } from 'webdriverio' import type { Services, Capabilities, Options, Reporters } from '@wdio/types' @@ -59,6 +83,13 @@ export default class DevToolsHookService implements Services.ServiceInstance { constructor(serviceOptions: ServiceOptions = {}) { this.#options = serviceOptions + const policyWarning = tracePolicyModeWarning( + serviceOptions.tracePolicy, + serviceOptions.mode + ) + if (policyWarning) { + log.warn(policyWarning) + } if (serviceOptions.mode === 'trace' && serviceOptions.screencast?.enabled) { log.warn('trace mode: ignoring screencast option (live-mode feature)') this.#screencastOptions = undefined @@ -73,19 +104,42 @@ export default class DevToolsHookService implements Services.ServiceInstance { */ #commandStack: CommandFrame[] = [] + /** Depth of the current expect-webdriverio matcher evaluation. While > 0, the + * matcher's internal WebDriver commands (getText/isExisting polling) are + * suppressed so only the `expect.<matcher>` assertion row is captured — + * matching Playwright and the Nightwatch native-assert behaviour. */ + #assertionDepth = 0 + + /** Matcher start timestamps (stack, paired with #assertionDepth) so a captured + * assertion spans [matcher start → end] — its poll duration — instead of + * collapsing to a zero-width point at completion, which left the screencast + * tracking the preceding command during the poll. */ + #assertionStartTimes: number[] = [] + + /** User `expect()` call sites (stack, paired with #assertionStartTimes), + * captured in beforeAssertion while a user frame is still synchronous. The + * matcher runs async, so afterAssertion's own stack resolves to the service + * bundle — this parallel stack lets each row keep its spec-file callSource. */ + #assertionCallSources: (string | undefined)[] = [] + /** Current test UID, set in beforeTest(), used by afterCommand() to tag commands. */ #currentTestUid?: string /** Map of testUid → metadata for trace group events and per-spec partitioning. */ #testMetadata: TestMetadataMap = new Map() + /** Per-test attempt counter. specFileRetries spawns a fresh worker (hence a + * fresh instance) per retry, so this only reflects same-process retries + * (Mocha this.retries(n)); cross-worker attempts rely on the WDIO result. */ + #attemptTracker = new TestAttemptTracker() + /** Index ranges into the session capturer's flat arrays, one per spec file. */ #specRanges: SpecRange[] = [] /** Set of spec files already flushed to disk. */ #flushedSpecs = new Set<string>() - /** Build the boundary context for recordSpecBoundary — the same shape is + /** Build the boundary context for recordSliceBoundary — the same shape is * needed in both beforeTest and beforeScenario. */ get #boundaryContext() { return { @@ -96,16 +150,65 @@ export default class DevToolsHookService implements Services.ServiceInstance { } } - /** Fire-and-forget flush of a previous spec's trace. The error log is - * inline so the spec-file reference stays precise. */ - #fireAndForgetFlush(prevRange: SpecRange): void { - void this.#flushSpecTrace(prevRange).catch((err) => - log.warn( - `Failed to flush trace for spec "${prevRange.specFile}": ${errorMessage(err)}` - ) + /** Record a trace-slice boundary. `spec` slices per file; `test` per test + * (retries keyed per attempt by core); `session` records nothing. The + * previous-slice flush fires for `spec`; `test` slices eager-flush at their + * own test end (see #eagerFlushTestSlice) so this is only a missed-slice net. */ + #recordBoundary(specFile: string | undefined, testUid?: string): void { + if (!specFile) { + return + } + const prevRange = recordSliceBoundary( + this.#boundaryContext, + this.#options.traceGranularity, + specFile, + testUid + ) + if (prevRange && this.#browser) { + flushPrevSlice(this.#traceContext(this.#browser), prevRange) + } + } + + /** Eager per-test flush at test end (test granularity only), run after the + * outcome is stamped so this attempt's metadata is written before a retry + * overwrites it; the end-of-run finalizer then dedupes it via the key set. */ + async #eagerFlushTestSlice(testUid: string): Promise<void> { + if ( + this.#options.traceGranularity !== 'test' || + this.#options.mode !== 'trace' || + !this.#browser + ) { + return + } + await flushTestSlice( + this.#traceContext(this.#browser), + this.#specRanges, + testUid ) } + /** Assemble the framework-agnostic trace-export context from this service's + * state. Output dir ignores the spec range — WDIO writes next to config. */ + #traceContext(browser: WebdriverIO.Browser): TraceExportContext { + return { + mode: this.#options.mode, + policy: this.#options.tracePolicy, + granularity: this.#options.traceGranularity, + format: this.#options.traceFormat, + capturer: this.#sessionCapturer, + actionSnapshots: this.#actionSnapshots, + sessionId: browser.sessionId, + capabilities: browser.capabilities, + testMetadata: this.#testMetadata, + attemptInfoAvailable: true, + ranges: this.#specRanges, + flushed: this.#flushedSpecs, + resolveOutputDir: () => this.#outputDir, + prepareSnapshots: dedupeSnapshotsByTimestamp, + log: (level, msg) => log[level](msg) + } + } + /** * allows to define the type of data being captured to hint the * devtools app which data to expect @@ -132,6 +235,13 @@ export default class DevToolsHookService implements Services.ServiceInstance { wdioCaps['wdio:devtoolsOptions'] ) + if (this.#options.captureAssertions !== false) { + wireAssertCapture( + () => this.#sessionCapturer, + () => this.#currentTestUid + ) + } + /** * Block until injection completes BEFORE any test commands. * Skip on native mobile — Appium sessions don't support WebDriver BiDi @@ -219,34 +329,31 @@ export default class DevToolsHookService implements Services.ServiceInstance { } } - /** - * Hook for Cucumber framework. - * Detects feature-file boundaries for per-spec tracing and tags commands - * with a stable testUid so tracingGroup spans render in the trace output. - * WDIO passes the Cucumber World as the first argument. - */ - beforeScenario(world?: { pickle?: { uri?: string; name?: string } }) { + /** Cucumber hook: records feature-file boundaries and tags commands with a stable testUid. */ + beforeScenario(world?: { + pickle?: { uri?: string; name?: string; astNodeIds?: readonly string[] } + }) { this.resetStack() const featureFile = world?.pickle?.uri const scenarioName = world?.pickle?.name + // Derived before recording the boundary so `test` granularity keys the + // slice on the same uid the metadata map uses. + const uid = + featureFile && scenarioName + ? cucumberScenarioUid( + featureFile, + scenarioName, + world?.pickle?.astNodeIds + ) + : undefined - // ── Per-spec boundary detection (Cucumber) ── - if (featureFile) { - const prevRange = recordSpecBoundary( - this.#boundaryContext, - featureFile, - this.#options.traceGranularity - ) - if (prevRange) { - this.#fireAndForgetFlush(prevRange) - } - } + this.#recordBoundary(featureFile, uid) // ── Test identity for command tagging ── - if (featureFile && scenarioName) { - const uid = deterministicUid(featureFile, scenarioName) + if (uid && scenarioName && featureFile) { this.#currentTestUid = uid + this.#attemptTracker.recordStart(uid) this.#testMetadata.set(uid, { title: scenarioName, specFile: featureFile @@ -254,56 +361,160 @@ export default class DevToolsHookService implements Services.ServiceInstance { } } - /** - * Hook for Mocha/Jasmine frameworks. - * It does the exact same thing as beforeScenario. - */ + /** Mocha/Jasmine hook: the beforeScenario equivalent for file-based specs. */ beforeTest(test?: { file?: string; title?: string; fullTitle?: string }) { this.resetStack() - const newSpec = test?.file - - // ── Per-spec boundary detection ── - // Only tracked when traceGranularity is 'spec'. Records array index - // ranges so #flushSpecTrace can slice the accumulated data per spec. - if (newSpec) { - const prevRange = recordSpecBoundary( - this.#boundaryContext, - newSpec, - this.#options.traceGranularity - ) - if (prevRange) { - this.#fireAndForgetFlush(prevRange) - } - } - // Track test identity for command tagging. Generate a stable UID // from file + title so commands can be partitioned across reruns. // WDIO's Test type always provides `fullTitle`; `title` is a - // fallback for non-WDIO frameworks. + // fallback for non-WDIO frameworks. Derived before the boundary so + // `test` granularity keys the slice on the metadata-map uid. const testTitle = test?.fullTitle || test?.title - if (test?.file && testTitle) { - const uid = deterministicUid(test.file, testTitle) + const uid = testTitle ? testMetadataUid(test?.file, testTitle) : undefined + + this.#recordBoundary(test?.file, uid) + + if (uid && testTitle) { this.#currentTestUid = uid + this.#attemptTracker.recordStart(uid) this.#testMetadata.set(uid, { title: testTitle, - specFile: test.file - }) - } else if (testTitle) { - this.#currentTestUid = testTitle - this.#testMetadata.set(testTitle, { - title: testTitle, - specFile: test.file ?? '' + specFile: test?.file ?? '' }) } } - async afterScenario() { + // afterStep fires right after each step, so the failing assertion lands next + // to the step's actions rather than after reloadSession at scenario end. + afterStep( + _step?: unknown, + _scenario?: unknown, + result?: { error?: unknown } + ) { + this.#captureExpectFailure(result?.error) + } + + /** Mark the failing action from a matcher error (afterStep for Cucumber, + * afterTest for Mocha route here). */ + #captureExpectFailure(error: unknown): void { + captureExpectFailure( + this.#sessionCapturer, + this.#currentTestUid, + error, + this.#options.captureAssertions !== false + ) + } + + /** Stamp final state + the resolved 0-based attempt onto the test's metadata + * entry, taking the max of the tracker count and WDIO's retry field. */ + #stampOutcome(uid: string, result?: TestOutcomeResult): void { + const fallback = this.#attemptTracker.attemptFor(uid) ?? 0 + const attempt = resolveTestAttempt(result, fallback) + stampTestState(this.#testMetadata, uid, result, attempt) + } + + async afterScenario( + world?: { + pickle?: { uri?: string; name?: string; astNodeIds?: readonly string[] } + }, + result?: TestOutcomeResult + ) { + const { uri, name, astNodeIds } = world?.pickle ?? {} + const uid = + uri && name ? cucumberScenarioUid(uri, name, astNodeIds) : undefined + if (uid) { + this.#stampOutcome(uid, result) + } await this.#finalizePerScenario() + // Flush now so this slice includes the final snapshot and stamped outcome. + if (uid) { + await this.#eagerFlushTestSlice(uid) + } } - async afterTest() { + async afterTest( + test?: { file?: string; title?: string; fullTitle?: string }, + _context?: unknown, + result?: TestOutcomeResult + ) { + this.#captureExpectFailure(result?.error) + const testTitle = test?.fullTitle || test?.title + const uid = testTitle ? testMetadataUid(test?.file, testTitle) : undefined + if (uid) { + this.#stampOutcome(uid, result) + } await this.#finalizePerScenario() + // Flush now so this slice includes the final snapshot and stamped outcome. + if (uid) { + await this.#eagerFlushTestSlice(uid) + } + } + + /** expect-webdriverio fires this before each matcher evaluates. The matcher's + * internal polling commands run inside the before→after window and must not + * surface as their own rows — only the `expect.<matcher>` assertion should. */ + beforeAssertion(): void { + // Matchers don't nest, so any residual depth here is a prior matcher whose + // afterAssertion was skipped by a hard throw. Reset before pushing so the + // suppression window can't stay stuck open across assertions. + if (this.#assertionDepth > 0) { + this.#assertionDepth = 0 + this.#assertionStartTimes = [] + this.#assertionCallSources = [] + } + this.#assertionDepth++ + this.#assertionStartTimes.push(Date.now()) + // Capture the user's expect() call site now — see #assertionCallSources. + this.#assertionCallSources.push( + resolveAssertionCallSource( + (file) => void this.#sessionCapturer.captureSource(file) + ) + ) + } + + async afterAssertion(params: ExpectAssertion): Promise<void> { + // Decrement first (even when capture is off) so the window stays balanced. + const startTime = this.#assertionStartTimes.pop() + const callSource = this.#assertionCallSources.pop() + if (this.#assertionDepth > 0) { + this.#assertionDepth-- + } + if (this.#options.captureAssertions === false) { + return + } + const entry = expectAssertionToCommandLog( + params, + this.#currentTestUid, + callSource + ) + // Span the matcher's poll window so the row's duration is real and the + // screencast tracks the assertion (not the preceding command) during it. + if (startTime !== undefined) { + entry.startTime = startTime + } + // The suppressed matcher-internal command carried the DOM screenshot; + // capture one here so the assertion row's Snapshot panel isn't blank. + if (this.#browser && !isNativeMobile(this.#browser)) { + try { + entry.screenshot = await this.#browser.takeScreenshot() + } catch (err) { + // best-effort: a missing screenshot must not fail the assertion hook + log.debug(`assertion screenshot skipped: ${errorMessage(err)}`) + } + } + // Trace mode: push a DOM action-snapshot stamped at this row's timestamp so + // the trace player's Snapshot tab renders it, exactly like a regular + // command's post-action snapshot (captureActionResult). + if (this.#options.mode === 'trace' && this.#browser) { + await pushActionSnapshotAt( + this.#browser, + entry.command, + entry.timestamp, + this.#actionSnapshots + ) + } + this.#sessionCapturer.captureAssertCommand(entry) } async #finalizePerScenario() { @@ -314,7 +525,10 @@ export default class DevToolsHookService implements Services.ServiceInstance { const snap = await captureActionSnapshot(this.#browser, '__final__') if (snap) { snap.timestamp = stamp - this.#actionSnapshots.push(snap) + // The last action's post-capture shares this timestamp and resources are + // named by timestamp, so keep only the richer screenshot — a blank + // end-of-scenario frame must not clobber the action's real result. + upsertRichestSnapshot(this.#actionSnapshots, snap) } } @@ -329,97 +543,15 @@ export default class DevToolsHookService implements Services.ServiceInstance { return Date.now() } - // Post-action capture: the state RESULTING from the action just completed. - // The pre-action capture in beforeCommand only records the state before the - // NEXT mapped action — when intervening commands (assertions, reloadSession) - // change the page first, an action's result (e.g. the page a click navigated - // to) is never captured. - // - // readyState alone is unreliable: right after a click the OLD document still - // reports 'complete', so a naive wait snapshots a blank mid-navigation frame. - // Instead, beforeCommand tags the document; if the tag is gone the action - // navigated, so we wait for the NEW document to finish loading AND render - // content before screenshotting its destination. Stamped at this command's - // own end (the latest logged action). - async #captureActionResult(command: string): Promise<void> { - if ( - this.#options.mode !== 'trace' || - !this.#browser || - !mapCommandToAction(command) || - INTERNAL_COMMANDS.includes(command) - ) { - return - } - const browser = this.#browser - if (!isNativeMobile(browser)) { - await this.#waitForResult(browser) - } - const snap = await captureActionSnapshot(browser, command) - if (snap) { - snap.timestamp = this.#lastActionTimestamp() - this.#actionSnapshots.push(snap) - } - } - - async #waitForResult(browser: WebdriverIO.Browser): Promise<void> { - const navigated = await browser - .execute( - () => !(window as Window & { __wdioSnapMark?: boolean }).__wdioSnapMark - ) - .catch(() => true) - if (!navigated) { - return - } - // Action triggered a navigation — wait for the destination document to load - // and render content so we screenshot the result page, not a blank frame. - await browser - .waitUntil( - async () => - (await browser - .execute( - () => - document.readyState === 'complete' && - !!document.body && - document.body.childElementCount > 0 - ) - .catch(() => false)) === true, - { timeout: 8000, interval: 150 } - ) - .catch(() => undefined) - // Headless renderers can return a blank shot right after load; let it paint. - await browser.pause(250).catch(() => undefined) - } - private resetStack() { this.#commandStack = [] + this.#assertionDepth = 0 + this.#assertionStartTimes = [] + this.#assertionCallSources = [] this.#sessionCapturer.resetLastSelector() this.#sessionCapturer.resetRetryTracker() } - #resolveCallSourceFromFrame( - frame: ReturnType<typeof parse>[number] - ): string | undefined { - const rawFile = frame.getFileName() ?? undefined - let absPath = rawFile - if (rawFile?.startsWith('file://')) { - try { - const url = new URL(rawFile) - absPath = decodeURIComponent(url.pathname) - } catch { - absPath = rawFile - } - } - if (absPath?.includes('?')) { - absPath = absPath.split('?')[0] - } - if (absPath === undefined) { - return undefined - } - const line = frame.getLineNumber() ?? undefined - const column = frame.getColumnNumber() ?? undefined - return `${absPath}:${line ?? 0}:${column ?? 0}` - } - #pushTopLevelCommandFrame( command: string, callSource: string | undefined @@ -457,10 +589,33 @@ export default class DevToolsHookService implements Services.ServiceInstance { Error.stackTraceLimit = 20 const stack = parse(new Error('')).reverse() const source = stack.find((frame) => isUserSpecFile(frame.getFileName())) - if (source && this.#commandStack.length === 0) { + // #assertionDepth > 0 → we believe we're inside an expect matcher; its + // internal commands trace back to the user's expect() line but must not + // become their own rows (only the expect.<matcher> assertion should). + // expect-webdriverio doesn't wrap afterAssertion in try/finally, though, so + // a matcher whose internal command hard-throws skips it and leaves the depth + // stuck ≥1 — which would then suppress every later user command. When a + // top-level user command arrives with the depth stuck but no live + // expect-webdriverio frame on the stack, the matcher already ended: self-heal + // the leftover depth + parallel stacks so this command captures normally. + if (source && this.#commandStack.length === 0 && this.#assertionDepth > 0) { + const inMatcher = stack.some((frame) => + frame.getFileName()?.includes('expect-webdriverio') + ) + if (!inMatcher) { + this.#assertionDepth = 0 + this.#assertionStartTimes = [] + this.#assertionCallSources = [] + } + } + if ( + source && + this.#commandStack.length === 0 && + this.#assertionDepth === 0 + ) { this.#pushTopLevelCommandFrame( command, - this.#resolveCallSourceFromFrame(source) + resolveCallSourceFromFrame(source) ) // Pre-action capture: state BEFORE this action executes. Will be @@ -522,7 +677,14 @@ export default class DevToolsHookService implements Services.ServiceInstance { frame.startTimestamp, this.#currentTestUid ) - await this.#captureActionResult(command) + if (this.#options.mode === 'trace') { + await captureActionResult( + this.#browser, + command, + this.#actionSnapshots, + () => this.#lastActionTimestamp() + ) + } return captured } } @@ -533,38 +695,11 @@ export default class DevToolsHookService implements Services.ServiceInstance { } } - /** - * Slice the session capturer's accumulated arrays for a single spec file - * and write a standalone trace artifact. Called at spec boundaries and - * from after() for the final spec. - */ - async #flushSpecTrace( - range: SpecRange, - nextRange?: SpecRange - ): Promise<string | undefined> { - if (!this.#browser || this.#flushedSpecs.has(range.specFile)) { - return undefined - } - this.#flushedSpecs.add(range.specFile) - - const tracePath = await writeSpecTrace({ - range, - nextRange, - capturer: this.#sessionCapturer, - actionSnapshots: this.#actionSnapshots, - sessionId: this.#browser.sessionId, - outputDir: this.#outputDir, - format: this.#options.traceFormat, - testMetadata: this.#testMetadata, - capabilities: this.#browser.capabilities - }) - log.info(`Trace for spec "${range.specFile}" saved to ${tracePath}`) - return tracePath - } - /** * after hook is triggered at the end of every worker session, therefore - * we can use it to write all trace information to a file + * we can use it to write all trace information to a file. `trace` mode + * writes the shareable trace.zip (opened via `pnpm show-trace`); `live` + * mode streams to the dashboard over WS and persists nothing to disk. */ async after() { if (!this.#browser) { @@ -574,47 +709,7 @@ export default class DevToolsHookService implements Services.ServiceInstance { // Stop and encode the screencast for the current session. await this.#finalizeScreencast(this.#browser.sessionId) - // `trace` mode writes the shareable trace.zip (opened via `pnpm - // show-trace`); `live` mode streams to the dashboard over WS and persists - // nothing to disk. - if (this.#options.mode === 'trace') { - if ( - this.#options.traceGranularity === 'spec' && - this.#specRanges.length > 0 - ) { - // Per-spec traces — flush each detected spec range. - for (const range of this.#specRanges) { - await this.#flushSpecTrace(range) - } - } else { - if (this.#options.traceGranularity === 'spec') { - log.warn( - 'traceGranularity is "spec" but no spec boundaries were ' + - 'detected (framework may not support service-level test ' + - 'hooks). Falling back to session-level trace.' - ) - } - // Session-level trace. Snapshots can share a timestamp (an action's - // post-action result plus the next action's pre-capture and the - // per-scenario final capture); the writer keys resources by timestamp, - // so keep the richest per timestamp — a navigated action's result wins - // over a blank mid-navigation frame. - const snapshots = dedupeSnapshotsByTimestamp(this.#actionSnapshots) - try { - const tracePath = await writeTraceZip(this.#sessionCapturer, { - outputDir: this.#outputDir, - sessionId: this.#browser.sessionId, - capabilities: this.#browser.capabilities, - actionSnapshots: snapshots.length ? snapshots : undefined, - format: this.#options.traceFormat, - testMetadata: this.#testMetadata - }) - log.info(`Trace saved to ${tracePath}`) - } catch (err) { - log.error(`Trace write failed: ${errorMessage(err)}`) - } - } - } + await finalizeTraceExport(this.#traceContext(this.#browser)) // Clean up console patching this.#sessionCapturer.cleanup() diff --git a/packages/service/src/reporter.ts b/packages/service/src/reporter.ts index ad792131..ee22f23b 100644 --- a/packages/service/src/reporter.ts +++ b/packages/service/src/reporter.ts @@ -26,8 +26,13 @@ function isScenario(item: SuiteStats | TestStats): boolean { // Generate stable UID for a WDIO suite/test stats object. Handles WDIO's // Cucumber-specific shapes (scenarios with featureFile/featureLine, or with // numeric uid + example-row fallback), then delegates the Mocha/Jasmine path -// to core's generateStableUid. -function generateStableUid(item: SuiteStats | TestStats): string { +// to core's generateStableUid. `parentScope` (the owning scenario's stable +// uid) disambiguates Cucumber steps so identical step text in sibling +// scenarios yields distinct, rerun-stable uids. +function generateStableUid( + item: SuiteStats | TestStats, + parentScope?: string +): string { // For Cucumber scenarios, prefer the feature file URI:line as the stable // discriminator. The Cucumber pickle carries the actual line of the example // row, which is stable across reruns regardless of how many examples run. @@ -60,11 +65,25 @@ function generateStableUid(item: SuiteStats | TestStats): string { ) } + // Cucumber step: scope by the owning scenario's stable uid via + // deterministicUid (no run-order counter), so two scenarios sharing step + // text — and scenario-outline example rows — get distinct, rerun-stable uids. + const stepFile = 'file' in item ? (item.file ?? '') : '' + if (parentScope) { + return deterministicUid( + stepFile, + parentScope, + String(item.fullTitle || item.title) + ) + } + // For Mocha/Jasmine tests and suites, use only stable identifiers // that don't change between full and partial runs // DO NOT use cid or parent as they can vary based on run context - const file = 'file' in item ? (item.file ?? '') : '' - return generateStableUidByFileName(file, String(item.fullTitle || item.title)) + return generateStableUidByFileName( + stepFile, + String(item.fullTitle || item.title) + ) } /** @@ -154,6 +173,9 @@ export class TestReporter extends WebdriverIOReporter { #loadSource: (location: string) => void #currentSpecFile?: string #suitePath: string[] = [] + /** Stable uid of the Cucumber scenario currently open, used to scope its + * step uids. Undefined outside a scenario (Mocha/Jasmine). */ + #currentScenarioUid?: string constructor( options: Reporters.Options, @@ -205,6 +227,11 @@ export class TestReporter extends WebdriverIOReporter { // Generate stable UID for consistent identification across reruns suiteStats.uid = generateStableUid(suiteStats) + // Track the open Cucumber scenario so its steps scope their uids to it. + if (isScenario(suiteStats)) { + this.#currentScenarioUid = suiteStats.uid + } + this.#currentSpecFile = suiteStats.file setCurrentSpecFile(suiteStats.file) @@ -252,8 +279,10 @@ export class TestReporter extends WebdriverIOReporter { this.#loadSource(testStats.file) } - // Generate stable UID after enriching metadata for consistent test identification - testStats.uid = generateStableUid(testStats) + // Generate stable UID after enriching metadata for consistent test + // identification. Cucumber steps are scoped by their scenario's uid so + // identical step text across scenarios stays distinct. + testStats.uid = generateStableUid(testStats, this.#currentScenarioUid) this.#sendUpstream() } @@ -285,11 +314,19 @@ export class TestReporter extends WebdriverIOReporter { matcherResult: rawErr.matcherResult } as Error } + // WDIO stamps the 0-based attempt on each test:start, so the final attempt's + // stat already carries the retry count; the field is optional upstream, so + // pin it to a number for the shared TestStats.retries contract. + testStats.retries = testStats.retries ?? 0 this.#sendUpstream() } onSuiteEnd(suiteStats: SuiteStats): void { super.onSuiteEnd(suiteStats) + // Stop scoping steps once the owning scenario closes. + if (isScenario(suiteStats) && suiteStats.uid === this.#currentScenarioUid) { + this.#currentScenarioUid = undefined + } // Pop the suite we pushed on start if ( suiteStats.title && diff --git a/packages/service/src/session.ts b/packages/service/src/session.ts index 2925c4b0..58a90a24 100644 --- a/packages/service/src/session.ts +++ b/packages/service/src/session.ts @@ -248,6 +248,12 @@ export class SessionCapturer extends SessionCapturerBase { this.#retryTracker.reset() } + /** Ingest an assertion entry (node:assert capture or synthesized expect + * failure) through the same retry-collapsing path driver commands use. */ + captureAssertCommand(entry: CommandLog): void { + this.#captureOrReplace(entry) + } + /** * Run the shared Performance API capture script and attach the result to * the given CommandLog entry. Same `CAPTURE_PERFORMANCE_SCRIPT` + diff --git a/packages/service/src/snapshot-dedupe.ts b/packages/service/src/snapshot-dedupe.ts index d7e14a00..5e51cccd 100644 --- a/packages/service/src/snapshot-dedupe.ts +++ b/packages/service/src/snapshot-dedupe.ts @@ -19,3 +19,23 @@ export function dedupeSnapshotsByTimestamp( } return [...best.values()].sort((a, b) => a.timestamp - b.timestamp) } + +/** Insert a snapshot, or — when one already shares its timestamp — keep only + * the richer screenshot, replacing in place to preserve any index ranges into + * the list. Applies the dedupe heuristic at capture time so a blank + * end-of-scenario frame can't clobber the last action's real result on export + * paths that don't run dedupeSnapshotsByTimestamp (e.g. per-spec traces). */ +export function upsertRichestSnapshot( + snapshots: ActionSnapshot[], + snap: ActionSnapshot +): void { + const idx = snapshots.findIndex((s) => s.timestamp === snap.timestamp) + if (idx === -1) { + snapshots.push(snap) + return + } + const existing = snapshots[idx]! + if ((snap.screenshot?.length ?? 0) > (existing.screenshot?.length ?? 0)) { + snapshots[idx] = snap + } +} diff --git a/packages/service/src/test-metadata.ts b/packages/service/src/test-metadata.ts new file mode 100644 index 00000000..dc2babad --- /dev/null +++ b/packages/service/src/test-metadata.ts @@ -0,0 +1,77 @@ +// Per-test metadata helpers shared between the before* and after* hooks so the +// state stamp in afterTest/afterScenario lands on the entry beforeTest created. + +import { deterministicUid } from '@wdio/devtools-core' +import type { TestMetadataMap, TestStatus } from '@wdio/devtools-shared' + +/** The subset of a WDIO afterTest/afterScenario hook result this adapter reads: + * pass/skip state, the error, and WDIO's authoritative 0-based retry count. */ +export interface TestOutcomeResult { + error?: unknown + passed?: boolean + skipped?: boolean + retries?: { attempts?: number } +} + +/** Stable per-test key. beforeTest and afterTest derive it identically so keys + * match. File-less runners (some non-WDIO frameworks) key on the title alone. */ +export function testMetadataUid( + file: string | undefined, + title: string +): string { + return file ? deterministicUid(file, title) : title +} + +/** Scenario key that separates scenario-outline example rows: they share a + * name, so the pickle's astNodeIds (distinct per row, stable across reruns) + * are folded in. beforeScenario and afterScenario derive it identically. */ +export function cucumberScenarioUid( + uri: string, + name: string, + astNodeIds?: readonly string[] +): string { + return astNodeIds?.length + ? deterministicUid(uri, name, astNodeIds.join(':')) + : deterministicUid(uri, name) +} + +/** Canonical test state from a WDIO afterTest/afterScenario result. */ +export function resultToState(result: { + passed?: boolean + skipped?: boolean +}): TestStatus { + if (result.skipped) { + return 'skipped' + } + return result.passed ? 'passed' : 'failed' +} + +/** Stamp the final state (and, when known, the 0-based attempt) onto the + * metadata entry beforeTest/beforeScenario created, so retention can gate its + * trace per attempt. No-op when there's no entry. */ +export function stampTestState( + metadata: TestMetadataMap, + uid: string, + result?: Pick<TestOutcomeResult, 'passed' | 'skipped'>, + attempt?: number +): void { + const entry = result && metadata.get(uid) + if (entry) { + entry.state = resultToState(result) + if (attempt !== undefined) { + entry.attempt = attempt + } + } +} + +/** Resolve the 0-based attempt for a test. WDIO's mocha framework reports + * `retries.attempts` as 0 even on a retry, so it can't override the in-process + * tracker — take the max so a present-but-zero runner field never clobbers a + * real retry count, while a genuine runner value still wins when it's higher. */ +export function resolveTestAttempt( + result: Pick<TestOutcomeResult, 'retries'> | undefined, + fallback: number +): number { + const attempts = result?.retries?.attempts + return Math.max(typeof attempts === 'number' ? attempts : 0, fallback) +} diff --git a/packages/service/src/trace-slices.ts b/packages/service/src/trace-slices.ts new file mode 100644 index 00000000..fcc36962 --- /dev/null +++ b/packages/service/src/trace-slices.ts @@ -0,0 +1,49 @@ +// Trace-slice flushing for the WDIO adapter: previous-slice flush at boundary +// changes plus the eager per-test flush. Kept out of index.ts so the slice +// selection and flush I/O are unit-testable and the god-file stays lean. + +import { + flushRangeLogged, + type SpecRange, + type TraceExportContext +} from '@wdio/devtools-core' + +/** The range for the test that just ended is the most recent slice recorded + * under this base testUid — retries push a new range under the same testUid, + * so reverse-scanning finds the attempt whose afterTest is now firing. */ +export function findCurrentTestRange( + ranges: readonly SpecRange[], + testUid: string +): SpecRange | undefined { + for (let i = ranges.length - 1; i >= 0; i--) { + if (ranges[i]!.testUid === testUid) { + return ranges[i] + } + } + return undefined +} + +/** Fire-and-forget flush of the previous unflushed slice at a boundary change + * (spec granularity, or a test slice whose eager flush was missed). Errors are + * logged, never thrown, so a failed flush can't abort the next test. */ +export function flushPrevSlice( + ctx: TraceExportContext, + range: SpecRange +): void { + void flushRangeLogged(ctx, range) +} + +/** Awaited flush of the just-ended test's slice (test granularity), so this + * attempt's just-stamped metadata is written before a retry's beforeTest + * overwrites the entry. No-op when the test recorded no range. */ +export async function flushTestSlice( + ctx: TraceExportContext, + ranges: readonly SpecRange[], + testUid: string +): Promise<void> { + const range = findCurrentTestRange(ranges, testUid) + if (!range) { + return + } + await flushRangeLogged(ctx, range) +} diff --git a/packages/service/src/types.ts b/packages/service/src/types.ts index f5cfa50d..ce10a42e 100644 --- a/packages/service/src/types.ts +++ b/packages/service/src/types.ts @@ -21,36 +21,24 @@ export { type Viewport } from '@wdio/devtools-shared' +import type { BaseDevToolsOptions } from '@wdio/devtools-shared' + // ScreencastFrame, ScreencastOptions hoisted to @wdio/devtools-shared; re-exported // here for backwards compatibility with existing service-internal imports. -import type { - DevToolsMode, - ScreencastOptions, - TraceFormat, - TraceGranularity -} from '@wdio/devtools-shared' export type { DevToolsMode, ScreencastFrame, ScreencastOptions, TraceFormat, - TraceGranularity + TraceGranularity, + TraceRetentionPolicy } from '@wdio/devtools-shared' export interface ExtendedCapabilities extends WebdriverIO.Capabilities { 'wdio:devtoolsOptions'?: ServiceOptions } -export interface ServiceOptions { - /** - * port to launch the application on (default: random) - */ - port?: number - /** - * hostname to launch the application on - * @default localhost - */ - hostname?: string +export interface ServiceOptions extends BaseDevToolsOptions { /** * capabilities used to launch the devtools application * @default @@ -63,21 +51,6 @@ export interface ServiceOptions { * } */ devtoolsCapabilities?: WebdriverIO.Capabilities - /** - * Screencast recording options. When enabled, a continuous video of the - * browser session is recorded and saved as a .webm file. Chrome/Chromium - * uses CDP push mode; all other browsers fall back to screenshot polling. - */ - screencast?: ScreencastOptions - /** `live` (default) launches the DevTools UI; `trace` skips it. */ - mode?: DevToolsMode - /** Trace output layout — `zip` (default) writes a single archive, - * `ndjson-directory` unpacks into `trace-<id>/`. Only applies in trace mode. */ - traceFormat?: TraceFormat - /** Trace output granularity — `session` (default) writes one trace per - * worker session; `spec` writes one trace per spec file. Only applies in - * trace mode. */ - traceGranularity?: TraceGranularity } declare namespace WebdriverIO { diff --git a/packages/service/tests/action-snapshot.test.ts b/packages/service/tests/action-snapshot.test.ts new file mode 100644 index 00000000..f81be33c --- /dev/null +++ b/packages/service/tests/action-snapshot.test.ts @@ -0,0 +1,29 @@ +import { describe, it, expect, vi } from 'vitest' +import type { ActionSnapshot } from '@wdio/devtools-shared' +import { pushActionSnapshotAt } from '../src/action-snapshot.js' + +const mockBrowser = () => + ({ + execute: vi.fn().mockResolvedValue([]), + takeScreenshot: vi.fn().mockResolvedValue('SHOT'), + getUrl: vi.fn().mockResolvedValue('http://example.com/'), + getTitle: vi.fn().mockResolvedValue('Example') + }) as unknown as WebdriverIO.Browser + +describe('pushActionSnapshotAt', () => { + it('captures a DOM snapshot and stamps it at the row timestamp', async () => { + const snapshots: ActionSnapshot[] = [] + await pushActionSnapshotAt( + mockBrowser(), + 'expect.toExist', + 12345, + snapshots + ) + expect(snapshots).toHaveLength(1) + // Stamped at the row's own timestamp — not the capture time — so the trace + // player's FrameSnapshotIndex.claimAfter(cmd.timestamp) matches it. + expect(snapshots[0]!.timestamp).toBe(12345) + expect(snapshots[0]!.command).toBe('expect.toExist') + expect(snapshots[0]!.screenshot).toBe('SHOT') + }) +}) diff --git a/packages/service/tests/assert-capture.test.ts b/packages/service/tests/assert-capture.test.ts new file mode 100644 index 00000000..534945e3 --- /dev/null +++ b/packages/service/tests/assert-capture.test.ts @@ -0,0 +1,255 @@ +import { describe, it, expect, vi, afterAll } from 'vitest' +import assert from 'node:assert' +import { + ASSERT_PATCHED_SYMBOL, + TRACKED_ASSERT_METHODS +} from '@wdio/devtools-core' +import { + captureExpectFailure, + expectAssertionToCommandLog, + resolveAssertionCallSource, + toCommandError, + wireAssertCapture +} from '../src/assert-capture.js' +import type { SessionCapturer } from '../src/session.js' + +const stackFrames = vi.hoisted(() => ({ + value: [] as Array<{ + getFileName: () => string | null + getLineNumber: () => number | null + getColumnNumber: () => number | null + }> +})) + +// Only resolveAssertionCallSource reads 'stack-trace'; node:assert capture uses +// core's stacktrace-parser path, so mocking here doesn't affect those tests. +vi.mock('stack-trace', () => ({ parse: () => stackFrames.value })) + +const frame = (file: string | null, line = 1, column = 1) => ({ + getFileName: () => file, + getLineNumber: () => line, + getColumnNumber: () => column +}) + +describe('toCommandError', () => { + it('normalizes a plain Error object (ANSI stripped)', () => { + // Matcher errors are skipped now (afterAssertion owns them); a plain + // thrown Error still routes through failLastAction, so test that path. + const error = new Error('something broke') + expect(toCommandError(error)).toMatchObject({ + name: 'Error', + message: 'something broke' + }) + }) + + it('wraps a Cucumber string message, stripping ANSI', () => { + const raw = 'Expect to have text\n\nExpected: "a"' + expect(toCommandError(raw)).toEqual({ + name: 'Error', + message: 'Expect to have text\n\nExpected: "a"' + }) + }) + + it('returns undefined for self-captured errors (AssertionError / matcher), empties and non-errors', () => { + const assertionError = Object.assign(new Error('a !== b'), { + name: 'AssertionError' + }) + expect(toCommandError(assertionError)).toBeUndefined() + // expect-webdriverio matcher errors carry matcherResult and are already + // captured by afterAssertion — must not double-mark via failLastAction. + const matcherError = Object.assign(new Error('expected a to be b'), { + matcherResult: { pass: false } + }) + expect(toCommandError(matcherError)).toBeUndefined() + expect(toCommandError(' ')).toBeUndefined() + expect(toCommandError(undefined)).toBeUndefined() + expect(toCommandError(42)).toBeUndefined() + }) +}) + +describe('captureExpectFailure', () => { + function fakeCapturer() { + return { + failLastAction: vi.fn().mockReturnValue(true) + } as unknown as SessionCapturer & { + failLastAction: ReturnType<typeof vi.fn> + } + } + + it('marks the last action with the normalized error', () => { + const capturer = fakeCapturer() + captureExpectFailure(capturer, 'test-1', 'boom', true) + expect(capturer.failLastAction).toHaveBeenCalledWith('test-1', { + name: 'Error', + message: 'boom' + }) + }) + + it('is a no-op when disabled or when there is no error', () => { + const capturer = fakeCapturer() + captureExpectFailure(capturer, 'test-1', 'boom', false) + captureExpectFailure(capturer, 'test-1', undefined, true) + expect(capturer.failLastAction).not.toHaveBeenCalled() + }) + + it('does not mark an action for a self-captured matcher error', () => { + const capturer = fakeCapturer() + // A failing expect-webdriverio matcher (carries matcherResult) is already + // its own command via afterAssertion — failLastAction must stay off it. + const matcherError = Object.assign(new Error('expected'), { + matcherResult: { pass: false } + }) + captureExpectFailure(capturer, 'test-1', matcherError, true) + expect(capturer.failLastAction).not.toHaveBeenCalled() + }) +}) + +describe('wireAssertCapture', () => { + // Snapshot real methods so the process-wide patch is undone after this file. + const ASSERT_MUT = assert as unknown as Record<string | symbol, unknown> + const originals: Record<string, unknown> = {} + for (const method of TRACKED_ASSERT_METHODS) { + originals[method] = ASSERT_MUT[method] + } + afterAll(() => { + delete ASSERT_MUT[ASSERT_PATCHED_SYMBOL] + for (const method of TRACKED_ASSERT_METHODS) { + ASSERT_MUT[method] = originals[method] + } + }) + + it('routes patched asserts into the capturer with the current test uid', () => { + const entries: CommandLog[] = [] + const live: { capturer?: SessionCapturer; uid?: string } = {} + wireAssertCapture( + () => live.capturer as SessionCapturer, + () => live.uid + ) + + // Fake narrowed to the single method the wiring uses. + live.capturer = { + captureAssertCommand: (entry: CommandLog) => entries.push(entry) + } as unknown as SessionCapturer + live.uid = 'uid-1' + assert.equal(1, 1) + expect(entries[0]).toMatchObject({ + command: 'assert.equal', + args: [1, 1], + result: 'passed', + testUid: 'uid-1' + }) + + live.uid = 'uid-2' + expect(() => assert.strictEqual('a', 'b')).toThrow() + const failed = entries[1] + expect(failed.command).toBe('assert.strictEqual') + expect(failed.testUid).toBe('uid-2') + expect(failed.error).toMatchObject({ name: 'AssertionError' }) + expect(failed.result).toBeUndefined() + }) + + it('is a no-op wiring when invoked twice (per-process patch guard)', () => { + const spy = vi.fn() + wireAssertCapture( + () => ({ captureAssertCommand: spy }) as unknown as SessionCapturer, + () => undefined + ) + assert.ok(true) + expect(spy).not.toHaveBeenCalled() + }) +}) + +describe('expectAssertionToCommandLog', () => { + it('captures a passing matcher as an expect.<matcher> command', () => { + const entry = expectAssertionToCommandLog( + { + matcherName: 'toHaveTitle', + expectedValue: 'The Internet', + result: { pass: true, message: () => 'ok' } + }, + 'uid-1' + ) + expect(entry).toMatchObject({ + command: 'expect.toHaveTitle', + args: ['The Internet'], + result: 'passed', + testUid: 'uid-1' + }) + expect(entry.error).toBeUndefined() + }) + + it('captures a failing matcher with its ANSI-stripped message as the error', () => { + const entry = expectAssertionToCommandLog( + { + matcherName: 'toHaveText', + expectedValue: 'foo', + result: { pass: false, message: () => 'expected foo' } + }, + undefined + ) + expect(entry.result).toBeUndefined() + expect(entry.error).toMatchObject({ message: 'expected foo' }) + }) + + it('spreads an array expectedValue and reads the typed `result` flag', () => { + // @wdio/types declares the pass flag on `result`, not `pass` — read both. + const entry = expectAssertionToCommandLog( + { + matcherName: 'toHaveAttribute', + expectedValue: ['href', '/x'], + result: { result: true } + }, + undefined + ) + expect(entry).toMatchObject({ + command: 'expect.toHaveAttribute', + args: ['href', '/x'], + result: 'passed' + }) + }) + + it('treats a matcher with no expectedValue as a no-arg assertion', () => { + const entry = expectAssertionToCommandLog( + { matcherName: 'toBeClickable', result: { pass: true } }, + undefined + ) + expect(entry).toMatchObject({ command: 'expect.toBeClickable', args: [] }) + }) + + it('forwards the captured callSource onto the assertion row', () => { + const entry = expectAssertionToCommandLog( + { matcherName: 'toExist', result: { pass: true } }, + 'uid-1', + '/proj/specs/login.e2e.ts:30:7' + ) + expect(entry.callSource).toBe('/proj/specs/login.e2e.ts:30:7') + }) +}) + +describe('resolveAssertionCallSource', () => { + it('returns the outermost user-spec frame and loads its source', () => { + // innermost → outermost: service bundle, expect-webdriverio matcher, the + // user spec, then node internals. The user spec must win, not the bundle. + stackFrames.value = [ + frame('/repo/packages/service/dist/index.js', 4045, 12), + frame('/repo/node_modules/expect-webdriverio/lib/matchers.js', 9, 3), + frame('/proj/specs/login.e2e.ts', 30, 7), + frame('node:internal/process/task_queues', 95, 5) + ] + const captured: string[] = [] + const callSource = resolveAssertionCallSource((f) => captured.push(f)) + expect(callSource).toBe('/proj/specs/login.e2e.ts:30:7') + expect(callSource).not.toContain('/dist/') + expect(captured).toEqual(['/proj/specs/login.e2e.ts']) + }) + + it('returns undefined and loads nothing when only dependency frames exist', () => { + stackFrames.value = [ + frame('/repo/node_modules/expect-webdriverio/lib/matchers.js', 9, 3), + frame('node:internal/process/task_queues', 95, 5) + ] + const captured: string[] = [] + expect(resolveAssertionCallSource((f) => captured.push(f))).toBeUndefined() + expect(captured).toEqual([]) + }) +}) diff --git a/packages/service/tests/assertion-rows.test.ts b/packages/service/tests/assertion-rows.test.ts new file mode 100644 index 00000000..5a5b09ac --- /dev/null +++ b/packages/service/tests/assertion-rows.test.ts @@ -0,0 +1,157 @@ +import { describe, it, expect, vi, beforeEach, afterAll } from 'vitest' +import assert from 'node:assert' +import { + ASSERT_PATCHED_SYMBOL, + TRACKED_ASSERT_METHODS +} from '@wdio/devtools-core' + +// Controlled synchronous stack for the beforeAssertion call-source walk. +const stackFrames = vi.hoisted(() => ({ + value: [] as Array<{ + getFileName: () => string | null + getLineNumber: () => number | null + getColumnNumber: () => number | null + }> +})) +vi.mock('stack-trace', () => ({ parse: () => stackFrames.value })) + +const capturer = vi.hoisted(() => ({ + captureAssertCommand: vi.fn(), + captureSource: vi.fn().mockResolvedValue(undefined), + injectScript: vi.fn().mockResolvedValue(undefined), + sendUpstream: vi.fn(), + cleanup: vi.fn(), + resetLastSelector: vi.fn(), + resetRetryTracker: vi.fn(), + commandsLog: [] as unknown[], + sources: new Map<string, string>() +})) +vi.mock('../src/session.js', () => ({ + SessionCapturer: vi.fn(function (this: unknown) { + return capturer + }) +})) + +const pushActionSnapshotAt = vi.hoisted(() => + vi.fn().mockResolvedValue(undefined) +) +vi.mock('../src/action-snapshot.js', () => ({ + pushActionSnapshotAt, + captureActionSnapshot: vi.fn().mockResolvedValue(null), + captureActionResult: vi.fn().mockResolvedValue(undefined) +})) + +import DevToolsHookService from '../src/index.js' +import type { ExpectAssertion } from '../src/assert-capture.js' + +const userFrame = { + getFileName: () => '/proj/specs/login.e2e.ts', + getLineNumber: () => 30, + getColumnNumber: () => 7 +} + +const mockBrowser = { + isBidi: true, + sessionId: 's1', + options: {}, + capabilities: {}, + takeScreenshot: vi.fn().mockResolvedValue('SHOT'), + execute: vi + .fn() + .mockResolvedValue({ width: 1, height: 1, offsetLeft: 0, offsetTop: 0 }), + on: vi.fn(), + emit: vi.fn() +} as unknown as WebdriverIO.Browser + +// before() wires node:assert capture; restore the real methods afterwards. +const ASSERT_MUT = assert as unknown as Record<string | symbol, unknown> +const originals: Record<string, unknown> = {} +for (const method of TRACKED_ASSERT_METHODS) { + originals[method] = ASSERT_MUT[method] +} +afterAll(() => { + delete ASSERT_MUT[ASSERT_PATCHED_SYMBOL] + for (const method of TRACKED_ASSERT_METHODS) { + ASSERT_MUT[method] = originals[method] + } +}) + +const capturedEntry = () => capturer.captureAssertCommand.mock.calls[0]![0] + +describe('DevtoolsService — expect.* assertion rows', () => { + beforeEach(() => { + vi.clearAllMocks() + stackFrames.value = [userFrame] + }) + + it('trace mode: user-spec callSource, spec source loaded, DOM snapshot pushed', async () => { + const service = new DevToolsHookService({ mode: 'trace' }) + await service.before({} as never, [], mockBrowser) + + service.beforeAssertion() + const params: ExpectAssertion = { + matcherName: 'toExist', + result: { pass: true, message: () => 'ok' } + } + await service.afterAssertion(params) + + const entry = capturedEntry() + expect(entry.command).toBe('expect.toExist') + // The regression: callSource must point at the user's spec, not the + // service bundle (…/service/dist/index.js). + expect(entry.callSource).toBe('/proj/specs/login.e2e.ts:30:7') + expect(entry.callSource).not.toContain('/dist/') + // Source of that file is loaded so the Source tab can render it. + expect(capturer.captureSource).toHaveBeenCalledWith( + '/proj/specs/login.e2e.ts' + ) + // Live-mode screenshot kept for the CommandLog. + expect(entry.screenshot).toBe('SHOT') + // Trace-player Snapshot tab: a DOM snapshot stamped at the row timestamp. + expect(pushActionSnapshotAt).toHaveBeenCalledWith( + mockBrowser, + 'expect.toExist', + entry.timestamp, + expect.any(Array) + ) + // WDIO reconciles rows by timestamp (like every regular WDIO command), + // so assertion rows carry no public id — parity with regular commands. + expect(entry.id).toBeUndefined() + }) + + it('live mode: keeps the screenshot but pushes no DOM snapshot', async () => { + const service = new DevToolsHookService({ mode: 'live' }) + await service.before({} as never, [], mockBrowser) + + service.beforeAssertion() + await service.afterAssertion({ + matcherName: 'toHaveText', + expectedValue: 'Hi', + result: { pass: false, message: () => 'nope' } + }) + + const entry = capturedEntry() + expect(entry.command).toBe('expect.toHaveText') + expect(entry.callSource).toBe('/proj/specs/login.e2e.ts:30:7') + expect(entry.error).toMatchObject({ message: 'nope' }) + expect(entry.screenshot).toBe('SHOT') + expect(pushActionSnapshotAt).not.toHaveBeenCalled() + }) + + it('captureAssertions: false suppresses the row but keeps the window balanced', async () => { + const service = new DevToolsHookService({ + mode: 'trace', + captureAssertions: false + }) + await service.before({} as never, [], mockBrowser) + + service.beforeAssertion() + await service.afterAssertion({ + matcherName: 'toExist', + result: { pass: true } + }) + + expect(capturer.captureAssertCommand).not.toHaveBeenCalled() + expect(pushActionSnapshotAt).not.toHaveBeenCalled() + }) +}) diff --git a/packages/service/tests/dedupe-snapshots.test.ts b/packages/service/tests/dedupe-snapshots.test.ts index c33b9b38..d1ac50ee 100644 --- a/packages/service/tests/dedupe-snapshots.test.ts +++ b/packages/service/tests/dedupe-snapshots.test.ts @@ -1,6 +1,13 @@ +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' import { describe, it, expect } from 'vitest' -import type { ActionSnapshot } from '@wdio/devtools-shared' -import { dedupeSnapshotsByTimestamp } from '../src/snapshot-dedupe.js' +import { writeTraceZip, type TraceCapturer } from '@wdio/devtools-core' +import { TraceType, type ActionSnapshot } from '@wdio/devtools-shared' +import { + dedupeSnapshotsByTimestamp, + upsertRichestSnapshot +} from '../src/snapshot-dedupe.js' function snap(timestamp: number, screenshot: string): ActionSnapshot { return { timestamp, command: 'click', screenshot } @@ -34,3 +41,116 @@ describe('dedupeSnapshotsByTimestamp', () => { expect(dedupeSnapshotsByTimestamp([])).toEqual([]) }) }) + +describe('upsertRichestSnapshot', () => { + const blank = 'AA' + const content = 'A'.repeat(100) + + it("does not let a blank __final__ clobber the last action's real frame", () => { + // The last action's post-capture (real) and the end-of-scenario __final__ + // (blank) share the last action's timestamp; the real frame must survive. + const list: ActionSnapshot[] = [snap(50, content), snap(100, content)] + const final: ActionSnapshot = { + timestamp: 100, + command: '__final__', + screenshot: blank + } + upsertRichestSnapshot(list, final) + expect(list).toHaveLength(2) + expect(list[1].screenshot).toBe(content) + expect(list[1].command).toBe('click') + }) + + it('replaces in place when the final capture is richer', () => { + // Mirror the opposite case: the action was screenshotted mid-navigation + // (blank) and the settled __final__ is the real frame. + const list: ActionSnapshot[] = [snap(50, content), snap(100, blank)] + const final: ActionSnapshot = { + timestamp: 100, + command: '__final__', + screenshot: content + } + upsertRichestSnapshot(list, final) + expect(list).toHaveLength(2) + expect(list[1].screenshot).toBe(content) + expect(list[1].command).toBe('__final__') + }) + + it('preserves array length/indices on a timestamp collision', () => { + // Spec-range slicing indexes into this array, so a collision must never + // change its length — only replace in place or skip. + const list: ActionSnapshot[] = [snap(100, content)] + upsertRichestSnapshot(list, snap(100, blank)) + expect(list).toHaveLength(1) + }) + + it('appends when the timestamp is new', () => { + const list: ActionSnapshot[] = [snap(100, content)] + upsertRichestSnapshot(list, snap(200, blank)) + expect(list.map((s) => s.timestamp)).toEqual([100, 200]) + }) +}) + +describe('final-frame regression (capture → export)', () => { + it("exports the last action's real result, not the blank __final__ frame", async () => { + // Base64 payloads chosen so byte-length ranks blank < real < result and + // each round-trips cleanly through the resource writer's base64 decode. + const blank = 'AA' + const real = 'R'.repeat(200) + const result = 'B'.repeat(400) + + // The service builds pre/post captures per action; the last action's + // post-capture (result) collides on timestamp with the trailing __final__. + const snapshots: ActionSnapshot[] = [ + { timestamp: 1200, command: 'url', screenshot: real }, + { timestamp: 1200, command: 'click', screenshot: real }, + { timestamp: 1400, command: 'click', screenshot: result } + ] + upsertRichestSnapshot(snapshots, { + timestamp: 1400, + command: '__final__', + screenshot: blank + }) + const prepared = dedupeSnapshotsByTimestamp(snapshots) + + const capturer: TraceCapturer = { + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + commandsLog: [ + { command: 'url', args: [], timestamp: 1200, startTime: 1150 }, + { command: 'click', args: [], timestamp: 1400, startTime: 1350 } + ], + sources: new Map(), + metadata: { + type: TraceType.Testrunner, + viewport: { + width: 800, + height: 600, + offsetLeft: 0, + offsetTop: 0, + scale: 1 + } + }, + startWallTime: 1000 + } + + const outputDir = await fs.mkdtemp(path.join(os.tmpdir(), 'final-frame-')) + const dir = await writeTraceZip(capturer, { + outputDir, + sessionId: 'sess1234', + format: 'ndjson-directory', + actionSnapshots: prepared + }) + + const frame = await fs.readFile( + path.join(dir, 'resources', 'page@sess1234-1400.jpeg') + ) + // The last action's frame is the real result, never the blank capture. + expect(frame.toString('base64')).toBe(result) + expect(frame.length).not.toBe(Buffer.from(blank, 'base64').length) + + await fs.rm(outputDir, { recursive: true, force: true }) + }) +}) diff --git a/packages/service/tests/index.test.ts b/packages/service/tests/index.test.ts index 6d2a56ff..d46921e0 100644 --- a/packages/service/tests/index.test.ts +++ b/packages/service/tests/index.test.ts @@ -15,6 +15,8 @@ const mockSessionCapturerInstance = { afterCommand: vi.fn(), sendUpstream: vi.fn(), injectScript: vi.fn().mockResolvedValue(undefined), + captureSource: vi.fn(), + captureAssertCommand: vi.fn(), cleanup: vi.fn(), commandsLog: [], sources: new Map(), @@ -105,13 +107,7 @@ describe('DevtoolsService - Internal Command Filtering', () => { describe('beforeCommand', () => { it('should not add internal commands to command stack', () => { - const internalCommands = [ - 'getTitle', - 'waitUntil', - 'getUrl', - 'execute', - 'findElement' - ] + const internalCommands = ['getTitle', 'getUrl', 'execute', 'findElement'] internalCommands.forEach((cmd) => service.beforeCommand(cmd as any, [])) expect(true).toBe(true) }) @@ -137,19 +133,34 @@ describe('DevtoolsService - Internal Command Filtering', () => { executeCommand('url', ['https://example.com']) executeCommand('getTitle', [], 'Page Title') // internal executeCommand('click', ['.button']) - executeCommand('waitUntil', [expect.any(Function)], true) // internal + executeCommand('waitUntil', [expect.any(Function)], true) // a user wait executeCommand('getText', ['.result'], 'Success') - // Only user commands (url, click, getText) should be captured - expect(mockSessionCapturerInstance.afterCommand).toHaveBeenCalledTimes(3) + // getTitle is internal; the rest — including the wait — are user actions. + expect(mockSessionCapturerInstance.afterCommand).toHaveBeenCalledTimes(4) const capturedCommands = mockSessionCapturerInstance.afterCommand.mock.calls.map( (call) => call[1] ) - expect(capturedCommands).toEqual(['url', 'click', 'getText']) + expect(capturedCommands).toEqual(['url', 'click', 'waitUntil', 'getText']) expect(capturedCommands).not.toContain('getTitle') - expect(capturedCommands).not.toContain('waitUntil') + }) + + it('captures a wait once and suppresses the commands it polls', () => { + // waitUntil opens; its predicate polls isDisplayed while the wait is + // still on the stack, so those polls are top-level-suppressed. + service.beforeCommand('waitUntil' as any, [expect.any(Function)]) + service.beforeCommand('isDisplayed' as any, []) + service.afterCommand('isDisplayed' as any, [], false) + service.beforeCommand('isDisplayed' as any, []) + service.afterCommand('isDisplayed' as any, [], true) + service.afterCommand('waitUntil' as any, [expect.any(Function)], true) + + const captured = mockSessionCapturerInstance.afterCommand.mock.calls.map( + (call) => call[1] + ) + expect(captured).toEqual(['waitUntil']) }) // Service-fired commands (preload injection, Puppeteer handle for CDP, @@ -269,3 +280,65 @@ describe('DevtoolsService - Screencast Integration', () => { expect(mockScreencastRecorder.start).toHaveBeenCalledWith(mockBrowser) }) }) + +describe('DevtoolsService - assertion suppression self-heal', () => { + let service: DevToolsHookService + const mockBrowser = { + isBidi: true, + sessionId: 'heal-session', + scriptAddPreloadScript: vi.fn().mockResolvedValue(undefined), + takeScreenshot: vi.fn().mockResolvedValue('screenshot'), + execute: vi.fn().mockResolvedValue({ + width: 1200, + height: 800, + offsetLeft: 0, + offsetTop: 0 + }), + on: vi.fn(), + emit: vi.fn() + } as any + + const capturedCommands = () => + mockSessionCapturerInstance.afterCommand.mock.calls.map((call) => call[1]) + + beforeEach(async () => { + vi.clearAllMocks() + service = new DevToolsHookService() + await service.before({} as any, [], mockBrowser) + vi.clearAllMocks() + }) + + // expect-webdriverio doesn't wrap afterAssertion in try/finally, so a matcher + // whose internal command hard-throws runs beforeAssertion but skips + // afterAssertion, leaving #assertionDepth stuck ≥ 1. Without the self-heal + // that stuck depth suppresses every later user command in the test. + it('captures a top-level command after a matcher throw left the assertion depth stuck', async () => { + service.beforeAssertion() + service.beforeAssertion() // depth now 2, no matching afterAssertion + + // The mocked stack is a user-spec frame with no expect-webdriverio frame, + // so the command is recognised as top-level and the leftover depth resets. + service.beforeCommand('click' as any, ['.button']) + await service.afterCommand('click' as any, ['.button'], undefined) + + expect(capturedCommands()).toEqual(['click']) + }) + + it('a fresh beforeAssertion resets a stuck depth so the window stays balanced', async () => { + service.beforeAssertion() + service.beforeAssertion() // leftover depth from a prior throw + + // A new matcher starts (resetting the stuck depth) and completes normally. + service.beforeAssertion() + await service.afterAssertion({ + matcherName: 'toBeDisplayed', + result: { pass: true } + } as any) + + // Depth is balanced again, so the next top-level command is captured. + service.beforeCommand('setValue' as any, ['.input', 'hi']) + await service.afterCommand('setValue' as any, ['.input', 'hi'], undefined) + + expect(capturedCommands()).toEqual(['setValue']) + }) +}) diff --git a/packages/service/tests/reporter.test.ts b/packages/service/tests/reporter.test.ts index 03461046..bdaccdca 100644 --- a/packages/service/tests/reporter.test.ts +++ b/packages/service/tests/reporter.test.ts @@ -324,4 +324,76 @@ describe('TestReporter - Rerun & Stable UID', () => { expect(() => reporter.report).not.toThrow() }) }) + + describe('Cucumber step uid scoping (no cross-scenario collision)', () => { + const FEATURE = '/proj/features/login.feature' + const STEP = + 'I should see a flash message saying You logged into a secure area!' + + const scenario = (title: string, line: number): SuiteStats => + ({ + uid: `raw-${title}`, + title, + fullTitle: `Login ${title}`, + file: FEATURE, + type: 'scenario', + argument: { uri: FEATURE, line } + }) as unknown as SuiteStats + + const step = (line: number): TestStats => + ({ + uid: 'raw-step', + title: STEP, + fullTitle: STEP, + file: FEATURE, + type: 'test', + argument: { uri: FEATURE, line } + }) as unknown as TestStats + + // Drive a scenario's step through the reporter and return the assigned uid. + const runStep = ( + r: TestReporter, + scen: SuiteStats, + stepStats: TestStats + ): string => { + r.onSuiteStart(scen) + r.onTestStart(stepStats) + r.onTestEnd(stepStats) + r.onSuiteEnd(scen) + return stepStats.uid + } + + it('gives identical step text in sibling scenarios distinct uids', () => { + const scenA = scenario('Scenario A', 5) + const scenB = scenario('Scenario B', 20) + const uidA = runStep(reporter, scenA, step(8)) + const uidB = runStep(reporter, scenB, step(23)) + expect(uidA).not.toBe(uidB) + }) + + it('keeps a step uid stable when its scenario is rerun alone', () => { + // Full run: scenario A then B. + const uidBFull = (() => { + runStep(reporter, scenario('Scenario A', 5), step(8)) + return runStep(reporter, scenario('Scenario B', 20), step(23)) + })() + + // Rerun scenario B on its own (fresh reporter resets the counter). A + // run-order-counter uid would shift to A's slot here; scoping by the + // scenario keeps it stable. + const reporter2 = new TestReporter( + { logFile: '/tmp/test.log' }, + sendUpstream as any + ) + const uidBAlone = runStep(reporter2, scenario('Scenario B', 20), step(23)) + + expect(uidBAlone).toBe(uidBFull) + }) + + it('distinguishes scenario-outline example rows (same title, different line)', () => { + const row1 = runStep(reporter, scenario('greet <name>', 10), step(11)) + const row2 = runStep(reporter, scenario('greet <name>', 14), step(15)) + expect(row1).not.toBe(row2) + }) + }) }) diff --git a/packages/service/tests/trace-granularity.test.ts b/packages/service/tests/trace-granularity.test.ts new file mode 100644 index 00000000..c43b6b66 --- /dev/null +++ b/packages/service/tests/trace-granularity.test.ts @@ -0,0 +1,240 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import type * as DevtoolsCore from '@wdio/devtools-core' +import { deterministicUid, type SpecRange } from '@wdio/devtools-core' +import { findCurrentTestRange } from '../src/trace-slices.js' + +// Records the key/state observed at each per-slice flush and replays the real +// dedupe (flushed.add) so recordSliceBoundary's prev-slice logic behaves as in +// production — otherwise a boundary change would re-flush an already-flushed +// slice. Capturing state at call time is what proves the eager flush sees this +// attempt's outcome before a retry's beforeTest overwrites the entry. +const flushedSlices: Array<{ key: string; testUid?: string; state?: string }> = + [] + +type FlushCtx = { + flushed: Set<string> + testMetadata: Map<string, { state?: string }> +} + +const flushRangeTrace = vi.fn( + (ctx: FlushCtx, range: { key: string; testUid?: string }) => { + ctx.flushed.add(range.key) + flushedSlices.push({ + key: range.key, + testUid: range.testUid, + state: range.testUid + ? ctx.testMetadata.get(range.testUid)?.state + : undefined + }) + return Promise.resolve(undefined) + } +) + +const finalizeTraceExport = vi.fn().mockResolvedValue([]) + +vi.mock('stack-trace', () => ({ parse: () => [] })) + +const mockSessionCapturerInstance = { + afterCommand: vi.fn(), + sendUpstream: vi.fn(), + injectScript: vi.fn().mockResolvedValue(undefined), + captureAssertCommand: vi.fn(), + failLastAction: vi.fn(), + resetLastSelector: vi.fn(), + resetRetryTracker: vi.fn(), + cleanup: vi.fn(), + commandsLog: [] as unknown[], + sources: new Map(), + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + metadata: { url: 'http://test.com', viewport: {} } +} + +vi.mock('../src/session.js', () => ({ + SessionCapturer: vi.fn(function (this: unknown) { + return mockSessionCapturerInstance + }) +})) + +vi.mock('../src/action-snapshot.js', () => ({ + captureActionSnapshot: vi.fn().mockResolvedValue(null), + captureActionResult: vi.fn().mockResolvedValue(undefined), + waitForActionResult: vi.fn().mockResolvedValue(undefined) +})) + +vi.mock('@wdio/devtools-core', async (importOriginal) => { + const actual = await importOriginal<typeof DevtoolsCore>() + return { + ...actual, + finalizeTraceExport: (ctx: unknown) => finalizeTraceExport(ctx), + // The adapter now flushes via core's flushRangeLogged wrapper; route it to + // the same spy so call-count/argument assertions still observe each flush. + flushRangeLogged: (ctx: unknown, range: unknown) => + flushRangeTrace( + ctx as FlushCtx, + range as { key: string; testUid?: string } + ), + flushRangeTrace: (ctx: unknown, range: unknown) => + flushRangeTrace( + ctx as FlushCtx, + range as { key: string; testUid?: string } + ) + } +}) + +// Imported after the mocks are declared so the mocked core module is used. +const { default: DevToolsHookService } = await import('../src/index.js') + +describe('findCurrentTestRange', () => { + const mk = (key: string, testUid?: string): SpecRange => ({ + specFile: 'f', + key, + testUid, + commandStartIdx: 0, + consoleStartIdx: 0, + networkStartIdx: 0, + mutationStartIdx: 0, + traceLogStartIdx: 0, + snapshotCount: 0 + }) + + it('returns the most recent range recorded under the base testUid', () => { + const ranges = [mk('a', 'a'), mk('b', 'b'), mk('b-retry1', 'b')] + // A retry pushes a new range under the same testUid; the latest one wins. + expect(findCurrentTestRange(ranges, 'b')?.key).toBe('b-retry1') + expect(findCurrentTestRange(ranges, 'a')?.key).toBe('a') + }) + + it('returns undefined when no range matches (spec/session slices)', () => { + expect( + findCurrentTestRange([mk('spec.ts', undefined)], 'x') + ).toBeUndefined() + expect(findCurrentTestRange([], 'x')).toBeUndefined() + }) +}) + +describe('DevtoolsService - trace granularity slicing', () => { + const file = '/proj/specs/login.spec.ts' + const mockBrowser = { + isBidi: true, + sessionId: 'sess-1', + scriptAddPreloadScript: vi.fn().mockResolvedValue(undefined), + takeScreenshot: vi.fn().mockResolvedValue('shot'), + execute: vi + .fn() + .mockResolvedValue({ width: 1, height: 1, offsetLeft: 0, offsetTop: 0 }), + on: vi.fn(), + emit: vi.fn(), + options: { rootDir: '/proj' }, + capabilities: { browserName: 'chrome' } + } as never + + beforeEach(() => { + vi.clearAllMocks() + finalizeTraceExport.mockResolvedValue([]) + flushedSlices.length = 0 + }) + + async function newService(granularity: 'session' | 'spec' | 'test') { + const service = new DevToolsHookService({ + mode: 'trace', + tracePolicy: 'retain-on-failure', + traceGranularity: granularity + }) + await service.before({} as never, [], mockBrowser) + return service + } + + it('test granularity: records a per-test slice at beforeTest and eager-flushes it at afterTest', async () => { + const service = await newService('test') + const title = 'login works' + const uid = deterministicUid(file, title) + + service.beforeTest({ file, fullTitle: title }) + expect(flushRangeTrace).not.toHaveBeenCalled() + + await service.afterTest({ file, fullTitle: title }, {}, { passed: true }) + + expect(flushRangeTrace).toHaveBeenCalledTimes(1) + // The flushed range is the one recorded at beforeTest (found via testUid), + // proving both the start-boundary and the eager end-flush. + expect(flushRangeTrace.mock.calls[0]![1]).toMatchObject({ + key: uid, + testUid: uid + }) + expect(flushedSlices).toEqual([{ key: uid, testUid: uid, state: 'passed' }]) + }) + + it('test granularity: a retried test produces a distinct retry slice, each with its own attempt outcome', async () => { + const service = await newService('test') + const title = 'flaky login' + const uid = deterministicUid(file, title) + + // Attempt 1 fails, then a same-process retry (Mocha) passes. + service.beforeTest({ file, fullTitle: title }) + await service.afterTest( + { file, fullTitle: title }, + {}, + { passed: false, error: new Error('boom') } + ) + service.beforeTest({ file, fullTitle: title }) + await service.afterTest({ file, fullTitle: title }, {}, { passed: true }) + + expect(flushRangeTrace).toHaveBeenCalledTimes(2) + // Each attempt is its own slice, and each slice was written with that + // attempt's just-stamped state — the failed first attempt survives the + // retry's beforeTest overwrite (the retain-on-first-failure fix). + expect(flushedSlices).toEqual([ + { key: uid, testUid: uid, state: 'failed' }, + { key: `${uid}-retry1`, testUid: uid, state: 'passed' } + ]) + }) + + it('test granularity: Cucumber scenario eager-flushes its slice at afterScenario', async () => { + const service = await newService('test') + const uri = '/proj/features/login.feature' + const name = 'log in' + const uid = deterministicUid(uri, name) + + service.beforeScenario({ pickle: { uri, name } }) + await service.afterScenario({ pickle: { uri, name } }, { passed: true }) + + expect(flushRangeTrace).toHaveBeenCalledTimes(1) + expect(flushRangeTrace.mock.calls[0]![1]).toMatchObject({ + key: uid, + testUid: uid + }) + }) + + it('spec granularity: no eager flush at afterTest; flushes only when the spec file changes', async () => { + const service = await newService('spec') + const other = '/proj/specs/cart.spec.ts' + + service.beforeTest({ file, fullTitle: 't1' }) + await service.afterTest({ file, fullTitle: 't1' }, {}, { passed: true }) + // Two tests in the same spec: neither the end-flush nor a boundary fires. + service.beforeTest({ file, fullTitle: 't2' }) + await service.afterTest({ file, fullTitle: 't2' }, {}, { passed: true }) + expect(flushRangeTrace).not.toHaveBeenCalled() + + // A new spec file flushes the previous spec's slice (fire-and-forget). + service.beforeTest({ file: other, fullTitle: 't3' }) + expect(flushRangeTrace).toHaveBeenCalledTimes(1) + const range = flushRangeTrace.mock.calls[0]![1] as SpecRange + expect(range.key).toBe(file) + expect(range.testUid).toBeUndefined() + }) + + it('session granularity: records no slices and never flushes per test', async () => { + const service = await newService('session') + + service.beforeTest({ file, fullTitle: 't1' }) + await service.afterTest({ file, fullTitle: 't1' }, {}, { passed: true }) + service.beforeTest({ file, fullTitle: 't2' }) + await service.afterTest({ file, fullTitle: 't2' }, {}, { passed: false }) + + expect(flushRangeTrace).not.toHaveBeenCalled() + }) +}) diff --git a/packages/service/tests/trace-metadata.test.ts b/packages/service/tests/trace-metadata.test.ts new file mode 100644 index 00000000..9b2ee653 --- /dev/null +++ b/packages/service/tests/trace-metadata.test.ts @@ -0,0 +1,235 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import type * as DevtoolsCore from '@wdio/devtools-core' +import { deterministicUid } from '@wdio/devtools-core' +import { + cucumberScenarioUid, + resultToState, + testMetadataUid +} from '../src/test-metadata.js' + +// Captures the ctx handed to finalizeTraceExport so the test can inspect the +// state stamped onto testMetadata and the policy that flowed in. +const finalizeTraceExport = vi.fn().mockResolvedValue([]) + +vi.mock('stack-trace', () => ({ parse: () => [] })) + +const mockSessionCapturerInstance = { + afterCommand: vi.fn(), + sendUpstream: vi.fn(), + injectScript: vi.fn().mockResolvedValue(undefined), + captureAssertCommand: vi.fn(), + failLastAction: vi.fn(), + resetLastSelector: vi.fn(), + resetRetryTracker: vi.fn(), + cleanup: vi.fn(), + commandsLog: [] as unknown[], + sources: new Map(), + mutations: [], + traceLogs: [], + consoleLogs: [], + networkRequests: [], + metadata: { url: 'http://test.com', viewport: {} } +} + +vi.mock('../src/session.js', () => ({ + SessionCapturer: vi.fn(function (this: unknown) { + return mockSessionCapturerInstance + }) +})) + +// Keep the after* hooks from touching a real browser/CDP. +vi.mock('../src/action-snapshot.js', () => ({ + captureActionSnapshot: vi.fn().mockResolvedValue(null), + captureActionResult: vi.fn().mockResolvedValue(undefined), + waitForActionResult: vi.fn().mockResolvedValue(undefined) +})) + +vi.mock('@wdio/devtools-core', async (importOriginal) => { + const actual = await importOriginal<typeof DevtoolsCore>() + return { + ...actual, + finalizeTraceExport: (ctx: unknown) => finalizeTraceExport(ctx) + } +}) + +// Imported after the mocks are declared so the mocked core module is used. +const { default: DevToolsHookService } = await import('../src/index.js') + +describe('test-metadata helpers', () => { + it('resultToState maps a WDIO result to the canonical state', () => { + expect(resultToState({ passed: true })).toBe('passed') + expect(resultToState({ passed: false })).toBe('failed') + expect(resultToState({ passed: false, skipped: true })).toBe('skipped') + // skipped wins even if passed is somehow set alongside it. + expect(resultToState({ passed: true, skipped: true })).toBe('skipped') + }) + + it('testMetadataUid keys on file+title, falling back to title alone', () => { + const file = '/proj/specs/a.spec.ts' + expect(testMetadataUid(file, 'renders')).toBe( + deterministicUid(file, 'renders') + ) + expect(testMetadataUid(undefined, 'renders')).toBe('renders') + }) + + it('cucumberScenarioUid separates outline rows sharing a name by astNodeIds', () => { + const uri = '/proj/features/login.feature' + const row1 = cucumberScenarioUid(uri, 'log in', ['node-1']) + const row2 = cucumberScenarioUid(uri, 'log in', ['node-2']) + // Distinct example rows → distinct uids (so they render as separate groups). + expect(row1).not.toBe(row2) + // A rerun of the same row → same uid (retry-coalescing stays intact). + expect(cucumberScenarioUid(uri, 'log in', ['node-1'])).toBe(row1) + // No astNodeIds → plain name-based uid. + expect(cucumberScenarioUid(uri, 'log in')).toBe( + deterministicUid(uri, 'log in') + ) + }) +}) + +describe('DevtoolsService - afterTest state stamping', () => { + const file = '/proj/specs/login.spec.ts' + const mockBrowser = { + isBidi: true, + sessionId: 'sess-1', + scriptAddPreloadScript: vi.fn().mockResolvedValue(undefined), + takeScreenshot: vi.fn().mockResolvedValue('shot'), + execute: vi + .fn() + .mockResolvedValue({ width: 1, height: 1, offsetLeft: 0, offsetTop: 0 }), + on: vi.fn(), + emit: vi.fn(), + options: { rootDir: '/proj' }, + capabilities: { browserName: 'chrome' } + } as never + + beforeEach(() => { + vi.clearAllMocks() + finalizeTraceExport.mockResolvedValue([]) + }) + + async function runTest(title: string, result: Record<string, unknown>) { + const service = new DevToolsHookService({ + mode: 'trace', + tracePolicy: 'retain-on-failure' + }) + await service.before({} as never, [], mockBrowser) + service.beforeTest({ file, fullTitle: title }) + await service.afterTest({ file, fullTitle: title }, {}, result) + await service.after() + const ctx = finalizeTraceExport.mock.calls.at(-1)?.[0] as { + policy?: string + testMetadata: Map<string, { state?: string }> + } + return ctx + } + + it('stamps passed / failed / skipped from the WDIO result', async () => { + const passed = await runTest('login works', { passed: true }) + expect( + passed.testMetadata.get(deterministicUid(file, 'login works'))?.state + ).toBe('passed') + + const failed = await runTest('login fails', { + passed: false, + error: new Error('boom') + }) + expect( + failed.testMetadata.get(deterministicUid(file, 'login fails'))?.state + ).toBe('failed') + + const skipped = await runTest('login skipped', { + passed: false, + skipped: true + }) + expect( + skipped.testMetadata.get(deterministicUid(file, 'login skipped'))?.state + ).toBe('skipped') + }) + + it('flows the tracePolicy and a failed state into the finalizer ctx', async () => { + const ctx = await runTest('checkout fails', { + passed: false, + error: new Error('x') + }) + // Both halves of "retain-on-failure is no longer a no-op for WDIO": + // the policy reached the finalizer, and the failing state is on the entry + // its retention evaluator reads. + expect(ctx.policy).toBe('retain-on-failure') + expect( + ctx.testMetadata.get(deterministicUid(file, 'checkout fails'))?.state + ).toBe('failed') + }) + + it('flags attemptInfoAvailable so retry-aware policies use per-test attempt', async () => { + const ctx = (await runTest('trace opts', { passed: true })) as { + attemptInfoAvailable?: boolean + } + expect(ctx.attemptInfoAvailable).toBe(true) + }) + + it('uses the tracker attempt even when WDIO reports retries.attempts:0', async () => { + const service = new DevToolsHookService({ + mode: 'trace', + tracePolicy: 'retain-on-failure-and-retries' + }) + await service.before({} as never, [], mockBrowser) + const title = 'flaky login' + // Two starts before a single end simulate a same-process (Mocha) retry. + // WDIO's mocha framework reports retries.attempts:0 even on the retry, so + // the runner field must NOT clobber the tracker's real count of 1. + service.beforeTest({ file, fullTitle: title }) + service.beforeTest({ file, fullTitle: title }) + await service.afterTest( + { file, fullTitle: title }, + {}, + { passed: true, retries: { attempts: 0 } } + ) + await service.after() + const ctx = finalizeTraceExport.mock.calls.at(-1)?.[0] as { + testMetadata: Map<string, { attempt?: number }> + } + expect(ctx.testMetadata.get(deterministicUid(file, title))?.attempt).toBe(1) + }) + + it('uses the runner attempts count when it exceeds the tracker', async () => { + const service = new DevToolsHookService({ + mode: 'trace', + tracePolicy: 'retain-on-failure-and-retries' + }) + await service.before({} as never, [], mockBrowser) + const title = 'checkout retries' + service.beforeTest({ file, fullTitle: title }) + await service.afterTest( + { file, fullTitle: title }, + {}, + { passed: true, retries: { attempts: 3 } } + ) + await service.after() + const ctx = finalizeTraceExport.mock.calls.at(-1)?.[0] as { + testMetadata: Map<string, { attempt?: number }> + } + expect(ctx.testMetadata.get(deterministicUid(file, title))?.attempt).toBe(3) + }) + + it('stamps a Cucumber scenario state in afterScenario', async () => { + const service = new DevToolsHookService({ + mode: 'trace', + tracePolicy: 'retain-on-failure' + }) + await service.before({} as never, [], mockBrowser) + const uri = '/proj/features/cart.feature' + service.beforeScenario({ pickle: { uri, name: 'add to cart' } }) + await service.afterScenario( + { pickle: { uri, name: 'add to cart' } }, + { passed: false } + ) + await service.after() + const ctx = finalizeTraceExport.mock.calls.at(-1)?.[0] as { + testMetadata: Map<string, { state?: string }> + } + expect( + ctx.testMetadata.get(deterministicUid(uri, 'add to cart'))?.state + ).toBe('failed') + }) +}) diff --git a/packages/shared/src/trace-actions.ts b/packages/shared/src/trace-actions.ts index e971b53f..c3bd7462 100644 --- a/packages/shared/src/trace-actions.ts +++ b/packages/shared/src/trace-actions.ts @@ -7,6 +7,41 @@ export interface TraceAction { method: string } +/** Trace action class assertion commands map to. */ +export const ASSERT_ACTION_CLASS = 'Assert' + +/** node:assert methods the core assert patcher wraps; the reader derives its + * `Assert.<m>` → `assert.<m>` reverse entries from the same list. */ +export const TRACKED_ASSERT_METHODS = [ + 'equal', + 'strictEqual', + 'deepEqual', + 'deepStrictEqual', + 'notEqual', + 'notStrictEqual', + 'notDeepEqual', + 'notDeepStrictEqual', + 'ok', + 'fail', + 'throws', + 'doesNotThrow', + 'rejects', + 'doesNotReject', + 'match', + 'doesNotMatch' +] as const + +// assert.<m> (node:assert), verify.<m> (nightwatch soft variants), and +// expect.<m> (synthesized failing-matcher entries) all render as Assert. +// Only FAILING expect-webdriverio matchers reach the command log today (via the +// reporter); recording passing matchers needs a per-adapter capture change. +const ASSERT_COMMAND_RE = /^(?:assert|verify|expect)\.(\w+)$/ + +export function mapAssertCommand(command: string): TraceAction | null { + const match = ASSERT_COMMAND_RE.exec(command) + return match ? { class: ASSERT_ACTION_CLASS, method: match[1] } : null +} + export const ACTION_MAP: Record<string, TraceAction> = { // WDIO browser-level url: { class: 'Page', method: 'navigate' }, @@ -35,5 +70,38 @@ export const ACTION_MAP: Record<string, TraceAction> = { execute: { class: 'Page', method: 'evaluate' }, executeAsync: { class: 'Page', method: 'evaluate' }, switchToFrame: { class: 'Frame', method: 'goto' }, - touchAction: { class: 'Element', method: 'tap' } + touchAction: { class: 'Element', method: 'tap' }, + // WDIO element reads — surfaced so query steps appear in the timeline the way + // locator queries do in standard trace viewers. Adapters already capture + // these; only the export allow-list kept them out. + getText: { class: 'Element', method: 'getText' }, + getValue: { class: 'Element', method: 'getValue' }, + getAttribute: { class: 'Element', method: 'getAttribute' }, + getProperty: { class: 'Element', method: 'getProperty' }, + getCSSProperty: { class: 'Element', method: 'getCSSProperty' }, + getTagName: { class: 'Element', method: 'getTagName' }, + getLocation: { class: 'Element', method: 'getLocation' }, + getSize: { class: 'Element', method: 'getSize' }, + isDisplayed: { class: 'Element', method: 'isDisplayed' }, + isExisting: { class: 'Element', method: 'isExisting' }, + isEnabled: { class: 'Element', method: 'isEnabled' }, + isSelected: { class: 'Element', method: 'isSelected' }, + isClickable: { class: 'Element', method: 'isClickable' }, + isFocused: { class: 'Element', method: 'isFocused' }, + // Explicit user-facing waits (not the internal polling loops behind them). + waitForDisplayed: { class: 'Element', method: 'waitForDisplayed' }, + waitForExist: { class: 'Element', method: 'waitForExist' }, + waitForEnabled: { class: 'Element', method: 'waitForEnabled' }, + waitForClickable: { class: 'Element', method: 'waitForClickable' }, + waitUntil: { class: 'Browser', method: 'waitForFunction' }, + // WDIO page/browser reads + getTitle: { class: 'Page', method: 'getTitle' }, + getUrl: { class: 'Page', method: 'getUrl' }, + getPageSource: { class: 'Page', method: 'getPageSource' }, + // Selenium read aliases — normalized onto the WDIO names above so both runners + // read identically. getText/getAttribute/getTagName/isDisplayed/isEnabled/ + // isSelected share the command name across runners and need no alias. + getCssValue: { class: 'Element', method: 'getCSSProperty' }, + getRect: { class: 'Element', method: 'getRect' }, + getCurrentUrl: { class: 'Page', method: 'getUrl' } } diff --git a/packages/shared/src/trace-player.ts b/packages/shared/src/trace-player.ts index 4001255d..ef0eaafc 100644 --- a/packages/shared/src/trace-player.ts +++ b/packages/shared/src/trace-player.ts @@ -18,6 +18,25 @@ export interface TracePlayerFrame { screenshot: string } +/** A structural step reconstructed from a trace.zip — a runner hook, step + * wrapper, or tracing group marker — rendered as a collapsible tree row. */ +export interface TraceActionGroupNode { + callId: string + title: string + /** Absolute wall-clock ms when the step opened. */ + startTime: number + /** Absolute wall-clock ms when the step closed. */ + endTime: number + /** Set when the step's own after errored or any descendant failed. */ + failed?: boolean + children: TraceActionChild[] +} + +/** One tree slot: a nested group or an index into `trace.commands`. */ +export type TraceActionChild = + | { group: TraceActionGroupNode } + | { commandIndex: number } + /** Payload served at `TRACE_API.get`. Carries the reconstructed TraceLog plus * the frame filmstrip and clock window the player's timeline needs. */ export interface TracePlayerData { @@ -27,4 +46,7 @@ export interface TracePlayerData { startTime: number /** Total span in ms from the first event to the last. */ duration: number + /** Root children of the action tree, chronological. Absent when the zip + * carried no structural steps — the player then renders the flat list. */ + groups?: TraceActionChild[] } diff --git a/packages/shared/src/types.ts b/packages/shared/src/types.ts index 23ad3c7f..f90db146 100644 --- a/packages/shared/src/types.ts +++ b/packages/shared/src/types.ts @@ -4,7 +4,15 @@ // these shapes. The backend stores and forwards them. The app consumes them. // See ARCHITECTURE.md §2 and CLAUDE.md §2.1. -export type LogLevel = 'trace' | 'debug' | 'log' | 'info' | 'warn' | 'error' +export const LOG_LEVELS = [ + 'trace', + 'debug', + 'log', + 'info', + 'warn', + 'error' +] as const +export type LogLevel = (typeof LOG_LEVELS)[number] /** Where a captured ConsoleLog entry originated. */ export type LogSource = 'browser' | 'test' | 'terminal' @@ -21,17 +29,64 @@ export type DevToolsMode = 'live' | 'trace' /** `zip` (default) writes a single `trace-<id>.zip`; `ndjson-directory` writes * the same `trace.trace` + `trace.network` + `resources/` layout unpacked - * into `trace-<id>/`. Both are consumable by `playwright show-trace` — the - * unpacked form skips the unzip step for agentic / scripted consumers. */ + * into `trace-<id>/`. Both open in any standard trace viewer — the unpacked + * form skips the unzip step for agentic / scripted consumers. */ export type TraceFormat = 'zip' | 'ndjson-directory' /** `session` (default) writes one trace per worker session; `spec` writes one - * trace per spec file, keyed on the spec's filename. Only applies in trace mode. */ -export type TraceGranularity = 'session' | 'spec' + * trace per spec file, keyed on the spec's filename; `test` writes one trace + * per test. Only applies in trace mode. */ +export type TraceGranularity = 'session' | 'spec' | 'test' + +/** Retention policy for written traces. Only applies in trace mode; `on` is + * the current always-write behavior (there is no `off` — that's simply not + * using trace mode). */ +export type TraceRetentionPolicy = + | 'on' + | 'retain-on-failure' + | 'retain-on-first-failure' + | 'on-first-retry' + | 'on-all-retries' + | 'retain-on-failure-and-retries' + +/** Video retention — video has no separate mode switch, so `off` is valid + * (and the default). */ +export type TraceVideoPolicy = 'off' | TraceRetentionPolicy + +/** One node in a test's ancestor chain, outermost first. */ +export interface TestAncestor { + uid: string + title: string + kind: 'feature' | 'scenario' | 'suite' | 'test' | 'step' | 'hook' +} + +/** Per-test metadata for Tracing.tracingGroup events in trace output. */ +export interface TestMetadataEntry { + title: string + specFile: string + state?: TestStatus + attempt?: number + ancestry?: TestAncestor[] +} /** Test metadata keyed by testUid — maps stable test IDs to human-readable * title + specFile for Tracing.tracingGroup events in trace output. */ -export type TestMetadataMap = Map<string, { title: string; specFile: string }> +export type TestMetadataMap = Map<string, TestMetadataEntry> + +/** Defaults for trace-mode options when not specified by the user. */ +export const TRACE_DEFAULTS = { + mode: 'live', + traceFormat: 'zip', + traceGranularity: 'session', + tracePolicy: 'on', + video: 'off' +} as const satisfies { + mode: DevToolsMode + traceFormat: TraceFormat + traceGranularity: TraceGranularity + tracePolicy: TraceRetentionPolicy + video: TraceVideoPolicy +} /** * Enum-style accessor for the canonical TestStatus values. Adapter code uses @@ -228,6 +283,53 @@ export const SCREENCAST_DEFAULTS: Required<ScreencastOptions> = { pollIntervalMs: 200 } +/** + * Options every framework adapter accepts. Each adapter's own options interface + * extends this and adds only its framework-specific fields (e.g. WDIO's + * devtoolsCapabilities, Selenium's openUi, Nightwatch's bidi). + */ +export interface BaseDevToolsOptions { + /** Port to launch the application on (default: random). */ + port?: number + /** Hostname to launch the application on. @default localhost */ + hostname?: string + /** Screencast recording options. When enabled, a continuous video of the + * browser session is recorded and saved as a .webm file. */ + screencast?: ScreencastOptions + /** Capture node:assert assertions (and framework `expect` matchers where + * supported) as first-class commands. Default true. */ + captureAssertions?: boolean + /** `live` (default) launches the DevTools UI; `trace` skips it. */ + mode?: DevToolsMode + /** Trace output layout — `zip` (default) writes a single archive, + * `ndjson-directory` unpacks into `trace-<id>/`. Only applies in trace mode. */ + traceFormat?: TraceFormat + /** Trace output granularity — `session` (default) writes one trace per + * worker session; `spec` writes one per spec file. Only applies in trace mode. */ + traceGranularity?: TraceGranularity + /** Trace retention policy — gates which traces are kept (e.g. + * `retain-on-failure`). Default `on` (keep all). Only applies in trace mode. */ + tracePolicy?: TraceRetentionPolicy +} + +/** Minimal Cucumber pickle-step shape — only the fields the adapters read. + * Cucumber's own types vary across versions, so we pin just these. */ +export interface CucumberPickleStep { + text?: string + astNodeIds?: string[] + location?: { line?: number } +} + +/** Minimal Cucumber pickle shape — only the fields the adapters read. `steps` + * is present only where the adapter walks step boundaries (Nightwatch). */ +export interface CucumberPickle { + name?: string + uri?: string + location?: { line?: number } + astNodeIds?: string[] + steps?: CucumberPickleStep[] +} + export interface Metadata { type: TraceType url?: string diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b338e0b8..528cbb11 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -109,7 +109,7 @@ importers: version: link:../../packages/nightwatch-devtools nightwatch: specifier: ^3.16.0 - version: 3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@148.0.4) + version: 3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@150.0.1) examples/selenium: dependencies: @@ -141,6 +141,9 @@ importers: '@wdio/local-runner': specifier: 9.28.0 version: 9.28.0(@wdio/globals@9.28.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/mocha-framework': + specifier: 9.28.0 + version: 9.28.0 '@wdio/spec-reporter': specifier: 9.28.0 version: 9.28.0 @@ -365,7 +368,7 @@ importers: version: 6.0.3 vitest: specifier: ^4.0.16 - version: 4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) + version: 4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) packages/nightwatch-devtools: dependencies: @@ -419,11 +422,11 @@ importers: specifier: workspace:^ version: link:../shared chromedriver: - specifier: ^148.0.4 - version: 148.0.4 + specifier: ^150.0.0 + version: 150.0.1 nightwatch: specifier: ^3.16.0 - version: 3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@148.0.4) + version: 3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@150.0.1) tsup: specifier: ^8.5.1 version: 8.5.1(@microsoft/api-extractor@7.58.9(@types/node@25.9.3))(jiti@2.7.0)(postcss@8.5.15)(tsx@4.22.4)(typescript@6.0.3)(yaml@2.9.0) @@ -1859,49 +1862,42 @@ packages: engines: {node: '>= 10'} cpu: [arm64] os: [linux] - libc: [glibc] '@napi-rs/nice-linux-arm64-musl@1.1.1': resolution: {integrity: sha512-+2Rzdb3nTIYZ0YJF43qf2twhqOCkiSrHx2Pg6DJaCPYhhaxbLcdlV8hCRMHghQ+EtZQWGNcS2xF4KxBhSGeutg==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] - libc: [musl] '@napi-rs/nice-linux-ppc64-gnu@1.1.1': resolution: {integrity: sha512-4FS8oc0GeHpwvv4tKciKkw3Y4jKsL7FRhaOeiPei0X9T4Jd619wHNe4xCLmN2EMgZoeGg+Q7GY7BsvwKpL22Tg==} engines: {node: '>= 10'} cpu: [ppc64] os: [linux] - libc: [glibc] '@napi-rs/nice-linux-riscv64-gnu@1.1.1': resolution: {integrity: sha512-HU0nw9uD4FO/oGCCk409tCi5IzIZpH2agE6nN4fqpwVlCn5BOq0MS1dXGjXaG17JaAvrlpV5ZeyZwSon10XOXw==} engines: {node: '>= 10'} cpu: [riscv64] os: [linux] - libc: [glibc] '@napi-rs/nice-linux-s390x-gnu@1.1.1': resolution: {integrity: sha512-2YqKJWWl24EwrX0DzCQgPLKQBxYDdBxOHot1KWEq7aY2uYeX+Uvtv4I8xFVVygJDgf6/92h9N3Y43WPx8+PAgQ==} engines: {node: '>= 10'} cpu: [s390x] os: [linux] - libc: [glibc] '@napi-rs/nice-linux-x64-gnu@1.1.1': resolution: {integrity: sha512-/gaNz3R92t+dcrfCw/96pDopcmec7oCcAQ3l/M+Zxr82KT4DljD37CpgrnXV+pJC263JkW572pdbP3hP+KjcIg==} engines: {node: '>= 10'} cpu: [x64] os: [linux] - libc: [glibc] '@napi-rs/nice-linux-x64-musl@1.1.1': resolution: {integrity: sha512-xScCGnyj/oppsNPMnevsBe3pvNaoK7FGvMjT35riz9YdhB2WtTG47ZlbxtOLpjeO9SqqQ2J2igCmz6IJOD5JYw==} engines: {node: '>= 10'} cpu: [x64] os: [linux] - libc: [musl] '@napi-rs/nice-openharmony-arm64@1.1.1': resolution: {integrity: sha512-6uJPRVwVCLDeoOaNyeiW0gp2kFIM4r7PL2MczdZQHkFi9gVlgm+Vn+V6nTWRcu856mJ2WjYJiumEajfSm7arPQ==} @@ -2027,42 +2023,36 @@ packages: engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] - libc: [glibc] '@rolldown/binding-linux-arm64-musl@1.0.3': resolution: {integrity: sha512-VWkUHwWriDciit80wleYwKILoR/KMvxh/IdwS/paX+ZgpuRpCrKLUdadJbc0NpBEiyhpYawsJ73j9aCvOH+f7Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] - libc: [musl] '@rolldown/binding-linux-ppc64-gnu@1.0.3': resolution: {integrity: sha512-5f1laC0SlIR0yDbFCd8acUhvJIag6N3zC5P7oUPN6wX0aOma+uKJ0wBDH5aq7I1PVI2ttTlhJwzwRIBnLiSGEg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] - libc: [glibc] '@rolldown/binding-linux-s390x-gnu@1.0.3': resolution: {integrity: sha512-Iq4ko0r4XsgbrF/LunNgHtAGLRRVE2kXonAXQ/MV0mC6jQpMOhW1SvtZja2EhC/kd05++bP78dsqBeIQyYJ6Yg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] - libc: [glibc] '@rolldown/binding-linux-x64-gnu@1.0.3': resolution: {integrity: sha512-B8m6tD5+/N5FeNQFbKlLA/2yVq9ycQP1SeedyEYYKWBNR3ZQbkvIUcNnDNM03lO1l5F2roiiFJGgvoLLyZXtSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] - libc: [glibc] '@rolldown/binding-linux-x64-musl@1.0.3': resolution: {integrity: sha512-pSdpdUJHkuCxun9LE7jvgUB9qsRgaiyNNCX7m/AvHTcq67AiT/Yhoxvw5zPfhrM8k/BfP8ce/hMOpthKDpEUow==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] - libc: [musl] '@rolldown/binding-openharmony-arm64@1.0.3': resolution: {integrity: sha512-OXXS3RKJgX2uLwM+gYyuH5omcH8fL1LJs96pZGgtetVCahON57+d4SJHzTgZiOjxgGkSnpXpOsWuPDGAKAigEg==} @@ -2133,79 +2123,66 @@ packages: resolution: {integrity: sha512-T1dMEQhXA/jkJ/jyMIw9IovK8bSUq7A8kLIlvZTb/6YIVsp2zLavr4F3oyllHWo7eIVJRyE5n3tUjQJEbE1IuQ==} cpu: [arm] os: [linux] - libc: [glibc] '@rollup/rollup-linux-arm-musleabihf@4.62.0': resolution: {integrity: sha512-2as0LgT7qQpyceQq6VUJYnumUMUrgGQCWIiDIN9DE0/tglsk6o66uCB4f3djRawAltvfCNLyZZrsqbPA6inCsA==} cpu: [arm] os: [linux] - libc: [musl] '@rollup/rollup-linux-arm64-gnu@4.62.0': resolution: {integrity: sha512-bVURMg+6eNN9C/yc0aVjooZcwTTtYF4YW3xta5pP0//r3o1V8gXEHXWCndj47w/HhwsFroZrFhR+6uQP5T0n0g==} cpu: [arm64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-arm64-musl@4.62.0': resolution: {integrity: sha512-Ful8pM/2yYI83PViWdFdpZhdI8HJ5qsXANe5atypbHDf+KIBBDsZsbyy8hbXnULVvW9NsTh5DHwbcBftyLTfiw==} cpu: [arm64] os: [linux] - libc: [musl] '@rollup/rollup-linux-loong64-gnu@4.62.0': resolution: {integrity: sha512-9Gp/DgrkzfUBmNPVTyPTvay+4xEP7M/clXpj3efXBcm6uTIVIgDg4rqUpqKXvLEuFRVuEpSAOkhgNeecvaZ4Cg==} cpu: [loong64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-loong64-musl@4.62.0': resolution: {integrity: sha512-m9tsJz54LUXkSYM8+8PG81B9IKK5r+2T0clMq4QrS16xFosufU7firBDAZEsDheDs7wTlP7h3++S7lMsU955HA==} cpu: [loong64] os: [linux] - libc: [musl] '@rollup/rollup-linux-ppc64-gnu@4.62.0': resolution: {integrity: sha512-3UvJ5PNVU16aJf6M3tFI24pWzAl2/ynfbyRN3ICyQajK1lSkrnVYNnLz3v04J32qKa0FczJc22zeToc0lr2A3w==} cpu: [ppc64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-ppc64-musl@4.62.0': resolution: {integrity: sha512-vRWUAbYLGHBZS6Q8Msb2sfnf1fvJf+47t8l/TwOerM2qArzy+IeNMTHrYLHXh95h8MoatPHI5hhSZNs+mGXKPg==} cpu: [ppc64] os: [linux] - libc: [musl] '@rollup/rollup-linux-riscv64-gnu@4.62.0': resolution: {integrity: sha512-c00T5SYENHAt86cfW47URaP3Us5vLC/4QO7GYud1G5VNRffCwwCuBspwqYrriuJB+5m0WFzClCn9wed0FBjKvg==} cpu: [riscv64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-riscv64-musl@4.62.0': resolution: {integrity: sha512-krrCDilhXOwFkSkO3Wm9I/f9H0L92XHHwy2fwxjukxIbh0dem8gZqOW5Y8BsHrpJv5qwlRBV+Wl4ZFyRWhUpwg==} cpu: [riscv64] os: [linux] - libc: [musl] '@rollup/rollup-linux-s390x-gnu@4.62.0': resolution: {integrity: sha512-7pfYFSTc4/rUC/FtAI0Qp6QthDBCIi6/AuP1xYqFk5vanI6KnL5dWKP60OM/05LOsbwTmIcvr6eXC4CJuJ75IA==} cpu: [s390x] os: [linux] - libc: [glibc] '@rollup/rollup-linux-x64-gnu@4.62.0': resolution: {integrity: sha512-7SDIalKeIpG0Ifogbbdn58HmSotYMlf23K3dCJEmiVd9Fg36Vmni82iPQec27N3wY4Bvbxftkxz6vSx9OcouTg==} cpu: [x64] os: [linux] - libc: [glibc] '@rollup/rollup-linux-x64-musl@4.62.0': resolution: {integrity: sha512-eRZevouTH2i1HeAVLqJuLnt256krQkGY0TN6WsTmsIhuzbh457HuWDMakKwmi0Cjadux983CoSr8Lim2QhUIFw==} cpu: [x64] os: [linux] - libc: [musl] '@rollup/rollup-openbsd-x64@4.62.0': resolution: {integrity: sha512-3oVS7FLGa4U1qcvao9ylGxrjXZyUQqR8UwxEcnUEyPX53O/C/mKDZegNXTdHCP+h3e6ta/f1EN38Yif1mmZHYg==} @@ -2327,28 +2304,24 @@ packages: engines: {node: '>= 20'} cpu: [arm64] os: [linux] - libc: [glibc] '@tailwindcss/oxide-linux-arm64-musl@4.3.1': resolution: {integrity: sha512-Bwv9KwOvE0VKa86xPFif9b9c3Y1NxOV1P0gLti/IYaWEsQYZXDlxfGEtA8mdDZ7SG3wyNXAWYT5SIn3giL57oA==} engines: {node: '>= 20'} cpu: [arm64] os: [linux] - libc: [musl] '@tailwindcss/oxide-linux-x64-gnu@4.3.1': resolution: {integrity: sha512-Ymi8O8T15HYQdOUWUtTI6ldN0neHP85FC+Qz32xTcZ7iJXtem/x8ITev0o1e9e5rkqj4lONZfTRLvkmin1+tKg==} engines: {node: '>= 20'} cpu: [x64] os: [linux] - libc: [glibc] '@tailwindcss/oxide-linux-x64-musl@4.3.1': resolution: {integrity: sha512-M+P/91qJ6uILLw4k2G93GMDRAXj61SMvFQYt39AqvUqYgExXpLL5aepfns7sj4HiAQeolirQF9E0lzRvdf4zPQ==} engines: {node: '>= 20'} cpu: [x64] os: [linux] - libc: [musl] '@tailwindcss/oxide-wasm32-wasi@4.3.1': resolution: {integrity: sha512-zsM8uOeqvVGHsAXsJxsT28ttosFahLJKCLOTUBqRAtKnVgGSRitds9T432QiT8b77Yga7JIBkulIRRlJPtYhRA==} @@ -2454,6 +2427,9 @@ packages: '@types/json5@0.0.29': resolution: {integrity: sha512-dRLjCWHYg4oaA77cxO64oO+7JwCwnIzkZPdrrC71jQmQtlhM556pwKo5bUzqvZndkVbeFLIIi+9TC40JNF5hNQ==} + '@types/mocha@10.0.10': + resolution: {integrity: sha512-xPyYSz1cMPnJQhl0CLMH68j3gprKZaTjG3s5Vi+fDgx+uhG9NOXwbVt52eFS8ECyXhyKcjDLCBEqBExKuiZb7Q==} + '@types/node@25.9.3': resolution: {integrity: sha512-603BddQMv3pUcr4U2dhujk83N2tTDVr/34wII2B6bJy6g+8WD6yUb11jszNs0gdi4PesVWl7ABt8nYMVpnLUcg==} @@ -2606,61 +2582,51 @@ packages: resolution: {integrity: sha512-zJc0H99FEPoFfSrNpa91HYfxzfAJCr502oxNK1cfdC9hlaFI43RT+JFCann9JUgZmLzzntChHyn13Sgn9ljHNg==} cpu: [arm64] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-arm64-musl@1.12.2': resolution: {integrity: sha512-KQ3Lki6l+Pz1k/eBipN41ES+YUK30beLGb9YqcB1O542cyLCNE6GaxrfcY3T6EezmGGk84wb5XyO9loTM9tkcA==} cpu: [arm64] os: [linux] - libc: [musl] '@unrs/resolver-binding-linux-loong64-gnu@1.12.2': resolution: {integrity: sha512-3SJGEh1DborhG6pyxvhPzCT4bbSIVihsvgJc13P1bHG7KLdNDaF9T3gsTwFc7Jw/5Y5/iWOjkEx7Zy0NvCGX3Q==} cpu: [loong64] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-loong64-musl@1.12.2': resolution: {integrity: sha512-jiuG/Obbel7uw1PwHNFfrkiKhLAF6mnyZ6aWlOAVN9WqKm8v0OFGnciJIHu8+CMvXLQ8AD51LPzAoUfT21D5Ew==} cpu: [loong64] os: [linux] - libc: [musl] '@unrs/resolver-binding-linux-ppc64-gnu@1.12.2': resolution: {integrity: sha512-q7xRvVpmcfeL+LlZg8Pbbo6QaTZwDU5BaGZbwfhkEsXJn3Was8xYfE0RBH266xZt0rM6B7i8xAYIvjthuUIWHg==} cpu: [ppc64] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-riscv64-gnu@1.12.2': resolution: {integrity: sha512-0CVdx6lcnT3Q9inOH8tsMIOJ6ImndllMjqJHg8RLVdB7Vq4SfkEXl9mCSsVNuNA4MCYycRicCUxPCabVHJRr6A==} cpu: [riscv64] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-riscv64-musl@1.12.2': resolution: {integrity: sha512-iOwlRo9vnp6R6ohHQS11n0NnfdXx/omhkocmIfaPRpQhKZ+3BDMkkdRVh53qjkFkpPddf+FETA28NwGN7l5l+w==} cpu: [riscv64] os: [linux] - libc: [musl] '@unrs/resolver-binding-linux-s390x-gnu@1.12.2': resolution: {integrity: sha512-HYJtLfXq94q8iZNFT1lknx258wlkkWhZeUXJRqzKBBUJ00CvZ+N33zgbCqimLjsyw5Va6uUxhVa12mI+kaveEw==} cpu: [s390x] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-x64-gnu@1.12.2': resolution: {integrity: sha512-mPsUhunKKDih5O96Y6enDQyHc1SqBPlY1E/SfMWDM3EdJ95Z9CArPeCVwCCqbP45ljvivdEk8Fxn+SIb1rDAJQ==} cpu: [x64] os: [linux] - libc: [glibc] '@unrs/resolver-binding-linux-x64-musl@1.12.2': resolution: {integrity: sha512-azrt6+5ydLd8Vt210AAFis/lZevSfPw93EJRIJG+xPu4WCJ8K0kppCTpMyLPcKT7H15M4Jnt2tMp5bOvCkRC6A==} cpu: [x64] os: [linux] - libc: [musl] '@unrs/resolver-binding-openharmony-arm64@1.12.2': resolution: {integrity: sha512-YZ9hP4O0X9PQb8eO980qmLNGH4zT3I9+SZTdt0Pr0YyuGQhYKoOZkV02VzrzyOZJ5xIJ3UFIenKkUkGg8GjgWQ==} @@ -2785,6 +2751,10 @@ packages: resolution: {integrity: sha512-HdzDrRs+ywAqbXGKqe1i/bLtCv47plz4TvsHFH3j729OooT5VH38ctFn5aLXgECmiAKDkmH/A6kOq2Zh5DIxww==} engines: {node: '>=18.20.0'} + '@wdio/mocha-framework@9.28.0': + resolution: {integrity: sha512-UdkgigdyxJu0Rpx2ZRSNuB8u8B5nmLBeRjn5JHYTXBp8Ubi+wgFR1+EdLg8k2z+CtuYe3bNHBw7vPpBYuQMLZw==} + engines: {node: '>=18.20.0'} + '@wdio/protocols@8.40.3': resolution: {integrity: sha512-wK7+eyrB3TAei8RwbdkcyoNk2dPu+mduMBOdPJjp8jf/mavd15nIUXLID1zA+w5m1Qt1DsT1NbvaeO9+aJQ33A==} @@ -3368,6 +3338,11 @@ packages: engines: {node: '>=22'} hasBin: true + chromedriver@150.0.1: + resolution: {integrity: sha512-vpYskeQFyZGPNMEwc/5fuvfnxzfjljCO1EvwuF0dZGQ8KE0repYQar7eThTlT1IjznwVrirYRgXrRsJts7DRIQ==} + engines: {node: '>=22'} + hasBin: true + chromium-bidi@0.5.8: resolution: {integrity: sha512-blqh+1cEQbHBKmok3rVJkBlBxt9beKBgOsxbFgs7UJcoVbbeZ+K7+6liAsjgpc8l1Xd55cQUy14fXZdGSb4zIw==} peerDependencies: @@ -5262,28 +5237,24 @@ packages: engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] - libc: [glibc] lightningcss-linux-arm64-musl@1.32.0: resolution: {integrity: sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] - libc: [musl] lightningcss-linux-x64-gnu@1.32.0: resolution: {integrity: sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] - libc: [glibc] lightningcss-linux-x64-musl@1.32.0: resolution: {integrity: sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] - libc: [musl] lightningcss-win32-arm64-msvc@1.32.0: resolution: {integrity: sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==} @@ -7644,7 +7615,7 @@ snapshots: '@babel/types': 7.29.7 '@jridgewell/remapping': 2.3.5 convert-source-map: 2.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) gensync: 1.0.0-beta.2 json5: 2.2.3 semver: 6.3.1 @@ -7803,7 +7774,7 @@ snapshots: '@babel/parser': 7.29.7 '@babel/template': 7.29.7 '@babel/types': 7.29.7 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color @@ -8484,7 +8455,7 @@ snapshots: '@eslint/config-array@0.23.5': dependencies: '@eslint/object-schema': 3.0.5 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) minimatch: 10.2.5 transitivePeerDependencies: - supports-color @@ -9168,7 +9139,7 @@ snapshots: '@puppeteer/browsers@2.13.2': dependencies: - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) extract-zip: 2.0.1 progress: 2.0.3 proxy-agent: 6.5.0 @@ -9514,6 +9485,8 @@ snapshots: '@types/json5@0.0.29': {} + '@types/mocha@10.0.10': {} + '@types/node@25.9.3': dependencies: undici-types: 7.24.6 @@ -9584,7 +9557,7 @@ snapshots: '@typescript-eslint/types': 8.61.1 '@typescript-eslint/typescript-estree': 8.61.1(typescript@6.0.3) '@typescript-eslint/visitor-keys': 8.61.1 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) eslint: 10.5.0(jiti@2.7.0) typescript: 6.0.3 transitivePeerDependencies: @@ -9594,7 +9567,7 @@ snapshots: dependencies: '@typescript-eslint/tsconfig-utils': 8.61.1(typescript@6.0.3) '@typescript-eslint/types': 8.61.1 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) typescript: 6.0.3 transitivePeerDependencies: - supports-color @@ -9613,7 +9586,7 @@ snapshots: '@typescript-eslint/types': 8.61.1 '@typescript-eslint/typescript-estree': 8.61.1(typescript@6.0.3) '@typescript-eslint/utils': 8.61.1(eslint@10.5.0(jiti@2.7.0))(typescript@6.0.3) - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) eslint: 10.5.0(jiti@2.7.0) ts-api-utils: 2.5.0(typescript@6.0.3) typescript: 6.0.3 @@ -9628,7 +9601,7 @@ snapshots: '@typescript-eslint/tsconfig-utils': 8.61.1(typescript@6.0.3) '@typescript-eslint/types': 8.61.1 '@typescript-eslint/visitor-keys': 8.61.1 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) minimatch: 10.2.5 semver: 7.8.4 tinyglobby: 0.2.17 @@ -9767,14 +9740,6 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.0 - '@vitest/mocker@4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0))': - dependencies: - '@vitest/spy': 4.1.9 - estree-walker: 3.0.3 - magic-string: 0.30.21 - optionalDependencies: - vite: 8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0) - '@vitest/mocker@4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.9 @@ -9958,6 +9923,20 @@ snapshots: safe-regex2: 5.1.1 strip-ansi: 7.2.0 + '@wdio/mocha-framework@9.28.0': + dependencies: + '@types/mocha': 10.0.10 + '@types/node': 25.9.3 + '@wdio/logger': 9.18.0 + '@wdio/types': 9.28.0 + '@wdio/utils': 9.28.0 + mocha: 10.8.2 + transitivePeerDependencies: + - bare-abort-controller + - bare-buffer + - react-native-b4a + - supports-color + '@wdio/protocols@8.40.3': {} '@wdio/protocols@9.28.0': {} @@ -10084,7 +10063,7 @@ snapshots: agent-base@6.0.2: dependencies: - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color @@ -10682,6 +10661,19 @@ snapshots: - debug - supports-color + chromedriver@150.0.1: + dependencies: + '@testim/chrome-version': 1.1.4 + adm-zip: 0.5.17 + axios: 1.16.1 + compare-versions: 6.1.1 + proxy-agent: 8.0.1 + proxy-from-env: 2.1.0 + tcp-port-used: 1.0.2 + transitivePeerDependencies: + - debug + - supports-color + chromium-bidi@0.5.8(devtools-protocol@0.0.1232444): dependencies: devtools-protocol: 0.0.1232444 @@ -11515,7 +11507,7 @@ snapshots: '@types/estree': 1.0.9 ajv: 6.15.0 cross-spawn: 7.0.6 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) escape-string-regexp: 4.0.0 eslint-scope: 9.1.2 eslint-visitor-keys: 5.0.1 @@ -11632,7 +11624,7 @@ snapshots: extract-zip@2.0.1: dependencies: - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) get-stream: 5.2.0 yauzl: 2.10.0 optionalDependencies: @@ -11963,7 +11955,7 @@ snapshots: dependencies: basic-ftp: 5.3.1 data-uri-to-buffer: 6.0.2 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color @@ -11971,7 +11963,7 @@ snapshots: dependencies: basic-ftp: 5.3.1 data-uri-to-buffer: 8.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color @@ -12167,35 +12159,35 @@ snapshots: http-proxy-agent@7.0.2: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color http-proxy-agent@9.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color https-proxy-agent@5.0.1: dependencies: agent-base: 6.0.2 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color https-proxy-agent@7.0.6: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color https-proxy-agent@9.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color @@ -12492,7 +12484,7 @@ snapshots: istanbul-lib-source-maps@5.0.6: dependencies: '@jridgewell/trace-mapping': 0.3.31 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) istanbul-lib-coverage: 3.2.2 transitivePeerDependencies: - supports-color @@ -13306,7 +13298,7 @@ snapshots: dependencies: axe-core: 4.12.0 - nightwatch@3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@148.0.4): + nightwatch@3.16.0(@cucumber/cucumber@13.0.0)(chromedriver@150.0.1): dependencies: '@nightwatch/chai': 5.0.3 '@nightwatch/html-reporter-template': 0.3.0 @@ -13344,7 +13336,7 @@ snapshots: uuid: 8.3.2 optionalDependencies: '@cucumber/cucumber': 13.0.0 - chromedriver: 148.0.4 + chromedriver: 150.0.1 transitivePeerDependencies: - bufferutil - canvas @@ -13578,7 +13570,7 @@ snapshots: dependencies: '@tootallnate/quickjs-emscripten': 0.23.0 agent-base: 7.1.4 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) get-uri: 6.0.5 http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 @@ -13590,7 +13582,7 @@ snapshots: pac-proxy-agent@9.0.1: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) get-uri: 8.0.0 http-proxy-agent: 9.0.0 https-proxy-agent: 9.0.0 @@ -13880,7 +13872,7 @@ snapshots: proxy-agent@6.5.0: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 lru-cache: 7.18.3 @@ -13893,7 +13885,7 @@ snapshots: proxy-agent@8.0.1: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) http-proxy-agent: 9.0.0 https-proxy-agent: 9.0.0 lru-cache: 7.18.3 @@ -14413,7 +14405,7 @@ snapshots: socks-proxy-agent@10.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) socks: 2.8.9 transitivePeerDependencies: - supports-color @@ -14421,7 +14413,7 @@ snapshots: socks-proxy-agent@8.0.5: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) socks: 2.8.9 transitivePeerDependencies: - supports-color @@ -14630,7 +14622,7 @@ snapshots: cosmiconfig: 9.0.2(typescript@6.0.3) css-functions-list: 3.3.3 css-tree: 3.2.1 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) fast-glob: 3.3.3 fastest-levenshtein: 1.0.16 file-entry-cache: 11.1.3 @@ -14893,7 +14885,7 @@ snapshots: cac: 6.7.14 chokidar: 4.0.3 consola: 3.4.2 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) esbuild: 0.27.7 fix-dts-default-cjs-exports: 1.0.1 joycon: 3.1.1 @@ -15026,7 +15018,7 @@ snapshots: '@rollup/pluginutils': 5.4.0(rollup@4.62.0) '@volar/typescript': 2.4.28 compare-versions: 6.1.1 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) kolorist: 1.8.0 local-pkg: 1.2.1 magic-string: 0.30.21 @@ -15169,21 +15161,6 @@ snapshots: optionalDependencies: rollup: 4.62.0 - vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0): - dependencies: - lightningcss: 1.32.0 - picomatch: 4.0.4 - postcss: 8.5.15 - rolldown: 1.0.3 - tinyglobby: 0.2.17 - optionalDependencies: - '@types/node': 25.9.3 - esbuild: 0.27.7 - fsevents: 2.3.3 - jiti: 2.7.0 - tsx: 4.22.4 - yaml: 2.9.0 - vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0): dependencies: lightningcss: 1.32.0 @@ -15199,36 +15176,6 @@ snapshots: tsx: 4.22.4 yaml: 2.9.0 - vitest@4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)): - dependencies: - '@vitest/expect': 4.1.9 - '@vitest/mocker': 4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) - '@vitest/pretty-format': 4.1.9 - '@vitest/runner': 4.1.9 - '@vitest/snapshot': 4.1.9 - '@vitest/spy': 4.1.9 - '@vitest/utils': 4.1.9 - es-module-lexer: 2.1.0 - expect-type: 1.3.0 - magic-string: 0.30.21 - obug: 2.1.3 - pathe: 2.0.3 - picomatch: 4.0.4 - std-env: 4.1.0 - tinybench: 2.9.0 - tinyexec: 1.2.4 - tinyglobby: 0.2.17 - tinyrainbow: 3.1.0 - vite: 8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0) - why-is-node-running: 2.3.0 - optionalDependencies: - '@types/node': 25.9.3 - '@vitest/coverage-v8': 4.1.9(@vitest/browser@4.1.9)(vitest@4.1.9) - happy-dom: 20.10.4 - jsdom: 24.1.3 - transitivePeerDependencies: - - msw - vitest@4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.9 @@ -15271,7 +15218,7 @@ snapshots: dependencies: chalk: 4.1.2 commander: 9.5.0 - debug: 4.4.3(supports-color@10.2.2) + debug: 4.4.3(supports-color@5.5.0) transitivePeerDependencies: - supports-color