Repository navigation
fix(oracle-gaps): Windows static and e2e failures, by class (follow-up to #58) - #73
Draft
KyleAMathews wants to merge 58 commits into
Draft
KyleAMathews wants to merge 58 commits into
KyleAMathews wants to merge 58 commits into
Conversation
The framework, the eslint plugin, and the gap CLI have never run on Windows, and users report path problems there. Two diagnostic jobs run on windows-latest: the unit suites (framework, eslint plugin, gap fast oracles) and the gap CLI end to end (a fixture package written at run time, the plain and daemon paths, and one fixed-seed e2e unit). Both are continue-on-error and each step continues on error, so the first failure does not hide later ones. Neither is a required check or a dependency of preview. Co-authored-by: Isaac <no-reply@databricks.com>
Diagnostic only, not a fix. The first run showed the gap fast suites and the fixed-seed end-to-end unit hang on windows-latest (cancelled at the job timeout, with no output after the static oracle and the e2e calibrations started). The fast suites now run one file per step with a step timeout and verbose output for the static oracle, so the hang is located. The fixture gets the package.json it was missing (the CLI reported ENOENT for it). The job timeouts are raised to 45 minutes so the split steps can finish. Co-authored-by: Isaac <no-reply@databricks.com>
… run Diagnostic only. Adds a bash replay of the first failing static unit (its verdict is not printed by the run), a run of the fixture with the discovered config instead of --config, and a dump of the gap CLI's error.log files from the OS temp directory, which the daemon run's failure points to. Co-authored-by: Isaac <no-reply@databricks.com>
…ines Forward slashes at entry, native only at the file system. Paths enter the CLI as argv, as Vite and Vitest report them, and as realpath returns them; comparisons and keys use one spelling (oracle-gaps/paths.mjs). - The config wrapper imports the config by a POSIX specifier. A backslash in an ES module specifier is a character, so every run with a config was refused on Windows (check.mjs, wrapper). - Mutant copies are keyed and served in forward slashes, as Vite resolves and Vitest reports them. The copy map was keyed by native paths, so the copies were never served on Windows. - Module IDs are kept by isAbsolute, not by a leading slash: a Windows ID is `C:/...`. - The package root, the located packages, and the source names use the native real path on both sides (realPathOf). The JavaScript realpath kept 8.3 short names such as RUNNER~1 and refused every owns entry. - The daemon listens on a named pipe on win32 (no socket file to remove), keys by the same real path as the CLI, and its timeout names the end of its log. - A package with no package.json is refused, not an internal error. - Replay and rerun lines print PowerShell and cmd forms on Windows; the POSIX form is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
…e realpath - npm is an npm.cmd shim on Windows; spawnSync runs it through a shell there (package.test.ts, shipped-citations.test.js). - The probe-form oracle's model decides production by a directory named src, split on either separator, not by a substring with `/`. Production was right; the model was wrong on Windows. - The owns test uses the native real path, as the CLI does. Co-authored-by: Isaac <no-reply@databricks.com>
The diagnostic steps are gone. The Windows jobs mirror the Linux shards: the unit suites in one job, the static gap oracle and the fixed-seed end-to-end oracle in three shards each (pr-shards.mjs), and the gap CLI on a fixture, including the discovered config and the daemon. Each job has a 15-minute backstop. The jobs are still continue-on-error until every one is green; that line is removed then. Co-authored-by: Isaac <no-reply@databricks.com>
… path The owners and the instrumenter named files with the platform's separators, and TypeScript matches a file's name as given, so on Windows the owner map keyed `src\x.js` while the sources were `src/x.js`, and CommonJS sites lost their functions. The compiler now gets forward-slash names, and owner keys are forward-slash (schemata.mjs, owners.mjs, analysis.mjs). A path's drive letter is capitalised in the one spelling (paths.mjs), so a drive's case never decides a path. New oracle: gap-report-path-spelling (a metamorphic check). Law 1: the instrumenter gives the same mutant ids for a source under any spelling of its path. Law 2: relativeWithin answers from the segments alone. Evidence block - Law: a path's separators and a drive letter's case do not change what the gap CLI decides (paths.mjs, Windows support, 2026-11-13). - Reach: 200 generated histories (fixed seed 20261113 and random), over four module forms, drives in both cases, mixed separators, short and spaced names. Reach is not declared (states: []); the histories are not measured as reaching a state. - RED: on the parent commit (72b9495), the Windows jobs of #58 failed: the config wrapper import (check.mjs), the copy map, the owner keys (owners.mjs), and the 8.3 short names. On Linux the enumeration law does NOT reproduce the Windows failure: the instrumenter gave the same ids on the parent. The Windows failure is a platform path.relative difference, which Linux cannot produce. The RED for those classes is the Windows CI run. - GREEN: the Windows jobs (see the PR). Local, on macOS: the new oracle 5 of 5, the framework 114 of 114, the eslint plugin 7335 of 7335. - Mutant: relativeWithin as it was before (raw string comparison) fails the relation checkpoint; dropping the drive capitalisation fails it too. Both run on the parent tree with the oracle and both fail (exit 1). - Limits: no declared reach; Law 1 uses the forward-slash spelling as its reference, so it claims only that the spelling changes nothing; owners, realpath, and OS short names are judged by the Windows end-to-end run, not by this oracle. Co-authored-by: Isaac <no-reply@databricks.com>
… witnesses - P10: the Windows CLI runs assert the done line on both streams, and that it matches the exit code (scripts/windows-witness.mjs done). - A package with no package.json is refused with exit 2 and a done line that names it (Windows witness). - One early stop: SIGINT once the CLI prints its first line; the run must end with the stop's done line on both streams, or with no output (#57). - Per-shell replay (framework, accepted by the coordinator): a failing property's PowerShell and cmd lines each reproduce the failure in their own shell. The POSIX line is run in PowerShell and must not reproduce (the RED). A run that printed only the POSIX line fails here (the mutant). - The tsc link lookup in package.test.ts reads npm's Windows shim, not a symlink. - The eslint plugin suite runs even when the framework suite fails. - Job timeouts are 20 minutes, as high backstops (2026-11-12). Co-authored-by: Isaac <no-reply@databricks.com>
…ch packages have a manifest Windows CI failed the classify oracle's baseline law: the CLI prints a PowerShell and a cmd replay line there, and the law read only the POSIX `replay:` line (Difference: "the replay of tests/f0.test.ts is undefined"). The law is unchanged: each replay line the platform prints must read as `vitest run <file> -t <pattern>` in its own shell, and the owed test is the one it selects. On win32 the law owes both per-shell lines; elsewhere the POSIX line (replayLinesOf, powerShellWords, cmdWords). The static oracle's scratch packages had no package.json, which the CLI now refuses (args.mjs). Each fixture now writes a minimal manifest; its content and its histories are unchanged. Evidence block - Law: the replay line of each shell reads as the owed command (Law 9, baseline; the CLI's replay lines, stream.mjs). - Reach: the existing classify histories (276 tests); no new histories. - RED: Windows run 43478073715, classify shard 3 (baseline law, 11 violations) and the static shard 1 refusal (the missing manifest). - GREEN: local POSIX classify 276 of 276. The win32 branch runs in the Windows job (see the PR). - Mutant: not run; the win32 branch is exercised only by the Windows job. - Limits: the win32 branch is checked on Windows only, not by the oracle's own generated histories. Co-authored-by: Isaac <no-reply@databricks.com>
… shipped citations Run 43483101749 (Windows, the first with the witnesses) showed: - The CLI printed `report <pkg>\out\summary.md` and `tests\a.test.ts` in the done line and in the heartbeat. Report paths and display paths are now forward-slash (posix, check.mjs, ending.mjs), as the register's keys rule requires; the file system still gets native paths. - The classify oracle's cmd reader rejected `\"` inside a double-quoted argument, which Node's argument parser reads as a quote. It now reads it so. - The static and classify oracles' second scratch package had no manifest. - Shipped citations joined paths with backslashes, and npm names files with forward slashes. The comparison now uses forward slashes. - The refusal witness searches both streams (a refusal prints to stderr). - The static and fixed-seed e2e took 3 to 6 times their Linux time on Windows, and the Linux shards hit the limit. Windows runs six shards of each (pr-shards.mjs `oracle-gaps-static-windows`, `oracle-gaps-fixed-seed-windows`, checked by `pr-shards.mjs check`: each test in exactly one shard). The classify, probe-form, mutant-validity, and path-spelling oracles run in their own job. Evidence block - Law: report and display paths are forward-slash; a cmd line reads its quoted quotes as Node does (register: deterministic replay; the CLI's keys rule). - Reach: the existing e2e and static histories, unchanged. - RED: run 43483101749, which printed backslash report paths and failed classify on `\"`. - GREEN: pending the next Windows run; local POSIX suites pass (classify 276, shipped citations 2). - Mutant: not run for this commit. - Limits: the Windows e2e and static shard timings are measured on the next run before the budget claim (P12) is made. Co-authored-by: Isaac <no-reply@databricks.com>
…and its models Run 43493431349 (Windows) failed the static and e2e oracles on report paths: the done line names its report in forward slashes (the register's Windows paths rule), but the model built the expected line from the native output directory, and the declared oracle files were keyed by native relative paths (`tests\a.test.ts`), so a heartbeat's file was not an owner. - analysis.mjs, check.mjs: declared oracle files and copies are keyed by their forward-slash relative path (relativeWithin). - gap-report-model-oracle.mjs: the model's expected done line is forward-slash (inline, the model does not import production). - shipped-citations.test.js: the rejection test's strings are forward-slash. Evidence block - Law: keys and report paths use forward slashes on every OS (register, Windows paths, #58); filesystem calls get native paths. - Reach: the existing static and e2e histories; no new histories. - RED: run 43493431349: static 40 refusal-ending violations, e2e 17 baseline lead, 12 report-shaped ending, 4 streaming, 3 greppable stream. - GREEN: pending the next Windows run. Local POSIX: shipped citations 2 of 2, CLI lints clean; the Linux keys are unchanged. - Mutant: not run. - Limits: the Windows e2e and static timings are measured on the next run. Co-authored-by: Isaac <no-reply@databricks.com>
The early-stop step failed on Windows (SIGINT ends the process through TerminateProcess, so the CLI's stop handler never runs; the run ended with a null exit code after its done line), which skipped the replay witness behind it. The replay witness now runs regardless, so each witness reports. Co-authored-by: Isaac <no-reply@databricks.com>
The replay witness found no per-shell lines in the fixture's failing output. This prints the first lines of that output in the same step, so the next run shows what vitest printed. Diagnostic, to be removed with the witness fix. Co-authored-by: Isaac <no-reply@databricks.com>
The replay fixture failed to load (TypeError reading fixedSeeds[0]), so the witness saw no failure. It now declares fixedSeeds: [1]. Co-authored-by: Isaac <no-reply@databricks.com>
… forward-slash done lines - Stop (#57, all-or-nothing) on the unsupervised win32 path: a stop of a run that has started now writes its `oracle-gaps: done stopped (<signal>)` line on both streams and exits 128 + signal, as the supervisor does. A stop before the run starts still prints nothing (index.mjs, runStarted). - Test-only stop hook: when the file ORACLE_GAPS_TEST_STOP names appears, the CLI takes the same stop as a SIGINT (index.mjs). The Windows stop witness uses it (a non-console parent cannot deliver Ctrl+C there; child.kill on win32 is TerminateProcess, which is a SIGKILL-like limit, so no line is owed). - Static oracle: the `unknown` long-argument case uses 30,000 characters on win32. Windows caps a command line at 32,767 characters; a longer one never starts (status null). Narrowing, recorded in the PR: the POSIX runs keep 100,000. - e2e and static model: the expected done line's report path is forward-slash (inline in the model; the e2e corruption check likewise). - The diagnostic Get-Content line is removed. Evidence block - Law: the all-or-nothing stop law (#57) on every platform; keys and report paths forward-slash (register: Windows paths; stop window on every platform). - Reach: the existing static and e2e histories; the stop witness is one run. - RED: run 43498457502, stop witness `stop was not all-or-nothing (signal SIGINT, exit null)`; static `with unknown arguments, the refusal exited null`; e2e `the last stdout line is ... report D:/...`, not `...\out\summary.md`. - GREEN: pending the next Windows run. Local: the CLI lints and loads; the stop witness is queued on the e2e lock for a fixture run. - Mutant: pending (the replay witness, on a throwaway branch). - Limits: OS delivery of Ctrl+C is not exercised on Windows; TerminateProcess is treated like SIGKILL; the win32 argv cap narrows the long-argument case. Co-authored-by: Isaac <no-reply@databricks.com>
…et, and logs The default output directory is under the OS temporary directory, which on windows-latest is `C:\Users\RUNNER~1\...` (an 8.3 short name). The model's expected done line names the package by its real path, so the report path never matched (e2e: `the last stdout line is ... report C:/Users/RUNNER~1/...`). The temporary directory is now taken by its real path (realPathOf) for the state directory, the socket, and the error log. Evidence block - Law: keys and report paths use one spelling; a real path on both sides (register: Windows paths; the realpath rule). - Reach: the existing e2e histories (the done-line law). - RED: e2e run 43515364499, done line with RUNNER~1 vs the package path. - GREEN: pending the next Windows run. - Mutant: pending; the replay witness mutant runs on the throwaway PR. - Limits: the stdout gap-count mismatch (`2 gap lines for 0 gaps`) and the cold internal error in the incremental law are not yet explained. Co-authored-by: Isaac <no-reply@databricks.com>
… real path The e2e oracle realpaths the OS temporary directory with the JavaScript realpath, which keeps the 8.3 name (C:\Users\RUNNER~1). The CLI prints the native real path, so the expected report line never matched. Use realpathSync.native, as the CLI does. Evidence block - Law: one real path on both sides (the realpath rule; Windows paths). - Reach: the existing e2e histories. - RED: run 43515364499 and ee1a5fd: the done line names RUNNER~1. - GREEN: pending the next Windows run. - Limits: the static `output namespace` violations (82 in the last run) are not yet traced. Co-authored-by: Isaac <no-reply@databricks.com>
…es; fix the tsc lookup type - Blocking now: the Windows unit suites, the classify, probe-form, mutant-validity, and path-spelling job, and the gap CLI fixture job with its witnesses (done line, refusal, stop, replay). Their continue-on-error is removed. - Non-blocking, stated in the workflow: the Windows static and e2e jobs, which fail on Windows (tracked in the follow-up PR's body by class). - Linux `Test` failed the typecheck on package.test.ts (the tsc lookup's optional capture). Fixed; tsc --noEmit passes locally. Co-authored-by: Isaac <no-reply@databricks.com>
…al path (output namespace) The output-namespace law (checkpoint `static`) took the OS temp directory by JavaScript's realpath, which keeps an 8.3 short name (C:\Users\RUNNER~1), while the CLI's default output is under the native real path. So every run's output failed "not under the OS temp directory" on windows-latest (82 hits in run 43525797122). The oracle now uses realpathSync.native, the same rule the CLI uses, and drops the raw tmpdir() alternative. Evidence block - Law: a run's default output is under the OS temporary directory's real path, never in its package (maintainer decision, 2026-11-11; register: the realpath rule). - Reach: the existing static histories (the outputs of three runs per history, two of one package and one of another). - RED: Windows run 43525797122, static shards: 82 violations, law "output namespace", "a run's default output is not under the OS temp directory". - GREEN: pending the next Windows run on this branch. - Mutant (Windows-only): the CLI's temporary directory taken raw (tmpdir() without realpath) puts the output under RUNNER~1, which the fixed law rejects on Windows. On Linux the two spellings coincide, so the Linux run cannot kill this mutant; the Windows run is its witness. - Limits: Linux does not exercise the 8.3 spelling. Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…gs (baseline lead) The e2e driver replaces scratch paths with <package> and <cli> in the CLI's output, but only in the native spelling. The CLI now names its report and package paths in forward slashes (the Windows paths rule), so the done line kept `D:/a/.../.scratch-XXXX/out/summary.md` and the baseline-lead law (18 hits in run 43525797122) read it as not `<package>/out/summary.md`. Evidence block - Law: the CLI's done line names its report in the package's placeholder spelling (law: baseline lead; register: Windows paths). - Reach: the existing e2e histories. - RED: run 43525797122, e2e shards: 18 violations, law "baseline lead". - GREEN: pending the next Windows run on this branch. - Mutant: a CLI that prints the native spelling (backslashes) in the done line is now caught by the same check, since the placeholder is only applied to the spelling the driver names; not run in CI. - Limits: the driver names two spellings of each scratch path (native and forward-slash), not every possible form. Co-authored-by: Isaac <no-reply@databricks.com>
…river (streaming) The e2e driver's mutant observations named each owner file with path.relative, which returns backslashes on Windows (tests\a.test.ts). The CLI names a heartbeat's file in forward slashes, so every heartbeat failed "names sut.ts:M1 in tests/a.test.ts, which does not run it" (the streaming law, 4 hits in run 43525797122, plus greppable-stream instances). Evidence block - Law: the heartbeat names a test file that runs the mutant (streaming law); keys and report paths are forward-slash (register: Windows paths). - Reach: the existing e2e histories with a slow owner. - RED: run 43525797122, e2e shards: "a heartbeat names ... which does not run it". - GREEN: pending the next Windows run on this branch. - Mutant: not run. - Limits: the stdout gap-line count mismatch (`122 gap lines for 121 gaps`, greppable stream) is not yet explained; left for the next run. Co-authored-by: Isaac <no-reply@databricks.com>
commit: |
…d review items F1-F8 Review of #58 at ebc39e6 (NOT READY). This commit fixes: - F1 (stop never stops): the witness's stop file is absolute (the CLI's cwd is the package), a stop requires the stopped line on both streams with exit 130, and the stop lands 400 ms after the run starts (the fixture's output is written in one burst near its end, so the first output line is no clock). - F2/F3 (replay quoting): PowerShell reads U+2018 to U+201B as quotes, so every such character is doubled inside single quotes. cmd toggles quoting on every `"` and expands `%`; a name with either has no cmd form, and its cmd line says `use the PowerShell line for this name` (stream.mjs, src/oracle-property.ts). The classify oracle reads each shell's line by that shell's grammar (the PowerShell quote set; cmd's quote toggling, `^`, and `%`), checks the win32 printer on every host, and generates adversarial names (curly quotes, `& | < > ( ) ^ % !`, quotes, backslashes), drawn as the last field of the test record (the fixed seeds' later draws shift; recorded here). - F(f) (POSIX hook): the test-only stop hook lives in the supervisor and goes through its stop, which forwards to the child; a hooked stop gives `done stopped (SIGINT)`, exit 130 (local run: outcome (a)). Unsupervised runs keep the hook in the process. - F5: posix() acts only on win32; on POSIX a backslash is a name's character. - F6: check.mjs realpath of workspace links, report.mjs results path, and ending.mjs error log path are forward-slash and use the native real path. - F7: the blocking Windows jobs are in preview's needs. - F8: the non-blocking jobs point at #72. - The Windows replay witness runs each printed PowerShell line and its cmd line, for two fixtures with hostile names (curly quote and metacharacters; a double quote and `%`). Evidence block - Law: the printed replay line replays in the shell it names (register: deterministic replay across shells; each printed line must replay on Windows); a hooked stop is a stopped line with exit 130 (register: stop window on every platform). - Reach: the classify oracle's 276 generated histories (new adversarial name dimension, drawn last); the Windows replay fixtures; one stop per run. - RED: the parent printer (ebc39e6, with only the platform parameter added) fails the new classify oracle on two names: `"vitest run tests/f0.test.ts -t '^’ t0$'"` (the PowerShell line ends its quote at U+2019) and the cmd line for `q"t` (`which the shell does not read as plain words`). Run locally, exit 1. - GREEN: classify 276/276 on Linux (current code); POSIX stop witness outcome (a), exit 130 (local, under the e2e lock). Windows: pending CI on this head. - Mutant: the parent printer is the mutant (the curly-quote and cmd forms before this commit); the RED run above is its rejection. - Limits: - the stop timing (400 ms after spawn) is an assumption about the fixture's length; a run that ends first is reported as a failure, not passed; - cmd's `!` is literal (delayed expansion off), an assumption; - the classify fixed seeds' later draws shift (the new field is drawn last); - the win32 argv cap, the pipe name's machine scope, the POSIX state paths moving to the temporary directory's real path (an older daemon is found only after its idle timeout), and the unsupervised win32 process-tree exemption (#74) are recorded in the PR body. Co-authored-by: Isaac <no-reply@databricks.com>
… drives only on win32 The Linux run of 50b4ab5 failed gap-report-path-spelling: posix() acts only on win32 (review F5), so on POSIX a backslash is a name's character and a drive letter's case is a different name. The oracle now generates backslash joints and mixed drive letters only on win32; on POSIX it generates `/` joints and `C`. This narrows the law on POSIX (recorded): the POSIX law is the forward-slash spelling, and the separator and drive-case laws are judged on win32 (the Windows classify job runs the oracle there). Evidence block - Law: a path's separators and a drive's case change nothing the gap CLI decides on win32; on POSIX they are name characters (paths.mjs, F5). - Reach: the same 200 histories, with the separator and drive alphabets by platform. - RED: Linux shard 3 of 50b4ab5: "observed value differs" on the path-spelling oracle (2 of 5 tests). - GREEN: local POSIX 5/5. - Mutant: the win32 mutants run in the Windows classify job; not re-run on POSIX. - Limits: on POSIX the separator and drive laws are not generated. Co-authored-by: Isaac <no-reply@databricks.com>
…e POSIX F5 witness Coordinator rulings, applied: 1. Stop trigger: an event, not a clock. The witness writes the stop file on the CLI's first stderr line. Measured (2026-11-14): on a pipe the CLI streams its progress on stderr and prints the done summary (stdout) only at the end, so stdout is no start marker. No stream.log exists on this branch; the stderr line is the event. The 400 ms timer is gone. 2. Classify generator: the hostile names are derived from the test's own fields (HOSTILE[(ms+ti) % 7] for one test in four), not drawn. A drawn field would shift every later draw of the fixed seeds. Proof: `git diff origin/main` on the classify oracle changes no `fc.` line (zero), so the generator is unchanged. 3. Path spelling on POSIX: the oracle generates backslash joints and `x:` drives on every platform. The model splits the spelled strings per platform (win32: both separators, case-insensitive drive; POSIX: `/` only, a backslash is a name's character). On POSIX a disk witness creates the backslash name and asserts it exists, realPathOf keeps it native, and posix() leaves it unchanged (F5). This removes the earlier POSIX narrowing. Evidence block - Law: the stop is delivered at a run event and ends stopped (register: stop window on every platform); a backslash name on POSIX is kept native (review F5). - Reach: the path-spelling oracle's 200 histories, now with backslash and `x:` names on POSIX (disk-backed); the classify oracle's existing histories, with hostile names on one test in four. - RED: ebc39e6 with this oracle: 2 of 5 tests fail, "the observed value differs from the owed value" (the relation and disk checks). - GREEN: POSIX local 5/5 (path-spelling); stop witness local POSIX outcome (a), exit 130 (event trigger). Windows: pending CI on this head. - Mutant: the parent (ebc39e6) is the mutant for F5 (posix() rewrites backslashes on POSIX); its RED is above. - Limits: the stdout summary arrives only at the end on pipes (a finding, not a fix here); the stop trigger assumes the CLI prints a stderr line before its work completes (measured on the fixture only). Co-authored-by: Isaac <no-reply@databricks.com>
The derived hostile names reached constructed tests too (ms 0 at index 0), which
broke the name-relationship reach of two constructed histories ("a failing
baseline test whose name another test's contains" and "... ends another
test's"). A hostile name now applies to generated tests only (ms >= 1).
Evidence block
- Law: the constructed histories reach their declared states (the oracle's own
reach rule).
- RED: de8a373 run: "constructed history ... did not enter a baseline test whose
name another test's contains" (Linux shard 3 and the Windows classify job).
- GREEN: local classify 276/276.
- Mutant: not run.
- Limits: as recorded in the previous commit.
Co-authored-by: Isaac <no-reply@databricks.com>
…an argument Run 43548823073 (head 2b7b7f3): the Windows CLI job failed on Windows PowerShell 5.1 ("Replay 3 did not reproduce in powershell", a parse exception on the fixture with a double-quoted name). The witness passed the line as a -Command argument, so Node's argument quoting turned the double quote into \" and 5.1 does not read that as an escape. A typed line never passes through an argv: the witness now pipes the line to `-Command -` on stdin, for pwsh and powershell.exe alike. Evidence block - Law: each printed line replays in the shell it names, as a user types it (the register's per-shell rule). - RED: run 43548823073, witness step: "Replay 3 did not reproduce in powershell" (the Windows PowerShell 5.1 parse exception). - GREEN: pending the next Windows run. - Mutant: the argument passing is the mutant (the run above). - Limits: a cmd line is still passed verbatim as an argument (cmd has no stdin command mode here). Co-authored-by: Isaac <no-reply@databricks.com>
…ne that does not reproduce Run 43551132190 (abc8b45): "Replay 3 did not reproduce in powershell" with no reason. The failure now carries the shell's exit code and output, so the Windows PowerShell 5.1 error for the double-quoted name is visible. Diagnostic; the witness's pass and fail rules are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
… (option c) PowerShell 5.1 passes an embedded double quote in a native .cmd argument wrongly (run 43553461458: a `"`-named fixture selected no test, exit 0). The coordinator's ruling (option c): `-t` is a coarse regular-expression filter, and the exact test is carried by ORACLE_UNIT where a line sets it. - The `-t` pattern (CLI baseline and framework Rerun) replaces each character a shell reads unsafely (`"`, `%`, `!`, the curly quotes) with `.`: still anchored, only slightly broader. - Framework environment values: PowerShell (5.1 and 7) doubles the single and curly quotes, which works for any name; cmd's `set "NAME=value"` tolerates a double quote, and `%` and `!` have no cmd form (environment values only). - Classify oracle: a CLI line has no environment values, so a cmd fallback line is a breach; the -t argument must carry no shell-hostile character. - Witness: each PowerShell Replay line's ORACLE_UNIT and ORACLE_PROPERTY decode to the printed text (round-trip), in addition to the reproduction in pwsh, Windows PowerShell 5.1, and cmd. Evidence block - Law: a printed command replays exactly the unit it names, in each shell (register: deterministic replay across shells; the coordinator's option c). - Reach: the classify oracle's 276 histories (the adversarial names, derived); the Windows fixtures (`’`, `"`, `%`). - RED: run 43553461458: "Replay 3 did not reproduce in powershell", exit 0, PowerShell 5.1 with an embedded double quote in the -t argument. - GREEN: classify 276/276 and framework 114/114 locally (POSIX); the Windows witness runs on this head. - Mutant: the pre-fix pattern (a raw double quote in -t) is the mutant; the RED above is its rejection. - Limits: the -t pattern is slightly broader for names with a hostile character (the wildcard may select a test whose name differs in that position, which the classify oracle's names cannot produce). Co-authored-by: Isaac <no-reply@databricks.com>
…for a line that does not reproduce" This reverts commit 69c82eb.
… fixtures (round-3 review) - F1: a cmd environment value containing `"`, `%`, or `!` has no form (the register's rule); `set "VAR=value" &&` is only emitted for a plain value (oracle-property.ts commandOf). Fixtures: one with a curly-quote property and no `"%!` (a cmd set line runs); one with a single `"` in its property (its cmd line points to PowerShell). - F2: the PowerShell decoder's offset is the value's own position, and decoded values are compared exactly with a failing unit's property and unit, as vitest prints them (`FAIL <file> > <property> > <unit>`). cmd `set` lines decode too and must agree with the PowerShell line. - F3 (partial): Rerun lines carry no ORACLE_UNIT: the framework reads ORACLE_UNIT only for a random unit during a path replay, so a fixed-seed Rerun cannot be selected exactly. Reported to the coordinator. - F6: the path-spelling comment states what it means (no history to keep). - F7: the classify readers reject `!` and judge `set … &&` prefixes; the fallback is allowed wherever the printer uses it (a file path with a hostile character). - F9: control characters in rerunPatternOf are escaped as stream.mjs does (by code point, not a control-character regex). - The witness fails when a failing unit prints no per-shell line (FAIL lines are counted per unit). Evidence block - Law: a printed command replays the failing unit it names, in each shell (register: deterministic replay across shells; the set rule). - Reach: the classify oracle's 276 histories; the Windows fixtures. - RED: run 43555531142 and the coordinator's round-3 review at ae44184 (a `"` in a set value broke `&&`). - GREEN: classify 276/276, framework 114/114, locally (POSIX). Windows: this head. - Mutant: the pre-fix env rule (a `"` in a set value) is the mutant. - Limits: Rerun lines are not exact (see F3). Co-authored-by: Isaac <no-reply@databricks.com>
…uns on the package's PATH Run 43561333340 (9698922), Windows CLI job: "2 random units failed, but 4 Replay lines printed" (vitest repeats a failing unit's error text, so its replay lines repeat), and every line failed with "'vitest' is not recognized" (the rewritten runIn dropped the PATH that adds the package's node_modules/.bin). - Each distinct printed line counts once, in first-occurrence order. - Every shell runs with the package's binaries on PATH. Evidence block - Law: a printed command replays in each shell it names (the register's per-shell rule); each failing unit prints one line (witness rule). - RED: run 43561333340, the witness: "4 Replay lines printed" for 2 random units; "'vitest' is not recognized". - GREEN: pending the next Windows run. - Mutant: not run. - Limits: as in the previous commit (Rerun lines are not exact). Co-authored-by: Isaac <no-reply@databricks.com>
… line and checks coverage Run 43563021693 (257a5b0): "2 random units failed, but 4 Replay lines printed" and "Replay 4 has no cmd line": a failing fixed unit prints its replay under the random unit's name, so one failing unit can have more than one replay line, and index pairing breaks. A random unit's violation also prints its replay under the Rerun label, so a Rerun line can carry ORACLE_ variables, and running it with `vitest run` in front broke it. - Each distinct PowerShell line pairs with the cmd line printed after it. - A line that carries ORACLE_ variables is a replay and runs as printed; a bare -t line is a rerun and runs as `vitest run` plus its arguments. - Coverage: every failing unit is matched by a replay whose ORACLE_PROPERTY and ORACLE_UNIT decode to its names (a fixed unit by the random unit it replays). Evidence block - Law: every failing unit prints a replay that names it exactly, and each printed command replays in each shell it names (register: deterministic replay; the witness rule). - RED: run 43563021693: "2 random units failed, but 4 Replay lines printed"; "Replay 4 has no cmd line"; "Rerun 2 did not reproduce in cmd". - GREEN: pending the next Windows run. - Mutant: not run. - Limits: a bare Rerun line is not exact (see F3). Co-authored-by: Isaac <no-reply@databricks.com>
…ewrite removed Run 0c42c33 (the Windows CLI job): "ReferenceError: decodePs is not defined". The pairing rewrite replaced the block that held the FAIL map, the PowerShell and cmd decoders, and the no-form constant. They are restored, and a local smoke run of the replay branch on a fabricated output reaches the report (pwsh is absent on this Mac, so its runs fail as expected; no reference error). Evidence block - Law: the replay witness reports a failing unit's printed commands (witness rule). - RED: run 0c42c33: ReferenceError in the replay step. - GREEN: pending the next Windows run. - Mutant: not run. - Limits: as before. Co-authored-by: Isaac <no-reply@databricks.com>
… (F3 ruling b) The coordinator's ruling: Rerun lines stay coarse -t filters (a sibling whose name the anchored pattern matches also runs). The witness now counts a Rerun line only when the failing fixed unit whose exact name it selects shows its own FAIL line in its output, so a sibling's failure cannot satisfy it. An exact selector is tracked in #78; Replay lines select exactly, through ORACLE_PROPERTY and ORACLE_UNIT. Evidence block - Law: a Rerun line runs the failing unit it names, and only it counts (register amendment: Replay lines select exactly; Rerun lines are anchored, coarse filters). - Reach: the Windows fixtures (the `Q"1` unit has no Rerun line in CI; the sibling case is the Limit's example). - RED: none in CI yet; the sibling case is not exercised by the fixtures. - GREEN: pending the next Windows run. - Mutant: not run. - Limits: a sibling case is not in the CI fixtures (recorded in the PR body). Co-authored-by: Isaac <no-reply@databricks.com>
… 1: <name>`) Run 43569602268 (bd208ac): "Rerun 1 selects no failing fixed unit" for both fixtures. The FAIL regex expected `fixed seed: ` with no number, but vitest prints `fixed seed 1: <name>`, so no fixed unit was recorded. (The earlier fix attempt did not apply; this commit applies it and checks the match.) Evidence block - Law: a Rerun line counts only for the failing fixed unit whose exact name it selects (witness rule; register: Rerun lines are anchored coarse filters). - RED: run 43569602268: "Rerun 1 selects no failing fixed unit". - GREEN: pending the next Windows run. - Mutant: not run. - Limits: as recorded in the PR body (sibling case, #78). Co-authored-by: Isaac <no-reply@databricks.com>
Run 43571581118 (3b97795): each Rerun line reproduced (exit 1, "Failed Tests 1"), but the check that the failing unit's exact name appears in its output failed: vitest colors the output, so the name is split by color codes. The witness now strips the color codes before it matches a unit's name. Evidence block - Law: a Rerun line counts for the failing fixed unit whose exact name it selects and whose FAIL line its output shows (witness rule). - RED: run 43571581118: "Rerun 1 did not reproduce in pwsh (exit 1 ... Failed Tests 1)" with the name check failing. - GREEN: pending the next Windows run. - Mutant: not run. - Limits: as recorded in the PR body (sibling case, #78). Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # .github/workflows/pr.yml # packages/oracles/oracle-gaps/gap-report-classify-oracle.property.test.mjs # packages/oracles/scripts/windows-witness.mjs
…line on win32 Class 1 of the Windows static and e2e failures (merged head 5186cd5): the e2e oracle's replay check read the POSIX ` replay: ` line and ran it with `sh`. On Windows the CLI prints ` replay (PowerShell): ` (and a cmd line), so the replay command was undefined and the ownership law failed ("the replay command undefined ran undefined, owed only ..."; 3 violations). The check now reads the PowerShell line on win32 and runs it through pwsh on stdin, as a typed line runs; the comparison of the tests it ran is unchanged. Evidence block - Law: a baseline's replay command, pasted into the shell it names, runs the owed test (the ownership law; the register's per-shell rule). - Reach: the e2e histories with a failing owner (the baseline replays). - RED: run 43575409066 / 5186cd5: "the replay command undefined ran undefined, owed only [...]" (3 e2e violations, law ownership). - GREEN: pending the next Windows run. - Mutant: the POSIX-only reader is the mutant (the RED above). - Limits: the stdout gap-line count class (greppable stream) is not addressed here. Co-authored-by: Isaac <no-reply@databricks.com>
… lines A failure reported only the count of the stdout's gap lines, so a Windows run's extra lines could not be read from CI. The breach now carries the lines themselves. Co-authored-by: Isaac <no-reply@databricks.com>
… lines too Co-authored-by: Isaac <no-reply@databricks.com>
… on win32 An authored mutant is served by patching a file in a warm context, which invalidates the file first. Vite keys its module map by forward-slash paths, so invalidateFile given a backslash path found no module and the context kept the original module. The authored mutant then survived as not-observed on Windows, where the model expected it killed (the e2e shards' 'greppable stream' breaches: authored:a1 and authored:a2 on sut-less fixtures, in run 43585144306). posix() is a no-op off win32. Co-authored-by: Isaac <no-reply@databricks.com>
… spelling The refused run whose package lies inside the CLI's directory was tested with a string prefix of the CLI's directory plus a forward slash. On win32 the package's path is joined with backslashes, so the prefix never matched and the state was never reached (RED: run 43586666931, e2e shard 6/6, 'did not enter a refused run whose package lies inside the CLI's directory'). isWithin compares both paths in forward slashes. Co-authored-by: Isaac <no-reply@databricks.com>
…ellings The CLI prints its paths in forward slashes, but the static driver named only the package's join-spelled path. The scratch directory's random name stayed in the forward-slash lines, so two runs of one constructed history recorded different arguments traces on Windows (RED: run 43586666931, static shards, 'two runs ... recorded different traces at observation arguments'). The e2e driver already names both spellings. Co-authored-by: Isaac <no-reply@databricks.com>
All six shards of the Windows static oracle are green (run 43588402680), each under 3 minutes, so the job no longer continues on error and the preview waits for it. The end-to-end job stays non-blocking: two of its shards cannot pass on Windows yet (a held daemon cannot be paused, and a stop by SIGTERM or SIGINT cannot be delivered), which needs a maintainer decision on the declared reach. Co-authored-by: Isaac <no-reply@databricks.com>
…s that win32 lacks A held daemon is paused by a test-only pause at the daemon's reply (ORACLE_GAPS_TEST_PAUSE_AT=daemon-reply), which writes its pid to the check's output directory, instead of SIGSTOP (no signal pauses a process on win32). The pause fires once per check, so a daemon the CLI restarts after the death answers. On win32 a stop goes through the CLI's test stop hook (ORACLE_GAPS_TEST_STOP) and is observed as SIGINT. The SIGTERM stop state is required on POSIX only: win32 cannot deliver SIGTERM (its kill is TerminateProcess). Limits states the reason. RED (Windows): e2e shard 1/6 run 43588402680, 'a check whose daemon held it past the CLI's wait, then died' not reached; shard 4/6, 'every outcome class' sent SIGTERM and stderr had no done line. GREEN (macOS, local): 'an edit sequence through the daemon' and 'every outcome class' pass (fixed seed 20261105). Co-authored-by: Isaac <no-reply@databricks.com>
… job prints test durations A run that aborts by SIGABRT needs a signal win32 cannot deliver to a process, so its state and run are required on POSIX only, with the reason in Limits (RED: e2e shard 4/6, run 43591904999, 'the stopped run\x27s last line is done completed ... not done error, the run\x27s process exited with SIGABRT'). The Windows end-to-end step uses the verbose reporter, so each test\x27s duration is in the log for the re-shard (shard 1/6 hit the 900 s test timeout in run 43591904999). Co-authored-by: Isaac <no-reply@databricks.com>
…he idle clock A hang cut off for idleness is not reached on win32 (e2e shard 4/6, run 43593613669). The profile (ORACLE_GAPS_IDLE) now counts, per test, the samples that each condition blocked: late, io with its kinds, busy, a timer due, timers listed. The Windows end-to-end job writes it and prints it. The driver passes it through GAP_IDLE_PROFILE, since cleanEnv drops the ORACLE_GAPS_ prefix. Diagnostic only; no behavior change. Co-authored-by: Isaac <no-reply@databricks.com>
Reverts the POSIX-only scoping of the run whose process aborted: win32 has the mechanism (a V8 out-of-memory abort happens there too), so the state stays required everywhere. The witness still needs a win32 way to report the abort, which the CLI's supervisor (POSIX only) does today; that is open. The idle profile writes each sample of a cuttable copy as it happens, so a copy the watchdog kills keeps its samples (diagnostic). Co-authored-by: Isaac <no-reply@databricks.com>
…dows Removes the GAP_IDLE_PROFILE pass-through and the idle print step. Adds a temporary daemon-history-profile job that runs the one fixed-seed history on ubuntu and windows with GAP_PROFILE_ALL (a temporary route of the CLI's profile spans to one file) and prints per-kind totals. Diagnostic only; remove before merge, with the GAP_PROFILE_ALL route in check.mjs. Co-authored-by: Isaac <no-reply@databricks.com>
…mings (temporary) Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
The profile job, its probes, and the GAP_PROFILE_ALL route were diagnostics for the daemon history's cost; the numbers are in the handback. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #58 (Windows CI). Traces and fixes the Windows static and e2e failures class by class, oracle-first: each class's RED is the Windows run that failed, GREEN is the Windows run that passes. Tracked in #72. Work in progress; the first class is the output namespace (the static oracle's temp directory by its native real path).
This pull request and its description were written by Isaac.