From 737055551c1b931903b40a9448b6f8b08c45821e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:37:54 +0000 Subject: [PATCH 01/21] fix(glob): bound expansion while it runs; stop ** following symlinks (#191) ClusterFuzzLite's pr_fuzzing (address) job on PR #230 (check run 134118693669) hit libFuzzer's out-of-memory limit in fuzz_glob_expansion with the input `/**/*`: 2629 MB used against 2560 MB, live allocations in the glob crate's fill_todo, reached from expand_glob. The fault is on main. Cause, reproduced locally before any change (Linux 6.18, release build): - The glob crate follows symbolic links to directories under `**`, so a link back to a parent never ends the walk. `**/*` over a 7-entry tree with `d/up -> ..` and `self -> .` passed 2.5 GB in 15 s; `**/no-such-name` over the same tree never returned, at 2 MB, so a cap on matches alone cannot bound it. - expand_glob collected every path and only then compared the count with MAX_GLOB_RESULTS. `/**/*` from the fuzz target's empty base dir walked the runner's whole file system, including the loops under /proc and /sys, and grew to the limit in 16 s. The real ASan fuzz target replays the CI input to the same out-of-memory report. - When the cap did fire, external-command expansion treated the error as "no match" and ran the command with the literal pattern. Fix: - glob.rs walks the tree itself. Each component is still matched by glob::Pattern; the walk processes each (directory, pattern component) pair once, in name order. `**` lists a symbolic link to a directory but does not descend into it and skips hidden directories; other components still go through links, so `link/*` works. - New src/grade.rs: a Cost grade (sequence adds) and an Occupancy grade (high-water mark), and a Meter that refuses the charge over a limit. The walk meters file-system reads (10,000,000), matches held (100,000) and bytes of matches held (6 MiB, the most Linux execve accepts for argument strings at any stack limit, measured here: 6,279,840 bytes). The grade laws are property-tested, not proved. - Reaching a limit returns GlobLimitExceeded: the limit, the usage so far and the directory being read. Nothing partial is returned, and the shell no longer substitutes the literal word for a limit error. Comparison with bash 5.2.21 (globstar on, dotglob off) on the same trees: - 7-entry looped tree: bash gives 6 paths for `**/*` in under 1 ms; vsh now gives the same 6 (was: out of memory). - 328,420-entry tree: bash expands `**/*` to 328,420 paths in 1.7 s at 63 MB; vsh stops at the 100,000-match limit after 0.46 s at 11 MB. `**/*/**/*`: bash lists 976,400 paths (328,400 distinct) in 5.3 s at 182 MB; vsh lists each path once and stops at the limit in 0.33 s. - `/**/*`: bash expands it to 643,683 paths in 4.6 s at 205 MB; vsh stops at the 6 MiB byte limit at 20 MB. bash has no limits; these are deliberate departures. - `/usr/**/*`: vsh 0.48 s, bash 0.57 s. bash finds 2,956 more paths, all below symlinked directories such as /usr/bin/X11 -> . : when `**` is not the first component, bash enters such a link for one more component. vsh does not; for `**/...` patterns the two agree. Documented in glob.rs. Also: - `.*` and `.h*` now find hidden files as in bash (before: only `.`/`..`, or nothing), and a file name that is not UTF-8 no longer panics. - The fuzz target expands over a small fixed tree (two symlink loops, a hidden directory, a non-UTF-8 name), re-roots absolute inputs into it, skips `..`, runs under small limits and checks every result against the limits and the file system. 382,036 runs in 10 minutes with -rss_limit_mb=2560: no crash. - fuzz/Cargo.lock regenerated: it lacked hmac and subtle. Tests: tests/glob_bounds_tests.rs (14). The two symlink-loop tests fail on main (timeout after 20 s) and pass here. Limit, residue and determinism tests use injected limits; the read counts are checked against counts made directly from the fixtures. A differential property test checks the walker against the glob crate where the rules did not change; three planted bugs were each caught by it. security_recursive_glob_bounded now asserts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- CHANGELOG.adoc | 29 + EXPLAINME.adoc | 3 +- docs/POSIX_COMPLIANCE.adoc | 15 +- docs/PROOF-OPEN-FRONTIER.adoc | 1 + impl/rust-cli/fuzz/Cargo.lock | 20 + .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 140 ++-- impl/rust-cli/src/external.rs | 77 ++- impl/rust-cli/src/glob.rs | 632 ++++++++++++++++-- impl/rust-cli/src/grade.rs | 493 ++++++++++++++ impl/rust-cli/src/lib.rs | 1 + impl/rust-cli/tests/glob_bounds_tests.rs | 395 +++++++++++ impl/rust-cli/tests/security_tests.rs | 29 +- 12 files changed, 1713 insertions(+), 122 deletions(-) create mode 100644 impl/rust-cli/src/grade.rs create mode 100644 impl/rust-cli/tests/glob_bounds_tests.rs diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index de48c3d5..de152145 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -11,6 +11,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 == [Unreleased] +=== Fixed — 2026-11-18 (glob expansion out of memory, #191) + +- `**` no longer follows symbolic links to directories, so a link back to a + parent directory can no longer make an expansion grow without end. On the old + code `**/*` over a 7-entry tree with two such links passed 2.5 GB in 15 s, + and `**/no-such-name` over it never returned. ClusterFuzzLite hit the same + fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run + 134118693669). The rule now matches bash 5.2 with `globstar` for patterns + that start with `**`. +- Glob expansion is metered while it runs: at most 10,000,000 file-system + reads, 100,000 matches and 6 MiB of matches per pattern. Reaching a limit is + a `GlobLimitExceeded` error that names the limit, the usage so far and the + directory being read; the command is not run. Before, the match limit was + checked only after every path had been collected, and an external command + then received the literal pattern instead of an error. +- `.*` and `.h*` find hidden files, as bash does. +- A file name that is not valid UTF-8 no longer panics the expansion. +- The `fuzz_glob_expansion` target expands over a small fixed tree instead of + the runner's file system, and checks the limits on every input. +- `impl/rust-cli/fuzz/Cargo.lock` regenerated; it was missing `hmac` and + `subtle`. + +=== Added — 2026-11-18 + +- `src/grade.rs`: resource grades. A cost adds up in sequence; an occupancy + keeps a high-water mark. A `Meter` refuses the charge that would pass its + limit. Glob expansion is the first user; operation-history retention (#190) + is the intended second. The grade laws are property-tested, not proved. + === Changed — 2026-11-01 (assurance-boundary re-audit) - Corrected prose that had drifted across the assurance boundary. The audit diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 7f0fc4d8..11279c02 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -212,7 +212,8 @@ All share: *Multiple proof systems for cross-validation, honest status reporting | `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine) -| `impl/rust-cli/src/glob.rs` | POSIX-compliant glob expansion: *, ?, [...], {...}. Hidden file handling (requires explicit leading dot). +| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash globstar rules), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). +| `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved. | `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT). diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index cc34cfb5..e8202f26 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -152,7 +152,8 @@ creating empty exported var [width="100%",cols="29%,25%,46%",options="header",] |=== |Feature |Status |Implementation -|`+*+` (match any) |Done |`+glob+` crate +|`+*+` (match any) |Done |`+glob+` crate `+Pattern+` per component; +directory walk in `+glob.rs+` (2026-11-18) |`+?+` (single char) |Done | @@ -161,7 +162,17 @@ creating empty exported var |`+[!...]+` negation |Done | |Hidden file exclusion |Done |`+require_literal_leading_dot: true+` -(fixed 2026-02-12) +(fixed 2026-02-12); `+.*+` and `+.h*+` now find hidden files (fixed +2026-11-18; they previously gave only `+.+`/`+..+` or nothing) + +|Recursive `+**+` (bash globstar) |Done |Does not descend through symbolic +links or into hidden directories. One known difference from bash 5.2: bash +enters a symlinked directory for one more component when `+**+` is not the +first component (2026-11-18) + +|Expansion limits |Done |10,000,000 file-system reads, 100,000 matches, +6 MiB of matches; exceeding one is an error that names it, never a truncated +or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported diff --git a/docs/PROOF-OPEN-FRONTIER.adoc b/docs/PROOF-OPEN-FRONTIER.adoc index 0372b66f..418417b0 100644 --- a/docs/PROOF-OPEN-FRONTIER.adoc +++ b/docs/PROOF-OPEN-FRONTIER.adoc @@ -119,6 +119,7 @@ Pure induction on list length. | File | `impl/rust-cli/src/glob.rs` — no proof today. | Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property. | Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4. +| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Lean bound and no Rust contract. |=== [[A-7]] diff --git a/impl/rust-cli/fuzz/Cargo.lock b/impl/rust-cli/fuzz/Cargo.lock index ec6f8271..7d73278a 100644 --- a/impl/rust-cli/fuzz/Cargo.lock +++ b/impl/rust-cli/fuzz/Cargo.lock @@ -469,6 +469,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", "crypto-common", + "subtle", ] [[package]] @@ -724,6 +725,15 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1706,6 +1716,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + [[package]] name = "syn" version = "1.0.109" @@ -1984,13 +2000,17 @@ dependencies = [ "dirs", "gag", "glob", + "hex", + "hmac", "libc", + "nix 0.31.2", "nu-ansi-term", "ratatui", "reedline", "regex", "serde", "serde_json", + "sha2", "thiserror 2.0.18", "tokio", "uuid", diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index beecbd9d..0c8c4882 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -2,56 +2,114 @@ // Copyright (c) Jonathan D.A. Jewell //! Fuzz Target: Glob Expansion //! -//! Tests glob pattern matching for: -//! - Malicious patterns (DoS via backtracking) -//! - Deep recursion (**/**/...) -//! - Edge cases (empty patterns, special chars) -//! - Resource exhaustion +//! Expands arbitrary patterns over one small, fixed directory tree, so the +//! target exercises the shell's own walker and limits rather than the size of +//! the machine it runs on. (Until 2026-11-17 an absolute input such as `/**/*` +//! walked the runner's whole file system, following symbolic-link loops under +//! /proc and /sys, and hit libFuzzer's 2560 MB limit.) +//! +//! - Absolute patterns are re-rooted under the fixture. +//! - Patterns with a `..` component are skipped: they would leave the fixture. +//! - Expansion runs under small limits so that inputs reach them, and every +//! result is checked against the limits and against the file system. +//! +//! The fixture holds the cases that matter for termination and matching: +//! two symbolic-link loops, a dangling link, a hidden directory and a name +//! that is not valid UTF-8. #![no_main] use libfuzzer_sys::fuzz_target; +use std::fs; +use std::os::unix::ffi::OsStrExt; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::OnceLock; use tempfile::TempDir; -use vsh::glob::expand_glob; +use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; + +const LIMITS: GlobLimits = GlobLimits { + max_reads: 20_000, + max_matches: 2_000, + max_match_bytes: 256 * 1024, +}; + +fn fixture() -> &'static Path { + static FIXTURE: OnceLock = OnceLock::new(); + FIXTURE + .get_or_init(|| { + let t = TempDir::new().expect("create fixture"); + let root = t.path(); + for dir in ["d/sub", ".hd/x", "e"] { + fs::create_dir_all(root.join(dir)).unwrap(); + } + for file in [ + "a.txt", "b.txt", "c.log", ".hidden", "d/f.rs", "d/sub/g.rs", ".hd/x/y", "e/z", + ] { + fs::write(root.join(file), b"").unwrap(); + } + let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); + symlink("..", root.join("d/up")).unwrap(); // loop back to the root + symlink(".", root.join("self")).unwrap(); // loop to itself + symlink("d", root.join("link")).unwrap(); + symlink("missing", root.join("dangling")).unwrap(); + t + }) + .path() +} fuzz_target!(|data: &[u8]| { - // Convert to string - if let Ok(pattern) = std::str::from_utf8(data) { - // Limit length to prevent DoS - if pattern.len() > 200 { - return; - } + let Ok(input) = std::str::from_utf8(data) else { + return; + }; + if input.len() > 200 || input.split('/').any(|part| part == "..") { + return; + } - // Skip patterns with too many wildcards (prevent combinatorial explosion) - let wildcard_count = pattern.matches('*').count() + pattern.matches('?').count(); - if wildcard_count > 10 { - return; - } + let base = fixture(); + let pattern = if input.starts_with('/') { + format!("{}{}", base.display(), input) + } else { + input.to_string() + }; - // Skip patterns with excessive recursion - let recursive_count = pattern.matches("**").count(); - if recursive_count > 3 { - return; - } + match expand_glob_with_limits(&pattern, base, &LIMITS) { + Ok(expansion) => { + let usage = expansion.usage; + assert!(usage.reads.total() <= LIMITS.max_reads); + assert!(usage.matches.peak() <= LIMITS.max_matches); + assert!(usage.match_bytes.peak() <= LIMITS.max_match_bytes); + assert_eq!(expansion.paths.len() as u64, usage.matches.held()); + + let bytes: u64 = expansion + .paths + .iter() + .map(|p| p.as_os_str().len() as u64 + 1) + .sum(); + assert_eq!(bytes, usage.match_bytes.held()); - // Create temporary directory for glob expansion - let temp = match TempDir::new() { - Ok(t) => t, - Err(_) => return, - }; - - // Expand glob (should complete quickly or return error) - // Should NEVER: - // - Hang (exponential backtracking) - // - Panic - // - Use excessive memory - let _ = expand_glob(pattern, temp.path()); - - // Test cases include: - // - Simple wildcards: *.txt, file?.rs - // - Recursive: **/*.rs - // - Character classes: [abc], [!xyz] - // - Braces: {a,b,c} - // - Malicious: *, **, *{,/*}{,/*}{,/*}... (DoS) + // Sorted, each path once, and every path exists. + assert!(expansion.paths.windows(2).all(|w| w[0] < w[1])); + for path in &expansion.paths { + let real: PathBuf = if path.is_absolute() { + path.clone() + } else { + base.join(path) + }; + assert!( + fs::symlink_metadata(&real).is_ok(), + "{} does not exist", + real.display() + ); + } + } + Err(err) => { + if let Some(stopped) = err.downcast_ref::() { + assert!(stopped.exceeded.level > stopped.exceeded.limit); + assert!(stopped.usage.reads.total() <= LIMITS.max_reads); + assert!(stopped.usage.matches.peak() <= LIMITS.max_matches); + assert!(stopped.usage.match_bytes.peak() <= LIMITS.max_match_bytes); + } + } } }); diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 4dbc9f94..60f345a3 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -479,8 +479,11 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { /// /// For each argument: /// - If it contains glob metacharacters (*, ?, [, {), expand it -/// - If expansion succeeds, use matching paths -/// - If expansion fails or returns empty, use literal pattern (POSIX behavior) +/// - If expansion finds matches, use them +/// - If expansion finds nothing, or the word is not a valid pattern, use the +/// word literally (POSIX behavior) +/// - If expansion stops at a limit, fail: the command is not run with a +/// literal or partial argument list in place of the real expansion /// - If no metacharacters, use argument as-is /// /// # POSIX Behavior @@ -495,10 +498,22 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { /// expand_glob_args(&["rm", "*.xyz"], state) // -> ["rm", "*.xyz"] (no .xyz files) /// ``` fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> { - let mut expanded: Vec = Vec::new(); - // Get current working directory for glob expansion let cwd = std::env::current_dir().context("Failed to get current working directory")?; + expand_glob_args_in(args, &cwd, &glob::GlobLimits::DEFAULT) +} + +/// [`expand_glob_args`] with an explicit directory and limits. +/// +/// # Errors +/// Returns the [`glob::GlobLimitExceeded`] error of the first argument whose +/// expansion reaches a limit. +fn expand_glob_args_in( + args: &[String], + cwd: &std::path::Path, + limits: &glob::GlobLimits, +) -> Result> { + let mut expanded: Vec = Vec::new(); for arg in args { // Check if argument contains glob metacharacters @@ -507,25 +522,24 @@ fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> let brace_expanded = glob::expand_braces(arg); // Then expand each brace result for glob patterns - let mut glob_matches: Vec = Vec::new(); for pattern in &brace_expanded { // Expand glob pattern relative to current directory - match glob::expand_glob(pattern, &cwd) { - Ok(matches) if !matches.is_empty() => { - // Found matches - add them as strings - for path in matches { - glob_matches.push(path.to_string_lossy().to_string()); - } - } - _ => { - // No matches or error - use literal pattern (POSIX behavior) - glob_matches.push(pattern.clone()); + match glob::expand_glob_with_limits(pattern, cwd, limits) { + Ok(expansion) if !expansion.paths.is_empty() => { + expanded.extend( + expansion + .paths + .iter() + .map(|path| path.to_string_lossy().to_string()), + ); } + // A limit is not "no match": substituting the literal word + // would run the command on the wrong arguments. + Err(e) if e.is::() => return Err(e), + // No matches or not a valid pattern - use literal (POSIX behavior) + _ => expanded.push(pattern.clone()), } } - - // Add all matches (or literal if no matches) - expanded.extend(glob_matches); } else { // No glob pattern - use argument as-is expanded.push(arg.clone()); @@ -813,4 +827,31 @@ mod tests { let exit_code = execute_pipeline(&stages, &[], &mut state).unwrap(); assert_eq!(exit_code, 0, "Single-stage pipeline should work"); } + #[test] + fn glob_args_use_the_literal_word_when_nothing_matches() { + let dir = tempfile::TempDir::new().unwrap(); + std::fs::write(dir.path().join("a.txt"), b"").unwrap(); + let args = vec!["*.txt".to_string(), "*.none".to_string(), "[z-".to_string()]; + let out = expand_glob_args_in(&args, dir.path(), &glob::GlobLimits::DEFAULT).unwrap(); + assert_eq!(out, ["a.txt", "*.none", "[z-"]); + } + + #[test] + fn glob_args_fail_instead_of_substituting_the_literal_at_a_limit() { + let dir = tempfile::TempDir::new().unwrap(); + for i in 0..5 { + std::fs::write(dir.path().join(format!("f{i}.txt")), b"").unwrap(); + } + let limits = glob::GlobLimits { + max_matches: 4, + ..glob::GlobLimits::DEFAULT + }; + let args = vec!["-l".to_string(), "*.txt".to_string()]; + let err = expand_glob_args_in(&args, dir.path(), &limits).unwrap_err(); + let stopped = err + .downcast_ref::() + .expect("the limit error must reach the caller"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert!(err.to_string().contains("over the limit of 4"), "{err}"); + } } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 0a5ba18d..ae5919f8 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -2,8 +2,72 @@ // Copyright (c) Jonathan D.A. Jewell //! Glob pattern expansion (filename wildcards) //! -//! Implements POSIX-compliant glob expansion with *, ?, [...], and {...}. -//! Glob patterns are expanded to matching file paths at command execution time. +//! Expands `*`, `?`, `[...]` and the recursive `**` against the file system, +//! with fixed limits on how much work and memory one expansion may use. +//! Brace expansion (`{a,b}`) is a separate, purely textual step: see +//! [`expand_braces`]. +//! +//! # Matching rules +//! +//! The rules follow bash 5.2 with `shopt -s globstar`, `dotglob` off and +//! `globskipdots` on (the default): +//! +//! - `*`, `?` and `[...]` match within one path component. Single components +//! are matched by the `glob` crate's [`glob::Pattern`]. +//! - A name that starts with `.` is matched only by a component that itself +//! starts with a literal `.`. So `*` skips hidden files, `.*` and `.h*` find +//! them, and `.` and `..` are never produced by a wildcard. +//! - A component that is exactly `**` matches zero or more directories. It +//! lists a symbolic link to a directory but does not descend into it, and it +//! does not enter hidden directories. Other components do go through +//! symbolic links, so `link/*` lists the target directory. +//! One measured difference: when `**` is *not* the first component (an +//! absolute pattern, `./**/*`, `d/**/*`), bash 5.2 enters a symbolic link to +//! a directory found by `**` for one more component, and vsh does not. On +//! one machine `/usr/**/*` gave bash 2,956 more paths than vsh, every one of +//! them below a link such as `/usr/bin/X11 -> .`. +//! - A pattern that ends in `**` lists everything below its directory, and the +//! directory itself when it is named (`d/**` gives `d/`, `d/e`, ...). +//! - A pattern that ends in `/` matches directories only, and each result ends +//! in `/`. +//! - Results are sorted and each path appears once. (bash lists a path again +//! for every way the pattern matches it, so `**/*/**/*` can repeat a path +//! many times.) +//! - A pattern that matches nothing gives an empty list; the caller decides +//! whether to keep the word literally. +//! +//! # Limits +//! +//! One expansion is metered on three axes ([`GlobLimits`]): +//! +//! | Axis | Kind | Default | +//! |------|------|---------| +//! | file-system reads (directories opened, entries listed, names looked up) | cost: adds up | 10,000,000 | +//! | matches held | occupancy: high-water mark | 100,000 | +//! | bytes of matches held (path length + 1 for each match) | occupancy | 6 MiB | +//! +//! The meters are charged *while* the walk runs, so an expansion stops at the +//! first unit over a limit instead of collecting everything first. Stopping +//! returns a [`GlobLimitExceeded`] error that names the limit and carries the +//! usage so far and the directory being read; no partial result is returned. +//! Because entries are visited in sorted order, the same pattern over the same +//! tree always stops at the same place with the same report. +//! +//! The default byte limit is 6 MiB because Linux refuses any `execve` whose +//! argument strings take more than that, whatever the stack limit (measured on +//! Linux 6.18: 6,279,840 bytes accepted at both a 64 MiB and an unlimited +//! stack), so a larger expansion could never reach an external command. +//! +//! bash has no such limits: it expanded `/**/*` on a test machine to 643,683 +//! paths. The limits here are deliberate departures, made so that a single +//! word cannot exhaust the shell's memory. +//! +//! # Work bound +//! +//! Each pair (directory, pattern component) is processed at most once, so the +//! walk reads each directory at most once per component of the pattern. That +//! is the `O(|pattern| x |entries|)` shape stated as frontier item A-6 in +//! `docs/PROOF-OPEN-FRONTIER.adoc`; here it is tested, not proved. //! //! # Examples //! ```no_run @@ -14,12 +78,115 @@ //! // Returns: ["file1.txt", "file2.txt", ...] //! ``` -use anyhow::{bail, Context, Result}; +use crate::grade::{Cost, Exceeded, Meter, Occupancy}; +use anyhow::{Context, Result}; +use std::collections::{BTreeSet, HashSet}; +use std::ffi::OsString; +use std::fmt; +use std::fs; use std::path::{Path, PathBuf}; -/// Maximum number of results from glob expansion (prevents resource exhaustion from -/// patterns like `/**/*` on deep directory trees). -const MAX_GLOB_RESULTS: usize = 100_000; +/// Default limit on file-system reads in one expansion. +pub const MAX_GLOB_READS: u64 = 10_000_000; + +/// Default limit on the number of matches one expansion may hold. +pub const MAX_GLOB_RESULTS: u64 = 100_000; + +/// Default limit on the bytes of matches one expansion may hold: 6 MiB, the +/// most Linux `execve` accepts for argument strings. +pub const MAX_GLOB_MATCH_BYTES: u64 = 6 * 1024 * 1024; + +/// The limits one glob expansion runs under. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GlobLimits { + /// Most file-system reads: directories opened, entries listed, names looked up. + pub max_reads: u64, + /// Most matches held at once. + pub max_matches: u64, + /// Most bytes of matches held at once (path length + 1 per match). + pub max_match_bytes: u64, +} + +impl GlobLimits { + /// The limits [`expand_glob`] uses. + pub const DEFAULT: GlobLimits = GlobLimits { + max_reads: MAX_GLOB_READS, + max_matches: MAX_GLOB_RESULTS, + max_match_bytes: MAX_GLOB_MATCH_BYTES, + }; +} + +impl Default for GlobLimits { + fn default() -> Self { + GlobLimits::DEFAULT + } +} + +/// What one expansion used, on each metered axis. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct GlobUsage { + /// File-system reads: directories opened, entries listed, names looked up. + pub reads: Cost, + /// Matches held. + pub matches: Occupancy, + /// Bytes of matches held (path length + 1 per match). + pub match_bytes: Occupancy, +} + +impl fmt::Display for GlobUsage { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} file-system reads, {} matches, {} bytes of matches", + self.reads.total(), + self.matches.peak(), + self.match_bytes.peak() + ) + } +} + +/// A successful expansion: the matches and what finding them cost. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Expansion { + /// Matching paths, sorted, each once. + pub paths: Vec, + /// The resources the expansion used. + pub usage: GlobUsage, +} + +/// The error returned when an expansion reaches one of its [`GlobLimits`]. +/// +/// It is a record of where the walk stopped, not a partial result: it says +/// that more exists and where the walk was, without listing what was found. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GlobLimitExceeded { + /// The pattern as given. + pub pattern: String, + /// The limit that was reached: which axis, its value, and the level the + /// refused step would have reached. + pub exceeded: Exceeded, + /// What had been used when the walk stopped. + pub usage: GlobUsage, + /// The directory being read when the walk stopped, written the way the + /// matches would be (relative to the base directory unless the pattern is + /// absolute). + pub at: PathBuf, +} + +impl fmt::Display for GlobLimitExceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "glob pattern '{}' was not expanded: {} (stopped while reading '{}' after {})", + self.pattern, + self.exceeded, + self.at.display(), + self.usage + ) + } +} + +impl std::error::Error for GlobLimitExceeded {} /// Check if a string contains glob metacharacters /// @@ -58,16 +225,11 @@ pub fn contains_glob_pattern(s: &str) -> bool { false } -/// Expand a glob pattern to matching file paths +/// Expand a glob pattern to matching file paths, under [`GlobLimits::DEFAULT`]. /// -/// Uses the `glob` crate for POSIX-compliant pattern matching. -/// Returns paths sorted alphabetically (POSIX requirement). -/// -/// # POSIX Behavior -/// - Patterns that match nothing return empty vec (caller handles literal fallback) -/// - Hidden files (starting with .) require explicit match: .* or .config -/// - Results are sorted alphabetically -/// - * and ? do not match directory separators (/) +/// Relative patterns are matched below `base_dir` and give paths relative to +/// it; patterns that start with `/` give absolute paths. See the +/// [module documentation](self) for the matching rules and the limits. /// /// # Examples /// ```no_run @@ -85,56 +247,418 @@ pub fn contains_glob_pattern(s: &str) -> bool { /// ``` /// /// # Errors -/// Returns error if: -/// - Pattern is invalid -/// - Base directory cannot be accessed +/// Returns an error if the pattern is invalid, or a [`GlobLimitExceeded`] +/// (reachable with `anyhow::Error::downcast_ref`) if the expansion reaches a +/// limit. Directories that cannot be read are skipped, as bash does. pub fn expand_glob(pattern: &str, base_dir: &Path) -> Result> { - // Build full glob pattern (base_dir + pattern) - let full_pattern = if pattern.starts_with('/') { - // Absolute path - pattern.to_string() - } else { - // Relative to base_dir - base_dir.join(pattern).to_string_lossy().to_string() - }; + expand_glob_with_limits(pattern, base_dir, &GlobLimits::DEFAULT).map(|e| e.paths) +} - // Use glob crate for pattern matching with POSIX options - let options = glob::MatchOptions { - // POSIX: * and ? do not match leading dot (hidden files) - require_literal_leading_dot: true, - // Case-sensitive matching (POSIX default) - case_sensitive: true, - // Require literal separator - require_literal_separator: false, - }; +/// Expand a glob pattern under the given limits, reporting what it used. +/// +/// # Examples +/// ``` +/// use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; +/// +/// let dir = tempfile::tempdir().unwrap(); +/// for name in ["a.txt", "b.txt", "c.txt"] { +/// std::fs::write(dir.path().join(name), b"").unwrap(); +/// } +/// +/// let limits = GlobLimits { max_matches: 2, ..GlobLimits::DEFAULT }; +/// let err = expand_glob_with_limits("*.txt", dir.path(), &limits).unwrap_err(); +/// let stopped = err.downcast_ref::().unwrap(); +/// assert_eq!(stopped.exceeded.axis, "matches held"); +/// assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (2, 3)); +/// ``` +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_with_limits( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, +) -> Result { + let compiled = + Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; + let mut walker = Walker::new(pattern, base_dir, &compiled, limits); + walker.run().map_err(anyhow::Error::new)?; + Ok(Expansion { + usage: walker.usage(), + paths: walker.found.into_iter().collect(), + }) +} - let mut matches: Vec = glob::glob_with(&full_pattern, options) - .with_context(|| format!("Invalid glob pattern: {}", pattern))? - .filter_map(|entry| entry.ok()) - .collect(); +/// One path component of a compiled pattern. +#[derive(Debug)] +enum Component { + /// No metacharacters: looked up by name, without listing the directory. + Literal(String), + /// Matched against each name in the directory. + Wild(glob::Pattern), + /// `**`: zero or more directories. + Recursive, +} - // Sort results (POSIX requirement) - matches.sort(); +#[derive(Debug)] +struct Compiled { + components: Vec, + absolute: bool, + dirs_only: bool, +} - // Guard against resource exhaustion from overly broad patterns - if matches.len() > MAX_GLOB_RESULTS { - bail!( - "Glob pattern '{}' matched {} files (limit: {})", +impl Compiled { + fn new(pattern: &str) -> std::result::Result { + let mut components: Vec = Vec::new(); + for part in pattern.split('/').filter(|part| !part.is_empty()) { + let component = if part == "**" { + // `**/**` matches exactly what `**` matches. + if matches!(components.last(), Some(Component::Recursive)) { + continue; + } + Component::Recursive + } else if part.contains(['*', '?', '[']) { + Component::Wild(glob::Pattern::new(part)?) + } else { + Component::Literal(part.to_string()) + }; + components.push(component); + } + Ok(Compiled { + components, + absolute: pattern.starts_with('/'), + dirs_only: pattern.len() > 1 && pattern.ends_with('/'), + }) + } +} + +const MATCH_OPTIONS: glob::MatchOptions = glob::MatchOptions { + case_sensitive: true, + require_literal_separator: true, + require_literal_leading_dot: true, +}; + +/// The walk: a depth-first search over states (directory, component index), +/// each processed at most once. +struct Walker<'a> { + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + reads: Meter, + matches: Meter, + match_bytes: Meter, + found: BTreeSet, +} + +impl<'a> Walker<'a> { + fn new(pattern: &'a str, base: &'a Path, compiled: &'a Compiled, limits: &GlobLimits) -> Self { + Walker { pattern, - matches.len(), - MAX_GLOB_RESULTS, - ); + base, + compiled, + reads: Meter::new("file-system reads", limits.max_reads), + matches: Meter::new("matches held", limits.max_matches), + match_bytes: Meter::new("bytes of matches held", limits.max_match_bytes), + found: BTreeSet::new(), + } + } + + fn usage(&self) -> GlobUsage { + GlobUsage { + reads: self.reads.grade(), + matches: self.matches.grade(), + match_bytes: self.match_bytes.grade(), + } + } + + /// Where `path` (as it would be reported) is on the file system. + fn real(&self, path: &Path) -> PathBuf { + if self.compiled.absolute { + path.to_path_buf() + } else if path.as_os_str().is_empty() { + self.base.to_path_buf() + } else { + self.base.join(path) + } + } + + fn stopped(&self, exceeded: Exceeded, at: &Path) -> GlobLimitExceeded { + GlobLimitExceeded { + pattern: self.pattern.to_string(), + exceeded, + usage: self.usage(), + at: if at.as_os_str().is_empty() { + PathBuf::from(".") + } else { + at.to_path_buf() + }, + } + } + + fn charge_read(&mut self, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + self.reads.charge(1).map_err(|e| self.stopped(e, at)) + } + + /// Record a match, once. + fn emit(&mut self, path: PathBuf, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + let path = if self.compiled.dirs_only { + path.join("") + } else { + path + }; + if self.found.contains(&path) { + return Ok(()); + } + let bytes = path.as_os_str().len() as u64 + 1; + self.matches.charge(1).map_err(|e| self.stopped(e, at))?; + if let Err(e) = self.match_bytes.charge(bytes) { + self.matches.release(1); + return Err(self.stopped(e, at)); + } + self.found.insert(path); + Ok(()) + } + + /// List a directory in name order, charging the open and every entry. + /// An unreadable directory gives no entries. + fn list( + &mut self, + dir: &Path, + ) -> std::result::Result, GlobLimitExceeded> { + self.charge_read(dir)?; + let mut entries = Vec::new(); + if let Ok(read) = fs::read_dir(self.real(dir)) { + for entry in read { + self.charge_read(dir)?; + if let Ok(entry) = entry { + if let Ok(kind) = entry.file_type() { + entries.push((entry.file_name(), kind)); + } + } + } + } + entries.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(entries) + } + + /// Whether `path` is a directory, going through a symbolic link. + fn is_dir(&self, path: &Path, kind: Option) -> bool { + match kind { + Some(kind) if kind.is_dir() => true, + Some(kind) if !kind.is_symlink() => false, + _ => fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir()), + } + } + + fn run(&mut self) -> std::result::Result<(), GlobLimitExceeded> { + let count = self.compiled.components.len(); + if count == 0 { + return Ok(()); + } + let start = if self.compiled.absolute { + PathBuf::from("/") + } else { + PathBuf::new() + }; + let mut stack: Vec<(PathBuf, usize)> = vec![(start, 0)]; + let mut seen: HashSet<(PathBuf, usize)> = HashSet::new(); + + while let Some((dir, index)) = stack.pop() { + if !seen.insert((dir.clone(), index)) { + continue; + } + let last = index + 1 == count; + let compiled = self.compiled; + // Successor states, pushed in reverse so they are visited in name order. + let mut next: Vec<(PathBuf, usize)> = Vec::new(); + + match &compiled.components[index] { + Component::Literal(name) => { + self.charge_read(&dir)?; + let child = dir.join(name); + if last { + let exists = fs::symlink_metadata(self.real(&child)).is_ok(); + if exists && (!compiled.dirs_only || self.is_dir(&child, None)) { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, None) { + next.push((child, index + 1)); + } + } + Component::Wild(pattern) => { + for (name, kind) in self.list(&dir)? { + if !pattern.matches_with(&name.to_string_lossy(), MATCH_OPTIONS) { + continue; + } + let child = dir.join(&name); + if last { + if !compiled.dirs_only || self.is_dir(&child, Some(kind)) { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, Some(kind)) { + next.push((child, index + 1)); + } + } + } + Component::Recursive => { + // Zero directories: the pattern continues here. + if !last { + next.push((dir.clone(), index + 1)); + } else if !dir.as_os_str().is_empty() { + self.emit(dir.join(""), &dir)?; + } + // One more directory: descend into real, visible directories. + for (name, kind) in self.list(&dir)? { + if name.as_encoded_bytes().starts_with(b".") { + continue; + } + let child = dir.join(&name); + if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind))) { + self.emit(child.clone(), &dir)?; + } + if kind.is_dir() { + next.push((child, index)); + } + } + } + } + stack.extend(next.into_iter().rev()); + } + Ok(()) } +} - // Convert absolute paths back to relative if needed - if !pattern.starts_with('/') { - matches = matches +#[cfg(test)] +mod walk_tests { + use super::*; + use proptest::prelude::*; + use tempfile::TempDir; + + fn expand(pattern: &str, base: &Path) -> Vec { + expand_glob(pattern, base) + .unwrap() .into_iter() - .map(|p| p.strip_prefix(base_dir).unwrap_or(&p).to_path_buf()) - .collect(); + .map(|p| p.to_string_lossy().into_owned()) + .collect() } - Ok(matches) + #[test] + fn literal_and_wild_components() { + let t = TempDir::new().unwrap(); + fs::create_dir_all(t.path().join("src/sub")).unwrap(); + for f in ["src/a.rs", "src/b.txt", "src/sub/c.rs", "top.rs"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!(expand("src/*.rs", t.path()), ["src/a.rs"]); + assert_eq!(expand("*/*/*.rs", t.path()), ["src/sub/c.rs"]); + assert_eq!(expand("./*.rs", t.path()), ["./top.rs"]); + assert_eq!( + expand("**/*.rs", t.path()), + ["src/a.rs", "src/sub/c.rs", "top.rs"] + ); + assert_eq!(expand("src/**/c.rs", t.path()), ["src/sub/c.rs"]); + assert!(expand("*.none", t.path()).is_empty()); + } + + #[test] + fn absolute_patterns_give_absolute_paths() { + let t = TempDir::new().unwrap(); + fs::write(t.path().join("x.txt"), b"").unwrap(); + let pattern = format!("{}/*.txt", t.path().display()); + assert_eq!( + expand_glob(&pattern, Path::new("/nonexistent")).unwrap(), + [t.path().join("x.txt")] + ); + } + + #[test] + fn invalid_patterns_are_errors_not_limits() { + let t = TempDir::new().unwrap(); + for bad in ["a**", "***", "[z-"] { + let err = expand_glob(bad, t.path()).unwrap_err(); + assert!(err.downcast_ref::().is_none(), "{bad}"); + } + } + + #[test] + fn a_base_directory_with_metacharacters_is_taken_literally() { + let t = TempDir::new().unwrap(); + let base = t.path().join("we[ir]d*"); + fs::create_dir(&base).unwrap(); + fs::write(base.join("f.txt"), b"").unwrap(); + assert_eq!(expand("*.txt", &base), ["f.txt"]); + } + + #[test] + fn empty_pattern_matches_nothing() { + let t = TempDir::new().unwrap(); + assert!(expand("", t.path()).is_empty()); + } + + // ------------------------------------------------------------------ + // Differential check against the `glob` crate, which this walker + // replaces. On trees without symbolic links and for patterns that do not + // end in `**` and have no component starting with `.`, the two must find + // the same set of paths (the crate can list a path twice; this walker + // lists it once). Those exclusions are exactly where the rules changed. + // ------------------------------------------------------------------ + + fn crate_glob(pattern: &str, base: &Path) -> BTreeSet { + let options = glob::MatchOptions { + require_literal_leading_dot: true, + case_sensitive: true, + require_literal_separator: false, + }; + glob::glob_with(&base.join(pattern).to_string_lossy(), options) + .unwrap() + .filter_map(|p| p.ok()) + .map(|p| p.strip_prefix(base).unwrap().to_path_buf()) + .collect() + } + + fn tree() -> impl Strategy, bool)>> { + let name = prop::sample::select(vec!["a", "b", "ab", "ba", ".h"]); + prop::collection::vec((prop::collection::vec(name, 1..4), any::()), 0..25) + } + + fn pattern() -> impl Strategy { + let component = prop::sample::select(vec![ + "*", "?", "a*", "*b", "[ab]", "[!a]*", "a", "b", "ab", "**", + ]); + (prop::collection::vec(component, 1..5), any::()).prop_filter_map( + "pattern ends in **", + |(parts, slash)| { + if parts.last() == Some(&"**") { + return None; + } + let mut p = parts.join("/"); + if slash { + p.push('/'); + } + Some(p) + }, + ) + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(96))] + #[test] + fn agrees_with_the_glob_crate(entries in tree(), pattern in pattern()) { + let t = TempDir::new().unwrap(); + for (parts, is_dir) in &entries { + let path = parts.iter().collect::(); + let full = t.path().join(&path); + if let Some(parent) = full.parent() { + if fs::create_dir_all(parent).is_err() { + continue; + } + } + let _ = if *is_dir { fs::create_dir_all(&full) } else { fs::write(&full, b"") }; + } + let ours: BTreeSet = expand_glob(&pattern, t.path()).unwrap().into_iter().collect(); + prop_assert_eq!(ours, crate_glob(&pattern, t.path()), "pattern {}", pattern); + } + } } /// Expand brace patterns: a{b,c}d -> [abd, acd] diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs new file mode 100644 index 00000000..8f5143ff --- /dev/null +++ b/impl/rust-cli/src/grade.rs @@ -0,0 +1,493 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Resource grades: deterministic accounting for bounded work and bounded holdings. +//! +//! A shell operation can use a resource in two different ways, and the two +//! compose differently: +//! +//! - **Cost** is consumed and never given back: directory entries read, +//! comparisons made. Doing one thing and then another costs the *sum*. +//! - **Occupancy** is held for a while and may be released: paths kept in +//! memory, history entries retained. What matters is the *high-water mark*, +//! the most that was held at one time. +//! +//! A [`Grade`] is a value of one of those two kinds. A [`Meter`] measures a +//! grade while the work runs and refuses a charge that would pass its limit, +//! so a limit is enforced where the resource is used and not after the fact. +//! +//! Both kinds are plain counts. They depend only on the input and on the state +//! of the file system, never on the clock or on the machine, so a test can +//! compute the expected grade independently and compare it for equality. +//! +//! # Laws +//! +//! For both kinds, [`Grade::then`] is associative with [`Grade::ZERO`] as its +//! identity, [`Grade::either`] is associative, commutative and idempotent, and +//! `then` distributes over `either`. Metering a sequence of charges gives +//! exactly the grade obtained by composing those charges with `then`. The tests +//! at the end of this file check these laws on random inputs; they are tests, +//! not proofs. +//! +//! # Relation to the estate's type theories +//! +//! The split follows the vocabulary of `tropical-types` (cost: sequence adds, +//! choice takes the maximum) and of occupancy grades (high-water marks) as +//! summarised in `docs/THEORY-FEED.adoc`. Nothing is imported from those +//! repositories and no theorem of theirs is relied on here. +//! +//! # Examples +//! +//! ``` +//! use vsh::grade::{Cost, Grade, Meter, Occupancy}; +//! +//! // Cost adds up. +//! assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); +//! +//! // Occupancy keeps what is still held and remembers the peak. +//! let held = Occupancy::of(5).then(Occupancy::of(2)); +//! assert_eq!((held.held(), held.peak()), (7, 7)); +//! assert_eq!(held.released(6).peak(), 7); +//! +//! // A meter refuses the charge that would pass its limit. +//! let mut entries: Meter = Meter::new("directory entries read", 2); +//! assert!(entries.charge(2).is_ok()); +//! let refused = entries.charge(1).unwrap_err(); +//! assert_eq!((refused.limit, refused.level), (2, 3)); +//! assert_eq!(entries.level(), 2); // the refused charge was not recorded +//! ``` + +use std::fmt; + +/// A summary of how much of one resource a piece of work used. +/// +/// See the [module documentation](self) for the two kinds and their laws. +pub trait Grade: Copy + Eq + fmt::Debug { + /// The grade of doing nothing. + const ZERO: Self; + + /// The grade of using `amount` units in a single step. + fn of(amount: u64) -> Self; + + /// Sequential composition: this work, and then `next`. + /// + /// `next` starts from whatever this work still holds. + #[must_use] + fn then(self, next: Self) -> Self; + + /// The worst case of two alternatives, of which only one happens. + #[must_use] + fn either(self, other: Self) -> Self; + + /// The single number a limit is compared against. + fn level(self) -> u64; +} + +/// A cumulative cost: a resource that is consumed and never given back. +/// +/// Sequential composition adds; the worst case of two alternatives is the +/// larger. Addition saturates at `u64::MAX` instead of wrapping. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Cost(u64); + +impl Cost { + /// The total consumed. + pub const fn total(self) -> u64 { + self.0 + } +} + +impl Grade for Cost { + const ZERO: Self = Cost(0); + + fn of(amount: u64) -> Self { + Cost(amount) + } + + fn then(self, next: Self) -> Self { + Cost(self.0.saturating_add(next.0)) + } + + fn either(self, other: Self) -> Self { + Cost(self.0.max(other.0)) + } + + fn level(self) -> u64 { + self.0 + } +} + +impl fmt::Display for Cost { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +/// An occupancy: how much of a resource is held now, and the most that was +/// ever held at one time. +/// +/// Invariant: `peak >= held`. Every constructor and operation preserves it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Occupancy { + held: u64, + peak: u64, +} + +impl Occupancy { + /// How much is held at the end of the work. + pub const fn held(self) -> u64 { + self.held + } + + /// The high-water mark: the most that was held at one time. + pub const fn peak(self) -> u64 { + self.peak + } + + /// The same work after giving back `amount` units of what it holds. + /// + /// Releasing more than is held releases everything. The peak is unchanged: + /// a release cannot undo a high-water mark. + #[must_use] + pub fn released(self, amount: u64) -> Self { + Occupancy { + held: self.held.saturating_sub(amount), + peak: self.peak, + } + } + + /// The same work after giving back everything it holds. + #[must_use] + pub fn settled(self) -> Self { + self.released(self.held) + } +} + +impl Grade for Occupancy { + const ZERO: Self = Occupancy { held: 0, peak: 0 }; + + fn of(amount: u64) -> Self { + Occupancy { + held: amount, + peak: amount, + } + } + + /// `next` runs on top of what this work still holds, so its peak is + /// measured from there. + fn then(self, next: Self) -> Self { + Occupancy { + held: self.held.saturating_add(next.held), + peak: self.peak.max(self.held.saturating_add(next.peak)), + } + } + + fn either(self, other: Self) -> Self { + Occupancy { + held: self.held.max(other.held), + peak: self.peak.max(other.peak), + } + } + + fn level(self) -> u64 { + self.peak + } +} + +impl fmt::Display for Occupancy { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} (peak {})", self.held, self.peak) + } +} + +/// A charge that a [`Meter`] refused because it would pass the limit. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Exceeded { + /// What was being counted, for example `"directory entries read"`. + pub axis: &'static str, + /// The limit that was configured. + pub limit: u64, + /// The level the refused charge would have reached. + pub level: u64, +} + +impl fmt::Display for Exceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} would reach {}, over the limit of {}", + self.axis, self.level, self.limit + ) + } +} + +impl std::error::Error for Exceeded {} + +/// Measures one grade against a limit while the work runs. +/// +/// Invariant: `level() <= limit()`. A charge that would break it is refused +/// and leaves the meter unchanged. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Meter { + axis: &'static str, + limit: u64, + grade: G, +} + +impl Meter { + /// A meter at zero for the resource named `axis`, allowing at most `limit`. + pub const fn new(axis: &'static str, limit: u64) -> Self { + Meter { + axis, + limit, + grade: G::ZERO, + } + } + + /// Record the use of `amount` more units, or refuse if that passes the limit. + /// + /// # Errors + /// Returns [`Exceeded`] and records nothing when the charge would take the + /// level over the limit. + pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + let next = self.grade.then(G::of(amount)); + if next.level() > self.limit { + return Err(Exceeded { + axis: self.axis, + limit: self.limit, + level: next.level(), + }); + } + self.grade = next; + Ok(()) + } + + /// The grade measured so far. + pub fn grade(&self) -> G { + self.grade + } + + /// The level measured so far; never more than [`Meter::limit`]. + pub fn level(&self) -> u64 { + self.grade.level() + } + + /// The configured limit. + pub fn limit(&self) -> u64 { + self.limit + } + + /// What this meter counts. + pub fn axis(&self) -> &'static str { + self.axis + } +} + +impl Meter { + /// Record that `amount` units are no longer held. + /// + /// Only an occupancy can be released. There is deliberately no such method + /// on `Meter`: a cost that has been paid cannot be taken back. + pub fn release(&mut self, amount: u64) { + self.grade = self.grade.released(amount); + } + + /// How much is held right now. + pub fn held(&self) -> u64 { + self.grade.held() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use proptest::prelude::*; + + #[test] + fn cost_adds_in_sequence_and_takes_the_larger_alternative() { + assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); + assert_eq!(Cost::of(3).either(Cost::of(4)), Cost::of(4)); + assert_eq!(Cost::ZERO.total(), 0); + } + + #[test] + fn cost_saturates_instead_of_wrapping() { + assert_eq!(Cost::of(u64::MAX).then(Cost::of(1)), Cost::of(u64::MAX)); + } + + #[test] + fn occupancy_remembers_the_peak_after_a_release() { + let work = Occupancy::of(10).released(8).then(Occupancy::of(3)); + assert_eq!(work.held(), 5); + assert_eq!(work.peak(), 10); + } + + #[test] + fn occupancy_of_two_settled_phases_is_the_larger_peak() { + let first = Occupancy::of(10).settled(); + let second = Occupancy::of(4).settled(); + let both = first.then(second); + assert_eq!((both.held(), both.peak()), (0, 10)); + } + + #[test] + fn occupancy_of_two_retained_phases_adds() { + let both = Occupancy::of(10).then(Occupancy::of(4)); + assert_eq!((both.held(), both.peak()), (14, 14)); + } + + #[test] + fn meter_accepts_up_to_the_limit_and_refuses_the_next_unit() { + let mut meter: Meter = Meter::new("steps", 3); + assert!(meter.charge(3).is_ok()); + assert_eq!(meter.level(), 3); + + let refused = meter.charge(1).unwrap_err(); + assert_eq!( + refused, + Exceeded { + axis: "steps", + limit: 3, + level: 4 + } + ); + assert_eq!(meter.level(), 3, "a refused charge must not be recorded"); + assert_eq!( + refused.to_string(), + "steps would reach 4, over the limit of 3" + ); + } + + #[test] + fn occupancy_meter_can_reuse_what_was_released() { + let mut meter: Meter = Meter::new("items held", 10); + meter.charge(10).unwrap(); + assert!(meter.charge(1).is_err()); + + meter.release(4); + assert_eq!(meter.held(), 6); + assert!(meter.charge(4).is_ok(), "released room can be used again"); + assert!(meter.charge(1).is_err()); + assert_eq!(meter.grade().peak(), 10); + } + + #[test] + fn a_meter_with_a_zero_limit_refuses_any_use() { + let mut meter: Meter = Meter::new("items held", 0); + assert!(meter.charge(0).is_ok()); + assert!(meter.charge(1).is_err()); + } + + /// A well-formed occupancy: `peak >= held`. + fn occupancy() -> impl Strategy { + (any::(), any::()).prop_map(|(held, extra)| Occupancy { + held: u64::from(held), + peak: u64::from(held) + u64::from(extra), + }) + } + + fn cost() -> impl Strategy { + any::().prop_map(Cost::of) + } + + /// The laws stated in the module documentation, for any grade. + fn check_laws(a: G, b: G, c: G) -> Result<(), TestCaseError> { + // `then` is a monoid. + prop_assert_eq!(a.then(b).then(c), a.then(b.then(c))); + prop_assert_eq!(G::ZERO.then(a), a); + prop_assert_eq!(a.then(G::ZERO), a); + // `either` is a semilattice. + prop_assert_eq!(a.either(b).either(c), a.either(b.either(c))); + prop_assert_eq!(a.either(b), b.either(a)); + prop_assert_eq!(a.either(a), a); + // `then` distributes over `either` on both sides. + prop_assert_eq!(a.then(b.either(c)), a.then(b).either(a.then(c))); + prop_assert_eq!(a.either(b).then(c), a.then(c).either(b.then(c))); + // Doing more never lowers the level. + prop_assert!(a.then(b).level() >= a.level()); + prop_assert!(a.either(b).level() >= a.level()); + Ok(()) + } + + proptest! { + #[test] + fn cost_obeys_the_grade_laws(a in cost(), b in cost(), c in cost()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_obeys_the_grade_laws(a in occupancy(), b in occupancy(), c in occupancy()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_operations_keep_peak_at_least_held( + a in occupancy(), + b in occupancy(), + amount in any::(), + ) { + for grade in [a.then(b), a.either(b), a.released(amount), a.settled()] { + prop_assert!(grade.peak() >= grade.held()); + } + prop_assert_eq!(a.released(amount).peak(), a.peak()); + prop_assert_eq!(a.settled().held(), 0); + } + + /// Metering charges one by one gives exactly the grade obtained by + /// composing them, and never passes the limit. + #[test] + fn a_cost_meter_agrees_with_composition( + charges in prop::collection::vec(0u64..1_000, 0..40), + limit in 0u64..20_000, + ) { + let mut meter: Meter = Meter::new("steps", limit); + let mut composed = Cost::ZERO; + for amount in charges { + let wanted = composed.then(Cost::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.limit, limit); + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + } + } + + /// The same for an occupancy meter, with releases mixed in. + #[test] + fn an_occupancy_meter_agrees_with_composition( + steps in prop::collection::vec((any::(), 0u64..1_000), 0..40), + limit in 0u64..5_000, + ) { + let mut meter: Meter = Meter::new("items held", limit); + let mut composed = Occupancy::ZERO; + for (is_release, amount) in steps { + if is_release { + meter.release(amount); + composed = composed.released(amount); + } else { + let wanted = composed.then(Occupancy::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, wanted.level()); + } + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + prop_assert!(meter.held() <= meter.level()); + } + } + } +} diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index 2323f8db..fd5fa34a 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -21,6 +21,7 @@ pub mod friendly_errors; pub mod fs_pure; pub mod functions; pub mod glob; +pub mod grade; pub mod help; pub mod highlighter; pub mod history; diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs new file mode 100644 index 00000000..bd613cef --- /dev/null +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -0,0 +1,395 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Glob expansion: termination, limits and resource accounting (issue #191). +//! +//! The first two tests reproduce the out-of-memory failure that +//! ClusterFuzzLite found on 2026-11-17 (`fuzz_glob_expansion`, input `/**/*`): +//! the previous expansion followed symbolic links under `**`, so a single link +//! back to a parent directory made `**/*` grow without end. Each runs the +//! expansion on a worker thread and fails if it has not finished in time, +//! instead of hanging the test run. +//! +//! The other tests use small injected limits so they are exact and fast, and +//! check the reported usage against counts computed here directly from the +//! fixture. +//! +//! Run with: `cargo test --test glob_bounds_tests` + +#![cfg(unix)] + +use std::collections::BTreeSet; +use std::fs; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::mpsc; +use std::time::{Duration, Instant}; +use tempfile::TempDir; +use vsh::glob::{expand_glob, expand_glob_with_limits, Expansion, GlobLimitExceeded, GlobLimits}; + +const TIME_ALLOWED: Duration = Duration::from_secs(20); + +/// Run `expand_glob` on a worker thread; panic if it does not finish in time. +fn expand_within_time(pattern: &'static str, base: PathBuf) -> Vec { + let (send, receive) = mpsc::channel(); + std::thread::spawn(move || { + let _ = send.send(expand_glob(pattern, &base).map_err(|e| e.to_string())); + }); + match receive.recv_timeout(TIME_ALLOWED) { + Ok(result) => result + .expect("expansion failed") + .into_iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect(), + Err(_) => panic!("expand_glob({pattern:?}) did not finish within {TIME_ALLOWED:?}"), + } +} + +fn strings(paths: &[PathBuf]) -> Vec { + paths + .iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect() +} + +fn stopped(result: anyhow::Result) -> GlobLimitExceeded { + let err = result.expect_err("the expansion must stop at a limit"); + err.downcast_ref::() + .unwrap_or_else(|| panic!("expected a limit error, got: {err}")) + .clone() +} + +/// The 7-entry tree with two symbolic-link loops used in the reproduction: +/// `d/{f1,f2}`, `d/up -> ..`, `self -> .`, `top`. +fn looped_tree() -> TempDir { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("d")).unwrap(); + for f in ["d/f1", "d/f2", "top"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("..", t.path().join("d/up")).unwrap(); + symlink(".", t.path().join("self")).unwrap(); + t +} + +/// `dirs` directories with `files` files each, plus `files` files at the top. +fn broad_tree(dirs: usize, files: usize) -> TempDir { + let t = TempDir::new().unwrap(); + for d in 0..dirs { + let dir = t.path().join(format!("d{d:03}")); + fs::create_dir(&dir).unwrap(); + for f in 0..files { + fs::write(dir.join(format!("f{f:03}")), b"").unwrap(); + } + } + for f in 0..files { + fs::write(t.path().join(format!("f{f:03}")), b"").unwrap(); + } + t +} + +// --------------------------------------------------------------------------- +// Termination over symbolic-link loops (fails on the previous implementation) +// --------------------------------------------------------------------------- + +#[test] +fn recursive_glob_over_symlink_loops_terminates_with_the_bash_result() { + let t = looped_tree(); + // bash 5.2 with globstar gives exactly these six paths on this tree. + assert_eq!( + expand_within_time("**/*", t.path().to_path_buf()), + ["d", "d/f1", "d/f2", "d/up", "self", "top"] + ); +} + +#[test] +fn recursive_glob_with_no_match_over_symlink_loops_terminates() { + let t = looped_tree(); + assert!(expand_within_time("**/no-such-name", t.path().to_path_buf()).is_empty()); +} + +#[test] +fn non_recursive_components_still_go_through_symlinks() { + let t = looped_tree(); + assert_eq!( + expand_within_time("d/up/*", t.path().to_path_buf()), + ["d/up/d", "d/up/self", "d/up/top"] + ); + assert_eq!( + expand_within_time("self/self/top", t.path().to_path_buf()), + ["self/self/top"] + ); +} + +// --------------------------------------------------------------------------- +// Each limit stops the walk, names itself, and returns nothing partial +// --------------------------------------------------------------------------- + +#[test] +fn the_match_limit_stops_the_walk_and_names_itself() { + let t = broad_tree(30, 40); // 1,230 entries + let limits = GlobLimits { + max_matches: 1_000, + ..GlobLimits::DEFAULT + }; + let started = Instant::now(); + let s = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + assert!(started.elapsed() < Duration::from_secs(5)); + + assert_eq!(s.exceeded.axis, "matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (1_000, 1_001)); + assert_eq!(s.usage.matches.peak(), 1_000); + assert_eq!(s.pattern, "**/*"); + let message = s.to_string(); + assert!(message.contains("matches held would reach 1001, over the limit of 1000")); + assert!(message.contains("was not expanded"), "{message}"); +} + +#[test] +fn the_read_limit_stops_a_pattern_that_matches_nothing() { + // A cap on matches alone cannot stop this one: nothing ever matches. + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_reads: 500, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits( + "**/no-such-name", + t.path(), + &limits, + )); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (500, 501)); + assert_eq!(s.usage.matches.peak(), 0); +} + +#[test] +fn the_byte_limit_counts_path_length_plus_one() { + let t = TempDir::new().unwrap(); + for name in ["aaaa", "bbbb", "cccc"] { + fs::write(t.path().join(name), b"").unwrap(); + } + // Each match costs 4 + 1 = 5 bytes: two fit in 10, the third does not. + let limits = GlobLimits { + max_match_bytes: 10, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (10, 15)); + assert_eq!( + s.usage.matches.held(), + 2, + "the refused match is not counted" + ); + + let limits = GlobLimits { + max_match_bytes: 15, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("*", t.path(), &limits).unwrap(); + assert_eq!(ok.usage.match_bytes.peak(), 15); +} + +#[test] +fn the_report_is_the_same_on_every_run() { + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_matches: 777, + ..GlobLimits::DEFAULT + }; + let first = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + for _ in 0..3 { + assert_eq!( + stopped(expand_glob_with_limits("**/*", t.path(), &limits)), + first + ); + } + // Entries are visited in name order. The top level gives 30 directories + // and 40 files (70 matches), then d000..d016 give 40 each (750), so the + // 778th match, the first one refused, is the 28th file of d017. + assert_eq!(first.at, PathBuf::from("d017")); + assert_eq!(first.usage.matches.peak(), 777); +} + +#[test] +fn default_limits_are_the_documented_values() { + assert_eq!( + GlobLimits::default(), + GlobLimits { + max_reads: 10_000_000, + max_matches: 100_000, + max_match_bytes: 6 * 1024 * 1024, + } + ); +} + +// --------------------------------------------------------------------------- +// Accounting: the reported usage equals a count made directly from the tree +// --------------------------------------------------------------------------- + +/// Visible, real (not symbolic-link) directories at or below `dir`, with the +/// number of entries in each. +fn visible_dirs(dir: &Path, out: &mut Vec) { + let entries: Vec<_> = fs::read_dir(dir).unwrap().map(|e| e.unwrap()).collect(); + out.push(entries.len() as u64); + for e in entries { + let hidden = e.file_name().to_string_lossy().starts_with('.'); + if e.file_type().unwrap().is_dir() && !hidden { + visible_dirs(&e.path(), out); + } + } +} + +#[test] +fn reads_for_a_two_level_wildcard_match_a_direct_count() { + let t = broad_tree(7, 5); + let e = expand_glob_with_limits("*/*", t.path(), &GlobLimits::DEFAULT).unwrap(); + // Open the top (1) and list its 7 + 5 entries, then open and list each of + // the 7 directories (1 + 5 each). + assert_eq!(e.usage.reads.total(), 1 + (7 + 5) + 7 * (1 + 5)); + assert_eq!(e.paths.len(), 35); + assert_eq!(e.usage.matches.peak(), 35); +} + +#[test] +fn reads_for_a_recursive_pattern_match_a_direct_count() { + let t = broad_tree(4, 3); + fs::create_dir_all(t.path().join("d001/deep/er")).unwrap(); + fs::create_dir(t.path().join(".hidden")).unwrap(); + fs::write(t.path().join(".hidden/x.rs"), b"").unwrap(); + symlink("d000", t.path().join("link")).unwrap(); + + let e = expand_glob_with_limits("**/*.rs", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert!(e.paths.is_empty()); + + // `**` lists every visible real directory once, and `*.rs` lists each of + // them once more: 2 x (open + entries) per directory. + let mut dirs = Vec::new(); + visible_dirs(t.path(), &mut dirs); + let expected: u64 = dirs.iter().map(|entries| 2 * (1 + entries)).sum(); + assert_eq!(e.usage.reads.total(), expected); +} + +#[test] +fn repeated_recursive_components_read_each_directory_a_bounded_number_of_times() { + // A chain of 30 nested directories. bash lists a path once per way the + // pattern can match it (736,281 results for six `**`); each (directory, + // component) pair is processed once here, so the work stays linear. + let t = TempDir::new().unwrap(); + let mut p = t.path().to_path_buf(); + for _ in 0..30 { + p.push("a"); + } + fs::create_dir_all(&p).unwrap(); + fs::write(p.join("leaf"), b"").unwrap(); + + let pattern = "**/*/**/*/**/*/**/*/**/*/**/*"; + let e = expand_glob_with_limits(pattern, t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(e.paths.len(), 26); // the paths at depth 6 or more, once each + let entries_in_tree = 31; + let components = 12; + assert!( + e.usage.reads.total() <= 2 * components * entries_in_tree, + "{} reads", + e.usage.reads.total() + ); +} + +// --------------------------------------------------------------------------- +// Matching rules shared with bash 5.2 (globstar on, dotglob off) +// --------------------------------------------------------------------------- + +/// The tree on which bash 5.2 results were recorded: +/// `d/e/g d/f d/up -> .. top .hid .hd/x/y ld/z link -> d`. +fn bash_tree() -> TempDir { + let t = TempDir::new().unwrap(); + for d in ["d/e", ".hd/x", "ld"] { + fs::create_dir_all(t.path().join(d)).unwrap(); + } + for f in ["d/f", "d/e/g", "top", ".hid", ".hd/x/y", "ld/z"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("d", t.path().join("link")).unwrap(); + symlink("..", t.path().join("d/up")).unwrap(); + t +} + +#[test] +fn results_match_bash_on_the_recorded_tree() { + let t = bash_tree(); + let cases: &[(&str, &[&str])] = &[ + ( + "**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/", &["d/", "d/e/", "d/up/", "ld/", "link/"]), + ("d/**", &["d/", "d/e", "d/e/g", "d/f", "d/up"]), + ("d/**/", &["d/", "d/e/", "d/up/"]), + ( + "**/*", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/e", &["d/e"]), + ("*/", &["d/", "ld/", "link/"]), + (".*", &[".hd", ".hid"]), + (".h*", &[".hd", ".hid"]), + ("link/*", &["link/e", "link/f", "link/up"]), + ("**/g", &["d/e/g"]), + ("d/**/g", &["d/e/g"]), + ( + "**/**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("./*", &["./d", "./ld", "./link", "./top"]), + (".hd/**", &[".hd/", ".hd/x", ".hd/x/y"]), + ("nomatch*", &[]), + ]; + for (pattern, bash) in cases { + let ours: BTreeSet = strings(&expand_glob(pattern, t.path()).unwrap()) + .into_iter() + .collect(); + let bash: BTreeSet = bash.iter().map(|s| s.to_string()).collect(); + assert_eq!(ours, bash, "pattern {pattern}"); + } +} + +#[test] +fn hidden_files_need_a_leading_dot_in_the_pattern() { + let t = TempDir::new().unwrap(); + for f in [".hid1", ".hid2", "vis"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!( + strings(&expand_glob(".*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!( + strings(&expand_glob(".hid*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!(strings(&expand_glob("*", t.path()).unwrap()), ["vis"]); + assert_eq!( + strings(&expand_glob("?hid1", t.path()).unwrap()), + Vec::::new() + ); +} + +#[test] +fn a_name_that_is_not_utf8_is_matched_without_a_panic() { + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let odd = std::ffi::OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(odd), b"").is_err() { + return; // the file system refuses such names; nothing to test + } + fs::write(t.path().join("ok.txt"), b"").unwrap(); + let found = expand_glob("*.txt", t.path()).unwrap(); + assert_eq!(found.len(), 2); + assert!(found.contains(&PathBuf::from(odd))); +} diff --git a/impl/rust-cli/tests/security_tests.rs b/impl/rust-cli/tests/security_tests.rs index 5cee2c91..c9b1b59d 100644 --- a/impl/rust-cli/tests/security_tests.rs +++ b/impl/rust-cli/tests/security_tests.rs @@ -430,12 +430,29 @@ fn security_recursive_glob_bounded() { } // Total directories: 3^1 + 3^2 + ... + 3^5 = 363 - // Glob should have depth limit or timeout for larger structures - - // This would be a recursive glob like: **/* - // Should either limit depth or timeout gracefully - - // A production glob-depth limit remains tracked by #191. + let all = vsh::glob::expand_glob("**/*", temp.path()).unwrap(); + assert_eq!(all.len(), 363); + + // Under a smaller limit the same walk stops with a typed error that names + // the limit, instead of returning a truncated list (issue #191). + let limits = vsh::glob::GlobLimits { + max_matches: 100, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/*", temp.path(), &limits).unwrap_err(); + let stopped = err + .downcast_ref::() + .expect("a limit error"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert_eq!(stopped.exceeded.limit, 100); + + // Walk budget: a pattern that matches nothing is still bounded. + let limits = vsh::glob::GlobLimits { + max_reads: 50, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/none", temp.path(), &limits).unwrap_err(); + assert!(err.is::()); } // ============================================================ From e93624abf392a0969673c75ba57cba77bfcf2234 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:37:55 +0000 Subject: [PATCH 02/21] fix(confirmation): count files without following symlinks count_files_in_tree followed every symbolic link to a directory, and its only test walked the real /tmp. With a link back to a parent directory anywhere in /tmp the walk branches at every level and does not finish: the test ran for over 10 minutes on this machine, where it had passed before, because /tmp now held the glob reproduction trees. The glob fuzz target leaves its fixture (with two such links) in the temp directory on every run, because libFuzzer exits without running destructors, so any machine that ran the fuzzer would hang this test. Count links as entries and do not follow them, as `rm -r` does; the test uses its own fixture, and a second test covers a looped tree. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- impl/rust-cli/src/confirmation.rs | 32 +++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 57f6ec4f..3d1d5c07 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -209,6 +209,11 @@ fn prompt_yes_no(prompt: &str) -> Result { Ok(input == "y" || input == "yes") } +/// Count the files a recursive removal of `path` would delete. +/// +/// Symbolic links are counted as entries and never followed, as `rm -r` does: +/// following them would count files outside the tree and, with a link back to +/// a parent directory, never finish. fn count_files_in_tree(path: &str) -> Result { let mut count = 0; @@ -216,9 +221,8 @@ fn count_files_in_tree(path: &str) -> Result { if path.is_dir() { for entry in std::fs::read_dir(path)? { let entry = entry?; - let path = entry.path(); - if path.is_dir() { - visit_dirs(&path, count)?; + if entry.file_type()?.is_dir() { + visit_dirs(&entry.path(), count)?; } else { *count += 1; } @@ -338,6 +342,26 @@ mod tests { #[test] fn test_count_files() { - let _ = count_files_in_tree("/tmp").unwrap_or(0); + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::fs::write(t.path().join("top"), b"").unwrap(); + let root = t.path().to_str().unwrap(); + assert_eq!(count_files_in_tree(root).unwrap(), 2); + } + + /// A link back to a parent directory used to make the count recurse until + /// the path was too long, branching at every level. + #[cfg(unix)] + #[test] + fn test_count_files_does_not_follow_symlink_loops() { + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::os::unix::fs::symlink("..", t.path().join("d/up")).unwrap(); + std::os::unix::fs::symlink(".", t.path().join("self")).unwrap(); + let root = t.path().to_str().unwrap(); + // d/f, d/up and self: the two links are counted, not followed. + assert_eq!(count_files_in_tree(root).unwrap(), 3); } } From 4323498ce0e1b4fd144e07ac9ea135e2221d0a66 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:41:43 +0000 Subject: [PATCH 03/21] fix(fuzz): build the glob fixture without unwrap or expect The Hypatia code_safety scan on PR #242 flagged expect() (line 41) and six unwrap() calls (lines 44-55) in the fixture setup of fuzz_glob_expansion. Setup errors are now returned with `?`; if the fixture cannot be built, the target skips its inputs instead of panicking, so a temp-directory failure is never reported as a crash. Rebuilt under ASan and fuzzed for 60 s (31,108 runs) with -rss_limit_mb=2560: no crash. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 59 ++++++++++++------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index 0c8c4882..6f778d58 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -34,28 +34,41 @@ const LIMITS: GlobLimits = GlobLimits { max_match_bytes: 256 * 1024, }; -fn fixture() -> &'static Path { - static FIXTURE: OnceLock = OnceLock::new(); +/// Build the fixture tree. Errors are returned, not unwrapped: if the temp +/// directory cannot be set up, the target skips its inputs instead of panicking. +fn build_fixture() -> std::io::Result { + let t = TempDir::new()?; + let root = t.path(); + for dir in ["d/sub", ".hd/x", "e"] { + fs::create_dir_all(root.join(dir))?; + } + for file in [ + "a.txt", + "b.txt", + "c.log", + ".hidden", + "d/f.rs", + "d/sub/g.rs", + ".hd/x/y", + "e/z", + ] { + fs::write(root.join(file), b"")?; + } + // Some file systems refuse names that are not UTF-8; the rest still works. + let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); + symlink("..", root.join("d/up"))?; // loop back to the root + symlink(".", root.join("self"))?; // loop to itself + symlink("d", root.join("link"))?; + symlink("missing", root.join("dangling"))?; + Ok(t) +} + +fn fixture() -> Option<&'static Path> { + static FIXTURE: OnceLock> = OnceLock::new(); FIXTURE - .get_or_init(|| { - let t = TempDir::new().expect("create fixture"); - let root = t.path(); - for dir in ["d/sub", ".hd/x", "e"] { - fs::create_dir_all(root.join(dir)).unwrap(); - } - for file in [ - "a.txt", "b.txt", "c.log", ".hidden", "d/f.rs", "d/sub/g.rs", ".hd/x/y", "e/z", - ] { - fs::write(root.join(file), b"").unwrap(); - } - let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); - symlink("..", root.join("d/up")).unwrap(); // loop back to the root - symlink(".", root.join("self")).unwrap(); // loop to itself - symlink("d", root.join("link")).unwrap(); - symlink("missing", root.join("dangling")).unwrap(); - t - }) - .path() + .get_or_init(|| build_fixture().ok()) + .as_ref() + .map(TempDir::path) } fuzz_target!(|data: &[u8]| { @@ -66,7 +79,9 @@ fuzz_target!(|data: &[u8]| { return; } - let base = fixture(); + let Some(base) = fixture() else { + return; + }; let pattern = if input.starts_with('/') { format!("{}{}", base.display(), input) } else { From 5038be1c088169b2a5d3c2afa8a147f727048202 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Wed, 18 Nov 2026 15:04:00 +0000 Subject: [PATCH 04/21] fix(word): expand globs for every simple command via typed words (#244) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stacked on hyperpolymath/valence-shell#242 (`claude/gifted-planck-mha9c9`), which adds `glob::expand_glob_with_limits` and `GlobLimitExceeded`. Once #242 merges, retarget this PR to `main`. ## Problem (measured on the debug binary, directory with `a.txt b.txt c.log`) | Command | Before | Now (same as bash) | |---|---|---| | `wc -c *.txt` | `wc: '*.txt': No such file or directory` | `0 a.txt` / `0 b.txt` / `0 total` | | `ls *.txt \| wc -l` | `ls: cannot access '*.txt'` | `2` | | `echo *.txt` | `*.txt` | `a.txt b.txt` | | `echo '*.txt'` | `\*.txt` | `*.txt` | | `echo a{1,2}` / `echo $((2+3))` | `a{1,2}` / `$((2+3))` | `a1 a2` / `5` | Cause: argument globbing lived only in `external::expand_glob_args`, which only `execute_external_with_redirects` called. There was also a second bug: quoted metacharacters were marked with a backslash in the token string, and nothing ever removed it. ## Change: typed word IR - New `src/word.rs`. A `Word` records the quoting of each piece (unquoted, `'…'`, `"…"`, `\x`). `expand_words` runs, in order: tilde, parameter, command and arithmetic expansion, process substitution, then brace expansion, glob expansion and quote removal. - The tokenizer builds a `Word` next to the old string form. `Command::External`, `Command::Pipeline` and `Command::Echo` now carry `Vec`. - The new stage is used for external commands (foreground, background, with or without redirections), every pipeline stage, `echo`, function-call arguments, and the `External` branch of command substitution and process substitution. - Kept from #242: - A pattern with no match, or an invalid pattern, stays as the literal word. - A `GlobLimitExceeded` fails the command without running it (`vsh: prog: …`, exit 1). - `external::expand_glob_args` is removed. Its rules and tests moved to `word.rs`. - Two smaller fixes: - `$(…)` is now read to its matching `)`, so `$((1+(2*3)))` is a single substitution. - `$(echo …)` is supported. ### Behaviour changes - An expansion error used to come back from `execute()` as `Err`. In `-c` mode that printed `Error:` plus a backtrace. Now the command exits with status 1 and does not run. - An unquoted `$X` whose value contains glob characters is now globbed, as POSIX requires. A quoted `"$X"` is not. ### Not migrated in this PR These still read the old string encoding, so their behaviour is unchanged and still differs from bash: - builtin operands (`touch '*q'` creates `\*q`; `ls *.txt` as a builtin does not glob) - redirection targets - assignment values (`X='*.txt'` stores `\*.txt`) - `case` patterns, `test`/`[[ ]]` arguments, `for` word lists Field splitting is still not done. All of this is documented in `docs/POSIX_COMPLIANCE.adoc`, Milestones 5 and 6. ## Tests - `tests/word_expansion_e2e_tests.rs` runs the `vsh` binary. It covers: - an external command without a redirect, with a redirect, and in a pipeline - builtin `echo` - quoted patterns `'…'`, `"…"` and `\*`, which stay literal with no backslash - a pattern that matches nothing - a partly quoted word (`'*'x.*`) - braces - function arguments - Unit tests in `word.rs`: the limit error, an invalid pattern, a literal backslash inside a pattern, quoted vs unquoted parameters, and refused process substitution. - A proptest in `parser.rs`: `Display` then `tokenize` gives back the same `Word` (256 cases). The domain excludes backticks inside double quotes, because the tokenizer has no `\`` escape. - Differential run of 30 commands across the old binary, the new binary and bash: every changed result now matches bash, and no builtin case changed. - `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`: clean. - `cargo test`: 887 passed, 0 failed, 14 ignored (the same ignored set as before: stress tests, the model oracle and 2 in the lib). --- _Generated by [Claude Code](https://claude.ai/code/session_01J4QvDfX8Xoq6vedAKdWBif)_ --------- Co-authored-by: Claude --- CHANGELOG.adoc | 29 + docs/POSIX_COMPLIANCE.adoc | 12 + impl/rust-cli/src/commands.rs | 2 +- impl/rust-cli/src/executable.rs | 121 ++-- impl/rust-cli/src/external.rs | 107 +-- impl/rust-cli/src/lib.rs | 1 + impl/rust-cli/src/parser.rs | 210 ++++-- impl/rust-cli/src/process_sub.rs | 14 +- impl/rust-cli/src/word.rs | 667 ++++++++++++++++++ .../tests/word_expansion_e2e_tests.rs | 126 ++++ 10 files changed, 1075 insertions(+), 214 deletions(-) create mode 100644 impl/rust-cli/src/word.rs create mode 100644 impl/rust-cli/tests/word_expansion_e2e_tests.rs diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index de152145..fb3d88be 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -11,6 +11,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 == [Unreleased] +=== Fixed — 2026-11-18 (arguments: globs, quote removal) + +- Glob expansion now applies to the arguments of every external command, + every pipeline stage, `echo` and function calls. Before, only an external + command that also had a redirection expanded its arguments: `wc -c *.txt` + passed `*.txt` to `wc`, `ls *.txt | wc -l` and `echo *.txt` did not expand. +- Quoted arguments no longer reach the program with a backslash in front of + `*`, `?`, `[` or `{`: `echo '*.txt'` printed `\*.txt` and now prints + `*.txt`. A partly quoted word globs only its unquoted part (`'*'x.*`). +- `echo` performs command substitution and arithmetic expansion + (`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works. +- `$((1+(2*3)))` and other substitutions containing `(` are read to their + matching `)`. +- An expansion error (such as a glob limit from #191) fails the command with + exit status 1 and does not run it. With a redirection the status was 127. + +=== Changed — 2026-11-18 (typed words) + +- New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces + (unquoted, single, double, backslash), and `word::expand_words` performs + tilde, parameter, command, arithmetic and process substitution, brace + expansion, glob expansion and quote removal on it. `Command::External`, + `Command::Pipeline` and `Command::Echo` now carry `Vec` arguments. + Other commands still use the older string encoding; see + `docs/POSIX_COMPLIANCE.adoc` (Milestones 5 and 6) for what is not migrated. +- Removed `external::expand_glob_args`; its rules (no match or invalid + pattern keeps the word, a limit is an error) moved to `word.rs` with their + tests. + === Fixed — 2026-11-18 (glob expansion out of memory, #191) - `**` no longer follows symbolic links to directories, so a link back to a diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index e8202f26..175e144c 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -177,6 +177,12 @@ or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported |Sorted results |Done |POSIX requirement + +|Where globs expand |Partial |Arguments of external commands (with or +without redirections), every pipeline stage, `+echo+` and function calls, +through `+word::expand_words+` (2026-11-18; before, only external commands +with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...), +`+for+` word lists, redirection targets |=== ==== Milestone 6: Quote Processing — COMPLETE @@ -194,6 +200,12 @@ quotes |Unclosed quote detection |Done | |Line continuation |Done | + +|Quote removal |Partial |Done for the arguments that go through +`+word::expand_words+` (see Milestone 5), which keeps the quoting of each +piece of a word. Builtin operands, redirection targets, assignment values, +`+case+` patterns and `+test+` arguments still receive quoted `+*?[{~+` with +a backslash in front (`+touch '*q'+` creates `+\*q+`) |=== ==== Milestone 7: Command Substitution — COMPLETE diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 4f91625d..02918fca 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -2856,7 +2856,7 @@ mod tests { let state = ShellState::new(tmp.path().to_str().unwrap()).unwrap(); let cmd = crate::parser::Command::External { program: "git".to_string(), - args: vec!["status".to_string()], + args: vec!["status".into()], redirects: vec![], background: false, }; diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index aeb72a25..7148cec7 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -488,35 +488,36 @@ impl ExecutableCommand for Command { // Expand program name (no process substitutions in program name) let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + // Word expansion: variables, substitutions, braces, globs and + // quote removal. A failed expansion (for example a glob that + // reaches a limit) fails the command without running it. + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + // Check if the command is a shell function before external execution. // `if let Some(...)` performs a single map lookup; the former // double-lookup plus expect shape could panic and duplicated // the lookup. if let Some(func_def) = state.functions.get(&expanded_program).cloned() { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); - return execute_function_call(&func_def, &expanded_args, state); + let result = execute_function_call(&func_def, &expansion.fields, state); + expansion.cleanup(); + return result; } - // Expand arguments with process substitutions - let mut all_process_subs = Vec::new(); - let mut expanded_args = Vec::new(); - - for arg in args { - let (expanded_arg, proc_subs) = - crate::parser::expand_with_process_sub(arg, state)?; - expanded_args.push(expanded_arg); - all_process_subs.extend(proc_subs); - } + let expanded_args = &expansion.fields; // Execute main command if *background { // Background execution let _job_id = external::execute_external_background( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -525,24 +526,19 @@ impl ExecutableCommand for Command { 0 // No job created }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::Success) } else { // Foreground execution let exit_code = if redirects.is_empty() { // No redirections - use simple path - external::execute_external(&expanded_program, &expanded_args) + external::execute_external(&expanded_program, expanded_args) } else { // Has redirections - use redirection-aware execution external::execute_external_with_redirects( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -552,12 +548,7 @@ impl ExecutableCommand for Command { 127 }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -569,33 +560,42 @@ impl ExecutableCommand for Command { redirects, background, } => { - // Expand variables and command substitutions in all pipeline stages - let expanded_stages: Result)>> = stages - .iter() - .map(|(program, args)| { - let expanded_program = - crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - Ok((expanded_program, expanded_args?)) - }) - .collect(); - let expanded_stages = expanded_stages?; - - if *background && !expanded_stages.is_empty() { + if *background && !stages.is_empty() { return Err(anyhow::anyhow!( "Background pipelines are not supported yet; refusing to run them in the foreground" )); } + // Expand every stage before any stage runs, so a failed + // expansion runs no part of the pipeline. + let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); + let mut process_subs = Vec::new(); + for (program, args) in stages { + let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => { + expanded_stages.push((expanded_program, expansion.fields)); + process_subs.extend(expansion.process_subs); + } + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + } + } + let exit_code = external::execute_pipeline(&expanded_stages, redirects, state) .unwrap_or_else(|e| { eprintln!("Pipeline error: {}", e); 127 }); + crate::word::Expansion { + fields: Vec::new(), + process_subs, + } + .cleanup(); + Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -684,10 +684,15 @@ impl ExecutableCommand for Command { interpret_escapes, redirects, } => { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: echo: {}", e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + let expanded_args = &expansion.fields; let output = if *interpret_escapes { expanded_args @@ -719,6 +724,7 @@ impl ExecutableCommand for Command { Ok(()) })?; } + expansion.cleanup(); Ok(ExecutionResult::Success) } @@ -1291,7 +1297,7 @@ impl ExecutableCommand for Command { if args.is_empty() { program.clone() } else { - format!("{} {}", program, args.join(" ")) + format!("{} {}", program, join_words(args)) } } Command::Pipeline { stages, .. } => { @@ -1301,7 +1307,7 @@ impl ExecutableCommand for Command { if args.is_empty() { prog.clone() } else { - format!("{} {}", prog, args.join(" ")) + format!("{} {}", prog, join_words(args)) } }) .collect(); @@ -1377,7 +1383,7 @@ impl ExecutableCommand for Command { format!("{} {} {}", left.description(), op_str, right.description()) } - Command::Echo { args, .. } => format!("echo {}", args.join(" ")), + Command::Echo { args, .. } => format!("echo {}", join_words(args)), Command::True => "true".to_string(), Command::False => "false".to_string(), Command::Read { var_names, .. } => format!("read {}", var_names.join(" ")), @@ -1645,6 +1651,15 @@ fn execute_loop( Ok(last_result) } +/// Words written back as source and separated by spaces. +fn join_words(words: &[crate::word::Word]) -> String { + words + .iter() + .map(|w| w.to_string()) + .collect::>() + .join(" ") +} + /// Match a word against a case pattern (supports glob-style wildcards) fn case_pattern_matches(word: &str, pattern: &str) -> bool { if pattern == "*" { diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 60f345a3..070e5f99 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -11,7 +11,6 @@ use std::path::PathBuf; use std::process::{Command, Stdio}; use std::time::Duration; -use crate::glob; use crate::redirection::{RedirectSetup, Redirection}; use crate::signals; use crate::state::ShellState; @@ -37,8 +36,7 @@ pub fn execute_external_with_redirects( let (stdin_cfg, stdout_cfg, stderr_cfg) = stdio_config_from_redirects(redirects, &redirect_setup, state)?; - // Expand glob patterns in arguments (Phase 6 M12) - let expanded_args = expand_glob_args(args, state)?; + // The arguments arrive already expanded, globs included (`crate::word`). // PATH lookup let executable = find_in_path(program)?; @@ -46,7 +44,7 @@ pub fn execute_external_with_redirects( // Build command with redirected stdio let mut command = Command::new(&executable); command - .args(&expanded_args) + .args(args) .stdin(stdin_cfg) .stdout(stdout_cfg) .stderr(stderr_cfg); @@ -475,80 +473,6 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { Ok(Stdio::from(reader)) } -/// Expand glob patterns in command arguments (Phase 6 M12) -/// -/// For each argument: -/// - If it contains glob metacharacters (*, ?, [, {), expand it -/// - If expansion finds matches, use them -/// - If expansion finds nothing, or the word is not a valid pattern, use the -/// word literally (POSIX behavior) -/// - If expansion stops at a limit, fail: the command is not run with a -/// literal or partial argument list in place of the real expansion -/// - If no metacharacters, use argument as-is -/// -/// # POSIX Behavior -/// - Brace expansion happens first: {a,b} -> [a, b] -/// - Then glob expansion: *.txt -> [file1.txt, file2.txt] -/// - Empty matches return literal: "*.xyz" if no .xyz files -/// -/// # Examples -/// ```ignore -/// expand_glob_args(&["echo", "*.txt"], state) // -> ["echo", "file1.txt", "file2.txt"] -/// expand_glob_args(&["ls", "file?.rs"], state) // -> ["ls", "file1.rs", "file2.rs"] -/// expand_glob_args(&["rm", "*.xyz"], state) // -> ["rm", "*.xyz"] (no .xyz files) -/// ``` -fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> { - // Get current working directory for glob expansion - let cwd = std::env::current_dir().context("Failed to get current working directory")?; - expand_glob_args_in(args, &cwd, &glob::GlobLimits::DEFAULT) -} - -/// [`expand_glob_args`] with an explicit directory and limits. -/// -/// # Errors -/// Returns the [`glob::GlobLimitExceeded`] error of the first argument whose -/// expansion reaches a limit. -fn expand_glob_args_in( - args: &[String], - cwd: &std::path::Path, - limits: &glob::GlobLimits, -) -> Result> { - let mut expanded: Vec = Vec::new(); - - for arg in args { - // Check if argument contains glob metacharacters - if glob::contains_glob_pattern(arg) { - // First, expand braces: file{1,2}.txt -> [file1.txt, file2.txt] - let brace_expanded = glob::expand_braces(arg); - - // Then expand each brace result for glob patterns - for pattern in &brace_expanded { - // Expand glob pattern relative to current directory - match glob::expand_glob_with_limits(pattern, cwd, limits) { - Ok(expansion) if !expansion.paths.is_empty() => { - expanded.extend( - expansion - .paths - .iter() - .map(|path| path.to_string_lossy().to_string()), - ); - } - // A limit is not "no match": substituting the literal word - // would run the command on the wrong arguments. - Err(e) if e.is::() => return Err(e), - // No matches or not a valid pattern - use literal (POSIX behavior) - _ => expanded.push(pattern.clone()), - } - } - } else { - // No glob pattern - use argument as-is - expanded.push(arg.clone()); - } - } - - Ok(expanded) -} - /// Handle command exit status, converting signals to exit codes fn handle_exit_status(status: std::process::ExitStatus) -> Result { #[cfg(unix)] @@ -827,31 +751,4 @@ mod tests { let exit_code = execute_pipeline(&stages, &[], &mut state).unwrap(); assert_eq!(exit_code, 0, "Single-stage pipeline should work"); } - #[test] - fn glob_args_use_the_literal_word_when_nothing_matches() { - let dir = tempfile::TempDir::new().unwrap(); - std::fs::write(dir.path().join("a.txt"), b"").unwrap(); - let args = vec!["*.txt".to_string(), "*.none".to_string(), "[z-".to_string()]; - let out = expand_glob_args_in(&args, dir.path(), &glob::GlobLimits::DEFAULT).unwrap(); - assert_eq!(out, ["a.txt", "*.none", "[z-"]); - } - - #[test] - fn glob_args_fail_instead_of_substituting_the_literal_at_a_limit() { - let dir = tempfile::TempDir::new().unwrap(); - for i in 0..5 { - std::fs::write(dir.path().join(format!("f{i}.txt")), b"").unwrap(); - } - let limits = glob::GlobLimits { - max_matches: 4, - ..glob::GlobLimits::DEFAULT - }; - let args = vec!["-l".to_string(), "*.txt".to_string()]; - let err = expand_glob_args_in(&args, dir.path(), &limits).unwrap_err(); - let stopped = err - .downcast_ref::() - .expect("the limit error must reach the caller"); - assert_eq!(stopped.exceeded.axis, "matches held"); - assert!(err.to_string().contains("over the limit of 4"), "{err}"); - } } diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index fd5fa34a..e74b4b1f 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -40,6 +40,7 @@ pub mod state; pub mod test_command; pub mod verification; +pub mod word; /// Echidna property-based verification integration (optional feature) #[cfg(feature = "echidna")] diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index da677ce3..1b51f2d0 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -9,6 +9,7 @@ use anyhow::{anyhow, Context, Result}; use crate::redirection::Redirection; +use crate::word::{Quote, Word}; /// Quote type for a word or word part #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -82,6 +83,8 @@ struct ParameterExpansion { struct QuotedWord { parts: Vec, quote_type: QuoteType, + /// The same word with the quoting of each piece kept (see `crate::word`). + typed: Word, } impl QuotedWord { @@ -89,6 +92,7 @@ impl QuotedWord { Self { parts: Vec::new(), quote_type: QuoteType::None, + typed: Word::new(), } } @@ -102,19 +106,23 @@ impl QuotedWord { } } - fn push_variable(&mut self, name: String) { + fn push_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), false, quote); self.parts.push(WordPart::Variable(name)); } - fn push_braced_variable(&mut self, name: String) { + fn push_braced_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), true, quote); self.parts.push(WordPart::BracedVariable(name)); } - fn push_command_sub(&mut self, cmd: String) { + fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.typed.push_command_sub(cmd.clone(), quote); self.parts.push(WordPart::CommandSub(cmd)); } fn push_process_sub(&mut self, sub_type: ProcessSubType, cmd: String) { + self.typed.push_process_sub(sub_type, cmd.clone()); self.parts.push(WordPart::ProcessSub(sub_type, cmd)); } } @@ -285,7 +293,8 @@ pub enum Command { // Shell builtins /// Echo: write arguments to stdout Echo { - args: Vec, + /// Arguments after the options, with their quoting. + args: Vec, no_newline: bool, interpret_escapes: bool, redirects: Vec, @@ -335,7 +344,8 @@ pub enum Command { // External command External { program: String, - args: Vec, + /// Arguments with their quoting; expanded by [`crate::word::expand_words`]. + args: Vec, redirects: Vec, background: bool, }, @@ -345,7 +355,7 @@ pub enum Command { /// Each stage is a (program, args) pair. Intermediate stages use piped stdio. /// Final redirections apply to the last stage only. Pipeline { - stages: Vec<(String, Vec)>, + stages: Vec<(String, Vec)>, redirects: Vec, background: bool, }, @@ -599,6 +609,7 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Backslash); quote_state = QuoteState::None; } @@ -609,6 +620,7 @@ fn tokenize(input: &str) -> Result> { quote_state = QuoteState::None; } else { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Single); } } @@ -631,27 +643,31 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(next_ch); + current_word.typed.push_char(next_ch, Quote::Double); } else { // Not a special char, keep backslash current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } else { current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } '$' => { // Variable expansion in double quotes push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::Double)?; } '`' => { // Backtick command substitution in double quotes push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::Double); } _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Double); } } } @@ -678,14 +694,14 @@ fn tokenize(input: &str) -> Result> { // Variable expansion '$' => { push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::None)?; } // Backtick command substitution '`' => { push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::None); } // Whitespace: end current word @@ -766,6 +782,7 @@ fn tokenize(input: &str) -> Result> { } else { // Invalid: 2>&[not 1] current_literal.push_str("2>&"); + current_word.typed.push_str("2>&", Quote::None); } } _ => { @@ -775,6 +792,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '2' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -816,6 +834,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '[' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -828,12 +847,14 @@ fn tokenize(input: &str) -> Result> { } else { // Regular ']' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } // Regular character _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } } @@ -861,12 +882,13 @@ fn tokenize(input: &str) -> Result> { fn parse_variable( chars: &mut std::iter::Peekable, word: &mut QuotedWord, + quote: Quote, ) -> Result<()> { if chars.peek() == Some(&'(') { // Command substitution: $(cmd) chars.next(); // consume '(' let cmd = parse_command_sub_dollar(chars)?; - word.push_command_sub(cmd); + word.push_command_sub(cmd, quote); } else if chars.peek() == Some(&'{') { // Braced form: ${VAR} chars.next(); // consume '{' @@ -888,7 +910,7 @@ fn parse_variable( } } - word.push_braced_variable(var_name); + word.push_braced_variable(var_name, quote); } else if let Some(&next_ch) = chars.peek() { // Simple form: $VAR or special variables. The peek above already // bound next_ch by value; chars.next() is called purely to advance @@ -897,11 +919,11 @@ fn parse_variable( if next_ch == '?' || next_ch == '$' || next_ch == '#' { // Single-character special variable chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_ascii_digit() { // Positional parameter: $0, $1, $2, etc. chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_alphabetic() || next_ch == '_' { // Variable name let mut var_name = String::new(); @@ -915,18 +937,20 @@ fn parse_variable( } } - word.push_variable(var_name); + word.push_variable(var_name, quote); } else if next_ch == '@' || next_ch == '*' { // Special positional parameters: $@ or $* chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else { // $ not followed by variable, treat as literal word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } } else { // $ at end of string word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } Ok(()) @@ -935,15 +959,14 @@ fn parse_variable( /// Parse command substitution in $(cmd) form fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> Result { let mut cmd = String::new(); - let mut depth = 1; // Track nesting depth for nested $() + let mut depth = 1; // Parenthesis depth: nested $(), $((expr)) and (...) for ch in chars.by_ref() { match ch { '(' => { - // Check if it's $( to track nested command substitution - if cmd.ends_with('$') { - depth += 1; - } + // Every `(` nests, so the `)` that closes `$((1+2))` or + // `$(echo $(pwd))` belongs to the substitution. + depth += 1; cmd.push(ch); } ')' => { @@ -1228,28 +1251,28 @@ fn parse_pipeline(tokens: &[Token]) -> Result { stages[last_stage_idx] = last_command_tokens; // Parse each stage as (program, args) - let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); + let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); for stage_tokens in stages { if stage_tokens.is_empty() { return Err(anyhow!("Empty pipeline stage")); } - let words: Vec = stage_tokens + let words: Vec<&QuotedWord> = stage_tokens .iter() .filter_map(|t| match t { - Token::Word(w) => Some(quoted_word_to_string(w)), + Token::Word(w) => Some(w), _ => None, }) .collect(); - if words.is_empty() { + let Some((program, args)) = words.split_first() else { return Err(anyhow!("Pipeline stage has no command")); - } - - let program = words[0].clone(); - let args = words[1..].to_vec(); - parsed_stages.push((program, args)); + }; + parsed_stages.push(( + quoted_word_to_string(program), + args.iter().map(|w| w.typed.clone()).collect(), + )); } Ok(Command::Pipeline { @@ -1719,6 +1742,7 @@ pub fn parse_command(input: &str) -> Result { // Separate command tokens from redirections let mut command_tokens: Vec = Vec::new(); + let mut command_words: Vec = Vec::new(); let mut redirects = Vec::new(); let mut background = false; @@ -1727,6 +1751,7 @@ pub fn parse_command(input: &str) -> Result { match &tokens[i] { Token::Word(w) => { command_tokens.push(quoted_word_to_string(w)); + command_words.push(w.typed.clone()); i += 1; } @@ -1852,9 +1877,10 @@ pub fn parse_command(input: &str) -> Result { // Parse command from tokens let cmd = &command_tokens[0]; let args: Vec = command_tokens[1..].to_vec(); + let words: Vec = command_words[1..].to_vec(); // Parse base command with redirections - parse_base_command(cmd, args, redirects, background) + parse_base_command(cmd, args, words, redirects, background) } /// Check if a string is a valid variable name @@ -2369,7 +2395,7 @@ fn apply_substring(value: &str, offset: i32, length: Option) -> String { /// /// `~user` is not yet supported (requires getpwnam). An escaped `\~` /// (produced by `quoted_word_to_string` for quoted `~`) stays literal. -fn expand_tilde(input: &str) -> String { +pub(crate) fn expand_tilde(input: &str) -> String { // An escaped tilde `\~` means the tilde was inside quotes — leave it. if let Some(stripped) = input.strip_prefix("\\~") { let mut s = String::with_capacity(input.len()); @@ -2628,14 +2654,12 @@ pub fn expand_command_substitution( process_cmd.stdout(Stdio::piped()); process_cmd.stderr(Stdio::null()); - // Expand variables and command substitutions in args before execution - let expanded_args: Result> = args - .iter() - .map(|arg| expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + // Expand the arguments (variables, command substitution, globs) + // before execution. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; - process_cmd.args(&expanded_args); + process_cmd.args(&expansion.fields); let output_result = process_cmd .output() @@ -2651,6 +2675,22 @@ pub fn expand_command_substitution( } } + Command::Echo { + args, + no_newline, + interpret_escapes: false, + redirects, + } if redirects.is_empty() => { + // echo builtin: its output is the joined, expanded arguments. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + let mut out = expansion.fields.join(" "); + if !no_newline { + out.push('\n'); + } + out + } + Command::Pwd { .. } => { // pwd builtin std::env::current_dir() @@ -3253,9 +3293,13 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { block_depth > 0 } +/// `args` are the arguments in the older string encoding, which the builtins +/// still read; `words` are the same arguments with their quoting, used by +/// `echo` and external commands. The two lists have the same length. fn parse_base_command( cmd: &str, args: Vec, + words: Vec, redirects: Vec, background: bool, ) -> Result { @@ -3382,7 +3426,8 @@ fn parse_base_command( // Re-parse the inner command: first arg is the command name, rest are its args let inner_cmd = &args[0]; let inner_args = args[1..].to_vec(); - let inner = parse_base_command(inner_cmd, inner_args, vec![], false)?; + let inner_words = words[1..].to_vec(); + let inner = parse_base_command(inner_cmd, inner_args, inner_words, vec![], false)?; Ok(Command::Explain { inner: Box::new(inner), }) @@ -3541,7 +3586,7 @@ fn parse_base_command( let mut echo_args = Vec::new(); let mut skip_flags = true; - for arg in &args { + for (arg, word) in args.iter().zip(&words) { if skip_flags && arg.starts_with('-') { for ch in arg[1..].chars() { match ch { @@ -3551,14 +3596,14 @@ fn parse_base_command( _ => { // Unknown flag — treat as regular argument skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); break; } } } } else { skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); } } @@ -3729,7 +3774,7 @@ fn parse_base_command( // Everything else: external command _ => Ok(Command::External { program: cmd.to_string(), - args, + args: words, redirects, background, }), @@ -4246,7 +4291,7 @@ mod tests { match cmd { Command::External { program, args, .. } => { assert_eq!(program, "touch"); - let expanded = expand_variables(&args[0], &state); + let expanded = expand_variables(&args[0].to_string(), &state); assert_eq!(expanded, "file_Alice"); } Command::Touch { path, .. } => { @@ -5313,3 +5358,78 @@ mod semicolon_tests { assert_eq!(parts, vec!["", ""]); } } + +#[cfg(test)] +mod typed_word_tests { + use super::*; + use proptest::prelude::*; + + fn typed_words(input: &str) -> Vec { + tokenize(input) + .unwrap() + .into_iter() + .filter_map(|t| match t { + Token::Word(w) => Some(w.typed), + _ => None, + }) + .collect() + } + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + #[test] + fn the_tokenizer_records_the_quoting_of_each_piece() { + assert_eq!( + typed_words("echo '*'x\\?\"a $V\"*"), + vec![Word::literal("echo"), { + let mut w = word(&[("*", Quote::Single), ("x", Quote::None)]); + w.push_char('?', Quote::Backslash); + w.push_str("a ", Quote::Double); + w.push_parameter("V".into(), false, Quote::Double); + w.push_char('*', Quote::None); + w + }] + ); + } + + #[test] + fn arithmetic_expansion_is_one_command_substitution_piece() { + let mut expected = Word::new(); + expected.push_command_sub("(1+(2*3))".into(), Quote::None); + expected.push_char('x', Quote::None); + assert_eq!(typed_words("$((1+(2*3)))x"), vec![expected]); + } + + /// One literal piece: (text, quoting). Characters are limited to those + /// the tokenizer reads literally in each quoting. + fn piece() -> impl Strategy { + prop_oneof![ + "[a-z0-9._/*?-]{1,4}".prop_map(|s| (s, Quote::None)), + "[a-z *?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Single)), + "[a-z '*?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Double)), + "[a-z*?$'\"\\\\ ]".prop_map(|s| (s, Quote::Backslash)), + ] + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// `Display` writes a word as source that tokenizes back to the + /// same word, so the quoting survives a round trip. + #[test] + fn display_then_tokenize_gives_the_same_word(pieces in prop::collection::vec(piece(), 1..6)) { + let mut w = Word::new(); + for (text, quote) in &pieces { + w.push_str(text, *quote); + } + let source = w.to_string(); + prop_assert_eq!(typed_words(&source), vec![w], "source: {}", source); + } + } +} diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index 7387952b..131eee30 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -157,11 +157,8 @@ fn start_command_with_output_redirect( let cmd_string = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + let expanded_args = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; // Build command string: prog arg1 arg2 > fifo let mut cmd = expanded_program; @@ -210,11 +207,8 @@ fn start_command_with_input_redirect( let cmd_string = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + let expanded_args = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; // Build command string let mut cmd = expanded_program; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs new file mode 100644 index 00000000..98fc8fc9 --- /dev/null +++ b/impl/rust-cli/src/word.rs @@ -0,0 +1,667 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Typed shell words and the word-expansion stage for simple-command arguments. +//! +//! The tokenizer used to flatten each word into a `String` in which quoted +//! glob metacharacters were marked with a backslash (`'*'` became `\*`). +//! Nothing removed those backslashes again, so `echo '*.txt'` printed +//! `\*.txt`, and glob expansion could not tell a quoted `*` from an unquoted +//! one without re-parsing that encoding. +//! +//! A [`Word`] keeps the quoting of every piece instead: each [`Segment`] +//! records whether it was written unquoted, in single or double quotes, or +//! after a backslash. [`expand_words`] then runs the POSIX word-expansion +//! steps on that structure, in this order: +//! +//! 1. tilde expansion (an unquoted `~`, `~/`, `~+`, `~-` at the start), +//! parameter expansion, command substitution, arithmetic expansion and +//! process substitution; +//! 2. brace expansion (`{a,b}`) of unquoted literal text; +//! 3. pathname (glob) expansion of the characters that are not quoted; +//! 4. quote removal. +//! +//! # Rules kept from the previous expansion code +//! +//! - A pattern that matches nothing, or is not a valid pattern, stays as the +//! literal word. +//! - A pattern whose expansion reaches a [`GlobLimits`] limit fails the whole +//! expansion with the [`GlobLimitExceeded`] error; the caller must not run +//! the command. +//! +//! # Known differences from POSIX (not changed here) +//! +//! - No field splitting: an unquoted `$X` whose value is `a b` stays one +//! argument. +//! - An empty word (`''`, or an unquoted expansion that is empty) is dropped +//! by the tokenizer or kept as an empty argument, as before. +//! - Brace expansion runs after parameter expansion (bash runs it before); +//! braces that come out of an expansion are not expanded. +//! +//! Only the arguments of external commands, pipeline stages, `echo` and +//! function calls go through this stage so far. Builtin operands, redirection +//! targets, assignments, `case` patterns and `test` arguments still use the +//! older string encoding. + +use std::fmt; +use std::path::Path; + +use anyhow::{Context, Result}; + +use crate::glob::{self, GlobLimitExceeded, GlobLimits}; +use crate::parser::ProcessSubType; +use crate::process_sub::ProcessSubstitution; +use crate::state::ShellState; + +/// How a piece of a word was written in the source. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Quote { + /// Not quoted: subject to tilde, brace and glob expansion. + None, + /// Inside `'...'`. + Single, + /// Inside `"..."`. + Double, + /// A single character after an unquoted `\`. + Backslash, +} + +impl Quote { + fn is_quoted(self) -> bool { + self != Quote::None + } +} + +/// One piece of a [`Word`]. +#[derive(Debug, Clone, PartialEq)] +pub enum Segment { + /// Literal text. + Literal { text: String, quote: Quote }, + /// `$NAME` (`braced: false`) or `${EXPR}` (`braced: true`). + Parameter { + expr: String, + braced: bool, + quote: Quote, + }, + /// `$(CMD)` or `` `CMD` ``; `$((EXPR))` is stored as the command `(EXPR)`. + CommandSub { cmd: String, quote: Quote }, + /// `<(CMD)` or `>(CMD)`. + ProcessSub { kind: ProcessSubType, cmd: String }, +} + +/// A shell word with the quoting of each of its pieces. +#[derive(Debug, Clone, PartialEq, Default)] +pub struct Word { + segments: Vec, +} + +impl Word { + /// An empty word. + pub fn new() -> Self { + Self::default() + } + + /// A word that is a single unquoted literal. + /// + /// # Examples + /// ``` + /// use vsh::word::Word; + /// assert_eq!(Word::literal("a.txt").to_string(), "a.txt"); + /// ``` + pub fn literal(text: &str) -> Self { + let mut word = Self::new(); + for ch in text.chars() { + word.push_char(ch, Quote::None); + } + word + } + + /// The pieces of the word, in order. + pub fn segments(&self) -> &[Segment] { + &self.segments + } + + /// True when the word has no pieces. + pub fn is_empty(&self) -> bool { + self.segments.is_empty() + } + + /// Append one literal character, merging it into the last piece when that + /// piece is a literal with the same quoting. + pub fn push_char(&mut self, ch: char, quote: Quote) { + if quote != Quote::Backslash { + if let Some(Segment::Literal { text, quote: last }) = self.segments.last_mut() { + if *last == quote { + text.push(ch); + return; + } + } + } + self.segments.push(Segment::Literal { + text: ch.to_string(), + quote, + }); + } + + /// Append literal text. + pub fn push_str(&mut self, text: &str, quote: Quote) { + for ch in text.chars() { + self.push_char(ch, quote); + } + } + + /// Append a parameter expansion. + pub fn push_parameter(&mut self, expr: String, braced: bool, quote: Quote) { + self.segments.push(Segment::Parameter { + expr, + braced, + quote, + }); + } + + /// Append a command substitution. + pub fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.segments.push(Segment::CommandSub { cmd, quote }); + } + + /// Append a process substitution. + pub fn push_process_sub(&mut self, kind: ProcessSubType, cmd: String) { + self.segments.push(Segment::ProcessSub { kind, cmd }); + } +} + +impl From<&str> for Word { + fn from(text: &str) -> Self { + Word::literal(text) + } +} + +/// Write one literal piece back as source with its quoting. +fn fmt_literal(f: &mut fmt::Formatter<'_>, text: &str, quote: Quote) -> fmt::Result { + match quote { + Quote::None => f.write_str(text), + Quote::Backslash => text.chars().try_for_each(|ch| write!(f, "\\{ch}")), + // `'` cannot appear inside single quotes. + Quote::Single => write!(f, "'{}'", text.replace('\'', "'\\''")), + Quote::Double => { + let mut escaped = String::with_capacity(text.len() + 2); + for ch in text.chars() { + if matches!(ch, '"' | '$' | '`' | '\\') { + escaped.push('\\'); + } + escaped.push(ch); + } + write!(f, "\"{escaped}\"") + } + } +} + +/// Writes the word back as shell source that parses to the same word. +impl fmt::Display for Word { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for segment in &self.segments { + match segment { + Segment::Literal { text, quote } => fmt_literal(f, text, *quote)?, + Segment::Parameter { + expr, + braced, + quote, + } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + if *braced { + write!(f, "{open}${{{expr}}}{open}")?; + } else { + write!(f, "{open}${expr}{open}")?; + } + } + Segment::CommandSub { cmd, quote } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + write!(f, "{open}$({cmd}){open}")?; + } + Segment::ProcessSub { kind, cmd } => match kind { + ProcessSubType::Input => write!(f, "<({cmd})")?, + ProcessSubType::Output => write!(f, ">({cmd})")?, + }, + } + } + Ok(()) + } +} + +/// Compares the source form ([`fmt::Display`]) with `other`. +impl PartialEq for Word { + fn eq(&self, other: &str) -> bool { + let source = self.to_string(); + source.as_str() == other + } +} + +impl PartialEq<&str> for Word { + fn eq(&self, other: &&str) -> bool { + *self == **other + } +} + +/// Whether process substitutions may be started while expanding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ProcessSubs { + /// Start them; the caller cleans them up after the command finishes. + Allow, + /// Fail on them, as the code paths without FIFO handling always have. + Refuse, +} + +/// The result of expanding a list of words. +pub struct Expansion { + /// The arguments, after quote removal. + pub fields: Vec, + /// Process substitutions started for these arguments. + pub process_subs: Vec, +} + +impl Expansion { + /// Wait for the process substitutions and remove their FIFOs. + pub fn cleanup(self) { + for proc_sub in self.process_subs { + if let Err(e) = proc_sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + } +} + +/// Expand words relative to the current directory under the default limits. +/// +/// # Errors +/// Fails on an expansion error (a failing command substitution, a refused +/// process substitution) or when a glob reaches a limit +/// ([`GlobLimitExceeded`], reachable with `anyhow::Error::downcast_ref`). +pub fn expand_words( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, +) -> Result { + let cwd = std::env::current_dir().context("Failed to get current working directory")?; + expand_words_in(words, state, process_subs, &cwd, &GlobLimits::DEFAULT) +} + +/// [`expand_words`] with an explicit directory for relative patterns and +/// explicit glob limits. +/// +/// # Errors +/// As for [`expand_words`]. +pub fn expand_words_in( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, + cwd: &Path, + limits: &GlobLimits, +) -> Result { + let mut expansion = Expansion { + fields: Vec::new(), + process_subs: Vec::new(), + }; + for word in words { + // On error the substitutions started so far are dropped, and Drop + // stops them and removes their FIFOs. + let chars = expand_segments(word, state, process_subs, &mut expansion.process_subs)?; + glob_fields(&chars, cwd, limits, &mut expansion.fields)?; + } + Ok(expansion) +} + +/// What later steps may do with a character produced by step 1. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Role { + /// Quoted text or a tilde result: taken literally. + Literal, + /// Unquoted source text: brace and glob expansion apply. + Source, + /// Result of an unquoted expansion: glob expansion applies, braces do not. + Expanded, +} + +/// Step 1: tilde, parameter, command, arithmetic and process substitution. +fn expand_segments( + word: &Word, + state: &mut ShellState, + process_subs: ProcessSubs, + started: &mut Vec, +) -> Result> { + let mut out: Vec<(char, Role)> = Vec::new(); + let push = |out: &mut Vec<(char, Role)>, text: &str, role: Role| { + out.extend(text.chars().map(|ch| (ch, role))); + }; + + for (index, segment) in word.segments.iter().enumerate() { + match segment { + Segment::Literal { text, quote } => { + if !quote.is_quoted() { + if index == 0 && text.starts_with('~') { + let (prefix, rest) = text.split_at(text.find('/').unwrap_or(text.len())); + let expanded = crate::parser::expand_tilde(prefix); + if expanded != prefix { + push(&mut out, &expanded, Role::Literal); + push(&mut out, rest, Role::Source); + continue; + } + } + push(&mut out, text, Role::Source); + } else { + push(&mut out, text, Role::Literal); + } + } + Segment::Parameter { + expr, + braced, + quote, + } => { + let source = if *braced { + format!("${{{expr}}}") + } else { + format!("${expr}") + }; + let value = crate::parser::expand_variables(&source, state); + push(&mut out, &value, expansion_role(*quote)); + } + Segment::CommandSub { cmd, quote } => { + let value = crate::parser::expand_with_command_sub(&format!("$({cmd})"), state)?; + push(&mut out, &value, expansion_role(*quote)); + } + Segment::ProcessSub { kind, cmd } => { + if process_subs == ProcessSubs::Refuse { + anyhow::bail!( + "Process substitution not yet implemented: execution requires FIFO support ({})", + Word { + segments: vec![segment.clone()] + } + ); + } + let mut proc_sub = ProcessSubstitution::create(*kind, cmd.clone(), state)?; + proc_sub.start(state)?; + push( + &mut out, + &proc_sub.fifo_path.to_string_lossy(), + Role::Literal, + ); + started.push(proc_sub); + } + } + } + Ok(out) +} + +fn expansion_role(quote: Quote) -> Role { + if quote.is_quoted() { + Role::Literal + } else { + Role::Expanded + } +} + +/// Encode step-1 output for [`glob::expand_braces`] and +/// [`glob::contains_glob_pattern`], which both read `\x` as a literal `x`. +fn encode(chars: &[(char, Role)]) -> String { + let mut out = String::with_capacity(chars.len() * 2); + for &(ch, role) in chars { + let literal = match role { + Role::Literal => true, + Role::Source => ch == '\\', + Role::Expanded => matches!(ch, '\\' | '{' | '}' | ','), + }; + if literal { + out.push('\\'); + } + out.push(ch); + } + out +} + +/// Steps 2 to 4: brace expansion, glob expansion and quote removal. +fn glob_fields( + chars: &[(char, Role)], + cwd: &Path, + limits: &GlobLimits, + fields: &mut Vec, +) -> Result<()> { + let encoded = encode(chars); + for candidate in glob::expand_braces(&encoded) { + match glob_pattern(&candidate) { + None => fields.push(remove_quotes(&candidate)), + Some(pattern) => match glob::expand_glob_with_limits(&pattern, cwd, limits) { + Ok(found) if !found.paths.is_empty() => fields.extend( + found + .paths + .iter() + .map(|path| path.to_string_lossy().into_owned()), + ), + // A limit is not "no match": substituting the literal word + // would run the command on the wrong arguments. + Err(e) if e.is::() => return Err(e), + // No match, or not a valid pattern: keep the word (POSIX). + _ => fields.push(remove_quotes(&candidate)), + }, + } + } + Ok(()) +} + +/// The pattern for the `glob` crate, or `None` when no `*`, `?` or `[` is +/// left unquoted. Quoted metacharacters are written as one-character classes +/// (`[*]`), the only escape that crate has. +fn glob_pattern(encoded: &str) -> Option { + let mut pattern = String::with_capacity(encoded.len()); + let mut active = false; + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + if let Some(next) = chars.next() { + pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + } + } else { + active |= matches!(ch, '*' | '?' | '['); + pattern.push(ch); + } + } + active.then_some(pattern) +} + +/// Quote removal: `\x` becomes `x`. +fn remove_quotes(encoded: &str) -> String { + let mut out = String::with_capacity(encoded.len()); + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + if let Some(next) = chars.next() { + out.push(next); + } + } else { + out.push(ch); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + fn fixture(names: &[&str]) -> tempfile::TempDir { + let dir = tempfile::TempDir::new().unwrap(); + for name in names { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir + } + + fn expand_in(dir: &Path, words: &[Word]) -> Vec { + let mut state = ShellState::new(dir.to_str().unwrap()).unwrap(); + expand_words_in( + words, + &mut state, + ProcessSubs::Refuse, + dir, + &GlobLimits::DEFAULT, + ) + .unwrap() + .fields + } + + #[test] + fn unquoted_patterns_expand_and_quoted_ones_do_not() { + let dir = fixture(&["a.txt", "b.txt", "c.log"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + word(&[("*.txt", Quote::Single)]), + word(&[("*.txt", Quote::Double)]), + word(&[("*", Quote::Backslash), (".txt", Quote::None)]), + ], + ); + assert_eq!(out, ["a.txt", "b.txt", "*.txt", "*.txt", "*.txt"]); + } + + #[test] + fn a_partly_quoted_pattern_matches_its_quoted_part_literally() { + let dir = fixture(&["*x.txt", "ax.txt", "a.txt"]); + // `'*'x*` : a literal star followed by `x` and a wildcard. + let out = expand_in( + dir.path(), + &[word(&[("*", Quote::Single), ("x*", Quote::None)])], + ); + assert_eq!(out, ["*x.txt"]); + } + + #[test] + fn no_match_and_invalid_patterns_stay_literal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + Word::literal("*.none"), + Word::literal("[z-"), + ], + ); + assert_eq!(out, ["a.txt", "*.none", "[z-"]); + } + + #[test] + fn braces_expand_only_when_unquoted() { + let dir = fixture(&[]); + let out = expand_in( + dir.path(), + &[Word::literal("a{1,2}"), word(&[("a{1,2}", Quote::Single)])], + ); + assert_eq!(out, ["a1", "a2", "a{1,2}"]); + } + + #[test] + fn literal_backslashes_survive_quote_removal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + word(&[("a\\*", Quote::Single)]), + word(&[("\\", Quote::Backslash)]), + ], + ); + assert_eq!(out, ["a\\*", "\\"]); + } + + #[test] + fn a_quoted_backslash_is_a_literal_character_in_a_pattern() { + let dir = fixture(&["a\\b.txt", "ab.txt"]); + // `'a\'*` : names that start with `a\`. + let out = expand_in( + dir.path(), + &[word(&[("a\\", Quote::Single), ("*", Quote::None)])], + ); + assert_eq!(out, ["a\\b.txt"]); + } + + #[test] + fn unquoted_parameter_values_are_globbed_and_quoted_ones_are_not() { + let dir = fixture(&["a.txt", "b.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("P", "*.txt"); + let mut unquoted = Word::new(); + unquoted.push_parameter("P".into(), false, Quote::None); + let mut quoted = Word::new(); + quoted.push_parameter("P".into(), true, Quote::Double); + let out = expand_words_in( + &[unquoted, quoted], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap() + .fields; + assert_eq!(out, ["a.txt", "b.txt", "*.txt"]); + } + + #[test] + fn a_glob_limit_fails_the_expansion_instead_of_substituting_the_literal() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let words = [Word::literal("-l"), Word::literal("*.txt")]; + let err = expand_words_in(&words, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the limit must fail the expansion"); + let stopped = err + .downcast_ref::() + .expect("the limit error must reach the caller"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert!(err.to_string().contains("over the limit of 4"), "{err}"); + } + + #[test] + fn refused_process_substitution_is_an_error() { + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let mut w = Word::new(); + w.push_process_sub(ProcessSubType::Input, "true".into()); + let err = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .err() + .expect("refused"); + assert!(err.to_string().contains("<(true)"), "{err}"); + } + + #[test] + fn display_round_trips_quoting() { + let w = word(&[ + ("a b", Quote::Single), + ("*", Quote::Backslash), + ("x\"$", Quote::Double), + ("y", Quote::None), + ]); + assert_eq!(w.to_string(), "'a b'\\*\"x\\\"\\$\"y"); + assert_eq!(Word::literal("plain"), "plain"); + } + + #[test] + fn glob_pattern_escapes_quoted_metacharacters() { + assert_eq!(glob_pattern("\\*a*"), Some("[*]a*".to_string())); + assert_eq!(glob_pattern("\\*a"), None); + assert_eq!(glob_pattern("\\[x?"), Some("[[]x?".to_string())); + } +} diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs new file mode 100644 index 00000000..bcfa282b --- /dev/null +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! End-to-end tests for argument glob expansion, run through the `vsh` binary. +//! +//! Before the word-expansion stage (`vsh::word`), globs were expanded only +//! for external commands that also had a redirection: `wc -c *.txt` passed +//! `*.txt` to `wc`, pipelines and `echo` never expanded, and quoted patterns +//! reached the program with a stray backslash (`echo '*.txt'` printed +//! `\*.txt`). Each test runs `vsh -c` in a directory holding `a.txt`, +//! `b.txt` and `c.log`. + +use std::path::Path; +use std::process::{Command, Output}; + +use tempfile::TempDir; + +fn fixture() -> TempDir { + let dir = TempDir::new().unwrap(); + for name in ["a.txt", "b.txt", "c.log"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir +} + +fn vsh(dir: &Path, script: &str) -> Output { + Command::new(env!("CARGO_BIN_EXE_vsh")) + .arg("-c") + .arg(script) + .current_dir(dir) + .output() + .expect("vsh runs") +} + +fn stdout(dir: &Path, script: &str) -> String { + let out = vsh(dir, script); + assert!( + out.status.success(), + "`{script}` failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8(out.stdout).unwrap() +} + +#[test] +fn external_command_without_redirect_gets_the_matches() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "wc -c *.txt"), + "0 a.txt\n0 b.txt\n0 total\n" + ); +} + +#[test] +fn external_command_with_redirect_gets_the_matches() { + let dir = fixture(); + stdout(dir.path(), "wc -c *.txt > out.wc"); + assert_eq!( + std::fs::read_to_string(dir.path().join("out.wc")).unwrap(), + "0 a.txt\n0 b.txt\n0 total\n" + ); +} + +#[test] +fn every_pipeline_stage_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "ls *.txt | wc -l").trim(), "2"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.log | cat"), "c.log\n"); +} + +#[test] +fn builtin_echo_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.txt"), "a.txt b.txt\n"); + assert_eq!(stdout(dir.path(), "echo -n *.log"), "c.log"); +} + +#[test] +fn quoted_patterns_stay_literal_without_a_backslash() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo '*.txt'"), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \"*.txt\""), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \\*.txt"), "*.txt\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' \"*.txt\" \\*.txt"), + "*.txt\n*.txt\n*.txt\n" + ); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' | cat"), + "*.txt\n" + ); +} + +#[test] +fn a_pattern_that_matches_nothing_is_passed_literally() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.none"), "*.none\n"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.none"), "*.none\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' *.none > out.none; cat out.none"), + "*.none\n" + ); +} + +#[test] +fn a_partly_quoted_word_globs_only_its_unquoted_part() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + assert_eq!(stdout(dir.path(), "echo '*'x.*"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "echo *x.*"), "*x.txt ax.txt\n"); +} + +#[test] +fn braces_expand_unquoted_and_stay_literal_quoted() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo a{1,2} '{a,b}'"), "a1 a2 {a,b}\n"); +} + +#[test] +fn function_arguments_get_the_matches() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "f() { echo \"$#:$1:$2\"; }; f *.txt"), + "2:a.txt:b.txt\n" + ); +} From b600ca33a7963d6011cd30c606086562a5e10ef1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 15:28:41 +0000 Subject: [PATCH 05/21] fix(glob): one glob budget per command; address CodeRabbit review on #242 Verified findings from the CodeRabbit review of PR #242: - word.rs:430 (major): every brace alternative and every word started with a fresh glob budget. With up to 10,000 brace alternatives, one word could do 10,000 times the intended work, which defeats the bound #191 adds. glob::expand_glob_continuing starts the meters from the usage already spent; word expansion threads that usage through every word and alternative of a command, so the error still reports the configured limit. New test: five one-match patterns, as brace alternatives and as separate words, now exceed a four-match limit. - glob.rs:415: a match refused by the byte limit still raised the match peak by one. Meter::check tests a charge without recording it; both limits are checked before either is charged. The byte-limit test now asserts both peaks. - confirmation.rs:221: a symbolic link given as the root was followed. The root is now read with symlink_metadata and counted as one entry. - fuzz_glob_expansion.rs:69 (major): a fixture that could not be built made the target skip every input, a green run that tested nothing. It now stops with a clear panic message. - glob.rs:612: the differential test oracle now escapes the temporary directory path before using it as a glob prefix. - EXPLAINME.adoc: the bash compatibility claim names its one documented difference. cargo fmt --check, cargo clippy --all-targets --all-features -D warnings: clean. cargo test: 889 passed, 0 failed, 14 ignored. Fuzz target rebuilt under ASan; 23,410 runs in 60 s, no crash. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- EXPLAINME.adoc | 2 +- .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 22 ++++--- impl/rust-cli/src/confirmation.rs | 6 +- impl/rust-cli/src/glob.rs | 54 +++++++++++++--- impl/rust-cli/src/grade.rs | 41 ++++++++++-- impl/rust-cli/src/word.rs | 62 ++++++++++++++++--- impl/rust-cli/tests/glob_bounds_tests.rs | 2 + 7 files changed, 153 insertions(+), 36 deletions(-) diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 11279c02..df0c1b3c 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -212,7 +212,7 @@ All share: *Multiple proof systems for cross-validation, honest status reporting | `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine) -| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash globstar rules), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). +| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash 5.2 globstar rules, one documented difference for symlinked directories), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). | `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved. | `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT). diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index 6f778d58..ade8a883 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -34,8 +34,7 @@ const LIMITS: GlobLimits = GlobLimits { max_match_bytes: 256 * 1024, }; -/// Build the fixture tree. Errors are returned, not unwrapped: if the temp -/// directory cannot be set up, the target skips its inputs instead of panicking. +/// Build the fixture tree. fn build_fixture() -> std::io::Result { let t = TempDir::new()?; let root = t.path(); @@ -63,12 +62,17 @@ fn build_fixture() -> std::io::Result { Ok(t) } -fn fixture() -> Option<&'static Path> { - static FIXTURE: OnceLock> = OnceLock::new(); +/// The fixture, built once. A fixture that cannot be built stops the run +/// with a clear message: skipping every input would report a green run that +/// tested nothing. +fn fixture() -> &'static Path { + static FIXTURE: OnceLock = OnceLock::new(); FIXTURE - .get_or_init(|| build_fixture().ok()) - .as_ref() - .map(TempDir::path) + .get_or_init(|| match build_fixture() { + Ok(fixture) => fixture, + Err(e) => panic!("fuzz_glob_expansion: cannot build the fixture tree: {e}"), + }) + .path() } fuzz_target!(|data: &[u8]| { @@ -79,9 +83,7 @@ fuzz_target!(|data: &[u8]| { return; } - let Some(base) = fixture() else { - return; - }; + let base = fixture(); let pattern = if input.starts_with('/') { format!("{}{}", base.display(), input) } else { diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 3d1d5c07..5433c5db 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -218,7 +218,8 @@ fn count_files_in_tree(path: &str) -> Result { let mut count = 0; fn visit_dirs(path: &std::path::Path, count: &mut usize) -> Result<()> { - if path.is_dir() { + // symlink_metadata: a link given as the root is one entry, not its target. + if std::fs::symlink_metadata(path)?.is_dir() { for entry in std::fs::read_dir(path)? { let entry = entry?; if entry.file_type()?.is_dir() { @@ -363,5 +364,8 @@ mod tests { let root = t.path().to_str().unwrap(); // d/f, d/up and self: the two links are counted, not followed. assert_eq!(count_files_in_tree(root).unwrap(), 3); + // A link given as the root is one entry too. + let link = t.path().join("self"); + assert_eq!(count_files_in_tree(link.to_str().unwrap()).unwrap(), 1); } } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index ae5919f8..a0b63cf7 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -278,10 +278,28 @@ pub fn expand_glob_with_limits( pattern: &str, base_dir: &Path, limits: &GlobLimits, +) -> Result { + expand_glob_continuing(pattern, base_dir, limits, GlobUsage::default()) +} + +/// Expand a glob pattern under limits that it shares with earlier expansions. +/// +/// The meters start from `spent`, the usage an earlier expansion reported, so +/// several patterns (the brace alternatives of one word, or the words of one +/// command) together stay within one set of [`GlobLimits`]. The returned usage +/// includes `spent`, and the matches of earlier expansions count as still held. +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_continuing( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, + spent: GlobUsage, ) -> Result { let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; - let mut walker = Walker::new(pattern, base_dir, &compiled, limits); + let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); walker.run().map_err(anyhow::Error::new)?; Ok(Expansion { usage: walker.usage(), @@ -351,14 +369,24 @@ struct Walker<'a> { } impl<'a> Walker<'a> { - fn new(pattern: &'a str, base: &'a Path, compiled: &'a Compiled, limits: &GlobLimits) -> Self { + fn new( + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + limits: &GlobLimits, + spent: GlobUsage, + ) -> Self { Walker { pattern, base, compiled, - reads: Meter::new("file-system reads", limits.max_reads), - matches: Meter::new("matches held", limits.max_matches), - match_bytes: Meter::new("bytes of matches held", limits.max_match_bytes), + reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads), + matches: Meter::starting_at("matches held", limits.max_matches, spent.matches), + match_bytes: Meter::starting_at( + "bytes of matches held", + limits.max_match_bytes, + spent.match_bytes, + ), found: BTreeSet::new(), } } @@ -410,11 +438,16 @@ impl<'a> Walker<'a> { return Ok(()); } let bytes = path.as_os_str().len() as u64 + 1; + // Check both limits before recording either, so a refused match leaves + // no trace in the reported usage. + self.matches.check(1).map_err(|e| self.stopped(e, at))?; + self.match_bytes + .check(bytes) + .map_err(|e| self.stopped(e, at))?; self.matches.charge(1).map_err(|e| self.stopped(e, at))?; - if let Err(e) = self.match_bytes.charge(bytes) { - self.matches.release(1); - return Err(self.stopped(e, at)); - } + self.match_bytes + .charge(bytes) + .map_err(|e| self.stopped(e, at))?; self.found.insert(path); Ok(()) } @@ -609,7 +642,8 @@ mod walk_tests { case_sensitive: true, require_literal_separator: false, }; - glob::glob_with(&base.join(pattern).to_string_lossy(), options) + let base_pattern = glob::Pattern::escape(&base.to_string_lossy()); + glob::glob_with(&format!("{base_pattern}/{pattern}"), options) .unwrap() .filter_map(|p| p.ok()) .map(|p| p.strip_prefix(base).unwrap().to_path_buf()) diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs index 8f5143ff..ad56a78b 100644 --- a/impl/rust-cli/src/grade.rs +++ b/impl/rust-cli/src/grade.rs @@ -243,12 +243,21 @@ impl Meter { } } - /// Record the use of `amount` more units, or refuse if that passes the limit. + /// A meter for `axis` that starts from work already measured as `spent`, + /// so that several pieces of work can share one limit. + pub const fn starting_at(axis: &'static str, limit: u64, spent: G) -> Self { + Meter { + axis, + limit, + grade: spent, + } + } + + /// Whether [`Meter::charge`] would accept `amount`, without recording it. /// /// # Errors - /// Returns [`Exceeded`] and records nothing when the charge would take the - /// level over the limit. - pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + /// Returns the [`Exceeded`] that the charge would return. + pub fn check(&self, amount: u64) -> Result<(), Exceeded> { let next = self.grade.then(G::of(amount)); if next.level() > self.limit { return Err(Exceeded { @@ -257,7 +266,17 @@ impl Meter { level: next.level(), }); } - self.grade = next; + Ok(()) + } + + /// Record the use of `amount` more units, or refuse if that passes the limit. + /// + /// # Errors + /// Returns [`Exceeded`] and records nothing when the charge would take the + /// level over the limit. + pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + self.check(amount)?; + self.grade = self.grade.then(G::of(amount)); Ok(()) } @@ -370,6 +389,18 @@ mod tests { assert_eq!(meter.grade().peak(), 10); } + #[test] + fn a_meter_can_start_from_work_already_spent() { + let mut meter: Meter = Meter::starting_at("steps", 10, Cost::of(8)); + assert!(meter.check(2).is_ok()); + assert_eq!(meter.charge(3).unwrap_err().level, 11); + assert_eq!( + meter.level(), + 8, + "check and a refused charge record nothing" + ); + } + #[test] fn a_meter_with_a_zero_limit_refuses_any_use() { let mut meter: Meter = Meter::new("items held", 0); diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 98fc8fc9..6809c23d 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -47,7 +47,7 @@ use std::path::Path; use anyhow::{Context, Result}; -use crate::glob::{self, GlobLimitExceeded, GlobLimits}; +use crate::glob::{self, GlobLimitExceeded, GlobLimits, GlobUsage}; use crate::parser::ProcessSubType; use crate::process_sub::ProcessSubstitution; use crate::state::ShellState; @@ -300,11 +300,14 @@ pub fn expand_words_in( fields: Vec::new(), process_subs: Vec::new(), }; + // One glob budget for the whole command: every word and every brace + // alternative draws on the same limits. + let mut spent = GlobUsage::default(); for word in words { // On error the substitutions started so far are dropped, and Drop // stops them and removes their FIFOs. let chars = expand_segments(word, state, process_subs, &mut expansion.process_subs)?; - glob_fields(&chars, cwd, limits, &mut expansion.fields)?; + glob_fields(&chars, cwd, limits, &mut spent, &mut expansion.fields)?; } Ok(expansion) } @@ -421,19 +424,28 @@ fn glob_fields( chars: &[(char, Role)], cwd: &Path, limits: &GlobLimits, + spent: &mut GlobUsage, fields: &mut Vec, ) -> Result<()> { let encoded = encode(chars); for candidate in glob::expand_braces(&encoded) { match glob_pattern(&candidate) { None => fields.push(remove_quotes(&candidate)), - Some(pattern) => match glob::expand_glob_with_limits(&pattern, cwd, limits) { - Ok(found) if !found.paths.is_empty() => fields.extend( - found - .paths - .iter() - .map(|path| path.to_string_lossy().into_owned()), - ), + Some(pattern) => match glob::expand_glob_continuing(&pattern, cwd, limits, *spent) { + Ok(found) if !found.paths.is_empty() => { + *spent = found.usage; + fields.extend( + found + .paths + .iter() + .map(|path| path.to_string_lossy().into_owned()), + ); + } + // No match still spent reads. + Ok(found) => { + *spent = found.usage; + fields.push(remove_quotes(&candidate)); + } // A limit is not "no match": substituting the literal word // would run the command on the wrong arguments. Err(e) if e.is::() => return Err(e), @@ -628,6 +640,38 @@ mod tests { assert!(err.to_string().contains("over the limit of 4"), "{err}"); } + /// The brace alternatives of one word, and the words of one command, share + /// one glob budget: five patterns of one match each pass a limit of four. + #[test] + fn brace_alternatives_and_words_share_one_glob_budget() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let braces = [Word::literal("{f0,f1,f2,f3,f4}*")]; + let words: Vec = (0..5).map(|i| Word::literal(&format!("f{i}*"))).collect(); + for case in [&braces[..], &words[..]] { + let err = expand_words_in(case, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the shared budget must be exceeded"); + let stopped = err.downcast_ref::().unwrap(); + assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (4, 5)); + } + // Within the budget the same words expand normally. + let out = expand_words_in( + &words[..4], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &limits, + ) + .unwrap() + .fields; + assert_eq!(out, ["f0.txt", "f1.txt", "f2.txt", "f3.txt"]); + } + #[test] fn refused_process_substitution_is_an_error() { let dir = fixture(&[]); diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs index bd613cef..ead12100 100644 --- a/impl/rust-cli/tests/glob_bounds_tests.rs +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -181,6 +181,8 @@ fn the_byte_limit_counts_path_length_plus_one() { 2, "the refused match is not counted" ); + assert_eq!(s.usage.matches.peak(), 2, "nor does it raise the peak"); + assert_eq!(s.usage.match_bytes.peak(), 10); let limits = GlobLimits { max_match_bytes: 15, From 093e86aeeadda046ba0a3ec94bb0fc7312f989bc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 15:31:04 +0000 Subject: [PATCH 06/21] fix(fuzz): exit instead of panicking when the glob fixture cannot be built Hypatia's code_safety scan flagged the panic! added in b600ca3 (panic_macro, CWE-754). CodeRabbit had asked that a fixture failure fail the run rather than skip every input. Printing the error and calling std::process::exit(1) satisfies both: libFuzzer reports an exiting target as an error. Checked with TMPDIR pointing at a missing directory: the run prints "cannot build the fixture tree" and "libFuzzer: fuzz target exited" and exits non-zero. A normal 30 s run (10,012 inputs) finds nothing. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index ade8a883..c467597e 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -70,7 +70,12 @@ fn fixture() -> &'static Path { FIXTURE .get_or_init(|| match build_fixture() { Ok(fixture) => fixture, - Err(e) => panic!("fuzz_glob_expansion: cannot build the fixture tree: {e}"), + Err(e) => { + // Exit rather than panic: libFuzzer reports an exiting target + // as an error ("fuzz target exited"), so the run still fails. + eprintln!("fuzz_glob_expansion: cannot build the fixture tree: {e}"); + std::process::exit(1); + } }) .path() } From d9669b2fad3ba8c9558ef30db39cb7f7985e04db Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 23:31:09 +0000 Subject: [PATCH 07/21] fix(word): seven faults from the review of #242, with regression tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - parser: the statement splitters walk bytes and looped for ever on an unquoted non-ASCII letter (`echo é` hung); only an ASCII letter can start a keyword. - glob: brace expansion used character counts as byte offsets and panicked on `'é'{a,b}`. - parser: ${...} parsing sliced at character counts; it now uses byte offsets (${é}, ${é:-x}). - parser: ${X:2:N} with a huge N overflowed; the end is now saturating. - parser: a quoted or escaped parenthesis no longer ends $(...). - word: a quoted hyphen inside [...] is written last in the class, where it cannot form a range; quoted *, ? and [ stay bare inside a class. - word: a tilde whose prefix has a quoted piece is not expanded. - word, parser: an unbraced parameter followed by a literal is written back with braces, so an AND-OR list reads $A''b as ${A}b, not $Ab. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- CHANGELOG.adoc | 17 +++ impl/rust-cli/src/glob.rs | 3 +- impl/rust-cli/src/parser.rs | 97 ++++++++------ impl/rust-cli/src/word.rs | 123 ++++++++++++++++-- .../tests/word_expansion_e2e_tests.rs | 76 +++++++++++ 5 files changed, 264 insertions(+), 52 deletions(-) diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index fb3d88be..ff987c16 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -27,6 +27,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An expansion error (such as a glob limit from #191) fails the command with exit status 1 and does not run it. With a redirection the status was 127. +=== Fixed — 2026-11-18 (review of #242) + +- An unquoted non-ASCII letter no longer hangs the shell: `echo é` did not + return, because the statement splitter did not advance past it. +- `'é'{a,b}` and `${é:-x}` no longer panic or misread the word: brace and + parameter expansion used character counts as byte offsets. +- `${X:2:N}` with a very large `N` gives the rest of the value; it panicked + on overflow. +- A quoted or escaped parenthesis no longer ends a command substitution: + `$(echo ')')`. +- A quoted hyphen in a bracket expression is a member, not a range: + `[a'-'z]*.txt` no longer matches `m.txt`. +- `~'local'/x` keeps its tilde: a quoted piece in the tilde prefix stops + the expansion. +- `$A''suffix && true` expands `A`: in an AND-OR list the word was read + back as `$Asuffix`. + === Changed — 2026-11-18 (typed words) - New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index a0b63cf7..b43dd92c 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -728,7 +728,8 @@ fn expand_braces_limited(pattern: &str, remaining: usize) -> Vec { } // Find first unescaped { - let chars = pattern.chars().enumerate().peekable(); + // Byte offsets, not character counts: the slices below index bytes. + let chars = pattern.char_indices(); let mut brace_start = None; let mut brace_depth = 0; let mut escaped = false; diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index 1b51f2d0..3082b834 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -960,9 +960,31 @@ fn parse_variable( fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> Result { let mut cmd = String::new(); let mut depth = 1; // Parenthesis depth: nested $(), $((expr)) and (...) + // A parenthesis that is quoted or escaped is text, not nesting. + let mut single = false; + let mut double = false; + let mut escaped = false; for ch in chars.by_ref() { + if escaped { + escaped = false; + cmd.push(ch); + continue; + } match ch { + '\\' if !single => { + escaped = true; + cmd.push(ch); + } + '\'' if !double => { + single = !single; + cmd.push(ch); + } + '"' if !single => { + double = !double; + cmd.push(ch); + } + _ if single || double => cmd.push(ch), '(' => { // Every `(` nests, so the `)` that closes `$((1+2))` or // `$(echo $(pwd))` belongs to the substitution. @@ -1556,7 +1578,7 @@ fn split_on_top_level(input: &str, split_on_newline: bool) -> Vec<&str> { } _ if !in_single_quote && !in_double_quote && paren_depth == 0 // Check for keyword at word boundary - && (ch.is_alphabetic() || ch == '_') => + && (ch.is_ascii_alphabetic() || ch == '_') => { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { @@ -2139,12 +2161,12 @@ fn parse_parameter_expansion(content: &str) -> Result = content.chars().collect(); - #[allow(unused_assignments)] - let mut pos = 0; + // Offsets are bytes. Every operator is one ASCII byte, so a `+ 1` step + // stays on a character boundary whatever the name contains. + let bytes = content.as_bytes(); // Check for length operator: ${#VAR} - if chars[0] == '#' { + if bytes[0] == b'#' { let var_name = content[1..].to_string(); return Ok(ParameterExpansion { var_name, @@ -2153,28 +2175,27 @@ fn parse_parameter_expansion(content: &str) -> Result= chars.len() { + if pos >= content.len() { // ${VAR:} with nothing after - try substring with offset 0 return Ok(ParameterExpansion { var_name, @@ -2185,8 +2206,10 @@ fn parse_parameter_expansion(content: &str) -> Result) -> String { match length { Some(n) => { - let end = (start + n).min(chars.len()); + let end = start.saturating_add(n).min(chars.len()); chars[start..end].iter().collect() } None => chars[start..].iter().collect(), @@ -2574,31 +2597,13 @@ fn quoted_word_to_string(word: &QuotedWord) -> String { result.push_str(s); } } - WordPart::Variable(name) => { - // This shouldn't happen in single quotes (tokenizer prevents it) - // but handle it anyway - if word.quote_type != QuoteType::Single { - result.push('$'); - result.push_str(name); - } else { - // In single quotes, escape the $ - result.push('\\'); - result.push('$'); - result.push_str(name); - } - } - WordPart::BracedVariable(name) => { - // This shouldn't happen in single quotes - if word.quote_type != QuoteType::Single { - result.push_str("${"); - result.push_str(name); - result.push('}'); - } else { - // In single quotes, escape the $ - result.push_str("\\${"); - result.push_str(name); - result.push('}'); - } + // The tokenizer makes a variable part only outside single quotes, + // so it is always an expansion, whatever quotes follow in the word. + // Braces keep a following literal out of the name (`$A''b`). + WordPart::Variable(name) | WordPart::BracedVariable(name) => { + result.push_str("${"); + result.push_str(name); + result.push('}'); } WordPart::CommandSub(cmd) => { // Command substitution - keep as-is for now @@ -2856,7 +2861,10 @@ fn split_control_keywords<'a>(input: &'a str, keywords: &[&str]) -> Vec<(&'a str in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; @@ -3260,7 +3268,10 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 6809c23d..08f7573f 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -198,7 +198,7 @@ fn fmt_literal(f: &mut fmt::Formatter<'_>, text: &str, quote: Quote) -> fmt::Res /// Writes the word back as shell source that parses to the same word. impl fmt::Display for Word { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - for segment in &self.segments { + for (index, segment) in self.segments.iter().enumerate() { match segment { Segment::Literal { text, quote } => fmt_literal(f, text, *quote)?, Segment::Parameter { @@ -207,7 +207,13 @@ impl fmt::Display for Word { quote, } => { let open = if quote.is_quoted() { "\"" } else { "" }; - if *braced { + // `$A''b` must not come back as `$Ab`: brace the name when + // the text after it could continue it. + let runs_on = self.segments.get(index + 1).is_some_and(|next| { + matches!(next, Segment::Literal { text, quote: Quote::None } + if text.starts_with(|c: char| c.is_alphanumeric() || c == '_')) + }) && expr.starts_with(|c: char| c.is_alphabetic() || c == '_'); + if *braced || runs_on { write!(f, "{open}${{{expr}}}{open}")?; } else { write!(f, "{open}${expr}{open}")?; @@ -339,7 +345,14 @@ fn expand_segments( match segment { Segment::Literal { text, quote } => { if !quote.is_quoted() { - if index == 0 && text.starts_with('~') { + // The tilde prefix runs to the first unquoted `/`. It ends in this + // segment only if the segment has a `/` or is the whole word; + // otherwise a quoted piece or an expansion is part of the + // prefix, and the tilde stays literal (`~'local'/x`). + if index == 0 + && text.starts_with('~') + && (text.contains('/') || word.segments.len() == 1) + { let (prefix, rest) = text.split_at(text.find('/').unwrap_or(text.len())); let expanded = crate::parser::expand_tilde(prefix); if expanded != prefix { @@ -463,16 +476,46 @@ fn glob_fields( fn glob_pattern(encoded: &str) -> Option { let mut pattern = String::with_capacity(encoded.len()); let mut active = false; + // Inside an unquoted `[...]`: how many members so far, and whether a + // quoted `-` waits to be written last, where it cannot form a range. + let mut class: Option = None; + let mut hyphen = false; let mut chars = encoded.chars(); while let Some(ch) = chars.next() { if ch == '\\' { - if let Some(next) = chars.next() { - pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + let Some(next) = chars.next() else { continue }; + match (class.as_mut(), next) { + (Some(_), '-') => hyphen = true, + // `*`, `?` and `[` are already literal inside a class. + (Some(members), '*' | '?' | '[') => { + *members += 1; + pattern.push(next); + } + (members, _) => { + if let Some(members) = members { + *members += 1; + } + pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + } } - } else { - active |= matches!(ch, '*' | '?' | '['); - pattern.push(ch); + continue; } + active |= matches!(ch, '*' | '?' | '['); + match class.as_mut() { + None if ch == '[' => class = Some(0), + None => {} + // `]` first in a class (after `!` or `^`) is a member, not the end. + Some(members) if ch == ']' && *members > 0 => { + if std::mem::take(&mut hyphen) { + pattern.push('-'); + } + class = None; + } + Some(members) if *members == 0 && matches!(ch, '!' | '^') && pattern.ends_with('[') => { + } + Some(members) => *members += 1, + } + pattern.push(ch); } active.then_some(pattern) } @@ -702,6 +745,70 @@ mod tests { assert_eq!(Word::literal("plain"), "plain"); } + #[test] + fn display_keeps_a_literal_out_of_an_unbraced_name() { + let mut w = Word::new(); + w.push_parameter("A".to_string(), false, Quote::None); + w.push_str("suffix", Quote::None); + assert_eq!(w.to_string(), "${A}suffix"); + + std::env::remove_var("Asuffix"); + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("A", "x"); + let out = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap(); + assert_eq!(out.fields, ["xsuffix"]); + } + + #[test] + fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + // [a'-'z]*.txt + assert_eq!(glob_pattern("[a\\-z]*.txt"), Some("[az-]*.txt".to_string())); + let dir = fixture(&["-.txt", "a.txt", "m.txt", "z.txt"]); + let w = word(&[ + ("[a", Quote::None), + ("-", Quote::Single), + ("z]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt", "a.txt", "z.txt"]); + } + + #[test] + fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(&[]); + let quoted = word(&[ + ("~", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[quoted]), ["~local/x"]); + // An unquoted slash ends the prefix, so this one still expands. + let slash = word(&[ + ("~/", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + let got = expand_in(dir.path(), &[slash]); + assert!( + got[0].ends_with("/local/x") && !got[0].starts_with('~'), + "{got:?}" + ); + } + + #[test] + fn a_non_ascii_character_before_a_brace_does_not_panic() { + let dir = fixture(&[]); + let w = word(&[("é", Quote::Single), ("{a,b}", Quote::None)]); + assert_eq!(expand_in(dir.path(), &[w]), ["éa", "éb"]); + } + #[test] fn glob_pattern_escapes_quoted_metacharacters() { assert_eq!(glob_pattern("\\*a*"), Some("[*]a*".to_string())); diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs index bcfa282b..86deade1 100644 --- a/impl/rust-cli/tests/word_expansion_e2e_tests.rs +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -124,3 +124,79 @@ fn function_arguments_get_the_matches() { "2:a.txt:b.txt\n" ); } + +// Regressions from the review of #242. Each of these once panicked, hung or +// gave the wrong arguments. + +#[test] +fn non_ascii_text_does_not_hang_or_panic() { + let dir = fixture(); + // The statement splitter looped for ever on an unquoted non-ASCII letter. + assert_eq!(stdout(dir.path(), "echo é"), "é\n"); + assert_eq!(stdout(dir.path(), "if true; then echo ünï; fi"), "ünï\n"); + // Brace expansion sliced the word at a character count. + assert_eq!(stdout(dir.path(), "echo 'é'{a,b}"), "éa éb\n"); + // So did `${...}`. + assert_eq!( + stdout(dir.path(), "é=1; echo ${é} ${é:-f} ${u:-é} ${#é}"), + "1 1 é 1\n" + ); +} + +#[test] +fn a_quoted_parenthesis_does_not_end_a_command_substitution() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo $(echo ')')"), ")\n"); + assert_eq!( + stdout(dir.path(), "echo $(echo \"(\") $(echo \\))"), + "( )\n" + ); + assert_eq!( + stdout(dir.path(), "echo $(echo $(echo a)) $((1+(2*3)))"), + "a 7\n" + ); +} + +#[test] +fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + let dir = fixture(); + for name in ["-.txt", "m.txt", "z.txt"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + assert_eq!( + stdout(dir.path(), "echo [a'-'z]*.txt"), + "-.txt a.txt z.txt\n" + ); + assert_eq!( + stdout(dir.path(), "echo [a-z]*.txt"), + "a.txt b.txt m.txt z.txt\n" + ); +} + +#[test] +fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo ~'local'/x"), "~local/x\n"); +} + +#[test] +fn a_parameter_before_a_quoted_literal_keeps_its_name_in_a_list() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "A=x; echo $A''suffix '$A' \"$A\"b && true"), + "xsuffix $A xb\n" + ); + assert_eq!(stdout(dir.path(), "A=x; echo $A''suffix"), "xsuffix\n"); +} + +#[test] +fn a_substring_length_past_the_end_gives_the_rest() { + let dir = fixture(); + assert_eq!( + stdout( + dir.path(), + "X=abcdef; echo ${X:2:18446744073709551615} ${X:1:2}" + ), + "cdef bc\n" + ); +} From ad8f1d399a8664ff6ad08069cce06fa7a2aa96f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 00:12:13 +0000 Subject: [PATCH 08/21] fix(glob,word): the small findings from the review of #242 Code - parser: an AND-OR list writes its operands back from the typed word, so each piece keeps its quoting (`echo '*'x.* && true` lost the glob). - word, external: expanded fields are OsString; a glob match reaches an external command with the exact bytes of its file name. - glob: a name that is not UTF-8 is matched through a private-use mapping, not a lossy copy, so a literal U+FFFD in a pattern no longer matches it. - glob: resolving a symbolic link in is_dir is a charged read. - glob: new limit max_listing_bytes for the names of one directory listing. - glob: the walker's stop error is boxed; glob::escape is new. - grade: Meter::check computes the attempted level in u128 before the grades are composed; Exceeded::level is u128. - confirmation: the no-follow count is documented as holding for a tree that does not change. Fuzz - fuzz_glob_expansion removes its fixture at exit (libc::atexit), escapes the fixture root in absolute patterns and checks that /**/* reaches it. Tests - charged link lookups, the listing limit, exact name matching, the meter overflow, list operands, exact argument bytes, a glob in a later pipeline stage; wc output is compared without column padding. Documents - CHANGELOG: walker read units defined; the exit-status claim is limited to argument expansion. - POSIX_COMPLIANCE: Milestones 5, 6 and 12 are Partial in the overview and the detail; `for` word lists do not expand globs. - PROOF-OPEN-FRONTIER A-6 and the EXPLAINME proof ledger: fixture evidence is tested-only, the bound is open; Lean is model evidence only, the Rust walker needs Creusot or Verus. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- CHANGELOG.adoc | 38 +++++- EXPLAINME.adoc | 12 ++ docs/POSIX_COMPLIANCE.adoc | 30 +++-- docs/PROOF-OPEN-FRONTIER.adoc | 4 +- impl/rust-cli/fuzz/Cargo.lock | 1 + impl/rust-cli/fuzz/Cargo.toml | 1 + .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 50 +++++++- impl/rust-cli/src/confirmation.rs | 6 + impl/rust-cli/src/executable.rs | 6 +- impl/rust-cli/src/external.rs | 32 +++-- impl/rust-cli/src/glob.rs | 115 ++++++++++++++---- impl/rust-cli/src/grade.rs | 46 +++++-- impl/rust-cli/src/parser.rs | 6 +- impl/rust-cli/src/process_sub.rs | 6 +- impl/rust-cli/src/word.rs | 41 ++++--- impl/rust-cli/tests/glob_bounds_tests.rs | 75 ++++++++++++ .../tests/word_expansion_e2e_tests.rs | 49 +++++++- 17 files changed, 422 insertions(+), 96 deletions(-) diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index ff987c16..4898dcd0 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -24,8 +24,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works. - `$((1+(2*3)))` and other substitutions containing `(` are read to their matching `)`. -- An expansion error (such as a glob limit from #191) fails the command with - exit status 1 and does not run it. With a redirection the status was 127. +- An expansion error in an argument (such as a glob limit from #191) fails + the command with exit status 1 and does not run it. With a redirection the + status was 127. An error while the program name or a pipeline stage name + is expanded is not changed: it is returned to the caller, so `||` sees it. + +=== Fixed — 2026-11-19 (review of #242, small findings) + +- An AND-OR list keeps the quoting of each piece of a word: `echo '*'x.* && true` + printed `*x.*` and now expands the unquoted part. +- A glob match reaches an external command with the exact bytes of its file + name. A name that is not UTF-8 was replaced by a lossy copy that did not + exist. +- A name that is not UTF-8 is no longer matched by a literal U+FFFD in the + pattern. `*` and `?` still match it. +- Resolving a symbolic link to learn whether it is a directory is charged as + a read unit; it was free. +- New limit `max_listing_bytes` (64 MiB): the names held while one directory + is listed. Exceeding it is a `GlobLimitExceeded` that names the directory. +- A `Meter` charge that would pass `u64::MAX` is refused and reported + exactly; `Exceeded::level` is now `u128`. Before, the level saturated and + a limit of `u64::MAX` accepted the charge. +- `Expansion::fields` in `src/word.rs` is `Vec`; the external + command functions take `OsString` arguments. `glob::escape` is new. +- The `fuzz_glob_expansion` target removes its fixture at exit, escapes the + fixture root in absolute patterns, and checks that `/**/*` reaches the + fixture. +- Tests: `wc` output is compared without its column padding; a glob in a + later pipeline stage is covered. === Fixed — 2026-11-18 (review of #242) @@ -66,8 +92,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run 134118693669). The rule now matches bash 5.2 with `globstar` for patterns that start with `**`. -- Glob expansion is metered while it runs: at most 10,000,000 file-system - reads, 100,000 matches and 6 MiB of matches per pattern. Reaching a limit is +- Glob expansion is metered while it runs: at most 10,000,000 walker read + units, 100,000 matches and 6 MiB of matches per command. A read unit is + charged for each directory opened, each entry listed, each name looked up + and each symbolic link resolved. The type of an entry comes with the + listing and is not charged again. The limit is on these units; it is not + a cap on every file-system operation the shell makes. Reaching a limit is a `GlobLimitExceeded` error that names the limit, the usage so far and the directory being read; the command is not run. Before, the match limit was checked only after every path had been collected, and an external command diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index df0c1b3c..a6b4063f 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -126,6 +126,18 @@ reproduce. Absence of a tool is an open obligation, not a passing leg | `proofs/z3/` (`archive/proofs/` for Isabelle and Mizar) | §4 exception notes are now recorded in each `.smt2` header. `filesystem_operations` and `symlink_operations` were rewritten on 2026-11-09 to define the operations as array `store`s and negate each goal; the other three still rest on asserted axioms. Isabelle and Mizar were never CI-checked and are archived, unchecked, with a successor map +| Glob expansion stays within its read bound on the test fixtures (frontier A-6) +| tested-only +| `cargo test --test glob_bounds_tests`; `proptest` in `src/glob.rs` and `src/grade.rs` +| `impl/rust-cli/tests/glob_bounds_tests.rs`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| Fixed fixtures only: the reported read count equals a count made from the fixture, and a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. The grade laws are property-tested, not proved + +| Glob expansion terminates within `O(|pattern| x |entries|)` reads (frontier A-6) +| open +| - +| `impl/rust-cli/src/glob.rs` `expand_glob`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| No Creusot or Verus contract on the walker (PROOF-POLICY §1), and no Lean model bound. A Lean bound would be model evidence only + | Sandbox path normalisation never escapes the root | proven (model) / tested-only (code) | Lean 4 for the model; `proptest` for the code diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index 175e144c..0d382ebc 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -22,14 +22,14 @@ M12-14 not started |2 |Redirections |*Complete* |All 8 types including `+2>&1+` |3 |Pipelines |*Complete* |Multi-stage, exit code from last |4 |Variables |*Complete* |Scalars, arrays, parameter expansion -|5 |Glob Expansion |*Complete* |POSIX-compliant + brace expansion -|6 |Quote Processing |*Complete* |Single, double, backslash +|5 |Glob Expansion |*Partial* |Matching complete; not yet applied to builtin operands, `+for+` word lists, redirection targets +|6 |Quote Processing |*Partial* |Quote removal complete only for arguments that go through `+word::expand_words+` |7 |Command Substitution |*Complete* |`+$(cmd)+` and backticks |8 |Arithmetic |*Complete* |Full operator set including `+**+` |9 |Control Structures |*Complete* |if/elif/else, while, for, case/esac |10 |Job Control |*Partial* |No SIGCHLD, no Ctrl+Z |11 |Shell Builtins |*Partial* |~30 implemented, ~15 missing -|12 |Functions |*Not started* |No `+func() { ... }+` syntax +|12 |Functions |*Partial* |`+name() { ... }+` definitions and calls work; nesting incomplete |13 |Shell Scripts |*Not started* |No `+.sh+` file execution |14 |Advanced Features |*Partial* |Here docs, process sub, arrays done |=== @@ -147,7 +147,7 @@ quoted context) - `+${VAR:=default}+` does not actually assign — same as `+${VAR:-default}+` - `+export VAR+` on unset var errors instead of creating empty exported var -==== Milestone 5: Glob Expansion — COMPLETE +==== Milestone 5: Glob Expansion — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== @@ -170,8 +170,9 @@ links or into hidden directories. One known difference from bash 5.2: bash enters a symlinked directory for one more component when `+**+` is not the first component (2026-11-18) -|Expansion limits |Done |10,000,000 file-system reads, 100,000 matches, -6 MiB of matches; exceeding one is an error that names it, never a truncated +|Expansion limits |Done |10,000,000 walker read units (directories opened, +entries listed, names looked up, links resolved), 100,000 matches, 6 MiB of +matches, 64 MiB of names for one directory listing; exceeding one is an error that names it, never a truncated or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported @@ -185,7 +186,7 @@ with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...), `+for+` word lists, redirection targets |=== -==== Milestone 6: Quote Processing — COMPLETE +==== Milestone 6: Quote Processing — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== @@ -254,8 +255,9 @@ condition chains |`+while+`/`+do+`/`+done+` |Done |`+Command::While+` with break/continue -|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::For+` with glob -expansion in wordlist +|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::ForLoop+`; the word list gets +parameter expansion and `+$IFS+` splitting, but not glob expansion (see +Milestone 5) |`+case+`/`+esac+` |Done |`+Command::Case+` with glob pattern matching @@ -315,9 +317,13 @@ ops, numeric comparison `+hash+`, `+umask+`, `+wait+`, `+getopts+`, `+shift+`, `+break+`, `+continue+`, `+return+`, `+local+` -==== Milestone 12: Functions — NOT STARTED +==== Milestone 12: Functions — PARTIAL -No function definition or calling syntax exists in the parser. +`+name() { ... }+` and `+function name { ... }+` define a function, and a call +runs it with its own positional parameters, `+local+` and `+return+`. An +end-to-end test runs `+f() { echo "$#:$1:$2"; }; f *.txt+` through the built +shell (`+tests/word_expansion_e2e_tests.rs+`). Full nesting of functions is +not complete. ==== Milestone 13: Shell Scripts — NOT STARTED @@ -378,7 +384,7 @@ composition |0 === Critical Missing Features (by impact) [arabic] -. *Functions* — No `+func() { ... }+` — blocks modular scripts +. *Functions* — definitions and calls work; nesting is incomplete . *Shell script execution* — No `+.sh+` file running . *Word splitting (`+$IFS+`)* — Unquoted expansions not split . *Tilde expansion* — Only works for `+~/+` in `+cd+` diff --git a/docs/PROOF-OPEN-FRONTIER.adoc b/docs/PROOF-OPEN-FRONTIER.adoc index 418417b0..828f2075 100644 --- a/docs/PROOF-OPEN-FRONTIER.adoc +++ b/docs/PROOF-OPEN-FRONTIER.adoc @@ -118,8 +118,8 @@ Pure induction on list length. | Statement | `expand_glob(pattern, base_dir)` terminates in time `O(|pattern| × |files_in_base_dir|)`. | File | `impl/rust-cli/src/glob.rs` — no proof today. | Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property. -| Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4. -| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Lean bound and no Rust contract. +| Lift | Property test (`proptest`) for termination + complexity. A bound proved in Lean 4 would be about a Lean model of the walk: optional model evidence, not a statement about the Rust code. The Rust walker (`expand_glob`) needs source-level verification, which the proof policy assigns to Creusot with Pearlite contracts, and Verus. +| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Creusot or Verus contract exists on `expand_glob`, and none has been attempted; no Lean model bound exists either. Ledger rows: `EXPLAINME.adoc#proof-ledger` ("Glob expansion stays within its read bound on the test fixtures" and "Glob expansion terminates within `O(|pattern| x |entries|)` reads"). |=== [[A-7]] diff --git a/impl/rust-cli/fuzz/Cargo.lock b/impl/rust-cli/fuzz/Cargo.lock index 7d73278a..df29e3de 100644 --- a/impl/rust-cli/fuzz/Cargo.lock +++ b/impl/rust-cli/fuzz/Cargo.lock @@ -2022,6 +2022,7 @@ dependencies = [ name = "vsh-fuzz" version = "0.0.0" dependencies = [ + "libc", "libfuzzer-sys", "tempfile", "vsh", diff --git a/impl/rust-cli/fuzz/Cargo.toml b/impl/rust-cli/fuzz/Cargo.toml index 7ea8a865..008f5bb2 100644 --- a/impl/rust-cli/fuzz/Cargo.toml +++ b/impl/rust-cli/fuzz/Cargo.toml @@ -11,6 +11,7 @@ edition = "2021" cargo-fuzz = true [dependencies] +libc = "0.2" libfuzzer-sys = "0.4" tempfile = "3.8" walkdir = "2.4" diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index c467597e..c4f99e02 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -26,12 +26,13 @@ use std::os::unix::fs::symlink; use std::path::{Path, PathBuf}; use std::sync::OnceLock; use tempfile::TempDir; -use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; +use vsh::glob::{escape, expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; const LIMITS: GlobLimits = GlobLimits { max_reads: 20_000, max_matches: 2_000, max_match_bytes: 256 * 1024, + max_listing_bytes: 64 * 1024, }; /// Build the fixture tree. @@ -62,14 +63,40 @@ fn build_fixture() -> std::io::Result { Ok(t) } +static FIXTURE: OnceLock = OnceLock::new(); + +/// Remove the fixture when the process exits. A static is never dropped, so +/// without this every fuzz worker left its tree in the temporary directory. +/// A crash ends the process without running this; that tree stays, with the +/// crashing input, for the person who looks at it. +extern "C" fn remove_fixture() { + if let Some(root) = FIXTURE.get() { + let _ = fs::remove_dir_all(root); + } +} + /// The fixture, built once. A fixture that cannot be built stops the run /// with a clear message: skipping every input would report a green run that /// tested nothing. fn fixture() -> &'static Path { - static FIXTURE: OnceLock = OnceLock::new(); FIXTURE .get_or_init(|| match build_fixture() { - Ok(fixture) => fixture, + Ok(fixture) => { + let root = fixture.keep(); + // The pattern for an absolute input is built from this path; + // it is escaped below, and it must stay valid UTF-8 for that. + if root.to_str().is_none() { + eprintln!("fuzz_glob_expansion: the fixture path is not UTF-8"); + let _ = fs::remove_dir_all(&root); + std::process::exit(1); + } + // SAFETY: `remove_fixture` is a plain `extern "C"` function + // that takes no arguments and does not unwind. + unsafe { + libc::atexit(remove_fixture); + } + root + } Err(e) => { // Exit rather than panic: libFuzzer reports an exiting target // as an error ("fuzz target exited"), so the run still fails. @@ -77,7 +104,7 @@ fn fixture() -> &'static Path { std::process::exit(1); } }) - .path() + .as_path() } fuzz_target!(|data: &[u8]| { @@ -90,7 +117,9 @@ fuzz_target!(|data: &[u8]| { let base = fixture(); let pattern = if input.starts_with('/') { - format!("{}{}", base.display(), input) + // The fixture root is a path, not a pattern: escape it, so a `[` or + // `*` in the temporary directory name is taken literally. + format!("{}{}", escape(&base.to_string_lossy()), input) } else { input.to_string() }; @@ -110,6 +139,15 @@ fuzz_target!(|data: &[u8]| { .sum(); assert_eq!(bytes, usage.match_bytes.held()); + // The input that found the out-of-memory fault must reach the + // fixture, not pass with nothing matched. + if input == "/**/*" { + assert!( + expansion.paths.contains(&base.join("d/sub/g.rs")), + "/**/* did not reach the fixture" + ); + } + // Sorted, each path once, and every path exists. assert!(expansion.paths.windows(2).all(|w| w[0] < w[1])); for path in &expansion.paths { @@ -127,7 +165,7 @@ fuzz_target!(|data: &[u8]| { } Err(err) => { if let Some(stopped) = err.downcast_ref::() { - assert!(stopped.exceeded.level > stopped.exceeded.limit); + assert!(stopped.exceeded.level > u128::from(stopped.exceeded.limit)); assert!(stopped.usage.reads.total() <= LIMITS.max_reads); assert!(stopped.usage.matches.peak() <= LIMITS.max_matches); assert!(stopped.usage.match_bytes.peak() <= LIMITS.max_match_bytes); diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 5433c5db..ba45a541 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -214,6 +214,12 @@ fn prompt_yes_no(prompt: &str) -> Result { /// Symbolic links are counted as entries and never followed, as `rm -r` does: /// following them would count files outside the tree and, with a link back to /// a parent directory, never finish. +/// +/// This holds for a tree that does not change during the count. Each child +/// directory is opened again by its path, so a directory that another process +/// replaces with a link between the type check and the open is followed. The +/// count is only shown in the confirmation prompt; the removal does its own +/// checks. fn count_files_in_tree(path: &str) -> Result { let mut count = 0; diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index 7148cec7..80756c3a 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -505,7 +505,7 @@ impl ExecutableCommand for Command { // double-lookup plus expect shape could panic and duplicated // the lookup. if let Some(func_def) = state.functions.get(&expanded_program).cloned() { - let result = execute_function_call(&func_def, &expansion.fields, state); + let result = execute_function_call(&func_def, &expansion.lossy_fields(), state); expansion.cleanup(); return result; } @@ -568,7 +568,7 @@ impl ExecutableCommand for Command { // Expand every stage before any stage runs, so a failed // expansion runs no part of the pipeline. - let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); + let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); let mut process_subs = Vec::new(); for (program, args) in stages { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; @@ -692,7 +692,7 @@ impl ExecutableCommand for Command { return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); } }; - let expanded_args = &expansion.fields; + let expanded_args = &expansion.lossy_fields(); let output = if *interpret_escapes { expanded_args diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 070e5f99..07de86a2 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -6,6 +6,7 @@ //! and manages stdio. use anyhow::{Context, Result}; +use std::ffi::OsString; use std::fs::{File, OpenOptions}; use std::path::PathBuf; use std::process::{Command, Stdio}; @@ -25,7 +26,7 @@ use crate::state::ShellState; /// - File modification tracking for undo pub fn execute_external_with_redirects( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -116,15 +117,15 @@ pub fn execute_external_with_redirects( /// use vsh::state::ShellState; /// /// let stages = vec![ -/// ("ls".to_string(), vec!["-la".to_string()]), -/// ("grep".to_string(), vec!["test".to_string()]), +/// ("ls".to_string(), vec!["-la".into()]), +/// ("grep".to_string(), vec!["test".into()]), /// ]; /// let mut state = ShellState::new("/tmp")?; /// let exit_code = external::execute_pipeline(&stages, &[], &mut state)?; /// # Ok::<(), anyhow::Error>(()) /// ``` pub fn execute_pipeline( - stages: &[(String, Vec)], + stages: &[(String, Vec)], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -283,11 +284,11 @@ pub fn execute_pipeline( /// ```no_run /// use vsh::external; /// -/// let exit_code = external::execute_external("ls", &["-la".to_string()])?; +/// let exit_code = external::execute_external("ls", &["-la".into()])?; /// assert_eq!(exit_code, 0); /// # Ok::<(), anyhow::Error>(()) /// ``` -pub fn execute_external(program: &str, args: &[String]) -> Result { +pub fn execute_external(program: &str, args: &[OsString]) -> Result { // PATH lookup let executable = find_in_path(program)?; @@ -551,7 +552,7 @@ fn is_executable(path: &PathBuf) -> bool { /// Job ID for the background job pub fn execute_external_background( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -569,7 +570,14 @@ pub fn execute_external_background( let command_str = if args.is_empty() { format!("{} &", program) } else { - format!("{} {} &", program, args.join(" ")) + format!( + "{} {} &", + program, + args.iter() + .map(|arg| arg.to_string_lossy()) + .collect::>() + .join(" ") + ) }; // Build command with redirected stdio @@ -658,7 +666,7 @@ mod tests { #[test] fn test_external_command_with_args() { // Test command with arguments - let exit_code = execute_external("echo", &["hello".to_string()]).unwrap(); + let exit_code = execute_external("echo", &["hello".into()]).unwrap(); assert_eq!(exit_code, 0, "echo should return 0"); } @@ -694,7 +702,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["hello".to_string()]), + ("echo".to_string(), vec!["hello".into()]), ("cat".to_string(), vec![]), ("cat".to_string(), vec![]), ]; @@ -711,7 +719,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["test".to_string()]), + ("echo".to_string(), vec!["test".into()]), ("cat".to_string(), vec![]), ]; let redirects = vec![crate::redirection::Redirection::Output { @@ -735,7 +743,7 @@ mod tests { let temp = TempDir::new().unwrap(); let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); - let stages: Vec<(String, Vec)> = vec![]; + let stages: Vec<(String, Vec)> = vec![]; let result = execute_pipeline(&stages, &[], &mut state); assert!(result.is_err(), "Empty pipeline should return error"); } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index b43dd92c..158f65d2 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -38,11 +38,12 @@ //! //! # Limits //! -//! One expansion is metered on three axes ([`GlobLimits`]): +//! One expansion is metered on these axes ([`GlobLimits`]): //! //! | Axis | Kind | Default | //! |------|------|---------| -//! | file-system reads (directories opened, entries listed, names looked up) | cost: adds up | 10,000,000 | +//! | walker read units (one for each directory opened, entry listed, name looked up and symbolic link resolved; an entry type that the listing gave is not charged) | cost: adds up | 10,000,000 | +//! | bytes of one directory listing held (name length + 1 for each entry) | occupancy, per listing | 64 MiB | //! | matches held | occupancy: high-water mark | 100,000 | //! | bytes of matches held (path length + 1 for each match) | occupancy | 6 MiB | //! @@ -96,6 +97,10 @@ pub const MAX_GLOB_RESULTS: u64 = 100_000; /// most Linux `execve` accepts for argument strings. pub const MAX_GLOB_MATCH_BYTES: u64 = 6 * 1024 * 1024; +/// Default limit on the bytes of names held while one directory is listed +/// (name length + 1 for each entry): 64 MiB. +pub const MAX_GLOB_LISTING_BYTES: u64 = 64 * 1024 * 1024; + /// The limits one glob expansion runs under. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct GlobLimits { @@ -105,6 +110,9 @@ pub struct GlobLimits { pub max_matches: u64, /// Most bytes of matches held at once (path length + 1 per match). pub max_match_bytes: u64, + /// Most bytes of names held while one directory is listed (name length + 1 + /// per entry). The listing is sorted before use, so it is held whole. + pub max_listing_bytes: u64, } impl GlobLimits { @@ -113,6 +121,7 @@ impl GlobLimits { max_reads: MAX_GLOB_READS, max_matches: MAX_GLOB_RESULTS, max_match_bytes: MAX_GLOB_MATCH_BYTES, + max_listing_bytes: MAX_GLOB_LISTING_BYTES, }; } @@ -300,7 +309,9 @@ pub fn expand_glob_continuing( let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); - walker.run().map_err(anyhow::Error::new)?; + walker + .run() + .map_err(|stopped| anyhow::Error::new(*stopped))?; Ok(Expansion { usage: walker.usage(), paths: walker.found.into_iter().collect(), @@ -350,12 +361,52 @@ impl Compiled { } } +/// The text a name is matched as. +/// +/// The `glob` crate matches `str` only. A byte that is not valid UTF-8 becomes +/// one character from U+F780..U+F7FF (private use), so `?` and `*` still +/// match it, and it does not look like a literal U+FFFD in the name or the +/// pattern, as a lossy conversion would make it. +// snip: a pattern that contains one of those 128 private-use characters +// matches the invalid byte too; an exact matcher needs a byte-level `Pattern`. +fn match_text(name: &std::ffi::OsStr) -> std::borrow::Cow<'_, str> { + if let Some(text) = name.to_str() { + return std::borrow::Cow::Borrowed(text); + } + let mut out = String::new(); + for chunk in name.as_encoded_bytes().utf8_chunks() { + out.push_str(chunk.valid()); + for &byte in chunk.invalid() { + // Invalid bytes are 0x80..=0xFF, so this stays in U+F780..U+F7FF. + out.push( + char::from_u32(0xF700 + u32::from(byte)).unwrap_or(char::REPLACEMENT_CHARACTER), + ); + } + } + std::borrow::Cow::Owned(out) +} + +/// `text` as a pattern that matches exactly `text`: `*`, `?` and `[` in it +/// are taken literally. Use it for a directory prefix that is not a pattern. +/// +/// # Examples +/// ``` +/// assert_eq!(vsh::glob::escape("/tmp/a*b"), "/tmp/a[*]b"); +/// ``` +pub fn escape(text: &str) -> String { + glob::Pattern::escape(text) +} + const MATCH_OPTIONS: glob::MatchOptions = glob::MatchOptions { case_sensitive: true, require_literal_separator: true, require_literal_leading_dot: true, }; +/// The result of a step of the walk. The error is boxed: it is built once, on +/// the path that stops the walk, and is larger than the values the steps return. +type Walk = std::result::Result>; + /// The walk: a depth-first search over states (directory, component index), /// each processed at most once. struct Walker<'a> { @@ -365,6 +416,7 @@ struct Walker<'a> { reads: Meter, matches: Meter, match_bytes: Meter, + max_listing_bytes: u64, found: BTreeSet, } @@ -387,6 +439,7 @@ impl<'a> Walker<'a> { limits.max_match_bytes, spent.match_bytes, ), + max_listing_bytes: limits.max_listing_bytes, found: BTreeSet::new(), } } @@ -410,8 +463,8 @@ impl<'a> Walker<'a> { } } - fn stopped(&self, exceeded: Exceeded, at: &Path) -> GlobLimitExceeded { - GlobLimitExceeded { + fn stopped(&self, exceeded: Exceeded, at: &Path) -> Box { + Box::new(GlobLimitExceeded { pattern: self.pattern.to_string(), exceeded, usage: self.usage(), @@ -420,15 +473,15 @@ impl<'a> Walker<'a> { } else { at.to_path_buf() }, - } + }) } - fn charge_read(&mut self, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + fn charge_read(&mut self, at: &Path) -> Walk<()> { self.reads.charge(1).map_err(|e| self.stopped(e, at)) } /// Record a match, once. - fn emit(&mut self, path: PathBuf, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + fn emit(&mut self, path: PathBuf, at: &Path) -> Walk<()> { let path = if self.compiled.dirs_only { path.join("") } else { @@ -454,18 +507,23 @@ impl<'a> Walker<'a> { /// List a directory in name order, charging the open and every entry. /// An unreadable directory gives no entries. - fn list( - &mut self, - dir: &Path, - ) -> std::result::Result, GlobLimitExceeded> { + fn list(&mut self, dir: &Path) -> Walk> { self.charge_read(dir)?; let mut entries = Vec::new(); + // The listing is held whole to sort it; meter the names it holds. + let mut held = Meter::::new( + "bytes of one directory listing held", + self.max_listing_bytes, + ); if let Ok(read) = fs::read_dir(self.real(dir)) { for entry in read { self.charge_read(dir)?; if let Ok(entry) = entry { if let Ok(kind) = entry.file_type() { - entries.push((entry.file_name(), kind)); + let name = entry.file_name(); + held.charge(name.len() as u64 + 1) + .map_err(|e| self.stopped(e, dir))?; + entries.push((name, kind)); } } } @@ -475,15 +533,22 @@ impl<'a> Walker<'a> { } /// Whether `path` is a directory, going through a symbolic link. - fn is_dir(&self, path: &Path, kind: Option) -> bool { + /// + /// A type that the directory listing already gave costs nothing. Looking + /// the path up (no type known, or a symbolic link to resolve) is one read, + /// charged before the lookup. + fn is_dir(&mut self, path: &Path, kind: Option, at: &Path) -> Walk { match kind { - Some(kind) if kind.is_dir() => true, - Some(kind) if !kind.is_symlink() => false, - _ => fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir()), + Some(kind) if kind.is_dir() => Ok(true), + Some(kind) if !kind.is_symlink() => Ok(false), + _ => { + self.charge_read(at)?; + Ok(fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir())) + } } } - fn run(&mut self) -> std::result::Result<(), GlobLimitExceeded> { + fn run(&mut self) -> Walk<()> { let count = self.compiled.components.len(); if count == 0 { return Ok(()); @@ -511,24 +576,24 @@ impl<'a> Walker<'a> { let child = dir.join(name); if last { let exists = fs::symlink_metadata(self.real(&child)).is_ok(); - if exists && (!compiled.dirs_only || self.is_dir(&child, None)) { + if exists && (!compiled.dirs_only || self.is_dir(&child, None, &dir)?) { self.emit(child, &dir)?; } - } else if self.is_dir(&child, None) { + } else if self.is_dir(&child, None, &dir)? { next.push((child, index + 1)); } } Component::Wild(pattern) => { for (name, kind) in self.list(&dir)? { - if !pattern.matches_with(&name.to_string_lossy(), MATCH_OPTIONS) { + if !pattern.matches_with(&match_text(&name), MATCH_OPTIONS) { continue; } let child = dir.join(&name); if last { - if !compiled.dirs_only || self.is_dir(&child, Some(kind)) { + if !compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)? { self.emit(child, &dir)?; } - } else if self.is_dir(&child, Some(kind)) { + } else if self.is_dir(&child, Some(kind), &dir)? { next.push((child, index + 1)); } } @@ -546,7 +611,7 @@ impl<'a> Walker<'a> { continue; } let child = dir.join(&name); - if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind))) { + if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)?) { self.emit(child.clone(), &dir)?; } if kind.is_dir() { @@ -597,7 +662,7 @@ mod walk_tests { fn absolute_patterns_give_absolute_paths() { let t = TempDir::new().unwrap(); fs::write(t.path().join("x.txt"), b"").unwrap(); - let pattern = format!("{}/*.txt", t.path().display()); + let pattern = format!("{}/*.txt", escape(&t.path().to_string_lossy())); assert_eq!( expand_glob(&pattern, Path::new("/nonexistent")).unwrap(), [t.path().join("x.txt")] diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs index ad56a78b..66fc01ec 100644 --- a/impl/rust-cli/src/grade.rs +++ b/impl/rust-cli/src/grade.rs @@ -80,6 +80,10 @@ pub trait Grade: Copy + Eq + fmt::Debug { /// The single number a limit is compared against. fn level(self) -> u64; + + /// What a further `of(amount)` is added to: everything spent for a cost, + /// what is still held for an occupancy. + fn base(self) -> u64; } /// A cumulative cost: a resource that is consumed and never given back. @@ -114,6 +118,10 @@ impl Grade for Cost { fn level(self) -> u64 { self.0 } + + fn base(self) -> u64 { + self.0 + } } impl fmt::Display for Cost { @@ -191,6 +199,10 @@ impl Grade for Occupancy { fn level(self) -> u64 { self.peak } + + fn base(self) -> u64 { + self.held + } } impl fmt::Display for Occupancy { @@ -206,8 +218,9 @@ pub struct Exceeded { pub axis: &'static str, /// The limit that was configured. pub limit: u64, - /// The level the refused charge would have reached. - pub level: u64, + /// The level the refused charge would have reached. Wider than a limit, + /// so a charge that passes `u64::MAX` is reported exactly, not saturated. + pub level: u128, } impl fmt::Display for Exceeded { @@ -258,12 +271,15 @@ impl Meter { /// # Errors /// Returns the [`Exceeded`] that the charge would return. pub fn check(&self, amount: u64) -> Result<(), Exceeded> { - let next = self.grade.then(G::of(amount)); - if next.level() > self.limit { + // Computed in u128 before the grades are composed: composition + // saturates, and a saturated level would pass a limit of `u64::MAX`. + let level = + u128::from(self.grade.level()).max(u128::from(self.grade.base()) + u128::from(amount)); + if level > u128::from(self.limit) { return Err(Exceeded { axis: self.axis, limit: self.limit, - level: next.level(), + level, }); } Ok(()) @@ -401,6 +417,22 @@ mod tests { ); } + #[test] + fn a_charge_past_u64_max_is_refused_and_reported_exactly() { + // A saturated level would equal the limit and be accepted. + let mut cost = Meter::::new("units", u64::MAX); + cost.charge(u64::MAX - 1).unwrap(); + cost.charge(1).unwrap(); // reaches the limit exactly + let refused = cost.charge(5).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 5); + assert_eq!(cost.level(), u64::MAX); + + let mut held = Meter::::new("held", u64::MAX); + held.charge(u64::MAX).unwrap(); + let refused = held.charge(1).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 1); + } + #[test] fn a_meter_with_a_zero_limit_refuses_any_use() { let mut meter: Meter = Meter::new("items held", 0); @@ -481,7 +513,7 @@ mod tests { } Err(refused) => { prop_assert!(wanted.level() > limit); - prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.level, u128::from(wanted.level())); prop_assert_eq!(refused.limit, limit); } } @@ -511,7 +543,7 @@ mod tests { } Err(refused) => { prop_assert!(wanted.level() > limit); - prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.level, u128::from(wanted.level())); } } } diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index 3082b834..b111f0fe 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -1203,7 +1203,9 @@ fn tokens_to_string(tokens: &[Token]) -> Result { } match token { - Token::Word(w) => result.push_str("ed_word_to_string(w)), + // The typed word keeps the quoting of each piece, so `'*'x.*` + // comes back with only its unquoted part active. + Token::Word(w) => result.push_str(&w.typed.to_string()), Token::OutputRedirect => result.push('>'), Token::AppendRedirect => result.push_str(">>"), Token::InputRedirect => result.push('<'), @@ -2689,7 +2691,7 @@ pub fn expand_command_substitution( // echo builtin: its output is the joined, expanded arguments. let expansion = crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; - let mut out = expansion.fields.join(" "); + let mut out = expansion.lossy_fields().join(" "); if !no_newline { out.push('\n'); } diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index 131eee30..d20b400c 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -158,7 +158,8 @@ fn start_command_with_output_redirect( Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? + .lossy_fields(); // Build command string: prog arg1 arg2 > fifo let mut cmd = expanded_program; @@ -208,7 +209,8 @@ fn start_command_with_input_redirect( Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? + .lossy_fields(); // Build command string let mut cmd = expanded_program; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 08f7573f..b2b4c60e 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -42,8 +42,9 @@ //! targets, assignments, `case` patterns and `test` arguments still use the //! older string encoding. +use std::ffi::OsString; use std::fmt; -use std::path::Path; +use std::path::{Path, PathBuf}; use anyhow::{Context, Result}; @@ -258,13 +259,24 @@ pub enum ProcessSubs { /// The result of expanding a list of words. pub struct Expansion { - /// The arguments, after quote removal. - pub fields: Vec, + /// The arguments, after quote removal. A glob match keeps the exact bytes + /// of its file name, which need not be UTF-8. + pub fields: Vec, /// Process substitutions started for these arguments. pub process_subs: Vec, } impl Expansion { + /// The arguments as text, for a builtin or a function that works on + /// strings. Bytes that are not UTF-8 are replaced; external commands get + /// [`Expansion::fields`] unchanged. + pub fn lossy_fields(&self) -> Vec { + self.fields + .iter() + .map(|field| field.to_string_lossy().into_owned()) + .collect() + } + /// Wait for the process substitutions and remove their FIFOs. pub fn cleanup(self) { for proc_sub in self.process_subs { @@ -438,32 +450,27 @@ fn glob_fields( cwd: &Path, limits: &GlobLimits, spent: &mut GlobUsage, - fields: &mut Vec, + fields: &mut Vec, ) -> Result<()> { let encoded = encode(chars); for candidate in glob::expand_braces(&encoded) { match glob_pattern(&candidate) { - None => fields.push(remove_quotes(&candidate)), + None => fields.push(remove_quotes(&candidate).into()), Some(pattern) => match glob::expand_glob_continuing(&pattern, cwd, limits, *spent) { Ok(found) if !found.paths.is_empty() => { *spent = found.usage; - fields.extend( - found - .paths - .iter() - .map(|path| path.to_string_lossy().into_owned()), - ); + fields.extend(found.paths.into_iter().map(PathBuf::into_os_string)); } // No match still spent reads. Ok(found) => { *spent = found.usage; - fields.push(remove_quotes(&candidate)); + fields.push(remove_quotes(&candidate).into()); } // A limit is not "no match": substituting the literal word // would run the command on the wrong arguments. Err(e) if e.is::() => return Err(e), // No match, or not a valid pattern: keep the word (POSIX). - _ => fields.push(remove_quotes(&candidate)), + _ => fields.push(remove_quotes(&candidate).into()), }, } } @@ -566,7 +573,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap() - .fields + .lossy_fields() } #[test] @@ -660,7 +667,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap() - .fields; + .lossy_fields(); assert_eq!(out, ["a.txt", "b.txt", "*.txt"]); } @@ -711,7 +718,7 @@ mod tests { &limits, ) .unwrap() - .fields; + .lossy_fields(); assert_eq!(out, ["f0.txt", "f1.txt", "f2.txt", "f3.txt"]); } @@ -764,7 +771,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap(); - assert_eq!(out.fields, ["xsuffix"]); + assert_eq!(out.lossy_fields(), ["xsuffix"]); } #[test] diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs index ead12100..aff5dfa3 100644 --- a/impl/rust-cli/tests/glob_bounds_tests.rs +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -221,6 +221,7 @@ fn default_limits_are_the_documented_values() { max_reads: 10_000_000, max_matches: 100_000, max_match_bytes: 6 * 1024 * 1024, + max_listing_bytes: 64 * 1024 * 1024, } ); } @@ -395,3 +396,77 @@ fn a_name_that_is_not_utf8_is_matched_without_a_panic() { assert_eq!(found.len(), 2); assert!(found.contains(&PathBuf::from(odd))); } + +// --------------------------------------------------------------------------- +// Review of #242: charged lookups, the listing budget, exact name matching +// --------------------------------------------------------------------------- + +#[test] +fn resolving_a_symbolic_link_is_a_charged_read() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + for i in 0..5 { + symlink("real", t.path().join(format!("link{i}"))).unwrap(); + } + // `*/` lists 6 entries (1 open + 6 entries = 7 reads) and then resolves + // each of the 5 links to learn whether it is a directory: 12 reads. + let all = expand_glob_with_limits("*/", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(all.paths.len(), 6); + assert_eq!(all.usage.reads.total(), 12); + + // With room for the listing only, the first lookup is refused. Before, + // the lookups were free and the expansion finished. + let limits = GlobLimits { + max_reads: 7, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*/", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (7, 8)); +} + +#[test] +fn the_listing_limit_stops_a_directory_that_is_too_large_to_hold() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("big")).unwrap(); + for i in 0..10 { + fs::write(t.path().join(format!("big/f{i}")), b"").unwrap(); + } + // Each name is 2 bytes + 1, so 10 names need 30 bytes. + let limits = GlobLimits { + max_listing_bytes: 29, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("big/*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of one directory listing held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (29, 30)); + assert_eq!(s.at, PathBuf::from("big")); + + let limits = GlobLimits { + max_listing_bytes: 30, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("big/*", t.path(), &limits).unwrap(); + assert_eq!(ok.paths.len(), 10); +} + +#[test] +fn an_invalid_byte_is_not_matched_as_a_replacement_character() { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let invalid = OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(invalid), b"").is_err() { + return; // this file system refuses names that are not UTF-8 + } + fs::write(t.path().join("bad\u{fffd}.txt"), b"").unwrap(); + + // Broad patterns match both names. + for pattern in ["*.txt", "bad?.txt"] { + let found = expand_glob(pattern, t.path()).unwrap(); + assert_eq!(found.len(), 2, "{pattern}"); + } + // A literal U+FFFD matches only the name that really contains it. + let found = expand_glob("bad\u{fffd}*", t.path()).unwrap(); + assert_eq!(found, [PathBuf::from("bad\u{fffd}.txt")]); +} diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs index 86deade1..888adee8 100644 --- a/impl/rust-cli/tests/word_expansion_e2e_tests.rs +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -41,12 +41,20 @@ fn stdout(dir: &Path, script: &str) -> String { String::from_utf8(out.stdout).unwrap() } +/// The lines of `text` with runs of white space made one space, so the column +/// padding of `wc` does not matter. +fn rows(text: &str) -> Vec { + text.lines() + .map(|line| line.split_whitespace().collect::>().join(" ")) + .collect() +} + #[test] fn external_command_without_redirect_gets_the_matches() { let dir = fixture(); assert_eq!( - stdout(dir.path(), "wc -c *.txt"), - "0 a.txt\n0 b.txt\n0 total\n" + rows(&stdout(dir.path(), "wc -c *.txt")), + ["0 a.txt", "0 b.txt", "0 total"] ); } @@ -55,8 +63,8 @@ fn external_command_with_redirect_gets_the_matches() { let dir = fixture(); stdout(dir.path(), "wc -c *.txt > out.wc"); assert_eq!( - std::fs::read_to_string(dir.path().join("out.wc")).unwrap(), - "0 a.txt\n0 b.txt\n0 total\n" + rows(&std::fs::read_to_string(dir.path().join("out.wc")).unwrap()), + ["0 a.txt", "0 b.txt", "0 total"] ); } @@ -200,3 +208,36 @@ fn a_substring_length_past_the_end_gives_the_rest() { "cdef bc\n" ); } + +#[test] +fn a_mixed_quote_word_keeps_its_quoting_in_a_list() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + // The operands of && were written back with one quoting for the whole + // word, so the unquoted `.*` stopped being a pattern. + assert_eq!(stdout(dir.path(), "echo '*'x.* && true"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "false || echo *x.*"), "*x.txt ax.txt\n"); + assert_eq!(stdout(dir.path(), "echo '*.txt' && true"), "*.txt\n"); +} + +#[cfg(unix)] +#[test] +fn an_external_command_gets_the_exact_bytes_of_a_matched_name() { + use std::os::unix::ffi::OsStrExt; + let dir = fixture(); + let name = std::ffi::OsStr::from_bytes(b"bad\xff.bin"); + if std::fs::write(dir.path().join(name), b"12345").is_err() { + return; // this file system refuses names that are not UTF-8 + } + // A lossy name would not exist, and `cat` would fail. + assert_eq!(stdout(dir.path(), "cat *.bin"), "12345"); + assert_eq!(stdout(dir.path(), "cat *.bin | cat"), "12345"); +} + +#[test] +fn a_later_pipeline_stage_gets_the_matches() { + let dir = fixture(); + let out = stdout(dir.path(), "printf '' | wc -c *.txt"); + assert_eq!(rows(&out), ["0 a.txt", "0 b.txt", "0 total"]); +} From 217f593a2417cb9e4e347c628c06cb767cebff09 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 01:16:53 +0000 Subject: [PATCH 09/21] test(parser): expand a double-quoted parameter through the typed external path test_double_quotes_with_expansion parsed `touch`, which is a builtin, so its Command::External arm never ran, and that arm used the Display form of the word as if it were the expanded argument. The test now parses an external command, expands its typed words with word::expand_words, and checks parameter expansion and quote removal. The builtin form is checked for the exact value. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- impl/rust-cli/src/parser.rs | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index b111f0fe..f61888bd 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -4299,20 +4299,26 @@ mod tests { let mut state = ShellState::new("/tmp/vsh_test").unwrap(); state.set_variable("NAME", "Alice"); - // Test external command with variable expansion - let cmd = parse_command("touch \"file_$NAME\"").unwrap(); - match cmd { - Command::External { program, args, .. } => { - assert_eq!(program, "touch"); - let expanded = expand_variables(&args[0].to_string(), &state); - assert_eq!(expanded, "file_Alice"); - } + // An external command: its argument is a typed word, expanded by + // `word::expand_words`. `touch` is a builtin and would not take this + // path, so the old form of this test never ran its external arm. + let cmd = parse_command("wc \"file_$NAME\" \"a $NAME\"x").unwrap(); + let Command::External { program, args, .. } = cmd else { + panic!("Expected an external command"); + }; + assert_eq!(program, "wc"); + let expansion = + crate::word::expand_words(&args, &mut state, crate::word::ProcessSubs::Refuse).unwrap(); + // The parameter is expanded, the quotes are removed, and the spaces + // inside the quotes stay in one argument. + assert_eq!(expansion.lossy_fields(), ["file_Alice", "a Alicex"]); + + // The builtin still takes the older string form. + match parse_command("touch \"file_$NAME\"").unwrap() { Command::Touch { path, .. } => { - // Could also be parsed as Touch command - let expanded = expand_variables(&path, &state); - assert!(expanded.contains("file_")); + assert_eq!(expand_variables(&path, &state), "file_Alice"); } - _ => panic!("Expected External or Touch command"), + other => panic!("Expected Touch, got {other:?}"), } } From bb40fc5354ff945a75130313889116e3000deddf Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:28:51 +0000 Subject: [PATCH 10/21] fix(glob): charge an intermediate literal lookup once A non-final literal component was charged in run() and again in is_dir(), so real/file cost three reads for two lookups and a budget of two failed early. The literal is now charged only in the final-component branch; is_dir charges the intermediate lookup. Adds a regression test. Answers the CodeRabbit finding on glob.rs line 582 of #242. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0136P7ZpKjBuoCRahUbNPywX --- impl/rust-cli/src/glob.rs | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 158f65d2..f1b6bfa0 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -572,9 +572,11 @@ impl<'a> Walker<'a> { match &compiled.components[index] { Component::Literal(name) => { - self.charge_read(&dir)?; let child = dir.join(name); if last { + // One lookup. An intermediate component is charged by + // `is_dir` below, so it is not charged here as well. + self.charge_read(&dir)?; let exists = fs::symlink_metadata(self.real(&child)).is_ok(); if exists && (!compiled.dirs_only || self.is_dir(&child, None, &dir)?) { self.emit(child, &dir)?; @@ -658,6 +660,29 @@ mod walk_tests { assert!(expand("*.none", t.path()).is_empty()); } + #[test] + fn intermediate_literal_is_charged_once() { + // `real/file` needs two lookups: `real`, then `file`. A budget of two + // reads must be enough, and a budget of one must stop at the second. + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + fs::write(t.path().join("real/file"), b"").unwrap(); + + let two = GlobLimits { + max_reads: 2, + ..GlobLimits::DEFAULT + }; + let found = expand_glob_with_limits("real/file", t.path(), &two).unwrap(); + assert_eq!(found.paths, [PathBuf::from("real/file")]); + + let one = GlobLimits { + max_reads: 1, + ..GlobLimits::DEFAULT + }; + let err = expand_glob_with_limits("real/file", t.path(), &one).unwrap_err(); + assert!(err.downcast_ref::().is_some()); + } + #[test] fn absolute_patterns_give_absolute_paths() { let t = TempDir::new().unwrap(); From 6ff543cc5f839182914bf3de5262d819fbbcce79 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:29:54 +0000 Subject: [PATCH 11/21] fix(process_sub): keep non-UTF-8 argument bytes Both helpers turned the expanded arguments into text with lossy_fields(), so a name such as bad\xff.bin became bad.bin. The arguments now stay OsString values and reach sh as positional parameters, forwarded with "$@". Adds a regression test for each helper. Answers the CodeRabbit finding on process_sub.rs lines 162 and 213 of #242. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0136P7ZpKjBuoCRahUbNPywX --- impl/rust-cli/src/process_sub.rs | 105 +++++++++++++++++++------------ 1 file changed, 66 insertions(+), 39 deletions(-) diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index d20b400c..fe3bf2ae 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -15,6 +15,7 @@ //! ``` use anyhow::{anyhow, Result}; +use std::ffi::OsString; use std::path::{Path, PathBuf}; use std::process::{Child, Command as ProcessCommand}; @@ -154,30 +155,24 @@ fn start_command_with_output_redirect( // For FIFOs, we use shell redirection (sh -c "cmd > fifo") to avoid blocking // The shell handles the FIFO opening in the background properly - let cmd_string = match cmd { + // The arguments stay OsString values and reach the shell as positional + // parameters, forwarded with "$@". Quoting them into the script text would + // need UTF-8 and would lose the bytes of a name that is not UTF-8. + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? - .lossy_fields(); - - // Build command string: prog arg1 arg2 > fifo - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - // Simple shell escaping (single quotes) - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } - Command::Pwd { .. } => "pwd".to_string(), + Command::Pwd { .. } => ("pwd".to_string(), Vec::new()), Command::Ls { path, .. } => { let mut cmd = "ls".to_string(); if let Some(p) = path { let expanded = crate::parser::expand_with_command_sub(p, state)?; cmd.push_str(&format!(" '{}'", expanded.replace('\'', "'\\''"))); } - cmd + (cmd, Vec::new()) } _ => { return Err(anyhow!( @@ -188,13 +183,7 @@ fn start_command_with_output_redirect( // Use sh -c to run: cmd > fifo_path let shell_cmd = format!("{} > {}", cmd_string, output_path.display()); - - let child = ProcessCommand::new("sh") - .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + spawn_sh(&shell_cmd, &args) } /// Start command with stdin redirected from file (for >(cmd)) @@ -205,20 +194,13 @@ fn start_command_with_input_redirect( ) -> Result { // For FIFOs, use shell redirection: sh -c "cmd < fifo" - let cmd_string = match cmd { + // As above: OsString positional parameters forwarded with "$@". + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? - .lossy_fields(); - - // Build command string - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } _ => { return Err(anyhow!( @@ -229,13 +211,18 @@ fn start_command_with_input_redirect( // Use sh -c to run: cmd < fifo_path let shell_cmd = format!("{} < {}", cmd_string, input_path.display()); + spawn_sh(&shell_cmd, &args) +} - let child = ProcessCommand::new("sh") +/// Run `sh -c script sh args...`. The arguments become `$1`, `$2`, ... and are +/// passed as raw OS strings, so their bytes and their boundaries survive. +fn spawn_sh(script: &str, args: &[OsString]) -> Result { + Ok(ProcessCommand::new("sh") .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + .arg(script) + .arg("sh") + .args(args) + .spawn()?) } #[cfg(test)] @@ -289,4 +276,44 @@ mod tests { proc_sub1.cleanup().unwrap(); proc_sub2.cleanup().unwrap(); } + + /// A file named `bad\xff.bin` (not UTF-8) and a pattern that matches it. + #[cfg(unix)] + fn non_utf8_fixture(dir: &Path) -> String { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let name = OsStr::from_bytes(b"bad\xff.bin"); + std::fs::write(dir.join(name), "12345").unwrap(); + format!("{}/bad*.bin", crate::glob::escape(&dir.to_string_lossy())) + } + + #[cfg(unix)] + #[test] + fn output_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let cmd = parse_command(&format!("cat {pattern}")).unwrap(); + let out = temp_dir.path().join("out.txt"); + + let mut child = start_command_with_output_redirect(&cmd, &mut state, &out).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(&out).unwrap(), "12345"); + } + + #[cfg(unix)] + #[test] + fn input_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let copy = temp_dir.path().join("copy.txt"); + let cmd = parse_command(&format!("sort -o {} {pattern}", copy.display())).unwrap(); + let input = temp_dir.path().join("in.txt"); + std::fs::write(&input, "").unwrap(); + + let mut child = start_command_with_input_redirect(&cmd, &mut state, &input).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(©).unwrap(), "12345\n"); + } } From 281b135ac76bb84bc3b9e8244f64b859093a5a16 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:10:04 +0000 Subject: [PATCH 12/21] fix(shell): Correct quoting and glob edge cases --- impl/rust-cli/src/parser.rs | 63 ++++++++++++++++++++++++++++++++++--- impl/rust-cli/src/word.rs | 17 +++++++++- 2 files changed, 74 insertions(+), 6 deletions(-) diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index f61888bd..2b054384 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -965,7 +965,9 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> let mut double = false; let mut escaped = false; - for ch in chars.by_ref() { + // A nested $() has its own quotes, even inside outer double quotes. + let mut contexts = Vec::new(); + while let Some(ch) = chars.next() { if escaped { escaped = false; cmd.push(ch); @@ -984,6 +986,12 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> double = !double; cmd.push(ch); } + '$' if !single && chars.peek() == Some(&'(') => { + chars.next(); + cmd.push_str("$("); + contexts.push((depth, single, double)); + (depth, single, double) = (1, false, false); + } _ if single || double => cmd.push(ch), '(' => { // Every `(` nests, so the `)` that closes `$((1+2))` or @@ -994,7 +1002,11 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> ')' => { depth -= 1; if depth == 0 { - return Ok(cmd); + if let Some(context) = contexts.pop() { + (depth, single, double) = context; + } else { + return Ok(cmd); + } } cmd.push(ch); } @@ -2603,9 +2615,14 @@ fn quoted_word_to_string(word: &QuotedWord) -> String { // so it is always an expansion, whatever quotes follow in the word. // Braces keep a following literal out of the name (`$A''b`). WordPart::Variable(name) | WordPart::BracedVariable(name) => { - result.push_str("${"); - result.push_str(name); - result.push('}'); + if matches!(name.as_str(), "?" | "$" | "#" | "@" | "*") { + result.push('$'); + result.push_str(name); + } else { + result.push_str("${"); + result.push_str(name); + result.push('}'); + } } WordPart::CommandSub(cmd) => { // Command substitution - keep as-is for now @@ -4275,6 +4292,42 @@ mod tests { // Quote Parsing Tests (Phase 6 M5) // ========================================================================= + #[test] + fn command_substitution_quotes_have_separate_nested_contexts() { + for source in [ + r#"echo "$(echo ")")" tail"#, + r#"echo "$(echo '"')" tail"#, + r#"echo "$(echo "$(echo ')')")" tail"#, + r#"echo ')' "(" \) (echo nested) tail"#, + r#"echo $(echo "(") tail"#, + ] { + let input = format!("{source})suffix"); + let mut chars = input.chars().peekable(); + assert_eq!(parse_command_sub_dollar(&mut chars).unwrap(), source); + assert_eq!(chars.collect::(), "suffix"); + } + let mut chars = r#"echo "$(echo ")")""#.chars().peekable(); + assert!(parse_command_sub_dollar(&mut chars).is_err()); + } + + #[test] + fn special_parameters_render_unbraced_and_names_keep_their_boundary() { + for name in ["?", "$", "#", "@", "*"] { + for source in [format!("${name}"), format!("${{{name}}}")] { + let tokens = tokenize(&source).unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), format!("${name}")); + } + } + let tokens = tokenize("$A''suffix").unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), "${A}suffix"); + } + #[test] fn test_single_quotes_no_expansion() { use crate::state::ShellState; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index b2b4c60e..55550aa1 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -492,7 +492,10 @@ fn glob_pattern(encoded: &str) -> Option { if ch == '\\' { let Some(next) = chars.next() else { continue }; match (class.as_mut(), next) { - (Some(_), '-') => hyphen = true, + (Some(members), '-') => { + *members += 1; + hyphen = true; + } // `*`, `?` and `[` are already literal inside a class. (Some(members), '*' | '?' | '[') => { *members += 1; @@ -787,6 +790,18 @@ mod tests { assert_eq!(expand_in(dir.path(), &[w]), ["-.txt", "a.txt", "z.txt"]); } + #[test] + fn a_sole_quoted_hyphen_closes_a_bracket_class() { + // ['-']*.txt + let dir = fixture(&["-.txt", "a.txt"]); + let w = word(&[ + ("[", Quote::None), + ("-", Quote::Single), + ("]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt"]); + } + #[test] fn a_tilde_with_a_quoted_prefix_is_not_expanded() { let dir = fixture(&[]); From 9de7856e608c27d054cf68eb55f6c5b17ea3c9f2 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:42:29 +0000 Subject: [PATCH 13/21] fix(process_sub): release a substitution that no command opened CodeRabbit (PR 242, executable.rs line 509, Major): `echo <(pwd)` printed the FIFO path and then hung the shell. The substitution child was blocked in open(2) on a FIFO that no command opened, and cleanup() waited for it with a blocking wait that never returned. The same happened for >(cmd), and when a function or a pipeline stage did not open its FIFO. cleanup() runs after the last command that could open the FIFO has finished. While the child still runs, it now opens the end of the FIFO that the child waits for, without blocking, and closes it at once. A <(cmd) writer sees its reader leave; a >(cmd) reader sees end of file. A connected child is not cut short: cleanup keeps waiting until it exits, so a >(cmd) reader still finishes the work it was given. A child that ends on SIGPIPE after its reader left is not reported as a failure. Tests, written first: the two unopened cases failed with "cleanup waited forever" (and left their sh children blocked); they pass now. Two more pin the behaviour that must not change: a connected <(cmd) still delivers its output, and a connected >(cmd) finishes before cleanup returns. The built shell now runs `echo <(pwd); echo after` to the end. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/process_sub.rs | 133 ++++++++++++++++++++++++++++++- 1 file changed, 130 insertions(+), 3 deletions(-) diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index fe3bf2ae..dbaf4f2e 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -108,12 +108,30 @@ impl ProcessSubstitution { Ok(()) } - /// Wait for background process and cleanup + /// Wait for background process and cleanup. + /// + /// Callers run this after the last command that could open the FIFO has + /// finished. A child that no command opened is still blocked in its + /// `open(2)` of the FIFO, waiting for a partner, so a plain `wait` would + /// never return (`echo <(pwd)` prints the path and opens nothing). While + /// the child runs, the shell therefore opens the end that the child waits + /// for, without blocking, and closes it at once: a `<(cmd)` writer sees + /// its reader leave, and a `>(cmd)` reader sees end of file. A connected + /// child is not cut short: the shell keeps waiting until it exits, so a + /// `>(cmd)` reader still finishes the work it was given. pub fn cleanup(mut self) -> Result<()> { // Wait for background process to finish if let Some(mut child) = self.child_process.take() { - let status = child.wait()?; - if !status.success() { + let status = loop { + if let Some(status) = child.try_wait()? { + break status; + } + self.release_waiting_child(); + std::thread::sleep(std::time::Duration::from_millis(10)); + }; + // A `<(cmd)` writer whose reader has gone ends on SIGPIPE. That is + // the normal end of an unread substitution, not a failure. + if !status.success() && !ended_by_sigpipe(&status) { eprintln!( "Process substitution command failed: {} (exit code: {})", self.command, @@ -133,6 +151,40 @@ impl ProcessSubstitution { } } +impl ProcessSubstitution { + /// Open the end of the FIFO that the child waits for, without blocking, + /// and close it again at once. An error only means that the child has not + /// reached its own `open` yet (no reader for a `>(cmd)` FIFO); the caller + /// tries again on its next turn. + #[cfg(unix)] + fn release_waiting_child(&self) { + use std::os::unix::fs::OpenOptionsExt; + let mut options = std::fs::OpenOptions::new(); + match self.sub_type { + // The child writes, so the shell opens the read end. + ProcessSubType::Input => options.read(true), + // The child reads, so the shell opens the write end. + ProcessSubType::Output => options.write(true), + }; + let _ = options.custom_flags(libc::O_NONBLOCK).open(&self.fifo_path); + } + + #[cfg(not(unix))] + fn release_waiting_child(&self) {} +} + +/// True when the process ended on SIGPIPE. +#[cfg(unix)] +fn ended_by_sigpipe(status: &std::process::ExitStatus) -> bool { + use std::os::unix::process::ExitStatusExt; + status.signal() == Some(libc::SIGPIPE) +} + +#[cfg(not(unix))] +fn ended_by_sigpipe(_status: &std::process::ExitStatus) -> bool { + false +} + /// Safety net: remove FIFO on drop if cleanup() was never called impl Drop for ProcessSubstitution { fn drop(&mut self) { @@ -316,4 +368,79 @@ mod tests { assert!(child.wait().unwrap().success()); assert_eq!(std::fs::read_to_string(©).unwrap(), "12345\n"); } + + /// Run `f` on another thread. True when it finishes within `secs`. + /// A cleanup that waits forever fails the test instead of hanging the suite. + fn finishes_within(secs: u64, f: impl FnOnce() + Send + 'static) -> bool { + let (tx, rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + f(); + let _ = tx.send(()); + }); + rx.recv_timeout(std::time::Duration::from_secs(secs)) + .is_ok() + } + + fn started(sub_type: ProcessSubType, cmd: &str) -> (TempDir, ProcessSubstitution) { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let mut sub = ProcessSubstitution::create(sub_type, cmd.to_string(), &mut state).unwrap(); + sub.start(&mut state).unwrap(); + (temp_dir, sub) + } + + #[test] + fn cleanup_releases_an_input_substitution_that_no_command_opened() { + // `echo <(pwd)` prints the path and never opens it. + let (_dir, sub) = started(ProcessSubType::Input, "pwd"); + let path = sub.fifo_path.clone(); + assert!( + finishes_within(10, move || sub.cleanup().unwrap()), + "cleanup waited forever for an unopened <(pwd)" + ); + assert!(!path.exists()); + } + + #[test] + fn cleanup_releases_an_output_substitution_that_no_command_opened() { + // `echo >(cat)` prints the path and never writes to it. + let (_dir, sub) = started(ProcessSubType::Output, "cat"); + let path = sub.fifo_path.clone(); + assert!( + finishes_within(10, move || sub.cleanup().unwrap()), + "cleanup waited forever for an unopened >(cat)" + ); + assert!(!path.exists()); + } + + #[test] + fn a_connected_input_substitution_still_delivers_its_output() { + // `cat <(cat FILE)`: the reader gets everything, then cleanup returns. + // (`echo` is a builtin, which a substitution does not run.) + let src = TempDir::new().unwrap(); + let file = src.path().join("in.txt"); + std::fs::write(&file, "hi\n").unwrap(); + let (_dir, sub) = started(ProcessSubType::Input, &format!("cat {}", file.display())); + let text = std::fs::read_to_string(&sub.fifo_path).unwrap(); + assert_eq!(text, "hi\n"); + assert!(finishes_within(10, move || sub.cleanup().unwrap())); + } + + #[test] + fn a_connected_output_substitution_finishes_its_work_before_cleanup_returns() { + // `echo data > >(tee FILE)`: cleanup waits until tee has written FILE. + let dir = TempDir::new().unwrap(); + let out = dir.path().join("out.txt"); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let mut sub = ProcessSubstitution::create( + ProcessSubType::Output, + format!("tee {}", out.display()), + &mut state, + ) + .unwrap(); + sub.start(&mut state).unwrap(); + std::fs::write(&sub.fifo_path, "data\n").unwrap(); + assert!(finishes_within(10, move || sub.cleanup().unwrap())); + assert_eq!(std::fs::read_to_string(&out).unwrap(), "data\n"); + } } From 8b7ff4fe19f3bfef0b91dcd1b43a3df7cba71117 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:42:29 +0000 Subject: [PATCH 14/21] fix(grade): a meter cannot start above its limit CodeRabbit (PR 242, grade.rs line 265, Minor): Meter::starting_at stored any spent grade, so a meter could break its own invariant level() <= limit() before any charge, for example a hand-built GlobUsage passed to expand_glob_continuing. starting_at now returns Result and refuses a spent grade over the limit, with the existing Exceeded error. A start exactly at the limit, and every valid resumed grade, are unchanged. The glob walker passes the refusal up as the expansion's error. Tests: a_meter_cannot_start_above_its_limit (written first; it did not compile against the old signature, which could not refuse), and a_resumed_usage_over_a_limit_is_refused_before_any_read for the glob path. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/glob.rs | 35 +++++++++++++++++++++++++------- impl/rust-cli/src/grade.rs | 41 ++++++++++++++++++++++++++++++++++---- 2 files changed, 65 insertions(+), 11 deletions(-) diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index f1b6bfa0..2a04c7dc 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -308,7 +308,7 @@ pub fn expand_glob_continuing( ) -> Result { let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; - let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); + let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent)?; walker .run() .map_err(|stopped| anyhow::Error::new(*stopped))?; @@ -427,21 +427,23 @@ impl<'a> Walker<'a> { compiled: &'a Compiled, limits: &GlobLimits, spent: GlobUsage, - ) -> Self { - Walker { + ) -> Result { + // A usage already over a limit (a hand-built `GlobUsage`, for example) + // is refused here, so every meter starts within its limit. + Ok(Walker { pattern, base, compiled, - reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads), - matches: Meter::starting_at("matches held", limits.max_matches, spent.matches), + reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads)?, + matches: Meter::starting_at("matches held", limits.max_matches, spent.matches)?, match_bytes: Meter::starting_at( "bytes of matches held", limits.max_match_bytes, spent.match_bytes, - ), + )?, max_listing_bytes: limits.max_listing_bytes, found: BTreeSet::new(), - } + }) } fn usage(&self) -> GlobUsage { @@ -683,6 +685,25 @@ mod walk_tests { assert!(err.downcast_ref::().is_some()); } + #[test] + fn a_resumed_usage_over_a_limit_is_refused_before_any_read() { + // A hand-built usage that is already over a limit cannot start a walk: + // every meter must begin within its limit. + use crate::grade::{Cost, Exceeded, Grade}; + let t = TempDir::new().unwrap(); + let limits = GlobLimits { + max_reads: 2, + ..GlobLimits::DEFAULT + }; + let spent = GlobUsage { + reads: Cost::of(5), + ..GlobUsage::default() + }; + let err = expand_glob_continuing("*", t.path(), &limits, spent).unwrap_err(); + let refused = err.downcast_ref::().expect("an Exceeded refusal"); + assert_eq!((refused.limit, refused.level), (2, 5)); + } + #[test] fn absolute_patterns_give_absolute_paths() { let t = TempDir::new().unwrap(); diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs index 66fc01ec..de539384 100644 --- a/impl/rust-cli/src/grade.rs +++ b/impl/rust-cli/src/grade.rs @@ -258,12 +258,25 @@ impl Meter { /// A meter for `axis` that starts from work already measured as `spent`, /// so that several pieces of work can share one limit. - pub const fn starting_at(axis: &'static str, limit: u64, spent: G) -> Self { - Meter { + /// + /// # Errors + /// Returns [`Exceeded`] when `spent` is already over `limit`. Such a meter + /// would break the invariant `level() <= limit()` before any charge, so it + /// is refused rather than stored. + pub fn starting_at(axis: &'static str, limit: u64, spent: G) -> Result { + let level = spent.level(); + if level > limit { + return Err(Exceeded { + axis, + limit, + level: u128::from(level), + }); + } + Ok(Meter { axis, limit, grade: spent, - } + }) } /// Whether [`Meter::charge`] would accept `amount`, without recording it. @@ -405,9 +418,29 @@ mod tests { assert_eq!(meter.grade().peak(), 10); } + #[test] + fn a_meter_cannot_start_above_its_limit() { + // The invariant level() <= limit() holds from the start: a resumed + // grade over the limit is refused, not stored. + let refused = Meter::::starting_at("steps", 1, Cost::of(2)).unwrap_err(); + assert_eq!( + refused, + Exceeded { + axis: "steps", + limit: 1, + level: 2 + } + ); + let held = Occupancy::ZERO.then(Occupancy::of(5)).released(5); + assert!(Meter::::starting_at("held", 4, held).is_err()); + // Exactly at the limit is a valid start. + let at = Meter::::starting_at("steps", 2, Cost::of(2)).unwrap(); + assert_eq!(at.level(), 2); + } + #[test] fn a_meter_can_start_from_work_already_spent() { - let mut meter: Meter = Meter::starting_at("steps", 10, Cost::of(8)); + let mut meter: Meter = Meter::starting_at("steps", 10, Cost::of(8)).unwrap(); assert!(meter.check(2).is_ok()); assert_eq!(meter.charge(3).unwrap_err().level, 11); assert_eq!( From 6b37edc26dfbfe7b4beb41bf5271a9ea03c648bb Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 15:12:44 +0000 Subject: [PATCH 15/21] fix(glob, process_sub): structured refusal at start; keep a background job's FIFOs CodeRabbit review on 8b7ff4f: - expand_glob_continuing returned a raw grade::Exceeded when a resumed usage was over a limit. word.rs passes on only GlobLimitExceeded, so the refusal was taken as "no match" and the literal word was used. Walker::new now returns GlobLimitExceeded (usage as given, at "."), still before any read. - A background job gets its process substitutions handed to a detached thread that waits for each one without releasing it, then removes the FIFO. The release from 9de7856 could end the writer or remove the FIFO before the job opened it. When no job starts, the normal cleanup runs. - A SIGPIPE end reported by sh as exit code 141 now counts as normal too. Tests: the glob refusal test asserts the structured error; a late reader gets a handed-off substitution's output (fails with the release); the SIGPIPE rule covers the sh exit code. cargo test 921 passed, 0 failed, 14 ignored; fmt, clippy -D warnings and the fuzz check pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/executable.rs | 19 ++++++----- impl/rust-cli/src/glob.rs | 43 ++++++++++++++++++++----- impl/rust-cli/src/process_sub.rs | 54 +++++++++++++++++++++++++++++--- impl/rust-cli/src/word.rs | 21 +++++++++++++ 4 files changed, 116 insertions(+), 21 deletions(-) diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index 80756c3a..60ce06b0 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -515,18 +515,21 @@ impl ExecutableCommand for Command { // Execute main command if *background { // Background execution - let _job_id = external::execute_external_background( + match external::execute_external_background( &expanded_program, expanded_args, redirects, state, - ) - .unwrap_or_else(|e| { - eprintln!("{}: {}", expanded_program, e); - 0 // No job created - }); - - expansion.cleanup(); + ) { + // The job may open a FIFO later: do not release or + // remove one that it has not read yet. + Ok(_job_id) => expansion.cleanup_when_consumed(), + // No job: nothing will open the FIFOs. + Err(e) => { + eprintln!("{}: {}", expanded_program, e); + expansion.cleanup(); + } + } Ok(ExecutionResult::Success) } else { diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 2a04c7dc..4b21aec6 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -308,7 +308,8 @@ pub fn expand_glob_continuing( ) -> Result { let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; - let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent)?; + let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent) + .map_err(|stopped| anyhow::Error::new(*stopped))?; walker .run() .map_err(|stopped| anyhow::Error::new(*stopped))?; @@ -427,20 +428,33 @@ impl<'a> Walker<'a> { compiled: &'a Compiled, limits: &GlobLimits, spent: GlobUsage, - ) -> Result { + ) -> Walk { // A usage already over a limit (a hand-built `GlobUsage`, for example) - // is refused here, so every meter starts within its limit. + // is refused here, before any read, so every meter starts within its + // limit. The refusal is the same structured error as a limit met on + // the walk, stopped at the base directory with the usage given. + let refuse = |exceeded| { + Box::new(GlobLimitExceeded { + pattern: pattern.to_string(), + exceeded, + usage: spent, + at: PathBuf::from("."), + }) + }; Ok(Walker { pattern, base, compiled, - reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads)?, - matches: Meter::starting_at("matches held", limits.max_matches, spent.matches)?, + reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads) + .map_err(refuse)?, + matches: Meter::starting_at("matches held", limits.max_matches, spent.matches) + .map_err(refuse)?, match_bytes: Meter::starting_at( "bytes of matches held", limits.max_match_bytes, spent.match_bytes, - )?, + ) + .map_err(refuse)?, max_listing_bytes: limits.max_listing_bytes, found: BTreeSet::new(), }) @@ -700,8 +714,21 @@ mod walk_tests { ..GlobUsage::default() }; let err = expand_glob_continuing("*", t.path(), &limits, spent).unwrap_err(); - let refused = err.downcast_ref::().expect("an Exceeded refusal"); - assert_eq!((refused.limit, refused.level), (2, 5)); + // The structured glob error, so a caller that passes on only + // `GlobLimitExceeded` does not take the refusal for "no match". + let stopped = err + .downcast_ref::() + .expect("a GlobLimitExceeded refusal"); + assert_eq!( + stopped.exceeded, + Exceeded { + axis: "file-system reads", + limit: 2, + level: 5 + } + ); + assert_eq!(stopped.usage, spent); + assert_eq!(stopped.at, PathBuf::from(".")); } #[test] diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index dbaf4f2e..ea0cf9bb 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -119,14 +119,26 @@ impl ProcessSubstitution { /// its reader leave, and a `>(cmd)` reader sees end of file. A connected /// child is not cut short: the shell keeps waiting until it exits, so a /// `>(cmd)` reader still finishes the work it was given. - pub fn cleanup(mut self) -> Result<()> { - // Wait for background process to finish + pub fn cleanup(self) -> Result<()> { + self.finish(true) + } + + /// Wait for the substitution to end, then remove its FIFO, without + /// releasing the child. For a background job, which may open the FIFO at + /// any later time: a release could end the child or remove the FIFO first. + pub fn cleanup_when_consumed(self) -> Result<()> { + self.finish(false) + } + + fn finish(mut self, release: bool) -> Result<()> { if let Some(mut child) = self.child_process.take() { let status = loop { if let Some(status) = child.try_wait()? { break status; } - self.release_waiting_child(); + if release { + self.release_waiting_child(); + } std::thread::sleep(std::time::Duration::from_millis(10)); }; // A `<(cmd)` writer whose reader has gone ends on SIGPIPE. That is @@ -173,11 +185,12 @@ impl ProcessSubstitution { fn release_waiting_child(&self) {} } -/// True when the process ended on SIGPIPE. +/// True when the process ended on SIGPIPE, either itself or (through `sh`) +/// as exit code 128 + SIGPIPE. #[cfg(unix)] fn ended_by_sigpipe(status: &std::process::ExitStatus) -> bool { use std::os::unix::process::ExitStatusExt; - status.signal() == Some(libc::SIGPIPE) + status.signal() == Some(libc::SIGPIPE) || status.code() == Some(128 + libc::SIGPIPE) } #[cfg(not(unix))] @@ -426,6 +439,37 @@ mod tests { assert!(finishes_within(10, move || sub.cleanup().unwrap())); } + #[cfg(unix)] + #[test] + fn a_sigpipe_end_counts_as_normal_also_through_sh() { + // The substitution runs through `sh`, which reports a child killed by + // SIGPIPE as exit code 128 + SIGPIPE, not as a signal. + use std::os::unix::process::ExitStatusExt; + use std::process::ExitStatus; + assert!(ended_by_sigpipe(&ExitStatus::from_raw(libc::SIGPIPE))); + assert!(ended_by_sigpipe(&ExitStatus::from_raw( + (128 + libc::SIGPIPE) << 8 + ))); + assert!(!ended_by_sigpipe(&ExitStatus::from_raw(1 << 8))); + } + + #[test] + fn a_handed_off_input_substitution_waits_for_a_late_reader() { + // `cat <(cat FILE) &`: the shell hands the substitution off before the + // background job opens the FIFO. The writer must not be released and + // the FIFO must still exist when the job opens it. + let src = TempDir::new().unwrap(); + let file = src.path().join("in.txt"); + std::fs::write(&file, "hi\n").unwrap(); + let (_dir, sub) = started(ProcessSubType::Input, &format!("cat {}", file.display())); + let path = sub.fifo_path.clone(); + let handed_off = std::thread::spawn(move || sub.cleanup_when_consumed().unwrap()); + std::thread::sleep(std::time::Duration::from_millis(200)); + assert_eq!(std::fs::read_to_string(&path).unwrap(), "hi\n"); + assert!(finishes_within(10, move || handed_off.join().unwrap())); + assert!(!path.exists()); + } + #[test] fn a_connected_output_substitution_finishes_its_work_before_cleanup_returns() { // `echo data > >(tee FILE)`: cleanup waits until tee has written FILE. diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 55550aa1..a6497376 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -285,6 +285,27 @@ impl Expansion { } } } + + /// Hand the process substitutions of a background job to a detached + /// thread, which waits for each one to end and then removes its FIFO. + /// The job may open a FIFO at any later time, so nothing is released and + /// the shell does not wait. + // snip: a substitution that the job never opens stays blocked until the + // shell exits, and a FIFO is left in the temp directory when the shell + // exits first (`vsh -c 'cmd <(x) &'`). Release and remove them once the + // job table reaps background jobs. + pub fn cleanup_when_consumed(self) { + if self.process_subs.is_empty() { + return; + } + std::thread::spawn(move || { + for proc_sub in self.process_subs { + if let Err(e) = proc_sub.cleanup_when_consumed() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + }); + } } /// Expand words relative to the current directory under the default limits. From 905833b8262697eafcab5ac1385356b1bf64a5c7 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 20:24:54 +0000 Subject: [PATCH 16/21] feat(job): reap ended background jobs and release their process substitutions A background job's substitutions went to a detached thread that waited without releasing, so one that the job never opened stayed blocked until exit and left a FIFO in /tmp. The job table now keeps them and reaps ended jobs with a non-blocking waitpid at each safe point: before every command, before each REPL prompt (with a Done notice), and in fg. A reaped job's substitutions go through ProcessSubstitution::cleanup(). A job keeps them while it runs, so a late reader still gets its input. The detached thread (cleanup_when_consumed) is removed. At exit, ended jobs are reaped and released. A job that still runs keeps its FIFO, and the exit does not wait for it (marked snip in job.rs). Tests: three JobTable unit tests, three binary tests, and the late-reader unit test now goes through the job table. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0136P7ZpKjBuoCRahUbNPywX --- impl/rust-cli/src/commands.rs | 8 +- impl/rust-cli/src/enhanced_repl.rs | 6 + impl/rust-cli/src/executable.rs | 12 +- impl/rust-cli/src/job.rs | 186 ++++++++++++++++++ impl/rust-cli/src/process_sub.rs | 70 ++++--- impl/rust-cli/src/repl.rs | 6 + impl/rust-cli/src/state.rs | 27 +++ impl/rust-cli/src/word.rs | 21 -- .../tests/background_job_reaper_tests.rs | 173 ++++++++++++++++ 9 files changed, 457 insertions(+), 52 deletions(-) create mode 100644 impl/rust-cli/tests/background_job_reaper_tests.rs diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 02918fca..5c74e1f2 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -1767,7 +1767,13 @@ pub fn fg(state: &mut ShellState, job_spec: Option<&str>) -> Result<()> { job_cmd.trim_end_matches(" &") ); } else { - // Job completed - remove from table + // Job completed: nothing can open its FIFOs now, so release the + // process substitutions it held, then remove it from the table. + for sub in state.jobs.take_process_subs(job_id) { + if let Err(e) = sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } state.jobs.remove_job(job_id); } } diff --git a/impl/rust-cli/src/enhanced_repl.rs b/impl/rust-cli/src/enhanced_repl.rs index 3bf1f8e8..c1999e9d 100644 --- a/impl/rust-cli/src/enhanced_repl.rs +++ b/impl/rust-cli/src/enhanced_repl.rs @@ -291,6 +291,12 @@ pub fn run(state: &mut ShellState) -> Result<()> { let mut accumulated_input = String::new(); loop { + // Report the background jobs that ended since the last prompt. + state.reap_jobs(); + for notice in state.job_notices.drain(..) { + println!("{notice}"); + } + // Build prompt let txn_name = state.active_transaction.as_ref().map(|t| t.name.clone()); let undo_count = state.history.iter().filter(|o| !o.undone).count(); diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index 60ce06b0..3da59eb2 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -72,6 +72,9 @@ pub enum ExecutionResult { impl ExecutableCommand for Command { fn execute(&self, state: &mut ShellState) -> Result { + // Safe point: collect the background jobs that have ended, and + // release their process substitutions, before the next command runs. + state.reap_jobs(); match self { // Filesystem operations (reversible) Command::Mkdir { path, redirects } => { @@ -521,9 +524,12 @@ impl ExecutableCommand for Command { redirects, state, ) { - // The job may open a FIFO later: do not release or - // remove one that it has not read yet. - Ok(_job_id) => expansion.cleanup_when_consumed(), + // The job may open a FIFO later: the job table keeps + // the substitutions and releases them when it reaps + // the job. + Ok(job_id) => state + .jobs + .attach_process_subs(job_id, expansion.process_subs), // No job: nothing will open the FIFOs. Err(e) => { eprintln!("{}: {}", expanded_program, e); diff --git a/impl/rust-cli/src/job.rs b/impl/rust-cli/src/job.rs index 13418723..b1fa1eb8 100644 --- a/impl/rust-cli/src/job.rs +++ b/impl/rust-cli/src/job.rs @@ -11,6 +11,8 @@ //! - Job specifications (%1, %+, %-, %name, %?pattern) //! - Current and previous job markers +use crate::process_sub::ProcessSubstitution; + /// Job state #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum JobState { @@ -45,6 +47,19 @@ pub struct Job { pub state: JobState, /// PIDs in this job (for pipelines) pub pids: Vec, + /// PIDs of this job that have been reaped (ended and collected) + reaped: Vec, +} + +/// A job that has ended, with the process substitutions it still held. +#[derive(Debug)] +pub struct FinishedJob { + /// Job ID + pub id: usize, + /// Command string for display + pub command: String, + /// The substitutions to release now that nothing can open them + pub process_subs: Vec, } /// Job table managing all jobs @@ -54,6 +69,9 @@ pub struct JobTable { next_id: usize, current_job_id: Option, // %+ marker previous_job_id: Option, // %- marker + /// Process substitutions that a job may still open, by job id. They live + /// here, not in [`Job`], because a substitution cannot be cloned. + process_subs: std::collections::HashMap>, } impl JobTable { @@ -64,6 +82,7 @@ impl JobTable { next_id: 1, current_job_id: None, previous_job_id: None, + process_subs: std::collections::HashMap::new(), } } @@ -78,6 +97,7 @@ impl JobTable { command, state: JobState::Running, pids, + reaped: Vec::new(), }; self.jobs.push(job); @@ -135,6 +155,9 @@ impl JobTable { /// Remove job from table pub fn remove_job(&mut self, id: usize) { self.jobs.retain(|j| j.id != id); + // A substitution that is still here belongs to a job that is gone: + // dropping it ends its child and removes its FIFO. + self.process_subs.remove(&id); // Update markers if removed job was current/previous if self.current_job_id == Some(id) { @@ -181,6 +204,81 @@ impl JobTable { lines } + /// Keep the process substitutions of a job until it ends: the job may open + /// a FIFO at any later time, so nothing is released while it runs. + pub fn attach_process_subs(&mut self, id: usize, subs: Vec) { + if subs.is_empty() { + return; + } + self.process_subs.entry(id).or_default().extend(subs); + } + + /// Take the process substitutions of a job, for a caller that has seen the + /// job end (`fg` waits for it itself). + pub fn take_process_subs(&mut self, id: usize) -> Vec { + self.process_subs.remove(&id).unwrap_or_default() + } + + /// The `+` or `-` marker of a job, or a space. + pub fn marker(&self, id: usize) -> char { + if self.current_job_id == Some(id) { + '+' + } else if self.previous_job_id == Some(id) { + '-' + } else { + ' ' + } + } + + /// Reap the running jobs that have ended, without blocking. + /// + /// Each process of a running job is polled with `waitpid(WNOHANG)`. A job + /// whose processes have all ended is marked [`JobState::Done`] and returned + /// once, with the process substitutions it held. The job stays in the table + /// until [`JobTable::remove_job`]. The caller releases the substitutions. + #[cfg(unix)] + pub fn reap_finished(&mut self) -> Vec { + let mut finished = Vec::new(); + for job in &mut self.jobs { + if job.state != JobState::Running { + continue; + } + for &pid in &job.pids { + if job.reaped.contains(&pid) { + continue; + } + let mut status: i32 = 0; + // SAFETY: `status` is a live, exclusively borrowed i32 for the + // whole call; waitpid writes the child's status into it and + // nothing else. WNOHANG makes the call return at once. + let reaped = unsafe { libc::waitpid(pid, &mut status, libc::WNOHANG) }; + // 0: still running. A pid or -1 (no such child: someone else + // already collected it): it has ended. + if reaped != 0 { + job.reaped.push(pid); + } + } + if job.pids.iter().all(|pid| job.reaped.contains(pid)) { + job.state = JobState::Done; + finished.push((job.id, job.command.clone())); + } + } + finished + .into_iter() + .map(|(id, command)| FinishedJob { + id, + command, + process_subs: self.process_subs.remove(&id).unwrap_or_default(), + }) + .collect() + } + + /// Job control needs Unix process groups; nothing is reaped elsewhere. + #[cfg(not(unix))] + pub fn reap_finished(&mut self) -> Vec { + Vec::new() + } + /// Get all jobs pub fn jobs(&self) -> &[Job] { &self.jobs @@ -193,9 +291,97 @@ impl Default for JobTable { } } +/// When the shell exits, release the substitutions of the jobs that have +/// ended. A job that is still running may open its FIFO later, so its +/// substitutions are left alone: dropping one would end its child and remove +/// the FIFO under the job. +// snip: a substitution of a job that still runs at exit is never released. +// Its FIFO stays in the temp directory, and its child stays blocked until the +// job opens the FIFO. The exit does not wait for the job. +impl Drop for JobTable { + fn drop(&mut self) { + for finished in self.reap_finished() { + for sub in finished.process_subs { + if let Err(e) = sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + } + for (_, subs) in self.process_subs.drain() { + for sub in subs { + std::mem::forget(sub); + } + } + } +} + #[cfg(test)] mod tests { use super::*; + use std::process::Command; + use std::time::{Duration, Instant}; + + /// Reap until `id` has ended, for at most ten seconds. + fn reap_until_done(table: &mut JobTable, id: usize) -> Vec { + let started = Instant::now(); + loop { + let finished = table.reap_finished(); + if finished.iter().any(|job| job.id == id) { + return finished; + } + assert!( + started.elapsed() < Duration::from_secs(10), + "job did not end" + ); + std::thread::sleep(Duration::from_millis(10)); + } + } + + #[test] + fn reap_marks_a_finished_job_done_and_returns_it() { + // The reaper under test collects the child, so no handle is kept. + let pid = Command::new("true").spawn().unwrap().id() as i32; + let mut table = JobTable::new(); + let id = table.add_job(pid, "true &".to_string(), vec![pid]); + let finished = reap_until_done(&mut table, id); + assert_eq!(finished.len(), 1); + assert_eq!(finished[0].command, "true &"); + assert_eq!(table.jobs()[0].state, JobState::Done); + // A job is reported once. + assert!(table.reap_finished().is_empty()); + } + + #[test] + fn reap_leaves_a_running_job_alone() { + let mut child = Command::new("sleep").arg("30").spawn().unwrap(); + let pid = child.id() as i32; + let mut table = JobTable::new(); + table.add_job(pid, "sleep 30 &".to_string(), vec![pid]); + assert!(table.reap_finished().is_empty()); + assert_eq!(table.jobs()[0].state, JobState::Running); + child.kill().unwrap(); + child.wait().unwrap(); + } + + #[test] + fn a_pipeline_job_ends_when_all_its_processes_have_ended() { + let quick_pid = Command::new("true").spawn().unwrap().id() as i32; + let mut slow = Command::new("sleep").arg("30").spawn().unwrap(); + let slow_pid = slow.id() as i32; + let mut table = JobTable::new(); + table.add_job( + quick_pid, + "true | sleep 30 &".to_string(), + vec![quick_pid, slow_pid], + ); + std::thread::sleep(Duration::from_millis(200)); + assert!(table.reap_finished().is_empty()); + assert_eq!(table.jobs()[0].state, JobState::Running); + slow.kill().unwrap(); + slow.wait().unwrap(); + let finished = reap_until_done(&mut table, 1); + assert_eq!(finished.len(), 1); + } #[test] fn test_add_job() { diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index ea0cf9bb..f37a25e4 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -23,6 +23,7 @@ use crate::parser::{parse_command, Command, ProcessSubType}; use crate::state::ShellState; /// Process substitution instance +#[derive(Debug)] pub struct ProcessSubstitution { /// Path to the FIFO pub fifo_path: PathBuf, @@ -111,34 +112,22 @@ impl ProcessSubstitution { /// Wait for background process and cleanup. /// /// Callers run this after the last command that could open the FIFO has - /// finished. A child that no command opened is still blocked in its - /// `open(2)` of the FIFO, waiting for a partner, so a plain `wait` would - /// never return (`echo <(pwd)` prints the path and opens nothing). While - /// the child runs, the shell therefore opens the end that the child waits - /// for, without blocking, and closes it at once: a `<(cmd)` writer sees - /// its reader leave, and a `>(cmd)` reader sees end of file. A connected - /// child is not cut short: the shell keeps waiting until it exits, so a + /// finished (for a background job, when the job has been reaped). A child + /// that no command opened is still blocked in its `open(2)` of the FIFO, + /// waiting for a partner, so a plain `wait` would never return + /// (`echo <(pwd)` prints the path and opens nothing). While the child + /// runs, the shell therefore opens the end that the child waits for, + /// without blocking, and closes it at once: a `<(cmd)` writer sees its + /// reader leave, and a `>(cmd)` reader sees end of file. A connected child + /// is not cut short: the shell keeps waiting until it exits, so a /// `>(cmd)` reader still finishes the work it was given. - pub fn cleanup(self) -> Result<()> { - self.finish(true) - } - - /// Wait for the substitution to end, then remove its FIFO, without - /// releasing the child. For a background job, which may open the FIFO at - /// any later time: a release could end the child or remove the FIFO first. - pub fn cleanup_when_consumed(self) -> Result<()> { - self.finish(false) - } - - fn finish(mut self, release: bool) -> Result<()> { + pub fn cleanup(mut self) -> Result<()> { if let Some(mut child) = self.child_process.take() { let status = loop { if let Some(status) = child.try_wait()? { break status; } - if release { - self.release_waiting_child(); - } + self.release_waiting_child(); std::thread::sleep(std::time::Duration::from_millis(10)); }; // A `<(cmd)` writer whose reader has gone ends on SIGPIPE. That is @@ -455,18 +444,45 @@ mod tests { #[test] fn a_handed_off_input_substitution_waits_for_a_late_reader() { - // `cat <(cat FILE) &`: the shell hands the substitution off before the - // background job opens the FIFO. The writer must not be released and - // the FIFO must still exist when the job opens it. + // `cat <(cat FILE) &`: the shell hands the substitution to the job + // table before the background job opens the FIFO. The writer must not + // be released while the job runs, and the FIFO must still exist when + // the job opens it. Cleanup follows the reaping of the job. + use crate::job::JobTable; let src = TempDir::new().unwrap(); let file = src.path().join("in.txt"); std::fs::write(&file, "hi\n").unwrap(); let (_dir, sub) = started(ProcessSubType::Input, &format!("cat {}", file.display())); let path = sub.fifo_path.clone(); - let handed_off = std::thread::spawn(move || sub.cleanup_when_consumed().unwrap()); + + // The job table under test reaps this child, so no handle is kept. + let pid = std::process::Command::new("sleep") + .arg("1") + .spawn() + .unwrap() + .id() as i32; + let mut jobs = JobTable::new(); + let id = jobs.add_job(pid, "sleep 1 &".to_string(), vec![pid]); + jobs.attach_process_subs(id, vec![sub]); + + assert!(jobs.reap_finished().is_empty(), "the job still runs"); std::thread::sleep(std::time::Duration::from_millis(200)); assert_eq!(std::fs::read_to_string(&path).unwrap(), "hi\n"); - assert!(finishes_within(10, move || handed_off.join().unwrap())); + + let finished = loop { + let finished = jobs.reap_finished(); + if !finished.is_empty() { + break finished; + } + std::thread::sleep(std::time::Duration::from_millis(20)); + }; + assert!(finishes_within(10, move || { + for job in finished { + for sub in job.process_subs { + sub.cleanup().unwrap(); + } + } + })); assert!(!path.exists()); } diff --git a/impl/rust-cli/src/repl.rs b/impl/rust-cli/src/repl.rs index c861e84e..dc8e1d00 100644 --- a/impl/rust-cli/src/repl.rs +++ b/impl/rust-cli/src/repl.rs @@ -55,6 +55,12 @@ pub fn run(state: &mut ShellState) -> Result<()> { let mut stdout = io::stdout(); loop { + // Report the background jobs that ended since the last prompt. + state.reap_jobs(); + for notice in state.job_notices.drain(..) { + println!("{notice}"); + } + // Build prompt let prompt = build_prompt(state); print!("{}", prompt); diff --git a/impl/rust-cli/src/state.rs b/impl/rust-cli/src/state.rs index 7f75389b..88e4d7ab 100644 --- a/impl/rust-cli/src/state.rs +++ b/impl/rust-cli/src/state.rs @@ -311,6 +311,10 @@ pub struct ShellState { /// Job table for background job management pub jobs: crate::job::JobTable, + /// Messages about background jobs that have ended (`[1]+ Done cmd`), + /// printed by the interactive shell before its next prompt. + pub job_notices: Vec, + /// Named checkpoints mapping name → history index at checkpoint time pub checkpoints: HashMap)>, @@ -325,6 +329,28 @@ pub struct ShellState { } impl ShellState { + /// Reap the background jobs that have ended, release their process + /// substitutions and queue a `Done` notice for each in + /// [`ShellState::job_notices`]. Cheap: one non-blocking `waitpid` per + /// running process. Called at the safe points between commands. + pub fn reap_jobs(&mut self) { + for job in self.jobs.reap_finished() { + for sub in job.process_subs { + if let Err(e) = sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + self.job_notices.push(format!( + "[{}]{} {:<20} {}", + job.id, + self.jobs.marker(job.id), + "Done", + job.command.trim_end_matches(" &") + )); + self.jobs.remove_job(job.id); + } + } + /// Create new shell state with given root pub fn new(root: &str) -> Result { let root_path = PathBuf::from(root); @@ -354,6 +380,7 @@ impl ShellState { positional_params: Vec::new(), fifo_counter: std::sync::atomic::AtomicUsize::new(0), jobs: crate::job::JobTable::new(), + job_notices: Vec::new(), checkpoints: HashMap::new(), functions: FunctionTable::new(), traps: TrapTable::new(), diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index a6497376..55550aa1 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -285,27 +285,6 @@ impl Expansion { } } } - - /// Hand the process substitutions of a background job to a detached - /// thread, which waits for each one to end and then removes its FIFO. - /// The job may open a FIFO at any later time, so nothing is released and - /// the shell does not wait. - // snip: a substitution that the job never opens stays blocked until the - // shell exits, and a FIFO is left in the temp directory when the shell - // exits first (`vsh -c 'cmd <(x) &'`). Release and remove them once the - // job table reaps background jobs. - pub fn cleanup_when_consumed(self) { - if self.process_subs.is_empty() { - return; - } - std::thread::spawn(move || { - for proc_sub in self.process_subs { - if let Err(e) = proc_sub.cleanup_when_consumed() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } - }); - } } /// Expand words relative to the current directory under the default limits. diff --git a/impl/rust-cli/tests/background_job_reaper_tests.rs b/impl/rust-cli/tests/background_job_reaper_tests.rs new file mode 100644 index 00000000..9e5e70fb --- /dev/null +++ b/impl/rust-cli/tests/background_job_reaper_tests.rs @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! A background job that ends is reaped, and its process substitutions are +//! released and removed. Each test runs the `vsh` binary with `-c`. +//! +//! A hung test would leave `sh` children that hold pipes, so every run has a +//! time limit and kills the process group of the shell when it passes. + +#![cfg(unix)] + +use std::path::Path; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +use tempfile::TempDir; + +struct Run { + pid: u32, + stdout: String, + success: bool, + _guard: Cleanup, +} + +/// Kills the substitution children and removes the FIFOs of one shell when +/// the test ends, whether it passed or not. +struct Cleanup(u32); + +impl Drop for Cleanup { + fn drop(&mut self) { + leftovers_cleanup(self.0); + } +} + +/// Run `vsh -c script` in `dir`; fail the test if it does not end in `secs`. +fn vsh(dir: &Path, script: &str, secs: u64) -> Run { + // Stdout goes to a file, not a pipe: a substitution child that blocks + // inherits the descriptor, and a pipe reader would wait for it forever. + let out_path = dir.join(".stdout.txt"); + let mut child = Command::new(env!("CARGO_BIN_EXE_vsh")) + .arg("-c") + .arg(script) + .current_dir(dir) + .stdout(Stdio::from(std::fs::File::create(&out_path).unwrap())) + .stderr(Stdio::null()) + .spawn() + .expect("vsh runs"); + let pid = child.id(); + let guard = Cleanup(pid); + let started = Instant::now(); + loop { + if child.try_wait().unwrap().is_some() { + break; + } + if started.elapsed() > Duration::from_secs(secs) { + let _ = child.kill(); + let _ = child.wait(); + panic!("`{script}` did not finish within {secs}s"); + } + std::thread::sleep(Duration::from_millis(20)); + } + let status = child.wait().unwrap(); + Run { + pid, + stdout: std::fs::read_to_string(&out_path).unwrap_or_default(), + success: status.success(), + _guard: guard, + } +} + +/// FIFOs that the shell with this pid created. +fn fifos_of(pid: u32) -> Vec { + let prefix = format!("vsh-fifo-{pid}-"); + std::fs::read_dir("/tmp") + .unwrap() + .filter_map(|entry| entry.ok()) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.starts_with(&prefix)) + .collect() +} + +/// Processes whose command line names a FIFO of the shell with this pid. +fn substitution_children_of(pid: u32) -> Vec { + let needle = format!("vsh-fifo-{pid}-"); + let mut found = Vec::new(); + for entry in std::fs::read_dir("/proc").unwrap().flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + let Ok(proc_pid) = name.parse::() else { + continue; + }; + if let Ok(cmdline) = std::fs::read(entry.path().join("cmdline")) { + if String::from_utf8_lossy(&cmdline).contains(&needle) { + found.push(proc_pid); + } + } + } + found +} + +fn leftovers_cleanup(pid: u32) { + for child in substitution_children_of(pid) { + let _ = Command::new("kill") + .arg("-9") + .arg(child.to_string()) + .status(); + } + for name in fifos_of(pid) { + let _ = std::fs::remove_file(Path::new("/tmp").join(name)); + } +} + +fn data_file(dir: &Path) { + std::fs::write(dir.join("f.txt"), "data\n").unwrap(); +} + +#[test] +fn an_unopened_substitution_of_an_ended_job_is_released_before_the_next_command() { + // `/bin/true` never opens the FIFO. The job ends at once; the shell must + // reap it at the next command and release and remove the substitution, + // not at exit. + let dir = TempDir::new().unwrap(); + data_file(dir.path()); + let run = vsh( + dir.path(), + "/bin/true <(cat f.txt) &\nsleep 0.3\n/bin/true\n/bin/ls /tmp", + 20, + ); + assert!(run.success); + let listed: Vec<&str> = run.stdout.lines().collect(); + let prefix = format!("vsh-fifo-{}-", run.pid); + assert!( + !listed.iter().any(|name| name.starts_with(&prefix)), + "the FIFO was still there at the third command: {listed:?}" + ); + assert!(fifos_of(run.pid).is_empty(), "a FIFO is left after exit"); + assert!( + substitution_children_of(run.pid).is_empty(), + "the substitution child is still running" + ); +} + +#[test] +fn a_late_reader_still_gets_its_input() { + // The job opens the FIFO a second after it starts. The shell must keep + // the substitution until the job ends. + let dir = TempDir::new().unwrap(); + let run = vsh( + dir.path(), + "sh -c 'sleep 1; cat \"$1\"' sh <(pwd) > out &\nsleep 2\n/bin/true", + 20, + ); + assert!(run.success); + let out = std::fs::read_to_string(dir.path().join("out")).unwrap(); + let name = dir + .path() + .file_name() + .unwrap() + .to_string_lossy() + .into_owned(); + assert!(out.trim().ends_with(&name), "out holds {out:?}"); + assert!(fifos_of(run.pid).is_empty(), "a FIFO is left after exit"); +} + +#[test] +fn a_job_that_ends_is_not_listed_again() { + let dir = TempDir::new().unwrap(); + let run = vsh(dir.path(), "/bin/true &\nsleep 0.3\n/bin/true\njobs", 20); + assert!(run.success); + assert!( + !run.stdout.contains("Running"), + "an ended job is still listed as running: {}", + run.stdout + ); +} From 1d2c4a103ca3ec276966890cfa919b8437dad927 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 20:29:25 +0000 Subject: [PATCH 17/21] fix(job): hand a running job's substitutions over without mem::forget Hypatia flagged std::mem::forget in JobTable::drop (mem_forget, CWE-401, high). ProcessSubstitution::leave_running now drops the Child handle (which neither kills nor waits) and marks the substitution as not owned, so its Drop keeps the FIFO and the child. Behaviour at shell exit is unchanged: a job that still runs can still open its FIFO. Test written first: a_substitution_left_running_keeps_its_fifo_and_its_child (failed with a plain drop). cargo test 928 passed, 0 failed, 14 ignored; fmt, clippy -D warnings and the fuzz check pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/job.rs | 2 +- impl/rust-cli/src/process_sub.rs | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/impl/rust-cli/src/job.rs b/impl/rust-cli/src/job.rs index b1fa1eb8..0823f6d9 100644 --- a/impl/rust-cli/src/job.rs +++ b/impl/rust-cli/src/job.rs @@ -309,7 +309,7 @@ impl Drop for JobTable { } for (_, subs) in self.process_subs.drain() { for sub in subs { - std::mem::forget(sub); + sub.leave_running(); } } } diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index f37a25e4..c3238aa7 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -33,6 +33,9 @@ pub struct ProcessSubstitution { pub sub_type: ProcessSubType, /// Background process handle pub child_process: Option, + /// False once the substitution is left to a job that still runs: drop + /// then neither ends the child nor removes the FIFO. + owned: bool, } impl ProcessSubstitution { @@ -85,6 +88,7 @@ impl ProcessSubstitution { command: cmd, sub_type, child_process: None, + owned: true, }) } @@ -153,6 +157,14 @@ impl ProcessSubstitution { } impl ProcessSubstitution { + /// Leave the substitution to a job that still runs when the shell exits. + /// The child keeps running and the FIFO stays, so the job can still open + /// it. Dropping a `Child` handle neither kills nor waits for the process. + pub fn leave_running(mut self) { + self.child_process = None; + self.owned = false; + } + /// Open the end of the FIFO that the child waits for, without blocking, /// and close it again at once. An error only means that the child has not /// reached its own `open` yet (no reader for a `>(cmd)` FIFO); the caller @@ -190,6 +202,9 @@ fn ended_by_sigpipe(_status: &std::process::ExitStatus) -> bool { /// Safety net: remove FIFO on drop if cleanup() was never called impl Drop for ProcessSubstitution { fn drop(&mut self) { + if !self.owned { + return; + } // Kill lingering child process if let Some(ref mut child) = self.child_process { let _ = child.kill(); @@ -442,6 +457,21 @@ mod tests { assert!(!ended_by_sigpipe(&ExitStatus::from_raw(1 << 8))); } + #[test] + fn a_substitution_left_running_keeps_its_fifo_and_its_child() { + // At shell exit a job that still runs may open the FIFO later. + let src = TempDir::new().unwrap(); + let file = src.path().join("in.txt"); + std::fs::write(&file, "hi\n").unwrap(); + let (_dir, sub) = started(ProcessSubType::Input, &format!("cat {}", file.display())); + let path = sub.fifo_path.clone(); + sub.leave_running(); + assert!(path.exists()); + // The child was not killed: a late reader still gets its output. + assert_eq!(std::fs::read_to_string(&path).unwrap(), "hi\n"); + std::fs::remove_file(&path).unwrap(); + } + #[test] fn a_handed_off_input_substitution_waits_for_a_late_reader() { // `cat <(cat FILE) &`: the shell hands the substitution to the job From 1e6ba50ee38cfe83cf1a534042544408eb7688eb Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 21:14:26 +0000 Subject: [PATCH 18/21] feat(job): SIGCHLD flag gates the reaper; clean ended jobs before exit Owner ruling AD1: a SIGCHLD handler that only sets a flag, which the reaper checks first, with polling at safe points as the fallback. - job.rs: on_sigchld stores to a lock-free AtomicBool (async-signal-safe: no allocation, lock, I/O or errno change). install_sigchld_handler uses sigaction with SA_RESTART. reap_finished returns at once when the handler is installed and the flag is clear; it clears the flag before the poll, so a signal during the poll is seen at the next safe point. Without the handler it polls every time. - main.rs: install the handler at start (a failure only warns). Drop the job table before std::process::exit, which runs no destructors: ended jobs of a -c or script run were never cleaned at exit. Tests written first: the handler sets the flag (failed with a stub); a raw read(2) interrupted by SIGCHLD is restarted (failed with EINTR when SA_RESTART was left out); a script's read builtin gets its line while a job ends during the read (a guard); a job that ended before exit leaves no FIFO (failed: FIFO left). cargo test 932 passed, 0 failed, 14 ignored; fmt, clippy -D warnings and the fuzz check pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/job.rs | 117 ++++++++++++++++++ impl/rust-cli/src/main.rs | 10 ++ .../tests/background_job_reaper_tests.rs | 51 ++++++++ 3 files changed, 178 insertions(+) diff --git a/impl/rust-cli/src/job.rs b/impl/rust-cli/src/job.rs index 0823f6d9..740a20b4 100644 --- a/impl/rust-cli/src/job.rs +++ b/impl/rust-cli/src/job.rs @@ -12,6 +12,8 @@ //! - Current and previous job markers use crate::process_sub::ProcessSubstitution; +#[cfg(unix)] +use std::sync::atomic::{AtomicBool, Ordering}; /// Job state #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -238,6 +240,14 @@ impl JobTable { /// until [`JobTable::remove_job`]. The caller releases the substitutions. #[cfg(unix)] pub fn reap_finished(&mut self) -> Vec { + // With the handler installed, a clear flag means that no child has + // changed state since the last poll, so there is nothing to collect. + // The flag is cleared before the poll: a SIGCHLD that arrives during + // the poll sets it again for the next safe point. Without the + // handler, poll every time. + if SIGCHLD_HANDLER.load(Ordering::SeqCst) && !take_sigchld() { + return Vec::new(); + } let mut finished = Vec::new(); for job in &mut self.jobs { if job.state != JobState::Running { @@ -315,6 +325,55 @@ impl Drop for JobTable { } } +/// Set by the SIGCHLD handler: a child process has changed state since the +/// reaper last looked. It starts set, so the first safe point polls. +#[cfg(unix)] +static SIGCHLD_SEEN: AtomicBool = AtomicBool::new(true); + +/// True once the SIGCHLD handler is installed. Until then the reaper polls at +/// every safe point. +#[cfg(unix)] +static SIGCHLD_HANDLER: AtomicBool = AtomicBool::new(false); + +/// The SIGCHLD handler. It only stores to a lock-free atomic, which is +/// async-signal-safe: no allocation, no lock, no I/O and no change to errno. +#[cfg(unix)] +extern "C" fn on_sigchld(_signal: libc::c_int) { + SIGCHLD_SEEN.store(true, Ordering::SeqCst); +} + +/// Install the SIGCHLD handler, so that the reaper polls only after a child +/// has changed state. `SA_RESTART` makes the kernel restart a read that the +/// signal interrupts. Without the handler the reaper still polls at every safe +/// point, so a failure here costs only the saved polls. +#[cfg(unix)] +pub fn install_sigchld_handler() -> std::io::Result<()> { + let handler: extern "C" fn(libc::c_int) = on_sigchld; + // SAFETY: an all-zero `sigaction` is a valid value (empty mask, no + // flags); the fields set below make it a plain handler. sigemptyset and + // sigaction get pointers to this live, exclusively borrowed value, and + // the old action is not requested (null). The handler is + // async-signal-safe (see `on_sigchld`). + let result = unsafe { + let mut action: libc::sigaction = std::mem::zeroed(); + action.sa_sigaction = handler as libc::sighandler_t; + action.sa_flags = libc::SA_RESTART; + libc::sigemptyset(&mut action.sa_mask); + libc::sigaction(libc::SIGCHLD, &action, std::ptr::null_mut()) + }; + if result != 0 { + return Err(std::io::Error::last_os_error()); + } + SIGCHLD_HANDLER.store(true, Ordering::SeqCst); + Ok(()) +} + +/// Take the SIGCHLD flag: true if a child changed state since the last take. +#[cfg(unix)] +fn take_sigchld() -> bool { + SIGCHLD_SEEN.swap(false, Ordering::SeqCst) +} + #[cfg(test)] mod tests { use super::*; @@ -475,4 +534,62 @@ mod tests { assert!(lines[1].contains("[2]")); assert!(lines[1].contains("+")); // Current job marker } + + #[cfg(unix)] + #[test] + fn the_sigchld_handler_sets_the_flag() { + install_sigchld_handler().unwrap(); + take_sigchld(); + // SAFETY: getpid and kill take plain integers; the signal goes to + // this process, whose SIGCHLD handler only stores to an atomic. + unsafe { + libc::kill(libc::getpid(), libc::SIGCHLD); + } + let started = Instant::now(); + while !take_sigchld() { + assert!(started.elapsed() < Duration::from_secs(5), "flag not set"); + std::thread::sleep(Duration::from_millis(5)); + } + } + + #[cfg(unix)] + #[test] + fn a_read_interrupted_by_sigchld_is_restarted() { + // A raw read(2) is not retried by std: without SA_RESTART it returns + // EINTR when SIGCHLD arrives during it. + install_sigchld_handler().unwrap(); + let mut fds = [0; 2]; + // SAFETY: `fds` is a live array of two c_ints for pipe(2) to fill. + assert_eq!(unsafe { libc::pipe(fds.as_mut_ptr()) }, 0); + let (read_fd, write_fd) = (fds[0], fds[1]); + let (tx, rx) = std::sync::mpsc::channel(); + let reader = std::thread::spawn(move || { + // SAFETY: pthread_self takes no arguments. + tx.send(unsafe { libc::pthread_self() }).unwrap(); + let mut byte = 0u8; + // SAFETY: `byte` is a live, exclusively borrowed u8 and the length + // is 1; read_fd is the open read end of the pipe. + let n = unsafe { libc::read(read_fd, (&mut byte as *mut u8).cast(), 1) }; + let errno = std::io::Error::last_os_error(); + (n, byte, errno) + }); + let thread = rx.recv().unwrap(); + std::thread::sleep(Duration::from_millis(100)); + // SAFETY: `thread` is the live reader thread, blocked in read(2). + assert_eq!(unsafe { libc::pthread_kill(thread, libc::SIGCHLD) }, 0); + std::thread::sleep(Duration::from_millis(100)); + // SAFETY: write_fd is the open write end; one byte from a live u8. + assert_eq!( + unsafe { libc::write(write_fd, (&b'x' as *const u8).cast(), 1) }, + 1 + ); + let (n, byte, errno) = reader.join().unwrap(); + assert_eq!(n, 1, "read was interrupted: {errno}"); + assert_eq!(byte, b'x'); + // SAFETY: both descriptors are open and owned by this test. + unsafe { + libc::close(read_fd); + libc::close(write_fd); + } + } } diff --git a/impl/rust-cli/src/main.rs b/impl/rust-cli/src/main.rs index ebb288ac..fae9b8ba 100644 --- a/impl/rust-cli/src/main.rs +++ b/impl/rust-cli/src/main.rs @@ -149,6 +149,13 @@ fn main() -> Result<()> { let mut state = state::ShellState::new(&root)?; + // Reap background jobs when SIGCHLD says a child has ended. If the + // handler cannot be installed, the shell polls at every safe point. + #[cfg(unix)] + if let Err(e) = vsh::job::install_sigchld_handler() { + eprintln!("vsh: warning: could not install SIGCHLD handler ({e})"); + } + // Mode 1: Execute inline command string (-c "command") if let Some(ref cmd_string) = cli.command_string { // Set positional parameters: $0 = "vsh", $1.. = script_args @@ -304,6 +311,9 @@ fn execute_script_content(content: &str, state: &mut state::ShellState) -> Resul } } + // process::exit runs no destructors: drop the job table first, so the + // substitutions of ended jobs are released and running jobs keep theirs. + drop(std::mem::take(&mut state.jobs)); std::process::exit(state.last_exit_code); } diff --git a/impl/rust-cli/tests/background_job_reaper_tests.rs b/impl/rust-cli/tests/background_job_reaper_tests.rs index 9e5e70fb..0395d865 100644 --- a/impl/rust-cli/tests/background_job_reaper_tests.rs +++ b/impl/rust-cli/tests/background_job_reaper_tests.rs @@ -171,3 +171,54 @@ fn a_job_that_ends_is_not_listed_again() { run.stdout ); } + +#[test] +fn a_read_still_gets_its_input_when_a_job_ends_during_it() { + // The background job ends while `read` waits on stdin, so SIGCHLD + // arrives during the read. The read must restart and get the line. + use std::io::Write; + let dir = TempDir::new().unwrap(); + let out_path = dir.path().join(".stdout.txt"); + let mut child = Command::new(env!("CARGO_BIN_EXE_vsh")) + .arg("-c") + .arg("sleep 0.1 &\nread x\necho \"got:$x\"") + .current_dir(dir.path()) + .stdin(Stdio::piped()) + .stdout(Stdio::from(std::fs::File::create(&out_path).unwrap())) + .stderr(Stdio::null()) + .spawn() + .expect("vsh runs"); + let _guard = Cleanup(child.id()); + std::thread::sleep(Duration::from_millis(600)); + let mut stdin = child.stdin.take().unwrap(); + stdin.write_all(b"hello\n").unwrap(); + drop(stdin); + let started = Instant::now(); + while child.try_wait().unwrap().is_none() { + if started.elapsed() > Duration::from_secs(10) { + let _ = child.kill(); + let _ = child.wait(); + panic!("vsh did not finish"); + } + std::thread::sleep(Duration::from_millis(20)); + } + let stdout = std::fs::read_to_string(&out_path).unwrap(); + assert!(stdout.contains("got:hello"), "stdout: {stdout:?}"); +} + +#[test] +fn a_job_that_ended_before_the_shell_exits_is_cleaned_at_exit() { + // The job ends during the last command, so no later command reaps it: + // the exit must release its substitution and remove the FIFO. + let dir = TempDir::new().unwrap(); + let run = vsh( + dir.path(), + "sh -c 'sleep 0.3; cat \"$1\"' sh <(pwd) > late.out &\nsleep 0.8", + 10, + ); + assert!(run.success); + assert!(!std::fs::read_to_string(dir.path().join("late.out")) + .unwrap() + .is_empty()); + assert_eq!(fifos_of(run.pid), Vec::::new()); +} From 3e53e52289102368f65f04db985a9499e827042b Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 21:19:08 +0000 Subject: [PATCH 19/21] fix(job): reap killed jobs, never block on a working substitution, no notices in scripts CodeRabbit review on 905833b: - A job that kill marked Done was never polled, so its substitutions were kept until exit. The reaper now polls every job it has not reported yet, whatever its display state, and reports each job once. - A finished job's substitutions were cleaned with the blocking cleanup(), so a connected >(cmd) that worked on made the next command and the exit wait for it. They now go to a finishing list that is checked at each safe point with ProcessSubstitution::try_finish (one release, one try_wait, no wait). fg uses the same path. At exit a substitution that still works is left to finish, like one of a job that still runs (issue 245). - Done notices are queued only in an interactive shell; a script or -c run kept every notice. - The background-job binary tests read /proc, so they run on Linux only. - The SIGCHLD handler tests move to their own test binary: they install a process-wide handler and read a global flag. Tests written first and seen failing: a job marked Done before it was reaped is still reaped; a script queues no notices; a connected substitution of an ended job blocks no command (the shell took 2.0 s, now well under 1.5 s, and the substitution still writes its file). cargo test 935 passed, 0 failed, 14 ignored; fmt, clippy -D warnings and the fuzz check pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/src/commands.rs | 7 +- impl/rust-cli/src/enhanced_repl.rs | 3 + impl/rust-cli/src/job.rs | 123 +++++++++--------- impl/rust-cli/src/process_sub.rs | 23 +++- impl/rust-cli/src/repl.rs | 3 + impl/rust-cli/src/state.rs | 27 ++-- .../tests/background_job_reaper_tests.rs | 28 +++- impl/rust-cli/tests/sigchld_handler_tests.rs | 64 +++++++++ 8 files changed, 196 insertions(+), 82 deletions(-) create mode 100644 impl/rust-cli/tests/sigchld_handler_tests.rs diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 5c74e1f2..fbc5b283 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -1769,11 +1769,8 @@ pub fn fg(state: &mut ShellState, job_spec: Option<&str>) -> Result<()> { } else { // Job completed: nothing can open its FIFOs now, so release the // process substitutions it held, then remove it from the table. - for sub in state.jobs.take_process_subs(job_id) { - if let Err(e) = sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + let subs = state.jobs.take_process_subs(job_id); + state.jobs.finish_process_subs(subs); state.jobs.remove_job(job_id); } } diff --git a/impl/rust-cli/src/enhanced_repl.rs b/impl/rust-cli/src/enhanced_repl.rs index c1999e9d..049d4265 100644 --- a/impl/rust-cli/src/enhanced_repl.rs +++ b/impl/rust-cli/src/enhanced_repl.rs @@ -241,6 +241,9 @@ impl Prompt for VshPrompt { /// Run enhanced REPL pub fn run(state: &mut ShellState) -> Result<()> { + // Job notices have a reader here: they print before each prompt. + state.interactive = true; + // Set up history file let history_file = dirs::home_dir() .unwrap_or_else(|| PathBuf::from("/tmp")) diff --git a/impl/rust-cli/src/job.rs b/impl/rust-cli/src/job.rs index 740a20b4..acd57d8c 100644 --- a/impl/rust-cli/src/job.rs +++ b/impl/rust-cli/src/job.rs @@ -51,6 +51,8 @@ pub struct Job { pub pids: Vec, /// PIDs of this job that have been reaped (ended and collected) reaped: Vec, + /// True once the reaper has returned this job as finished + reported: bool, } /// A job that has ended, with the process substitutions it still held. @@ -74,6 +76,9 @@ pub struct JobTable { /// Process substitutions that a job may still open, by job id. They live /// here, not in [`Job`], because a substitution cannot be cloned. process_subs: std::collections::HashMap>, + /// Substitutions of jobs that have ended, whose children have not exited + /// yet. A connected `>(cmd)` can work on after its writer has ended. + finishing: Vec, } impl JobTable { @@ -85,6 +90,7 @@ impl JobTable { current_job_id: None, previous_job_id: None, process_subs: std::collections::HashMap::new(), + finishing: Vec::new(), } } @@ -100,6 +106,7 @@ impl JobTable { state: JobState::Running, pids, reaped: Vec::new(), + reported: false, }; self.jobs.push(job); @@ -221,6 +228,26 @@ impl JobTable { self.process_subs.remove(&id).unwrap_or_default() } + /// Release the substitutions of a job that has ended. Each one is checked + /// now and again at each later safe point, without waiting: a connected + /// child that still works is not blocked on and not killed. + pub fn finish_process_subs(&mut self, subs: Vec) { + self.finishing.extend(subs); + self.settle_process_subs(); + } + + /// Check the substitutions of ended jobs once, without waiting, and drop + /// those whose child has exited (their FIFO is already removed). + pub fn settle_process_subs(&mut self) { + self.finishing.retain_mut(|sub| match sub.try_finish() { + Ok(done) => !done, + Err(e) => { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + false + } + }); + } + /// The `+` or `-` marker of a job, or a space. pub fn marker(&self, id: usize) -> char { if self.current_job_id == Some(id) { @@ -250,7 +277,9 @@ impl JobTable { } let mut finished = Vec::new(); for job in &mut self.jobs { - if job.state != JobState::Running { + // Not the display state: `kill` marks a job Done before its + // processes are collected, and a stopped job can still end. + if job.reported { continue; } for &pid in &job.pids { @@ -270,6 +299,7 @@ impl JobTable { } if job.pids.iter().all(|pid| job.reaped.contains(pid)) { job.state = JobState::Done; + job.reported = true; finished.push((job.id, job.command.clone())); } } @@ -311,11 +341,13 @@ impl Default for JobTable { impl Drop for JobTable { fn drop(&mut self) { for finished in self.reap_finished() { - for sub in finished.process_subs { - if let Err(e) = sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + self.finish_process_subs(finished.process_subs); + } + self.settle_process_subs(); + // A substitution still working (its job has ended) is left to finish, + // like one whose job still runs. + for sub in std::mem::take(&mut self.finishing) { + sub.leave_running(); } for (_, subs) in self.process_subs.drain() { for sub in subs { @@ -370,7 +402,7 @@ pub fn install_sigchld_handler() -> std::io::Result<()> { /// Take the SIGCHLD flag: true if a child changed state since the last take. #[cfg(unix)] -fn take_sigchld() -> bool { +pub fn take_sigchld() -> bool { SIGCHLD_SEEN.swap(false, Ordering::SeqCst) } @@ -535,61 +567,36 @@ mod tests { assert!(lines[1].contains("+")); // Current job marker } - #[cfg(unix)] #[test] - fn the_sigchld_handler_sets_the_flag() { - install_sigchld_handler().unwrap(); - take_sigchld(); - // SAFETY: getpid and kill take plain integers; the signal goes to - // this process, whose SIGCHLD handler only stores to an atomic. - unsafe { - libc::kill(libc::getpid(), libc::SIGCHLD); - } - let started = Instant::now(); - while !take_sigchld() { - assert!(started.elapsed() < Duration::from_secs(5), "flag not set"); - std::thread::sleep(Duration::from_millis(5)); - } + fn a_job_marked_done_before_it_was_reaped_is_still_reaped() { + // `kill` marks a job Done before its process is collected. + let pid = Command::new("true").spawn().unwrap().id() as i32; + let mut table = JobTable::new(); + let id = table.add_job(pid, "true &".to_string(), vec![pid]); + table.update_job_state(pid, JobState::Done); + let finished = reap_until_done(&mut table, id); + assert_eq!(finished.len(), 1); + assert!(table.reap_finished().is_empty()); } - #[cfg(unix)] #[test] - fn a_read_interrupted_by_sigchld_is_restarted() { - // A raw read(2) is not retried by std: without SA_RESTART it returns - // EINTR when SIGCHLD arrives during it. - install_sigchld_handler().unwrap(); - let mut fds = [0; 2]; - // SAFETY: `fds` is a live array of two c_ints for pipe(2) to fill. - assert_eq!(unsafe { libc::pipe(fds.as_mut_ptr()) }, 0); - let (read_fd, write_fd) = (fds[0], fds[1]); - let (tx, rx) = std::sync::mpsc::channel(); - let reader = std::thread::spawn(move || { - // SAFETY: pthread_self takes no arguments. - tx.send(unsafe { libc::pthread_self() }).unwrap(); - let mut byte = 0u8; - // SAFETY: `byte` is a live, exclusively borrowed u8 and the length - // is 1; read_fd is the open read end of the pipe. - let n = unsafe { libc::read(read_fd, (&mut byte as *mut u8).cast(), 1) }; - let errno = std::io::Error::last_os_error(); - (n, byte, errno) - }); - let thread = rx.recv().unwrap(); - std::thread::sleep(Duration::from_millis(100)); - // SAFETY: `thread` is the live reader thread, blocked in read(2). - assert_eq!(unsafe { libc::pthread_kill(thread, libc::SIGCHLD) }, 0); - std::thread::sleep(Duration::from_millis(100)); - // SAFETY: write_fd is the open write end; one byte from a live u8. - assert_eq!( - unsafe { libc::write(write_fd, (&b'x' as *const u8).cast(), 1) }, - 1 - ); - let (n, byte, errno) = reader.join().unwrap(); - assert_eq!(n, 1, "read was interrupted: {errno}"); - assert_eq!(byte, b'x'); - // SAFETY: both descriptors are open and owned by this test. - unsafe { - libc::close(read_fd); - libc::close(write_fd); + fn a_script_does_not_queue_job_notices() { + let dir = tempfile::TempDir::new().unwrap(); + for interactive in [false, true] { + let mut state = crate::state::ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.interactive = interactive; + let pid = Command::new("true").spawn().unwrap().id() as i32; + state.jobs.add_job(pid, "true &".to_string(), vec![pid]); + let started = Instant::now(); + while !state.jobs.jobs().is_empty() { + assert!( + started.elapsed() < Duration::from_secs(10), + "job not reaped" + ); + state.reap_jobs(); + std::thread::sleep(Duration::from_millis(10)); + } + assert_eq!(state.job_notices.len(), usize::from(interactive)); } } } diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index c3238aa7..ae3746c2 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -126,14 +126,23 @@ impl ProcessSubstitution { /// is not cut short: the shell keeps waiting until it exits, so a /// `>(cmd)` reader still finishes the work it was given. pub fn cleanup(mut self) -> Result<()> { - if let Some(mut child) = self.child_process.take() { - let status = loop { - if let Some(status) = child.try_wait()? { - break status; - } + while !self.try_finish()? { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + Ok(()) + } + + /// One non-blocking step of [`ProcessSubstitution::cleanup`]: release a + /// child that waits for a partner, then check whether it has exited. True + /// when it has, and the FIFO is removed. A connected child that still + /// works is left to work; the caller asks again later. + pub fn try_finish(&mut self) -> Result { + if let Some(child) = self.child_process.as_mut() { + let Some(status) = child.try_wait()? else { self.release_waiting_child(); - std::thread::sleep(std::time::Duration::from_millis(10)); + return Ok(false); }; + self.child_process = None; // A `<(cmd)` writer whose reader has gone ends on SIGPIPE. That is // the normal end of an unread substitution, not a failure. if !status.success() && !ended_by_sigpipe(&status) { @@ -152,7 +161,7 @@ impl ProcessSubstitution { Err(e) => return Err(e.into()), } - Ok(()) + Ok(true) } } diff --git a/impl/rust-cli/src/repl.rs b/impl/rust-cli/src/repl.rs index dc8e1d00..96bfe2aa 100644 --- a/impl/rust-cli/src/repl.rs +++ b/impl/rust-cli/src/repl.rs @@ -43,6 +43,9 @@ use crate::state::ShellState; /// # Ok::<(), anyhow::Error>(()) /// ``` pub fn run(state: &mut ShellState) -> Result<()> { + // Job notices have a reader here: they print before each prompt. + state.interactive = true; + // Install SIGINT handler for graceful Ctrl+C handling; a failure only // means Ctrl-C keeps its default behaviour, so warn and continue. if let Err(e) = ctrlc::set_handler(move || { diff --git a/impl/rust-cli/src/state.rs b/impl/rust-cli/src/state.rs index 88e4d7ab..dfb0baa4 100644 --- a/impl/rust-cli/src/state.rs +++ b/impl/rust-cli/src/state.rs @@ -315,6 +315,10 @@ pub struct ShellState { /// printed by the interactive shell before its next prompt. pub job_notices: Vec, + /// True in an interactive shell, which prints job notices. A script or + /// `-c` run has no reader for them, so none are queued. + pub interactive: bool, + /// Named checkpoints mapping name → history index at checkpoint time pub checkpoints: HashMap)>, @@ -335,20 +339,20 @@ impl ShellState { /// running process. Called at the safe points between commands. pub fn reap_jobs(&mut self) { for job in self.jobs.reap_finished() { - for sub in job.process_subs { - if let Err(e) = sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } + self.jobs.finish_process_subs(job.process_subs); + if self.interactive { + self.job_notices.push(format!( + "[{}]{} {:<20} {}", + job.id, + self.jobs.marker(job.id), + "Done", + job.command.trim_end_matches(" &") + )); } - self.job_notices.push(format!( - "[{}]{} {:<20} {}", - job.id, - self.jobs.marker(job.id), - "Done", - job.command.trim_end_matches(" &") - )); self.jobs.remove_job(job.id); } + // Not gated by SIGCHLD: one cheap try_wait per substitution. + self.jobs.settle_process_subs(); } /// Create new shell state with given root @@ -381,6 +385,7 @@ impl ShellState { fifo_counter: std::sync::atomic::AtomicUsize::new(0), jobs: crate::job::JobTable::new(), job_notices: Vec::new(), + interactive: false, checkpoints: HashMap::new(), functions: FunctionTable::new(), traps: TrapTable::new(), diff --git a/impl/rust-cli/tests/background_job_reaper_tests.rs b/impl/rust-cli/tests/background_job_reaper_tests.rs index 0395d865..a454d1d1 100644 --- a/impl/rust-cli/tests/background_job_reaper_tests.rs +++ b/impl/rust-cli/tests/background_job_reaper_tests.rs @@ -6,7 +6,7 @@ //! A hung test would leave `sh` children that hold pipes, so every run has a //! time limit and kills the process group of the shell when it passes. -#![cfg(unix)] +#![cfg(target_os = "linux")] use std::path::Path; use std::process::{Command, Stdio}; @@ -222,3 +222,29 @@ fn a_job_that_ended_before_the_shell_exits_is_cleaned_at_exit() { .is_empty()); assert_eq!(fifos_of(run.pid), Vec::::new()); } + +#[test] +fn a_connected_substitution_of_an_ended_job_blocks_no_command() { + // The writer ends at once; its >(...) reader works on for 2 seconds. + // Later commands and the exit must not wait for it, and it must not be + // killed: it still writes its file. + let dir = TempDir::new().unwrap(); + let started = Instant::now(); + let run = vsh( + dir.path(), + "sh -c 'printf data > \"$1\"' sh >(sh -c 'sleep 2; cat > out2') &\nsleep 0.3\necho ready", + 10, + ); + let took = started.elapsed(); + assert!(run.success); + assert!(run.stdout.contains("ready")); + assert!( + took < Duration::from_millis(1500), + "the shell waited: {took:?}" + ); + std::thread::sleep(Duration::from_secs(3)); + assert_eq!( + std::fs::read_to_string(dir.path().join("out2")).unwrap(), + "data" + ); +} diff --git a/impl/rust-cli/tests/sigchld_handler_tests.rs b/impl/rust-cli/tests/sigchld_handler_tests.rs new file mode 100644 index 00000000..535a1d58 --- /dev/null +++ b/impl/rust-cli/tests/sigchld_handler_tests.rs @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! The SIGCHLD handler. It installs a process-wide handler and reads a global +//! flag, so it runs in its own test binary, apart from the job table tests. + +#![cfg(unix)] + +use std::time::{Duration, Instant}; + +#[test] +fn the_sigchld_handler_sets_the_flag() { + vsh::job::install_sigchld_handler().unwrap(); + vsh::job::take_sigchld(); + // SAFETY: getpid and kill take plain integers; the signal goes to + // this process, whose SIGCHLD handler only stores to an atomic. + unsafe { + libc::kill(libc::getpid(), libc::SIGCHLD); + } + let started = Instant::now(); + while !vsh::job::take_sigchld() { + assert!(started.elapsed() < Duration::from_secs(5), "flag not set"); + std::thread::sleep(Duration::from_millis(5)); + } +} + +#[test] +fn a_read_interrupted_by_sigchld_is_restarted() { + // A raw read(2) is not retried by std: without SA_RESTART it returns + // EINTR when SIGCHLD arrives during it. + vsh::job::install_sigchld_handler().unwrap(); + let mut fds = [0; 2]; + // SAFETY: `fds` is a live array of two c_ints for pipe(2) to fill. + assert_eq!(unsafe { libc::pipe(fds.as_mut_ptr()) }, 0); + let (read_fd, write_fd) = (fds[0], fds[1]); + let (tx, rx) = std::sync::mpsc::channel(); + let reader = std::thread::spawn(move || { + // SAFETY: pthread_self takes no arguments. + tx.send(unsafe { libc::pthread_self() }).unwrap(); + let mut byte = 0u8; + // SAFETY: `byte` is a live, exclusively borrowed u8 and the length + // is 1; read_fd is the open read end of the pipe. + let n = unsafe { libc::read(read_fd, (&mut byte as *mut u8).cast(), 1) }; + let errno = std::io::Error::last_os_error(); + (n, byte, errno) + }); + let thread = rx.recv().unwrap(); + std::thread::sleep(Duration::from_millis(100)); + // SAFETY: `thread` is the live reader thread, blocked in read(2). + assert_eq!(unsafe { libc::pthread_kill(thread, libc::SIGCHLD) }, 0); + std::thread::sleep(Duration::from_millis(100)); + // SAFETY: write_fd is the open write end; one byte from a live u8. + assert_eq!( + unsafe { libc::write(write_fd, (&b'x' as *const u8).cast(), 1) }, + 1 + ); + let (n, byte, errno) = reader.join().unwrap(); + assert_eq!(n, 1, "read was interrupted: {errno}"); + assert_eq!(byte, b'x'); + // SAFETY: both descriptors are open and owned by this test. + unsafe { + libc::close(read_fd); + libc::close(write_fd); + } +} From 8367afdb86172819c3c361f0b4d9cbe9274a4bf8 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 22:19:18 +0000 Subject: [PATCH 20/21] test(sigchld): take the reader's pthread_t from its JoinHandle CodeRabbit review on 3e53e52: the test sent a pthread_t through a channel. On musl pthread_t is *mut c_void, which is not Send, so the test would not compile there. JoinHandleExt::as_pthread_t gives the handle on the parent side; the reader stays blocked in read(2) until the byte is written after pthread_kill. Checked: the test still fails with EINTR when SA_RESTART is left out, and passes with it. cargo test 935 passed, 0 failed, 14 ignored; fmt, clippy -D warnings and the fuzz check pass. Not compiled for a musl target here (only x86_64-unknown-linux-gnu is installed). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/tests/sigchld_handler_tests.rs | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/impl/rust-cli/tests/sigchld_handler_tests.rs b/impl/rust-cli/tests/sigchld_handler_tests.rs index 535a1d58..ca083e10 100644 --- a/impl/rust-cli/tests/sigchld_handler_tests.rs +++ b/impl/rust-cli/tests/sigchld_handler_tests.rs @@ -5,6 +5,7 @@ #![cfg(unix)] +use std::os::unix::thread::JoinHandleExt; use std::time::{Duration, Instant}; #[test] @@ -32,10 +33,7 @@ fn a_read_interrupted_by_sigchld_is_restarted() { // SAFETY: `fds` is a live array of two c_ints for pipe(2) to fill. assert_eq!(unsafe { libc::pipe(fds.as_mut_ptr()) }, 0); let (read_fd, write_fd) = (fds[0], fds[1]); - let (tx, rx) = std::sync::mpsc::channel(); let reader = std::thread::spawn(move || { - // SAFETY: pthread_self takes no arguments. - tx.send(unsafe { libc::pthread_self() }).unwrap(); let mut byte = 0u8; // SAFETY: `byte` is a live, exclusively borrowed u8 and the length // is 1; read_fd is the open read end of the pipe. @@ -43,9 +41,12 @@ fn a_read_interrupted_by_sigchld_is_restarted() { let errno = std::io::Error::last_os_error(); (n, byte, errno) }); - let thread = rx.recv().unwrap(); + // The handle comes from the parent's JoinHandle: on musl a pthread_t is a + // raw pointer, which a channel cannot carry between threads. + let thread = reader.as_pthread_t(); std::thread::sleep(Duration::from_millis(100)); - // SAFETY: `thread` is the live reader thread, blocked in read(2). + // SAFETY: `thread` is the reader thread. It is still alive, blocked in + // read(2), because the byte that ends the read is written only below. assert_eq!(unsafe { libc::pthread_kill(thread, libc::SIGCHLD) }, 0); std::thread::sleep(Duration::from_millis(100)); // SAFETY: write_fd is the open write end; one byte from a live u8. From b534f427190d5567b363dd4288fa9f70bac6d00e Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 23:35:52 +0000 Subject: [PATCH 21/21] test(sigchld): cast the JoinHandle's pthread_t to libc::pthread_t CodeRabbit review on 8367afd: std's RawPthread is c_ulong on every Linux target, but libc::pthread_t is *mut c_void on musl, so pthread_kill would not type-check there. The cast is a no-op on gnu (clippy -D warnings accepts it). cargo test 935 passed, 0 failed, 14 ignored; fmt and clippy pass. Not compiled for a musl target here (only x86_64-unknown-linux-gnu). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_0136P7Zk179dJt4BhyPzDhFN --- impl/rust-cli/tests/sigchld_handler_tests.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/impl/rust-cli/tests/sigchld_handler_tests.rs b/impl/rust-cli/tests/sigchld_handler_tests.rs index ca083e10..fe6315e3 100644 --- a/impl/rust-cli/tests/sigchld_handler_tests.rs +++ b/impl/rust-cli/tests/sigchld_handler_tests.rs @@ -42,8 +42,9 @@ fn a_read_interrupted_by_sigchld_is_restarted() { (n, byte, errno) }); // The handle comes from the parent's JoinHandle: on musl a pthread_t is a - // raw pointer, which a channel cannot carry between threads. - let thread = reader.as_pthread_t(); + // raw pointer, which a channel cannot carry between threads. std gives a + // c_ulong on every Linux target, so cast it to libc's pthread_t. + let thread = reader.as_pthread_t() as libc::pthread_t; std::thread::sleep(Duration::from_millis(100)); // SAFETY: `thread` is the reader thread. It is still alive, blocked in // read(2), because the byte that ends the read is written only below.