diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 70f83ee43..11cdcd0c9 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1064,7 +1064,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together. * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`). When the lock already holds a crates.io block for the same `name@version` (a later dependent locked its own copy, #679), the Socket block merges into it instead: it is dropped (with a v1 lock's `[metadata]` line), and dependents' references take the spelling cargo writes, so no duplicate block is left that cargo cannot parse (#863), and no index lookup is needed; every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. - * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `static_urls` `pdm.lock` whose project installs from an index other than PyPI (a `[[tool.pdm.source]]` in the sibling `pyproject.toml`, a project `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, with neither, another package's file URL in the lock off PyPI's file host), since restoring PyPI's file URLs would make `pdm sync` bypass that index (#413); a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `==` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back. + * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `static_urls` `pdm.lock` whose project installs from an index other than PyPI (a `[[tool.pdm.source]]` in the sibling `pyproject.toml`, a project `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, with neither, another package's file URL in the lock off PyPI's file host), since restoring PyPI's file URLs would make `pdm sync` bypass that index (#413); a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`, or an `exclude-newer-package` cutoff for the restored package) (#1604); a pylock from `uv pip compile` whose header command line (`--no-build`, `--only-binary`, `--no-binary`, `--exclude-newer`, `--exclude-newer-package`) or uv settings (the nearest `uv.toml`, else a pyproject's `[tool.uv]` / `[tool.uv.pip]`, then the user's and the system's `uv.toml` unless `UV_NO_CONFIG` is set, merged as uv does with `[pip]` over the top level; none under `--no-config`, and any `--config-file` refuses; `UV_CONFIG_FILE` or `UV_EXCLUDE_NEWER` set in the restoring environment also refuses, since the header does not record uv's environment) filter the restored package's files, or whose header records no `uv pip compile` command line (`--no-header`, `UV_CUSTOM_COMPILE_COMMAND`), since uv 0.9.16 records only the files those filters let through (#1613); a uv lock whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `==` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back. * **gem** — `Gemfile.lock` / `gems.locked` + `Gemfile` / `gems.rb`: the spec moves back into the upstream `GEM` section (or the Socket remote leaves a merged section), the `source "" do … end` block is undone, the `CHECKSUMS` entry is re-pinned from the rubygems.org compact index (`SOCKET_RUBYGEMS_URL`) and the `DEPENDENCIES` pin loses its `!`, unless the declaration comes back inside the user's own `source "…" do` block (Bundler writes the `!` for any dependency declared in one, rubygems.org included), where the `!` stays. The declaration's original constraint is not recorded, so it comes back as the exact pin `gem "", ""`. A transitive gem (one the manifest never declared) gets an appended block with a blank line before it; the restore removes that block, its blank line and the `DEPENDENCIES` entry, so the pair comes back byte for byte. An appended block with no blank line before it (written by a release before this one) can't be told apart from an in-place rewrite, so it still comes back as the exact pin. Refused: an ambiguous upstream section, an upstream remote other than rubygems.org. The manifest pair can't make a committed bundler cache upstream: a `-.gem` left in Bundler's cache dir that isn't the upstream archive is named by `upstream_gem_stale_cache`, never deleted. * **composer** — `composer.lock`: `dist` and the deleted `source` block from packagist's composer v2 metadata (`SOCKET_PACKAGIST_URL`). Refused unless the entry is packagist-sourced and packagist still serves the lock's `dist.reference` for the version. * **maven** — `pom.xml` (the `-socket.` version suffix, the added `` / `` entry) and the `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` lines hosted mode writes: **no network**, so it restores under `--offline` too. `.mvn` files holding anything else keep the resolver lines (`maven_trusted_checksums_left`). diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/uv.rs b/crates/socket-patch-core/src/patch/redirect/upstream/uv.rs index 648282aae..a9d6a5c7f 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/uv.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/uv.rs @@ -22,9 +22,13 @@ //! only the wheels its `requires-python` and environments can install, so //! a release with any wheel that is not pure Python 3 is refused, as is a //! lock whose `[options]` / `[tool.uv]` filter files (`exclude-newer`, -//! `no-binary`, `no-build`). PEP 751 allows both TOML spellings of the -//! artifacts: uv's inline `wheels = [{ … }]`, and the standard tables -//! `pip lock` writes (`[[packages.wheels]]` with a +//! `no-binary`, `no-build`, an `exclude-newer-package` cutoff for the +//! package, #1604), and a `uv pip compile` pylock whose header command +//! line or uv settings (project, user and system) filter the package's +//! files, or whose header records no command line (uv 0.9.16+ records +//! only the files they let through, #1613). PEP 751 allows both TOML +//! spellings of the artifacts: uv's inline `wheels = [{ … }]`, and the +//! standard tables `pip lock` writes (`[[packages.wheels]]` with a //! `[packages.wheels.hashes]` sub-table, `[packages.sdist]`); the entry //! comes back in its siblings' spelling. `pip lock` records only the one //! artifact pip selected — the wheel, or the sdist of a release with no @@ -334,20 +338,104 @@ async fn restore_lock( if !version_ok { return refuse_hits(result, "unsupported lock version"); } - if doc - .get("options") - .and_then(Item::as_table_like) - .is_some_and(|o| o.contains_key("exclude-newer")) - { + let options = doc.get("options").and_then(Item::as_table_like); + if let Some(key) = options.and_then(|o| { + o.iter() + .map(|(k, _)| k) + .find(|k| k.starts_with("exclude-newer") && *k != "exclude-newer-package") + }) { return refuse_hits( result, - "the lock's `exclude-newer` option filters release files by upload time", + &format!("the lock's `{key}` option filters release files by upload time"), + ); + } + // A per-package cutoff (#1604) filters only the packages it names. + let cutoffs = options.and_then(|o| o.get("exclude-newer-package")); + for hit in hits + .iter() + .filter(|h| cutoffs.is_some_and(|c| names_package(c, &h.name))) + { + result.refuse( + &hit.uuid, + format!( + "{rel}: the lock's `exclude-newer-package` cutoff for {} filters its release \ + files by upload time", + hit.name + ), ); } let shape = match lock_shape(&doc, collection, pep751, &hits) { Ok(shape) => shape, Err(why) => return refuse_hits(result, &why), }; + if pep751 && pip_compiled(&doc, &text, &shape) { + // Without uv's command line (`--no-header`, or a header replaced by + // `UV_CUSTOM_COMPILE_COMMAND`) its filter flags are unknowable. + let Some(command) = uv_header_command(&text).filter(|c| is_pip_compile(c)) else { + return refuse_hits( + result, + &format!( + "{rel}: the lock does not record the `uv pip compile` command line, whose \ + flags may filter which release files it records" + ), + ); + }; + // `--no-config` reads no settings file; `--config-file` names one + // relative to wherever uv ran, which the lock does not record. + let settings = if command.contains(&"--no-config") { + Vec::new() + } else if command + .iter() + .any(|a| *a == "--config-file" || a.starts_with("--config-file=")) + { + return refuse_hits( + result, + "`uv pip compile --config-file` read settings that may filter which release \ + files the lock records, and that file is not derivable from the lock", + ); + } else if env_set("UV_CONFIG_FILE") { + // The header never records uv's environment; one set here is + // the best evidence of the one `uv pip compile` ran in. + return refuse_hits( + result, + "`UV_CONFIG_FILE` is set, and settings read from it may filter which release \ + files `uv pip compile` records", + ); + } else { + let project = match pip_settings(view, rel).await { + Ok(settings) => settings, + Err(why) => return refuse_hits(result, &why), + }; + let global = + match global_uv_settings(&|name| std::env::var_os(name), cfg!(windows)).await { + Ok(settings) => settings, + Err(why) => return refuse_hits(result, &why), + }; + project.into_iter().chain(global).collect() + }; + for hit in &hits { + let why = if let Some(flag) = compile_filter(&command, &hit.name) { + format!("`uv pip compile {flag}` filters") + } else if std::env::var_os("UV_EXCLUDE_NEWER") + // Empty is unset, and `false` turns the cutoff off. + .is_some_and(|v| !v.is_empty() && v != "false") + { + "the `UV_EXCLUDE_NEWER` cutoff set in this environment filters".to_string() + } else if let Some((path, key)) = settings_filter(&settings, &hit.name) { + format!("{path}'s uv `{key}` setting filters") + } else { + continue; + }; + result.refuse( + &hit.uuid, + format!( + "{rel}: {why} which of {}'s release files `uv pip compile` records, so \ + they are not re-derivable", + hit.name + ), + ); + } + } // The paired metadata (a project lock with no pyproject beside it was // edited alone). @@ -568,6 +656,284 @@ fn pylock_unindexed_registry(package: &toml_edit::Table) -> Option bool { + if shape.selected_only || doc.get("created-by").and_then(Item::as_str) != Some("uv") { + return false; + } + match uv_header_command(text) { + Some(command) if is_pip_compile(&command) => true, + Some(command) if command.first() == Some(&"export") => false, + // No header, or one `UV_CUSTOM_COMPILE_COMMAND` replaced. + _ => !shape.index, + } +} + +/// Whether the uv environment variable `name` is set to a value. +fn env_set(name: &str) -> bool { + std::env::var_os(name).is_some_and(|v| !v.is_empty()) +} + +/// Whether a header command line is `uv pip compile`'s. uv records global +/// options where they were given, before `pip` or between it and +/// `compile` (`uv --no-config pip compile …`). +fn is_pip_compile(command: &[&str]) -> bool { + command + .iter() + .position(|a| *a == "pip") + .is_some_and(|pip| command[pip + 1..].contains(&"compile")) +} + +/// The arguments of the `uv …` command line uv's autogenerated header +/// records, after `uv`. +fn uv_header_command(text: &str) -> Option> { + text.lines() + .take_while(|line| line.starts_with('#')) + .find_map(|line| { + let mut words = line.trim_start_matches('#').split_whitespace(); + (words.next() == Some("uv")).then(|| words.collect()) + }) +} + +/// Whether a filter setting's value reaches `name`: `true`, a cutoff date, +/// a list naming it (or `:all:`), or a per-package cutoff table naming it +/// with anything but `false` (uv's "no cutoff"). +fn names_package(item: &Item, name: &str) -> bool { + if let Some(cutoffs) = item.as_table_like() { + return cutoffs.iter().any(|(package, cutoff)| { + canonicalize_pypi_name(package) == name && cutoff.as_bool() != Some(false) + }); + } + match item.as_value() { + Some(Value::Boolean(on)) => *on.value(), + Some(Value::String(_)) | Some(Value::Datetime(_)) => true, + Some(Value::Array(list)) => list + .iter() + .filter_map(Value::as_str) + .any(|v| v == ":all:" || canonicalize_pypi_name(v) == name), + _ => false, + } +} + +/// The `uv pip compile` flag in `command` that filters `name`'s files: +/// `--no-build`, `--only-binary` / `--no-binary` naming it or `:all:`, +/// `--exclude-newer`, or an `--exclude-newer-package` cutoff for it. +fn compile_filter(command: &[&str], name: &str) -> Option { + let mut args = command.iter(); + while let Some(arg) = args.next() { + let (flag, inline) = match arg.split_once('=') { + Some((flag, value)) if flag.starts_with("--") => (flag, Some(value)), + _ => (*arg, None), + }; + let applies = match flag { + "--no-build" => true, + // `false` turns the cutoff off (uv 0.11.24+). + "--exclude-newer" => inline.or_else(|| args.next().copied()) != Some("false"), + "--only-binary" | "--no-binary" | "--exclude-newer-package" => { + let Some(value) = inline.or_else(|| args.next().copied()) else { + continue; + }; + if flag == "--exclude-newer-package" { + value.split_once('=').is_some_and(|(package, cutoff)| { + canonicalize_pypi_name(package) == name && cutoff != "false" + }) + } else { + value + .split(',') + .any(|v| v == ":all:" || canonicalize_pypi_name(v) == name) + } + } + _ => false, + }; + if applies { + return Some(flag.to_string()); + } + } + None +} + +/// The uv settings `uv pip compile` reads beside a pylock, as `(settings +/// file, its uv table)`: the nearest `uv.toml`, else a pyproject with a +/// `[tool.uv]` table, in the lock's directory or above it. Like uv, a +/// pyproject that does not parse is passed over. +async fn pip_settings( + view: &mut View<'_>, + rel: &str, +) -> Result, String> { + let mut dir = rel.rsplit_once('/').map_or("", |(d, _)| d); + loop { + let path = |file: &str| { + if dir.is_empty() { + file.to_string() + } else { + format!("{dir}/{file}") + } + }; + let uv_toml = path("uv.toml"); + if let Some(text) = view.read(&uv_toml).await? { + let doc = text + .parse::() + .map_err(|e| format!("{uv_toml} is not valid TOML: {e}"))?; + return Ok(Some((uv_toml, doc.as_table().clone()))); + } + let pyproject = path("pyproject.toml"); + if let Some(text) = view.read(&pyproject).await? { + let uv = text.parse::().ok().and_then(|doc| { + let uv = doc.get("tool")?.get("uv")?; + uv.as_table() + .cloned() + .or_else(|| uv.as_inline_table().map(|t| t.clone().into_table())) + }); + if let Some(uv) = uv { + return Ok(Some((pyproject, uv))); + } + } + if dir.is_empty() { + return Ok(None); + } + dir = dir.rsplit_once('/').map_or("", |(d, _)| d); + } +} + +/// The user- and system-level `uv.toml` files uv reads after the project's +/// settings, as `(path, table)` in that order: `$XDG_CONFIG_HOME/uv` (else +/// `~/.config/uv`, or `%APPDATA%\uv` on Windows), then the first +/// `$XDG_CONFIG_DIRS` entry holding one (default `/etc/xdg`), else +/// `/etc/uv` (`%PROGRAMDATA%\uv` on Windows). A relative or empty variable +/// counts as unset. +async fn global_uv_settings( + var: &impl Fn(&str) -> Option, + windows: bool, +) -> Result, String> { + use std::path::PathBuf; + // uv's boolish `UV_NO_CONFIG` reads no settings file. + let no_config = var("UV_NO_CONFIG") + .and_then(|v| v.into_string().ok()) + .is_some_and(|v| ["1", "true", "yes", "y", "on"].contains(&v.to_lowercase().as_str())); + if no_config { + return Ok(Vec::new()); + } + let absolute = |name: &str| { + var(name) + .map(PathBuf::from) + .filter(|path| path.is_absolute()) + }; + let file = |dir: PathBuf| dir.join("uv").join("uv.toml"); + let (user, system) = if windows { + let system = absolute("PROGRAMDATA").map(file).into_iter().collect(); + (absolute("APPDATA").map(file), system) + } else { + let user = absolute("XDG_CONFIG_HOME") + .or_else(|| absolute("HOME").map(|home| home.join(".config"))) + .map(file); + // Like uv, the entries end at the first empty one. + let dirs = var("XDG_CONFIG_DIRS") + .and_then(|dirs| dirs.into_string().ok()) + .unwrap_or_else(|| "/etc/xdg".into()); + let mut system: Vec<_> = dirs + .split(':') + .take_while(|dir| !dir.is_empty()) + .map(|dir| file(PathBuf::from(dir))) + .collect(); + system.push(PathBuf::from("/etc/uv/uv.toml")); + (user, system) + }; + let mut settings = Vec::new(); + // uv reads the first system file that exists. + let candidates = user + .into_iter() + .map(|p| (p, false)) + .chain(system.into_iter().map(|p| (p, true))); + for (path, system) in candidates { + // uv passes over a candidate that is not a file (opening a + // directory fails with `PermissionDenied` on Windows, so check + // first); the regular-file opener also never blocks on a FIFO. + if !tokio::fs::metadata(&path).await.is_ok_and(|m| m.is_file()) { + continue; + } + let text = match crate::utils::fs::read_regular_to_string(&path).await { + Ok(text) => text, + Err(e) + if matches!( + e.kind(), + std::io::ErrorKind::NotFound | std::io::ErrorKind::InvalidInput + ) => + { + continue + } + Err(e) => return Err(format!("could not read {}: {e}", path.display())), + }; + let doc = text + .parse::() + .map_err(|e| format!("{} is not valid TOML: {e}", path.display()))?; + settings.push((path.display().to_string(), doc.as_table().clone())); + if system { + break; + } + } + Ok(settings) +} + +/// The uv setting that filters which of `name`'s files `uv pip compile` +/// records (`no-build`, `no-binary`, `only-binary`, `exclude-newer` and +/// their per-package forms), as `(file that sets it, key)`. Like uv, the +/// files merge key by key: lists from every file add up, and otherwise the +/// first file setting a key wins. Then `[pip]`'s value of a key wins over +/// the top level's. `no-build-isolation` changes only how an sdist is +/// built. +fn settings_filter(files: &[(String, toml_edit::Table)], name: &str) -> Option<(String, String)> { + fn pip(uv: &toml_edit::Table) -> Option<&dyn TableLike> { + uv.get("pip").and_then(Item::as_table_like) + } + let keys = files.iter().flat_map(|(_, uv)| { + uv.iter() + .map(|(k, _)| k) + .filter(|k| *k != "pip") + .chain(pip(uv).into_iter().flat_map(|p| p.iter().map(|(k, _)| k))) + }); + for key in keys { + let filters = ["no-build", "no-binary", "only-binary", "exclude-newer"] + .iter() + .any(|f| key.starts_with(f)) + && !key.starts_with("no-build-isolation"); + if !filters { + continue; + } + let in_pip: Vec<_> = files + .iter() + .filter_map(|(path, uv)| Some((path, pip(uv)?.get(key)?))) + .collect(); + let set = if in_pip.is_empty() { + files + .iter() + .filter_map(|(path, uv)| Some((path, uv.get(key)?))) + .collect() + } else { + in_pip + }; + let decides = if set.iter().all(|(_, v)| v.as_array().is_some()) { + set.iter().find(|(_, v)| names_package(v, name)) + } else if set.iter().all(|(_, v)| v.is_table_like()) { + // Per-package tables merge entry by entry, the first one + // naming the package winning. + set.iter().find(|(_, v)| { + v.as_table_like() + .is_some_and(|t| t.iter().any(|(k, _)| canonicalize_pypi_name(k) == name)) + }) + } else { + set.first() + }; + if let Some((path, _)) = decides.filter(|(_, v)| names_package(v, name)) { + return Some((path.to_string(), key.to_string())); + } + } + None +} + /// The registry and artifact shape the lock's other registry packages show. fn lock_shape( doc: &DocumentMut, @@ -1872,6 +2238,162 @@ fn hide_header_over_sub_tables(table: &mut toml_edit::Table) { mod tests { use super::*; + fn uv_file(path: &str, text: &str) -> (String, toml_edit::Table) { + let doc = text.parse::().unwrap(); + (path.to_string(), doc.as_table().clone()) + } + + #[test] + fn settings_files_merge_key_by_key_with_pip_over_the_top_level() { + let project = uv_file("uv.toml", "[pip]\nno-build = false\n"); + let user = uv_file("user/uv.toml", "no-build = true\nonly-binary = [\"six\"]\n"); + let system = uv_file("/etc/uv/uv.toml", "[pip]\nno-binary = [\"six\"]\n"); + let files = [project.clone(), user.clone(), system.clone()]; + // The project's `[pip] no-build = false` wins over the user's + // top-level `no-build`; the user's `only-binary` still applies. + assert_eq!( + settings_filter(&files, "six"), + Some(("user/uv.toml".into(), "only-binary".into())) + ); + assert_eq!(settings_filter(&files, "idna"), None); + // A system `[pip]` value wins over a user top-level one. + let files = [uv_file("user/uv.toml", "no-binary = []\n"), system]; + assert_eq!( + settings_filter(&files, "six"), + Some(("/etc/uv/uv.toml".into(), "no-binary".into())) + ); + assert_eq!( + settings_filter(&[user], "idna"), + Some(("user/uv.toml".into(), "no-build".into())) + ); + // Lists from every file add up, as uv combines them. + let files = [ + uv_file("uv.toml", "[pip]\nno-binary = []\n"), + uv_file("user/uv.toml", "[pip]\nno-binary = [\"six\"]\n"), + ]; + assert_eq!( + settings_filter(&files, "six"), + Some(("user/uv.toml".into(), "no-binary".into())) + ); + // Per-package cutoffs merge entry by entry. + let files = [ + uv_file( + "uv.toml", + "exclude-newer-package = { idna = \"2024-01-01\" }\n", + ), + uv_file( + "user/uv.toml", + "exclude-newer-package = { six = \"2024-01-01\" }\n", + ), + uv_file( + "/etc/uv/uv.toml", + "exclude-newer-package = { Six = false }\n", + ), + ]; + assert_eq!( + settings_filter(&files, "six"), + Some(("user/uv.toml".into(), "exclude-newer-package".into())) + ); + assert_eq!(settings_filter(&files[2..], "six"), None); + } + + #[tokio::test] + async fn global_uv_settings_reads_the_user_then_first_system_file() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + let write = |rel: &str, text: &str| { + let path = root.join(rel); + std::fs::create_dir_all(path.parent().unwrap()).unwrap(); + std::fs::write(path, text).unwrap(); + }; + write("xdg/uv/uv.toml", "[pip]\nno-build = true\n"); + write("home/.config/uv/uv.toml", "no-binary = true\n"); + write("dirs2/uv/uv.toml", "only-binary = [\"six\"]\n"); + write("dirs3/uv/uv.toml", "no-build = true\n"); + let env = |pairs: Vec<(&'static str, std::path::PathBuf)>| { + move |name: &str| { + pairs + .iter() + .find(|(key, _)| *key == name) + .map(|(_, value)| value.clone().into_os_string()) + } + }; + let paths = |files: Vec<(String, toml_edit::Table)>| -> Vec { + files.into_iter().map(|(path, _)| path).collect() + }; + let dirs = + std::env::join_paths([root.join("dirs1"), root.join("dirs2"), root.join("dirs3")]) + .unwrap(); + let var = env(vec![ + ("XDG_CONFIG_HOME", root.join("xdg")), + ("HOME", root.join("home")), + ("XDG_CONFIG_DIRS", dirs.into()), + ]); + assert_eq!( + paths(global_uv_settings(&var, false).await.unwrap()), + [ + root.join("xdg/uv/uv.toml").display().to_string(), + root.join("dirs2/uv/uv.toml").display().to_string(), + ] + ); + // A relative XDG_CONFIG_HOME counts as unset. + let var = env(vec![ + ("XDG_CONFIG_HOME", "rel".into()), + ("HOME", root.join("home")), + ("XDG_CONFIG_DIRS", root.join("none")), + ]); + let found = paths(global_uv_settings(&var, false).await.unwrap()); + assert_eq!( + found[0], + root.join("home/.config/uv/uv.toml").display().to_string() + ); + // Windows reads %APPDATA% and %PROGRAMDATA%. + write("appdata/uv/uv.toml", "no-build = true\n"); + let var = env(vec![ + ("APPDATA", root.join("appdata")), + ("PROGRAMDATA", root.join("dirs3")), + ]); + assert_eq!( + paths(global_uv_settings(&var, true).await.unwrap()), + [ + root.join("appdata/uv/uv.toml").display().to_string(), + root.join("dirs3/uv/uv.toml").display().to_string(), + ] + ); + // Like uv, the system entries end at the first empty one, and a + // directory named `uv.toml` is passed over. + write("dirs4/uv/uv.toml/x", ""); + let dirs = format!( + "{}:{}::{}", + root.join("dirs4").display(), + root.join("dirs1").display(), + root.join("dirs2").display() + ); + let var = env(vec![("XDG_CONFIG_DIRS", dirs.into())]); + let found = paths(global_uv_settings(&var, false).await.unwrap()); + assert!(found.iter().all(|p| !p.contains("dirs")), "{found:?}"); + // `UV_NO_CONFIG` reads none. + let var = env(vec![ + ("XDG_CONFIG_HOME", root.join("xdg")), + ("UV_NO_CONFIG", "1".into()), + ]); + assert!(global_uv_settings(&var, false).await.unwrap().is_empty()); + #[cfg(unix)] + { + // A FIFO is passed over rather than read, which would block. + let fifo = root.join("fifo/uv/uv.toml"); + std::fs::create_dir_all(fifo.parent().unwrap()).unwrap(); + let c = std::ffi::CString::new(fifo.to_str().unwrap()).unwrap(); + assert_eq!(unsafe { libc::mkfifo(c.as_ptr(), 0o644) }, 0); + let var = env(vec![("XDG_CONFIG_HOME", root.join("fifo"))]); + let found = paths(global_uv_settings(&var, false).await.unwrap()); + assert!(found.iter().all(|p| !p.contains("fifo")), "{found:?}"); + } + write("bad/uv/uv.toml", "[pip\n"); + let var = env(vec![("XDG_CONFIG_HOME", root.join("bad"))]); + assert!(global_uv_settings(&var, false).await.is_err()); + } + #[test] fn upload_times_are_milliseconds_without_trailing_zeros() { assert_eq!( diff --git a/crates/socket-patch-core/tests/upstream_restore_golden.rs b/crates/socket-patch-core/tests/upstream_restore_golden.rs index 0d2f98faf..d9017e5a1 100644 --- a/crates/socket-patch-core/tests/upstream_restore_golden.rs +++ b/crates/socket-patch-core/tests/upstream_restore_golden.rs @@ -1202,7 +1202,11 @@ async fn pypi_mock(releases: &[Release<'_>]) -> (MockServer, EnvGuard) { .mount(&server) .await; } - let env = EnvGuard::set(&[("SOCKET_PYPI_JSON_API", format!("{}/pypi", server.uri()))]); + // The host's user- and system-level uv settings stay out of the tests. + let env = EnvGuard::set(&[ + ("SOCKET_PYPI_JSON_API", format!("{}/pypi", server.uri())), + ("UV_NO_CONFIG", "1".to_string()), + ]); (server, env) } @@ -2212,6 +2216,178 @@ async fn pylock_whole_second_upload_times_round_trip() { assert_pypi_round_trip("uv export pylock", &input, &[urllib3_dep()], None).await; } +/// #1613: since uv 0.9.16, `uv pip compile --format pylock.toml` applies +/// `--no-build` / `--only-binary` / `--no-binary` (and the `exclude-newer` +/// cutoffs) to the files it records, so a filter that reaches the patched +/// package makes the entry's files not re-derivable from PyPI. The command +/// line in uv's header and the uv settings `uv pip` reads beside the lock +/// show the filter; one that names another package, or that only changes +/// how sdists are built, restores byte for byte. +#[tokio::test] +#[serial] +async fn pip_compile_pylock_build_filters_refuse_the_filtered_entry() { + let (_server, _env) = pypi_mock(&[urllib3_release()]).await; + let header = "# uv pip compile --format pylock.toml req.in -o pylock.toml\n"; + let with_flags = |flags: &str| { + uv_pylock(false).replace( + header, + &format!("# uv pip compile --format pylock.toml req.in{flags} -o pylock.toml\n"), + ) + }; + assert!(with_flags(" --no-build").contains("--no-build")); + for flags in [ + " --no-build", + " --only-binary urllib3", + " --only-binary=:all:", + " --no-binary urllib3", + " --no-binary=idna,urllib3", + " --no-binary Urllib3", + " --exclude-newer 2024-01-01", + " --exclude-newer-package urllib3=2023-10-17T17:46:22Z", + ] { + let input = tree(&[("pylock.toml", with_flags(flags))]); + let (why, rewritten, after) = + pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert_eq!( + after, rewritten, + "{flags}: a refused pin leaves the lock hosted" + ); + assert!(why.contains("pylock.toml"), "{flags}: {why}"); + let flag = flags.trim().split([' ', '=']).next().unwrap(); + assert!(why.contains(flag), "{flags}: {why}"); + } + for flags in [ + " --no-binary idna", + " --only-binary :none:", + " --no-build-isolation", + " --exclude-newer-package idna=2023-01-01T00:00:00Z", + " --exclude-newer-package urllib3=false", + " --exclude-newer false", + " --exclude-newer=false", + ] { + let input = tree(&[("pylock.toml", with_flags(flags))]); + assert_pypi_round_trip(flags, &input, &[urllib3_dep()], None).await; + } + + // uv's settings files: a `uv.toml` (top level or `[pip]`), else a + // pyproject's `[tool.uv]` / `[tool.uv.pip]`. + let plain = uv_pylock(false); + for (rel, settings) in [ + ("uv.toml", "[pip]\nno-build = true\n"), + ("uv.toml", "only-binary = [\"urllib3\"]\n"), + ( + "pyproject.toml", + "[project]\nname = \"p\"\nversion = \"0\"\n\n[tool.uv]\nno-binary-package = [\"urllib3\"]\n", + ), + ( + "pyproject.toml", + "[project]\nname = \"p\"\nversion = \"0\"\n\n[tool.uv.pip]\nexclude-newer = \"2024-01-01T00:00:00Z\"\n", + ), + ] { + let input = tree(&[("pylock.toml", plain.clone()), (rel, settings.to_string())]); + let (why, _, after) = + pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!(after["pylock.toml"].contains("patch.socket.dev"), "{rel}"); + assert!(why.contains(rel), "{rel}: {why}"); + } + for (rel, settings) in [ + ("uv.toml", "no-build-isolation = true\n"), + ("uv.toml", "[pip]\nno-binary = [\"idna\"]\n"), + // `uv pip` reads `[pip]` over the top level. + ("uv.toml", "no-build = true\n\n[pip]\nno-build = false\n"), + // uv passes over a pyproject that does not parse. + ("pyproject.toml", "[project\n"), + // A pyproject without `[tool.uv]` sets nothing. + ( + "pyproject.toml", + "[project]\nname = \"p\"\nversion = \"0\"\n", + ), + ] { + let input = tree(&[("pylock.toml", plain.clone()), (rel, settings.to_string())]); + assert_pypi_round_trip(rel, &input, &[urllib3_dep()], None).await; + } + let input = tree(&[ + ("pylock.toml", plain.clone()), + ("uv.toml", "offline = false\n".to_string()), + ( + "pyproject.toml", + "[project]\nname = \"p\"\nversion = \"0\"\n\n[tool.uv]\nno-build = true\n".to_string(), + ), + ]); + assert_pypi_round_trip("uv.toml over pyproject", &input, &[urllib3_dep()], None).await; + // `--no-config` read no settings file; `--config-file` read one the + // lock does not name. + let no_build = ("uv.toml", "no-build = true\n".to_string()); + let input = tree(&[ + ("pylock.toml", with_flags(" --no-config")), + no_build.clone(), + ]); + assert_pypi_round_trip("--no-config", &input, &[urllib3_dep()], None).await; + // uv records global options before `pip` or before `compile`. + for command in [ + "uv --native-tls pip compile --format pylock.toml req.in --no-build", + "uv --offline pip --quiet compile --format pylock.toml req.in --no-build", + ] { + let lock = uv_pylock(false).replace(header, &format!("# {command}\n")); + let input = tree(&[("pylock.toml", lock)]); + let (why, _, _) = pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!(why.contains("--no-build"), "{command}: {why}"); + } + let lock = uv_pylock(false).replace(header, "# uv --no-config pip compile req.in\n"); + let input = tree(&[("pylock.toml", lock), no_build.clone()]); + assert_pypi_round_trip("global --no-config", &input, &[urllib3_dep()], None).await; + let input = tree(&[("pylock.toml", with_flags(" --config-file ci/uv.toml"))]); + let (why, _, _) = pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!(why.contains("--config-file"), "{why}"); + // uv's environment is not in the header: a cutoff or settings file + // set in the restoring environment refuses. + for (var, value, expect) in [ + ( + "UV_EXCLUDE_NEWER", + "2023-01-01T00:00:00Z", + "UV_EXCLUDE_NEWER", + ), + ("UV_CONFIG_FILE", "ci/uv.toml", "UV_CONFIG_FILE"), + ] { + let _env = EnvGuard::set(&[(var, value.to_string())]); + let input = tree(&[("pylock.toml", uv_pylock(false))]); + let (why, _, _) = pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!(why.contains(expect), "{why}"); + } + // Empty is unset, and `false` turns the cutoff off. + for value in ["", "false"] { + let _env = EnvGuard::set(&[("UV_EXCLUDE_NEWER", value.to_string())]); + let input = tree(&[("pylock.toml", uv_pylock(false))]); + assert_pypi_round_trip("UV_EXCLUDE_NEWER off", &input, &[urllib3_dep()], None).await; + } + // With no uv command line recorded (`--no-header`, or a header that + // `UV_CUSTOM_COMPILE_COMMAND` replaced) the flags cannot be ruled out. + let comment = "# This file was autogenerated by uv via the following command:\n"; + for lock in [ + uv_pylock(false).replace(comment, "").replace(header, ""), + uv_pylock(false).replace(header, "# make lock\n"), + uv_pylock(false).replace(header, "# uv run poe lock\n"), + ] { + let input = tree(&[("pylock.toml", lock)]); + let (why, _, _) = pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!(why.contains("does not record"), "{why}"); + } + + // `uv export --format pylock.toml` records every file whatever the + // filters (it carries an `index`), so its restore is unaffected. + let input = tree(&[ + ( + "pylock.toml", + uv_pylock(true).replace( + header, + "# uv export --format pylock.toml --no-build -o pylock.toml\n", + ), + ), + ("uv.toml", "no-build = true\n".to_string()), + ]); + assert_pypi_round_trip("uv export", &input, &[urllib3_dep()], None).await; +} + /// A pylock as `pip lock` writes it (#804): PEP 751's array-of-tables /// spelling (`[[packages.wheels]]` with a `[packages.wheels.hashes]` /// sub-table), no `index`, and only the one artifact pip selected. @@ -2418,6 +2594,66 @@ async fn uv_refusals() { assert!(why.contains("interpreter-specific wheels"), "{why}"); } +/// #1604: uv 0.12+ applies a per-package `exclude-newer-package` cutoff to +/// each file, and records it as `[options.exclude-newer-package]`. A cutoff +/// on the patched package makes its files not re-derivable, in project and +/// script locks alike; one on another package, or `false` (no cutoff), +/// restores byte for byte. +#[tokio::test] +#[serial] +async fn uv_exclude_newer_package_refuses_only_its_package() { + let (_server, _env) = pypi_mock(&[urllib3_release()]).await; + let with_options = |lock: String, options: &str| { + lock.replacen( + "requires-python = \">=3.8\"\n", + &format!("requires-python = \">=3.8\"\n\n[options]\n\n{options}"), + 1, + ) + }; + let direct = uv_lock( + " { name = \"idna\" },\n { name = \"urllib3\" },\n", + " { name = \"idna\", specifier = \">=3\" },\n { name = \"urllib3\", specifier = \"==1.26.18\" },\n", + "", + ); + let pyproject = "[project]\nname = \"proj\"\nversion = \"0.1.0\"\ndependencies = [\"idna>=3\", \"urllib3==1.26.18\"]\n"; + let script = "#!/usr/bin/env python3\n# /// script\n# dependencies = [\"idna>=3\", \"urllib3==1.26.18\"]\n# ///\nprint('hi')\n"; + let script_lock = uv_lock("", "", "\n[manifest]\nrequirements = [\n { name = \"idna\", specifier = \">=3\" },\n { name = \"urllib3\", specifier = \"==1.26.18\" },\n]\n") + .replace( + "[[package]]\nname = \"proj\"\nversion = \"0.1.0\"\nsource = { virtual = \".\" }\ndependencies = [\n]\n\n[package.metadata]\nrequires-dist = [\n]\n\n", + "", + ); + let lanes = [ + ("uv.lock", direct, Some(("pyproject.toml", pyproject))), + ("tool.py.lock", script_lock, Some(("tool.py", script))), + ]; + for (rel, lock, meta) in &lanes { + let input_with = |options: &str| { + let mut files = vec![(*rel, with_options(lock.clone(), options))]; + files.extend(meta.map(|(m, text)| (m, text.to_string()))); + tree(&files) + }; + for options in [ + "[options.exclude-newer-package]\nurllib3 = \"2023-10-17T17:46:22Z\"\n", + "[options.exclude-newer-package]\nidna = false\nurllib3 = \"2023-10-17T17:46:22Z\"\n", + "exclude-newer-span = \"P7D\"\n", + ] { + let input = input_with(options); + let (why, rewritten, after) = + pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert_eq!(after, rewritten, "{rel} {options}: the lock stays hosted"); + assert!(why.contains("exclude-newer"), "{rel} {options}: {why}"); + } + for options in [ + "[options.exclude-newer-package]\nidna = \"2022-01-01T00:00:00Z\"\n", + "[options.exclude-newer-package]\nurllib3 = false\n", + ] { + let input = input_with(options); + assert_pypi_round_trip(&format!("{rel} {options}"), &input, &[urllib3_dep()], None) + .await; + } + } +} + /// Serve each override's npm version document with the slot-[2] integrity /// its `input/` vlt-lock.json node records. async fn vlt_mock(case: &Case) -> MockServer {