From b2207e406f2b418e23ba0ced0e28386f010e8c77 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Wed, 19 Aug 2026 13:23:12 -0400 Subject: [PATCH 1/2] fix(vendor): fall back to local build when the served gem stub gemspec is invalid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Defense-in-depth for defect D4 from the 2026-08-19 gem live-matrix campaign (verified 6/6 live across bundler 1.17/2.7/4.0): production's gem-stub-gemspec secondary artifact omits the rubygems-required `summary` and `authors` attributes (and `licenses`). The CLI sha512-verified the stub and wrote it verbatim as the vendored path-source `.gemspec`, and every bundler major validates path-source gemspecs, so any post-vendor or fresh-checkout `bundle install` exited 1 with `missing value for attribute summary`. A depscan-side fix for the stub generator is in flight, but every currently-published gem stub is invalid, so the CLI hardens now. An INVALID served stub now follows the existing MISSING-stub policy, under its own code (additive/MINOR): - `--vendor-source auto`: loud `vendor_prebuilt_stub_invalid` warning naming the missing attributes, then fall back to the local build (installed gem + locally derived stub); - `--vendor-source service` (explicit): refuse with `vendor_prebuilt_stub_invalid`, naming the attributes and the remedy, before anything is written (no partial artifacts). Validation is a conservative textual heuristic (assignment-line presence for `summary` / `authors`|`author`, obviously-empty spellings rejected — no ruby parsing); a legitimate stub always passes and is still written byte-verbatim. A missing `licenses` is only mentioned in the message (rubygems warns, not fails). CLI_CONTRACT.md documents the new code in the fallback ladder and the PatchAction vocabulary. Tests: - hermetic (wiremock, RED->GREEN): auto+invalid-stub falls back to the local build with the loud warning and the LOCAL stub on disk; service+invalid-stub refuses with `vendor_prebuilt_stub_invalid` leaving no partial artifacts and an untouched lock; the valid-stub byte-verbatim write is pinned (the service-success fixture now carries the required attributes); plus a unit table for the heuristic's spellings. - live regression leg: `e2e_vendored_production.rs`'s gem leg is upgraded to `gem_bundler_vendored_install_proof` — a full fresh-dir frozen `bundle install` delivery proof with the failure tolerance and the `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT` knob deleted. It passes against real production TODAY via the auto fallback (the served stub is still invalid), and exercises the service artifact directly once the depscan fix deploys and the artifacts rebuild. The leg installs in bundler's deployment layout (`vendor/bundle` inside the project) so the crawler-visible install can feed the local-build fallback its stub gemspec. Verified live against production: scan --mode vendored applied=1 via the fallback with the `vendor_prebuilt_stub_invalid` event, vendored gemspec carries real summary/authors; --vendor-source service refuses with the new code and leaves no .socket/vendor. Stacked on #217 (test/gem-e2e-restore). Co-Authored-By: Claude Fable 5 --- crates/socket-patch-cli/CLI_CONTRACT.md | 11 +- .../tests/e2e_vendored_production.rs | 283 ++++++++++++------ crates/socket-patch-core/src/vendor/gem.rs | 263 +++++++++++++++- docs/testing/vendored-production-e2e.md | 35 ++- 4 files changed, 479 insertions(+), 113 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 100838d2..631da07e 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -397,6 +397,7 @@ per service outcome: | integrity mismatch | cargo/maven/nuget: **refuse** (`vendor_prebuilt_integrity_mismatch`) — tampered bytes never fall back; other ecosystems (to be aligned): local build + `vendor_prebuilt_integrity_mismatch` | refuse (cargo/maven/nuget: `vendor_prebuilt_integrity_mismatch`; others: `vendor_prebuilt_required`) | | still building (`pending_build` / serve 408) | local build + `vendor_prebuilt_pending` | refuse | | not built / withdrawn / not found / no usable artifact | local build (quiet) | refuse | +| gem stub gemspec missing / invalid | local build + `vendor_prebuilt_stub_missing` / `vendor_prebuilt_stub_invalid` | refuse (`vendor_prebuilt_required` / `vendor_prebuilt_stub_invalid`) | | 401 / 403 grant / 5xx / network error | local build + `vendor_prebuilt_unavailable` | refuse | | `--offline` | local build | refuse (`vendor_service_offline_conflict`) | @@ -416,7 +417,14 @@ needs an eval-able stub gemspec that the `.gem` archive doesn't carry in bundler alongside the `.gem`, and the gem backend downloads + integrity-verifies both. A served gem whose stub is missing (a native-extension gem, for which the converter emits no stub, or a patch built before the stub rollout) is treated as a service miss — `auto` falls back to the local build, -`service` refuses (`vendor_prebuilt_required`). For any ecosystem with no service path at all +`service` refuses (`vendor_prebuilt_required`). A served stub that is present but INVALID — it +never assigns the rubygems-required `summary`/`authors`, so every bundler major would reject the +vendored path source at install time (a defect the 2026-08-19 live matrix found in every +then-published gem stub) — follows the same miss policy under its own code (additive/MINOR): +`auto` falls back to the local build with a loud `vendor_prebuilt_stub_invalid` warning naming +the missing attributes, `service` refuses with `vendor_prebuilt_stub_invalid`. The check is a +conservative textual heuristic (assignment-line presence, no ruby parsing); a valid stub is +still written byte-verbatim. For any ecosystem with no service path at all `auto`/`build` build locally as before, and `service` refuses with `vendor_service_unsupported_ecosystem`. A successful service vend emits `vendor_prebuilt_downloaded`. Unrelated to `--download-mode` (which selects the patch-CONTENT format for the local build). @@ -894,6 +902,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_lock_checksums_unsupported` / `vendor_stale_lock_checksum` | `failed` | vendor (gem): an ambiguous/platform CHECKSUMS entry, or a v1-wired lock whose stale token blocks the hot path (run `vendor --revert` + re-vendor). | | `redirect_gem_stale_install` | `redirect.warnings[]` (warning) | scan `--mode hosted` (gem): a stale UNPATCHED materialization (installed gem, or committed `vendor/cache` archive) that `bundle install` will reuse instead of fetching the redirected patch; the detail carries the verified remedy. Full rules and flavors: the "Gem stale-install guard" section. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | +| `vendor_prebuilt_stub_invalid` | `failed` / `skipped` (warning) | vendor (gem, `--vendor-source`): the served stub gemspec never assigns the rubygems-required `summary`/`authors`, so bundler would refuse the vendored path source at install time. `service`: refusal naming the missing attributes; `auto`: loud warning + local-build fallback. | | `vendor_*` / `pypi_*` / `gemfile_*` / `lock_*` / `locked_version_mismatch` / `user_authored_*` / `native_extensions_unsupported` / `platform_gem_unsupported` | `failed`/`skipped` | vendor: per-ecosystem refusal + drift vocabulary; see the Vendor command contract section. New tags are additive (MINOR). | ### Top-level `EnvelopeError` codes diff --git a/crates/socket-patch-cli/tests/e2e_vendored_production.rs b/crates/socket-patch-cli/tests/e2e_vendored_production.rs index 41e0f036..ebbe588f 100644 --- a/crates/socket-patch-cli/tests/e2e_vendored_production.rs +++ b/crates/socket-patch-cli/tests/e2e_vendored_production.rs @@ -52,22 +52,22 @@ //! | npm | `pkg:npm/minimist@1.2.2` | `80630680-4da6-45f9-bba8-b888e0ffd58c` | `Socket Community Patch` header | //! | PyPI | `pkg:pypi/urllib3@1.26.18` | *any of three* (see [`PYPI_UUIDS`]) | `Socket Community Patch` header | //! | Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | advisory id `GHSA-pp8r-vv2j-9j5v` | -//! | gem | `pkg:gem/activestorage@6.0.3` | `15e960b5-f432-4b6c-b8aa-534a2b419323` | `Socket Community Patch` header *(not yet asserted — see below)* | +//! | gem | `pkg:gem/activestorage@6.0.3` | `15e960b5-f432-4b6c-b8aa-534a2b419323` | `Socket Community Patch` header | //! //! # Ecosystems with no full coverage, and why //! -//! * **gem** — the old `platform_gem_unsupported` refusal for -//! `?platform=ruby` purls was fixed in the CLI (#172 — only non-`ruby` -//! platform qualifiers are refused), and the 2026-08-18 catalog republish -//! restored the pinned patch, so the VENDOR itself now succeeds. -//! The full fresh-dir `bundle install` delivery proof is DEFERRED to the -//! stacked stub-hardening PR: the gem-stub-gemspec artifact production -//! serves is currently invalid (missing `summary`/`authors`), so rubygems -//! validation rejects the vendored `path:` source and `bundle install` -//! exits 1 on every bundler major (discovered 2026-08-19). -//! [`gem_bundler_vendored_known_platform_defect`] asserts redirect+download -//! hard and reports the vendor success. Promote with -//! `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT=1`. +//! * **gem** — RESOLVED: full coverage. The old `platform_gem_unsupported` +//! refusal for `?platform=ruby` purls was fixed in the CLI (#172 — only +//! non-`ruby` platform qualifiers are refused), and the 2026-08-18 catalog +//! republish restored the pinned patch, so +//! [`gem_bundler_vendored_install_proof`] now runs the complete vendored +//! loop including the fresh-dir `bundle install` delivery proof. While +//! production's served `gem-stub-gemspec` remains invalid (D4: missing the +//! rubygems-required `summary`/`authors`), the leg passes via the CLI's +//! invalid-stub hardening — `--vendor-source auto` detects the defect and +//! falls back to the local build (`vendor_prebuilt_stub_invalid` warning); +//! once the server-side stub fix deploys and the artifacts rebuild, the +//! same leg exercises the service artifact directly. //! * **golang** — vendored mode *works* (directory `replace`), but production //! publishes no free golang patches, so there is nothing to vendor. //! [`golang_vendored_finds_no_free_patches`] asserts exactly that (zero @@ -1670,38 +1670,64 @@ fn cargo_package_block(lock_text: &str, name: &str) -> Option { } // =========================================================================== -// RubyGems — vendor succeeds; full install proof deferred (invalid stub gemspec) +// RubyGems — bundler `path:` source // =========================================================================== -/// The gem redirect+download are asserted hard, and the vendor itself now -/// SUCCEEDS: #172 fixed the `platform_gem_unsupported` gate (only non-empty, -/// non-`ruby` platform qualifiers are refused) and the 2026-08-18 catalog -/// republish restored the pinned patch, so the success branch below — the -/// "appears FIXED" NOTE — is the live path today and the failure-tolerance -/// branch is vestigial. +/// Full vendored install proof for RubyGems: `scan --mode vendored` resolves +/// the free activestorage patch, materializes the patched gem under +/// `.socket/vendor/gem//-/`, and lands the mandatory +/// pair edit (Gemfile exact pin + `path:`, lock `PATH` section + +/// `(= )!` DEPENDENCIES pin). The delivery proof copies ONLY the +/// committable files into a fresh dir and runs a frozen `bundle install` +/// against a fresh empty `BUNDLE_PATH`: activestorage MUST resolve from the +/// vendored path source (its dependencies legitimately come from +/// rubygems.org — a path source only pins the one gem), and the file the +/// patch rewrites must carry the `Socket Community Patch` header. /// -/// The upgrade to a full fresh-dir `bundle install` delivery proof is -/// DEFERRED to the stacked stub-hardening PR: the gem-stub-gemspec artifact -/// production serves is currently invalid (missing `summary`/`authors`), so -/// rubygems validation rejects the vendored `path:` source and -/// `bundle install` exits 1 on every bundler major (discovered 2026-08-19). -/// That PR carries the install proof as its regression test; until it lands, -/// `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT=1` still promotes any vendor -/// failure here to a hard failure. +/// # How the leg passes while production's served stub is invalid (D4) +/// +/// The `gem-stub-gemspec` artifact production currently serves omits the +/// rubygems-required `summary`/`authors`, so writing it verbatim would make +/// the frozen `bundle install` below exit 1 on every bundler major. The CLI's +/// invalid-stub hardening is what this leg regression-tests live: under the +/// default `--vendor-source auto` the scan detects the defective stub, warns +/// (`vendor_prebuilt_stub_invalid`), and falls back to the LOCAL build +/// (installed gem + locally derived stub), which installs green. That is also +/// why the leg installs in bundler's deployment layout (`vendor/bundle` +/// inside the project): the crawler only sees a project-local install, and +/// the fallback needs the install's `specifications/` stub. Once the +/// server-side stub fix (depscan) deploys and the artifacts rebuild, the same +/// leg exercises the service artifact directly — no test change needed. +/// +/// # History +/// +/// This leg used to tolerate a `platform_gem_unsupported` vendor refusal: +/// production publishes the gem purl platform-qualified (`?platform=ruby`) +/// and the old vendor gate refused every platform qualifier. #172 fixed the +/// gate to refuse only non-empty, non-`ruby` platforms, and the 2026-08-18 +/// catalog republish restored the pinned patch, so the leg was upgraded to +/// this full install proof per its own auto-retire NOTE. #[test] #[ignore = "live production API + real rubygems.org. Run with --ignored."] -fn gem_bundler_vendored_known_platform_defect() { - const LEG: &str = "gem_bundler_vendored_known_platform_defect"; +fn gem_bundler_vendored_install_proof() { + const LEG: &str = "gem_bundler_vendored_install_proof"; if !has_command("ruby") || !has_command("bundle") { soft_skip!(LEG, "`ruby` and/or `bundle` not on PATH"); } let tmp = tempfile::tempdir().expect("tempdir"); let proj = tmp.path().join("proj"); std::fs::create_dir_all(&proj).expect("mkdir proj"); - let bundle_path = tmp.path().join("bundle").display().to_string(); + // Deployment layout — `vendor/bundle` INSIDE the project: the gem crawler + // probes `vendor/bundle//*/gems/` in local mode, and the vendor + // backend's local-build fallback needs the crawler-visible install (the + // `specifications/` sibling carries the local stub gemspec) while + // production's served stub is invalid (D4). The app config stays OUTSIDE + // the project so no `.bundle/config` joins the committable set. + let bundle_path = proj.join("vendor/bundle").display().to_string(); + let bundle_config = tmp.path().join("bundle-config").display().to_string(); let env = [ ("BUNDLE_PATH", bundle_path.as_str()), - ("BUNDLE_APP_CONFIG", bundle_path.as_str()), + ("BUNDLE_APP_CONFIG", bundle_config.as_str()), ]; std::fs::write( @@ -1718,77 +1744,148 @@ fn gem_bundler_vendored_known_platform_defect() { soft_skip!(LEG, "upstream `bundle install` failed:\n{}", dump(&install)); } - // Raw invocation: this run is EXPECTED to exit non-zero on the known defect. - let (code, stdout, stderr) = run_socket( - &proj, - &[ - "scan", - "--json", - "--yes", - "--mode", - "vendored", - "--cwd", - proj.to_str().unwrap(), - ], + // Anti-vacuity: the upstream install must be pristine. `bundle info + // --path` reports the exact directory bundler resolved for the gem. + let info = tool(&proj, "bundle", &["info", GEM_NAME, "--path"], &env); + assert!( + ok(&info), + "{LEG}: `bundle info {GEM_NAME} --path` failed after the upstream install:\n{}", + dump(&info) + ); + let installed_dir = PathBuf::from(String::from_utf8_lossy(&info.stdout).trim()); + assert_pristine( + &installed_dir.join("lib/active_storage/service/s3_service.rb"), + PATCH_MARKER, + LEG, ); - let env_json: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|e| { - panic!("{LEG}: scan --mode vendored did not emit JSON ({e}).\nstdout:\n{stdout}\nstderr:\n{stderr}") - }); + let gemfile_before = std::fs::read(proj.join("Gemfile")).unwrap(); + let lock_before = std::fs::read(proj.join("Gemfile.lock")).unwrap(); - // The download must have succeeded regardless — that half is not defective. + let env_json = scan_vendored(&proj, &[]); assert_download_uuid(&env_json, &[GEM_UUID], LEG); - let dl_failed = env_json["download"]["failed"].as_u64().unwrap_or(0); assert_eq!( - dl_failed, 0, - "{LEG}: the gem patch download itself failed — that is a regression, not the known \ - vendor-platform defect.\nenvelope:\n{env_json:#}" + env_json["download"]["failed"].as_u64().unwrap_or(99), + 0, + "{LEG}: the gem patch download failed.\nenvelope:\n{env_json:#}" ); + assert_vendor_applied(&env_json, &format!("{GEM_NAME}@{GEM_VERSION}"), LEG); - let gem_strict = std::env::var("SOCKET_PATCH_VENDORED_E2E_GEM_STRICT") - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) - .unwrap_or(false); + // The committable artifact + the mandatory pair edit. + let copy_rel = format!(".socket/vendor/gem/{GEM_UUID}/{GEM_NAME}-{GEM_VERSION}"); + assert_patched( + &proj + .join(©_rel) + .join("lib/active_storage/service/s3_service.rb"), + PATCH_MARKER, + LEG, + ); + let gemfile = read(&proj.join("Gemfile")); + assert!( + gemfile.contains(&format!( + "gem \"{GEM_NAME}\", \"{GEM_VERSION}\", path: \"{copy_rel}\"" + )), + "{LEG}: Gemfile line not rewritten to the exact-pin + path: form:\n{gemfile}" + ); + let lock = read(&proj.join("Gemfile.lock")); + assert!( + lock.contains(&format!( + "PATH\n remote: {copy_rel}\n specs:\n {GEM_NAME} ({GEM_VERSION})" + )), + "{LEG}: canonical PATH section missing from Gemfile.lock:\n{lock}" + ); + assert!( + lock.contains(&format!("\n {GEM_NAME} (= {GEM_VERSION})!")), + "{LEG}: DEPENDENCIES pin ` {GEM_NAME} (= {GEM_VERSION})!` missing from Gemfile.lock:\n{lock}" + ); + + // DELIVERY PROOF: ONLY the committable files (Gemfile, Gemfile.lock, + // .socket/ — the leg's BUNDLE_APP_CONFIG lives outside the project, so + // there is no .bundle/config to commit), a fresh EMPTY BUNDLE_PATH, and + // a frozen `bundle install`. Frozen mode makes bundler enforce the + // committed lock — including its vendored PATH source — and fail rather + // than re-resolve, mirroring docker_e2e_vendor_gem's stage 2. The gem's + // dependencies come from rubygems.org (a path source pins only the one + // gem), so the install is online; the point is that activestorage itself + // can only come from `.socket/vendor/`. + let fresh = tmp.path().join("fresh"); + std::fs::create_dir_all(&fresh).unwrap(); + std::fs::copy(proj.join("Gemfile"), fresh.join("Gemfile")).unwrap(); + std::fs::copy(proj.join("Gemfile.lock"), fresh.join("Gemfile.lock")).unwrap(); + copy_dir_recursive(&proj.join(".socket"), &fresh.join(".socket")); + let fresh_bundle = tmp.path().join("fresh-bundle").display().to_string(); + let fresh_env = [ + ("BUNDLE_PATH", fresh_bundle.as_str()), + ("BUNDLE_APP_CONFIG", fresh_bundle.as_str()), + ("BUNDLE_FROZEN", "true"), + ]; + let lock_committed = std::fs::read(fresh.join("Gemfile.lock")).unwrap(); + let fresh_install = tool(&fresh, "bundle", &["install"], &fresh_env); + assert!( + ok(&fresh_install), + "{LEG}: frozen `bundle install` from the committable files failed:\n{}", + dump(&fresh_install) + ); + assert_eq!( + std::fs::read(fresh.join("Gemfile.lock")).unwrap(), + lock_committed, + "{LEG}: frozen `bundle install` churned the committed Gemfile.lock" + ); + // Bundler must have resolved the gem FROM the vendored path source, and + // the bytes it will load must carry the patch marker. + let fresh_info = tool(&fresh, "bundle", &["info", GEM_NAME, "--path"], &fresh_env); + assert!( + ok(&fresh_info), + "{LEG}: `bundle info {GEM_NAME} --path` failed after the fresh install:\n{}", + dump(&fresh_info) + ); + let resolved = String::from_utf8_lossy(&fresh_info.stdout) + .trim() + .to_string(); + assert!( + resolved.contains(©_rel), + "{LEG}: bundler resolved {GEM_NAME} from `{resolved}`, not the vendored \ + path `{copy_rel}` — the pair edit did not take effect in the fresh dir" + ); + assert_patched( + &PathBuf::from(&resolved).join("lib/active_storage/service/s3_service.rb"), + PATCH_MARKER, + LEG, + ); - let applied = env_json["vendor"]["summary"]["applied"] - .as_u64() - .unwrap_or(0); - if code == 0 && applied >= 1 { - // The CLI now vendors platform gems. Say so loudly — this leg should be - // promoted to a full delivery proof (bundle install frozen) and this - // tolerance branch deleted. - println!( - "NOTE {LEG}: `scan --mode vendored` now SUCCEEDS for {GEM_PURL} (applied={applied}). \ - The platform_gem_unsupported vendor gap appears FIXED — upgrade this leg to a full \ - fresh-checkout `bundle install` delivery proof and delete the tolerance branch." - ); - return; - } + // Idempotency + revert (mirrors the pip/uv legs). + let gemfile_wired = std::fs::read(proj.join("Gemfile")).unwrap(); + let lock_wired = std::fs::read(proj.join("Gemfile.lock")).unwrap(); + let env2 = scan_vendored(&proj, &[]); + assert_eq!( + env2["vendor"]["summary"]["applied"].as_u64().unwrap_or(99), + 0, + "{LEG}: re-run must vendor nothing new:\n{env2:#}" + ); + assert_eq!( + std::fs::read(proj.join("Gemfile")).unwrap(), + gemfile_wired, + "{LEG}: re-run must leave the Gemfile byte-identical" + ); + assert_eq!( + std::fs::read(proj.join("Gemfile.lock")).unwrap(), + lock_wired, + "{LEG}: re-run must leave Gemfile.lock byte-identical" + ); - // Otherwise: it must be exactly the known `platform_gem_unsupported` failure. - let events = env_json["vendor"]["events"] - .as_array() - .cloned() - .unwrap_or_default(); - let is_known = events - .iter() - .any(|e| e["action"] == "failed" && e["errorCode"] == "platform_gem_unsupported"); - assert!( - !gem_strict, - "{LEG}: SOCKET_PATCH_VENDORED_E2E_GEM_STRICT=1 and vendoring {GEM_PURL} did not succeed \ - (exit {code}).\nenvelope:\n{env_json:#}\nstderr:\n{stderr}" + assert_eq!(vendor_revert(&proj, LEG), 1, "{LEG}: one entry reverted"); + assert_eq!( + std::fs::read(proj.join("Gemfile")).unwrap(), + gemfile_before, + "{LEG}: revert must restore the Gemfile byte-identical" + ); + assert_eq!( + std::fs::read(proj.join("Gemfile.lock")).unwrap(), + lock_before, + "{LEG}: revert must restore Gemfile.lock byte-identical" ); assert!( - is_known, - "{LEG}: `scan --mode vendored` failed for {GEM_PURL}, but NOT with the known \ - `platform_gem_unsupported` vendor code — this is a new regression (exit {code}).\n\ - envelope:\n{env_json:#}\nstderr:\n{stderr}" - ); - println!( - "KNOWN CLI GAP {LEG}: `scan --mode vendored` downloads the {GEM_NAME} patch but the \ - vendor backend refuses the platform-qualified purl \ - (pkg:gem/{GEM_NAME}@{GEM_VERSION}?platform=ruby) with `platform_gem_unsupported`. \ - Redirect+download asserted; the delivery proof is blocked until the CLI learns to \ - vendor platform gems. This leg starts asserting a real install automatically once the \ - vendor succeeds." + !proj.join(".socket/vendor").exists(), + "{LEG}: .socket/vendor must be gone after revert" ); } diff --git a/crates/socket-patch-core/src/vendor/gem.rs b/crates/socket-patch-core/src/vendor/gem.rs index de369999..a0f4326b 100644 --- a/crates/socket-patch-core/src/vendor/gem.rs +++ b/crates/socket-patch-core/src/vendor/gem.rs @@ -667,7 +667,10 @@ enum GemServiceCopy { /// emits no stub — bundler can't build extensions for a path source) or a gem /// patch built before the stub rollout (the invalidation migration rebuilds /// those). The downloaded stub is re-checked for native extensions as defense -/// in depth. +/// in depth, and an INVALID stub — one missing the rubygems-required +/// `summary`/`authors` assignments, the D4 served-artifact defect — follows +/// the same miss policy under its own `vendor_prebuilt_stub_invalid` code +/// (always loud, even under `auto`). async fn gem_service_copy( service: Option<&VendorServiceConfig>, record: &PatchRecord, @@ -773,6 +776,48 @@ async fn gem_service_copy( ); } + // Defense in depth (D4, gem live matrix 2026-08-19): production's stub + // generator omitted the rubygems-required `summary`/`authors`, and every + // bundler major validates path-source gemspecs — writing such a stub + // verbatim makes every later `bundle install` exit 1 (`missing value for + // attribute summary`). An INVALID stub follows the MISSING-stub policy + // (fall back under `auto`, refuse under `service`) but under its own + // `vendor_prebuilt_stub_invalid` code, and always loudly — the served + // artifact is defective, not merely absent. Nothing has been written yet, + // so the refusal leaves no partial artifacts. + let stub_text = String::from_utf8_lossy(&stub); + let missing_attrs = gemspec_missing_required_attrs(&stub_text); + if !missing_attrs.is_empty() { + let licenses_note = if gemspec_assigns_attr(&stub_text, "licenses") + || gemspec_assigns_attr(&stub_text, "license") + { + "" + } else { + " (it also omits `licenses`, a rubygems warning)" + }; + let reason = format!( + "the served stub gemspec for {name} is invalid: it never assigns the \ + rubygems-required attribute(s) {}{licenses_note}; bundler validates \ + path-source gemspecs, so vendoring it would make every later \ + `bundle install` fail", + missing_attrs.join(", "), + ); + if cfg.source.requires_service() { + return hard( + "vendor_prebuilt_stub_invalid", + format!( + "{reason}. Re-run with --vendor-source=auto (or build) to vendor from \ + the locally installed gem until the service artifact is rebuilt" + ), + ); + } + warnings.push(VendorWarning::new( + "vendor_prebuilt_stub_invalid", + format!("{reason}; building locally instead"), + )); + return GemServiceCopy::FallBack; + } + // Extract the patched `.gem`'s data.tar.gz into a clean copy dir, then add // the stub as `.gemspec` (a `.gem`'s data.tar.gz never carries one — // the gemspec lives in metadata.gz). @@ -2193,6 +2238,56 @@ fn gemspec_declares_extensions(spec_text: &str) -> bool { false } +/// Textual heuristic: does any line (comment-stripped) assign a NON-EMPTY +/// value to `.{attr}`? Same conservative bar as +/// [`gemspec_declares_extensions`] — no real ruby parsing. "Non-empty" filters +/// the obviously-empty spellings (`""`, `''`, `[]`, `nil`, any of them +/// `.freeze`-d); anything else counts, so a legitimate assignment can never be +/// missed by over-cleverness. +fn gemspec_assigns_attr(spec_text: &str, attr: &str) -> bool { + let needle = format!(".{attr}"); + for raw in spec_text.lines() { + let line = raw.split('#').next().unwrap_or(""); + if let Some(idx) = line.find(&needle) { + let after = line[idx + needle.len()..].trim_start(); + if after.starts_with('=') && !after.starts_with("==") { + let value: String = after[1..] + .replace(".freeze", "") + .chars() + .filter(|c| !c.is_whitespace() && !matches!(c, '"' | '\'' | '[' | ']' | ',')) + .collect(); + if !value.is_empty() && value != "nil" { + return true; + } + } + } + } + false +} + +/// The rubygems-REQUIRED attributes a served stub gemspec must assign for +/// bundler to accept it as a path source, returned as the list it is missing +/// (empty = valid). Every bundler major validates path-source gemspecs and +/// hard-fails on a nil `summary` or `authors` (`missing value for attribute +/// summary`), so a stub without them bricks every later `bundle install` — +/// the D4 defect the 2026-08-19 gem live matrix found in ALL served stubs. +/// `authors` also accepts rubygems' singular `author =` alias. A missing +/// `licenses` is only a rubygems WARNING, so it is deliberately not checked +/// here (callers may mention it in advisory text via +/// [`gemspec_assigns_attr`]). Fail-open by construction: only a stub where +/// the assignment lines are demonstrably absent is flagged, so a legitimate +/// stub always passes and is written byte-verbatim. +fn gemspec_missing_required_attrs(spec_text: &str) -> Vec<&'static str> { + let mut missing = Vec::new(); + if !gemspec_assigns_attr(spec_text, "summary") { + missing.push("summary"); + } + if !gemspec_assigns_attr(spec_text, "authors") && !gemspec_assigns_attr(spec_text, "author") { + missing.push("authors"); + } + missing +} + #[cfg(test)] mod tests { use super::*; @@ -2529,6 +2624,58 @@ mod tests { assert!(!root.join(".socket").exists()); } + /// The required-attribute heuristic ([`gemspec_missing_required_attrs`]): + /// flag ONLY a demonstrably-absent (or demonstrably-empty) assignment — + /// a legitimate stub must always pass, whatever its spelling. + #[test] + fn required_attrs_heuristic() { + // The D4 production shape: no summary, no authors. + assert_eq!( + gemspec_missing_required_attrs( + "Gem::Specification.new do |s|\n s.name = \"rack\".freeze\n s.version = \"3.2.6\".freeze\n s.require_paths = [\"lib\".freeze]\nend\n" + ), + vec!["summary", "authors"] + ); + // A real converter/to_ruby stub: `.freeze`-d scalar + array. Valid. + assert_eq!( + gemspec_missing_required_attrs( + "Gem::Specification.new do |s|\n s.summary = \"web server interface\".freeze\n s.authors = [\"A. Person\".freeze, \"B. Person\".freeze]\nend\n" + ), + Vec::<&str>::new() + ); + // Alternate spellings a valid stub may use: another block variable, + // no space around `=`, the singular `author =` alias, %w arrays. + assert_eq!( + gemspec_missing_required_attrs( + "Gem::Specification.new do |spec|\n spec.summary=\"x\"\n spec.author = \"A. Person\"\nend\n" + ), + Vec::<&str>::new() + ); + assert_eq!( + gemspec_missing_required_attrs("s.summary = \"x\"\ns.authors = %w[alice bob]\n"), + Vec::<&str>::new() + ); + // One present, one absent → only the absent one is named. + assert_eq!( + gemspec_missing_required_attrs("s.summary = \"x\".freeze\n"), + vec!["authors"] + ); + // Demonstrably-empty spellings count as missing… + assert_eq!( + gemspec_missing_required_attrs("s.summary = \"\".freeze\ns.authors = [].freeze\n"), + vec!["summary", "authors"] + ); + assert_eq!( + gemspec_missing_required_attrs("s.summary = nil\ns.authors = [\"\"]\n"), + vec!["summary", "authors"] + ); + // …and so do commented-out assignments and `==` comparisons. + assert_eq!( + gemspec_missing_required_attrs("# s.summary = \"x\"\nraise if s.authors == [\"x\"]\n"), + vec!["summary", "authors"] + ); + } + #[tokio::test] async fn test_refuses_native_extensions() { let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; @@ -4002,10 +4149,17 @@ mod tests { use crate::api::client::{ApiClient, ApiClientOptions}; use crate::vendor::VendorSource; - /// A valid path-source stub (no native extensions). - const SERVICE_STUB: &[u8] = b"# -*- encoding: utf-8 -*-\n# stub: rack 3.2.6 ruby lib\n\nGem::Specification.new do |s|\n s.name = \"rack\".freeze\n s.version = \"3.2.6\".freeze\n s.require_paths = [\"lib\".freeze]\nend\n"; + /// A valid path-source stub (no native extensions; assigns the + /// rubygems-required `summary` + `authors`, which every bundler major + /// validates on path-source gemspecs). + const SERVICE_STUB: &[u8] = b"# -*- encoding: utf-8 -*-\n# stub: rack 3.2.6 ruby lib\n\nGem::Specification.new do |s|\n s.name = \"rack\".freeze\n s.version = \"3.2.6\".freeze\n s.summary = \"a modular Ruby web server interface\".freeze\n s.authors = [\"Rack maintainers\".freeze]\n s.licenses = [\"MIT\".freeze]\n s.require_paths = [\"lib\".freeze]\nend\n"; /// A stub that declares native extensions (must be refused). const SERVICE_STUB_NATIVE: &[u8] = b"Gem::Specification.new do |s|\n s.name = \"rack\".freeze\n s.version = \"3.2.6\".freeze\n s.extensions = [\"ext/rack/extconf.rb\"]\nend\n"; + /// The DEFECTIVE stub shape production served as of 2026-08-19 (gem + /// live-matrix defect D4): it never assigns the rubygems-required + /// `summary` / `authors` (nor `licenses`), so bundler's path-source + /// validation rejects it and every post-vendor `bundle install` exits 1. + const SERVICE_STUB_INVALID: &[u8] = b"# -*- encoding: utf-8 -*-\n# stub: rack 3.2.6 ruby lib\n\nGem::Specification.new do |s|\n s.name = \"rack\".freeze\n s.version = \"3.2.6\".freeze\n s.require_paths = [\"lib\".freeze]\nend\n"; fn sri_sha512(bytes: &[u8]) -> String { use base64::Engine as _; @@ -4146,8 +4300,10 @@ mod tests { } /// Service success: the prebuilt `.gem` is extracted into the copy dir, the - /// served stub is written as `rack.gemspec`, the Gemfile + lock are wired, - /// and a `vendor_prebuilt_downloaded` advisory is emitted — WITHOUT a local + /// served stub is written as `rack.gemspec` BYTE-VERBATIM (a valid stub — + /// one assigning `summary`/`authors` — must pass the required-attribute + /// validation untouched), the Gemfile + lock are wired, and a + /// `vendor_prebuilt_downloaded` advisory is emitted — WITHOUT a local /// install (a deliberately-missing `installed_dir`). #[tokio::test] async fn service_success_extracts_gem_and_wires_lock() { @@ -4333,6 +4489,103 @@ mod tests { ); } + /// D4 (gem live-matrix 2026-08-19): explicit `service` mode + a served stub + /// that never assigns the rubygems-required `summary`/`authors` refuses + /// with its own `vendor_prebuilt_stub_invalid` code, naming the missing + /// attributes — writing it verbatim would make every later `bundle install` + /// exit 1 (all bundler majors validate path-source gemspecs). No partial + /// artifacts are left and the lock is untouched. + #[tokio::test] + async fn service_stub_invalid_service_mode_hard_fails() { + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let gem = make_gem(&[("lib/rack.rb", PATCHED)]); + let sri = sri_sha512(&gem); + let stub_sri = sri_sha512(SERVICE_STUB_INVALID); + let server = wiremock::MockServer::start().await; + mount_gem_granted(&server, &gem, &sri, Some((SERVICE_STUB_INVALID, &stub_sri))).await; + let sources = PatchSources::blobs_only(&blobs); + + let outcome = vendor_gem( + PURL, + &installed, + &root, + &record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&gem_service_cfg( + &server.uri(), + VendorSource::Service, + false, + )), + ) + .await; + let (code, detail) = unwrap_refused(outcome); + assert_eq!(code, "vendor_prebuilt_stub_invalid"); + assert!( + detail.contains("summary") && detail.contains("authors"), + "the refusal must name the missing attributes: {detail}" + ); + assert!(!root.join(format!(".socket/vendor/gem/{UUID}")).exists()); + // The lock is untouched. + assert_eq!( + tokio::fs::read_to_string(root.join(GEMFILE_LOCK)) + .await + .unwrap(), + LOCK_DIRECT + ); + } + + /// D4 under the default `auto`: an INVALID served stub is treated exactly + /// like a MISSING one — fall back to the LOCAL build (installed gem + + /// locally derived stub) — but with a LOUD `vendor_prebuilt_stub_invalid` + /// warning naming the served-stub defect. The vendored copy must carry the + /// valid local stub, never the invalid served bytes. + #[tokio::test] + async fn service_stub_invalid_auto_falls_back_to_build() { + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let gem = make_gem(&[("lib/rack.rb", PATCHED)]); + let sri = sri_sha512(&gem); + let stub_sri = sri_sha512(SERVICE_STUB_INVALID); + let server = wiremock::MockServer::start().await; + mount_gem_granted(&server, &gem, &sri, Some((SERVICE_STUB_INVALID, &stub_sri))).await; + let sources = PatchSources::blobs_only(&blobs); + + let outcome = vendor_gem( + PURL, + &installed, + &root, + &record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&gem_service_cfg(&server.uri(), VendorSource::Auto, false)), + ) + .await; + let (result, entry, warnings) = unwrap_done(outcome); + assert!(result.success, "auto must fall back: {:?}", result.error); + assert!(entry.is_some()); + assert_eq!(tokio::fs::read(copy_lib(&root)).await.unwrap(), PATCHED); + assert_eq!( + tokio::fs::read_to_string(copy_gemspec(&root)) + .await + .unwrap(), + GEMSPEC, + "the vendored gemspec must be the LOCAL stub, not the invalid served bytes" + ); + let warning = warnings + .iter() + .find(|w| w.code == "vendor_prebuilt_stub_invalid") + .expect("auto fallback must warn loudly about the invalid served stub"); + assert!( + warning.detail.contains("summary") && warning.detail.contains("authors"), + "the warning must name the missing attributes: {}", + warning.detail + ); + } + /// `auto` + a not-built service status falls back to the local build. #[tokio::test] async fn service_unavailable_auto_falls_back_to_build() { diff --git a/docs/testing/vendored-production-e2e.md b/docs/testing/vendored-production-e2e.md index 5667713b..4fcd6939 100644 --- a/docs/testing/vendored-production-e2e.md +++ b/docs/testing/vendored-production-e2e.md @@ -68,7 +68,7 @@ hosted suite). | pip (requirements.txt) | urllib3@1.26.18 | `pip install --no-index -r requirements.txt` | ✅ full | | uv (uv.lock) | urllib3@1.26.18 | `uv sync --frozen --offline` | ✅ full | | cargo (`[patch.crates-io]`) | traitobject@0.1.1 | `cargo fetch --offline --locked` (see note) | ✅ full | -| bundler | activestorage@6.0.3 | — (deferred: production's served gem-stub gemspec is invalid, see below) | ⚠️ vendor succeeds; delivery proof deferred to the stub-hardening fix PR | +| bundler | activestorage@6.0.3 | frozen `bundle install`, fresh empty `BUNDLE_PATH` | ✅ full | | go | — | — | zero-patch assertion (no free golang patches) | | deno | — | — | negative assertion (unsupported) | | maven / nuget / composer | — | — | canary (no free production patches) | @@ -84,7 +84,8 @@ directory. ## Known issues this suite surfaced All were found against real production + real toolchains; none is a test bug. -The first two are fixed; the third is open with a stacked fix PR pending. +The first two are fixed; the third is mitigated CLI-side (the served artifact +is still defective server-side). ### 1. `pnpm` >= 11 — vendored `overrides` land in the wrong file (CLI) — FIXED @@ -118,21 +119,28 @@ vendors like a bare purl. The suite's original pin (`activestorage@7.0.2.2` / `2535d43d-67ce-4944-be27-c19e113997fb`) was withdrawn on 2026-08-14; the 2026-08-18 catalog republish REPLACED it, and the suite was re-pinned to `activestorage@6.0.3` / -`15e960b5-f432-4b6c-b8aa-534a2b419323`. The vendor now succeeds live and the -leg's failure-tolerance branch is vestigial. The leg's upgrade to a full fresh-dir -`bundle install` delivery proof is deferred to the stacked stub-hardening fix -PR because of issue 3 below; until then the leg keeps its tolerant shape and -its `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT` knob. +`15e960b5-f432-4b6c-b8aa-534a2b419323`. The vendor succeeds live and the leg +was upgraded to the full fresh-dir `bundle install` delivery proof +(`gem_bundler_vendored_install_proof`), retiring its failure-tolerance branch +and its `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT` knob. -### 3. `gem` — the served gem-stub gemspec is invalid (SERVER + CLI hardening) — OPEN +### 3. `gem` — the served gem-stub gemspec is invalid (SERVER; CLI mitigated) Discovered 2026-08-19 while upgrading the gem leg to a full delivery proof: the gem-stub-gemspec artifact production serves is invalid — it is missing -`summary`/`authors`, which rubygems validation requires — so bundler rejects -the vendored `path:` source and `bundle install` exits 1 on every bundler -major. The fix (server-side stub correction plus CLI-side hardening) lands in -a separate stacked PR; that PR carries the full -fresh-dir-frozen-`bundle install` install proof as its regression test. +`summary`/`authors`, which rubygems validation requires — so writing it +verbatim makes bundler reject the vendored `path:` source and +`bundle install` exit 1 on every bundler major. + +**Mitigated CLI-side**: the gem vendor backend now validates the served stub +(conservative textual check for the required assignment lines). +`--vendor-source auto` detects the defect, warns +(`vendor_prebuilt_stub_invalid`), and falls back to the local build — which is +how `gem_bundler_vendored_install_proof` passes against production today — +while explicit `--vendor-source service` refuses with +`vendor_prebuilt_stub_invalid`. The server-side stub-generator fix plus the +rebuild of all published gem artifacts are tracked in depscan; once deployed, +the same leg exercises the service artifact directly. ## Running @@ -154,7 +162,6 @@ installs from contending on the shared cache sandbox. | Variable | Effect | |----------|--------| | `SOCKET_PATCH_VENDORED_E2E_STRICT=1` | Turn every "toolchain missing" soft-skip into a hard failure. | -| `SOCKET_PATCH_VENDORED_E2E_GEM_STRICT=1` | Promote any gem vendor failure to a hard failure (the leg's tolerance branch is vestigial since #172; the knob stays until the stub-hardening PR lands the full install proof). | | `SOCKET_PATCH_VENDORED_E2E_CANARY_STRICT=1` | Fail when maven/nuget/composer gain their first free published patch. | The suite forces `SOCKET_NO_CONFIG=true` and scrubs every ambient `SOCKET_*` From 80e610cc82eae3a98929ffd13d813b1bc5466a5d Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Wed, 19 Aug 2026 14:52:02 -0400 Subject: [PATCH 2/2] =?UTF-8?q?fix(vendor):=20review=20round=20=E2=80=94?= =?UTF-8?q?=20gem-home=20guard,=20empirical=20rubygems=20bar,=20both-arm?= =?UTF-8?q?=20stub=20validation,=20truthful=20dead-ends,=20on-disk=20heal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial-review fix round for the D4 stub-hardening PR. SECURITY (1): the local-stub derivation walked two parents up from installed_dir unconditionally. For a registry auto-fetch staging dir (/-) that escapes into the SHARED temp root, making $TMPDIR/specifications/.gemspec a predictable, attacker-plantable path whose contents would be committed and eval'd as Ruby by every later `bundle install`. The derivation now requires installed_dir's parent to be a literal `gems/` dir (a real gem-home layout); staging dirs have no local stub. Test plants a valid spec at the old derivation target and proves it is never consumed. Scanner rewrite (2): comment-stripping at `#` truncated inside string literals (s.summary = "#1 Ruby web server" judged missing → valid stubs refused under service). The scanner now examines raw lines anchored to the line start (receiver ident + .attr + assignment), where a preceding comment marker is impossible. The emptiness policy is now EMPIRICAL, verified against rubygems 3.3/3.5/3.6 in the bundler 1.17/2.7/4.0 era images: summary hard-fails only when never assigned (nil/"" are writer-coerced, warning at most); authors hard-fails when never assigned or collapsing to no String elements ([], nil, [nil], %w[] — while [""] passes). The old code refused rubygems-tolerated stubs and passed %w[] which hard-fails. Both-arm validation (3): the local-build arm wrote the local stub verbatim; it now validates at the same write choke point and refuses `gem_spec_invalid` naming the file (new CLI_CONTRACT vocabulary row). The GEMSPEC/GEMSPEC_318/GEMSPEC_PUMA and CLI-suite fixtures now carry summary+authors like every healthy rubygems-written stub. Truthful dead-end (4): auto + invalid served stub + gem not installed used to refuse gem_spec_missing with circular advice ("use --vendor-source=service" <-> service says "use auto") and the D4 diagnostic never reached the envelope (Refused carries no warnings). The FallBack variant now carries the served-stub defect; the refusal is `vendor_prebuilt_stub_invalid`, names the defect, and advises installing the gem. Tests cover auto and service, both not-installed. Heal existing victims (5): the idempotent hot path only checked the vendored gemspec EXISTS, silently re-blessing pre-fix invalid stubs. copy_ok now re-validates the on-disk stub; invalid routes into the existing artifact-only rebuild (test: pre-seeded invalid stub on disk → re-scan rewrites a valid one, pair edit + ledger untouched). Dedupe (11 + addendum B): one shared attr_mention line-scanner under both gemspec_declares_extensions and the attr checks; the miss closure widened with the (hard code, remedy) pair instead of a re-implemented branch; licenses/license alias fan-out folded into the alias-list helper; stub_text bound once. e2e leg robustness (6-10): bundler invocations scrub ambient BUNDLE_*/GEM_*/RUBYOPT (sibling-suite pattern) and set USE_FREEDESKTOP_PLACEHOLDER=true (mimemagic shared-mime-info hazard, mirrors #217 round 2); delivery proof asserts canonicalized starts_with(fresh-dir) provenance and compares installed bytes against captured pristine bytes; a route-attribution assertion requires exactly one of {vendor_prebuilt_downloaded, vendor_prebuilt_stub_invalid} so the leg auto-retires the fallback expectation when the depscan stub fix deploys; applied/failed/idempotency/revert assertions are scoped to the activestorage purl so future catalog additions cannot red the leg; stale doc comments fixed. Contract (addendum A): documented why the service-mode refusal on an invalid stub rides a MINOR — the prior exit-0 wrote a stub bundler rejects (an uninstallable project); the refusal is the bug fix. Rebased on test/gem-e2e-restore @ 63531d9 (PR #217 review round 2). Co-Authored-By: Claude Fable 5 --- crates/socket-patch-cli/CLI_CONTRACT.md | 29 +- .../tests/e2e_vendored_production.rs | 175 ++++- .../tests/in_process_vendor.rs | 2 +- .../tests/repair_vendor_e2e.rs | 7 +- crates/socket-patch-core/src/vendor/gem.rs | 612 +++++++++++++++--- docs/testing/vendored-production-e2e.md | 20 +- 6 files changed, 702 insertions(+), 143 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 631da07e..75fef147 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -397,7 +397,7 @@ per service outcome: | integrity mismatch | cargo/maven/nuget: **refuse** (`vendor_prebuilt_integrity_mismatch`) — tampered bytes never fall back; other ecosystems (to be aligned): local build + `vendor_prebuilt_integrity_mismatch` | refuse (cargo/maven/nuget: `vendor_prebuilt_integrity_mismatch`; others: `vendor_prebuilt_required`) | | still building (`pending_build` / serve 408) | local build + `vendor_prebuilt_pending` | refuse | | not built / withdrawn / not found / no usable artifact | local build (quiet) | refuse | -| gem stub gemspec missing / invalid | local build + `vendor_prebuilt_stub_missing` / `vendor_prebuilt_stub_invalid` | refuse (`vendor_prebuilt_required` / `vendor_prebuilt_stub_invalid`) | +| gem stub gemspec missing / invalid | local build + `vendor_prebuilt_stub_missing` / `vendor_prebuilt_stub_invalid` (invalid + gem not installed: refuse `vendor_prebuilt_stub_invalid` — no stub source exists) | refuse (`vendor_prebuilt_required` / `vendor_prebuilt_stub_invalid`) | | 401 / 403 grant / 5xx / network error | local build + `vendor_prebuilt_unavailable` | refuse | | `--offline` | local build | refuse (`vendor_service_offline_conflict`) | @@ -418,13 +418,23 @@ alongside the `.gem`, and the gem backend downloads + integrity-verifies both. A stub is missing (a native-extension gem, for which the converter emits no stub, or a patch built before the stub rollout) is treated as a service miss — `auto` falls back to the local build, `service` refuses (`vendor_prebuilt_required`). A served stub that is present but INVALID — it -never assigns the rubygems-required `summary`/`authors`, so every bundler major would reject the -vendored path source at install time (a defect the 2026-08-19 live matrix found in every -then-published gem stub) — follows the same miss policy under its own code (additive/MINOR): -`auto` falls back to the local build with a loud `vendor_prebuilt_stub_invalid` warning naming -the missing attributes, `service` refuses with `vendor_prebuilt_stub_invalid`. The check is a -conservative textual heuristic (assignment-line presence, no ruby parsing); a valid stub is -still written byte-verbatim. For any ecosystem with no service path at all +fails the rubygems `summary`/`authors` bar, so every bundler major would reject the vendored +path source at install time (a defect the 2026-08-19 live matrix found in every then-published +gem stub) — follows the same miss policy under its own code (additive/MINOR): `auto` falls back +to the local build with a loud `vendor_prebuilt_stub_invalid` warning naming the missing +attributes, `service` refuses with `vendor_prebuilt_stub_invalid`. (Semver note: before the +hardening, `service` mode exited 0 here while writing a stub bundler rejects — an UNINSTALLABLE +project. The refusal is the bug fix; the exit-0 was the defect, so this rides a MINOR.) When the +invalid-stub fallback finds the gem is ALSO not installed locally (no `specifications/` stub to +derive), the vendor refuses with the same `vendor_prebuilt_stub_invalid` code, naming the served +defect and the install-the-gem remedy. The locally-derived stub is validated at the same write +choke point: a corrupted local `specifications/` stub failing the bar refuses with +`gem_spec_invalid` naming the file. The bar is a conservative textual heuristic matched to what +rubygems 3.3–3.6 actually hard-fails (no assignment of `summary`; no `authors`/`author` +assignment, or one that collapses to no String elements — `[]`/`nil`/`[nil]`/`%w[]`; nil/empty +strings are rubygems-tolerated and pass); a valid stub is still written byte-verbatim, and the +idempotent re-vendor path re-checks the ON-DISK stub, routing a pre-hardening invalid one into +the artifact rebuild. For any ecosystem with no service path at all `auto`/`build` build locally as before, and `service` refuses with `vendor_service_unsupported_ecosystem`. A successful service vend emits `vendor_prebuilt_downloaded`. Unrelated to `--download-mode` (which selects the patch-CONTENT format for the local build). @@ -902,7 +912,8 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_lock_checksums_unsupported` / `vendor_stale_lock_checksum` | `failed` | vendor (gem): an ambiguous/platform CHECKSUMS entry, or a v1-wired lock whose stale token blocks the hot path (run `vendor --revert` + re-vendor). | | `redirect_gem_stale_install` | `redirect.warnings[]` (warning) | scan `--mode hosted` (gem): a stale UNPATCHED materialization (installed gem, or committed `vendor/cache` archive) that `bundle install` will reuse instead of fetching the redirected patch; the detail carries the verified remedy. Full rules and flavors: the "Gem stale-install guard" section. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | -| `vendor_prebuilt_stub_invalid` | `failed` / `skipped` (warning) | vendor (gem, `--vendor-source`): the served stub gemspec never assigns the rubygems-required `summary`/`authors`, so bundler would refuse the vendored path source at install time. `service`: refusal naming the missing attributes; `auto`: loud warning + local-build fallback. | +| `vendor_prebuilt_stub_invalid` | `failed` / `skipped` (warning) | vendor (gem, `--vendor-source`): the served stub gemspec fails the rubygems `summary`/`authors` bar, so bundler would refuse the vendored path source at install time. `service`: refusal naming the missing attributes; `auto`: loud warning + local-build fallback — or, when the gem is also not installed locally (no stub to derive), a refusal naming the served defect and the install-the-gem remedy. | +| `gem_spec_invalid` | `failed` | vendor (gem): the LOCAL `specifications/` stub gemspec fails the same rubygems `summary`/`authors` bar (a corrupted or hand-edited gem home); the refusal names the file — reinstall the gem (`gem pristine ` / fresh `bundle install`). | | `vendor_*` / `pypi_*` / `gemfile_*` / `lock_*` / `locked_version_mismatch` / `user_authored_*` / `native_extensions_unsupported` / `platform_gem_unsupported` | `failed`/`skipped` | vendor: per-ecosystem refusal + drift vocabulary; see the Vendor command contract section. New tags are additive (MINOR). | ### Top-level `EnvelopeError` codes diff --git a/crates/socket-patch-cli/tests/e2e_vendored_production.rs b/crates/socket-patch-cli/tests/e2e_vendored_production.rs index ebbe588f..2450bc12 100644 --- a/crates/socket-patch-cli/tests/e2e_vendored_production.rs +++ b/crates/socket-patch-cli/tests/e2e_vendored_production.rs @@ -54,7 +54,7 @@ //! | Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | advisory id `GHSA-pp8r-vv2j-9j5v` | //! | gem | `pkg:gem/activestorage@6.0.3` | `15e960b5-f432-4b6c-b8aa-534a2b419323` | `Socket Community Patch` header | //! -//! # Ecosystems with no full coverage, and why +//! # Ecosystem coverage notes (gaps, and one resolved gap) //! //! * **gem** — RESOLVED: full coverage. The old `platform_gem_unsupported` //! refusal for `?platform=ruby` purls was fixed in the CLI (#172 — only @@ -341,9 +341,11 @@ fn scan_vendored(cwd: &Path, extra: &[&str]) -> serde_json::Value { /// /// `applied >= 1` is the anti-vacuity guard: a run that discovered nothing also /// exits 0 with `"status": "success"`, so without this a broken crawler would -/// look identical to a working vendor. `failed == 0` catches partial failures -/// (the gem leg deliberately does NOT go through here). The `applied` event's -/// purl may carry qualifiers (`?artifact_id=…`), so a substring match is used. +/// look identical to a working vendor. `failed == 0` catches partial failures. +/// The gem leg uses the purl-SCOPED [`assert_vendor_applied_for`] instead: its +/// fixture has many transitive gems, so run-wide counts would couple the leg +/// to the production catalog's future patches. The `applied` event's purl may +/// carry qualifiers (`?artifact_id=…`), so a substring match is used. fn assert_vendor_applied(env: &serde_json::Value, purl_needle: &str, leg: &str) { let vendor = &env["vendor"]; assert!( @@ -376,6 +378,52 @@ fn assert_vendor_applied(env: &serde_json::Value, purl_needle: &str, leg: &str) ); } +/// The events for purls containing `purl_needle` (substring — purls may carry +/// qualifiers) with the given `action`. +fn vendor_events_for<'a>( + env: &'a serde_json::Value, + purl_needle: &str, + action: &str, +) -> Vec<&'a serde_json::Value> { + env["vendor"]["events"] + .as_array() + .map(|events| { + events + .iter() + .filter(|e| { + e["action"] == action + && e["purl"] + .as_str() + .map(|p| p.contains(purl_needle)) + .unwrap_or(false) + }) + .collect() + }) + .unwrap_or_default() +} + +/// Purl-SCOPED variant of [`assert_vendor_applied`]: the named purl must have +/// an `applied` event and NO `failed` event. Other purls' outcomes are the +/// production catalog's business — a future free patch on a transitive gem of +/// the fixture must not red this leg — so run-wide `summary` counts are +/// deliberately not asserted. +fn assert_vendor_applied_for(env: &serde_json::Value, purl_needle: &str, leg: &str) { + assert!( + !env["vendor"].is_null(), + "{leg}: scan --mode vendored emitted no `vendor` sub-object — the CLI omits it \ + when discovery found nothing, so this means the crawler did not see the \ + installed dependency.\nenvelope:\n{env:#}" + ); + assert!( + !vendor_events_for(env, purl_needle, "applied").is_empty(), + "{leg}: no `applied` event for a purl containing `{purl_needle}`.\nenvelope:\n{env:#}" + ); + assert!( + vendor_events_for(env, purl_needle, "failed").is_empty(), + "{leg}: a `failed` event for `{purl_needle}`.\nenvelope:\n{env:#}" + ); +} + /// Assert the download phase resolved one of the expected patch UUIDs. fn assert_download_uuid(env: &serde_json::Value, uuids: &[&str], leg: &str) { let patches = env["download"]["patches"] @@ -424,6 +472,29 @@ fn vendor_revert(cwd: &Path, leg: &str) -> u64 { // Toolchain invocation // --------------------------------------------------------------------------- +/// Run `bundle ` with the ambient `BUNDLE_*`/`GEM_*`/`RUBYOPT` state +/// scrubbed first (a developer's global bundler config — frozen mode, a +/// custom BUNDLE_PATH or gem home, a RUBYOPT require — must not leak into a +/// leg that hard-asserts bundler outcomes; mirrors e2e_vendor_gem_build.rs). +/// The cache sandbox re-pins its own BUNDLE_USER_HOME/GEM_SPEC_CACHE after +/// the scrub, and the per-leg `env` is applied last so the leg's pins win. +fn bundle(cwd: &Path, args: &[&str], env: &[(&str, &str)]) -> Output { + let mut cmd = Command::new("bundle"); + cmd.args(args).current_dir(cwd); + for (k, _) in std::env::vars_os() { + let key = k.to_string_lossy().into_owned(); + if key.starts_with("BUNDLE_") || key.starts_with("GEM_") || key == "RUBYOPT" { + cmd.env_remove(&k); + } + } + cache_env::isolate(&mut cmd); + for (k, v) in env { + cmd.env(k, v); + } + cmd.output() + .unwrap_or_else(|e| panic!("failed to spawn `bundle`: {e}")) +} + /// Run an external package manager. Returns the `Output` without asserting, so /// callers can distinguish "the registry was unreachable" (soft-skip material /// during fixture setup) from "the install of the vendored lock failed" (always @@ -1725,9 +1796,13 @@ fn gem_bundler_vendored_install_proof() { // the project so no `.bundle/config` joins the committable set. let bundle_path = proj.join("vendor/bundle").display().to_string(); let bundle_config = tmp.path().join("bundle-config").display().to_string(); + // mimemagic (pulled via activestorage → marcel) builds against the system + // shared-mime-info DB, which not every host installs — use the gem's + // bundled placeholder instead of depending on a host package. let env = [ ("BUNDLE_PATH", bundle_path.as_str()), ("BUNDLE_APP_CONFIG", bundle_config.as_str()), + ("USE_FREEDESKTOP_PLACEHOLDER", "true"), ]; std::fs::write( @@ -1736,28 +1811,26 @@ fn gem_bundler_vendored_install_proof() { ) .expect("write Gemfile"); - if !ok(&tool(&proj, "bundle", &["lock"], &env)) { + if !ok(&bundle(&proj, &["lock"], &env)) { soft_skip!(LEG, "`bundle lock` failed"); } - let install = tool(&proj, "bundle", &["install", "--quiet"], &env); + let install = bundle(&proj, &["install", "--quiet"], &env); if !ok(&install) { soft_skip!(LEG, "upstream `bundle install` failed:\n{}", dump(&install)); } // Anti-vacuity: the upstream install must be pristine. `bundle info // --path` reports the exact directory bundler resolved for the gem. - let info = tool(&proj, "bundle", &["info", GEM_NAME, "--path"], &env); + let info = bundle(&proj, &["info", GEM_NAME, "--path"], &env); assert!( ok(&info), "{LEG}: `bundle info {GEM_NAME} --path` failed after the upstream install:\n{}", dump(&info) ); let installed_dir = PathBuf::from(String::from_utf8_lossy(&info.stdout).trim()); - assert_pristine( - &installed_dir.join("lib/active_storage/service/s3_service.rb"), - PATCH_MARKER, - LEG, - ); + let patched_file_rel = "lib/active_storage/service/s3_service.rb"; + assert_pristine(&installed_dir.join(patched_file_rel), PATCH_MARKER, LEG); + let pristine = std::fs::read(installed_dir.join(patched_file_rel)).unwrap(); let gemfile_before = std::fs::read(proj.join("Gemfile")).unwrap(); let lock_before = std::fs::read(proj.join("Gemfile.lock")).unwrap(); @@ -1768,7 +1841,38 @@ fn gem_bundler_vendored_install_proof() { 0, "{LEG}: the gem patch download failed.\nenvelope:\n{env_json:#}" ); - assert_vendor_applied(&env_json, &format!("{GEM_NAME}@{GEM_VERSION}"), LEG); + // Purl-scoped (NOT run-wide counts): a future free patch on one of the + // fixture's transitive Rails gems must not red this leg. + assert_vendor_applied_for(&env_json, &format!("{GEM_NAME}@{GEM_VERSION}"), LEG); + + // Route attribution: EXACTLY one of the two markers must be present for + // this purl — the service artifact was used (`vendor_prebuilt_downloaded`) + // or the invalid-served-stub fallback built locally + // (`vendor_prebuilt_stub_invalid`). Asserting exactly one auto-retires the + // fallback expectation the moment the depscan stub fix deploys and the + // rebuilt artifacts serve valid stubs (and catches both-or-neither as a + // defect either way). + let route_markers = env_json["vendor"]["events"] + .as_array() + .cloned() + .unwrap_or_default() + .iter() + .filter(|e| { + e["purl"] + .as_str() + .map(|p| p.contains(GEM_NAME)) + .unwrap_or(false) + && matches!( + e["errorCode"].as_str(), + Some("vendor_prebuilt_stub_invalid") | Some("vendor_prebuilt_downloaded") + ) + }) + .count(); + assert_eq!( + route_markers, 1, + "{LEG}: expected exactly one route marker (vendor_prebuilt_downloaded XOR \ + vendor_prebuilt_stub_invalid) for {GEM_NAME}.\nenvelope:\n{env_json:#}" + ); // The committable artifact + the mandatory pair edit. let copy_rel = format!(".socket/vendor/gem/{GEM_UUID}/{GEM_NAME}-{GEM_VERSION}"); @@ -1817,9 +1921,11 @@ fn gem_bundler_vendored_install_proof() { ("BUNDLE_PATH", fresh_bundle.as_str()), ("BUNDLE_APP_CONFIG", fresh_bundle.as_str()), ("BUNDLE_FROZEN", "true"), + // Same shared-mime-info hazard as the upstream install above. + ("USE_FREEDESKTOP_PLACEHOLDER", "true"), ]; let lock_committed = std::fs::read(fresh.join("Gemfile.lock")).unwrap(); - let fresh_install = tool(&fresh, "bundle", &["install"], &fresh_env); + let fresh_install = bundle(&fresh, &["install"], &fresh_env); assert!( ok(&fresh_install), "{LEG}: frozen `bundle install` from the committable files failed:\n{}", @@ -1830,9 +1936,13 @@ fn gem_bundler_vendored_install_proof() { lock_committed, "{LEG}: frozen `bundle install` churned the committed Gemfile.lock" ); - // Bundler must have resolved the gem FROM the vendored path source, and - // the bytes it will load must carry the patch marker. - let fresh_info = tool(&fresh, "bundle", &["info", GEM_NAME, "--path"], &fresh_env); + // Bundler must have resolved the gem FROM the vendored path source INSIDE + // the fresh dir, and the bytes it will load must carry the patch marker + // and differ from the captured pristine registry bytes. (`contains` + // alone would also match the ORIGINAL project's vendored path; + // canonicalize both sides — macOS reports tempdirs via /var symlinked to + // /private/var.) + let fresh_info = bundle(&fresh, &["info", GEM_NAME, "--path"], &fresh_env); assert!( ok(&fresh_info), "{LEG}: `bundle info {GEM_NAME} --path` failed after the fresh install:\n{}", @@ -1841,25 +1951,35 @@ fn gem_bundler_vendored_install_proof() { let resolved = String::from_utf8_lossy(&fresh_info.stdout) .trim() .to_string(); + let resolved_canon = std::fs::canonicalize(&resolved) + .unwrap_or_else(|e| panic!("{LEG}: cannot canonicalize `{resolved}`: {e}")); + let fresh_canon = std::fs::canonicalize(&fresh).unwrap(); assert!( - resolved.contains(©_rel), + resolved_canon.starts_with(&fresh_canon) && resolved.contains(©_rel), "{LEG}: bundler resolved {GEM_NAME} from `{resolved}`, not the vendored \ - path `{copy_rel}` — the pair edit did not take effect in the fresh dir" + path `{copy_rel}` inside the fresh dir — the pair edit did not take \ + effect in the fresh dir" ); assert_patched( - &PathBuf::from(&resolved).join("lib/active_storage/service/s3_service.rb"), + &PathBuf::from(&resolved).join(patched_file_rel), PATCH_MARKER, LEG, ); + assert_ne!( + std::fs::read(PathBuf::from(&resolved).join(patched_file_rel)).unwrap(), + pristine, + "{LEG}: the reinstalled bytes equal the PRISTINE registry bytes — the vendored \ + artifact was not the one installed" + ); - // Idempotency + revert (mirrors the pip/uv legs). + // Idempotency + revert (mirrors the pip/uv legs; purl-scoped so a future + // free patch on a transitive gem cannot red the leg). let gemfile_wired = std::fs::read(proj.join("Gemfile")).unwrap(); let lock_wired = std::fs::read(proj.join("Gemfile.lock")).unwrap(); let env2 = scan_vendored(&proj, &[]); - assert_eq!( - env2["vendor"]["summary"]["applied"].as_u64().unwrap_or(99), - 0, - "{LEG}: re-run must vendor nothing new:\n{env2:#}" + assert!( + vendor_events_for(&env2, GEM_NAME, "applied").is_empty(), + "{LEG}: re-run must vendor nothing new for {GEM_NAME}:\n{env2:#}" ); assert_eq!( std::fs::read(proj.join("Gemfile")).unwrap(), @@ -1872,7 +1992,10 @@ fn gem_bundler_vendored_install_proof() { "{LEG}: re-run must leave Gemfile.lock byte-identical" ); - assert_eq!(vendor_revert(&proj, LEG), 1, "{LEG}: one entry reverted"); + assert!( + vendor_revert(&proj, LEG) >= 1, + "{LEG}: at least the {GEM_NAME} entry reverted" + ); assert_eq!( std::fs::read(proj.join("Gemfile")).unwrap(), gemfile_before, diff --git a/crates/socket-patch-cli/tests/in_process_vendor.rs b/crates/socket-patch-cli/tests/in_process_vendor.rs index a07ff897..e54baf87 100644 --- a/crates/socket-patch-cli/tests/in_process_vendor.rs +++ b/crates/socket-patch-cli/tests/in_process_vendor.rs @@ -1734,7 +1734,7 @@ const GEM_PURL: &str = "pkg:gem/demo-gem@1.0.0"; const GEM_PURL_QUALIFIED: &str = "pkg:gem/demo-gem@1.0.0?platform=ruby"; const GEM_ORIG: &[u8] = b"module DemoGem\n STATUS = \"orig\"\nend\n"; const GEM_PATCHED: &[u8] = b"module DemoGem\n STATUS = \"patched\"\nend\n"; -const GEM_GEMSPEC: &str = "Gem::Specification.new do |s|\n s.name = \"demo-gem\"\n s.version = \"1.0.0\"\n s.summary = \"in-process scan --vendor fixture\"\n s.require_paths = [\"lib\"]\nend\n"; +const GEM_GEMSPEC: &str = "Gem::Specification.new do |s|\n s.name = \"demo-gem\"\n s.version = \"1.0.0\"\n s.summary = \"in-process scan --vendor fixture\"\n s.authors = [\"socket-patch e2e\"]\n s.require_paths = [\"lib\"]\nend\n"; const GEM_GEMFILE: &str = "source \"https://rubygems.org\"\n\ngem \"demo-gem\", \"~> 1.0\"\n"; /// Hand-pinned bundler lock grammar (no CHECKSUMS — the 2.x/3.x default). const GEM_LOCK: &str = "GEM\n remote: https://rubygems.org/\n specs:\n demo-gem (1.0.0)\n\nPLATFORMS\n ruby\n\nDEPENDENCIES\n demo-gem (~> 1.0)\n\nBUNDLED WITH\n 2.6.2\n"; diff --git a/crates/socket-patch-cli/tests/repair_vendor_e2e.rs b/crates/socket-patch-cli/tests/repair_vendor_e2e.rs index 41a5d869..faab137f 100644 --- a/crates/socket-patch-cli/tests/repair_vendor_e2e.rs +++ b/crates/socket-patch-cli/tests/repair_vendor_e2e.rs @@ -1150,7 +1150,10 @@ const GEM_NAME: &str = "padlock"; const GEM_VERSION: &str = "1.2.0"; const GEM_PURL: &str = "pkg:gem/padlock@1.2.0"; const GEM_ENCODED: &str = "pkg%3Agem%2Fpadlock%401.2.0"; -const GEMSPEC_STUB: &[u8] = b"Gem::Specification.new do |s|\n s.name = \"padlock\"\n s.version = \"1.2.0\"\n s.require_paths = [\"lib\"]\nend\n"; +// Assigns the rubygems-required `summary` + `authors` (as every healthy +// rubygems-written stub does): the vendor/rebuild write choke point validates +// them since the D4 invalid-stub hardening. +const GEMSPEC_STUB: &[u8] = b"Gem::Specification.new do |s|\n s.name = \"padlock\"\n s.version = \"1.2.0\"\n s.summary = \"repair fixture\"\n s.authors = [\"socket-patch e2e\"]\n s.require_paths = [\"lib\"]\nend\n"; fn gem_copy_rel() -> String { format!(".socket/vendor/gem/{GEM_UUID}/{GEM_NAME}-{GEM_VERSION}") @@ -1811,7 +1814,7 @@ async fn repair_gem_dir_tamper_matrix_and_vex_refusal() { /// repair loops the same failure. #[tokio::test] async fn repair_refreshes_stale_inventory_from_service_provenance() { - const SERVICE_STUB: &[u8] = b"# converter-generated stub\nGem::Specification.new do |s|\n s.name = \"padlock\"\n s.version = \"1.2.0\"\n s.require_paths = [\"lib\"]\nend\n"; + const SERVICE_STUB: &[u8] = b"# converter-generated stub\nGem::Specification.new do |s|\n s.name = \"padlock\"\n s.version = \"1.2.0\"\n s.summary = \"repair fixture\"\n s.authors = [\"socket-patch e2e\"]\n s.require_paths = [\"lib\"]\nend\n"; let mock = MockServer::start().await; mount_gem_patch_api(&mock).await; let tmp = tempfile::tempdir().unwrap(); diff --git a/crates/socket-patch-core/src/vendor/gem.rs b/crates/socket-patch-core/src/vendor/gem.rs index a0f4326b..46dcb29a 100644 --- a/crates/socket-patch-core/src/vendor/gem.rs +++ b/crates/socket-patch-core/src/vendor/gem.rs @@ -51,7 +51,7 @@ //! sources and the missing `.so` only fails at `require` time with a //! confusing error — refusing up front is the honest failure. -use std::path::Path; +use std::path::{Path, PathBuf}; use serde_json::Value; @@ -254,20 +254,32 @@ pub async fn vendor_gem( }; // ── stub gemspec (local) ───────────────────────────────────────────── - // `specifications/` is a sibling of `gems/`; derive it from installed_dir. + // `specifications/` is a sibling of `gems/`; derive it from installed_dir + // ONLY when installed_dir actually sits inside a gem home's `gems/` dir. + // SECURITY: the registry auto-fetch ladder stages a not-installed gem at + // `/-` (registry_fetch::fetch_gem); + // walking two parents up from THERE escapes the private dir into the + // SHARED temp root, making `$TMPDIR/specifications/.gemspec` a + // predictable, attacker-plantable path on multi-user hosts — one whose + // contents would be committed into the project and later eval'd as Ruby + // by every `bundle install`. A staging dir has no local stub, period. + // // The read is non-fatal: the LOCAL build needs this stub, but the service // path brings its own (the converter-generated `gem-stub-gemspec`), so an // auto-fetched (not-installed) gem whose only `installed_dir` is a bare // `data.tar.gz` extraction can still vendor via the service. The // `gem_spec_missing` refusal moves into the local-build fallback, where the // stub is actually required. - let spec_src = installed_dir - .parent() - .and_then(Path::parent) - .map(|home| home.join("specifications").join(format!("{leaf}.gemspec"))); - let spec_text: Option = match &spec_src { - Some(p) => tokio::fs::read_to_string(p).await.ok(), - None => None, + let local_stub: Option<(PathBuf, String)> = { + let spec_src = installed_dir + .parent() + .filter(|gems| gems.file_name().is_some_and(|n| n == "gems")) + .and_then(Path::parent) + .map(|home| home.join("specifications").join(format!("{leaf}.gemspec"))); + match spec_src { + Some(p) => tokio::fs::read_to_string(&p).await.ok().map(|t| (p, t)), + None => None, + } }; // Textual heuristic, deliberately fail-closed on a match: bundler skips // extension builds for path sources entirely, so a native gem would @@ -275,7 +287,7 @@ pub async fn vendor_gem( // Only the local stub is checked here (when present); the service stub is // re-checked in `gem_service_copy`, and a native gem emits no service stub // at all (the converter refuses it), so the service path also misses. - if let Some(text) = &spec_text { + if let Some((_, text)) = &local_stub { if gemspec_declares_extensions(text) { return refused( "native_extensions_unsupported", @@ -294,10 +306,17 @@ pub async fn vendor_gem( let remote_line = format!(" remote: {copy_rel}"); let lock_wired = lock_text.split('\n').any(|l| l == remote_line) && gemfile_text.contains(©_rel); - let copy_ok = copy_matches_after_hashes(©_dir, &record.files).await - && tokio::fs::metadata(copy_dir.join(format!("{name}.gemspec"))) - .await - .is_ok(); + // D4 heal: a project vendored before the invalid-stub hardening carries + // the defective SERVED stub on disk, so EXISTS is not enough — an on-disk + // stub that fails the required-attribute bar routes into the artifact + // rebuild below (which re-materialises a valid stub) instead of the + // silent `already_vendored` no-op. + let copy_stub_ok = + match tokio::fs::read_to_string(copy_dir.join(format!("{name}.gemspec"))).await { + Ok(text) => gemspec_missing_required_attrs(&text).is_empty(), + Err(_) => false, + }; + let copy_ok = copy_matches_after_hashes(©_dir, &record.files).await && copy_stub_ok; if lock_wired { if lock_checksum_in_sync(&lock_text, name, version) { if copy_ok { @@ -325,7 +344,7 @@ pub async fn vendor_gem( &uuid_dir, name, version, - spec_text.as_deref(), + local_stub.as_ref().map(|(p, t)| (p.as_path(), t.as_str())), record, sources, force, @@ -408,7 +427,7 @@ pub async fn vendor_gem( &uuid_dir, name, version, - spec_text.as_deref(), + local_stub.as_ref().map(|(p, t)| (p.as_path(), t.as_str())), record, sources, force, @@ -652,7 +671,13 @@ enum GemServiceCopy { /// Bubble this terminal outcome (boxed — `VendorOutcome` is large). HardFail(Box), /// Fall back to copying the installed gem + local stub and patching it. - FallBack, + /// When the service DID serve a stub but it failed validation (the D4 + /// defect), the payload carries the defect reason so a stub-less local + /// fallback can refuse truthfully — naming the served defect and the + /// install-the-gem remedy — instead of `gem_spec_missing`'s circular + /// "use --vendor-source=service" advice (a `Refused` outcome carries no + /// warnings, so without this the diagnostic never reaches the envelope). + FallBack(Option), } /// Download the prebuilt `.gem` + its `gem-stub-gemspec` secondary artifact, @@ -680,23 +705,39 @@ async fn gem_service_copy( warnings: &mut Vec, ) -> GemServiceCopy { let Some(cfg) = service else { - return GemServiceCopy::FallBack; + return GemServiceCopy::FallBack(None); }; if !cfg.service_enabled() { - return GemServiceCopy::FallBack; + return GemServiceCopy::FallBack(None); } fn hard(code: &'static str, detail: String) -> GemServiceCopy { GemServiceCopy::HardFail(Box::new(refused(code, detail))) } - let miss = |warnings: &mut Vec, code: &'static str, reason: String| { + // One policy for every service miss: explicit `service` refuses (the + // `refusal` tuple names the terminal code and an optional remedy sentence + // for its detail), `auto` warns under `code` and falls back to the local + // build. `is_stub_defect` marks the misses where the service DID serve a + // stub that failed validation — the reason then rides the `FallBack` + // payload (see [`GemServiceCopy::FallBack`]). + let miss = |warnings: &mut Vec, + code: &'static str, + refusal: (&'static str, &str), + reason: String, + is_stub_defect: bool| { if cfg.source.requires_service() { - hard("vendor_prebuilt_required", reason) + let (hard_code, remedy) = refusal; + let detail = if remedy.is_empty() { + reason + } else { + format!("{reason}. {remedy}") + }; + hard(hard_code, detail) } else { warnings.push(VendorWarning::new( code, format!("{reason}; building locally instead"), )); - GemServiceCopy::FallBack + GemServiceCopy::FallBack(is_stub_defect.then_some(reason)) } }; @@ -707,14 +748,18 @@ async fn gem_service_copy( return miss( warnings, "vendor_prebuilt_integrity_mismatch", + ("vendor_prebuilt_required", ""), format!("prebuilt .gem failed integrity ({reason})"), + false, ); } ServiceArtifact::Pending => { return miss( warnings, "vendor_prebuilt_pending", + ("vendor_prebuilt_required", ""), "prebuilt .gem is still building".to_string(), + false, ); } ServiceArtifact::Unavailable(reason) => { @@ -724,13 +769,15 @@ async fn gem_service_copy( format!("prebuilt .gem unavailable: {reason}"), ); } - return GemServiceCopy::FallBack; + return GemServiceCopy::FallBack(None); } ServiceArtifact::Failed(reason) => { return miss( warnings, "vendor_prebuilt_unavailable", + ("vendor_prebuilt_required", ""), format!("patch service request failed ({reason})"), + false, ); } }; @@ -742,31 +789,38 @@ async fn gem_service_copy( return miss( warnings, "vendor_prebuilt_stub_missing", + ("vendor_prebuilt_required", ""), "the patch service served no stub gemspec for this gem (a native-extension \ gem, or a patch built before the stub rollout)" .to_string(), + false, ); } SecondaryArtifactResult::IntegrityMismatch(reason) => { return miss( warnings, "vendor_prebuilt_integrity_mismatch", + ("vendor_prebuilt_required", ""), format!("prebuilt stub gemspec failed integrity ({reason})"), + false, ); } SecondaryArtifactResult::Failed(reason) => { return miss( warnings, "vendor_prebuilt_unavailable", + ("vendor_prebuilt_required", ""), format!("could not fetch the stub gemspec ({reason})"), + false, ); } }; + let stub_text = String::from_utf8_lossy(&stub); // Defense in depth: the converter does not emit a stub for native gems, but // refuse one here too — bundler silently skips extension builds for path // sources, so a native gem would install and then fail at `require` time. - if gemspec_declares_extensions(&String::from_utf8_lossy(&stub)) { + if gemspec_declares_extensions(&stub_text) { return hard( "native_extensions_unsupported", format!( @@ -785,37 +839,31 @@ async fn gem_service_copy( // `vendor_prebuilt_stub_invalid` code, and always loudly — the served // artifact is defective, not merely absent. Nothing has been written yet, // so the refusal leaves no partial artifacts. - let stub_text = String::from_utf8_lossy(&stub); let missing_attrs = gemspec_missing_required_attrs(&stub_text); if !missing_attrs.is_empty() { - let licenses_note = if gemspec_assigns_attr(&stub_text, "licenses") - || gemspec_assigns_attr(&stub_text, "license") - { + let licenses_note = if gemspec_assigns_attr(&stub_text, &["licenses", "license"]) { "" } else { " (it also omits `licenses`, a rubygems warning)" }; let reason = format!( - "the served stub gemspec for {name} is invalid: it never assigns the \ + "the served stub gemspec for {name} is invalid: it does not assign the \ rubygems-required attribute(s) {}{licenses_note}; bundler validates \ path-source gemspecs, so vendoring it would make every later \ `bundle install` fail", missing_attrs.join(", "), ); - if cfg.source.requires_service() { - return hard( - "vendor_prebuilt_stub_invalid", - format!( - "{reason}. Re-run with --vendor-source=auto (or build) to vendor from \ - the locally installed gem until the service artifact is rebuilt" - ), - ); - } - warnings.push(VendorWarning::new( + return miss( + warnings, "vendor_prebuilt_stub_invalid", - format!("{reason}; building locally instead"), - )); - return GemServiceCopy::FallBack; + ( + "vendor_prebuilt_stub_invalid", + "Re-run with --vendor-source=auto (or build) to vendor from the locally \ + installed gem until the service artifact is rebuilt", + ), + reason, + true, + ); } // Extract the patched `.gem`'s data.tar.gz into a clean copy dir, then add @@ -854,10 +902,12 @@ async fn gem_service_copy( return miss( warnings, "vendor_prebuilt_layout_mismatch", + ("vendor_prebuilt_required", ""), format!( "prebuilt .gem for {name} extracted to an unexpected layout \ (patched files absent at their recorded paths)" ), + false, ); } warnings.push(VendorWarning::new( @@ -885,7 +935,7 @@ async fn materialise_patched_copy( uuid_dir: &Path, name: &str, version: &str, - spec_text: Option<&str>, + local_stub: Option<(&Path, &str)>, record: &PatchRecord, sources: &PatchSources<'_>, force: bool, @@ -899,19 +949,63 @@ async fn materialise_patched_copy( Ok(already_patched_result(purl, copy_dir, &record.files)) } GemServiceCopy::HardFail(outcome) => Err(outcome), - GemServiceCopy::FallBack => { + GemServiceCopy::FallBack(served_stub_defect) => { // The local build needs the stub gemspec from the installed gem's // `specifications/` dir — absent for an auto-fetched (not-installed) // gem, whose only route is the service path. - let Some(spec_text) = spec_text else { + let Some((spec_path, spec_text)) = local_stub else { + return Err(Box::new(match served_stub_defect { + // The service DID serve a stub — a defective one (D4). Say + // so: the generic advice below would send the user in a + // circle (`--vendor-source=service` refuses on the same + // defect), and a `Refused` outcome carries no warnings, so + // this detail is the diagnostic's only route into the + // envelope. + Some(defect) => refused( + "vendor_prebuilt_stub_invalid", + format!( + "{defect}; and {name}@{version} is not installed locally, so the \ + local-build fallback has no stub gemspec to derive from — install \ + the gem (e.g. `bundle install`) and re-run, or wait for the \ + rebuilt service artifact" + ), + ), + None => refused( + "gem_spec_missing", + format!( + "no local stub gemspec for {name}@{version} (a path source cannot \ + be wired without one); install the gem or use \ + --vendor-source=service" + ), + ), + })); + }; + // The write choke point validates BOTH stub sources: the served + // stub is checked in `gem_service_copy`, and the locally-derived + // stub here — bundler rejects a path-source gemspec missing the + // required attributes wherever it came from. (A healthy rubygems + // install always writes a valid `specifications/` stub, so this + // only fires on a corrupted or hand-edited gem home.) + let missing = gemspec_missing_required_attrs(spec_text); + if !missing.is_empty() { + let served_note = match served_stub_defect { + Some(defect) => { + format!("; the patch service cannot supply one either ({defect})") + } + None => String::new(), + }; return Err(Box::new(refused( - "gem_spec_missing", + "gem_spec_invalid", format!( - "no local stub gemspec for {name}@{version} (a path source cannot be \ - wired without one); install the gem or use --vendor-source=service" + "the local stub gemspec at {} does not assign the rubygems-required \ + attribute(s) {} — bundler would refuse the vendored path source at \ + install time; reinstall the gem (`gem pristine {name}` or a fresh \ + `bundle install`) and re-run{served_note}", + spec_path.display(), + missing.join(", "), ), ))); - }; + } if let Err(e) = fresh_copy(installed_dir, copy_dir, None).await { return Ok(synthesized_result( purl, @@ -2215,16 +2309,53 @@ fn is_plain_gem_token(s: &str) -> bool { .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-')) } +/// The one shared gemspec line-scanner: locate a `.{attr}` mention in `line` +/// and return what follows it (leading-whitespace-trimmed), or `None`. +/// +/// `anchored` additionally requires the mention to OPEN the line as +/// `.{attr}` with a plain-identifier receiver (`s.summary = …`, +/// ` spec.authors= …`). Anchoring makes a preceding comment marker +/// impossible, so anchored callers scan RAW lines with no comment-stripping — +/// stripping at `#` would truncate inside string literals and misjudge +/// `s.summary = "#1 Ruby web server"` as missing. A mention whose attr +/// continues as a longer identifier (`.extensions_dir`, `.authors` when +/// looking for `.author`) is never a match. Only the FIRST mention per line +/// is examined — one attribute per line is the shape `Specification#to_ruby` +/// emits. Parsing ruby for real would need a ruby. +fn attr_mention<'a>(line: &'a str, attr: &str, anchored: bool) -> Option<&'a str> { + let needle = format!(".{attr}"); + let idx = line.find(&needle)?; + if anchored { + let receiver = line[..idx].trim_start(); + if receiver.is_empty() + || !receiver + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '_' | '@')) + { + return None; + } + } + let after = &line[idx + needle.len()..]; + if after + .chars() + .next() + .is_some_and(|c| c.is_ascii_alphanumeric() || c == '_') + { + return None; + } + Some(after.trim_start()) +} + /// Textual heuristic for `s.extensions = […]` / `spec.extensions << …` style -/// declarations (comment-stripped per line). A match always refuses -/// (fail-closed); a miss — e.g. extensions assigned through interpolation -/// tricks — falls through, which only loses the refusal's nicer error, not -/// safety. Parsing ruby for real would need a ruby. +/// declarations (comment-stripped per line — a commented-out declaration is +/// not one; the truncation caveat in [`attr_mention`] only loses this +/// refusal's nicer error, never safety, because a match REFUSES). A miss — +/// e.g. extensions assigned through interpolation tricks — falls through, +/// which likewise only loses the nicer error. fn gemspec_declares_extensions(spec_text: &str) -> bool { for raw in spec_text.lines() { let line = raw.split('#').next().unwrap_or(""); - if let Some(idx) = line.find(".extensions") { - let after = line[idx + ".extensions".len()..].trim_start(); + if let Some(after) = attr_mention(line, "extensions", false) { if (after.starts_with('=') && !after.starts_with("==")) || after.starts_with("<<") || after.starts_with("+=") @@ -2238,51 +2369,83 @@ fn gemspec_declares_extensions(spec_text: &str) -> bool { false } -/// Textual heuristic: does any line (comment-stripped) assign a NON-EMPTY -/// value to `.{attr}`? Same conservative bar as -/// [`gemspec_declares_extensions`] — no real ruby parsing. "Non-empty" filters -/// the obviously-empty spellings (`""`, `''`, `[]`, `nil`, any of them -/// `.freeze`-d); anything else counts, so a legitimate assignment can never be -/// missed by over-cleverness. -fn gemspec_assigns_attr(spec_text: &str, attr: &str) -> bool { - let needle = format!(".{attr}"); +/// Every RHS assigned to any of the `attrs` aliases at a line start +/// (assignments only — `==` comparisons don't count), via [`attr_mention`]. +fn gemspec_attr_rhs<'a>(spec_text: &'a str, attrs: &[&str]) -> Vec<&'a str> { + let mut out = Vec::new(); for raw in spec_text.lines() { - let line = raw.split('#').next().unwrap_or(""); - if let Some(idx) = line.find(&needle) { - let after = line[idx + needle.len()..].trim_start(); - if after.starts_with('=') && !after.starts_with("==") { - let value: String = after[1..] - .replace(".freeze", "") - .chars() - .filter(|c| !c.is_whitespace() && !matches!(c, '"' | '\'' | '[' | ']' | ',')) - .collect(); - if !value.is_empty() && value != "nil" { - return true; + for attr in attrs { + if let Some(after) = attr_mention(raw, attr, true) { + if let Some(rhs) = after.strip_prefix('=') { + if !rhs.starts_with('=') { + out.push(rhs.trim()); + } } } } } - false + out +} + +/// Does any line assign one of the `attrs` aliases? Pass every alias rubygems +/// accepts for the attribute (`["authors", "author"]`, `["licenses", +/// "license"]`). +fn gemspec_assigns_attr(spec_text: &str, attrs: &[&str]) -> bool { + !gemspec_attr_rhs(spec_text, attrs).is_empty() } -/// The rubygems-REQUIRED attributes a served stub gemspec must assign for -/// bundler to accept it as a path source, returned as the list it is missing -/// (empty = valid). Every bundler major validates path-source gemspecs and -/// hard-fails on a nil `summary` or `authors` (`missing value for attribute -/// summary`), so a stub without them bricks every later `bundle install` — -/// the D4 defect the 2026-08-19 gem live matrix found in ALL served stubs. -/// `authors` also accepts rubygems' singular `author =` alias. A missing -/// `licenses` is only a rubygems WARNING, so it is deliberately not checked +/// Textually: does this `authors` RHS collapse to NO String elements? +/// Rubygems' `authors=` writer keeps only Strings (`grep(String)`), so `[]`, +/// `nil`, `[nil]`, and empty word-arrays (`%w[]`) all yield an empty authors +/// list — the hard `authors may not be empty` error — while `[""]` keeps its +/// String and validates. Fail-open: anything not demonstrably empty passes +/// (a `[42]` would slip through, but `to_ruby` never emits one and bundler +/// still reports it — the heuristic only loses the nicer error). +fn authors_rhs_collapses_empty(rhs: &str) -> bool { + let cleaned = rhs.replace(".freeze", ""); + let cleaned = cleaned.trim(); + let body = cleaned + .strip_prefix("%w") + .or_else(|| cleaned.strip_prefix("%W")) + .unwrap_or(cleaned); + !body + .split(|c: char| c.is_whitespace() || matches!(c, '[' | ']' | '(' | ')' | ',')) + .any(|tok| !tok.is_empty() && tok != "nil") +} + +/// The rubygems-REQUIRED attributes a stub gemspec must assign for bundler to +/// accept it as a path source, returned as the list it is missing (empty = +/// valid). Every bundler major validates path-source gemspecs, so a stub +/// missing these bricks every later `bundle install` — the D4 defect the +/// 2026-08-19 gem live matrix found in ALL served stubs. +/// +/// The bar is EMPIRICAL, verified against rubygems 3.3 / 3.5 / 3.6 +/// (`Gem::Specification#validate`, both packaging modes, in the bundler +/// 1.17 / 2.7 / 4.0 era images): +/// +/// * `summary` — hard `missing value for attribute summary` ONLY when never +/// assigned. The `summary=` writer coerces `nil`/`""` to a present value +/// (empty is at most a warning), so ANY assignment line satisfies it. +/// * `authors` — hard `authors may not be empty` when never assigned (the +/// singular `author =` alias counts) or when every assignment textually +/// collapses to no String elements ([`authors_rhs_collapses_empty`]). +/// +/// A missing `licenses` is only a rubygems WARNING, deliberately not checked /// here (callers may mention it in advisory text via -/// [`gemspec_assigns_attr`]). Fail-open by construction: only a stub where -/// the assignment lines are demonstrably absent is flagged, so a legitimate -/// stub always passes and is written byte-verbatim. +/// [`gemspec_assigns_attr`]). Fail-open by construction: only a stub that +/// demonstrably fails the bar is flagged, so a legitimate stub always passes +/// and is written byte-verbatim. fn gemspec_missing_required_attrs(spec_text: &str) -> Vec<&'static str> { let mut missing = Vec::new(); - if !gemspec_assigns_attr(spec_text, "summary") { + if !gemspec_assigns_attr(spec_text, &["summary"]) { missing.push("summary"); } - if !gemspec_assigns_attr(spec_text, "authors") && !gemspec_assigns_attr(spec_text, "author") { + let author_rhs = gemspec_attr_rhs(spec_text, &["authors", "author"]); + if author_rhs.is_empty() + || author_rhs + .iter() + .all(|rhs| authors_rhs_collapses_empty(rhs)) + { missing.push("authors"); } missing @@ -2303,7 +2466,10 @@ mod tests { const PRISTINE: &[u8] = b"module Rack\n VERSION = \"3.2.6\"\nend\n"; const PATCHED: &[u8] = b"module Rack\n SOCKET_PATCHED = true\n VERSION = \"3.2.6\"\nend\n"; - const GEMSPEC: &str = "Gem::Specification.new do |s|\n s.name = \"rack\"\n s.version = \"3.2.6\"\n s.summary = \"a modular Ruby web server interface\"\n s.require_paths = [\"lib\"]\nend\n"; + // Every local-stub fixture assigns the rubygems-required `summary` + + // `authors` — as any healthy rubygems-written `specifications/` stub does + // — because the local-build write choke point validates them too. + const GEMSPEC: &str = "Gem::Specification.new do |s|\n s.name = \"rack\"\n s.version = \"3.2.6\"\n s.summary = \"a modular Ruby web server interface\"\n s.authors = [\"Rack maintainers\"]\n s.require_paths = [\"lib\"]\nend\n"; const GEMFILE_DIRECT: &str = "source \"https://rubygems.org\"\n\ngem \"puma\"\ngem \"rack\", \"~> 3.1\"\n"; @@ -2625,8 +2791,9 @@ mod tests { } /// The required-attribute heuristic ([`gemspec_missing_required_attrs`]): - /// flag ONLY a demonstrably-absent (or demonstrably-empty) assignment — - /// a legitimate stub must always pass, whatever its spelling. + /// flag ONLY what real rubygems hard-fails on (empirically verified + /// against rubygems 3.3/3.5/3.6, see the fn doc) — a stub rubygems + /// tolerates must always pass, whatever its spelling. #[test] fn required_attrs_heuristic() { // The D4 production shape: no summary, no authors. @@ -2655,23 +2822,53 @@ mod tests { gemspec_missing_required_attrs("s.summary = \"x\"\ns.authors = %w[alice bob]\n"), Vec::<&str>::new() ); + // A `#` inside a string literal is CONTENT, not a comment — this + // valid stub must never be judged missing (scanning raw lines, + // anchored to the line start, instead of comment-stripping). + assert_eq!( + gemspec_missing_required_attrs( + "s.summary = \"#1 Ruby web server\".freeze\ns.authors = [\"D. #2 Person\".freeze]\n" + ), + Vec::<&str>::new() + ); // One present, one absent → only the absent one is named. assert_eq!( gemspec_missing_required_attrs("s.summary = \"x\".freeze\n"), vec!["authors"] ); - // Demonstrably-empty spellings count as missing… + // Rubygems TOLERATES nil/empty summary (the writer coerces; empty is + // a warning) and an empty-STRING author ([""] keeps its String), so + // none of these flag. assert_eq!( - gemspec_missing_required_attrs("s.summary = \"\".freeze\ns.authors = [].freeze\n"), - vec!["summary", "authors"] + gemspec_missing_required_attrs("s.summary = \"\".freeze\ns.authors = [\"\"]\n"), + Vec::<&str>::new() ); assert_eq!( - gemspec_missing_required_attrs("s.summary = nil\ns.authors = [\"\"]\n"), + gemspec_missing_required_attrs("s.summary = nil\ns.authors = [\"a\"]\n"), + Vec::<&str>::new() + ); + // Rubygems HARD-FAILS an authors list with no String elements + // (`authors may not be empty`): [], nil, [nil], %w[] all flag. + for empty_authors in ["[]", "[].freeze", "nil", "[nil]", "%w[]", "%W()"] { + assert_eq!( + gemspec_missing_required_attrs(&format!( + "s.summary = \"x\"\ns.authors = {empty_authors}\n" + )), + vec!["authors"], + "authors = {empty_authors} must flag" + ); + } + // Commented-out assignments, `==` comparisons, and mid-line mentions + // are not assignments. + assert_eq!( + gemspec_missing_required_attrs( + "# s.summary = \"x\"\nraise if s.authors == [\"x\"]\nfoo(s.summary = \"x\")\n" + ), vec!["summary", "authors"] ); - // …and so do commented-out assignments and `==` comparisons. + // Longer identifiers are not the attribute (`.authors` != `.author`). assert_eq!( - gemspec_missing_required_attrs("# s.summary = \"x\"\nraise if s.authors == [\"x\"]\n"), + gemspec_missing_required_attrs("s.summary_text = \"x\"\ns.author_email = \"x\"\n"), vec!["summary", "authors"] ); } @@ -3135,7 +3332,7 @@ mod tests { const PRISTINE_318: &[u8] = b"module Rack\n VERSION = \"3.1.8\"\nend\n"; const PATCHED_318: &[u8] = b"module Rack\n SOCKET_PATCHED = true\n VERSION = \"3.1.8\"\nend\n"; - const GEMSPEC_318: &str = "Gem::Specification.new do |s|\n s.name = \"rack\"\n s.version = \"3.1.8\"\n s.require_paths = [\"lib\"]\nend\n"; + const GEMSPEC_318: &str = "Gem::Specification.new do |s|\n s.name = \"rack\"\n s.version = \"3.1.8\"\n s.summary = \"a modular Ruby web server interface\"\n s.authors = [\"Rack maintainers\"]\n s.require_paths = [\"lib\"]\nend\n"; // Embedded VERBATIM from the spike pair // `spikes/gem-checksums/path-with-checksums/{before,after}/` (bundler @@ -3668,7 +3865,7 @@ mod tests { const PRISTINE_PUMA: &[u8] = b"module Puma\n VERSION = \"6.4.2\"\nend\n"; const PATCHED_PUMA: &[u8] = b"module Puma\n SOCKET_PATCHED = true\n VERSION = \"6.4.2\"\nend\n"; - const GEMSPEC_PUMA: &str = "Gem::Specification.new do |s|\n s.name = \"puma\"\n s.version = \"6.4.2\"\n s.require_paths = [\"lib\"]\nend\n"; + const GEMSPEC_PUMA: &str = "Gem::Specification.new do |s|\n s.name = \"puma\"\n s.version = \"6.4.2\"\n s.summary = \"a fast, concurrent web server\"\n s.authors = [\"Puma maintainers\"]\n s.require_paths = [\"lib\"]\nend\n"; fn puma_rel() -> String { format!(".socket/vendor/gem/{UUID_PUMA}/puma-6.4.2") @@ -4586,6 +4783,223 @@ mod tests { ); } + /// D4 + `auto` + the gem NOT installed (a `missing_install` staging-style + /// dir): the local-build fallback has no stub to derive, and the refusal + /// must be TRUTHFUL — it carries the served-stub defect (a `Refused` + /// outcome has no warnings channel, so the detail is the diagnostic's + /// only route into the envelope) and the install-the-gem remedy, never + /// `gem_spec_missing`'s circular "use --vendor-source=service" advice + /// (service refuses on the same defect). + #[tokio::test] + async fn service_stub_invalid_auto_not_installed_refuses_truthfully() { + let (_tmp, root, _installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let gem = make_gem(&[("lib/rack.rb", PATCHED)]); + let sri = sri_sha512(&gem); + let stub_sri = sri_sha512(SERVICE_STUB_INVALID); + let server = wiremock::MockServer::start().await; + mount_gem_granted(&server, &gem, &sri, Some((SERVICE_STUB_INVALID, &stub_sri))).await; + let sources = PatchSources::blobs_only(&blobs); + + let outcome = vendor_gem( + PURL, + &missing_install(&root), + &root, + &record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&gem_service_cfg(&server.uri(), VendorSource::Auto, false)), + ) + .await; + let (code, detail) = unwrap_refused(outcome); + assert_eq!(code, "vendor_prebuilt_stub_invalid"); + assert!( + detail.contains("summary") && detail.contains("authors"), + "the refusal must carry the served-stub defect: {detail}" + ); + assert!( + detail.contains("not installed locally") && detail.contains("install the gem"), + "the refusal must advise installing the gem: {detail}" + ); + assert!( + !detail.contains("--vendor-source=service"), + "circular advice (service refuses on the same defect): {detail}" + ); + assert!(!root.join(".socket").exists()); + } + + /// D4 + explicit `service` + the gem NOT installed: the hard refusal is + /// the same as the installed case — installation is irrelevant to + /// `service` mode, which never falls back. + #[tokio::test] + async fn service_stub_invalid_service_mode_not_installed_hard_fails() { + let (_tmp, root, _installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let gem = make_gem(&[("lib/rack.rb", PATCHED)]); + let sri = sri_sha512(&gem); + let stub_sri = sri_sha512(SERVICE_STUB_INVALID); + let server = wiremock::MockServer::start().await; + mount_gem_granted(&server, &gem, &sri, Some((SERVICE_STUB_INVALID, &stub_sri))).await; + let sources = PatchSources::blobs_only(&blobs); + + let outcome = vendor_gem( + PURL, + &missing_install(&root), + &root, + &record, + &sources, + "2026-06-09T00:00:00Z", + false, + false, + Some(&gem_service_cfg( + &server.uri(), + VendorSource::Service, + false, + )), + ) + .await; + let (code, detail) = unwrap_refused(outcome); + assert_eq!(code, "vendor_prebuilt_stub_invalid"); + assert!( + detail.contains("--vendor-source=auto"), + "the service refusal names the auto/build remedy: {detail}" + ); + assert!(!root.join(".socket").exists()); + } + + /// SECURITY: the local stub gemspec is derived from `installed_dir` ONLY + /// when it sits inside a real gem home's `gems/` dir. For an auto-fetch + /// staging dir (`/-`), walking two + /// parents up would escape into the SHARED temp root, where + /// `specifications/.gemspec` is a predictable, attacker-plantable + /// path whose contents would be committed into the project and eval'd as + /// Ruby by every later `bundle install`. The planted spec must never be + /// consumed: with no service configured the vendor refuses + /// `gem_spec_missing`. + #[tokio::test] + async fn planted_spec_outside_gem_home_is_not_consumed() { + let (tmp, root, _installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let base = tmp.path(); + // The auto-fetch staging shape: /stage/-, with + // the pristine bytes present (the parent is NOT named `gems`). + let staged = base.join("stage/rack-3.2.6"); + tokio::fs::create_dir_all(staged.join("lib")).await.unwrap(); + tokio::fs::write(staged.join("lib/rack.rb"), PRISTINE) + .await + .unwrap(); + // The attacker's plant, at exactly where an unguarded + // parent-of-parent derivation would look: a VALID stub, so consuming + // it would "succeed". + tokio::fs::create_dir_all(base.join("specifications")) + .await + .unwrap(); + tokio::fs::write(base.join("specifications/rack-3.2.6.gemspec"), GEMSPEC) + .await + .unwrap(); + + let (code, detail) = + unwrap_refused(run_vendor_purl(PURL, &root, &blobs, &staged, &record, false).await); + assert_eq!( + code, "gem_spec_missing", + "the planted spec outside a gem home must not be consumed: {detail}" + ); + assert!(!root.join(".socket").exists()); + } + + /// The write choke point validates the LOCALLY-derived stub too: a + /// corrupted `specifications/` stub missing the required attributes is an + /// honest `gem_spec_invalid` refusal naming the file — never a vendored + /// copy bundler will reject at install time. + #[tokio::test] + async fn local_stub_invalid_refuses_with_gem_spec_invalid() { + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let spec = installed + .parent() + .unwrap() + .parent() + .unwrap() + .join("specifications/rack-3.2.6.gemspec"); + tokio::fs::write( + &spec, + "Gem::Specification.new do |s|\n s.name = \"rack\"\n s.version = \"3.2.6\"\n s.require_paths = [\"lib\"]\nend\n", + ) + .await + .unwrap(); + + let (code, detail) = + unwrap_refused(run_vendor(&root, &blobs, &installed, &record, false).await); + assert_eq!(code, "gem_spec_invalid"); + assert!( + detail.contains("summary") + && detail.contains("authors") + && detail.contains("rack-3.2.6.gemspec"), + "the refusal names the file and the missing attributes: {detail}" + ); + assert!(!root.join(".socket").exists()); + assert_eq!( + tokio::fs::read_to_string(root.join(GEMFILE)).await.unwrap(), + GEMFILE_DIRECT + ); + assert_eq!( + tokio::fs::read_to_string(root.join(GEMFILE_LOCK)) + .await + .unwrap(), + LOCK_DIRECT + ); + } + + /// D4 heal: a project vendored PRE-hardening carries the invalid served + /// stub on disk. The idempotent hot path must not re-bless it as + /// `already_vendored`: the on-disk stub fails the required-attribute bar, + /// routing into the artifact-only rebuild, which rewrites a valid stub + /// with the pair edit and the ledger entry untouched. + #[tokio::test] + async fn wired_copy_with_invalid_stub_is_rebuilt() { + let (_tmp, root, installed, blobs, record) = fixture(GEMFILE_DIRECT, LOCK_DIRECT).await; + let (result, entry, _) = + unwrap_done(run_vendor(&root, &blobs, &installed, &record, false).await); + assert!(result.success, "{:?}", result.error); + assert!(entry.is_some()); + let gemfile_wired = tokio::fs::read(root.join(GEMFILE)).await.unwrap(); + let lock_wired = tokio::fs::read(root.join(GEMFILE_LOCK)).await.unwrap(); + + // Simulate the pre-fix victim: the served invalid stub on disk. + tokio::fs::write(copy_gemspec(&root), SERVICE_STUB_INVALID) + .await + .unwrap(); + + let (result2, entry2, warnings2) = + unwrap_done(run_vendor(&root, &blobs, &installed, &record, false).await); + assert!(result2.success, "{:?}", result2.error); + assert!( + entry2.is_none(), + "artifact-only rebuild must not re-record a ledger entry" + ); + assert!( + warnings2 + .iter() + .any(|w| w.code == "vendor_artifact_rebuilt"), + "the heal must surface as a rebuild, not a silent no-op: {warnings2:?}" + ); + assert_eq!( + tokio::fs::read_to_string(copy_gemspec(&root)) + .await + .unwrap(), + GEMSPEC, + "the invalid on-disk stub must be replaced with the valid local stub" + ); + assert_eq!( + tokio::fs::read(root.join(GEMFILE)).await.unwrap(), + gemfile_wired, + "the heal must not touch the Gemfile" + ); + assert_eq!( + tokio::fs::read(root.join(GEMFILE_LOCK)).await.unwrap(), + lock_wired, + "the heal must not touch Gemfile.lock" + ); + } + /// `auto` + a not-built service status falls back to the local build. #[tokio::test] async fn service_unavailable_auto_falls_back_to_build() { diff --git a/docs/testing/vendored-production-e2e.md b/docs/testing/vendored-production-e2e.md index 4fcd6939..695152e6 100644 --- a/docs/testing/vendored-production-e2e.md +++ b/docs/testing/vendored-production-e2e.md @@ -132,15 +132,23 @@ the gem-stub-gemspec artifact production serves is invalid — it is missing verbatim makes bundler reject the vendored `path:` source and `bundle install` exit 1 on every bundler major. -**Mitigated CLI-side**: the gem vendor backend now validates the served stub -(conservative textual check for the required assignment lines). -`--vendor-source auto` detects the defect, warns +**Mitigated CLI-side**: the gem vendor backend now validates BOTH stub sources +at the write choke point (a conservative textual check matched to what +rubygems 3.3–3.6 actually hard-fails — empirically verified in the bundler-era +docker images). `--vendor-source auto` detects the served defect, warns (`vendor_prebuilt_stub_invalid`), and falls back to the local build — which is how `gem_bundler_vendored_install_proof` passes against production today — while explicit `--vendor-source service` refuses with -`vendor_prebuilt_stub_invalid`. The server-side stub-generator fix plus the -rebuild of all published gem artifacts are tracked in depscan; once deployed, -the same leg exercises the service artifact directly. +`vendor_prebuilt_stub_invalid`. When the gem is also not installed (no local +stub to derive), `auto` refuses with the same code, the served defect named, +and the install-the-gem remedy; a corrupted LOCAL `specifications/` stub +refuses `gem_spec_invalid`. Re-vendoring a project that committed a defective +stub pre-hardening re-validates the ON-DISK stub and rebuilds the artifact +with a valid one. The leg's route-attribution assertion (exactly one of +`vendor_prebuilt_downloaded` / `vendor_prebuilt_stub_invalid`) auto-retires +the fallback expectation once the depscan stub-generator fix deploys and the +rebuilt artifacts serve valid stubs — the same leg then exercises the service +artifact directly. ## Running