From 2986ebbf59b466bb2a7de31a451cb384154ceb68 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 20:14:33 +0000 Subject: [PATCH 1/2] subtree tool: merge by default, add the deletion audit, rename it Two changes, one because the tool was awkward to use and one because it had a hole. `--apply` is gone; merging is what it does. `--dry-run` prints a real unified diff (`git diff --no-index`) instead of the custom before/after format the old default emitted, so the output is a diff you can read, grade, pipe to `patch`, or hand to another tool. Nothing here needs a bespoke rendering of a diff. Almost every call is an agent finishing a pull; making the common path the default and the preview the flag matches that. The hole: the old tool was driven entirely by conflicts, and there is a way to lose an upstream change with no conflict at all. When upstream DELETES a file we had moved out of the prefix, both sides deleted that path -- git raises nothing, `git status` says nothing, the pull succeeds, and we go on carrying a file upstream removed. Nothing conflict-driven can see it, because there is no conflict to see. So the tool now also audits the merge itself: it reads upstream's own diff against the previous split point and reports deletions whose file we still have. `-M` is load-bearing there -- without it every upstream rename reads as a delete and each one is a false positive. That audit has to run when nothing conflicted, which is precisely when the old tool did not run at all. So the mode is detected rather than flagged: MERGE_HEAD means a conflicted pull in progress, a merge commit at HEAD means one that just finished. Either way, run it; hence the name, `finish-subtree-pull.sh`, and hence the section heading in AGENTS.md being an instruction rather than a condition. Exit is non-zero if anything was left for a human OR anything was flagged by the audit. Verified on synthetic subtree fixtures, both modes: - upstream modifies a file we moved out -> merged to its new home; conflicted result gets diff3 markers in the file and stages 1/2/3 at the new path, so `git status` shows UU and `git add` resolves it - upstream deletes a file we moved out -> flagged STILL HERE, exit 1, in both the conflicted-pull and completed-pull modes - upstream RENAMES a file we moved out -> mapped back through MERGE_HEAD and merged cleanly, and NOT flagged as a deletion (the -M case) - upstream deletes a file we never moved -> git deletes it, nothing flagged, exit 0 - --dry-run leaves the working tree and index untouched Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt --- ...ee-conflicts.sh => finish-subtree-pull.sh} | 89 ++++++++++++++++--- AGENTS.md | 40 ++++++--- 2 files changed, 103 insertions(+), 26 deletions(-) rename .github/scripts/{subtree-conflicts.sh => finish-subtree-pull.sh} (63%) diff --git a/.github/scripts/subtree-conflicts.sh b/.github/scripts/finish-subtree-pull.sh similarity index 63% rename from .github/scripts/subtree-conflicts.sh rename to .github/scripts/finish-subtree-pull.sh index 2d8de024..990fda77 100755 --- a/.github/scripts/subtree-conflicts.sh +++ b/.github/scripts/finish-subtree-pull.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash -# Finish the merge a `git subtree pull` could not follow. +# Run this with every `git subtree pull` -- during one that stopped with +# conflicts, or straight after one that did not. +# +# It does two things, because a subtree pull can lose an upstream change in two +# different ways and only one of them is loud. # # When we move a file out of a subtree prefix, git's rename detection stops # following it -- it does not see outside the prefix -- so every later upstream @@ -18,8 +22,17 @@ # there `git status` reports UU, mergetool works, and `git add` resolves it. # Nothing downstream has to be told this script was involved. # -# subtree-conflicts.sh merge, and write the results -# subtree-conflicts.sh --dry-run print the diffs, change nothing +# THE SECOND THING, and the reason this is not called subtree-conflicts.sh any +# more: when upstream DELETES a file we had moved out of the prefix, both sides +# deleted that path, so git raises no conflict at all. The pull succeeds in +# silence and we go on carrying a file upstream removed. Nothing driven by +# conflicts can see that, so the audit below reads the merge itself -- upstream's +# own diff against the previous split point -- and reports deletions whose file +# we still have. It runs in both modes, including after a pull that had no +# conflicts whatsoever, which is exactly when it is the only thing looking. +# +# finish-subtree-pull.sh merge, write the results, audit +# finish-subtree-pull.sh --dry-run print the diffs, change nothing # # stdout is diffs and nothing else, so --dry-run can be read, graded or piped. # Notes and problems go to stderr. Exit 1 if anything needed a human. @@ -35,13 +48,26 @@ while [ $# -gt 0 ]; do esac done -prefix=${1:?usage: subtree-conflicts.sh [--dry-run] (run during a conflicted subtree pull)} +prefix=${1:?usage: finish-subtree-pull.sh [--dry-run] (run during or right after a subtree pull)} prefix=${prefix%/} -git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1 || { - echo "No merge in progress. Run this while a subtree pull is conflicted." >&2 +# Which mode we are in is detected, not flagged: a conflicted pull leaves +# MERGE_HEAD, and a finished one leaves a merge commit at HEAD whose second +# parent is the subtree split. Asking the caller to say which would be one more +# thing to get wrong on a day that is already going badly. +if git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1; then + pull_mode=conflicted + upstream=MERGE_HEAD + base=$(git merge-base HEAD MERGE_HEAD) +elif git rev-parse -q --verify 'HEAD^2' >/dev/null 2>&1; then + pull_mode=merged + upstream=HEAD^2 + base=$(git merge-base 'HEAD^1' 'HEAD^2') +else + echo "No subtree pull to finish: no merge in progress, and HEAD is not a merge." >&2 + echo "Run this during a conflicted pull, or immediately after any pull." >&2 exit 2 -} +fi cd "$(git rev-parse --show-toplevel)" note() { printf '%s\n' "$*" >&2; } @@ -155,12 +181,51 @@ while IFS= read -r conflicted; do note "merged $conflicted -> $dest ($rc conflict(s); markers in the file, UU in git status)" fi merged=$((merged + 1)); rm -rf "$tmp" -done < <(git status --porcelain | awk '/^DU /{print substr($0,4)}') +done < <( + if [ "$pull_mode" = conflicted ]; then + git status --porcelain | awk '/^DU /{print substr($0,4)}' + fi +) -if [ "$merged" = 0 ] && [ "$unresolved" = 0 ]; then +if [ "$pull_mode" = conflicted ] && [ "$merged" = 0 ] && [ "$unresolved" = 0 ]; then note "No 'deleted by us, modified by them' conflicts. Anything else here is an" note "ordinary content conflict: resolve it normally." - exit 0 fi -note "$merged merged, $unresolved left for a human." -[ "$unresolved" = 0 ] + +# THE AUDIT. Everything above reacts to a conflict; this reads the merge. +# +# When upstream deletes a file we had moved out of the prefix, BOTH sides +# deleted that path, so git raises nothing -- no conflict, no status entry, +# nothing for a conflict-driven tool to react to. The pull succeeds and we keep +# carrying a file upstream removed. This is the only thing that looks. +# +# Upstream's own history is in its own path space (no prefix), so its diff +# against the previous split point names deletions as upstream saw them, and we +# map each back through our prefix. -M matters: without it a rename upstream +# reads as a delete and every one would be a false positive. +stale=0 +while IFS= read -r gone; do + [ -n "$gone" ] || continue + old=$prefix/$gone + dest=$(destination_of "$old") + if [ -n "$dest" ]; then + note "STILL HERE upstream deleted $gone; we moved it to $dest and still have it" + stale=$((stale + 1)) + fi +done < <(git diff -M --diff-filter=D --name-only "$base" "$upstream" 2>/dev/null || true) + +if [ "$stale" -gt 0 ]; then + note "" + note "$stale file(s) upstream deleted are still in this tree because we had moved" + note "them. git raised no conflict for these -- both sides deleted the old path --" + note "so nothing else would have mentioned them. Decide each one: upstream may have" + note "deleted dead code we are still carrying, or may have moved it somewhere this" + note "audit cannot see." +fi + +if [ "$pull_mode" = merged ]; then + note "Audited a completed pull: $stale upstream deletion(s) we still carry." +else + note "$merged merged, $unresolved left for a human, $stale upstream deletion(s) we still carry." +fi +[ "$unresolved" = 0 ] && [ "$stale" = 0 ] diff --git a/AGENTS.md b/AGENTS.md index 6a545f64..642b7cb3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -170,19 +170,23 @@ rule 5 says ship none rather than a partial one. Keep them in a module's `testutil` package with a named const and an env override, not inline in a `_test.go`. -## When a `git subtree pull` conflicts, finish the merge it could not follow +## Run this with every `git subtree pull` ``` -.github/scripts/subtree-conflicts.sh merge and write -.github/scripts/subtree-conflicts.sh --dry-run print the diffs only +.github/scripts/finish-subtree-pull.sh merge, write, audit +.github/scripts/finish-subtree-pull.sh --dry-run print the diffs only ``` -Run it **while the merge is still conflicted**. For every `deleted by us, -modified by them` entry it finds where that file lives now and performs the -three-way merge git would have performed if its rename detection had seen -outside the prefix. Everything it needs is already in the index: stage 1 is the -merge base, stage 3 is upstream, both at the old path, and our side is the file -at its new home — an ordinary three-way merge, so `git merge-file` does it. +Run it **during a pull that stopped with conflicts, or straight after one that +did not** — it detects which rather than being told. A subtree pull can lose an +upstream change in two ways and only one of them is loud, so it does two things. + +**The loud one.** For every `deleted by us, modified by them` entry it finds +where that file lives now and performs the three-way merge git would have +performed if its rename detection had seen outside the prefix. Everything it +needs is already in the index: stage 1 is the merge base, stage 3 is upstream, +both at the old path, and our side is the file at its new home — an ordinary +three-way merge, so `git merge-file` does it. **It hands the result back in git's own terms.** A clean merge is written and staged. A conflicted one gets `--diff3` markers in the file *and* index stages @@ -217,11 +221,19 @@ Things that are true and worth not rediscovering: - **A rename/delete is reported at upstream's NEW path**, which never existed in our history. `MERGE_HEAD` is in upstream's path space (no prefix), so its own diff names the rename; the script uses that to map back. -- **One gap it cannot cover.** When upstream *deletes* a file we moved out, both - sides deleted the path, so git raises no conflict at all — the pull succeeds - silently and we keep carrying a file upstream removed. Nothing in a - conflict-driven tool can see that; it needs an audit that runs when nothing - conflicted. + +**The quiet one, and the reason the name changed.** When upstream *deletes* +a file we had moved out of the prefix, **both sides deleted that path, so git +raises no conflict at all** — no status entry, nothing to react to. The pull +succeeds in silence and we keep carrying a file upstream removed. So the audit +does not wait to be asked: it reads upstream's own diff against the previous +split point and reports deletions whose file we still have. It runs in both +modes, including after a pull with no conflicts whatsoever, which is precisely +when it is the only thing looking. `-M` is load-bearing there — without it a +rename upstream reads as a delete and every one is a false positive. + +Exit is non-zero if anything was left for a human *or* anything was flagged by +the audit, so a caller can act on the status rather than parse prose. ## Conventions From 1a17fdc5cfea69c138af0c90ddc3fd6b8fd10b3c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 18 Sep 2026 20:54:10 +0000 Subject: [PATCH 2/2] subtree tool: a second script for the conflicts the rewrite itself causes finish-subtree-pull.sh handles files we moved OUT of a prefix, which git's rename detection cannot follow. That is the rare case. The common one is the conflict this repository creates by existing: every service's imports were rewritten to github.com/fil-forge/forge/, so a pull conflicts on every file where upstream touched that import block -- fourteen of them in the resync of eight prefixes, each the same non-decision. resolve-rewrite-conflicts.sh takes upstream's file and re-applies the rewrite, but only where that is provably safe, and THE CHECK IS THE POINT. It verifies per file that rewrite(base) is exactly ours -- meaning our side carries nothing upstream could disagree with -- and refuses with a diff when it is not. gofmt normalises both sides for .go files, which is load-bearing rather than tidy: the rewrite makes a path longer, which can move it within its import group, and gofmt sorts groups. Without it, a file whose only difference IS the rewrite compares unequal byte for byte and gets refused as a hand merge. Verified against the real thing rather than a fixture. Replaying the sprue pull of the resync branch, from origin/main, with up-sprue at 506f5f6: - 3 resolved, 8 left, matching the hand resolution exactly - all three resolved files are BYTE-IDENTICAL to what was committed after resolving them by hand, which is the strongest available evidence that those hand resolutions were right - go.mod and go.sum refused, with the sibling pins and the libforge difference printed as the reason - the six modify/delete workflow conflicts reported as not its business And proved to fail on the defect: appending one unrelated comment line to our side of ucan_conclude_http_put.go makes it refuse that file and print the added line, while still taking its sibling in the same run. --- .github/scripts/resolve-rewrite-conflicts.sh | 110 +++++++++++++++++++ AGENTS.md | 33 ++++++ 2 files changed, 143 insertions(+) create mode 100755 .github/scripts/resolve-rewrite-conflicts.sh diff --git a/.github/scripts/resolve-rewrite-conflicts.sh b/.github/scripts/resolve-rewrite-conflicts.sh new file mode 100755 index 00000000..9af23700 --- /dev/null +++ b/.github/scripts/resolve-rewrite-conflicts.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# Resolve the subtree-pull conflicts that are ONLY the monorepo's module-path +# rewrite, and refuse the rest. +# +# Every service arrived here with its import paths rewritten from +# github.com/fil-forge/ to github.com/fil-forge/forge/. Upstream keeps +# editing those same import blocks, so a pull conflicts on every file where it +# touched one -- 14 of them across the eight prefixes of the last resync. Each +# is the same non-decision: our side says forge/, theirs says , and +# the actual change is somewhere else in the file. +# +# THE CHECK IS THE POINT, NOT THE FIX. Taking upstream's file and re-applying +# the rewrite is only safe if our side carries no judgement upstream could +# disagree with -- that is, if rewrite(base) is EXACTLY ours. This verifies that +# per file before touching any of them, and reports the ones that fail with what +# else is in there. go.mod and go.sum fail it every time, which is the check +# earning its keep: those carry real decisions (siblings pinned to v0.0.0 with +# `replace ../`, a unified libforge) that must be re-applied by hand. +# +# gofmt normalises both sides for .go files, and it is load-bearing rather than +# tidiness: the rewrite makes an import path longer, which can move it within +# its group, and gofmt sorts groups. Without normalising, a file whose only +# difference IS the rewrite compares unequal byte for byte and gets refused. +# +# resolve-rewrite-conflicts.sh resolve what qualifies, stage it +# resolve-rewrite-conflicts.sh --dry-run say what it would do, change nothing +# +# Exit 1 if anything was left for a human. Run it after finish-subtree-pull.sh, +# which handles a different class: files we moved out of the prefix, which git's +# rename detection cannot follow. The two do not overlap -- that one works on +# `deleted by us, modified by them`, this one on ordinary content conflicts -- +# and neither sees the third class, a hunk that merged CLEANLY while carrying a +# polyrepo import path. Nothing reports that but a sweep of the tree afterwards. +set -uo pipefail + +dry=0 +case "${1:-}" in + --dry-run|-n) dry=1 ;; + "") ;; + *) echo "usage: resolve-rewrite-conflicts.sh [--dry-run]" >&2; exit 2 ;; +esac + +cd "$(git rev-parse --show-toplevel)" + +git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1 || { + echo "No merge in progress. Run this during a conflicted subtree pull." >&2 + exit 2 +} + +# The services are the top-level directories with a go.mod -- the same set +# go.work lists. Nothing here needs updating when one is added. +svcs=$(for d in */; do [ -f "$d/go.mod" ] && printf '%s\n' "${d%/}"; done | paste -sd'|') +[ -n "$svcs" ] || { echo "no modules found -- is this the repository root?" >&2; exit 2; } + +# Two rules, in order. The URL one first, because the import rule would +# otherwise turn a repository URL into .../forge/, which is a 404 that +# services serve out of GET /. (?!forge/) makes it idempotent; libforge is +# untouched because it is not one of the prefixes. +rewrite() { + perl -pe " + s{https://github\\.com/fil-forge/(?:$svcs)\\b}{https://github.com/fil-forge/forge}g; + s{github\\.com/fil-forge/(?!forge/)($svcs)\\b}{github.com/fil-forge/forge/\$1}g; + " +} + +took=0 left=0 +while IFS= read -r f; do + [ -n "$f" ] || continue + t=$(mktemp -d) + + norm() { case "$f" in *.go) gofmt "$1" 2>/dev/null || cat "$1" ;; *) cat "$1" ;; esac; } + + if ! git show ":1:$f" > "$t/base" 2>/dev/null; then + echo "LEFT $f: no merge base (add/add conflict)" + left=$((left + 1)); rm -rf "$t"; continue + fi + git show ":2:$f" > "$t/ours" 2>/dev/null || { echo "LEFT $f: deleted on our side; finish-subtree-pull.sh reports these"; left=$((left+1)); rm -rf "$t"; continue; } + git show ":3:$f" > "$t/theirs" 2>/dev/null || { echo "LEFT $f: deleted upstream; resolve by hand"; left=$((left+1)); rm -rf "$t"; continue; } + + rewrite < "$t/base" > "$t/base_rw" + norm "$t/base_rw" > "$t/base_n" + norm "$t/ours" > "$t/ours_n" + + if ! cmp -s "$t/base_n" "$t/ours_n"; then + echo "LEFT $f: ours differs from the base by more than the rewrite:" + diff "$t/base_n" "$t/ours_n" | head -12 | sed 's/^/ /' + left=$((left + 1)); rm -rf "$t"; continue + fi + + rewrite < "$t/theirs" > "$t/out_raw" + norm "$t/out_raw" > "$t/out" + + if [ "$dry" = 1 ]; then + echo "TAKE $f: ours is base+rewrite only; would take upstream and re-apply it" + else + cat "$t/out" > "$f" + git add -- "$f" + echo "TAKE $f: ours is base+rewrite only; took upstream and re-applied it" + fi + took=$((took + 1)); rm -rf "$t" +done < <(git diff --name-only --diff-filter=U) + +echo "--- $took resolved, $left left for a human ---" +if [ "$left" -gt 0 ]; then + echo + echo "Each one above carries something besides the rewrite; merge it by hand." + echo "Then sweep the whole prefix for polyrepo import paths before trusting a" + echo "build: a hunk that merged cleanly can carry one, and raises no conflict." +fi +[ "$left" = 0 ] diff --git a/AGENTS.md b/AGENTS.md index 642b7cb3..8710e5d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -175,6 +175,9 @@ rule 5 says ship none rather than a partial one. Keep them in a module's ``` .github/scripts/finish-subtree-pull.sh merge, write, audit .github/scripts/finish-subtree-pull.sh --dry-run print the diffs only + +.github/scripts/resolve-rewrite-conflicts.sh take upstream + rewrite +.github/scripts/resolve-rewrite-conflicts.sh --dry-run say what it would take ``` Run it **during a pull that stopped with conflicts, or straight after one that @@ -235,6 +238,36 @@ rename upstream reads as a delete and every one is a false positive. Exit is non-zero if anything was left for a human *or* anything was flagged by the audit, so a caller can act on the status rather than parse prose. +## The other half of the conflicts: `resolve-rewrite-conflicts.sh` + +Everything above is about files we moved out of a prefix. The far more common +conflict is the one this repository creates by existing: every service's +imports were rewritten from `github.com/fil-forge/` to +`github.com/fil-forge/forge/`, so a pull conflicts on every file where +upstream touched that import block. **Fourteen of them in one resync**, each +the same non-decision. + +`resolve-rewrite-conflicts.sh` takes upstream's file and re-applies the +rewrite — but only where that is provably safe. + +**The check is the point, not the fix.** It is safe only if our side carries +nothing upstream could disagree with, so the script verifies per file that +`rewrite(base)` is *exactly* ours, and refuses with a diff when it is not. +`go.mod` and `go.sum` fail it every time, which is the check working: those +carry real decisions (siblings at `v0.0.0` with `replace ../`, a unified +libforge) that have to be re-applied by hand. + +`gofmt` normalises both sides for `.go` files, and that is load-bearing, not +tidiness: the rewrite makes a path longer, which can move it within its import +group, and `gofmt` sorts groups. Without normalising, a file whose only +difference *is* the rewrite compares unequal and gets refused. + +**Neither tool sees the third class.** A hunk can merge *cleanly* and still +carry a polyrepo import path, because it never touched a line we had rewritten +— true of a file upstream added and of an existing file that merely gained an +import. Nothing conflicts, so nothing reports it. `check-module-paths.sh` is +what does; run it after every pull, before trusting a build. + ## Conventions - **Never hand-transcribe a digest or a sha.** Resolve and apply it with one