Skip to content

subtree tool: merge by default, add the deletion audit, rename it #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Peeja wants to merge 2 commits into
base: main
Choose a base branch
from
+246 −26
Open

subtree tool: merge by default, add the deletion audit, rename it #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2986ebb
subtree tool: merge by default, add the deletion audit, rename it
claude Sep 18, 2026
1a17fdc
subtree tool: a second script for the conflicts the rewrite itself ca…
claude Sep 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 77 additions & 12 deletions .github/scripts/subtree-conflicts.sh → .github/scripts/finish-subtree-pull.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/usr/bin/env bash
# Finish the merge a `git subtree pull` could not follow.
# Run this with every `git subtree pull` -- during one that stopped with
# conflicts, or straight after one that did not.
#
# It does two things, because a subtree pull can lose an upstream change in two
# different ways and only one of them is loud.
#
# When we move a file out of a subtree prefix, git's rename detection stops
# following it -- it does not see outside the prefix -- so every later upstream
Expand All @@ -18,8 +22,17 @@
# there `git status` reports UU, mergetool works, and `git add` resolves it.
# Nothing downstream has to be told this script was involved.
#
# subtree-conflicts.sh <prefix> merge, and write the results
# subtree-conflicts.sh --dry-run <prefix> print the diffs, change nothing
# THE SECOND THING, and the reason this is not called subtree-conflicts.sh any
# more: when upstream DELETES a file we had moved out of the prefix, both sides
# deleted that path, so git raises no conflict at all. The pull succeeds in
# silence and we go on carrying a file upstream removed. Nothing driven by
# conflicts can see that, so the audit below reads the merge itself -- upstream's
# own diff against the previous split point -- and reports deletions whose file
# we still have. It runs in both modes, including after a pull that had no
# conflicts whatsoever, which is exactly when it is the only thing looking.
#
# finish-subtree-pull.sh <prefix> merge, write the results, audit
# finish-subtree-pull.sh --dry-run <prefix> print the diffs, change nothing
#
# stdout is diffs and nothing else, so --dry-run can be read, graded or piped.
# Notes and problems go to stderr. Exit 1 if anything needed a human.
Expand All @@ -35,13 +48,26 @@ while [ $# -gt 0 ]; do
esac
done

prefix=${1:?usage: subtree-conflicts.sh [--dry-run] <prefix> (run during a conflicted subtree pull)}
prefix=${1:?usage: finish-subtree-pull.sh [--dry-run] <prefix> (run during or right after a subtree pull)}
prefix=${prefix%/}

git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1 || {
echo "No merge in progress. Run this while a subtree pull is conflicted." >&2
# Which mode we are in is detected, not flagged: a conflicted pull leaves
# MERGE_HEAD, and a finished one leaves a merge commit at HEAD whose second
# parent is the subtree split. Asking the caller to say which would be one more
# thing to get wrong on a day that is already going badly.
if git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1; then
pull_mode=conflicted
upstream=MERGE_HEAD
base=$(git merge-base HEAD MERGE_HEAD)
elif git rev-parse -q --verify 'HEAD^2' >/dev/null 2>&1; then
pull_mode=merged
upstream=HEAD^2
base=$(git merge-base 'HEAD^1' 'HEAD^2')
else
echo "No subtree pull to finish: no merge in progress, and HEAD is not a merge." >&2
echo "Run this during a conflicted pull, or immediately after any pull." >&2
exit 2
}
fi
cd "$(git rev-parse --show-toplevel)"

note() { printf '%s\n' "$*" >&2; }
Expand Down Expand Up @@ -155,12 +181,51 @@ while IFS= read -r conflicted; do
note "merged $conflicted -> $dest ($rc conflict(s); markers in the file, UU in git status)"
fi
merged=$((merged + 1)); rm -rf "$tmp"
done < <(git status --porcelain | awk '/^DU /{print substr($0,4)}')
done < <(
if [ "$pull_mode" = conflicted ]; then
git status --porcelain | awk '/^DU /{print substr($0,4)}'
fi
)

if [ "$merged" = 0 ] && [ "$unresolved" = 0 ]; then
if [ "$pull_mode" = conflicted ] && [ "$merged" = 0 ] && [ "$unresolved" = 0 ]; then
note "No 'deleted by us, modified by them' conflicts. Anything else here is an"
note "ordinary content conflict: resolve it normally."
exit 0
fi
note "$merged merged, $unresolved left for a human."
[ "$unresolved" = 0 ]

# THE AUDIT. Everything above reacts to a conflict; this reads the merge.
#
# When upstream deletes a file we had moved out of the prefix, BOTH sides
# deleted that path, so git raises nothing -- no conflict, no status entry,
# nothing for a conflict-driven tool to react to. The pull succeeds and we keep
# carrying a file upstream removed. This is the only thing that looks.
#
# Upstream's own history is in its own path space (no prefix), so its diff
# against the previous split point names deletions as upstream saw them, and we
# map each back through our prefix. -M matters: without it a rename upstream
# reads as a delete and every one would be a false positive.
stale=0
while IFS= read -r gone; do
[ -n "$gone" ] || continue
old=$prefix/$gone
dest=$(destination_of "$old")
if [ -n "$dest" ]; then
note "STILL HERE upstream deleted $gone; we moved it to $dest and still have it"
stale=$((stale + 1))
fi
done < <(git diff -M --diff-filter=D --name-only "$base" "$upstream" 2>/dev/null || true)

if [ "$stale" -gt 0 ]; then
note ""
note "$stale file(s) upstream deleted are still in this tree because we had moved"
note "them. git raised no conflict for these -- both sides deleted the old path --"
note "so nothing else would have mentioned them. Decide each one: upstream may have"
note "deleted dead code we are still carrying, or may have moved it somewhere this"
note "audit cannot see."
fi

if [ "$pull_mode" = merged ]; then
note "Audited a completed pull: $stale upstream deletion(s) we still carry."
else
note "$merged merged, $unresolved left for a human, $stale upstream deletion(s) we still carry."
fi
[ "$unresolved" = 0 ] && [ "$stale" = 0 ]
110 changes: 110 additions & 0 deletions .github/scripts/resolve-rewrite-conflicts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# Resolve the subtree-pull conflicts that are ONLY the monorepo's module-path
# rewrite, and refuse the rest.
#
# Every service arrived here with its import paths rewritten from
# github.com/fil-forge/<svc> to github.com/fil-forge/forge/<svc>. Upstream keeps
# editing those same import blocks, so a pull conflicts on every file where it
# touched one -- 14 of them across the eight prefixes of the last resync. Each
# is the same non-decision: our side says forge/<svc>, theirs says <svc>, and
# the actual change is somewhere else in the file.
#
# THE CHECK IS THE POINT, NOT THE FIX. Taking upstream's file and re-applying
# the rewrite is only safe if our side carries no judgement upstream could
# disagree with -- that is, if rewrite(base) is EXACTLY ours. This verifies that
# per file before touching any of them, and reports the ones that fail with what
# else is in there. go.mod and go.sum fail it every time, which is the check
# earning its keep: those carry real decisions (siblings pinned to v0.0.0 with
# `replace ../<svc>`, a unified libforge) that must be re-applied by hand.
#
# gofmt normalises both sides for .go files, and it is load-bearing rather than
# tidiness: the rewrite makes an import path longer, which can move it within
# its group, and gofmt sorts groups. Without normalising, a file whose only
# difference IS the rewrite compares unequal byte for byte and gets refused.
#
# resolve-rewrite-conflicts.sh resolve what qualifies, stage it
# resolve-rewrite-conflicts.sh --dry-run say what it would do, change nothing
#
# Exit 1 if anything was left for a human. Run it after finish-subtree-pull.sh,
# which handles a different class: files we moved out of the prefix, which git's
# rename detection cannot follow. The two do not overlap -- that one works on
# `deleted by us, modified by them`, this one on ordinary content conflicts --
# and neither sees the third class, a hunk that merged CLEANLY while carrying a
# polyrepo import path. Nothing reports that but a sweep of the tree afterwards.
set -uo pipefail

dry=0
case "${1:-}" in
--dry-run|-n) dry=1 ;;
"") ;;
*) echo "usage: resolve-rewrite-conflicts.sh [--dry-run]" >&2; exit 2 ;;
esac

cd "$(git rev-parse --show-toplevel)"

git rev-parse -q --verify MERGE_HEAD >/dev/null 2>&1 || {
echo "No merge in progress. Run this during a conflicted subtree pull." >&2
exit 2
}

# The services are the top-level directories with a go.mod -- the same set
# go.work lists. Nothing here needs updating when one is added.
svcs=$(for d in */; do [ -f "$d/go.mod" ] && printf '%s\n' "${d%/}"; done | paste -sd'|')
[ -n "$svcs" ] || { echo "no modules found -- is this the repository root?" >&2; exit 2; }

# Two rules, in order. The URL one first, because the import rule would
# otherwise turn a repository URL into .../forge/<svc>, which is a 404 that
# services serve out of GET /. (?!forge/) makes it idempotent; libforge is
# untouched because it is not one of the prefixes.
rewrite() {
perl -pe "
s{https://github\\.com/fil-forge/(?:$svcs)\\b}{https://github.com/fil-forge/forge}g;
s{github\\.com/fil-forge/(?!forge/)($svcs)\\b}{github.com/fil-forge/forge/\$1}g;
"
}

took=0 left=0
while IFS= read -r f; do
[ -n "$f" ] || continue
t=$(mktemp -d)

norm() { case "$f" in *.go) gofmt "$1" 2>/dev/null || cat "$1" ;; *) cat "$1" ;; esac; }

if ! git show ":1:$f" > "$t/base" 2>/dev/null; then
echo "LEFT $f: no merge base (add/add conflict)"
left=$((left + 1)); rm -rf "$t"; continue
fi
git show ":2:$f" > "$t/ours" 2>/dev/null || { echo "LEFT $f: deleted on our side; finish-subtree-pull.sh reports these"; left=$((left+1)); rm -rf "$t"; continue; }
git show ":3:$f" > "$t/theirs" 2>/dev/null || { echo "LEFT $f: deleted upstream; resolve by hand"; left=$((left+1)); rm -rf "$t"; continue; }

rewrite < "$t/base" > "$t/base_rw"
norm "$t/base_rw" > "$t/base_n"
norm "$t/ours" > "$t/ours_n"

if ! cmp -s "$t/base_n" "$t/ours_n"; then
echo "LEFT $f: ours differs from the base by more than the rewrite:"
diff "$t/base_n" "$t/ours_n" | head -12 | sed 's/^/ /'
left=$((left + 1)); rm -rf "$t"; continue
fi

rewrite < "$t/theirs" > "$t/out_raw"
norm "$t/out_raw" > "$t/out"

if [ "$dry" = 1 ]; then
echo "TAKE $f: ours is base+rewrite only; would take upstream and re-apply it"
else
cat "$t/out" > "$f"
git add -- "$f"
echo "TAKE $f: ours is base+rewrite only; took upstream and re-applied it"
fi
took=$((took + 1)); rm -rf "$t"
done < <(git diff --name-only --diff-filter=U)

echo "--- $took resolved, $left left for a human ---"
if [ "$left" -gt 0 ]; then
echo
echo "Each one above carries something besides the rewrite; merge it by hand."
echo "Then sweep the whole prefix for polyrepo import paths before trusting a"
echo "build: a hunk that merged cleanly can carry one, and raises no conflict."
fi
[ "$left" = 0 ]
73 changes: 59 additions & 14 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,19 +170,26 @@ rule 5 says ship none rather than a partial one. Keep them in a module's
`testutil` package with a named const and an env override, not inline in a
`_test.go`.

## When a `git subtree pull` conflicts, finish the merge it could not follow
## Run this with every `git subtree pull`

```
.github/scripts/subtree-conflicts.sh <prefix> merge and write
.github/scripts/subtree-conflicts.sh --dry-run <prefix> print the diffs only
.github/scripts/finish-subtree-pull.sh <prefix> merge, write, audit
.github/scripts/finish-subtree-pull.sh --dry-run <prefix> print the diffs only

.github/scripts/resolve-rewrite-conflicts.sh take upstream + rewrite
.github/scripts/resolve-rewrite-conflicts.sh --dry-run say what it would take
```

Run it **while the merge is still conflicted**. For every `deleted by us,
modified by them` entry it finds where that file lives now and performs the
three-way merge git would have performed if its rename detection had seen
outside the prefix. Everything it needs is already in the index: stage 1 is the
merge base, stage 3 is upstream, both at the old path, and our side is the file
at its new home — an ordinary three-way merge, so `git merge-file` does it.
Run it **during a pull that stopped with conflicts, or straight after one that
did not** — it detects which rather than being told. A subtree pull can lose an
upstream change in two ways and only one of them is loud, so it does two things.

**The loud one.** For every `deleted by us, modified by them` entry it finds
where that file lives now and performs the three-way merge git would have
performed if its rename detection had seen outside the prefix. Everything it
needs is already in the index: stage 1 is the merge base, stage 3 is upstream,
both at the old path, and our side is the file at its new home — an ordinary
three-way merge, so `git merge-file` does it.

**It hands the result back in git's own terms.** A clean merge is written and
staged. A conflicted one gets `--diff3` markers in the file *and* index stages
Expand Down Expand Up @@ -217,11 +224,49 @@ Things that are true and worth not rediscovering:
- **A rename/delete is reported at upstream's NEW path**, which never existed in
our history. `MERGE_HEAD` is in upstream's path space (no prefix), so its own
diff names the rename; the script uses that to map back.
- **One gap it cannot cover.** When upstream *deletes* a file we moved out, both
sides deleted the path, so git raises no conflict at all — the pull succeeds
silently and we keep carrying a file upstream removed. Nothing in a
conflict-driven tool can see that; it needs an audit that runs when nothing
conflicted.

**The quiet one, and the reason the name changed.** When upstream *deletes*
a file we had moved out of the prefix, **both sides deleted that path, so git
raises no conflict at all** — no status entry, nothing to react to. The pull
succeeds in silence and we keep carrying a file upstream removed. So the audit
does not wait to be asked: it reads upstream's own diff against the previous
split point and reports deletions whose file we still have. It runs in both
modes, including after a pull with no conflicts whatsoever, which is precisely
when it is the only thing looking. `-M` is load-bearing there — without it a
rename upstream reads as a delete and every one is a false positive.

Exit is non-zero if anything was left for a human *or* anything was flagged by
the audit, so a caller can act on the status rather than parse prose.

## The other half of the conflicts: `resolve-rewrite-conflicts.sh`

Everything above is about files we moved out of a prefix. The far more common
conflict is the one this repository creates by existing: every service's
imports were rewritten from `github.com/fil-forge/<svc>` to
`github.com/fil-forge/forge/<svc>`, so a pull conflicts on every file where
upstream touched that import block. **Fourteen of them in one resync**, each
the same non-decision.

`resolve-rewrite-conflicts.sh` takes upstream's file and re-applies the
rewrite — but only where that is provably safe.

**The check is the point, not the fix.** It is safe only if our side carries
nothing upstream could disagree with, so the script verifies per file that
`rewrite(base)` is *exactly* ours, and refuses with a diff when it is not.
`go.mod` and `go.sum` fail it every time, which is the check working: those
carry real decisions (siblings at `v0.0.0` with `replace ../<svc>`, a unified
libforge) that have to be re-applied by hand.

`gofmt` normalises both sides for `.go` files, and that is load-bearing, not
tidiness: the rewrite makes a path longer, which can move it within its import
group, and `gofmt` sorts groups. Without normalising, a file whose only
difference *is* the rewrite compares unequal and gets refused.

**Neither tool sees the third class.** A hunk can merge *cleanly* and still
carry a polyrepo import path, because it never touched a line we had rewritten
— true of a file upstream added and of an existing file that merely gained an
import. Nothing conflicts, so nothing reports it. `check-module-paths.sh` is
what does; run it after every pull, before trusting a build.

## Conventions

Expand Down
Loading