diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..943fa27 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,44 @@ +# github + +A nightly mirror of every repository under `OWNERS` into one Proton Drive folder, one +git bundle each. `README.md` says what it mirrors, how to restore a bundle, how it works +and how to fork it; [katoptra/lib](https://github.com/katoptra/lib)'s README is the +manual for the proton engine and everything the mirrors share. This file is what a +change must not break. + +Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. +`Taskfile.yml` and its comments are the design of what is this mirror's own: `OWNERS` +and `LIST_FLOOR` in root vars, `list`, the engine's two hooks `stage` and `prune`, +`report-mirror` and `offline`. The pipeline is the engine's. + +## Must knows + +- **The skip rests on reproducible bundles.** `git bundle create --all` from two mirror + clones of an unchanged repository is byte-identical, which is why the engine's upload + skips it and why nothing but the session is kept in the bucket. `offline` proves it; + do not add anything to a bundle that varies between clones. +- **One token per owner, read-only, and `list` refuses a listing that names another + owner.** A fine-grained token has one resource owner. A token that lists the wrong + owner is a swapped credential, and a listing under `LIST_FLOOR` is a broken token or + API, never an emptied account; either fails the run before `prune` can trash on it. +- **The session is this mirror's own.** Two mirrors sharing one race its rotating refresh + token. Never run `sync` or `plan` from a laptop while an Actions run may be going. +- **The logs are public.** git's stderr goes to `.run/git.err`, the CLI's to + `.run/pd.err`, a failure names a repository by its position in the listing, and the + token reaches git as a header through its environment, never on a command line. +- **A renamed repository is trashed under its old name and uploaded under the new.** + Proton's trash keeps it; `empty-trash` is by hand and asks first. +- **Unverified until a real run shows it:** whether the CLI uploads the dotfile bundle + `katoptra/.github.bundle`, and the exact JSON shape of `filesystem list -j`, which + `destination` and `prune` unwrap as `{"value": ...}` the way dropbox's provider does. + +## Verifying a change + +- `task check` renders every command of the pipeline inside the image and diffs it + against `render.txt`; `task render-update` accepts a change. The `check` workflow does + the same on every pull request, then `task run -- task offline`. +- `task run -- task offline`: a throwaway repository bundled twice inside the image; the + two must match. +- `task plan`: the real listing, clones and bundles, nothing uploaded. Needs the vault + and the session. +- The engine's own checks run in lib: `cd ../lib/examples/proton && task run -- task offline`. diff --git a/README.md b/README.md index a532b6e..1702c54 100644 --- a/README.md +++ b/README.md @@ -1,173 +1,178 @@ -## 🪞 github - -A nightly GitHub Actions run that mirrors every repository under -[jshvn](https://github.com/jshvn) and [katoptra](https://github.com/katoptra) into Proton -Drive. Each repository becomes one git bundle, `/.bundle`, under one folder of -your choosing: a mirror clone's every ref (branches, tags, pull-request heads) in one file -that `git clone name.bundle` restores. A repository that did not change uploads nothing, -because its bundle's bytes are the same and Proton's CLI skips a file whose content it -already holds; one that changed becomes a new revision, and Proton's version history is -the history of the mirror. A repository that left GitHub has its bundle moved to Proton's -trash. +# github -``` -GitHub (read-only tokens) - -> GitHub Actions run (this repo, inside one pinned toolbox image) - list -> clone --mirror, bundle each -> upload (one CLI call) -> confirm -> prune -> report -> ping - -> Proton Drive //.bundle - -> R2 .state/session.tar.age (the CLI session, and nothing else) -``` +[![sync](https://github.com/katoptra/github/actions/workflows/sync.yml/badge.svg)](https://github.com/katoptra/github/actions/workflows/sync.yml) +[![license](https://img.shields.io/github/license/katoptra/github)](LICENSE) + +A nightly mirror of every repository under the GitHub owners it names into one Proton +Drive folder. Each repository becomes one git bundle, `/.bundle`: a mirror +clone's every ref, branches, tags and pull-request heads, in one file that `git clone` +restores. A repository that did not change uploads nothing, because its bundle's bytes +are the same and Proton's CLI skips a file whose content it already holds; one that +changed becomes a new revision, and Proton's version history is the history of the +mirror. Nothing is kept between runs but the Proton CLI session, encrypted in a bucket. The repository is public and holds no account: every credential and every account -identifier lives in one 1Password vault and reaches a run by name. Nothing is kept between -runs but the Proton CLI session, encrypted in a Cloudflare R2 bucket. +identifier lives in one vault and reaches a run by name. + +## How to use -The toolbox that runs this, its image, the Proton engine and the two workflows this -repository calls are [katoptra/lib](https://github.com/katoptra/lib)'s, at `v2`. What is -here is the mirror's own: which owners, the floor under a listing, the `stage` and `prune` -hooks the engine leaves to a mirror, its rows of the report, and one offline check. +A bundle is a repository. To get one back: -## 🧭 How it works +```sh +git clone owner/name.bundle name # every branch, tag and pull-request head +git -C name remote set-url origin https://github.com/owner/name.git +git bundle list-heads owner/name.bundle # what it carries, without cloning +``` -One run is `task pipeline`, executed inside the toolbox image. The engine's verbs, in -order, with this mirror's two hooks: +Last night's copy is the current revision in Proton Drive; earlier nights are its +version history, as many as the plan keeps. A repository that left GitHub is in Proton's +trash under its old name, and a renamed one is trashed under the old name and uploaded +under the new. -| Step | Does | -|---|---| -| `clock` | Records the run's start. | -| `session` | Pulls the encrypted Proton CLI session from R2 and decrypts it into `.run/session`. Every CLI call afterwards seals it back when its refresh token rotated, whatever the call's exit. | -| `destination` | Lists the parent of the destination folder and refuses the run unless exactly one folder of that name exists and its UID is the one the vault names. | -| `stage` | This mirror's. `list` asks the API for every repository under each owner, with that owner's token, and refuses a listing under `LIST_FLOOR`. Then each repository is cloned as a mirror, bundled with `git bundle create --all` into `staging//.bundle`, and the clone deleted. | -| `upload` | One `proton-drive filesystem upload -f create-new-revision -d merge` of `staging/*` into the destination. The CLI skips identical files and makes a revision of changed ones. | -| `confirm` | The CLI's own summary must account for every staged file and folder with no failure, or the run fails and the next night retries everything, which costs nothing for what already landed. | -| `prune` | This mirror's. Lists each owner's folder in Proton and trashes bundles whose repository is no longer listed. A renamed repository is trashed under its old name and uploaded under the new. | -| `report`, `ping` | The run summary on the Actions job page, then the healthchecks.io ping. | - -Bundles are byte-for-byte reproducible for an unchanged repository; `task run -- task -offline` bundles a throwaway repository twice inside the image and compares, since the -skip rests on it. - -## 📦 Requirements - -- **A container engine**, running: Apple `container` on macOS, or Docker. Every command - here runs inside `ghcr.io/katoptra/toolbox:proton-v2`. Nothing else is installed on the - host. -- **[go-task](https://taskfile.dev/)**: `brew install go-task`. -- **The 1Password CLI** `op`, signed in, for anything that needs the vault on the laptop. -- **Accounts**: two GitHub tokens, Proton Drive, a Cloudflare R2 bucket, a healthchecks.io - check, and a 1Password vault. - -## 🚀 First-time setup - -Every value that names an account is stored in the vault and referenced in -[op.env](op.env), ten `op:///github/
/` lines, one item with five -sections: - -| Section | Fields | Reaches a run as | -|---|---|---| -| `github` | `token_jshvn`, `token_katoptra` | `MIRROR_GITHUB_TOKEN_JSHVN`, `MIRROR_GITHUB_TOKEN_KATOPTRA` | -| `proton` | `destination`, `destination_uid` | `MIRROR_PROTON_DESTINATION`, `MIRROR_PROTON_DESTINATION_UID` | -| `r2` | `access_key_id`, `secret_access_key`, `endpoint`, `bucket` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3`, `MIRROR_R2_BUCKET` | -| `age` | `identity` | `MIRROR_AGE_IDENTITY` | -| `healthcheck` | `url` | `HEALTHCHECK_URL` | - -1. **1Password.** An item `github` with the five sections above, in the vault `op.env` - names by UUID. This organization keeps one vault, `Katoptra`, with one item per mirror, - and one service account that reads it, stored as the organization secret - `OP_SERVICE_ACCOUNT_TOKEN`. A fork makes its own vault and service account and puts - the token in a repository secret of the same name. -2. **GitHub.** One fine-grained personal access token per owner, since a fine-grained token - has exactly one resource owner: one with jshvn as the owner, one with katoptra, each - for all repositories with `Contents: read` and `Metadata: read` and nothing else. The - mirror can never write to GitHub. An owner whose token lists another owner's - repositories fails the run. Store them as the two fields of section `github`. -3. **Proton Drive.** The CLI can only be seeded by a browser sign-in, so the session is - made once on the laptop and carried to CI encrypted. It is this mirror's own: two - mirrors sharing one session race its rotating refresh token, and the loser needs a - fresh login. Install the `proton-drive` CLI at the version - [katoptra/lib's lock](https://github.com/katoptra/lib/blob/main/toolchain.lock.toml) - pins, then: - - ```bash - PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive auth login - PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem create-folder /my-files GitHub - PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file proton-drive filesystem list -j /my-files - ``` - - Store `/my-files/GitHub` as `destination` and, from the listing, the `uid` of the entry - whose `name.value` is `GitHub` as `destination_uid`. Then, once the bucket exists, - `task session-seal -- .run/pd`. -4. **age.** `age-keygen` once; the `AGE-SECRET-KEY-...` line is `identity`. -5. **Cloudflare R2.** A bucket and an API token scoped to it. It holds one object. -6. **healthchecks.io.** A check on the nightly schedule with a grace period of an hour or - two; its ping URL is `url`. -7. **Prove it.** - - ```bash - task image # pull the toolbox image once - task check # every pipeline command rendered inside the image, diffed against render.txt - task run -- task offline # the bundle reproducibility check - task plan # the real thing, read-only: session, destination, list, clone and bundle; nothing uploaded - ``` - -8. **First run.** Dispatch the `sync` workflow from the Actions tab or with `gh workflow - run sync.yml`. The first run uploads every bundle; a quiet night afterwards uploads - none. -9. **Schedule.** Register `sync.yml` in jshvn/dispatch, once a day. - -## ▶️ Running it +## How it works -`task` alone prints the menu. Everything runs inside the toolbox. +Once a night a GitHub Actions job runs this pipeline inside the toolbox image from +[katoptra/lib](https://github.com/katoptra/lib). Every solid box is a verb of lib's +proton engine; the dashed ones are this mirror's own. -```bash -task plan # read-only: list, clone and bundle; prints what an upload would carry -task sync # one run, the same thing CI runs -task session-seal -- .run/pd # encrypt a laptop Proton CLI session into R2 -task empty-trash # permanently delete Proton trash; asks first; never scheduled -task check # render every pipeline command inside the image, diff against render.txt -task run -- task offline # bundle a throwaway repository twice; the two must match -task clean # delete .run, staging and the Taskfile cache +```mermaid +flowchart LR + clock --> session --> destination --> stage --> upload --> confirm --> prune --> report --> ping + stage -.-> list["list"] + report -.-> rm["report-mirror"] + classDef own stroke-dasharray: 5 5 + class stage,list,prune,rm own ``` -Never run `task sync` or `task plan` from the laptop while a CI run may be in progress: -both hold the one Proton session. +What this mirror owns, in [`Taskfile.yml`](Taskfile.yml): -## ⚙️ GitHub Actions +- **Its identity**, in root vars: `OWNERS`, the GitHub users or organizations to mirror, + and `LIST_FLOOR`, the count under which a listing is refused, so a broken token can + never turn an empty listing into a trash list. +- **`stage`**, the engine's hook for filling the staging tree. It runs `list`, which asks + the API for every repository under each owner with that owner's token and refuses a + listing that names another owner's repositories, then clones each as a mirror, bundles + it with `git bundle create --all`, and deletes the clone. +- **`prune`**, the engine's hook for what upstream dropped: list each owner's folder in + Proton and trash the bundles whose repository is no longer listed. +- **Its rows of the run summary**, and `offline`, which bundles a throwaway repository + twice inside the image and compares: the skip rests on bundles being byte-for-byte + reproducible for an unchanged repository. -[sync.yml](.github/workflows/sync.yml) is dispatch-only and calls katoptra/lib's reusable -`sync.yml`, which installs go-task and the 1Password CLI at the versions in lib's lock, -pulls the image and runs `task sync`. `concurrency: {group: sync, cancel-in-progress: -false}` queues a second dispatch behind a running one. A pipeline that fails still reports -and pings `/fail` inside the same container. Nothing in this repo starts a run; the nightly -dispatch comes from jshvn/dispatch. +The session, the destination check, the one-call upload and its confirmation are the +engine's and are documented once in +[lib's README](https://github.com/katoptra/lib#the-proton-engine). -[check.yml](.github/workflows/check.yml) runs on pull requests: the render diff and the -offline check, neither with access to the vault. +## Want your own? -On a public repository the run logs are public. They carry counts, the phase lines and the -report table. They never carry a repository name, a path in Proton, a token or an account -identifier: git's stderr goes to a file, a failure names a repository by its position in -the listing, the CLI's stderr goes to `.run/pd.err`, and `op run` masks every value it -resolved. +### 1. Fork it -## 📊 Reading a run +Fork [katoptra/github](https://github.com/katoptra/github). One line of `Taskfile.yml` is +yours to change, `OWNERS`; each owner needs a token line in `op.env` named +`MIRROR_GITHUB_TOKEN_`. Set `LIST_FLOOR` to about half the repositories you have. -The step summary is counts alone: repositories listed and bundled, files and MB staged, -what Proton reported (uploaded, skipped as identical, failed), bundles pruned, whether the -session was restored. A quiet night reads "0 uploaded, 24 skipped as identical": the 22 -bundles and their two folders. +### 2. Storage -## 🔧 Configuration +The bucket holds one object, the encrypted Proton CLI session, under `.state/`. Proton +holds the mirror. -Three root vars in [Taskfile.yml](Taskfile.yml): `OWNERS`, the space-separated owners, -each needing a `MIRROR_GITHUB_TOKEN_` line in `op.env`; `LIST_FLOOR`, the count -under which a listing is refused; and the includes' `IMAGE`. The account is the -environment, all of it `op.env`. +| What | Why | +|---|---| +| An R2 bucket, or any S3-compatible bucket | The session | +| An API token with Object Read & Write, scoped to that bucket | The `r2` values in step 3 | + +What the engine keeps in a bucket and why the session gets no history: +[lib, Storage](https://github.com/katoptra/lib#storage). + +### 3. Secrets + +Ten values, in one vault item named `github`, one section per service: + +| Section | Field | What it is | Reaches the run as | +|---|---|---|---| +| `github` | `token_`, one per owner | A fine-grained token, step 4 | `MIRROR_GITHUB_TOKEN_` | +| `proton` | `destination` | The CLI path of the folder, `/my-files/GitHub` | `MIRROR_PROTON_DESTINATION` | +| `proton` | `destination_uid` | That folder's UID | `MIRROR_PROTON_DESTINATION_UID` | +| `r2` | `access_key_id`, `secret_access_key` | The token from step 2 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | +| `r2` | `endpoint` | `https://.r2.cloudflarestorage.com` | `AWS_ENDPOINT_URL_S3` | +| `r2` | `bucket` | The bucket's name | `MIRROR_R2_BUCKET` | +| `age` | `identity` | An `AGE-SECRET-KEY-...` line from `age-keygen` | `MIRROR_AGE_IDENTITY` | +| `healthcheck` | `url` | Optional: a healthchecks.io ping URL | `HEALTHCHECK_URL` | + +Put your vault's UUID into the references in [`op.env`](op.env), make a service account +that can read that vault, and store its token as the `OP_SERVICE_ACCOUNT_TOKEN` secret, +on the organization or on the repository. Finding a vault's UUID and why a UUID and not +a name: [lib, Secrets](https://github.com/katoptra/lib#secrets). + +### 4. GitHub and Proton + +**Tokens.** One fine-grained personal access token per owner, since a fine-grained token +has exactly one resource owner: for all repositories, with `Contents: read` and +`Metadata: read` and nothing else. The mirror can never write to GitHub. Store each as +the `token_` field of section `github`. + +**The session.** The Proton CLI can only be seeded by a browser sign-in, so the session +is made once on a laptop and carried to every run encrypted; the commands, the folder and +its UID, and `task session-seal -- .run/pd` are in +[lib, The session](https://github.com/katoptra/lib#the-session). Store the folder's CLI +path as `destination` and its `uid` from the listing as `destination_uid`. This mirror's +session is its own: two mirrors sharing one race its rotating refresh token. + +### 5. Prove it, run it, schedule it + +On a laptop with go-task, the 1Password CLI and Docker or Apple `container`: + +```sh +task check # every pipeline command rendered inside the image, diffed against render.txt +task run -- task offline # the bundle reproducibility check; no network +task plan # the real thing, read-only: session, destination, list, clone and bundle; nothing uploaded +``` + +Then Actions, sync, Run workflow. The first run uploads every bundle; a quiet night +afterwards uploads none. Nothing in this repository schedules a run: add a `schedule:` +trigger to `.github/workflows/sync.yml`, or dispatch it from outside as this mirror is. + +## Operating it -## 🚫 What is not mirrored +`task` alone prints the menu. Everything runs inside the toolbox. + +```sh +task sync # one run, the same thing Actions runs +task plan # read-only: list, clone and bundle; prints what an upload would carry +task session-seal -- .run/pd # encrypt a laptop Proton CLI session into the bucket +task empty-trash # permanently delete Proton's trash; asks first; never scheduled +``` -Issues, pull-request discussion, release notes and attachments are not in git and GitHub's -migration export API is closed to ordinary accounts, so they are not here. Wikis would be -one more clone each; no repository under either owner has one today. +Never run `task sync` or `task plan` from a laptop while an Actions run may be in +progress: both hold the one Proton session, and the loser of a race needs a fresh login. + +Every run appends one table to its job page: when it started, the files and MB staged, +what Proton reported (uploaded, skipped as identical, failed), whether the session was +restored, repositories listed and bundled, bundles pruned. A quiet night reads "0 +uploaded, N skipped as identical", N being the bundles plus their owner folders. A failed +run is the only alert: healthchecks.io emails when a night passes without a ping. + +On a public repository the run logs are public. They carry counts and never a repository +name, a path in Proton, a token or an account identifier: git's stderr goes to a file, a +failure names a repository by its position in the listing, and the CLI's stderr goes to +`.run/pd.err`. + +- **The run fails at `session`.** The session is gone or its token was rotated out from + under it. Sign in again on a laptop and `task session-seal -- .run/pd`. +- **`destination` refuses the run.** The folder is not a direct child of its parent, or + its UID differs from the vault's. List the parent with `filesystem list -j` and fix the + field or the folder. +- **`list` refuses a listing.** Under `LIST_FLOOR`, or the token listed another owner's + repositories: a wrong or expired token. Nothing was trashed. +- **`confirm` fails.** Proton's summary did not account for every staged file. The next + night retries everything, which costs nothing for what already landed. + +## Reference + +Issues, pull-request discussion, release notes and attachments are not in git and +GitHub's migration export API is closed to ordinary accounts, so they are not here. A +wiki would be one more clone each; no repository under either owner has one today. + +Pull requests are welcome. MIT licensed. Built by [Josh Vaughen](https://ijosh.com). diff --git a/op.env b/op.env index 719fdb3..d5ae549 100644 --- a/op.env +++ b/op.env @@ -1,5 +1,5 @@ # op:// references only; `op run --env-file=op.env` resolves them at run time. -# Vault Katoptra, item github, addressed by UUID so a rename cannot break it. +# One vault, addressed by UUID so a rename cannot break it; item github. See katoptra/lib, Secrets. MIRROR_GITHUB_TOKEN_JSHVN=op://y6y6b6l2zjbpv7szc5ym5sprne/github/github/token_jshvn MIRROR_GITHUB_TOKEN_KATOPTRA=op://y6y6b6l2zjbpv7szc5ym5sprne/github/github/token_katoptra MIRROR_PROTON_DESTINATION=op://y6y6b6l2zjbpv7szc5ym5sprne/github/proton/destination