Sanitized, generalized write-ups of infrastructure patterns I've designed and operated in production - distilled into Architecture Decision Records (ADRs) so the reasoning is visible even where the code can't be.
Most of this comes out of running a multi-portal PHP/MySQL reservations platform plus its surrounding tooling (a Python agent-orchestration host, Docker-based Manager Sandboxes, a server-side deploy pipeline) as its lead engineer. Those repositories are private; these patterns are the parts that generalize, written at the architecture level - no hostnames, addresses, credentials, or vendor specifics.
- A Manager Sandbox is a role-specific Tourbot instance running in a Docker container. It gives a manager and AI assistant an isolated place to explore data and prototype changes before human-reviewed promotion.
- A Factory Worker is one ETA Factory agent attempt running inside a short-lived Firecracker microVM. It is not a Manager Sandbox.
- The Kubernetes Demo is a separate single-node k3s learning and portfolio environment. It is not part of ETA production, and Manager Sandboxes do not run on it.
- The Unattended Loop is a single-operator coding loop that picks up a ticket, runs each agent iteration in a fresh microVM on a dedicated low-credential box, and opens a draft pull request with nobody watching. It is not a Manager Sandbox and not a Factory Worker; it is the third execution plane, and ADRs 0023-0026 are about it.
Most of these patterns cluster on one seam: safely running autonomous AI agents against revenue-critical legacy systems. Per-tenant isolation (0001), snapshot-fed writable sandboxes (0003), a scoped agent identity (0005), and default-deny, host-pinned data access (0009) are four boundaries around the same problem: give an agent real production data and real reach without giving it the ability to damage the business. ADR 0010 covers the legacy-modernization case itself - proving an agent's edits to opaque, generated markup moved nothing visible before a human is asked to approve.
A second cluster, ADRs 0023 through 0027, is the unattended version of the same problem: when nobody is watching the agent, which controls survive single-operator scale (a per-iteration microVM boundary and a short, asserted credential inventory) and which do not (the rest of the multi-tenant factory).
Each decision below is one where a non-obvious trade-off was made and can be defended. That's the useful unit: not "what I used," but "what I chose, what I gave up, and when I'd choose differently."
These also read on the web, rendered from this repo, at stephens.page/decisions.
| # | Decision | Trade-off in one line |
|---|---|---|
| 0001 | Docker over bare-metal for per-tenant isolation | Pay image/ops overhead to get strong filesystem + DB-user isolation cheaply |
| 0002 | External managed DB over a containerized one | Give up "one compose up" simplicity for durable, backup-friendly state |
| 0003 | Periodic snapshot over live replication for Manager Sandboxes | Accept some staleness to gain isolation, reset-ability, and no prod write-path risk |
| 0004 | A guarded shell deploy over a hosted CI runner | Forgo ecosystem features for a dependency-free, auditable single-server deploy |
| 0005 | A scoped system user over a shared service account for an autonomous agent | More host setup in exchange for clean per-action auditing and least privilege |
| 0006 | Binlog-tailing daemons over database triggers for denormalization | Accept eventual consistency to keep derive-logic in versioned code, off the hot write path |
| 0007 | Pull-based health probing over push agents for a small fleet | Forgo deep metrics/history to keep the monitored fleet agent-free and the failure domain legible |
| 0008 | Embedded SQLite over a networked DB for single-node tooling | Give up cross-host sharing for zero operational surface on state one process owns |
| 0009 | Default-deny, host-pinned DB access over a trusted network | Take on provisioning friction so a leaked credential isn't portable off its host |
| 0010 | A pixel-equality gate over diff review for changes to generated markup | Pay a render/diff harness to safely change markup you can't audit by eye - it proves visual, not semantic, equality |
| 0011 | A pull-based metrics stack over a bespoke prober, data plane kept private | Take on a TSDB to run for real metrics/history/alerting - and bind the unauthenticated parts to loopback, exposing only one read-only pane |
| 0012 | Per-app copy-deployed sync services over a shared multi-tenant backend | Run N near-identical small services to gain physical blast-radius isolation and per-app sync-model freedom, at the cost of hand-applying shared-auth fixes across copies |
| 0013 | Staged declarative provisioning (Terraform + cloud-init + separate deploy) over one imperative bootstrap script | Take on Terraform state and a deliberately create-only provisioning token for a single box to get a reproducible, reviewable host and a clean provision/configure/deploy seam |
| 0014 | Import ~220 live DNS records into Terraform over recreating them from a desired-state list | Accept verbose generated config and provider quirks to adopt traffic-serving records with zero downtime and a no-op baseline plan, instead of risking duplicate-creates and silent deletion of forgotten records |
| 0015 | Keep Terraform state on a different provider than the compute it provisions | Take on a second vendor + scoped IAM key so a provider-level outage can't destroy both the infrastructure and the state needed to rebuild it |
| 0016 | Enforce cluster posture at admission (OPA/Gatekeeper + a VAP) over trusting reviewed manifests | Run a policy controller so the hardened posture is rejected-if-violated at the API server instead of relying on review - the control that matters once a second actor or an agent can apply to the cluster |
| 0017 | Self-host the signing instrument and its audit trail over a SaaS signature service | Take on one container plus its own cert and upkeep so the legally-binding document and its audit trail stay on infra you govern, with a named-human gate on every consequential action - the highest-stakes case of keeping the authoritative copy where you control it |
| 0018 | Runtime-injected secrets over plaintext config, with the store matched to the team | Hold one invariant (encrypted at rest, injected into the environment at runtime) and pay for two backends - a broker where a team needs sharing/revocation/audit, SOPS + age where a solo fleet needs offline zero-vendor recovery - rather than force one tool to fit both |
| 0019 | Reuse an existing passkey session to gate a second app (Caddy forward_auth) over standing up a second auth stack | Gate a second internal app by delegating to one hardened passkey session via Caddy forward_auth - ~40 lines of Caddyfile, zero new auth code, every tool inheriting the gate's future hardening - at the cost of concentrating trust in a single session |
| 0020 | A private mesh for operator shells, public MFA-gated endpoints for browser consoles, over one VPN for everything | Put SSH behind a WireGuard mesh and drop public :22, but keep browser admin consoles public behind per-audience MFA - size the boundary to who uses it, rather than hide the consoles non-technical staff reach by URL behind a VPN client they can't maintain |
| 0021 | A shared OpenTelemetry collector seam over direct backend wiring | Add one collector as the ingestion seam so new telemetry sources and backends are configuration, not new architecture - agent usage metrics only, never prompt content |
| 0022 | Complete the logs pillar with a co-located local-disk Loki behind the collector seam, over a dedicated or hosted log stack | Add single-binary Loki beside Prometheus/Tempo on the one monitoring node, ingested only through the shared collector seam and bound to localhost - one query surface and native trace-to-logs for near-zero new infrastructure, at the cost of a deeper single-box blast radius and a 31-day local-disk retention ceiling with object storage as the written-down escape hatch |
| 0023 | Attendedness is a fourth trust axis: a per-iteration microVM boundary for unattended agents, even at single-operator scale | Re-earn exactly one subsystem - the execution boundary - because "I notice and fix" is the premise that lets solo-scale isolation collapse, and an unattended run is defined by nobody noticing; everything else in the heavy factory stays deleted |
| 0024 | A short, script-asserted credential inventory on the agent box; the inventory, not the network hop, is the isolation seam | Three base credentials plus one repo token per target, asserted in both directions before every dispatch - so the box's reach is enumerable, single-host mode is gated on passing the same check, and every "just add a mail credential" reopens the two lists whose value is being short |
| 0025 | The model credential inside the boundary: a long-lived subscription token by environment, over per-iteration renewal or a metered key | Keep the flat-rate cost control and give up proxy-injected isolation - bounded instead by deny-all egress, a sandbox that dies per iteration, a token that revokes alone, and a diff scan that refuses to push the token pattern |
| 0026 | The agent is a substitutable command: one boundary harness, vendor facts in a leaf | Pay a harness/leaf split and double-asserted structural tests so the loop is a claim about a technique rather than a vendor, and a third agent is a leaf, not a second copy of the lifecycle |
| 0027 | Enroll an unsigned third-party executable by pinning the exact artifact and keying its acceptance proof to the pin | Accept an unsignable IaC provider and a self-updating agent runner by pinning bytes, proving the pin with a real lifecycle, refusing work until the proof matches the installed pin, and never letting the dependency be the interface - a false pause is recoverable, a silently dropped boundary is not |
| 0028 | One secret manifest per consumer, generated from a tracked declaration; validate every manifest the machine reads | Take on a generator, a startup assertion, and a constants file so a vault rename kills one consumer instead of the box - after a 252-reference shared manifest took down four units for six hours while the validator printed green |
| 0029 | Pin the shared serving checkout fast-forward-only, refuse-and-alert on a dirty tree, edit only in worktrees | Give up "restart and it's live" on one service to stop the inverted-signal failure where the newest commit carries the oldest content - the pin is a detector, the default worktree and a stale-path pre-commit hook are the barrier |
| 0030 | Immutable per-commit releases, atomic promotion, an append-only journal; re-promotion is a named attended operation; root-pinned authority drift is reported, never a failure | Pay a promoter and a journal so no exit leaves production unnamed, hand-editing the symlink is the thing the design refuses, and a merge can never expand what root pinned - drift is shown, not paged |
| 0031 | A CI deploy key that reaches a shared host, restricted to one installed forced command, over having the host pull | Let a CI secret reach a host that serves client work because it cannot open a shell - forced command outside the deployed tree, pinned host key - in exchange for sub-minute deploys without a listener or a timer |
| 0032 | The first unauthenticated-ingestion workload gets its own host; store private by construction; admin gated by a named permission; masking proof per property; abort written before launch | Pay a dedicated small host and a volume so browser-driven POSTs never land beside fleet keys or prod sandboxes, and prove masking by grepping the store for planted strings rather than looking at the player |
| 0033 | Grow the monitoring host's own disk over a block volume, and cap retention by size | Take a one-way, power-off resize because a grandfathered allocation made it free - and accept that the bigger disk is not the guard; the size-based retention cap and log hygiene are |
| 0034 | A strict root allowlist returning 404 over a growing blocklist returning 403, for a legacy repository-as-webroot | Fail closed against files nobody anticipated and stop confirming their existence to scanners, without the multi-quarter refactor of moving the document root - which the allowlist now guards |
| 0035 | Let the hardened service create namespaces so the sandbox can confine the model; hold the boundary in AppArmor; prove from the live unit | Loosen one systemd restriction that no narrower setting can express, keep the kernel and profile layers that make it safe, and never again certify a sandbox from an attended shell that the service could not start |
A concrete, sanitized companion to ADR 0011 lives in
observability/: the Prometheus/Alertmanager config and
Grafana dashboards that make the pattern reproducible. A companion to ADRs
0013-0015 - the full Terraform/Cloudflare/Ansible DNS-as-code repo, sanitized -
lives at
terraform-cloudflare-dns.
The runtime-secrets launcher that ADRs 0018 and 0028 describe - manifest
validation over every file a machine reads - lives at
vaulted-agent.
The unattended loop behind ADRs 0023-0026 lives at
tracewake.
A companion to ADR 0016 - the separate, non-production Kubernetes Demo whose
posture the policy set enforces - lives at k3s-demo
(k3s-demo.stephens.page), with the manifests under
policy/.
The ADRs answer why. These sit alongside them to answer how it all fits together:
case-study/— a sanitized end-to-end write-up of the specialty-travel platform these patterns come from: what the system looked like before, which ADRs tied together to modernize it under agents without a rewrite, and what generalizes.threat-model/agent-sandbox.md— the threat model I use to reason about running semi-autonomous AI agents against a revenue-critical legacy system: scope, assets, actors, threats, controls (mapped to specific ADRs), and residual risks. Loosely follows OWASP Threat Modeling.checklist/operational-review.md— the operational-review checklist I actually run before signing off that a system is ready to hold real money in production: blast radius, restore, secrets, auth, observability, deploy, cluster posture, agent-specific, supply chain, and decision documentation.
Each ADR uses a short, consistent shape: Context → Decision → Consequences → When I'd revisit. They're deliberately terse.