Close audit issues #20–#27: agent engine, worker & email, profiles, external clients, web, data layer, docs, deploy

.dockerignore

@@ -3,6 +3,7 @@
 certbot
 logs
 data
+profiles
 *.log
 __pycache__
 **/__pycache__
@@ -15,3 +16,41 @@ venv
 # Secrets — never in an image layer (compose injects these at runtime via env_file)
 .env
 .env.*
+
+# Bloat + secrets that COPY . . would otherwise bake into every image layer.
+# backups/ is the sharp one: it holds full production pg_dumps AND a
+# plaintext prod .env (SLACK_BOT_TOKEN_* per agent, POSTGRES_PASSWORD,
+# SLACK_CONFIG_TOKEN/_REFRESH_TOKEN) that the .env/.env.* patterns above do
+# not catch — it has no leading dot and sits two directories down.
+backups
+.venv-test
+.venv*
+tests
+.notes
+mutants
+build
+.hypothesis
+.ruff_cache
+.coverage
+.mutmut-cache
+# Created by ci.sh's mypy stage — large relative to the built image's /app
+# tree, and not something any runtime process reads.
+.mypy_cache
+# A stray uv.lock (a `uv` command run without --no-project writes one) is a second,
+# competing lockfile; requirements.lock is the only one the image installs.
+uv.lock
+.playwright-mcp
+copi.egg-info
+.superpowers
+**/.env*
+docs/specs
+# The one-off implementation plans and their evidence subtrees — a large
+# fraction of /app's tree for text no process reads. Nothing under src/,
+# scripts/, templates/ or alembic/ opens a path below docs/; every reference
+# is a comment or a docstring (`grep -rn 'docs/' src/ scripts/ templates/
+# alembic/`), which is the same reasoning that already excludes docs/specs
+# above. docs/ itself is deliberately NOT excluded: docs/production-migration.md
+# and docs/inbound-email.md are operator runbooks worth having next to the
+# code in a shell inside the container.
+docs/plans
+docs/superpowers

.gitignore

@@ -37,6 +37,7 @@ profiles/**/*.md
 # Sweep backups of agent memory (same prose as the .md files — never commit;
 # this repo is public and a blanket `git add -A` must not pick them up).
 profiles/**/*.pre-sweep
+profiles/**/*.tmp
 profiles/*.log
 
 # Generated/scratch data and static asset bundles (not source)
@@ -63,6 +64,17 @@ certbot/
 .coverage
 htmlcov/
 
+# mypy's incremental cache (scripts/ci.sh's mypy stage, #27 I5) — 58 MB of
+# per-module analysis state, no reason to ever commit it.
+.mypy_cache/
+
+# Stray uv-native lockfile. requirements.lock (pip-compile, hash-pinned) is
+# the one actually installed by the Dockerfile and checked by ci.sh's
+# freshness gate; `uv lock`/`uv sync` commands run against this repo would
+# otherwise leave a second, competing lockfile that is a public-repo secret
+# risk if it ever picks up an internal index URL or credentials (#27 I5).
+uv.lock
+
 # Generated by scripts/export_agent_roster.py (roster snapshot for host provisioning)
 data/agent_roster.json
 

AGENT.md

@@ -4,24 +4,31 @@
 
 Python implementation of the CoPI researcher collaboration platform combined with the LabAgent multi-agent Slack system. ORCID OAuth, profile generation pipeline, profile editing UI, admin dashboard, and Slack-based AI agent simulation.
 
-**GitHub:** https://github.com/andrewsu/coPI-python-opus
+**GitHub:** https://github.com/SuLab/coPI.science
 **Target domain:** copi.science
-**Pilot:** 10 labs at Scripps Research
+**Pilot:** Scripps Research labs — the live roster and count are at **/admin/agents** (see CLAUDE.md "Adding New PIs"); the number changes as PIs are provisioned.
 
 ## What's In Scope
 
 - ORCID OAuth authentication
 - Profile ingestion pipeline (ORCID → PubMed → PMC → Claude Opus synthesis)
 - Profile review/editing web UI (FastAPI + Jinja2)
 - Admin dashboard (users, profiles, jobs, agent activity)
-- Slack agent system (8 bots, simulation engine)
+- Slack agent system (bot count tracked live at /admin/agents; simulation engine)
 
 ## What's Out of Scope
 
 - Matching engine (pairwise proposal generation)
 - Swipe interface
-- Notifications (email)
-- Daily digest
+
+## What Email Actually Does (in scope, built)
+
+`src/services/email.py`, `email_inbound.py`, `email_notifications.py` —
+proposal-review emails, reply intake, unsubscribe/settings, and a periodic
+status-overview digest (`check_and_send_status_overviews`, called from the worker's
+~300 s poll loop in `src/worker/main.py`; each PI's own cadence comes from
+`email_notification_preferences.frequency`, which defaults to weekly —
+`daily` is one option on that ladder, not the schedule).
 
 ## Key Specs
 
@@ -73,8 +80,8 @@ templates/                  # Jinja2 HTML templates
 Decisions made autonomously during implementation are recorded here for human review.
 
 ### 2026-03-20: Admin impersonation endpoint location
-**Decision:** Admin impersonation routes placed at `/api/admin/impersonate` (POST) and `/api/admin/impersonate/stop` (POST) rather than inside the `/admin` router prefix.
-**Reason:** The impersonate stop button posts from any page (including non-admin pages when impersonating), so a clean `/api/admin/` prefix was clearer. Both routes still require is_admin verification.
+**Decision:** Admin impersonation routes are `POST /admin/impersonate` and `POST /admin/impersonate/stop` (`src/routers/admin.py`, mounted under the `/admin` prefix in `src/main.py`). *(Corrected 2026-09: this entry originally described an `/api/`-prefixed design that was never what shipped — see issue #26 DOC-3.)*
+**Reason:** The impersonate-stop button posts from any page (including non-admin pages when impersonating). Both routes still require is_admin verification.
 
 ### 2026-03-20: Login page GET /login serves both redirect and HTML
 **Decision:** `/login` GET route redirects directly to ORCID OAuth if not already logged in. The login.html page has its sign-in button also pointing to `/login` (which re-triggers the redirect).
@@ -102,7 +109,7 @@ Decisions made autonomously during implementation are recorded here for human re
 **Reason:** Avoids Node.js build step in a Python project. Acceptable for pilot; switch to compiled Tailwind for production if performance matters.
 
 ### 2026-03-20: Profile markdown export
-**Decision:** When a ResearcherProfile is saved/updated in the DB, automatically export it to `profiles/public/{lab}.md` if the user is one of the 8 pilot labs (matched by ORCID).
+**Decision:** When a ResearcherProfile is saved/updated in the DB, automatically export it to `profiles/public/{lab}.md` (matched by ORCID). Written when there were 8 pilot labs; the roster is now whatever `AgentRegistry` holds — see `/admin/agents` for the live count, and `orcids.txt` (48 entries) for the seeding list. Do not treat any number in this file as the current roster size.
 **Reason:** Keeps the DB (source of truth) and filesystem (agent input) in sync without a separate sync step.
 
 ## Implementation Status
@@ -115,11 +122,15 @@ Decisions made autonomously during implementation are recorded here for human re
 - [x] Admin dashboard
 - [x] Worker process
 - [x] Agent system (Slack bots, simulation engine)
-- [x] Agent profiles (8 pilot labs, auto-generated structure)
+- [x] Agent profiles (auto-generated structure; current roster at /admin/agents)
 - [x] Prompt files
 
 ## Pilot Lab ORCIDs
 
+**Historical snapshot — the original pilot cohort, not the current roster.** The live roster is
+`AgentRegistry` (see `/admin/agents`); the seeding list is `orcids.txt`, which now holds 48 ORCIDs.
+This table is kept because the decisions above refer to it.
+
 | PI | ORCID |
 |---|---|
 | Andrew Su | 0000-0002-9859-4104 |

CLAUDE.md

@@ -17,6 +17,12 @@ docker compose exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postg
   app python -m pytest tests/ -v
 ```
 
+**Note (2026-09, #27 I3):** the command above works against the **dev** compose file, whose
+`.:/app` bind mount supplies `tests/`. `tests/` is excluded from the built image by
+`.dockerignore`, so the same command against a prod-built container reports
+"file or directory not found: tests/". Run the suite on the host (`./scripts/ci.sh`) or with the
+dev compose file.
+
 The named database must already exist — the suite migrates it, it does not create
 it. Add a fresh scratch DB with
 `docker compose exec -T postgres createdb -U copi copi_xN`, and give concurrent
@@ -72,9 +78,11 @@ docker compose $C --profile agent run -d --name agent-run agent python -m src.ag
 
 On resume the sim fetches Slack history for each bot in roster order before reaching
 turn 1. Slack throttles this hard — expect ~10 minutes of
-`[<first-agent>] Rate limited, retrying in 10s (attempt 1/3)` before the first
-`=== Turn 1 ===`. Repeated `attempt 1/3` (never `2/3`) means each call 429s once then
-succeeds on retry — that is forward progress, not a hang.
+`[<first-agent>] Rate limited, retrying in 10s (attempt 1/8, 10.0s/180.0s of wait
+budget used)` before the first `=== Turn 1: <agent> ===`. Repeated `attempt 1/8`
+(never `2/8`) means each call 429s once then succeeds on retry — that is forward
+progress, not a hang (RC-3 on this branch raised the ceiling from 3 attempts to 8 and
+added the 180s cumulative wait-budget figure logged alongside it).
 
 **Before restarting**, always save logs and rebuild containers:
 
@@ -88,12 +96,36 @@ ls -t logs/run_*.log | tail -n +11 | xargs rm -f
 # 2. Stop the old container — GRACEFULLY. `docker rm -f` sends SIGKILL, which
 #    skips the shutdown flush and permanently loses the in-flight turn's
 #    messages (the DB, not Slack, is the durable store). `docker stop` sends
-#    SIGTERM; -t 30 leaves room for an in-flight LLM call to finish.
+#    SIGTERM; -t 30 leaves room for an in-flight LLM call to finish. One
+#    SIGTERM stops after the current turn and aborts Slack retry sleeps 20 s
+#    later; a SECOND signal aborts Slack immediately (the DB flush still
+#    runs); a THIRD terminates the process at once and can lose the flush --
+#    never send a third unless the process is wedged.
 docker stop -t 30 agent-run
 docker rm agent-run
 
-# 3. Rebuild app + worker (picks up code changes)
-docker compose $C up -d --build app worker
+# 3. Redeploy app + worker + grantbot against the migrated schema — via
+#    scripts/redeploy.sh, NOT a bare `up -d --build`. `depends_on: migrate: condition:
+#    service_completed_successfully` only orders container CREATION: on an
+#    already-running stack, an existing exited `migrate` container from the last
+#    deploy can satisfy that condition without being re-run against the freshly
+#    built image, so old code can keep serving requests against a schema the new
+#    migration hasn't applied yet (audit 2026-09-08 RC-6, #27 I2; audit 2026-09-10
+#    R-4 added grantbot, which has the identical depends_on shape and was
+#    otherwise left running the old image). redeploy.sh builds
+#    migrate+app+worker+grantbot, STOPS app/worker/grantbot first, runs migrate
+#    and checks its exit code, only then starts the new app/worker/grantbot, then
+#    reloads nginx (the recreated app container gets a new IP — see the
+#    nginx-stale-upstream-ip memory note). `agent` is NOT part of this — it is a
+#    one-off with its own restart runbook above. It refuses to run unless both prod compose files are visible
+#    (via $COMPOSE_FILE or -f) and never passes an orphan-removal flag. The image
+#    runs as UID 10001, so profiles/ and data/ on the host must already be owned
+#    by 10001:10001 (never prompts/ — see docs/production-migration.md §10.8 and
+#    Part R.5 of docs/plans/2026-09-02-close-issues-20-27.md) or the services
+#    that mount them fail to write into their bind mounts — profiles/ and
+#    prompts/ are mounted on app/worker (and agent/grantbot); data/ is mounted
+#    only on agent/grantbot, not app/worker.
+./scripts/redeploy.sh $C
 
 # 4. Rebuild the agent image too — prod bakes code into the image, so skipping
 #    this silently runs whatever source was current at the last build.
@@ -115,6 +147,18 @@ they can decide whether to restart.** Roster changes — activating/inactivating
 setting a new `slack_bot_token` in `AgentRegistry` — do NOT need a restart; they're
 picked up live by `_sync_roster_from_db`.
 
+**One-time Slack-ts repair (legacy rows).** A workspace that predates the DB-primary
+conversation model may have `agent_messages` rows with `slack_ts IS NULL`. Replies to
+threads rooted on those rows are silently kept off Slack — `_slack_parent_ts`
+(`src/agent/simulation.py`) returns `None` for a legacy root and callers skip the
+mirror rather than guess a timestamp Slack never issued. Run
+`docker compose exec -e PYTHONPATH=/app app python scripts/backfill_slack_ts.py --apply` once, before your
+next restart, to ask Slack which timestamps actually exist and repair them (safe to
+re-run; read-only against Slack otherwise). `docs/production-migration.md` §8 Step 8
+walks through this as an ordered step after the migration and before the app-code
+deploy for a *fresh* migration; if your workspace is already at head and has never run
+it, run it manually — nothing else will prompt you to.
+
 ## Adding New PIs
 
 **The `AgentRegistry` table is the single source of truth for the agent roster.**
@@ -139,7 +183,7 @@ Each agent needs an `AgentRegistry` row with a unique `agent_id` (lowercase last
 and `bot_name` (`{LastName}Bot`), created `status='pending'`. Self-service signups
 (`src/routers/agent_page.py`) and the backfill scripts both create these automatically.
 
-**Last-name collisions:** If a last name is already taken (e.g., Chunlei Wu = `wu`), prefix with the first initial (e.g., Peng Wu = `pwu` / `PWuBot`). The web UI applies this logic automatically.
+**Last-name collisions:** If a last name is already taken (e.g., Chunlei Wu = `wu`), prefix with the first initial (e.g., Peng Wu = `pwu` / `PWuBot`). If that prefixed id is *also* taken (a third same-initial namesake), append a numeric suffix to the prefixed candidate (e.g. `pwu2` / `PWu2Bot`). The web UI applies this logic automatically.
 
 ### 3. Provision the Slack bot + activate (admin UI)
 

Dockerfile

@@ -1,23 +1,68 @@
-FROM python:3.11-slim
+FROM python:3.11-slim AS builder
 
 WORKDIR /app
 
-# Install build dependencies
+# Build-time only: gcc/libpq-dev compile any dependency that ships as an sdist
+# for this platform/Python combination. Not present in the runtime image
+# below — asyncpg itself needs none of this, it bundles its own wire protocol
+# implementation rather than linking libpq.
 RUN apt-get update && apt-get install -y --no-install-recommends \
     gcc \
     libpq-dev \
     && rm -rf /var/lib/apt/lists/*
 
-# Install Python dependencies
-COPY pyproject.toml .
+COPY pyproject.toml requirements.lock ./
+RUN pip install --no-cache-dir --require-hashes -r requirements.lock
 COPY src/ src/
-RUN pip install --no-cache-dir .
+# --no-build-isolation: build isolation would otherwise fetch a fresh,
+# unhashed setuptools/wheel from PyPI at build time just to satisfy
+# pyproject.toml's [build-system] requires; the base image's preinstalled
+# setuptools/wheel already satisfy it. This local `pip install .` therefore
+# is not hash-verified the way the `-r requirements.lock` install above is
+# — accepted, since it installs only this repo's own source, not a
+# third-party artifact off the network.
+RUN pip install --no-cache-dir --no-deps --no-build-isolation .
 
-# Copy source
+FROM python:3.11-slim AS runtime
+
+WORKDIR /app
+
+# libpq5 only: the runtime client library a compiled wheel may dlopen. Nothing
+# currently links it — asyncpg is pure-protocol — this is insurance for a
+# future psycopg dependency. No compiler, no -dev headers, no build toolchain
+# of any kind in this stage. Deliberately avoids naming the builder-stage
+# packages here — the structural test in tests/unit/test_dockerfile_build.py
+# asserts their names are absent from this section.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libpq5 \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
+COPY --from=builder /usr/local/bin /usr/local/bin
 COPY . .
 
-# Create directories for profiles and prompts
-RUN mkdir -p profiles/public profiles/private prompts logs static
+# Bake the bytecode cache while root still owns src/ — UID 10001 (set below)
+# cannot write __pycache__ into root-owned src/, so without this every
+# process start pays a first-import compile cost (~0.9s, measured). Must run
+# AFTER src/ lands (COPY . . above) and BEFORE USER drops root.
+RUN python -m compileall -q src
+
+# Fixed UID so it matches whatever the prod host chowns the bind-mounted
+# profiles/data trees to — a plain chown target on the host, not a real host
+# account. Ownership is scoped to the directories the runtime user actually
+# writes to (profiles/data/logs); src/, templates/, alembic/, scripts/ and
+# static/ stay root-owned and read-only to this user, so a compromised
+# process cannot rewrite its own code. static/ is deliberately excluded:
+# StaticFiles only ever reads it, nothing under src/ writes to it, so a write
+# grant there would be a needless stored-XSS surface on assets served
+# straight to the browser.
+RUN groupadd --gid 10001 copi \
+    && useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin copi \
+    && mkdir -p profiles/public profiles/private profiles/memory data logs \
+    && chown -R 10001:10001 profiles data logs
+ENV HOME=/app
+
+USER 10001
 
 EXPOSE 8000