Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
362f13c
Add except pd.errors.ParserError for uploading malformed CSV files an…
EstelleDa Oct 15, 2026
1691b1c
build(ruff): pin Ruff 0.17.1 with an explicit rule set
bencap Oct 15, 2026
dcf4e38
style: apply Ruff 0.17.1 formatting
bencap Oct 15, 2026
de7c253
chore: skip the Ruff formatting commit in git blame
bencap Oct 15, 2026
747c0fb
build: add bin/check and bin/t entry points and run CI through them
bencap Oct 15, 2026
b8e32a9
docs: add AGENTS.md and path-scoped agent rules
bencap Oct 15, 2026
4b03a86
chore: remove Copilot instruction files and the Makefile
bencap Oct 15, 2026
9e7c6c1
build(pre-commit): format Python code blocks in Markdown
bencap Oct 15, 2026
1517be0
docs: trim AGENTS.md to what applies everywhere
bencap Oct 15, 2026
c7b0792
docs: move cross-repo rules to the workspace AGENTS.md
bencap Oct 16, 2026
50cf7df
docs: generalize the stricter-than-parent permissions rule
bencap Oct 16, 2026
23a54d2
build(claude): add shared permission rules
bencap Oct 16, 2026
7aa0dae
build(claude): run AWS as the read-only agent role and block ~/.ssh
bencap Oct 16, 2026
e0444da
build(claude): ask before edits to instruction and settings files
bencap Oct 16, 2026
3924b1c
build(claude): ask only before hard-to-undo actions
bencap Oct 16, 2026
694aef0
style(claude): format settings.json with Prettier
bencap Oct 16, 2026
fd1d5d0
Merge release-2026.3.2 into feature/bencap/agent-harness
bencap Oct 19, 2026
e52f665
Merge pull request #926 from VariantEffect/feature/bencap/agent-harness
bencap Oct 19, 2026
77ee328
ci: move linked issues across the backlog board from PR events
bencap Oct 19, 2026
b0eceb2
ci: read Closes lines from PR descriptions
bencap Oct 19, 2026
b43c559
refactor(scripts): move the allele substrate backfill out of manual_m…
bencap Oct 20, 2026
5e44491
refactor(db): squash the migration history into a 2026.3.0 baseline
bencap Oct 20, 2026
7b1d6c5
ci: keep the migration history replayable from an empty database
bencap Oct 20, 2026
c45c3a4
docs(db): document the migration rules and the cycle-boundary squash
bencap Oct 20, 2026
e9f20e8
Merge pull request #936 from VariantEffect/feature/bencap/issue-statu…
bencap Oct 20, 2026
85cfa13
ci: install the server extra in the migration replay job
bencap Oct 20, 2026
ef6bd18
fix(db): restore check_function_bodies after the baseline
bencap Oct 20, 2026
8617473
ci: reject enum ADD VALUE in migrations
bencap Oct 20, 2026
96137ed
feat: store API keys as hashes
bencap Oct 21, 2026
034385b
chore(deps): bump certifi to 2026.10.19 to drop emSign ECC Root CA - G3
bencap Oct 21, 2026
8412abe
Merge pull request #938 from VariantEffect/feature/bencap/877/migrati…
bencap Oct 21, 2026
086390c
fix(migrations): chain the access key hash migration after the varian…
bencap Oct 21, 2026
ded2b7b
perf(db): cover the variant number index with urn
bencap Oct 21, 2026
c813bd8
chore(deps): bump fsspec to 2026.9.0 and Mako to 1.4.3 for security f…
bencap Oct 21, 2026
14dfa83
perf(export): page score set downloads by index-only id selection
bencap Oct 21, 2026
f91d8e3
Merge pull request #942 from VariantEffect/maintenance/bencap/certifi…
bencap Oct 21, 2026
d1d7da7
Merge pull request #941 from VariantEffect/bugfix/bencap/901/paged-do…
bencap Oct 21, 2026
ced505d
Merge pull request #943 from VariantEffect/feature/bencap/store-api-k…
bencap Oct 21, 2026
cd5fab9
chore: bump version to 2026.3.2
bencap Oct 21, 2026
0b58f9b
Merge pull request #925 from VariantEffect/feature/estelle/723/csvRet…
EstelleDa Oct 21, 2026
5a0d10d
test: build the test schema once and clone it for each test
bencap Oct 22, 2026
a8fb09e
test: read data files in place and shut down worker process pools
bencap Oct 22, 2026
60b3734
test: drop -v from pytest's default options
bencap Oct 22, 2026
cf72381
chore(deps): add pytest-xdist to the dev group
bencap Oct 22, 2026
f31fd16
test: label dataframe subtests with strings
bencap Oct 22, 2026
05ce5cc
test: run tests against a tmpfs Postgres when bin/test-db is up
bencap Oct 22, 2026
82a621b
test: reuse one database connection per test
bencap Oct 22, 2026
8efb8d7
ci: run tests in parallel with pytest-xdist
bencap Oct 22, 2026
e58dbcd
docs: tell agents how to use bin/test-db and parallel test runs
bencap Oct 22, 2026
4dddd50
test: label the genomic-variant subtests that run without hgvs
bencap Oct 22, 2026
67365bb
ci: use every logical core and keep test temp files in memory
bencap Oct 22, 2026
133564b
fix(worker): stamp job and pipeline start and finish times in UTC
bencap Oct 22, 2026
dfc62bd
test: harden the shared test database
bencap Oct 22, 2026
39c9fe3
Merge pull request #947 from VariantEffect/maintenance/bencap/test-perf
bencap Oct 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .claude/rules/database.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
paths:
- "src/mavedb/models/**"
- "src/mavedb/lib/**"
- "src/mavedb/routers/**"
- "src/mavedb/worker/**"
- "src/mavedb/scripts/**"
---

# Database

- Write new queries in SQLAlchemy 2.0 `select()` style. Older code still uses `db.query()`; don't copy it.
- ValidTime tables never return only live rows by default. Filter with `Model.current` or `Model.live_at(...)`.
- Replace ValidTime rows with `supersede_with` or `supersede_live_where`, never a separate retire-then-insert, so the handoff has no gap.
14 changes: 14 additions & 0 deletions .claude/rules/migrations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
paths:
- "alembic/**"
---

# Migrations

- Generate with `poetry run alembic revision --autogenerate -m "<what changes>"`, then read the result: autogenerate misses or mangles enum changes, server defaults, views and functions.
- Set `down_revision` to the current head so the chain stays linear. Check with `poetry run alembic heads` (one head).
- Never edit a revision that has run on staging or production. Write a new one, even for a fix. The one exception is the squash at a cycle boundary (`alembic/README.md`).
- One logical change per revision. The message says what changes ("add variant_count to scoresets"), not "update model".
- Every migration must replay from an empty database inside one transaction: no imports from `mavedb.*`, no network calls, ORM sessions or commits, no `CONCURRENTLY` or `autocommit_block()`, and no `ALTER TYPE ... ADD VALUE` (enums are text with a `CHECK` constraint). `bin/check-migrations` enforces this; the rules are in `alembic/README.md`.
- Production tables are large (`variants`, `mapping_records`, `mapping_record_alleles`, `alleles`, `annotation_event` run to millions of rows). A full-table `UPDATE` holds its lock for minutes. For an index on one of them, put the `CREATE INDEX CONCURRENTLY` to run by hand before the deploy in the docstring, and create the same index with `IF NOT EXISTS` in the migration.
- Data work that needs application code or external services is an operator script in `src/mavedb/scripts/`, not a migration.
12 changes: 12 additions & 0 deletions .claude/rules/permissions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
paths:
- "src/mavedb/routers/**"
- "src/mavedb/lib/**"
---

# Permissions

- Import from `mavedb.lib.permissions`. Check the entity a route names with `assert_permission(user, entity, Action.X)`.
- When a read fans out from that entity to related ones (score set → calibrations, allele → score sets), get a `Principal` from the `get_principal` dependency and filter with its `Viewer` for that entity type. Some entities are stricter than their parent: for instance, reading a score set doesn't grant reading its calibrations. Never assume that permission on one entity implies permission on its related entities.
- `Principal` and `Viewer` are request-scoped. Never use one as a default argument.
- Read enforcement is moving into Postgres row-level security (#808). Until it lands, these application checks are the only enforcement.
15 changes: 15 additions & 0 deletions .claude/rules/routes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
paths:
- "src/mavedb/routers/**"
- "src/mavedb/lib/authentication.py"
- "src/mavedb/lib/authorization.py"
---

# Routes

FastAPI runs `def` handlers and dependencies in a thread pool and `async def` ones on the event loop, so a synchronous database call inside `async def` stalls every request on that process (#881).

- New routes, dependencies and helpers are `def`.
- Use `async def` only for real awaits (ARQ enqueue, async HTTP), and run the synchronous database work in the thread pool.
- Start worker pipelines only through `enqueue_pipeline_for_score_set` (`lib/workflow/kickoff.py`).
- Route tests go in `tests/routers/`.
14 changes: 14 additions & 0 deletions .claude/rules/tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
paths:
- "tests/**"
---

# Tests

- Mock only at system boundaries: external services, Redis/ARQ, S3, Slack. Don't mock internal helpers, validators or transforms.
- Assert on outcomes: return values, responses, database state, and calls at system boundaries. Don't assert on internal calls, call counts or log and progress message strings; tests should survive a refactor.
- Unit tests cover edge cases and error paths with external services mocked. Integration tests cover happy paths end to end against the real test database.
- `session.refresh()` before asserting on a modified ORM object. Include negative assertions where they matter (the record that must not exist).
- Name tests `test_<function>_<condition>_<expected_outcome>` when you write or modify them; don't rename untouched tests.
- Use `@pytest.mark.parametrize` with `ids` instead of near-identical copies.
- Put a fixture in the most specific `conftest.py` that needs it, and lift it to the nearest common one when it's shared.
23 changes: 23 additions & 0 deletions .claude/rules/worker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
paths:
- "src/mavedb/worker/**"
- "tests/worker/**"
---

# Worker

Read `src/mavedb/worker/README.md` first; `best_practices.md`, `job_decorators.md`, `job_managers.md` and `pipeline_management.md` beside it cover job structure, decorators and pipelines.

## Rules

- Job signature: `async def job(ctx, job_id, job_manager) -> JobExecutionOutcome`. Register jobs in `worker/jobs/registry.py`.
- Job code never commits. The management decorators and `update_progress()` own commits.
- Start pipelines only through `enqueue_pipeline_for_score_set` (`lib/workflow/kickoff.py`).
- `max_jobs = 2` is deliberate: psycopg2 is synchronous and blocks the worker's event loop.

## Testing jobs

- Assert on `JobExecutionOutcome.status` and `.data` in every job test, plus database state for the objects the job changes. For external-service jobs, assert the boundary calls (ClinGen CAR/LDH, UniProt, gnomAD/Athena, ClinVar, S3).
- Let `update_progress()` run unpatched. It commits as a checkpoint in production, and tests should exercise that.
- Use `TransactionSpy` (`tests/helpers/transaction_spy.py`) only in manager and decorator tests, where commit and rollback are the contract. In job tests, check database state instead.
- `mock_database_flush_failure` and `mock_database_rollback_failure` are for infrastructure error-recovery paths that are hard to trigger otherwise.
32 changes: 32 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"env": {
"AWS_PROFILE": "mavedb-agent-readonly"
},
"permissions": {
"deny": ["Bash(* --no-verify*)", "Read(~/.aws/**)", "Read(~/.pgpass)", "Read(~/.ssh/**)"],
"ask": [
"Edit(**/.claude/settings*.json)",
"Bash(git push *--force*)",
"Bash(git push * -f*)",
"Bash(git push * main*)",
"Bash(git push * mavedb-main*)",
"Bash(git push * mavedb-dev*)",
"Bash(git push * release-*)",
"Bash(git -C * push *--force*)",
"Bash(git -C * push * -f*)",
"Bash(git -C * push * main*)",
"Bash(git -C * push * mavedb-main*)",
"Bash(git -C * push * mavedb-dev*)",
"Bash(git -C * push * release-*)",
"Bash(aws *--profile*)",
"Bash(*AWS_PROFILE=*)",
"Bash(eb *)",
"Bash(*MODE=prodapi*)",
"Bash(poetry add *)",
"Bash(poetry -C * add *)",
"Bash(npm install *)",
"Bash(npm --prefix * install *)",
"Bash(*pip install *)"
]
}
}
5 changes: 5 additions & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Formatting-only commits, skipped by `git blame` (GitHub reads this file automatically).
# Locally: git config blame.ignoreRevsFile .git-blame-ignore-revs

# style: apply Ruff 0.17.1 formatting
dcf4e382d2e0bea48bd6c094141727275cbf262f
165 changes: 0 additions & 165 deletions .github/instructions/api.instructions.md

This file was deleted.

Loading
Loading