Repository navigation
Release 2026.3.2 - #946
Merged
Merged
Release 2026.3.2#946
Conversation
…d add some related tests.
Write out the lint rule set (Ruff's pre-0.16 defaults) so upgrades no longer change which rules apply. Run the pre-commit hooks from the Poetry environment so poetry.lock is the single source of Ruff's version.
Formatting only; no behavior change. Includes Python code blocks in Markdown, which Ruff now formats.
bin/check runs Ruff lint, the Ruff format check and mypy, reporting every failure in one pass. bin/t wraps pytest and requires a target, so the full suite only runs when asked for. CI's lint and type jobs merge into a single bin/check job, which adds the format check CI was missing.
AGENTS.md holds the setup, commands, domain terms and rules a contributor (human or agent) can't derive from the code. CLAUDE.md imports it so Claude Code loads it. Path-scoped guidance for migrations, tests and the worker lives in .claude/rules/. README's install command now matches CI.
AGENTS.md and .claude/rules/ replace .github/instructions/, which had drifted from the code (wrong permissions import, a visibility example that leaks). bin/check and bin/t replace the Makefile targets.
Ruff formats them and bin/check enforces it, so the hook has to cover them too.
AGENTS.md loads in every session, so it keeps only setup, commands, terms and the rules that apply whatever file is open, including one-line forms of the permission and async rules. Path-specific detail moves to new permissions, database and routes rules and into the worker and tests rules. Lines the code already enforces or shows are dropped. One-time contributor setup moves to the README.
The domain terms are dropped rather than moved: the model docstrings already define them, and a copy here had gone stale. The legacy-table note becomes a rule, since the models don't say it.
Calibrations are one case; permission on an entity never implies permission on its related entities.
Deny --no-verify and reads of ~/.aws and ~/.pgpass. Ask before pushes, AWS and Elastic Beanstalk commands, the prod-API dev server and dependency changes. The same file lives in the workspace and in each repo, because Claude Code reads settings only from the folder a session starts in. These are guardrails, not a security boundary: a rewritten command can get past them.
Sessions get AWS_PROFILE=mavedb-agent-readonly, so aws commands use the read-only role (see deployment/environments/aws/agent-access.md) and fail rather than fall back to full credentials where the profile is missing. Choosing another profile asks first, replacing the blanket prompt on aws. Reading ~/.ssh is now blocked, alongside ~/.aws and ~/.pgpass.
Edits to AGENTS.md, CLAUDE.md and anything under .claude/ now ask first. Those files shape every later session, and settings.json holds the guardrails. The rules cover Edit and Write; a shell redirect into one of these files is not caught.
Prompts now cover force pushes, pushes to main, mavedb-main, mavedb-dev or release-*, AWS with an explicit profile, eb, adding a dependency and edits to settings.json itself. Ordinary feature-branch pushes, dependency updates and edits to AGENTS.md, CLAUDE.md and .claude/rules no longer ask; the never rule in the workspace AGENTS.md still covers those edits, and PR review sees the diffs.
The ui checks it with Prettier, so every copy uses that layout.
Takes the PyPI variant-annotation dependency and the 2026.3.1 version bump. poetry.lock conflicted; it is release-2026.3.2's lockfile re-locked with poetry lock, which changes only ruff, 0.6.8 to 0.17.1.
Agent harness: AGENTS.md, bin/check and bin/t, Ruff 0.17.1
Opening or pushing to a PR moves its linked issues to In review, and merging it into a release branch moves them to Pending release. A release PR into the default branch gets a Closes line for every issue its branch picked up, so GitHub closes them when the release merges. Issues only ever move forward. Adds a pull request template whose Closes line feeds the automation.
GitHub turns a Closes line into an issue link only on PRs into the default branch. On a PR into a release branch it stays a plain mention, so the workflow missed issues named that way. The script now reads closing lines from PR descriptions as well as the sidebar links.
…igrations The RT backfill (#747) and the 2026.3.0 runbook still run migrate_mapped_variants_to_allele_substrate, so it moves to src/mavedb/scripts/ and runs as python -m mavedb.scripts.migrate_mapped_variants_to_allele_substrate. Every other script in alembic/manual_migrations/ was a one-off that has already run against production, so the directory and its Ruff exclude go. Under src/ the script is now type-checked. The mypy fixes are casts and annotations only; the columns behind them are NOT NULL.
alembic upgrade head couldn't build a database from empty: 22e2d92d602e called PubMed and bioRxiv through the ORM, 398067c53257 and a7c3e5f9b1d2 committed mid-upgrade through autocommit_block(), and b8f2c5a1d3e4 imported today's view definition from the models. RLS needs upgrades that apply all or nothing, and the new test setup needs a replayable history. Every revision up to 2026.3.0's head is replaced by one migration that keeps that head's ID (c5e1a9d4b7f2), so production and staging need no restamp. It executes alembic/baseline/c5e1a9d4b7f2.sql, which bin/build-baseline wrote from a schema-only dump of production plus the fixed vocabulary an empty database needs (controlled keywords, licenses, roles and the generic MONDO term) and a refresh of the materialized view. Replayed on an empty database, the schema matches production's; 15 CHECK constraints differ only in how Postgres prints their arrays. The baseline can't be downgraded; the rollback across it is the pre-deploy snapshot. alembic_helpers.py had no remaining callers.
A new replay-migrations job runs alembic upgrade head on an empty Postgres 15 for every PR and checks that there is exactly one head. bin/check-migrations, run by bin/check, rejects the patterns that broke the old history: imports from mavedb.*, ORM, network or commit calls, autocommit_block(), and CONCURRENTLY outside a docstring. It skips docstrings because they hold the build-by-hand step for large indexes.
alembic/README.md gives the rules a migration must follow to replay, how to build a large index by hand before a deploy, and how to squash at each cycle boundary, with a log of squashes. The migrations rule and AGENTS.md no longer point at manual_migrations or tell authors to build indexes CONCURRENTLY inside autocommit_block().
…s-automation ci: move linked issues across the backlog board from PR events
Alembic is in the server extra, so the job failed with "Command not found: alembic". Also pass -U postgres to the healthcheck, which otherwise connects as root and logs a FATAL on every probe.
The upgrade runs in one transaction, so SET LOCAL left function-body validation off for every migration replayed after the baseline.
The replay runs every migration in one transaction, and Postgres won't use an enum value in the transaction that added it, so a later migration using the value would fail only on replay. The schema has no native enums; they're text with a CHECK constraint.
Access keys are now stored as the hex SHA-256 of the key, and requests
are authenticated by hashing the presented key and looking it up. Keys
are long random tokens, so an unsalted fast hash is enough. A new
key_hint column keeps the key's last four characters so a user can tell
their keys apart.
The migration hashes existing keys in place and fills their hints, so
issued keys keep working without being reissued. It can't be
downgraded, since the keys can't be recovered from their hashes.
The full key is returned once, when it is created. Listings return the
hint and the key's numeric id instead, and keys are deleted by that id.
BREAKING CHANGE: GET /users/me/access-keys no longer returns keyId; it
returns id and keyHint. DELETE /users/me/access-keys/{access_key_id}
takes the numeric id instead of the key.
eMudhra's intermediates under emSign ECC Root CA - G3 were found to have weak keys, so anyone holding them can issue a certificate for any domain. httpx and requests trust only certifi's bundle and don't check revocation, which left the api trusting forged certificates; the ORCID token exchange, which sends the client secret, is the request most exposed. certifi 2026.10.19 removes G3 and keeps the other emSign roots. 2026.10.18 is skipped because it removed every emSign root, which breaks UCSC (chains to emSign Root CA - G1).
Postgres won't use an index-only scan on an expression index unless the expression's columns are in the index too. Without urn, selecting a deep page's variant ids still read every skipped row from the table, so the planner preferred a full scan and sort of the score set. The migration builds ix_variants_scoreset_number_covering with INCLUDE (urn) and drops the old index. On a populated database, build the new one and drop the old one CONCURRENTLY by hand first (commands in the docstring), which makes the migration a no-op.
…ixes fsspec < 2026.6.0 allows server-side template injection in ReferenceFileSystem, and Mako <= 1.4.1 allows path traversal through drive-letter URIs in TemplateLookup on Windows. Both are transitive and unpinned (pyathena and alembic), so only the lock changes. The other open alerts are left alone on purpose. setuptools stays on ~81.0 because 82 and later drop pkg_resources, which biocommons, eutils and isbnlib import at runtime; its flaw only affects building sdists on macOS. python-jose and ecdsa have no patched release. Our only jwt.decode pins algorithms=["RS256"], which the python-jose bypass needs to be absent, and nothing signs with ecdsa.
A paged CSV request applied OFFSET to the full variant query, so every row before start was read, joined to its mapping record, allele and projection group, and then discarded. Deep pages of a 648k-variant score set took minutes. Select the page's variant ids first, from the covering variant number index, and join the full query to just those ids. The outer query has no score set filter of its own: with one, the planner could serve its ORDER BY by walking the whole score set in index order and reading every row. Output is unchanged. start and limit are now checked against None, so paging an explicit variant id list raises even with a 0 bound (its order was never defined in SQL, and no caller pages one), and the build limiter counts limit=0 as no rows. The route already rejects limit <= 0.
…-drop-emsign-g3 chore(deps): security updates for certifi, fsspec and Mako
…wnload-ids perf(export): page score set downloads by index-only id selection
…eys-as-hashes feat: store API keys as hashes
…urnError Add except pd.errors.ParserError for uploading malformed CSV files and add some related tests
Coverage Report for CI Build 39734580226Coverage decreased (-0.07%) to 90.34%Details
Uncovered Changes
Coverage Regressions13 previously-covered lines in 4 files lost coverage.
Coverage Stats
💛 - Coveralls |
The session fixture ran create_all and drop_all on a fresh database for every test: about a second of setup and teardown, and some 4,600 files created and deleted per test. pytest-postgresql now loads the schema into its template database once per server, and each test's database is cloned from it. Dropping the database replaces drop_all.
data_files copied the whole data directory into tmp_path for each of its ~1,000 uses, though no test writes to it. arq_worker made a ProcessPoolExecutor for every job and standalone_worker_context one for every test, and neither was ever shut down.
Verbose mode prints a line for each of ~5,400 tests, which floods agent context and CI logs. Failures are still reported in full; pass -v to get the per-test lines back.
Adds pytest-xdist ~3.8 (3.8.0) and its dependency execnet 2.1.2, to run tests in parallel. pytest-postgresql already supports xdist: each worker starts its own server, or uses its own database names on a shared one.
Under xdist, subtest labels travel between processes through execnet, which can't serialize a pandas Series, so these subtests errored. The label is now the column's name and values as a string. It can go once these tests move to pytest's parametrize.
bin/test-db starts Postgres 15 in Docker with its data on tmpfs, and bin/t points tests at it through MAVEDB_TEST_DB_PORT while it runs. Per-test databases then never touch the host filesystem, where on macOS fseventsd and endpoint scanning react to every file. One container serves every checkout, and a per-run id keeps concurrent runs apart. The container runs in the host's time zone, as pytest-postgresql's own server does, because pipeline manager tests depend on it. Without the container, tests run as before.
The session engine used NullPool, so it reconnected after every commit: about 10 connections per test. Under xdist against the Docker database that rate outran Docker's port forwarding, and tests failed with "connection refused". NullPool kept connections from blocking drop_all; pytest-postgresql now drops the database and terminates connections first, so a pooled engine disposed at teardown is safe. Cuts it to about 4 connections per test.
-n auto uses each runner's cores; every worker starts its own Postgres.
These two subtests also pass a pandas Series as their label, and only run when hgvs isn't installed, so the core-dependencies jobs hit the same xdist serialization error.
-n auto counts physical cores, so GitHub's 4-vCPU runners got only two xdist workers. -n logical uses all four. --basetemp=/dev/shm/pytest puts each worker's pytest-postgresql data directory on tmpfs, the CI equivalent of bin/test-db, so per-test database clones never touch disk.
started_at and finished_at on job_runs and pipelines are timestamptz, but the managers set them from a naive datetime.now(). Postgres reads a naive value in the session's time zone, so any worker whose process time zone differed from the database's recorded times off by the difference. A finished_at set in the same session was also naive, so the pipeline duration calculation would raise TypeError subtracting the aware created_at from it. The two unit tests that bracketed these times with naive datetimes now use aware ones, as the integration tests already did.
- bin/test-db: run the container in its default UTC, like CI and RDS, now that the worker no longer depends on the server's time zone. Give up after 30s with the container's logs instead of waiting forever for a server that failed to start. Note that a restart clears databases left behind by interrupted runs. - Connect to 127.0.0.1, where the container is published, so libpq doesn't try ::1 first on each connection.
test: clone test databases from a template and run tests in parallel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
GET /users/me/access-keysreturnsidandkeyHint(last four characters) and no longer returnskeyId;DELETE /users/me/access-keys/{access_key_id}takes the numericid. The full key is returned only once, when it's created. Ships with feat: show a new API key once, at creation mavedb-ui#751.Bug Fixes
start/limit) select the page's variant ids from an index-only scan first, instead of reading and joining every row beforestart. On a 648k-variant score set, a late page dropped from 16–31 s to 1.9 s (scores only) and from 104–141 s to 17.9 s (with mapping namespaces).Maintenance
bin/test-dbruns the test Postgres on tmpfs in Docker, and tests run in parallel with pytest-xdist (new dev dependency). The PR test set takes about 2 minutes locally instead of about 20, and CI's slowest job 12–14 minutes instead of 25–33. Also records job and pipelinestarted_at/finished_atin UTC; they were naive local times.c5e1a9d4b7f2, the same ID as 2026.3.0's head), built from production's schema.alembic upgrade headnow builds a database from empty, and a CI job andbin/check-migrationskeep it that way.manual_migrations/is gone; the allele substrate backfill now runs aspython -m mavedb.scripts.migrate_mapped_variants_to_allele_substrate.emSign ECC Root CA - G3), fsspec 2026.9.0 and Mako 1.4.3.AGENTS.mdand.claude/rules/,bin/checkandbin/t, CI through them, and Ruff 0.17.1 with one formatting commit.Breaking API changes
AccessKeyno longer haskeyIdin list responses; it hasidandkeyHint.keyIdappears only in the create response.DELETE /users/me/access-keys/{access_key_id}takes the key's numericidinstead of its key ID.Deployment
Three revisions above
main's head:c5e1a9d4b7f2(baseline) →d7a3e9c1f5b2→6987f253e02a. Production and staging are already atc5e1a9d4b7f2, so they need no restamp.Take an RDS snapshot first.
6987f253e02ahashesaccess_keys.key_idin place and its downgrade raises, and there's no Alembic downgrade across 2026.3.0 any more; the snapshot is the rollback.Before deploying, build the covering index by hand so
d7a3e9c1f5b2is a no-op:Deploy with Release 2026.3.1 mavedb-ui#752. The settings page in the current ui breaks against the new access key schema.
Merge with a merge commit, not a squash, so the formatting commit in
.git-blame-ignore-revskeeps its hash. Expect apyproject.toml/poetry.lockconflict on Ruff (mainpins~0.15); resolve it to~0.17.1.After merge: rebase
release-2026.4.0ontomain, andpoetry publish --buildfrommain.