epic: Allele centric mapping, storage, and service#791
Draft
bencap wants to merge 68 commits into
Draft
Conversation
- add shared seqrepo and seqrepo data proxy providers in services - reuse shared seqrepo provider in deps to remove duplicate config - align vrs sequence proxy behavior with dcd-mapping expectations
- add shared helpers to translate hgvs, normalize alleles, and compute ids - clear cached merkle digests before identification so mutated alleles do not keep stale ga4gh identifiers - document the invariant that all allele identification must route through the helper to prevent digest correctness regressions
Introduce a reusable declarative mixin implementing transaction-time SCD Type 2 versioning for rows that change over time, used by either versioned entities or link/association rows. - A row is live while valid_to is NULL; current/as_of express the half-open [valid_from, valid_to) point-in-time predicate so call sites never hand-roll it - supersede_with (single row) and supersede_live_where (bulk) stamp the retired valid_to and successor valid_from with one timestamp for a gap-free handoff regardless of transaction boundaries - retire/retire_live_where are withdrawal primitives; retire cascades to live child links named in __retire_cascade__ - bulk supersede refuses to run on cascade-bearing classes since it cannot fire the cascade - consumers must add a partial unique index over their natural key WHERE valid_to IS NULL as a backstop against duplicate live rows Cover the mixin with unit tests against purpose-built models, using explicit timestamps far from the transaction clock to prove the gap-free handoff and verify the partial unique index backstop.
Add a lib module with reusable helpers for parsing and rewriting HGVS strings ahead of VRS translation. - extract_accession returns the reference accession (substring before the first colon), tolerant of whitespace and missing separators - split_cis_phased_hgvs expands a bracketed multivariant expression into fully-qualified components carrying the original accession and coordinate prefix, since ga4gh's AlleleTranslator only yields a single Allele per call and each component must translate alone - join_cis_phased_hgvs is the inverse, recombining components into one bracketed block and returning None when they do not share a single accession and coordinate prefix Cover the helpers with unit tests including the split/join round trip and the mixed-accession and mixed-prefix rejection cases.
Add VRS translation support for cis-phased multivariant HGVS, building on the new hgvs split helpers. - translate_hgvs_to_variation translates each component HGVS to an Allele independently, returning a bare Allele for a single component and wrapping two or more in a CisPhasedBlock, mirroring dcd_mapping's vrs_map._construct_vrs_allele; the reverse-translation job emits bracketed genomic forms the AlleleTranslator cannot translate directly - identify_variation generalizes identify_allele to blocks, clearing every member and location digest plus the block's own before identifying so a stale member digest never propagates into the block id; the block digest is order-independent so a set dedups to one row Cover both with unit tests, including the order-independent block digest and the stale-digest clearing.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
Introduce parallel mapping tables for the Better Reverse Translation epic (#746). The existing mapped_variants table is left untouched (frozen serving) while the new schema is built out separately. - add alleles, mapping_records, and mapping_record_alleles tables with their ORM models; alleles are content-addressed by vrs_digest and shared across mapping records via the link table - mapping_record_alleles.is_authoritative distinguishes the assay's actual measurement from translator-derived links, since the same VRS allele can be authoritative for one record and derived for another - put mapping_records and mapping_record_alleles on valid-time versioning (ValidTime mixin): a re-map retires the prior live row by closing valid_to instead of deleting, so history is retained and point-in-time queries are a single predicate; partial unique indexes promote "one live row per key" to the database - derive Allele.transcript and MappingRecord.transcript as hybrid properties from the HGVS columns rather than storing them, so they cannot drift; drop the stored alleles.transcript column - add the cross_level_translation annotation type, written once per variant to record whether filling unmapped levels succeeded, was skipped, or failed - annotate Variant and TargetGeneMapping with mapping_records relationships and typed primary keys
Add type stubs for the VRS and hgvs APIs used by the reverse translation work so callers can be type-checked without casts. - ga4gh.core: type ga4gh_identify and re-export PrevVrsVersion - ga4gh.vrs: stub the data proxy, AlleleTranslator, and normalize; translate_from returns Any so callers annotate the concrete VRS subtype they expect without a redundant cast - hgvs.assemblymapper: stub AssemblyMapper with the g_to_t/c_to_g/c_to_p conversions used for cross-level translation
Wire the variant-annotation package into the server extra as a local editable (PEP 660) dependency so the API can drive the reverse translation pipeline. - declare variant-annotation as a develop path dependency in pyproject.toml and the server extra; lock pulls in its transitive deps (variant-translation, biopython, openpyxl, et-xmlfile) - mount ../variant-annotation into the dev and worker containers and prepend it to PYTHONPATH so the editable install resolves - pass --no-directory to poetry install in the Dockerfile so the build does not try to install the not-yet-copied editable sibling - ignore_missing_imports for variant_annotation.* in mypy: its editable import hook is unfollowable and it ships no py.typed, so treat it as untyped rather than maintaining drift-prone local stubs - add a Makefile with dev/test/lint/format targets
Rework the variant mapping job to persist its results into the new MappingRecord / Allele / MappingRecordAllele schema instead of MappedVariant, building on the closure tables for the Better Reverse Translation epic. - create one MappingRecord per variant (including failed variants, with null VRS data); successfully post-mapped variants additionally get-or-create an authoritative Allele and link it via an is_authoritative MappingRecordAllele - supersede a variant's prior live record through ValidTime supersede_with — retiring it and its allele links and inserting the new record under one timestamp — instead of flipping a current flag, so the old/new handoff is gap-free - require a TargetGeneMapping for every mapped score, raising on a miss rather than tolerating a null link, since dcd-mapping guarantees one - combine cis-phased members when deriving hgvs_assay_level - update the mapping job tests to assert against MappingRecord and the authoritative-allele link, including the gap-free supersession and cascade-retire of the prior link TODO#765: mapping is not idempotent, so each run always creates a new authoritative link.
Add a seqrepo_data_proxy to the worker context in both the standalone context and the on_job_start hook, so jobs performing VRS translation have a SeqRepo-backed data proxy available alongside the cdot HGVS data provider. Mirror it in the mock worker context used by tests.
Add a reverse_translate_variants_for_score_set worker job that builds the cross-level HGVS equivalence class for every mapped variant in a score set, persisting the candidates as non-authoritative alleles. - for each current authoritative MappingRecord, collapse the assay-level HGVS to its protein consequence and expand to all coding/genomic candidates via a single batched construct_equivalent_variants call from the variant-annotation library, run off-loop in a thread - resolve each record's coding (NM_) transcript from its target gene's cdna alignment, falling back to a batched UTA NP_→NM_ lookup for protein-level mappings; records with no coding transcript are skipped - translate each candidate to VRS and write it as a get-or-created Allele linked non-authoritatively to the record, deduping by vrs_digest and never relinking the record's authoritative allele - supersede the prior live derived links with the new set in one gap-free operation via ValidTime.supersede_live_where - record per-variant cross_level_translation annotation status (success / failed / skipped), retaining per-candidate translation errors as metadata; the job fails only when every variant fails - add WorkerCoordinateTranslator and NullTranscriptSource adapters for the library's translation ports, deferring AssemblyMapper init so its network calls do not fire under mocked unit tests - get_or_create_allele helper, job registration, and a pipeline dependency placing reverse translation after mapping and before CAR submission TODO#765: a re-run supersedes the whole derived set wholesale because re-mapping re-mints the records; idempotent records would let unchanged derived links stay live.
…n date Previously the cdna-transcript lookup was keyed by target_gene_id alone, which meant a re-mapped score set could bind a stale NM_ transcript from an earlier run rather than the one the current run emitted. - Key cdna_transcript_by_run on (target_gene_id, mapped_date); within a key the highest-id row wins, so a same-run replacement takes precedence. - Carry mapped_date through the MappingRecord query so each record anchors to its own run's cdna row. - Add _TranscriptResolutionSkipReason to distinguish recoverable skips (protein-coding target, transcript unresolved) from correct skips (non-coding/regulatory target, no protein consequence). Emit skip_category in annotation_metadata. - Add target_gene_id to _TranscriptResolution so skip classification can look up the target's TargetCategory. - Add Mapped[] type annotations to TargetGene.id and .category columns. - New tests: genomic-accession coding target RT, latest-cdna-row-within- run selection, stale-cdna-row isolation, skip classification (coding recoverable vs. regulatory correct), and cdna TGM persistence in the mapping job. Refs mavedb-api#763
Forward translation emits predicted protein consequences in parentheses (e.g. p.(Ala222Val)). The parens denote inference, not a distinct variant form, so normalize to the bare p. form before storing. Strings without prediction parens are returned unchanged. Includes parametrized unit tests.
…st collisions - Re-identify each component via normalize_and_identify after translate_from so the stored vrs_digest reflects current content, not a stale value cached by the reused AlleleTranslator's Merkle tree. Without this, distinct biological variants at the same position (e.g. A>C and A>T) share one digest and are merged by the digest-keyed get_or_create_allele. - Use ga4gh_identify with in_place="always" in identify_allele and identify_variation so an allele that already carries a translator-stamped id has it recomputed, not retained. - Coerce ReferenceLengthExpression -> LiteralSequenceExpression in normalize_and_identify, mirroring dcd_mapping's _rle_to_lse, so RT-built alleles hash identically to the mapper's authoritative alleles and dedup correctly across sources. - Add regression tests: stale-id overwrite, distinct-alt digest isolation, cis-phased ordering canonicalization, and RLE->LSE coercion.
…tighten comments - Emit the protein consequence (result.hgvs_p) as a protein-level member of the equivalence set alongside the coding/genomic candidates. Prediction parens (p.(Ala222Val)) are stripped via strip_protein_prediction_parens before translation and storage. Protein-assay inputs are excluded (the protein is already the authoritative allele). - Trim all inline and docstring comments to essential why; remove redundant prose throughout reverse_translation.py. - Add tests: protein allele persistence, skip-classification for all three skip categories (no_assay_level_hgvs, transcript_unresolved, no_coding_transcript).
…rom failures Previously, variant mapping success was determined solely by the presence of pre/post-mapped alleles. This conflated genuinely failed mappings with benign absences (intronic variants, no-protein-consequence variants) that legitimately produce no allele. - Add `MappingOutcome` enum to `lib/mapping/schema.py` mirroring `dcd_mapping.schemas.MappingOutcome`, with an `is_benign_absence` helper to classify INTRONIC and NO_PROTEIN_CONSEQUENCE outcomes - Rewrite the per-record outcome logic in `map_variants_for_score_set` to branch on the typed outcome rather than allele presence: MAPPED -> SUCCESS, benign absence -> SKIPPED, FAILED -> FAILED - Replace the `successful_mapped_variants` scalar with a typed `Counter[MappingOutcome]` that feeds `mapped_count`, `failed_count`, and `skipped_count` tallies in logs and final state decision - Derive `mapping_state` from genuine failures only; all-benign result sets are treated as complete, not failed - Raise `NonexistentMappingResultsError` when a score annotation has no `outcome` field (malformed/older payload) - Expand test coverage for intronic and no-protein-consequence scenarios, verifying correct status and failure-category assignment
… UTA-backed source WtCodonMode.ALL reads the reference codon via TranscriptSource.codon_at, which requires a real UTA connection. The previous NullTranscriptSource always returned None, silently breaking WT-codon resolution. - Extract `uta_transcript_source()` context manager into `translation_ports.py`; removes the ad-hoc UTA connection setup that was duplicated in the job and the now-deleted NullTranscriptSource - Scope the live UTA client around the full `run_in_executor` call so the connection outlives the synchronous executor block - Remove NullTranscriptSource; callers that only need transcript_for_protein (already resolved by the job) also benefit from the real client without extra cost - Add a TODO noting that non-substitution consequences (del/ins/delins/ fs/ext) are miscounted as FAILED rather than SKIPPED (#767)
…ndle null gracefully - `translate_hgvs_to_variation` now attaches an `Expression` to each allele produced from a cis-phased or single HGVS string, mirroring the dcd_mapping authoritative-allele convention so `post_mapped` is self-describing without a separate round-trip. - `hgvs_from_vrs_allele` returns `None` instead of crashing when `expressions` is null or empty (valid for cis-phased block members), and `get_hgvs_from_post_mapped` propagates that as a `None` result. - `post_mapped` is now serialized with `exclude_none=True`, matching the mapper's output format. - Minor formatting clean-ups in reverse_translation.py (no logic change).
… model - `submit_score_set_mappings_to_car` now operates on Allele rows (authoritative + RT-derived) rather than MappedVariant, deduplicating by allele_id so each VRS allele is registered exactly once regardless of how many variants share it. Adds `force_reregister` param and per-allele outcome counters. - `submit_score_set_mappings_to_ldh` queries MappingRecord + Allele for pre/post-mapped data instead of the deprecated MappedVariant join. - `construct_ldh_submission_entity` signature updated to accept MappingRecord and Allele separately, since those fields now live on different models. - `warm_clingen_cache` switched to the shared `get_alleles_for_score_set` helper to keep allele scope consistent across all three jobs. - Extracts `get_alleles_for_score_set` and `ScoreSetAlleleRow` into `lib/clingen/alleles.py` as the single canonical query for both CAR and cache jobs.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
…-keyed refresh Replace MappedVariant-based gnomAD linkage with valid-time GnomadAlleleLink rows keyed on the deduplicated Allele. Linking covers every current allele of a score set (authoritative and RT-derived), so protein/coding score sets — whose genomic allele is RT-derived — are no longer dropped; per-variant annotation status flows through the _annotate_gnomad choke point against authoritative links only (interim bandaid, the AnnotationEvent migration seam). Refresh / idempotency model: - One live link per allele (unique index on allele_id); a gnomAD version bump supersedes rather than accumulating one live link per version. - Supersede only on change — an unchanged re-run leaves the live link untouched, so the valid-time history records no spurious boundary. - Version-keyed skip avoids re-fetching alleles already current at the version; a force param bypasses the skip (re-ingestion / heal) without churning unchanged links. - Per-variant status is a per-run audit event: created / preexisting / skipped. Normalize CAIDs across the Athena join: the gnomAD Hail dump drops leading zeros (CA025094 -> CA25094), so exact-string matching silently missed zero-padded CAIDs. Extract group_alleles_for_annotation as the shared allele-grouping primitive for allele-subject annotation jobs (adopted by gnomAD; VEP/ClinVar to follow). Refs #742. Partially addresses #722 (CAID-completeness tracking there stays open).
Migrate the VEP functional-consequence job (Step 2 of the annotation infrastructure migration) off MappedVariant onto the deduplicated allele model. - New ValidTime vep_allele_consequences table: a single allele-keyed row collapses record + link (the consequence is a scalar with no shared external entity, unlike gnomAD/ClinVar). One live consequence per allele via a partial unique index. - Job runs over the score set's full allele set (authoritative + RT-derived) via get_alleles_for_score_set + the shared grouping primitive; VAS still fans only to authoritative variants (the bandaid seam). - Version-key the refresh on the Ensembl release (/info/software): skip alleles already current, abort if the release can't be fetched. Supersede is value-keyed, not version-keyed — an unchanged consequence at a new release bumps source_version/access_date in place to avoid churning history. force bypasses the skip (e.g. after editing VEP_CONSEQUENCES). - A no-result is treated as a non-answer, never a negative: held consequences are not retired on an empty/failed VEP fetch. - Annotation links stay one-directional to Allele (no reverse back-ref). Adds lib + job tests covering linkage, RT-derived scope, version skip, in-place bump, supersede-on-change, no-result handling, and release-fetch failure.
Step 3 of the #742 external-annotation migration. ClinVar linkage moves off MappedVariant onto the deduplicated allele model. - Rename clinical_controls -> clinvar_controls (ORM model + table + unique constraint). Internal only: the ClinicalControl* view models, the /clinical-controls serving endpoints, and the frozen mapped_variants_clinical_controls association are unchanged, since the view-model name is both the OpenAPI schema name and the record_type discriminator consumed by the UI. - New clinvar_allele_links ValidTime table + ClinvarAlleleLink model. Multi-live: partial unique index (allele_id, clinvar_control_id) WHERE valid_to IS NULL, so an allele accumulates one live link per release. - Refactor refresh_clinvar_controls onto get_alleles_for_score_set + group_alleles_for_annotation (payload = CAID, full allele scope). Links are get-or-create; a same-version re-resolution to a different control supersedes newest-wins (gap-free retire+insert) rather than leaving two live links. VAS writes funnel through the _annotate_clinvar choke point, fanned only to authoritative_variant_ids and version-scoped. - Additively capture ClinVar's VariationID: nullable clinvar_variation_id column populated forward from the variant_summary TSV (parse degrades to None on archival schemas lacking the column). Unserved; the dedicated clinvar_variants remodel is deferred to the read-cutover. Tests rewritten for the allele model, covering the multi-live link writes, the version-scoped supersede guard, and the authoritative-only VAS fan-out.
Steps 4 & 5 of the #742 migration. Both jobs are redundant under the allele model: Allele.hgvs_g/c/p are populated by the mapping job, and the reverse-translation equivalence space (genomic/coding/protein alleles linked per MappingRecord) replaces the ClinGen PA<->CA translation table. - Remove populate_hgvs_for_score_set and populate_variant_translations_for_score_set from the pipeline DAG (both were leaf nodes — no dependency edges to repair), the worker registry (BACKGROUND_FUNCTIONS + STANDALONE_JOB_DEFINITIONS), and the external_services package exports. - Delete the two job modules and the now-orphaned ClinGen HGVS helpers (extract_hgvs_from_ca_allele_data / extract_hgvs_from_pa_allele_data), used only by the HGVS job. - Keep lib/variant_translations.py and the variant_translations table/model, marked FROZEN (serving-only) — they back old-model serving and are dropped at read-cutover. - Delete the obsolete job tests and their conftest fixtures.
get_allele_translations(db, allele_id, *, as_of=None) returns an allele's full cross-layer equivalence set (genomic/coding/protein) by traversing the MappingRecordAllele link graph: allele -> its live links -> mapping record(s) -> all co-linked alleles. The relation is co-membership in a MappingRecord's allele set, not a shared identifier — ClinGen's CAID spans only the nucleotide layers (the protein allele carries a distinct PA), so the link graph is the only thing tying all layers together. This replaces what the retired variant_translations PA<->CA table provided. Forward-compatible with temporal reads: the same half-open valid-time predicate is applied at both the anchor and fan-out hops, so passing as_of reconstructs the equivalence set as of any instant. Defaults to the currently-live set.
…bulary Introduces the AnnotationEvent log: a single append-only event table whose subjects are Variant and Allele, selected by annotation_type via a polymorphic CHECK. "Current" is derived (DISTINCT ON … id DESC), never stored. Adds the shared Disposition (present/absent/not_applicable/failed) and EventReason vocabulary, the v_current_annotation_events view, and the supporting migrations.
normalize_and_identify/_rle_to_lse assume an inlined SequenceReference and integer coordinates, but the VRS models type location.sequenceReference and location.start/rle.length as unions (iriReference/Range/None). Add narrowing assertions documenting that contract — restoring a green `mypy src/` (a required CI job) and turning the unreachable bad-input case into a clear AssertionError instead of a deeper AttributeError. Adds guard tests.
The newest-wins ClinvarAlleleLink supersede stamped valid_to/valid_from with a tz-naive datetime.now() into timestamptz columns, skewing the valid-time axis against every other job's func.now()-stamped rows. Use supersede_with so retire and insert share one DB timestamp. Test now asserts the boundary is timezone-aware.
Every other FK on annotation_event was indexed except score_set_id. Add it (model __table_args__ + migration) to back the ON DELETE SET NULL cascade on score-set deletion and score-set-scoped audit queries.
Revision a1b2c3d4e5f6 creates alleles/mapping_records/mapping_record_alleles — there is no allele-closure table anywhere (equivalence is a runtime query). Rename the file so it stops misleading readers; the revision id is unchanged.
Re-add cases dropped in the manager rewrite: None/empty reads, no-op flush, auto-flush-before-read for both query methods, a FAILED-disposition round-trip through the manager, and per-subject current-status isolation.
…ents split_cis_phased_hgvs raised ValueError on bracketed input lacking an accession; it now degrades to a single-element list. join_cis_phased_hgvs emits components in coordinate order so CSV exports are deterministic regardless of VRS member ordering (the block digest is order-independent).
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
Step 3 of the #742 external-annotation migration. ClinVar linkage moves off MappedVariant onto the deduplicated allele model. - Rename clinical_controls -> clinvar_controls (ORM model + table + unique constraint). Internal only: the ClinicalControl* view models, the /clinical-controls serving endpoints, and the frozen mapped_variants_clinical_controls association are unchanged, since the view-model name is both the OpenAPI schema name and the record_type discriminator consumed by the UI. - New clinvar_allele_links ValidTime table + ClinvarAlleleLink model. Multi-live: partial unique index (allele_id, clinvar_control_id) WHERE valid_to IS NULL, so an allele accumulates one live link per release. - Refactor refresh_clinvar_controls onto get_alleles_for_score_set + group_alleles_for_annotation (payload = CAID, full allele scope). Links are get-or-create; a same-version re-resolution to a different control supersedes newest-wins (gap-free retire+insert) rather than leaving two live links. VAS writes funnel through the _annotate_clinvar choke point, fanned only to authoritative_variant_ids and version-scoped. - Additively capture ClinVar's VariationID: nullable clinvar_variation_id column populated forward from the variant_summary TSV (parse degrades to None on archival schemas lacking the column). Unserved; the dedicated clinvar_variants remodel is deferred to the read-cutover. Tests rewritten for the allele model, covering the multi-live link writes, the version-scoped supersede guard, and the authoritative-only VAS fan-out.
… allele links Add the Cat-VRS transit foundation for the #743 variant + annotation API. The CategoricalVariant served on /variants/{urn} is assembled per request from a variant's live MappingRecordAllele links, not materialized: a stored skeleton on MappingRecord (the design's D33) would be a denormalized cache of the link graph with a sync cost and no benefit, since the link graph already derives it on demand and full-scan paths never assemble Cat-VRS. This reverses D33. - lib/alleles.py: get_live_record_allele_links — record-scoped live link set for a variant (defining + members, as_of-aware), distinct from the cross-record union of get_allele_translations. - lib/cat_vrs.py: build_categorical_variant (pure over links) -> CategoricalVariantTransit (spec-pure CategoricalVariant + mode + per-member digest->relation map), plus the categorical_variant_for_variant DB wrapper that threads as_of through the fetch. - ga4gh-cat-vrs promoted to a direct dependency (package 0.7.2 implements Cat-VRS spec 1.0.0); members are bare variations with relations on DefiningAlleleConstraint. - Anomaly logging when an authoritative or member allele has no post_mapped. - 16 tests (7 unit / 9 integration), mypy + ruff clean.
Add score_from_variant_data / variant_score to lib/variants.py as the single accessor for a variant's score_data.score — float-coercing (accepts numeric strings, rejects bool/None), robust to malformed JSONB. Replaces scattered variant.data["score_data"]["score"] reach-ins, several of which carried a # type: ignore. Migrate score_calibrations, annotation/util (variant_can_be_annotated), and annotation/study_result to the helper — dropping the duplicated defensive dict/float handling and the type: ignore suppressions. Unit-tested over the coercion / NA / non-dict matrix.
…variants)
Add the lean per-variant dataset the score-set page composes its table, heatmap, and histograms from: variant_urn, score, representative VEP consequence, ClinGen id + assay-level digest, and HGVS in both frames — submitted hgvs_nt/pro/splice (target) and mapped assay_level_hgvs + protein_level_hgvs (reference). Each slot carries the canonical HGVS string plus a parsed {position,ref,alt} block when it is a simple substitution (splice/indels/multivariants keep the string alone).
Assembled in one bulk query, one row per variant: the authoritative allele is joined directly for digest/ClinGen/consequence; the protein HGVS comes from a correlated scalar subquery so there is no allele fan-out. Opt-in ?as_of= reconstructs the annotation layer over the immutable scores/submitted HGVS and is echoed via the X-As-Of header. response_model_exclude_none keeps the payload lean; READ-gated via fetch_score_set_by_urn.
Adds simple-substitution HGVS parsers to lib/hgvs.py. A single canonical coding projection for genomic/coding assays is deferred to #784 (the degenerate reverse-translation set has no canonical member): the hover tooltip uses this strict model, the selection tooltip uses the detail endpoint's full set.
Tests: parser units + lib integration (nt/protein assays, as_of reconstruction, unmapped, unparseable-string, ordering) + router (serialization, exclude_none, X-As-Of, and the READ permission matrix).
…ering Add a live_at(as_of) hybrid_method to the ValidTime mixin — the one-call form of `as_of(ts) if ts is not None else current`. It is alias-aware (like as_of/current, its column refs adapt to an aliased() class), so a per-layer subquery can filter an aliased ValidTime table directly rather than hand-rolling a valid_to predicate. Migrate the repeated ternary in alleles.py to it. Covered by TestLiveAt: instance both branches, expression agrees with current/as_of, and alias-awareness.
Rework get_lean_score_set_variants to simplify ValidTime handling and the
post-mapped protein HGVS projection.
- Replace the inline record/link/vep liveness predicates with a shared
live_at(as_of) helper on each ValidTime model, removing the ad-hoc
and_/or_ valid_from/valid_to comparisons (and the now-unused or_ import)
- Swap the correlated scalar subquery for protein_hgvs for a single
DISTINCT ON subquery scoped to the score set and joined once, so its
cost scales with the response rather than per-row correlation
- Expand comments to document why live_at rides in each ON clause (a WHERE
would collapse the LEFT joins to inner joins for unmapped variants)
The protein-level HGVS projection was a plain subquery. Postgres badly under-counted its DISTINCT ON cardinality (est. ~5 vs ~2.8k actual) and buried it on the inner side of a nested loop, re-running the whole score-set-scoped projection once per variant. That is O(N^2) in the score set size and made the whole-set variants query hang on large sets. Rendering it as a MATERIALIZED CTE forces single evaluation, collapsing the query back to O(N). Results are unchanged; only the plan differs.
The module serves the whole-set variant view for a score set, not a generic 'lean variant' concept; the new name pairs it with lib/score_sets.py and lib/alleles.py. Pure rename plus reference/docstring updates.
Replace the flat measurement response with a two-tier envelope: flat assay-level fields (assay level, target/reference HGVS, digest, ClinGen id), per-calibration classifications, the spec-pure Cat-VRS molecular representation, and a digest-keyed annotation map (VEP/gnomAD/ClinVar). Adds ?as_of= reconstruction of the molecular layer (scores/classifications carry no ValidTime, so they are as-of-invariant) and folds in supersession self-describe (is_current/superseded_by) on this single-resource route. New lib/allele_annotations.py (digest-keyed annotation assembler, reused by the allele page) and lib/variant_detail.py (envelope composition); view models split into allele_annotation + variant_detail. Model columns gain Mapped[...] typing to back the annotation reads. Deletes the dead VariantEffectMeasurementWithScoreSet.
Rework the score-set clinical-controls routes off the legacy MappedVariant.clinical_controls association onto the new-model substrate (ClinvarAlleleLink -> Allele -> MappingRecordAllele -> MappingRecord), matching how the annotation pipeline now writes links. Adds ?as_of= (X-As-Of echo) to reconstruct the allele->ClinVar link state at an instant, and sorts control options newest-version-first. Test infra: add the reusable new-substrate seeder tests/helpers/util/annotation.py (seed_mapping_record + AlleleSpec building the MappingRecord/Allele/link graph), migrate the clinical-controls fixtures (link_clinical_controls_to_alleles) and the lean-view seeder onto it, and retire ClinvarAlleleLinks (not MappedVariant.current) to exercise the live-link filter.
… module The list and options handlers each carried the same ClinvarAlleleLink -> Allele -> MappingRecordAllele -> MappingRecord -> Variant join chain inline, plus an ad-hoc MM_YYYY version sort. Extract the traversal, grouping, and sort into lib/clinical_controls.py so the two endpoints share one definition and cannot drift, matching how the rest of the redesign pushes query logic into lib/. Drop the now-unused model imports and add a unit test pinning the version-sort contract the UI parses in parallel.
…riant detail The field carries the superseding *score set*'s URN, not a variant URN, so the old name read as if a consumer could jump straight to a superseding variant. Supersession is versioned at the score-set level and a newer version may add, drop, or renumber variants, so no stable superseding-variant pointer exists; document that alongside the rename. Also modernize get_variant's fetch to a 2.0-style select (preserving the MultipleResultsFound handling).
The lean whole-set view joins the authoritative MappingRecordAllele expecting it 1:1 with the variant. The existing (mapping_record, allele) unique index cannot stop two different alleles both being flagged authoritative for one record, so a second live authoritative link would silently duplicate the variant and double-count its score rather than raise. Add a partial unique index on (mapping_record_id) WHERE is_authoritative AND valid_to IS NULL so the mistake fails loud in the mapping job at write time. Detection query is clean on dev data.
…dence helpers Calibration selection had diverged across the serving layer (variant_detail ordered by primary/id; the new allele-measurements list used a fuller cascade). Add two pure helpers to lib/score_calibrations: - calibration_preference_key: the UI's default-calibration cascade (primary -> investigator_provided -> non-research-use-only), so the API and UI agree on the default calibration for a score set. - classification_evidence_strength: the (magnitude, direction) evidence primitive (ACMG points -> |ln(oddspath)| -> functional call). Repoint variant_detail's classification ordering onto the shared key so the variant page's first-listed classification matches the measurement card and the UI default. Annotate ScoreCalibration.id/primary as Mapped for the typed helpers.
GET /clingen-alleles/{caid}/measurements — the ClinGen-allele-centric variant page's
entrypoint. Resolves a CA/PA to its anchor alleles and lists every measurement whose
cross-layer equivalence class (MappingRecordAllele co-membership) touches it, not just
the exact allele. Each card is labeled by its assayed level and its relationship to the
query (direct / protein_consequence / nucleotide_encoding); the split falls out of which
allele anchors the co-membership.
Carries the primary readable classification inline, orders direct-first then by strongest
pathogenic-biased evidence, and opts into superseded measurements and as_of. Authorization
is has_permission in the lib (score-set READ gates inclusion, calibration READ gates the
inline classification).
…elope
Replace the UI-unused memberRelations map on GET /variants/{urn} with an
alleles sidecar keyed by VRS digest: one entry per linked allele (the
record-scoped, all-levels set, sharing keys with annotations) carrying level,
reference-frame HGVS, ClinGen id, and its relation to the measured allele.
This lets the selection panel label annotations at every level by level + HGVS
rather than by digest, without per-digest /alleles/{digest} fetches.
…y to avoid O(N²) planner trap The protein HGVS subquery uses DISTINCT ON, whose output cardinality is opaque to the planner (~18 estimated vs ~11.5k actual). Joined in, the planner buried it on the inner side of a nested loop and rescanned it in full once per variant — accounting for ~97% of the endpoint's total time. MATERIALIZED in a CTE fixed construction but not the per-variant rescan. Pulling the protein projection out as a standalone query and stitching it back by mapping_record_id in Python keeps both queries O(N) — the same batched-join pattern SQLAlchemy's selectin loader uses internally for the same class of cardinality estimation problem.
Add a nullable projection_group column to mapping_record_alleles and thread the coding/genomic pairing it records through the reverse- translation worker and the serving layer. - migration e7b2c9a1f4d3: add nullable projection_group (no index, no backfill; existing rows stay NULL and re-group on next re-map) - RT worker: consume the library's ProjectionPair list, stamp both links of a pair with a shared per-record group id, leave the protein apex ungrouped, and fold a pair member that equals the record's authoritative allele onto the existing link instead of duplicating it - lean variant view: replace assayLevelHgvs/proteinLevelHgvs with an assayLevel pointer plus a mapped MappedTriple (genomic/cdna/protein), filling the other nucleotide slot from the authoritative link's projection_group sibling via one indexed 1:1 join - variant detail: add derivation (authoritative/projection/candidate) and projection_of (the sibling's VRS digest) to AlleleIdentity Unpopulated projection_group (pre-RT data, protein apex) degrades gracefully to a null sibling / empty slot.
…emove legacy lookup endpoint
- Add `include_nucleotide_siblings` parameter to `get_allele_measurements`
and expose it as a query param on `GET /clingen-alleles/{id}/measurements`;
for a CA query, widens the equivalence class through the queried change's
protein consequence to surface sibling nt variants encoding the same
amino-acid change (relationship=nucleotide_encoding)
- Fix relationship labeling to key off measured-allele level rather than
entry level, correcting the sibling-nt branch so sibling nt records
get nucleotide_encoding instead of protein_consequence
- Remove `POST /variants/clingen-allele-id-lookups` and its associated
view models (ClingenAlleleIdVariantLookupsRequest, ClingenAlleleVariants,
ClingenAlleleIdVariantLookupResponse, VariantEffectMeasurementWithShortScoreSet)
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.