Skip to content

Concurrent gnomAD linking jobs fail on the live-link unique index #889

Description

@bencap

Problem

gnomAD links are shared across score sets, because alleles are. On RT (the allele-centric mapping work on feature/bencap/allele-centric-mapping-and-storage), gnomad_allele_links has a partial unique index allowing one live link per allele, and link_gnomad_variants_to_alleles (lib/gnomad.py) retires the old link and inserts the new one. Two gnomAD jobs on score sets with overlapping alleles can both try to insert the live link for the same allele. The second then fails on the unique index, and the job fails for work the first job already did.

Scope

  • Make the retire-and-insert for each allele safe under concurrency: insert with ON CONFLICT DO NOTHING on the live-link index, and treat an existing identical live link as done.
  • Take the per-allele writes in a consistent order, sorted by allele id, so concurrent jobs lock rows in the same order.

Acceptance criteria

  • A test runs two linking jobs over overlapping alleles on separate connections, and both succeed with one live link per allele.
  • A test asserts a job that finds an identical live link leaves it unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: databaseTask implementation requires database changesapp: workerTask implementation touches the worker

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions