Skip to content

No public API (or index) for querying in-flight jobs by concurrency_key #805

Description

@epugh

Summary

SolidQueue::Job#concurrency_key (set via limits_concurrency) already encodes exactly the "is a business-level operation currently running" signal applications need — but there's no supported way to query it, and the column isn't indexed on solid_queue_jobs itself.

The app I work on Quepid has a long running jobs that do "LLM as a Judge"... I want to make sure that I don't accidentally cue up more than one Judge per specific Book. It's okay to have multiple judges work on the same Book. It's okay to have the same judge running twice, for two seperate Books. But not have the same Judge running twice for the same Book.

We wrote some janky code to look this up, and I think it would be nicer if it was part of SolidQueue.

A small part of me wondered if I have to send 1000's of query/doc pairs to evaluate to an LLM, should it be a queue, or is there a different architecture I should have picked?

Current behavior

Any job using limits_concurrency:

class RunJudgeJudyJob < ApplicationJob
  limits_concurrency to: 1, key: ->(book, judge, *) { "run_judge_judy_#{book.id}_#{judge.id}" }
end

gets a concurrency_key column populated on its solid_queue_jobs row (confirmed in our own app's DB: "RunJudgeJudyJob/run_judge_judy_1_8"). That's precisely what an app needs to answer "is this book+judge combination currently being judged," or "which judges are active for this book" via a prefix match. But today we have to:

  1. Recompute the key format ourselves ("#{class_name}/#{key}"), relying on an undocumented internal format (see ActiveJob::ConcurrencyControls#concurrency_key) that could change without notice.
  2. Query it with WHERE concurrency_key LIKE '...' against an unindexed column — a full table scan on solid_queue_jobs as job volume grows. (solid_queue_blocked_executions already indexes concurrency_key; solid_queue_jobs does not.)
  3. For anything the key alone doesn't capture (e.g., extracting which argument matched a prefix), fall back to parsing the raw serialized arguments JSON per row anyway.

Why this belongs in SolidQueue, not application code

The key is computed and persisted entirely by SolidQueue's own ActiveJob::ConcurrencyControls extension (injected into ActiveJob::Base via SolidQueue::Engine's on_load(:active_job) hook) — there's no adapter-portable way to derive it from outside the gem. Every app using limits_concurrency for a "one active run per business entity" pattern hits this same gap and re-derives the same fragile format independently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions