Fix catalog pricing saves failing on the queue mutex

[sjcallender]

This one has been nagging us for a long time. And it's about to bite us again on a new build, so we dove in deeper to offer what we see is a solid fix.

Specifically, the build pulls its products in from an ERP. Running the sync piles up so many catalog pricing jobs that the sync itself starts failing on the mutex. Pausing the queue is the only way to get through it. We see the same thing on another build whenever a lot of products get created at once.

Pausing the queue works for the console command, but we're importing off of webhooks, so the same saves will be running in a web request where we can't pause anything.

[edited]
#4123 was closed asking for a new ticket if it persisted. On 5.7.3 we still get 600 jobs for 600 saves.

This doesn't fix the database deadlocks in #4003 and #4141, which are in generateCatalogPrices() rather than the queue table. Fewer jobs means fewer of them running at once, so it should reduce how often those happen.
[/end edit]

Problem

N saves push N jobs against the one row they all merged into. All but a few reserve nothing, each taking the catalogpricingqueue mutex on the way.

The two sides of that mutex disagree:

	Timeout	On failure
reserveCatalogPricingQueueRow()	0	returns null
_queueCatalogPricingIds()	5	throw new RuntimeException

So a background job can fail a foreground save.

Changes

Merge by compare-and-swap	Write back conditional on the row being unchanged and still unreserved. A losing merge affects no rows and retries
Claim by conditional update	UPDATE ... SET reserved = true WHERE id = ? AND reserved = false, checking affected rows
Push only when nothing is pending	A job already working the queue picks up what arrives while it runs
Job loops over every pending row	Re-queues if a row arrives as it exits
Release reservations older than the queue's ttr	Nothing frees a row when a worker is killed rather than caught, so that work was stranded silently

We first tried dropping the merge and inserting a row per save. That made the write path much faster, 0.26s against 7.0s for 1200 saves, but it put generateCatalogPrices() on every row: 60 IDs in one call measured 0.025s against 0.081s as 60 single-ID calls. We kept the merge.

We also tried claiming every pending row up front. Killing a worker mid-drain left 200 rows reserved with nothing to free them, so we kept claiming one row at a time.

Our changes don't include any schema change, or SKIP LOCKED or RETURNING.

Verification

Postgres, runQueueAutomatically off, no workers running.
Six processes, 100 createCatalogPricingJob() calls each with a distinct purchasable ID.

	5.7.2	Patched
Queue rows	1	1
IDs preserved	600	600
Errors	0	0
Jobs pushed	600	3

Mutex contention. Holding catalogpricingqueue and then saving throws RuntimeException: Unable to acquire the catalog pricing queue mutex. on 5.7.2, succeeds patched.

Lost updates. Compare-and-swap against an unguarded read-modify-write of the same row, six processes merging 30 IDs each:

	Expected	Survived
Read-modify-write	180	66
Compare-and-swap	180	180

Reservation. Eight processes against 400 seeded rows: 400 claims, 400 distinct, 0 double-claimed, 0 left unreserved.

Crash. A worker that is killed after claiming leaves just its 1 row reserved; others drain the rest, and the expired-reservation release recovers it.

[sjcallender]

I should add that we really only see the error on staging/production. Locally, it finishes too fast, so our test forces the issue.

[lukeholder]

Thanks for the PR, we will review and give feedback soon. Thanks!

[sjcallender]

Thanks @lukeholder. While we have hopes for you all restructuring how all this works, this PR stays closer to the current setup. It is AI-assisted output for sure, but human-guided in shape and thought. Hope there's something valuable in here to help alleviate our builds' issues.