OpenConceptLab/ocl_issues#2849 | Auto Match waits out a busy server (429 Retry-After), bounds $rerank, preview concurrency 2

[paynejd]

Makes the Mapper safe for a server-side capacity limit on heavy calls (semantic $match, $rerank) before the server ever enforces one (OpenConceptLab/ocl_online#275 ships it in shadow mode; enforcement is OpenConceptLab/ocl_online#340). Under load, Auto Match gets slower, never broken. Other 429s (DRF throttles) already reach these paths today.

Closes OpenConceptLab/ocl_issues#2849 (via 0d0221d).

What changes

Waiting out a 429 (src/services/capacity.js, new, dependency-free)

• requestWithCapacityRetry waits for Retry-After (header, HTTP date, or the body's retry_after) plus 0–50% jitter, however often a 429 comes, until a 30-minute cap. It then ends throttled. A 429 without Retry-After backs off from 5 s, doubling to a minute.
• 502/503/504 and network errors are retried twice with backoff (a 503's Retry-After is honoured), then end error. A 500 or 4xx is an error at once.
• Stop ends any wait within 250 ms. It never rejects: it returns {ok, reason: throttled|cancelled|error}.
• One gate per server per tab. A 429 on any request pauses the others to that server, and the bulk scheduler sends no new batch while the gate is paused, so a throttled run doesn't drain its queue into refusals.
• A limiter holds $rerank to 2 in flight per tab, including the per-row reranks fired as each batch finishes (a 10-row batch used to fire 10 at once). The end-of-run sweep waits for a row's queued rerank, reruns it for late candidates, and then proposes the row's mapping.
• A run stops asking once refused for the whole cap. When an algorithm's request (or a rerank) stays refused for 30 minutes, the rest of its rows end throttled without being sent, instead of each waiting another 30 minutes. A Retry-After longer than the cap ends that request at once without pausing the whole server, so other algorithms carry on.
• Transport timeouts bound a send that never answers. Heavy calls get 11 minutes, longer than the API itself runs a request, so legitimate $match work is never cut short or re-metered. Light calls (logs, run records, lookups, search) get 60 s. Stop never waits on a hung send: in-flight sends finish in the background.
• The X-OCL-Capacity-* headers are parsed and logged (in the row log for a wait; console.debug otherwise). Adapting to them is #340.

Callers moved off APIService.post/get (which never settled on a 429 and resolved a 5xx as its body)

• $rerank: a 429 used to freeze the run; a 5xx made the row look reranked with no scores.
• ScispaCy $match: its warm-up handling is kept, and its 2-minute warm-up wait now stops on Stop.
• Bridge $match: oclmap hands the Bridge Match component its service, whose post() now waits out 429s and always settles, so no react-bridge-match release is needed.
• Row-panel search and facets.
• The logs POSTs: one at a time per project, reading the log at send time, so a retried older POST can't overwrite a newer one.
• Also now waiting out 429s: bulk and single-row $match, concept lookups, $resolveReference, and AutomatchRun create/close.
• These calls pass a new handlesThrottle option to APIService.request, so the full-screen "Too many requests" countdown (which also covered Stop) stays closed for them. Other callers still get it.

Rows and runs

• A waiting row shows "Waiting for capacity: results may take longer due to demand" in its panel; the toolbar shows it while anything waits.
• A row still refused at the cap is throttled (new stage -4): "not run, retry", never failed. It isn't saved as a failed response, the end-of-run notice lists it apart from failures, and an AutomatchRun with throttled rows is partial, not failed.
• A stopped or failed run still autosaves and closes its AutomatchRun: the close PATCH retries and ignores Stop, and Stop also ends the wait for the create call (a run the server opens late is closed as cancelled).
• A superseded run's late results don't touch the newer run: no batch merge, rerank scores, or stage writes, and a stopped run proposes no more mappings.
• When ScispaCy or the bridge fails, or stays throttled, for a row matched by hand, the row goes on to its next algorithm and its rerank, as other algorithms do (for errors this predates the PR).
• A save keeps log entries written while it was in flight (this also predates the PR).
• A bridge call the component never sends (bridging unavailable to the user) marks the algorithm n/a (-3) instead of leaving the row "Running" forever.

Preview concurrency (own commit, ec304f0; the number is being confirmed)

• requestLimits.js caps by tier: core and staff keep the full range, early access 10 rows × 5, preview (or no tier) 10 rows × 2.
• A preview project has at most 25 rows (3 batches of 10), so 2 in flight costs one extra batch-time per run.

Acceptance criteria (#2849)

• Every listed caller uses service.request with a bounded retry that honours 429 Retry-After and backs off on 502/503/504.
• A 5xx or network error shows as an error on the row, never as "no candidates".
• Bulk Auto Match retries a throttled batch after Retry-After and pauses scheduling while throttled.
• Tests: 429 with Retry-After, 503, network failure, Stop during a wait, the rerank bound.
• A throttled row is never marked failed, shows "Waiting for capacity…", resumes on its own, and after the 30-minute cap is "throttled, retry".
• Stop always breaks out of a wait; the run still autosaves and closes its AutomatchRun.
• At most 2 $rerank in flight per run, the per-batch reranks included.
• Preview accounts: at most 2 $match batches in flight; early access stays at 5.

Tests

• npm run test:all: 396 unit + 6 integration, all passing. ESLint (CI's command) is clean.
• New: services/__tests__/capacity.test.js (44 tests), __tests__/rowProgress.test.js, and new cases in matchBatch, autosave, autoMatchRows, attribution and requestLimits.
• Manual end-to-end check: the real app against a local mock API that forces 429s, a 503, long refusals and Stop-then-restart, with no production traffic. Results are in two comments below; the last re-run was on 70497ea.
• Codex adversarial reviews: 12 passes are posted on this PR, each with the commit that fixed its findings. Pass 12 (7519657) is clean.
• Deliberate choices, explained in the passes:
  • no tight client timeout on metered $match;
  • a late result after Stop is still kept, as on main;
  • a Retry-After past the cap doesn't pause the whole server.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LxsPK3oiPfG85myNX2VDSj

[paynejd]

Codex adversarial review, pass 1 (codex-cli 0.159.2, commit ec304f0)

Scope: git diff origin/main...HEAD at ec304f0, read-only, adversarial prompt covering the ticket's acceptance criteria.

Outcome: fixed in ec0fb65, except item 1, which is only partly taken:

1. Hung sends: Stop no longer waits on a send that never answers. The scheduler, the rerank sweep, and the bridge and ScispaCy loops watch Stop while awaiting (untilCancelled), and in-flight sends finish in the background ("Stopping gracefully"). Not added: a per-attempt client timeout. $match is metered and can legitimately take minutes, so timing out and retrying it would double-charge and add load. Axios has had no timeout in these paths before either.
2. A batch that answers after a newer run started isn't merged into it, and a superseded run's rerank leaves the new run's stages alone.
3. The sweep awaits a row's already-queued rerank and then proposes the row's mapping. With the 2-rerank limit this was a real regression: the sweep's own call skipped most rows.
4. Gate waits have a deadline, so extended pauses can't carry a request past its cap.
5. A Retry-After past the cap pauses the gate before the request gives up. The scheduler doesn't hold the queue for a pause longer than the cap. Once an algorithm (or rerank) stays refused for the whole cap, the rest of its rows end throttled without being sent.
6. A throttled rerank shows "Server busy" in the row panel.
7. Each logs sender keeps its own payload, and the save path sends the snapshot taken when the save started.

On the Bridge Match contract: I read react-bridge-match main (93fc0a5). fetchBridgeCandidates calls service.post synchronously, and its only other call is the one-time get for the bridge repo, which this PR leaves unchanged. The "uncommitted edits" and the interrupted suite Codex mentions were my in-progress work at the time; the full suite passes at ec0fb65.

---

I would request changes. I reviewed git diff origin/main...HEAD; no files were modified.

1. High — Requests can still hang forever, blocking Stop and leaking rerank slots.
   src/services/capacity.js:219
await send() has neither a timeout nor cancellation. The migrated calls supply no Axios timeout or signal. If a connection stays open without answering, Stop cannot settle the request; two such reranks occupy both limiter slots indefinitely. Run finalization may never reach autosave or the closing PATCH. I reproduced a send remaining pending after cancellation.
   Fix: Give attempts a finite timeout and an AbortSignal tied to the run. Ensure cancellation settles the wrapper and releases slots.

2. High — A stopped batch can write results into a subsequent run.
   src/components/map-projects/MapProject.jsx:2310, src/services/capacity.js:249
   Cancellation is checked before sending, but not after the response arrives. The scheduler can return on Stop while other requests remain active. Starting another run resets abortRef; an old successful response then reaches the unguarded candidate merge and setMatchedConcepts. Only its stage writes have the run-ticket guard. Rerank result writes and cancellation stage writes have similar missing guards at MapProject.jsx:4206 and :4247.
   Fix: Capture immutable run context and guard every result, stage, log, quota, and mapping side effect. Discard obsolete responses after settlement; drain cancelled work before finalizing.

3. High — Run completion does not await reranks already launched by finished batches.
   src/components/map-projects/MapProject.jsx:4135, :2477
   When the final rerank sweep encounters a row already reranking, rerank() returns null immediately. If the debounced path already queued all rows, the sweep finishes while those requests are still waiting for capacity. The run then clears loading, schedules autosave, and closes its AutomatchRun before scoring finishes. Later scores may miss that save, and Stop can disappear while work remains outstanding.
   Fix: Track rerank promises per row and run. Return the existing promise when appropriate, and await all run-owned reranks before saving and closing.

4. Medium — Shared-gate waits bypass the 30-minute cap.
   src/services/capacity.js:205
   The cap is checked against the initial pause, then gate.wait() can follow unlimited extensions from other requests. After it finally opens, elapsed waiting is recorded, but the code can send successfully even after exceeding the cap. My reproduction configured a 2-second cap and obtained success after 8.5 seconds of waiting. Continuous extensions can wait indefinitely.
   Fix: Enforce the remaining budget during gate polling and jitter, and check it before sending.

5. Medium — A long Retry-After does not pause bulk scheduling.
   src/services/capacity.js:230
   When Retry-After plus jitter exceeds the remaining budget, the helper returns before calling gate.pause(). For example, a 429 with Retry-After exceeding 30 minutes marks each batch throttled immediately while the scheduler continues sending the remaining queue to the same busy server.
   Fix: Publish the server pause before terminating the individual request. Bound the scheduler’s wait separately and retain unscheduled rows without issuing further requests.

6. Medium — Rerank throttling is invisible in the row progress label.
   src/components/map-projects/rowProgress.js:19
   Progress examines only configured matching algorithms. With matching stages all 1 and rerank: -4, it returns “done,” leaving no retry label despite the rerank being throttled. The new tests cover waiting reranks but omit terminal rerank throttling.
   Fix: Include rerank’s running/error/throttled state when deriving row progress, and test the rendered Candidates chip.

7. Medium — Retried logs can overwrite one project with another project’s logs.
   src/components/map-projects/MapProject.jsx:1728
   The sender captures project A’s URL but reads component-wide log refs on every retry. Navigating directly to project B reuses the route component and replaces those refs. A delayed retry for A can consequently POST B’s logs to A. Coalescing prevents overlapping sends within one sender, but does not preserve project ownership.
   Fix: Keep log payloads and sender state keyed by project identity; each trigger should update only that project’s payload.

The ordinary Retry-After parsing, jitter, gateway-error backoff, limiter release logic, preview/early-access caps, and unchanged core/staff request limits look fine. Capacity headers are parsed and logged without adaptive behavior, and the new translation keys exist in English, Spanish, and Chinese. Disabling ambiguous-error retries for AutomatchRun creation is also appropriate.

The premium Bridge Match implementation is loaded externally, so its synchronous post() assumption and other service calls remain unverified; the local stub does not prove that contract.

The focused tests passed 174/174, but mostly exercise helpers rather than MapProject’s lifecycle. Existing uncommitted edits in matchBatch.js and its tests were excluded from the PR findings; the test run included those edits. The broader suite did not finish and was interrupted.

[paynejd]

Codex adversarial review, pass 2 (codex-cli 0.159.2, commit ec0fb65)

Scope: git diff origin/main...HEAD at ec0fb65, read-only; verify the pass-1 fixes, then look for new problems.

Outcome: fixed in bd81b0f.
1, 3 and 7. A send that never answers is now bounded by transport timeouts. Heavy calls get 11 minutes, longer than the API itself runs a request, so a legitimate $match is never cut short or re-metered. Light calls (logs, run records, lookups, search) get 60 s. Stop also ends the wait for the AutomatchRun create call, and a run the server opens after that is closed as cancelled.
2. A superseded run's bridge, ScispaCy and rerank outcomes leave the new run's stages alone, and a stopped run proposes no more mappings. Kept on purpose: a late result after Stop in the same run is still merged, as on main ("Stopping gracefully"). Those candidates are real, and the next save persists them.
4. The sweep reruns a row after its queued rerank ends, so candidates that arrived meanwhile are scored before the row's mapping is proposed.
5. The scheduler's hold on a paused gate is 30 minutes in all.
6. A save keeps the entries logged while it was in flight, instead of resetting the log to its starting snapshot (this predates the PR). A save that lands after another project was opened leaves that project's log alone.
8. When ScispaCy or the bridge fails, or stays throttled, for a row matched by hand, the row goes on to its next algorithm and rerank, like other algorithms. This also predates the PR, for errors.

The local end-to-end check (below, in a separate comment) also found that my pass-1 fix for item 5, pausing the whole server for a Retry-After past the cap, let one refused algorithm block the others, the reranks and later runs for as long. bd81b0f reverts it: that request gives up alone, and the run stops asking for its algorithm only.

On the Bridge Match contract, see pass 1: react-bridge-match main calls service.post synchronously.

---

Pass 2 still finds blocking issues. I reviewed git diff origin/main...HEAD and ran the focused tests: 187 passed. No files were modified.

1. High — Stop still cannot escape a hung AutomatchRun creation request.
   MapProject.jsx:1863, capacity.js:230
   createAutomatchRun() directly awaits requestWithCapacityRetry(), which directly awaits send(). Cancellation is checked during backoff, but not during an in-flight send. If the creation POST never answers, Stop leaves the UI stopping indefinitely; execution never reaches the pipeline’s finally.
   Fix: Make awaiting creation cancellable without retrying an ambiguous POST. Handle a late successful creation separately so its returned run ID is closed. Give the idempotent completion PATCH a bounded transport timeout as well; its current wait budget does not bound a hung send.

2. High — Stopped or superseded runs can still overwrite current results and decisions.
   MapProject.jsx:2614, MapProject.jsx:4024, MapProject.jsx:4259
   Stop abandons the outer wait but leaves its request running. A late bridge or ScispaCy success still invokes callbacks that mark stages and merge candidates. Rerank checks supersession only in cancellation/throttle branches; successful responses and terminal errors still update the newer run. Its delayed setAutoMatched() can overwrite a subsequent manual decision. Ordinary batches guard against a newer ticket, but Stop itself does not invalidate their ticket, so late results can arrive after the stopped run’s autosave and completion counts were captured.
   Fix: Capture an immutable run context and guard every result, error callback, stage/log update, and delayed proposal against cancellation and supersession. Apply the guard after every awaited send.

3. High — Hung reranks permanently occupy the shared limiter.
   MapProject.jsx:4217, MapProject.jsx:4308
   Two hung $rerank sends hold both slots. Stop lets the sweep return, but neither rerank reaches finally, so neither releases its slot or settles its row’s in-flight promise. Every subsequent run’s reranks queue forever behind them. Same-row sweep calls also await those old promises.
   Fix: Scope limiter and in-flight bookkeeping to the run, with separate manual-request bookkeeping, or implement explicit cancellation of abandoned requests. Preserve the deliberate no-timeout/no-retry policy for metered $match; merely abandoning the sweep is insufficient.

4. Medium — The sweep waits for the first rerank, but not a required follow-up rerank.
   MapProject.jsx:4153, MapProject.jsx:4311
   If candidates arrive while a rerank is running, the sweep awaits that promise and returns. finally settles the promise and schedules the follow-up through a debounce. The run can finish, invoke AI, autosave, and propose a mapping while the follow-up rerank is still pending. That follow-up uses isBulk=false, so it does not correct the proposal after better scores arrive.
   Fix: Make the row’s completion promise encompass required reruns, or have the sweep repeatedly await/recheck until eligible candidates are scored. Propose only after that final completion.

5. Medium — Scheduler gate waits still have a renewable 30-minute budget.
   matchBatch.js:140
   If the scheduler starts while its shared gate is paused, it may have no active request. Other requests can keep extending that pause. gate.wait() eventually returns 'timeout', but the scheduler ignores the result and starts another wait with a fresh maxHoldMs. No request has entered requestWithCapacityRetry(), so no row sets the run-level throttled flag. The queue can remain pending indefinitely. I reproduced repeated timeout results receiving a fresh budget each time.
   Fix: Track a cumulative scheduler deadline and handle 'timeout' by marking the queued rows throttled and ending scheduling.

6. Medium — A late save can overwrite newer logs for the same project.
   MapProject.jsx:1684, autosave.js:173
   Save captures log snapshot A. While it is pending, new entries produce snapshot B and trigger its logs POST. When save finishes, it triggers A plus the save entry. The sender treats the latest trigger as the latest content, so the older snapshot replaces B—even during B’s retry. projectLogsRef is also reset to the older snapshot.
   Fix: Maintain project-scoped log state with revisions. Merge the save entry into that project’s latest logs, and reject older snapshots instead of ordering solely by trigger arrival.

7. Medium — One hung logs POST blocks every subsequent logs update.
   MapProject.jsx:1731, autosave.js:157
   Serialization introduces a new failure mode: a POST that never settles keeps the sender’s chain active forever. Later log changes only set pending; none can be transmitted. The five-minute retry budget bounds sleeps, not the send itself.
   Fix: Bound and cancel the logs transport attempt, then send the latest snapshot. Preserve ordering so an abandoned older request cannot subsequently overwrite newer logs.

8. Medium — ScispaCy throttle/error outcomes do not advance the single-row algorithm chain.
   MapProject.jsx:3932, MapProject.jsx:4032
   fetchAllCandidatesForRow() advances to the next algorithm through onResponse. ScispaCy invokes that callback only on success. A capacity-cap outcome or terminal error marks ScispaCy and returns directly, leaving subsequent algorithms unrun and preventing the normal rerank continuation.
   Fix: Return terminal outcomes through the common callback, or centralize chain advancement so every outcome settles and advances exactly once.

The Pass 1 fixes that are fine: the request-level gate deadline, pausing the gate before rejecting an oversized Retry-After, the throttled-rerank label, and isolating retry payloads by project URL. Preview concurrency is capped at two; early access remains five; core/staff saved request settings remain uncapped. The added translation keys exist in all three locale files, and capacity headers remain observational.

The helper tests do not prove React run isolation, Stop during run creation, limiter recovery after abandoned sends, completion of follow-up reranks, or preservation of logs across late saves. The premium bridge implementation is externally loaded, so its synchronous post() assumption and other service calls remain unverified from this checkout.

[paynejd]

Codex adversarial review, pass 3 (codex-cli 0.159.2, commit bd81b0f)

Scope: git diff origin/main...HEAD at bd81b0f, read-only; verify the pass-2 fixes, then report only new problems of real consequence.

Outcome: both fixed in 0d0221d. A superseded run's rerank no longer writes its scores. The orphan-run close retries like the normal close.

---

Two new medium-severity findings:

1. Superseded reranks still overwrite the newer run’s scores — MapProject.jsx:4347
   Scenario: Stop run A with a rerank in flight, change the input or encoder, and start run B. A’s late success patches B’s current ConceptRows; the superseded check protects only stages. B can then skip those candidates as already scored and propose mappings using A’s scores. A read-only harness executing the actual rerank function reproduced a newer score changing from 20 to 99 after A answered.
   Fix: Check isSuperseded() before processing or merging successful results. Preserve graceful late merging only within the same run.

2. Orphan-run closure abandons transient failures — MapProject.jsx:1881
   Scenario: Stop while AutomatchRun creation is pending. The POST subsequently succeeds, but its cleanup PATCH receives 429 or 503. This path sends once and silently catches rejection, leaving the server record open permanently. Normal run closure correctly uses bounded retries.
   Fix: Use requestWithCapacityRetry for this idempotent cleanup PATCH, with the light transport timeout and bounded capacity wait, independent of Stop.

The affected helper suites pass 190/190 tests, but they do not exercise these lifecycle paths. The transport bounds, cumulative scheduler hold, limiter release, log coalescing, tier limits, translations, and split-view chip look fine. No files were modified.

[paynejd]

Manual end-to-end check: forced 429s, the real app, a local mock API

• Setup:
  • The app was built from the branch and pointed at a local mock API; headless Chromium drove it.
  • No production or staging traffic: the browser blocked every non-local request.
  • The test user is a preview user with a 25-row project. It was saved with 5 concurrent requests, so the new preview cap had to clamp it to 2.
• Runs: first on ec304f0 (the original head), then re-run on ec0fb65 and bd81b0f after the review fixes. The results below are for bd81b0f. 0d0221d changes only the superseded-run rerank write and the orphan-run close, which these scenarios don't exercise.

Scenario	What the mock does	Result
A: wait, then finish	First 2 $match → 429 Retry-After: 4; one later → 503; first $rerank → 429 Retry-After: 3	Pass.
B: Stop during a wait	Every $match → 429 Retry-After: 30; Stop pressed mid-wait	Pass.
M: multi-algorithm auto-proposal	2 algorithms, top rerank score 99.5 (above the "Recommended" threshold)	Pass, twice.
T: one algorithm refused past the cap	ocl-semantic → 429 Retry-After: 4000 (past the 30-min cap); ocl-search answered	Pass.

A

• "Waiting for capacity: results may take longer due to demand" showed in the toolbar (short form in the split view, full text on hover) and in the open row's panel.
• The full-screen "Too many requests" overlay never appeared.
• The next batch was held until the pause ended.
• The 429'd batches retried after 4 s plus jitter, and the 503 was retried.
• All 25 rows saved with reranked candidates. The run closed completed, then autosaved.
• Max in flight: 2 $match, 2 $rerank.

B

• No $match went out in the 40 s after Stop.
• The run closed cancelled, and the autosave still followed.
• After Stop, only the close PATCH, the logs POSTs and the autosave went out.

M

• 25/25 rows auto-proposed, against 10–12 on main (whose reranks raced the end-of-run sweep). ec304f0 got 2/25; ec0fb65 fixed it.
• Max in flight: 2 $match, 2 $rerank.
• The close PATCH went out after the last rerank.

T

• ocl-semantic sent its first 2 requests and nothing more; its rows are listed as throttled ("Nothing failed").
• ocl-search matched all 25 rows, and the reranks ran.
• The run closed partial, and autosaved.
• A second run in the same tab with the mock answering everything matched both algorithms and closed completed.
• On ec0fb65, T showed one refused algorithm blocking the others for the whole Retry-After. bd81b0f fixed it.

Not exercised here: ScispaCy and the bridge (their services aren't mocked), and the real 30-minute cap. Unit tests cover the cap with a virtual clock.

[paynejd]

Codex adversarial review, pass 4 (codex-cli 0.159.2, commit 0d0221d)

Scope: git diff origin/main...HEAD at 0d0221d, read-only; verify the pass-3 fixes, then report only new problems of real consequence.

Outcome: both fixed in f22d7d7. rerank() captures its run on entry, checks Stop after waiting for lookups, and releases only its own in-flight entry. A run whose reranks stayed throttled closes as partial.

---

Two new medium-severity findings at 0d0221d:

1. A stopped rerank can overwrite a newer run’s stage. MapProject.jsx:4274
   Run A’s rerank waits for pending concept lookups. Stop returns through the scheduler; run B starts before those lookups settle. A then resumes, captures B’s ticket as its supersession baseline, writes rerank: 0, and writes rerank: -1 when its original cancellation check rejects acquisition. It can also clear B’s debounce and overwrite its in-flight entry. I reproduced the stage writes using the extracted current function.
   Fix: Capture the ticket at function entry, check cancellation immediately after lookup waits and before shared-state mutations, and delete in-flight entries only if they still belong to that invocation.

2. Throttled reranks are recorded as completed runs. MapProject.jsx:1924
   When retrieval succeeds for every row but $rerank reaches its capacity cap, rows correctly receive rerank: -4. However, completion summarization receives only retrieval algorithm IDs, so it ignores rerank throttling and PATCHes completion_status: 'completed'. Direct reproduction returned {completed_rows: 1, failed_rows: 0, completion_status: 'completed'} for a successfully retrieved, throttled-rerank row.
   Fix: Include rerank throttling in terminal-status determination while preserving the intended retrieval-based row counts.

The pass-3 fixes themselves look correct: superseded response scores are blocked before merging, and orphan-run closure retries. All 396 unit tests pass, but they do not cover these two scenarios. No files changed.

[paynejd]

Codex adversarial review, pass 5 (codex-cli 0.159.2, commit f22d7d7)

Scope: git diff origin/main...HEAD at f22d7d7, read-only; verify the pass-4 fixes, then report only new problems of real consequence.

Outcome: both fixed in 063f85f.

• A rerank re-checks the run's throttle cutoff once it has its limiter slot.
• A run resets a rerank left throttled by an earlier run.
• A row whose candidates are all scored already has its rerank marked done.

---

Two new findings at f22d7d7:

1. High — queued reranks bypass the run’s throttle cutoff. MapProject.jsx:4289
   The throttle check happens before acquiring the limiter slot. When a batch queues ten reranks, the first two can exhaust their 30-minute budgets, but the eight already queued still acquire slots and start fresh retry budgets. Persistent 429s can prolong the run for hours while continuing requests for an algorithm the run supposedly stopped asking for.
   Fix: recheck run throttling after acquiring the slot and before sending; mark queued rows -4 and release through finally. Reproduced using the actual rerank function body: the third queued call started after the first had recorded run-level throttling.

2. Medium — a successful retry can retain the previous run’s throttled rerank stage. MapProject.jsx:2481, MapProject.jsx:4262
   Run initialization resets retrieval stages but preserves rerank: -4. After a throttled run, configure only semantic matching and retry successfully with inline rerank scores. The sweep finds nothing unscored and returns without clearing -4. The row still shows “Server busy,” and pass-4’s summary reports this successful run as partial.
   Fix: reset rerank state for intended rows at run start, and mark rerank complete when all eligible candidates already have scores. Reproduced the stale-stage path and resulting partial summary.

The specified pass-4 guards are present and look correct. All 191 targeted tests passed; they don’t cover these two wiring scenarios. No files were modified.

[paynejd]

Codex adversarial review, pass 6 (codex-cli 0.159.2, commit 063f85f)

Scope: git diff origin/main...HEAD at 063f85f, read-only; verify the pass-5 fixes, then report only new problems of real consequence.

Outcome: both fixed in 891fa8a.

• A result that lands after Stop is still kept, and now schedules an autosave.
• A superseded batch's preview limit is ignored, so it can't stop the newer run.

---

Two new findings, both medium:

1. Late results after Stop can escape autosave. MapProject.jsx:2579
   Stop releases the scheduler and schedules autosave after five seconds. If an in-flight batch returns afterward, its results are still merged at lines 2353–2387, but no further save is scheduled. Reopening the project loses those visible results.
   Fix: Schedule autosave when a late result is merged into its stopped, still-current run, including late rerank scores. Preserve the intended graceful merge.

2. A superseded batch’s preview-limit response can stop the new run. MapProject.jsx:2325
   Stop run A while its request is outstanding, then start run B. If A subsequently returns a preview-limit 403—for example, mapper_custom_algorithms_denied—onPreviewLimit calls the shared handler without checking A’s ticket. The handler reads B’s bulk-run state, sets B’s matchQuotaStopRef, and drops B’s remaining matching work. Stage and result guards don’t protect this callback.
   Fix: Guard onPreviewLimit with isCurrentRun() before changing quota state or displaying its dialog.

Both paths reproduced in memory using the actual request/scheduler helpers. All 397 unit tests pass, but they don’t cover these lifecycle scenarios.

The three pass-5 fixes are present and look correct. No files modified.

[paynejd]

Codex adversarial review, pass 7 (codex-cli 0.159.2, commit 891fa8a)

Scope: git diff origin/main...HEAD at 891fa8a, read-only; verify the pass-6 fixes, then report only new problems of real consequence.

Outcome: no change. This is a deliberate decision.

After Stop, a result that lands late is kept (graceful stop), and scoring its unscored candidates is the point of keeping it. main does the same, and it's the path the row panel already takes when a row with unscored candidates is opened. These reranks are:

• idempotent and unmetered;
• held to 2 in flight by the limiter;
• bounded by the transport timeout and the capacity cap.

Carrying a run ticket through merges, lookups and debounce timers would add plumbing to stop work whose results we want. The pass-6 fixes are confirmed.

---

One new finding:

• Medium — automatic reranks lose their run identity after Stop. MapProject.jsx:4458, with the call at line 4496.

  Scenario: Run two algorithms; let one finish, then Stop while the other is in flight. Once the stopped pipeline clears its bulk flags, the second result lands and is gracefully merged. With both algorithm stages complete, scheduleRerank starts rerank(index, false, false). That treats this automatic work as manual: it ignores Stop and the run’s throttle cutoff, allowing fresh $rerank requests and another capacity wait after the run closed. A delayed lookup completion can trigger the same path.

  Fix: Carry the originating run ticket through merges, lookup completions and debounce callbacks into rerank. Check that ticket’s cancellation and throttle state before scheduling or sending. Keep late-result merging and autosaving.

  Evidence: Executing the extracted scheduler with completed stages and cleared bulk flags produced rerank(0, false, false). The targeted tests cover helpers, not this React lifecycle path.

The two pass-6 fixes are present and correctly guarded on inspection. All 191 targeted tests passed. No other new consequential findings; no files modified.

[paynejd]

Codex adversarial review, pass 8 (codex-cli 0.159.2, commit 891fa8a)

Scope: git diff origin/main...HEAD at 891fa8a, read-only; report only new problems of real consequence (the pass-7 decision stated as deliberate).

Outcome: fixed in 70497ea.

• Every phase of a run checks the run's own Stop (stopped, or superseded by a newer run): the scheduler, the algorithm loop, the bridge and ScispaCy loops, the sweep, the AI step and the end-of-run tail.
• Each run closes the AutomatchRun record it opened, passed explicitly.
• Only the current run clears the shared loading state.
• Beyond Codex's report: the same abortRef reset also let a stopped run carry on into its next algorithm, which the same change covers.

---

Found one new high-severity issue.

• High — stopped pipeline can resume AI work and close the next run’s record. MapProject.jsx:2405, MapProject.jsx:2528

  Scenario: Enable automatic AI analysis, stop run A while matching is waiting, then start run B during A’s one-second delay before AI analysis. The cancelled scheduler clears loadingMatches, allowing B to start. B resets abortRef.current to false, so A’s unguarded continuation enters runBulkAIAnalysis and processes A’s rows. If B’s AutomatchRun opens before A finishes, A’s finally calls completeAutomatchRun, which reads and clears the shared automatchRunRef: A closes B’s record using A’s row counts, while A’s record remains open. It also clears B’s loading state and can consume AI allowance on unintended rows.

  Suggested fix: Carry A’s captured ticket/cancellation check through every pipeline phase, including the delayed AI transition and AI loop. Capture the AutomatchRun record per invocation and pass it explicitly to completion; only the current run should change shared loading state.

  A reduced reproduction using the actual runWithConcurrency produced: old AI starts with ticket B → B’s record opens → old finalization closes B. The added tests do not exercise this component-level overlap.

The 191 relevant tests pass. I found no other new concrete problems; the deliberately accepted late-result and post-Stop rerank behavior is fine. No files changed.

[paynejd]

Codex adversarial review, pass 9 (codex-cli 0.159.2, commit 70497ea)

Scope: git diff origin/main...HEAD at 70497ea, read-only; verify the pass-8 fix, then report only new problems of real consequence.

Outcome: fixed in 3b2e0e7.

1. The AI step's awaits (the prompt template and each row's AI call) no longer hold Stop: the run goes on to its autosave and closes its record, and the call finishes in the background. Out of scope: moving the AI Assistant calls themselves to a capacity-aware retry is OpenConceptLab/ocl_online#206.
2. A superseded ScispaCy answer no longer touches the alert or the panel's loading state.

---

Found two new issues at 70497ea:

1. High — Stop can still hang forever during the AI phase.
   MapProject.jsx:2625, also lines 2642 and 5460.
   Scenario: Enable automatic AI analysis; let matching finish; return 429 from /prompts/<key>/. Prompt retrieval still uses .get(), and APIService.sendRequest returns a never-settling promise on 429. Stop cannot interrupt the await, so the pipeline never reaches its autosave or AutomatchRun close. An unresponsive AI invocation likewise has no transport timeout or cancellable outer await.
   Fix: Use service.request with bounded capacity retry and a transport timeout for prompt retrieval. Make both AI-phase awaits cancellable using the captured run’s stop check. Guard detached outcomes against supersession and preserve the invocation’s idempotency key.

2. Medium — A superseded ScispaCy response clears the current panel’s loading state.
   MapProject.jsx:4104, particularly line 4111.
   Scenario: Stop run A while its ScispaCy request is in flight. Start run B and open a row whose candidates are loading. A’s request then succeeds. Its bulk callback correctly rejects the superseded result, but the surrounding success handler still clears the shared alert and calls setIsLoadingInDecisionView(false). B’s panel can show an empty/non-loading state before its request finishes.
   Fix: Check isSuperseded() before success-handler UI mutations; likewise guard the unconditional loading reset in endStopped().

Pass-8’s algorithm-loop checks and explicitly passed AutomatchRun record are correct. Its “only the current run clears loading state” fix remains incomplete.

All 191 focused tests passed; they do not cover these React pipeline scenarios. No files modified.

[paynejd]

Codex adversarial review, pass 10 (codex-cli 0.159.2, commit 3b2e0e7)

Scope: git diff origin/main...HEAD at 3b2e0e7, read-only; verify the pass-9 fixes, then report only new problems of real consequence.

Outcome: fixed in eec23c5. A run's pending rerank debounce remembers its run and is dropped if a newer run has started; the newer run reranks its own rows. No unit test: the path lives in MapProject.jsx's component lifecycle, which the node test runner can't mount. The local end-to-end restart scenario (R) covers Stop followed by a new run.

---

One new finding at 3b2e0e7:

• Medium — A pending rerank debounce can adopt the next run and disable its reranks. MapProject.jsx:4530, with ticket capture at MapProject.jsx:4263.

  Scenario: Run A finishes a batch and schedules its 300 ms rerank debounce. Stop A and start B before that callback executes. The callback retains A’s isRunTraffic=true, but rerank() captures B’s ticket. Its supersession checks therefore accept it as B’s work. If this request receives a 429 whose Retry-After exceeds the cap, it sets B’s shared rerank throttle flag; B’s subsequent reranks are skipped as throttled, including rows unrelated to A.

  I reproduced the ownership error in memory using the actual scheduleRerank function: scheduling under A and firing under B produced runTraffic: true, capturedTicket: "run-B".

  Fix: Capture the originating ticket when scheduling and discard run-owned callbacks after supersession, or pass that ticket and its cancellation check into rerank(). Add a regression covering schedule → Stop → new run → timer fires. This concerns work scheduled before Stop adopting another run; the deliberate late-result non-run reranks remain fine.

The pass-9 fixes look correct: both AI awaits are cancellation-wrapped, and superseded ScispaCy successes leave panel state alone. All 191 targeted tests passed, but they do not cover this debounce/run-transition boundary.

No other new concrete findings. No files modified.

[paynejd]

Manual end-to-end check, re-run on 70497ea (same setup: the real app, a local mock API, no production traffic)

All five scenarios pass: A, B, M and T as before, plus a new one, R (Stop, then restart).

R

• Setup: $match answers 429 Retry-After: 30. Stop is clicked 2.5 s into the wait, and a second Auto Match starts 0.2 s later with the mock answering everything.
• The first run:
  • closed cancelled 75 ms after Stop, under its own AutomatchRun id;
  • sent nothing under that id after Stop.
• The second run:
  • opened its own record, and all its traffic (3 $match, 25 $rerank) carried that id;
  • closed completed (25/0);
  • was autosaved with all 25 rows.
• As designed: the second run held off until the first run's 30-second Retry-After had passed, with the "Waiting for capacity" chip showing.

A, B, M, T

• M: 25/25 rows auto-proposed; max 2 $match and 2 $rerank in flight.
• A: every row saved and reranked; the run closed completed.
• B: nothing sent after Stop; the run closed cancelled, and the autosave followed.
• T: only the refused algorithm's rows ended throttled; the run closed partial; a second run matched everything.

The commits after 70497ea change the AI step (not mocked here), a superseded ScispaCy answer, and a superseded rerank debounce.

[paynejd]

Codex adversarial review, pass 11 (codex-cli 0.159.2, commit eec23c5)

Scope: git diff origin/main...HEAD at eec23c5, read-only; verify the pass-10 fix, then report only new problems of real consequence.

Outcome: fixed in 7519657. A rerank's follow-up goes through scheduleRerankRef.current, the current render's scheduler, not its own closure's. No unit test: this is MapProject.jsx component-lifecycle code.

---

One new finding at eec23c5:

• High — A superseded rerank can schedule work for the new run using the old configuration. MapProject.jsx:4466

  Scenario: Run A has a rerank in flight. Stop A, change the encoder, and start run B. B’s debounced rerank encounters A’s in-flight entry and sets rerankRerunNeededRef. When A’s response arrives, its scores are correctly discarded—but its finally calls the old closure’s scheduleRerank(index). That scheduler captures B’s current ticket, so the pass-10 guard accepts it. It then sends and merges scores using A’s encoder or row input. B’s sweep subsequently skips those candidates because they already have scores, potentially proposing mappings from the wrong configuration.

  Reproduction: An in-memory harness using the actual extracted rerank and scheduleRerank functions produced ["OLD-ENCODER", "OLD-ENCODER"] after switching to NEW-ENCODER and ticket 2; the second response wrote score 91 and marked B’s rerank completed.

  Suggested fix: Dispatch the completion-triggered rerun through scheduleRerankRef.current, so it uses the current closure. Preserve ownership of pending rerun flags across runs, and add a regression test covering this completion-after-restart sequence.

The pass-10 fix is fine for timers scheduled before a newer run starts; this finding concerns a timer newly scheduled by an old call’s completion.

All 191 targeted helper tests passed, but they don’t exercise this React closure race. No files were modified.

[paynejd]

Codex adversarial review, pass 12 (codex-cli 0.159.2, commit 7519657)

Scope: git diff origin/main...HEAD at 7519657, read-only; verify the pass-11 fix, then report only new problems of real consequence.

Outcome: clean, nothing to change.

---

No new, concrete findings of real consequence at 7519657.

Verified the pass-11 fix: rerank completion invokes scheduleRerankRef.current(index), using the current render’s scheduler.

Reviewed the diff and prior-pass dispositions; all 191 focused tests passed. Those tests validate the helpers, but don’t prove the full React lifecycle or externally loaded Bridge Match integration.

No files modified.

[paynejd]

Codex adversarial review of the capacity_aware commits, pass 1 (codex-cli 0.159.2, commit 43649ce)

These commits were added overnight in the integration test of this PR with OpenConceptLab/oclapi2#922 (OpenConceptLab/ocl_online#275). Reviewed: 43649ce (the capacity_aware flag). Posted verbatim; read-only.

1. Low — src/components/map-projects/MapProject.jsx:2311: bulk fallback to OCL remains unflagged. capacityAware: algo.type !== 'custom' differs from URL routing: getMatchAPIService falls back to OCL when a custom algorithm has no URL. A single-algorithm bulk run then sends reranker: true to OCL without declaring capacity awareness, bypassing enforce_for=aware. The single-row path correctly flags this fallback. Fix: use !(algo.type === 'custom' && algo.url) for bulk too, and add a regression test.

2. Low — src/services/__tests__/attribution.test.js:243: new tests verify serialization, but not call-site wiring. They would pass if a flag disappeared from any request, or attribution headers leaked onto a custom single-row URL. Failure scenario: future routing changes silently bypass enforcement or introduce CORS failures. Fix: test the built-in/custom URL matrix and flag propagation through bulk, single-row, bridge, and rerank requests.

No significant regression found in the normal configured paths.

What I checked and found correct:

• Bulk: built-in requests go to the selected OCL API; runMatchBatch waits out 429s and marks exhausted waits throttled (-4), not failed.
• Single-row: built-in requests use requestSingleMatch; throttled responses reach the callback, clear loading, and continue the row’s remaining algorithms. Custom algorithms with a URL receive neither new attribution header.
• Bridge: the local react-bridge-match source forwards the headers into service.post with semantic: true. Mapper’s overridden post retries capacity responses and returns a throttled body that the bridge failure callback handles. The script’s hosting origin does not determine the API request origin.
• Rerank: targets OCL’s $rerank/, waits through requestWithCapacityRetry, and handles exhaustion as throttled.
• CORS: local oclapi2 settings allow both attribution headers. Single-row requests already require preflight through existing headers. Bulk and bridge already sent attribution headers; adding a JSON field creates no new header requirement. Other API deployments must allow these headers; their deployed configuration was not verified.
• Analytics: single-row requests now deliberately carry mapper-ui-manual, project/row/algorithm metadata, and no run ID, including when overlapping a bulk run.
• Encoding: metadata contains the string "true", matching the supplied server contract. Flagging ungated lexical calls is harmless under that contract.
• CI: eslint src --ext .jsx,.js passed. npm run test:all passed 400 unit tests and 6 integration tests.

Reviewed only 43649ce over 7519657, using local source. No files modified; no network used.

[paynejd]

Codex adversarial review of the capacity_aware commits, pass 2 (codex-cli 0.159.2, commit 9d1bb52)

These commits were added overnight in the integration test of this PR with OpenConceptLab/oclapi2#922 (OpenConceptLab/ocl_online#275). Reviewed: 43649ce + 9d1bb52. Posted verbatim; read-only.

No significant findings in 7519657..9d1bb52. Both pass-1 findings are addressed.

Checked and found correct:

• Routing: matchesOnOCL() matches getMatchAPIService(). Custom algorithms without a URL receive the flag on bulk and single-row calls; custom algorithms with a URL do not.
• 429 handling: Bulk $match, single-row $match, bridge $match, and $rerank use rejecting requests with handlesThrottle: true and capacity retries. Exhausted waits reach throttled handling rather than failed-row or frozen-promise handling.
• Coverage: No additional gated OCL call site found unflagged among the reviewed paths. Flags on lexical-only OCL calls are harmless under the stated backend gates.
• CORS: Local oclapi2 settings allow both attribution headers and all origins. Custom single-row URLs receive neither new header. Bulk custom URLs retain their existing attribution headers. The local bridge implementation forwards headers through the retry wrapper.
• Analytics: Single-row OCL requests now correctly carry mapper-ui-manual, project, row, and algorithm metadata, without an automatch run ID.
• Serialization/tests: capacity_aware is the string "true". Tests cover opt-in, omission, manual attribution, and the custom-without-URL routing case. They test the helper rather than actual request wiring; that wiring was checked manually.
• CI: ESLint passed; npm run test:all passed 401 unit + 6 integration tests; diff whitespace checks passed.

No files modified or network used. CORS and bridge conclusions rely on local source, not deployed configuration.

[paynejd]

Codex adversarial review across both PRs (codex-cli 0.159.2): OpenConceptLab/oclapi2#922 at 2701b02e + #86 at 7519657

From the overnight integration test of the two PRs as a set (OpenConceptLab/ocl_online#275). Focus: the contract between them. Posted verbatim; read-only. Since then: finding 1 (the capacity_aware flag) is fixed in oclmap#86 43649ce + 9d1bb52, and finding 5 in oclapi2#922 2f399a12. Findings 2–4 are listed for OpenConceptLab/ocl_online#340, before enforcement.

Significant issues remain. The planned shadow rollout is compatible, but switching only runtime config to enforcement will not enforce against this Mapper under the default enforce_for=aware.

1. High — Enforcement prerequisite missing.
   oclmap:src/services/attribution.js:64; src/components/map-projects/MapProject.jsx:3811
   No capacity_aware occurs anywhere in Mapper’s src. The attribution builder omits it, and the interactive single-row $match path sends no attribution metadata at all.

   Failure: changing the API to mode=enforce, enforce_for=aware leaves all these Mapper calls effectively in shadow: they acquire leases even when lanes are full. Switching to all activates refusals, including for old tabs that cannot handle them safely.

   Fix: before #340 activation, merge "capacity_aware":"true" into metadata for every retry-capable heavy OCL call: bulk and interactive semantic $match, $match with inline reranking, and standalone $rerank. Preserve existing metadata; use a JSON string, not boolean true. Do not rely solely on modifying the builder—the interactive path must actually send its headers. This is an activation blocker, not a blocker for shipping shadow mode.

2. High — Ambiguous failures can duplicate heavy work and consumption.
   oclmap:src/services/capacity.js:37,260; oclapi2:core/concepts/views.py:1223
   Mapper retries network failures and 502/503/504 twice, without request deduplication. An 11-minute browser timeout does not guarantee that every earlier gateway failure means the original operation stopped.

   Failure: an ALB/gateway returns 504 or the connection breaks while Django continues searching. Mapper sends another $match; both execute and consume operations if successful. During Redis failure, both are admitted uncounted. Even when Gunicorn actually kills the worker at 600 seconds, the precharged operations have no guaranteed refund because process death bypasses the exception handler.

   Fix: add server-backed idempotency/deduplication for $match, including consumption and result recovery; otherwise avoid automatic retries after ambiguous execution failures. Validate the actual ALB timeout alongside Gunicorn’s timeout. The ALB configuration was unavailable locally.

3. Medium — The “per-tab” gate and rerank limit reset on project unmount.
   oclmap:src/components/map-projects/MapProject.jsx:236,361,1804
   Gates and the two-slot rerank limiter are component refs. Unmount does not cancel capacity waits or invalidate the run ticket; cancellation checks only Stop and ticket changes in that component.

   Failure: start work in project A, navigate to project B, then start another run. A’s waits and outstanding calls can continue, while B creates an independent gate and two more rerank slots. One tab can exceed two concurrent $rerank calls and ignore A’s server pause.

   Fix: keep server gates and the rerank limiter in a tab-scoped service. On unmount, cancel queued work and waits through a component/run cancellation token. Outstanding server work may finish, but must retain its shared limiter slot until settlement.

4. Medium — Capacity refusals consume DRF rate allowance; Mapper conflates the two 429s.
   oclapi2:core/concepts/views.py:952,1213; oclmap:src/services/capacity.js:242
   DRF throttle checks run before the view’s capacity gate. A request subsequently refused for capacity has already passed—and consumed—the rate allowance. Mapper treats every 429 identically, without inspecting error_code, scope, or rate-limit headers.

   Failure: at a five-second capacity delay, one waiting request can make roughly 288 attempts in 30 minutes with average jitter. Repeated runs or parallel calls can burn substantial portions of the standard 5,000/day $match allowance without matching anything. Eventually a DRF daily throttle replaces capacity refusals; Mapper still describes the condition as capacity throttling. A minute throttle also pauses unrelated heavy endpoints sharing the server gate.

   Fix: ensure capacity-refused calls do not retain DRF allowance consumption, using a concurrency-safe design. Classify error_code === "capacity_exceeded" separately from DRF throttles; retain bounded waiting for both, but show the correct reason and apply appropriately scoped pauses.

5. Low — paused can contradict the reason for the long retry delay.
   oclapi2:core/capacity/limiter.py:405,460
   Retry delay considers any full lane with limit zero, but scope and paused describe only the first full lane.

   Failure: the cluster lane is full and the preview tier is paused at zero. The response uses the paused delay of 120 seconds but reports scope:"api_heavy", paused:false. Mapper currently ignores these fields, so waiting works, but consumers cannot reliably distinguish a configured pause from temporary congestion.

   Fix: prioritize a zero-limit lane as the reported scope, or define paused as “any applicable lane is paused” and expose all blocking scopes.

What I checked and found correct:

• Headers/CORS: all seven emitted names match Mapper’s case-insensitive parser. Decision/tier are strings; counts, limits and suggested concurrency parse as numbers. Retry-After and all seven capacity headers are explicitly exposed through CORS; X-OCL-Event-Metadata is allowed for preflight. Mapper’s optional suggested-spacing header is absent on this API and safely omitted.
• Response coverage: gated 200s, capacity 429s and DRF-handled errors after acquisition receive capacity headers. Errors before acquisition—authentication, DRF throttling, $rerank input validation—do not. Off mode, ungated calls and preflight have no capacity decision headers. Proxy-generated errors cannot receive Django’s headers.
• 429 shape: detail, error_code, and scope are strings; paused is boolean; retry_after is an integer in seconds. Capacity refusal precedes capability consumption. DRF throttles have a different body and ordinarily no capacity headers.
• Retry parsing: numeric seconds and HTTP dates work; zero/past dates get a one-second minimum wait. Missing headers fall back to numeric body seconds, then exponential delays of 5–60 seconds. An invalid present header bypasses body fallback. Huge delays immediately end as throttled without poisoning the shared gate. Header-driven waits receive 0–50% jitter.
• Bounds and Stop: Auto Match accumulates up to 30 minutes of waiting, not total elapsed execution time. Single-row $match uses five minutes; interactive $rerank currently uses the default 30 minutes. Stop interrupts configured waits within a 250-ms poll and prevents further run retries; already executing server work continues.
• Leases: 60 seconds is a renewable lease, not a request duration limit. Healthy 20-second renewal keeps a long call counted. Renewal is atomic across all lanes and never revives expired leases. With default settings, Gunicorn kills an overlong sync request at roughly 600 seconds, before the 900-second renewal ceiling; its count then expires within 60 seconds of the last renewal. If a call survives beyond the renewal ceiling, or renewal is lost during a Redis outage, it eventually continues uncounted. Fail-open therefore provides availability, not a hard concurrency guarantee.
• Mixed versions: old Mapper + new shadow API remains compatible. Old tabs + aware enforcement remain admitted; old tabs + all can hang on their legacy 429 path. New Mapper + old API tolerates absent capacity headers and handles ordinary DRF 429s. During a mixed API rollout, old tasks perform uncounted work, so cluster limits cannot be authoritative until every task is upgraded. New workers also converge on runtime config over the cache interval, rather than instantly.
• Verification: confirmed both requested HEADs and ran 130 focused Mapper tests: all passed. API behavior was reviewed from source and existing tests; no live Django/Redis/ALB verification was performed. No files were modified and no network was used.

[paynejd]

Independent Claude review across both PRs (a read-only Claude subagent, run in the overnight integration test of OpenConceptLab/oclapi2#922 + #86, OpenConceptLab/ocl_online#275). Reviewed: oclapi2#922 at 2701b02e, oclmap#86 at 7519657. Posted verbatim. Since then: finding 1 is fixed in oclmap#86 (43649ce, 9d1bb52), and finding 7 in oclapi2#922 (040a6891: skip window 15 s). Findings 2–6 are listed for OpenConceptLab/ocl_online#340, before enforcement.

Cross-repo contract review: oclapi2 PR 922 and oclmap PR 86

I only read code. No files were changed and no APIs were called. Nothing here breaks the shadow-mode deploy. Findings 1–3 must be fixed before enforcement is turned on.

Findings

1. HIGH: the Mapper never sends capacity_aware, so enforcement can't reach it.

• Where: oclmap src/services/attribution.js:52-86 (searched all of src/, no capacity_aware anywhere); oclapi2 core/capacity/limiter.py:398-403 and config.py:26.
• Under enforce_for=aware (the default), Mapper calls are never refused. The cap never applies to the main heavy-call client.
• Under enforce_for=all, old Mapper tabs break:
  • On origin/main, $rerank goes through service.post. On a 429, sendRequest returns a promise that never settles and opens the full-screen throttle dialog, so runs freeze.
  • Old $match batches treat a 429 as not retryable and mark the rows failed (-2).
• So right now neither setting is usable.
• Fix: send "capacity_aware": "true" (a string, per the metadata convention) only on gated calls:
  • the bulk batch $match (MapProject.jsx:2309)
  • the single-row $match (MapProject.jsx:3811-3833). It currently sends no X-OCL-Event-Metadata at all, so the header has to be added.
  • the bridge $match (merge it into the component's headers in getBridgeMatchService, :2101)
  • $rerank (:4373)
• Don't send it on ScispaCy, logs or lookups. Also add the key to event-metadata-vocabulary.md.

2. HIGH (only once enforced): the Mapper runs more calls at once than the server allows per user, and the retrying batch starves.

• Where: oclmap requestLimits.js:20-22 and autoMatchRows.js RERANK_MAX_IN_FLIGHT=2, against oclapi2 config.py:34 (per_user: preview 1, early access 2, core 3).
• A preview user runs 2 $match batches plus 2 $rerank calls against a limit of 1. Early access runs 5+2 against 2.
• On a 429, capacity.js:248-256 pauses the gate for exactly Retry-After, but the refused request sleeps Retry-After × 1–1.5.
• runWithConcurrency (matchBatch.js:134-147) sends the next queued batch the moment the gate reopens, with no jitter. So a new batch always takes the freed slot first.
• Failure scenario: early-access run, 3 batches stuck retrying.
  • Each gets a 429 every ~6 s until the queue drains, or until its 30-minute cap runs out.
  • At the cap the batch ends throttled. isRunThrottled(algo.id) then marks every remaining row throttled, even though the server was serving the user the whole time.
• Server side: a refused call takes no lease, so a full lane's count never goes above its limit. ahead (limiter.py:409) is therefore always 1 and Retry-After is always 5 s, however many clients are waiting.
• Fix, before enforcing:
  • Cap heavy calls in flight per tab at X-OCL-Capacity-Suggested-Concurrency (ocl_online#340), or line the per_user defaults up with the Mapper's caps.
  • Let a refused request retry ahead of new sends: the scheduler should wait the pause plus the maximum jitter, or use a FIFO slot per server.

3. MEDIUM: a capacity 429 and a DRF throttle 429 look the same to the Mapper, and capacity retries use up DRF quota.

• Where: capacity.js:242 (any 429), APIService.js:172.
• $match uses DRF throttles (views.py:952; match_standard_day is 5000/day).
• DRF counts each attempt in initial(), before the gate in post(). Every capacity-refused retry therefore uses one daily hit: about 1,700 per hour with three starving batches (finding 2).
• When the daily limit is reached, the Retry-After is hours long, which is over the cap. Every batch then ends "throttled — server busy" at once.
• handlesThrottle also hides the old countdown dialog, so the user is told the server is busy when they actually hit a rate limit.
• Fix:
  • Mapper: branch on data.error_code === 'capacity_exceeded', and show any other 429 as a rate limit.
  • Server: don't count calls the capacity limit refuses against the DRF throttles (gate before the throttle check, or refund the hit).

4. MEDIUM: heavy calls are re-sent after long failures.

• Where: capacity.js:32-40, retry.js:6-10.
• When gunicorn kills a worker at 600 s, the ALB returns a 502. The Mapper re-sends twice more, which can mean up to 3 × 600 s of worker time.
• Each attempt is charged match units, and nothing is refunded when the worker is killed.
• The 11-minute axios timeout doesn't count time spent queued at the ALB or in the backlog. The resulting ECONNABORTED (no response) is treated as transient, so the call is re-sent while the server is still working on it. An ALB idle timeout under 11 minutes causes the same with 504s.
• Fix: retry 502/504/timeouts only when the attempt failed quickly (say under 30 s, which is the deploy/restart case), or send an idempotency key.

5. MEDIUM-LOW: one gate per server ignores scope.

• Where: MapProject.jsx:1749-1760, capacity.js:213-233.
• A bulk batch's user or es2_knn 429 also holds back:
  • lexical $match, which the server doesn't gate
  • the row panel's single-row $match, which the server reserves a cluster slot for (reserve_single_row)
  • $rerank
• Interactive calls end up waiting behind bulk pauses.
• Fix: pause by the 429 body's scope, or let single-row calls skip the gate's pause.

6. LOW: a manual $rerank waits up to 30 minutes and can't be stopped, while holding a rerank slot.

• Where: MapProject.jsx:4261, 4351, 4370-4378. There's no maxWaitMs, and isCancelled is always false when it isn't run traffic.
• Two manual reranks waiting block every run rerank for up to 30 minutes.
• Fix: use INTERACTIVE_WAIT_CAP_MS for manual reranks, and wait on the gate before taking a limiter slot.

7. LOW: a single Redis error can lose a long call's lease.

• Where: limiter.py:197-203, 424-431, CAPACITY_REDIS_RETRY_SECONDS=30.
• A renewal fails at T+20, so the process skips Redis until about T+51 and the T+40 renewal is skipped. The T+60 renewal arrives at expiry, the lease is lost, and the renewer exits.
• The call goes uncounted for its remaining minutes. This fails open, but it skews the counts.
• Fix: make the skip window shorter than lease − 2×renew, or let renewals retry sooner.

8. NIT: the Mapper parses x-ocl-capacity-suggested-spacing-ms (capacity.js:88). The server neither sends nor exposes it.

Checked and found correct

• Header names: all 7 X-OCL-Capacity-* headers plus Retry-After are in CORS_EXPOSE_HEADERS. axios 1.x header lookup ignores case, and the Mapper's lowercase names resolve.
• Which responses carry them: every gated response, including 4xx/5xx raised inside the gate (finalize_response). DRF throttle 429s, ungated calls and mode=off carry none. Retry-After is only set on real refusals.
• 429 body: {detail: str, error_code: "capacity_exceeded", scope: str, paused: bool, retry_after: int}. The refusal comes before any quota charge.
• Retry-After parsing: 0 waits at least 1 s; missing waits 5 s, doubling to 60 s; HTTP-dates work; negative values fall back to the default; very large values end the request throttled without pausing the gate; the body's retry_after is used when the header is missing.
• No gated Mapper path still uses the never-settling sendRequest: bulk, single-row and bridge $match and $rerank all go through request + handlesThrottle + requestWithCapacityRetry. A 429 ends rows -4 (throttled), not -2 (failed). A 500 is an error; 502/503/504 get bounded retries.
• Stop: checked every 250 ms in waits, in the scheduler (untilCancelled) and in the limiter queue.
• Fail-open: Redis errors mean no 429s, so the Mapper can't get into a retry storm. Enforce falls back to shadow if the config can't be read.
• Leases vs the 600 s gunicorn timeout: a killed worker's leases free within 60 s, max_hold (900 s) never applies, and renew/release run in order on the one Redis thread.
• Mixed versions:
  • Old Mapper + new API in shadow: no change in behaviour.
  • New Mapper + old API: works with no capacity headers.
  • Old and new API tasks mid-deploy: only the counts are low for that window (old tasks don't count). Config compatibility holds both ways (merge(known_only)), and a missing migration falls back to the defaults.

[paynejd]

Overnight integration test: OpenConceptLab/oclapi2#922 + #86 (2026-10-01)

The two PRs were tested together on a local production-like stack:

• 2 API containers with 4 gunicorn workers each, behind a round-robin proxy;
• Redis behind Sentinel;
• the real embedding and rerank models;
• production builds of this Mapper and of today's main, on their own origins, with headless browsers per user.

No merges or deploys. Every decisive row was re-run at the final heads (oclapi2#922 c5282f4a, oclmap#86 9d1bb52; CI green on both).

Compatibility (decides the deploy order): both directions work.

• This Mapper with today's API: Auto Match results are the same as with the new API (25-row and 100-row two-algorithm runs). Saved projects, closed run records, no alerts.
• Today's Mapper with the new API in shadow mode: results identical to today's API, nothing refused, capacity headers present, CORS correct.
• A rolling API deploy (old → new, one container at a time) during a run of today's Mapper: run completed, same results.

The new API in shadow mode:

• Off mode returns identical responses and quota usage to master.
• Overhead isn't measurable at this load.
• No lease leaks after Stop, client aborts, a killed worker or a container restart.
• Workers recycle cleanly.
• The runtime off switch takes effect within the 10 s config cache under traffic.
• Fail-open holds through a Redis hang and an unclean master loss: runs finish with the same results. Calls during the outage are uncounted.

This Mapper against real capacity 429s (enforce mode, which ships off):

• Runs wait out refusals and finish with the same results and no failed rows, including with 4 users at once.
• Stop works during a wait; a paused tier shows "Waiting for capacity".
• With both APIs down, rows in flight end as "Matching failed … Run Auto Match on those rows again", never as "no candidates".

Commits added overnight:

• oclmap#86:
  • 43649ce sends "capacity_aware": "true" on OCL's gated calls. Without it, enforcement could never reach this Mapper.
  • 9d1bb52 adds one tested rule for which calls go to OCL.
  • Codex reviewed both: two findings fixed, then clean.
• oclapi2#922:
  • 2701b02e: an acquire that runs after its caller gave up takes no lease (found by this test).
  • 2f399a12: the 429 scope names the paused lane.
  • 040a6891: a 15 s Redis skip window, so one error can't cost a long call its lease.

Before enforcement (OpenConceptLab/ocl_online#340), from the test and the reviews:

• The single-row reserve should cover the per-task and kNN lanes too: interactive calls were refused about half the time under saturation.
• Capacity refusals shouldn't use up DRF throttle allowance. The Mapper should tell the two 429s apart: today it shows "Waiting for capacity" for a rate limit as well.
• The Mapper's concurrency vs per-user limits, and fair retry order.
• The per-tab gate across project switches.
• Interactive calls waiting behind bulk pauses.
• Retrying ambiguous failures after long calls.
• Retrying skipped releases once Redis is back.

The full readout, reviews and evidence are on OpenConceptLab/ocl_online#275.