perf(memory-storage): make forefront request re-add O(1)

[anxkhn]

MemoryRequestQueueClient.add_batch_of_requests repositioned an already-pending
request to the forefront by calling self._pending_requests.remove(existing_request)
on the pending deque. deque.remove is O(n): it linearly scans the deque to find
the entry. Re-adding a batch of K already-pending requests to a pending deque of N
entries is therefore O(K*N).

Forefront re-adds of still-pending requests are a normal runtime path, not an edge
case:

• RequestManagerTandem.fetch_next_request re-enqueues every request it takes from
  the read-only loader with forefront=True
  (src/crawlee/request_loaders/_request_manager_tandem.py).
• Tiered-proxy retries set request.forefront = True before re-adding
  (src/crawlee/proxy_configuration.py).

This change removes the linear scan:

• The forefront reposition now just appendlefts the request and leaves the old
  entry in the deque. The new object is registered in _requests_by_unique_key,
  superseding the old one.
• fetch_next_request and is_empty skip any popped entry whose object is no longer
  the one currently registered for its unique_key (a stale tombstone). Because a
  forefront reposition always enqueues the live copy ahead of the entry it supersedes,
  is_empty can safely prune stale entries from the front until it reaches a live one.
• A regular (non-forefront) re-add of an already-pending request is left as a true
  no-op: neither the deque entry nor its registered object is touched, so the identity
  check never misfires and no request is dropped. This matches the sibling file-system
  client, which likewise does not re-store an already-pending request on a
  non-forefront re-add.

Forefront-LIFO / regular-FIFO ordering, deduplication, and the metadata counts
(total/pending/handled) are unchanged.

This is the memory-client counterpart to #2011, which fixes the same class of
quadratic reposition behavior in the file-system request queue client. The memory
client was not covered there.

Issues

• Related: perf(file-system): Fix quadratic file rescans in the FS request queue #2011 (same quadratic-reposition fix for the file-system client)

Testing

Added regression tests in
tests/unit/storage_clients/_memory/test_memory_rq_client.py:

• test_forefront_readd_repositions_without_deque_scan - wraps the pending deque in a
  deque subclass that counts remove calls; re-adding 20 still-pending requests with
  forefront=True now performs 0 remove calls (was 20 before the change). Guards the
  performance fix.
• test_forefront_readd_preserves_order_and_dedup - forefront re-add of a subset keeps
  LIFO order, no duplicates linger, and the queue drains to empty/finished.
• test_regular_readd_of_pending_request_is_not_dropped - a non-forefront re-add of a
  still-pending URL as a distinct Request object stays fetchable exactly once, with
  consistent counts (guards against the lazy-tombstone skip dropping a live request).
• test_regular_readd_does_not_reorder_pending_queue - a non-forefront re-add leaves
  FIFO order untouched.

Commands run locally (offline):

uv run pytest tests/unit/storage_clients/_memory/test_memory_rq_client.py -v   # 7 passed
uv run pytest tests/unit/storages/test_request_queue.py tests/unit/storage_clients/ \
  tests/unit/storages/test_request_manager_tandem.py                            # 404 passed, 2 skipped
uv run ruff check <changed files>                                              # clean
uv run ruff format --check <changed files>                                     # clean
uv run poe type-check                                                          # clean on changed files

Checklist

• CI passed

[Copilot]

Pull request overview

This PR optimizes the in-memory RequestQueue client by eliminating the O(n) deque.remove scan when re-adding already-pending requests with forefront=True, replacing it with an appendleft + “lazy stale entry” skipping approach in fetch_next_request() / is_empty().

Changes:

• Reworked MemoryRequestQueueClient.add_batch_of_requests() so forefront re-adds of pending requests no longer scan/remove from the pending deque.
• Added stale-entry skipping in fetch_next_request() and front-pruning in is_empty() to support the new “tombstone” behavior.
• Added regression tests to guard the performance fix and ordering/dedup semantics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/crawlee/storage_clients/_memory/_request_queue_client.py	Implements tombstone-based forefront repositioning and stale-entry skipping/pruning logic.
tests/unit/storage_clients/_memory/test_memory_rq_client.py	Adds regression tests validating no deque.remove usage and preserves order/dedup semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[anxkhn]

pushed a revision addressing the latest review and ci feedback.

[vdusek]

@anxkhn Thank you. For future reviews, I'd really appreciate it if you could address all the feedback in a single follow-up commit, rather than splitting the changes across multiple commits, merging them into earlier commits, and force-pushing it. Your current approach makes the changes really hard to review.

[anxkhn]

understood, i’ll keep future review fixes in a single follow-up commit without rewriting earlier commits.

[vdusek]

Thanks. It's much cleaner after the rewrite. Two things left.

[anxkhn]

@vdusek addressed both new threads, rebased, and re-requesting review. thank you.

[vdusek]

Summary:

• Store pending requests in an OrderedDict keyed by unique key instead of a deque. Moving a request to the forefront is now O(1) instead of an O(n) deque.remove() scan, so re-adding K already-pending requests is O(K) instead of O(K*N). This is a hot path: RequestManagerTandem re-adds every request taken from the read-only loader with forefront=True, and tiered-proxy retries do the same.
• Fix the same URL being crawled twice. The old scan silently failed whenever an earlier regular re-add had replaced the registered request object, leaving a stale entry behind and appending a second one for the same unique key. Duplicates are now impossible by construction.
• Keep the originally enqueued request on a re-add. A duplicate discovered later is typically a freshly built object and used to overwrite the queued one, resetting retry_count (a request could retry forever), label (dispatch to a different handler) and user_data. This matches the file-system, SQL and Redis clients.
• Make get_request consistent after a reclaim: it now returns the reclaimed object, the same one fetch_next_request hands back.
• Write the queue metadata once per batch instead of once per new request, which also makes a batch add atomic with respect to the event loop.

[vdusek]

@Mantisus could you please check this as well?