fix(system): do not fail memory snapshots on inaccessible processes

[anxkhn]

get_memory_info() in src/crawlee/_utils/system.py can crash with psutil.AccessDenied in restricted environments, which silently stops the autoscaler's CPU/memory snapshots.

• On Linux, _get_used_memory reads a process's PSS via process.memory_full_info() (/proc/<pid>/smaps). psutil documents that memory_full_info / PSS may require elevated privileges, so in a hardened container (hidepid, restricted /proc, seccomp) or for an unreadable child subprocess it raises psutil.AccessDenied.
• psutil.AccessDenied is not a subclass of psutil.NoSuchProcess (MRO: AccessDenied -> Error -> Exception), so the existing with suppress(psutil.NoSuchProcess) around the child loop did not catch it, and the current-process read at the top of the function was not guarded at all.
• When it propagates, it escapes the recurring SYSTEM_INFO task (RecurringTask._wrapper has no error handling), so the recurring task stops and the autoscaler quietly stops receiving snapshots.

Fix, kept minimal:

-    # Retrieve estimated memory usage of the current process.
-    current_size_bytes = _get_used_memory(current_process)
+    # Retrieve estimated memory usage of the current process. On Linux `_get_used_memory` reads PSS via
+    # `memory_full_info`, which can raise `AccessDenied` in restricted environments (e.g. hardened containers);
+    # fall back to RSS, which a process can always read for itself.
+    try:
+        current_size_bytes = _get_used_memory(current_process)
+    except psutil.AccessDenied:
+        current_size_bytes = int(current_process.memory_info().rss)

     # Sum memory usage by all children processes, try to exclude shared memory from the sum if allowed by OS.
     for child in current_process.children(recursive=True):
-        # Ignore any NoSuchProcess exception that might occur if a child process ends before we retrieve
-        # its memory usage.
-        with suppress(psutil.NoSuchProcess):
+        # Ignore a child that ends before we retrieve its memory usage (`NoSuchProcess`) or that we are not
+        # allowed to inspect (`AccessDenied`, e.g. an unreadable subprocess in a restricted environment).
+        with suppress(psutil.NoSuchProcess, psutil.AccessDenied):
             current_size_bytes += _get_used_memory(child)

A process can always read its own RSS, so the current-process fallback keeps a valid (slightly higher, RSS vs PSS) estimate instead of aborting the whole snapshot; an unreadable child is skipped just like a dead one.

Issues

• No existing GitHub issue. This is a self-identified defensive fix found while reviewing get_memory_info for cross-platform robustness (in the same spirit as the earlier PSS-on-Linux and macOS memory-estimation fixes to this file). Happy to open a tracking issue with the repro first if you'd prefer.
• Related but not a duplicate: fix: prevent silent RecurringTask death and swallowed timeouts in AutoscaledPool #2009 hardens the generic recurring-task pathway so a failing task logs and continues instead of dying silently. This PR fixes the root cause so get_memory_info does not raise AccessDenied in the first place (no file overlap; the two are complementary).

Testing

• Added two tests to tests/unit/_utils/test_system.py:
  • test_get_memory_info_skips_children_with_access_denied: a child that raises AccessDenied is skipped rather than aborting the snapshot.
  • test_get_memory_info_falls_back_to_rss_when_current_process_access_denied: PSS denial for the current process falls back to RSS.
• Verified fail-first: reverting only src/crawlee/_utils/system.py (keeping the new tests) makes both tests fail with psutil.AccessDenied propagating out of get_memory_info(); they pass with the fix.
• uv run pytest tests/unit/_utils/test_system.py -> 4 passed, 1 skipped (the skip is the Linux-only shared-memory test).
• Ran the downstream consumers too: uv run pytest tests/unit/events/ tests/unit/_autoscaling/ -> green.
• uv run poe lint (ruff format + check) and uv run ty check on the changed files both pass.

Checklist

• CI passed

[Copilot]

Pull request overview

This PR hardens get_memory_info() (used for autoscaler CPU/memory snapshots) against psutil.AccessDenied in restricted environments, preventing the periodic system-info task from crashing and silently stopping.

Changes:

• Catch psutil.AccessDenied when reading the current process’s memory via PSS and fall back to RSS.
• Suppress psutil.AccessDenied when aggregating child process memory usage.
• Add unit tests covering both the child-skip and current-process RSS fallback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/crawlee/_utils/system.py	Adds AccessDenied handling/fallbacks in memory snapshot collection.
tests/unit/_utils/test_system.py	Adds regression tests for AccessDenied scenarios in get_memory_info().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[anxkhn]

pushed a revision addressing the latest review and ci feedback.

[vdusek]

Same as here #2074 (comment)

@anxkhn Thank you. For future reviews, I'd really appreciate it if you could address all the feedback in a single follow-up commit, rather than splitting the changes across multiple commits, merging them into earlier commits, and force-pushing it. Your current approach makes the changes really hard to review.

[vdusek]

Thanks for the revision. Both of my earlier comments are correctly addressed - the logger.debug for the PSS->RSS fallback, and the child RSS fallback so an unreadable child is no longer dropped.

However, the change made for the Copilot comment about guarding children() introduces a regression: wrapping the entire loop in suppress means one child dying mid-iteration now aborts the loop and drops all remaining children. I verified this locally - a repro with a dead first child and a live second child worth 40 B yields 140 B on master and 100 B on this branch. CI does not catch it because that path is untested.

Details inline.

✍️ Drafted by Claude Code

[anxkhn]

@vdusek addressed all new threads, including the child-exit regression test, rebased, and re-requesting review. thank you.

[vdusek]

Summary:

• get_memory_info() no longer raises when a process cannot be inspected. In restricted environments (hidepid, hardened containers) it failed on every sample, which meant a traceback in the log every second and an autoscaler running on a stale memory history. Processes that deny inspection, exit mid-measurement, or lose their /proc entry are now skipped instead.
• Failure to list the child processes no longer aborts the snapshot either - the parent's own usage is still reported.
• Every degraded path warns once, so an estimate that misses a subprocess or falls back from PSS to RSS shows up in the log instead of being silently wrong.
• PSS is asked for only while the system exposes it: a machine without /proc/<pid>/smaps is detected once, not re-probed for every process on every sample. A single process denying PSS falls back to RSS just for itself.
• current_size is capped at the machine's total memory. Summed RSS counts shared memory repeatedly, so a browser process tree could report more than the machine has and leave the autoscaler permanently throttled as "critically overloaded".
• LoggerOnce is now thread-safe, since memory metrics are sampled in a worker thread.
• Tests cover the denied, exited, and missing-/proc cases, the PSS fallback and its warnings, the cap, and concurrent LoggerOnce calls.

[vdusek]

@Mantisus could you please check this as well?