gh-154194: Degrade frames in Tachyon instead of failing the sample by maurycy · Pull Request #154195 · python/cpython

maurycy · 2026-07-19T20:01:33Z

As always, see #154194 for the context. Shortly: right now remote inspection fails the whole sample when it hits the limits (and we're really not sure what are the best limits) OR always when reading a part fails. This is too strict, wasteful and introduces scary bias, as observed in https://discuss.python.org/t/tachyon-97-error-rate/107619.

The PR introduces graceful degradation with the sentinels like <unreadable frame> (funcname), <unknown function> (funcname), <unknown file> (filename). Also, attemps to better differentiate between transient failures in _Py_RemoteDebug_IsFatalReadError().

Importantly: degraded frames are never cached, while the truncated strings are cached.

Issue: _remote_debugging: Degrade gracefully reads instead of failing the whole read completely #154194

maurycy · 2026-07-19T21:26:53Z

I believe that the CI failures are caused by the fact that test_async_aware_running_sees_only_cpu_task does not terminate. It worked only because a torn read caused an exception. That's what inspired #153365 in the first place.

pablogsal

I still need more time but some nits

maurycy · 2026-07-24T22:37:41Z

            # Build coroutine stack string
            frames = [frame for coro in task_info.coroutine_stack
                     for frame in coro.call_stack]
-            coro_stack = " -> ".join(_format_stack_entry(x).split(" ")[0]


This does not work for the <unknown function> sentinel.

maurycy · 2026-07-24T22:38:27Z

 def _format_stack_entry(elem: str|FrameInfo) -> str:
    if not isinstance(elem, str):


I believe that isinstance(elem, str) is dead since #135436. Perhaps we should clean this defensive code everywhere.

maurycy · 2026-07-24T22:43:45Z

 # ─── indexing helpers ───────────────────────────────────────────
 def _format_stack_entry(elem: str|FrameInfo) -> str:
    if not isinstance(elem, str):
+        if elem.location is None:


Regular collectors rely on these helpers:

cpython/Lib/profiling/sampling/collector.py

Lines 46 to 47 in 40f7fbf

if location is None:

return 0

cpython/Lib/profiling/sampling/collector.py

Lines 29 to 30 in 40f7fbf

if location is None:

return DEFAULT_LOCATION

maurycy · 2026-07-24T23:03:59Z

    PyObject *func = NULL;
    PyObject *file = NULL;
    PyObject *linetable = NULL;
+    int code_metadata_incomplete = 0;


Perhaps parse_code_object() should return this information, and we should not store incomplete objects in the frame_cache?

Perhaps we should collect degradation stats? Important caveat is that we will likely have more types of degradation.

maurycy · 2026-07-24T23:13:56Z

+                "<GC>",
+                "<native>",
+                "<unknown function>",
+                "<unknown file>",
+                "<unreadable frame>",


Perhaps we could add something like is_synthetic to FrameInfo?

…cefully

maurycy · 2026-10-04T19:53:41Z

@pablogsal I've synced with main, and removed the (len=) suffix. The #157790 did not include it. #153365 still needs to go first, because of #154195 (comment)

pablogsal

LGTM!

miss-islington-app · 2026-10-05T09:19:59Z

Thanks @maurycy for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

miss-islington-app · 2026-10-05T09:20:07Z

Sorry, @maurycy and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 7d25916b41de4fc28e04b4944864ad1094a9e8e0 3.15

bedevere-app · 2026-10-05T09:24:16Z

GH-158831 is a backport of this pull request to the 3.15 branch.

bedevere-app · 2026-10-05T09:26:40Z

GH-158833 is a backport of this pull request to the 3.14 branch.

…mple (GH-154195) (#158831) * gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195) * degrade gracefully * news * better NEWS wording * do not raise on MAX_REMOTE_STR_READ * bye MAX_REMOTE_STR_READ * fix -m asyncio ps|pstree * test truncation and linetable sentinel * simpler * simpler * redundant now * respect #157790 in the news --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 7d25916) * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>

…4195) (#158833) * gh-154194: Degrade unreadable frames on 3.14 (cherry picked from commit 7d25916) Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com> * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>

degrade gracefully

8aaf6f0

bedevere-app Bot mentioned this pull request Jul 19, 2026

_remote_debugging: Degrade gracefully reads instead of failing the whole read completely #154194

Open

maurycy mentioned this pull request Jul 19, 2026

tachyon: degrade reads instead of failing the whole read completely maurycy/cpython#7

Closed

news

f45ef66

maurycy marked this pull request as ready for review July 19, 2026 20:17

maurycy requested a review from pablogsal as a code owner July 19, 2026 20:17

bedevere-app Bot added the awaiting review label Jul 19, 2026

pablogsal reviewed Jul 20, 2026

View reviewed changes

Comment thread Modules/_remote_debugging/object_reading.c Outdated

Comment thread Misc/NEWS.d/next/Library/2026-07-19-22-08-05.gh-issue-154194.1bqRdx.rst Outdated

better NEWS wording

ad797d5

maurycy mentioned this pull request Jul 21, 2026

gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded #153365

Merged

maurycy changed the title ~~gh-154194: Do not fail the whole sample in Tachyon if we can degrade~~ gh-154194: Degrade frames in Tachyon instead of failing the sample Jul 21, 2026

maurycy added 4 commits July 24, 2026 23:06

do not raise on MAX_REMOTE_STR_READ

3e58d13

bye MAX_REMOTE_STR_READ

55632e9

fix -m asyncio ps|pstree

6736133

test truncation and linetable sentinel

17b4197

maurycy requested review from 1st1, asvetlov, kumaraditya303 and willingc as code owners July 24, 2026 22:18

maurycy commented Jul 24, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into tachyon-degrade-gra…

8213bd3

…cefully

maurycy mentioned this pull request Oct 4, 2026

gh-157227: _remote_debugging: Validate lists and tuples in write_sample #157228

Open

maurycy and others added 5 commits October 4, 2026 22:21

simpler

443f5c1

simpler

bb28257

redundant now

be34b1f

respect python#157790 in the news

2a88472

pythongh-154194: Integrate bounded walks and simplify degradation tests

c6a44a8

pablogsal approved these changes Oct 5, 2026

View reviewed changes

pablogsal enabled auto-merge (squash) October 5, 2026 00:09

bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 5, 2026

pablogsal merged commit 7d25916 into python:main Oct 5, 2026
57 of 58 checks passed

bedevere-app Bot removed the awaiting merge label Oct 5, 2026

maurycy deleted the tachyon-degrade-gracefully branch October 5, 2026 04:28

pablogsal added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Oct 5, 2026

miss-islington-app Bot assigned pablogsal Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) #158831

Merged

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.14] gh-154194: Degrade frames instead of failing the sample (GH-154195) #158833

Merged

		def _format_stack_entry(elem: str\|FrameInfo) -> str:
		if not isinstance(elem, str):

Uh oh!

Conversation

maurycy commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

maurycy commented Jul 19, 2026 •

edited

Loading

maurycy commented Jul 19, 2026 •

edited

Loading

maurycy Jul 24, 2026 •

edited

Loading

maurycy Jul 24, 2026 •

edited

Loading

maurycy commented Oct 4, 2026 •

edited

Loading