Refine L3 promotion criteria for downstream repositories by fffrog · Pull Request #102 · pytorch/rfcs

fffrog · 2026-07-22T11:53:05Z

Summary

This PR clarifies the L3 promotion criteria for downstream repositories by:

Defining infrastructure metrics more precisely (queue time, execution time, timeout rate).
Clarifying that the pass rate refers to CI job success rate, not individual test pass rate..

fffrog · 2026-07-22T11:53:46Z

Hi @albanD @atalman, please help to take a look at this when you have a chance.

Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).

Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.

atalman · 2026-08-03T19:35:16Z

 > \[!NOTE\]
-> - The requirements above are an **initial reference** and may **be adjusted over time based on real-world conditions** (e.g., determining the specific values of `X`).
+> - The requirements above are an **initial reference** and may **be adjusted over time based on real-world conditions**.
 > - To maintain the PyTorch community's user experience, **downstream repos that no longer meet the requirements of their current level will be downgraded to the level that matches their actual status.**


HI @fffrog maybe we should also callout de-motion Criteria here. Also would be nice to document temporary downgrade process.

Another point to consider, and maybe add here is that we probably want to add possibility of temporary downgrades for certain repos. For example when dealing with SEV's.

Agree on that. We can also write a script to automatically upgrade or downgrade the repos so that we don't need to waste our efforts tracking whether a repo meets the criteria. WDYT?

Hi @atalman, thank you for those valuable suggestions, and I have added new sections named Promotion and Demotion, especially, add a new mechanism to achieve the temporary downgrade.

@can-gaa-hou Thank you.

Sure, based on the new mechanism of temporary downgrade, we can prepare a script to collect the necessary info to determine whether some repo should be downgraded.

Replace the "Flaky Jobs" stat card with a "Timeout Rate" percentage card on the CRCR downstream repo page, aligning with the L3 promotion criteria proposed in pytorch/rfcs#102 which defines timeout rate < 10% as an infrastructure metric. The timeout rate is computed as timed_out / total_jobs directly in the ClickHouse query, replacing the complex flaky-job subquery. The card shows the percentage with color coding: green (0%), orange (>0%), red (≥10% — exceeds L3 threshold).

Update the Pass Rate card color coding to match the L3 promotion criteria from pytorch/rfcs#102 (job pass rate > 90%): - Green: 100% (perfect) - Orange: 90–99.9% (meets L3 threshold) - Red: < 90% (below L3 threshold) Previously the thresholds were ≥95% green, ≥80% orange, <80% red.

…colors with L3 criteria (#8421) ## Summary Two changes to the CRCR per-repo dashboard page (`/crcr/{org}/{repo}`), aligning stat card thresholds with the L3 promotion criteria proposed in [pytorch/rfcs#102](pytorch/rfcs#102). ### 1. Replace Flaky Jobs card → Timeout Rate **ClickHouse query** (`crcr_backend_summary/query.sql`): - Removed the complex flaky-job subquery (correlated subquery scanning the table twice) - Added `timeout_rate = timed_out / total_jobs` **Frontend card**: - Shows percentage (e.g., `0.0%`, `1.2%`) - Sub-text: `X timed out / Y jobs` - Color: green (0%), orange (>0% but <10%), red (≥10% — exceeds L3 threshold) ### 2. Align Pass Rate color thresholds Updated to match L3 criteria (`job pass rate > 90%`): - **Green**: 100% - **Orange**: 90–99.9% (meets L3) - **Red**: < 90% (below L3) Previously: ≥95% green, ≥80% orange, <80% red. ## Context The L3 criteria in [pytorch/rfcs#102](pytorch/rfcs#102) defines: - **Timeout rate < 10%** as an infrastructure reliability signal - **Job pass rate > 90%** as a test quality signal These cards make L3 readiness directly visible on the per-repo dashboard. ## Test plan - [ ] Verify `crcr_backend_summary` query returns `timeout_rate` correctly - [ ] Verify Timeout Rate card renders with proper color coding - [ ] Verify Pass Rate card shows red when < 90%, orange when 90–99.9%, green at 100%

can-gaa-hou · 2026-08-05T03:13:41Z

+| Metric | Target | Description |
+| :--- | :--- | :--- |
+| Max execution time | < 4 h | The longest "run" phase of any single job (excludes queue wait). |
+| Avg queue time | < 30 min | Average time a job waits before a runner picks it up (excludes execution). |


Question here: for most backends, they need to build PyTorch first, then they can run their first test, which means most of their test queue time includes PyTorch building time. So the average queue time definitely exceeds 30 mins. cc @fffrog @albanD @atalman

Thank you for asking this question. I've already sought help in this channel (https://pytorch.slack.com/archives/C0BBJK4B1EJ/p1785912202488279).

Additionally, I'd like to share my personal thoughts.

I believe that avg queue time here is just a reference value, but it is indeed a necessary metric. So, what we need to discuss is how to calculate the correct value, right?

fffrog · 2026-08-05T10:58:03Z

Hi @albanD @atalman @can-gaa-hou , the new commit is ready, please help to take a look at it when you have a time. thank you in advance.

albanD

Looks pretty good. Mostly small comments to tune numbers and be more precise.

albanD · 2026-08-11T16:31:13Z

+Formal demotion is a permanent level reduction triggered when a downstream repo continuously fails to meet its current level's [demotion criteria](#criteria-for-every-level):
+
+1. **Collect data**: The PyTorch CI Maintainer checks HUD data against the current level's demotion thresholds to confirm whether the repo has triggered a demotion condition.
+2. **Notify downstream**: Notify the downstream repo maintainers via email or other established channels that their repo has triggered demotion, including the specific metrics and a remediation deadline (4 weeks).


4 weeks is way too long here. If it has impact on trunk and live PRs, I would expect a couple days.
I think 2 working days is ok.

albanD · 2026-08-11T16:33:28Z

+     - **Formal config**: synced from the `allowlist.yml` file in the PyTorch repo (Redis formal cache)
+   - **The temporary cache takes priority over the formal cache**: if an unexpired override record exists for a repo in the temporary cache, it wins; otherwise the formal cache is used.
+3. **Restoration**: There are two ways to restore the original level:
+   - **Automatic**: The override record expires from the temporary cache, and the repo's level automatically falls back to the formal config.


How long is that override for?

albanD · 2026-08-11T16:33:35Z

+
+Temporary downgrade is an **operational tool** for exceptional situations (e.g., an upstream PyTorch CI SEV, or a downstream repo causing large-scale disruption to the relay system). It must take effect **immediately** without modifying config files, and be **reversible** after the incident:
+
+1. **Admin page**: HUD provides an admin page for PyTorch CI Maintainers to perform temporary downgrade operations.


Does that page already exist?

albanD · 2026-08-11T16:34:08Z

+L1 is the experimental onboarding phase. Requirements:
+
+1. The GitHub App must be installed.
+2. Provide verifiable accelerator hardware information.


Not sure what this one is about? Maybe drop it?

albanD · 2026-08-11T16:36:16Z

+
+| Metric | Target | Description |
+| :--- | :--- | :--- |
+| end-to-end time | < 4.5 h | The longest wall-clock time of a single job, from webhook delivery to CI status report. |


@georgehong does that match our current TTS goal for the trunk job?

We'd like to reach a sub 3-hour P50 TTS for trunk, and this encompasses build to the longest test shard. Over the past seven days, this metric was hovering around 3.4 hours. 4.5 definitely exceeds that by quite a bit.

Ok, let's aim for 3 here then!

albanD · 2026-08-11T16:37:06Z

+| Timeout rate | < 1% | Percentage of jobs terminated due to timeout, measured over a 7-day window. |
+
+> [!NOTE]
+> Average queue time requirements may be relaxed for hardware-constrained accelerators with approval from the PyTorch CI maintainers.


nit: add here that the end-to-end time will not be relaxed though right?

albanD · 2026-08-11T16:37:35Z

+
+#### L3 Demotion
+
+Demotion is triggered when any of the following conditions are observed over a **4 week** evaluation window:


Can we do 1 or 2 weeks here?

albanD · 2026-08-11T16:38:19Z

+1. Prerequisites
+
+- Must have been operating at **L2 for at least 1 month** before applying for L3 promotion.
+- The [HUD](https://hud.pytorch.org/crcr) must have at least 2 weeks of recent data for the downstream repo.


Does this mean that all the metrics and reuqirement below must be met over a 2week period before the upgrade can happen?

albanD · 2026-08-11T16:39:32Z

+2. **Hardware usage**: the actual deployment scale of the accelerator among PyTorch users.
+3. **Test coverage**: whether the accelerator is required to run the PyTorch core test suite.
+4. **Test pass rate**: the stability of the accelerator's CI test results.
+5. **Oncall responsiveness**: how quickly the downstream team responds to and resolves CI failures.


CI failures and PR being blocked on this job.

expand the criteria for Level 3

80fba2b

meta-cla Bot added the cla signed label Jul 22, 2026

fffrog commented Jul 22, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

fffrog commented Jul 22, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

can-gaa-hou mentioned this pull request Jul 28, 2026

[CRCR] Draft L3/L4 admission criteria pytorch/test-infra#8313

Open

jathu requested review from albanD and atalman July 28, 2026 16:28

albanD reviewed Jul 28, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

update criteria for L3

f9e9cdb

fffrog requested a review from albanD August 3, 2026 08:40

subinz1 mentioned this pull request Aug 3, 2026

[CRCR] Replace Flaky Jobs card with Timeout Rate on per-repo page pytorch/test-infra#8420

Closed

3 tasks

subinz1 mentioned this pull request Aug 3, 2026

[CRCR] Replace Flaky Jobs card with Timeout Rate and align Pass Rate colors with L3 criteria pytorch/test-infra#8421

Merged

3 tasks

jewelkm89 mentioned this pull request Aug 3, 2026

[CRCR][HUD] Create / align metrics in CRCR repo level page to L3 criteria pytorch/test-infra#8423

Closed

atalman requested changes Aug 3, 2026

View reviewed changes

Comment thread RFC-0050-Cross-Repository-CI-Relay-for-PyTorch-Out-of-Tree-Backends.md Outdated

atalman reviewed Aug 3, 2026

View reviewed changes

can-gaa-hou reviewed Aug 5, 2026

View reviewed changes

add the demotion and improve the criteria for L3

05e24ba

fffrog requested a review from atalman August 5, 2026 10:57

albanD mentioned this pull request Aug 7, 2026

[RFC][docs] Add viable/strict-blocking jobs one-pager pytorch/pytorch#192132

Open

albanD reviewed Aug 11, 2026

View reviewed changes


		Temporary downgrade is an operational tool for exceptional situations (e.g., an upstream PyTorch CI SEV, or a downstream repo causing large-scale disruption to the relay system). It must take effect immediately without modifying config files, and be reversible after the incident:

		1. Admin page: HUD provides an admin page for PyTorch CI Maintainers to perform temporary downgrade operations.


		#### L3 Demotion

		Demotion is triggered when any of the following conditions are observed over a 4 week evaluation window:

Uh oh!

Conversation

fffrog commented Jul 22, 2026

Summary

Uh oh!

fffrog commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fffrog commented Aug 5, 2026

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants