Skip to content

QoR check: make the local merge-base pin fail safe - #4834

Draft
vvbandeira wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
vvbandeira:qor-check-pin-fixes
Draft

vvbandeira wants to merge 1 commit into
The-OpenROAD-Project:masterfrom
vvbandeira:qor-check-pin-fixes

Conversation

@vvbandeira

Copy link
Copy Markdown
Member

Follow-up to #4604.
For a commit that is on no remote branch, checkQorMetrics.py pins the baseline to a merge base that it computes locally.
Four cases made that pin hide or distort the comparison.
This PR fixes them.
CI is not affected, because DASHBOARD_JOB_NAME turns the pin off.

  • Merge base without a build. Before: the result was INCONCLUSIVE ("no baseline"), and the latest-master fallback that the docs describe was never tried. Now the check sends the run again without the pin, so the dashboard uses its normal fallback. An explicit --base-commit is still exact and is not sent again.
  • Fork clone. With gh repo fork --clone, origin/master is the fork's master, which can be far behind. Upstream changes since then counted as the commit's own, which could give a false FAIL or hide a regression. The default --merge-base is now upstream/master when that ref exists, else origin/master. Each applied pin is now a [WARN] that names the ref, not an [INFO].
  • No pin possible. Before: a commit with no merge base, or a --merge-base ref that does not exist, gave QoR check: ERROR and exit 1, so make metadata failed. A failed git for-each-ref was read as "on no remote" and pinned anyway. Now each case prints a warning and sends the run unpinned.
  • Test fixture. The FAIL report test used keys that the renderer does not read (baselineValue, currentValue, limit), so the row printed n/a and the test still passed. It now uses baseValue, targetValue, and thresholdValue, and asserts the rendered row.

Tests: cd flow/test && python3 test_checkQorMetrics.py has 53 tests, all pass.
The six new tests for the cases above fail on master.

The pin for an unpushed commit could hide or distort the comparison.

- A merge base with no build of the design gave no baseline, and the
  latest-master fallback was never tried. The check now sends the run
  again without the pin. An explicit --base-commit stays exact.
- In a fork clone, origin/master is the fork's old master, so upstream
  changes counted as the commit's own. The default ref is now
  upstream/master when it exists, else origin/master. Each applied pin
  prints a warning that names the ref.
- A commit with no merge base, a missing ref, or a failed remote lookup
  made the check fail or pin by mistake. These now warn and send the
  run unpinned.
- The FAIL report test used keys the renderer does not read, so the
  row printed n/a and the test still passed.

Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the QoR metrics check script to use upstream/master as the default merge-base reference if it exists, falling back to origin/master. Additionally, if an automatically pinned merge base does not have a build on the dashboard, the script now automatically retries the check without the pin (falling back to the latest master build) rather than failing. The tests, documentation, and makefiles have been updated to reflect these changes. I have no feedback to provide as there are no review comments.

@openroad-ci

Copy link
Copy Markdown
Member

🔍 QoR check

Metrics reflect the PR merge build — i.e. what will land on the target branch.

Commit 422e07a · Jenkins build #1 · Baseline: build · View build on dashboard

62 design(s) checked — 0 with regression(s), 0 without a comparable baseline.

@vvbandeira
vvbandeira marked this pull request as draft November 16, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants