Repository navigation
QoR check: make the local merge-base pin fail safe - #4834
vvbandeira wants to merge 1 commit into
Conversation
The pin for an unpushed commit could hide or distort the comparison. - A merge base with no build of the design gave no baseline, and the latest-master fallback was never tried. The check now sends the run again without the pin. An explicit --base-commit stays exact. - In a fork clone, origin/master is the fork's old master, so upstream changes counted as the commit's own. The default ref is now upstream/master when it exists, else origin/master. Each applied pin prints a warning that names the ref. - A commit with no merge base, a missing ref, or a failed remote lookup made the check fail or pin by mistake. These now warn and send the run unpinned. - The FAIL report test used keys the renderer does not read, so the row printed n/a and the test still passed. Signed-off-by: Vitor Bandeira <vvbandeira@precisioninno.com>
There was a problem hiding this comment.
Code Review
This pull request updates the QoR metrics check script to use upstream/master as the default merge-base reference if it exists, falling back to origin/master. Additionally, if an automatically pinned merge base does not have a build on the dashboard, the script now automatically retries the check without the pin (falling back to the latest master build) rather than failing. The tests, documentation, and makefiles have been updated to reflect these changes. I have no feedback to provide as there are no review comments.
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
Follow-up to #4604.
For a commit that is on no remote branch,
checkQorMetrics.pypins the baseline to a merge base that it computes locally.Four cases made that pin hide or distort the comparison.
This PR fixes them.
CI is not affected, because
DASHBOARD_JOB_NAMEturns the pin off.--base-commitis still exact and is not sent again.gh repo fork --clone,origin/masteris the fork's master, which can be far behind. Upstream changes since then counted as the commit's own, which could give a false FAIL or hide a regression. The default--merge-baseis nowupstream/masterwhen that ref exists, elseorigin/master. Each applied pin is now a[WARN]that names the ref, not an[INFO].--merge-baseref that does not exist, gaveQoR check: ERRORand exit 1, somake metadatafailed. A failedgit for-each-refwas read as "on no remote" and pinned anyway. Now each case prints a warning and sends the run unpinned.baselineValue,currentValue,limit), so the row printedn/aand the test still passed. It now usesbaseValue,targetValue, andthresholdValue, and asserts the rendered row.Tests:
cd flow/test && python3 test_checkQorMetrics.pyhas 53 tests, all pass.The six new tests for the cases above fail on master.