Repository navigation
Show the per-metric comparison table by default in checkQorMetrics.py - #4829
migueldalberto wants to merge 1 commit into
Conversation
The QoR check report now always lists every metric a rule evaluated, with a new Status column (PASS/FAIL), instead of only the failing ones. The new --quiet (-q) flag leaves the table out; failed rules, warnings and the verdict are still reported. Ruled metrics that are absent from the baseline build are now always named, one per line, rather than only counted. --verbose, which used to gate both, is removed. The failing-rule test fixture now uses the field names the dashboard sends (baseValue, targetValue, thresholdValue). Signed-off-by: Miguel Dalberto Pedro <miguel.pedro@precisioninno.com>
There was a problem hiding this comment.
Code Review
This pull request replaces the --verbose option with a --quiet (or -q) option in the checkQorMetrics.py script. By default, the script now prints a comprehensive table showing all evaluated metrics along with their PASS/FAIL status, whereas --quiet suppresses this table. Additionally, metrics absent from the baseline build are now always listed in the warnings rather than requiring a verbose flag. Unit tests and documentation have been updated to reflect these changes. I have no feedback to provide.
|
@AcKoucher FYI |
🔍 QoR checkMetrics reflect the PR merge build — i.e. what will land on the target branch. Commit 62 design(s) checked — 0 with regression(s), 0 without a comparable baseline. |
vvbandeira
left a comment
There was a problem hiding this comment.
When the dashboard cannot see a baseline for the design (the design is not ingested, or the platform is private and DASHBOARD_API_KEY is not set), it returns every metric sent with status missing_from_baseline.
The script sends every numeric metric in metadata.json.
With this change, that run prints one line per metric after the "Nothing was evaluated" warning, which is hundreds of lines per design, and --quiet does not hide them.
The heading calls them "ruled metric(s)", but in this case they are all the metrics sent.
The removed comment in render_run covered this case ("A design can have hundreds of absent metrics, so name them only on request").
Please keep the count line and name the metrics only up to a small limit, and add a test for an inconclusive reply where every metric is absent.
Optional: docs/contrib/Metrics.md line 39 still says that metadata-check.log lists every failed rule; it now lists every evaluated rule with a status.
Summary
The QoR check report now always shows how every rule-checked metric compares with the baseline, not only the failing ones, and states each metric's verdict in a new Status column.
--quiethides the table for a short report.--verboseis removed: the table it used to gate is now the default, and the absent metrics it used to name are always named.make metadata-checkis unchanged, sometadata-check.lognow holds the full table.Evidence
The same failing run, with one passing rule, one failing rule and one metric absent from the baseline build:
Before:
After:
New tests in
flow/test/test_checkQorMetrics.py: the default table lists every evaluated metric with its status; a passing run prints no[ERROR]line; absent metrics are named but not tabled;--quietand-qdrop only the table;--verboseis rejected.python3 -m unittest test_checkQorMetrics test_genReportpasses (62 tests).The failing-rule test fixture now uses the field names the dashboard actually sends (
baseValue,targetValue,thresholdValue); it passed before only because it never checked the values.Merge Danger
Door: two-way
Output format and one CLI flag; a revert restores both.
Blast Radius: logs
metadata-check.loggrows by one row per evaluated metric, plus one line per absent metric. CI reads only theQoR check:line andgenReport.pycounts only lines starting with[ERROR]; table rows start with|, so neither changes its verdict. Anyone calling the script with--verbosegets a usage error; no caller in this repository does.