Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 96 additions & 61 deletions docs/hydra/operations/sig-scores.mdx
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
---
title: SIG scores
sidebar_label: SIG scores
description: Our own estimate, per app and per check, of the maintainability stars Sigrid will give the fleet.
title: Quality matrix
sidebar_label: Quality matrix
description: One matrix per app with our estimate of the SIG stars, our own banded measures and every gate.
---

import SigMatrix, {SigFloors} from '@site/src/components/SigMatrix';
import SigMatrix, {SigBeyond, SigFloors} from '@site/src/components/SigMatrix';
import matrix from '@site/src/data/sig-matrix.json';

# SIG scores
# Quality matrix

This page is our own measurement, not Sigrid. We run gate 122
(`check_sig_maintainability.py --profile five-star`) on each app's
`development` branch. The result estimates what the Software Improvement Group
(SIG) will say in Sigrid. Sigrid uses its own parsers and scoping, so its
numbers can differ.
This page is our own measurement, not Sigrid. We measure each app's
`development` branch with our gates and tools. The SIG rows estimate what the
Software Improvement Group (SIG) will say in Sigrid. Sigrid uses its own
parsers and scoping, so its numbers can differ.

**Measured on {matrix.measured}.** The commit under each app name is the head
that was measured.
Expand All @@ -29,79 +28,115 @@ rise from 1 November.

## How to read the matrix

Each column is an app. Each row is one SIG check for one language. A cell shows
the star level the check earns, with the measured share under it. Hover a cell
for the value and both limits.

- **Unit size, complexity and interfacing:** the share of unit code above the
bound, for example the code in units longer than 15 lines.
- **Duplication:** the share of redundant lines, in fragments of 6 lines or more.
- **Module coupling:** the share of module code in files with more than 10, 20
or 50 incoming dependencies.
- **Component independence:** the share of code that no other component calls.
Higher is better here. For every other check, lower is better.

The 4-star limits come from the SIG/TUV NORD CERT Guidance for Producers v17.0.
SIG publishes no 5-star limits, so our 5-star yardstick is half of each 4-star
limit. The guidance also gives no limits for 3, 2 or 1 stars. A check that
misses 4 stars is therefore shown as `<4`, not as an estimated lower star.

The totals at the top count the checks per app. An app is only measured on the
languages it contains, so the totals differ between apps. The percentages and
the average above the matrix are our own figures, not SIG scores.

The matrix scrolls in both directions. Use the search box to show only the apps
you want to compare.

A cell with a ring and a lock is floored. Gate 123 holds that check: the app
met it when its floor was recorded, and a change that loses it fails. The
"floored checks" row counts them and shows the commit the floor was recorded
at. Floors come from each app's `.github/sig-floor.json` on `development`, read
on {matrix.floors.read}. They can be newer than the measurement, so a floored
cell can still show `<4` until the next measurement.
Each column is an app. The rows come in four sections.

1. **SIG maintainability.** Every check of gate 122, per language, rated from
1 to 5 stars.
2. **SIG's other models.** Open Source Health and Architecture Quality, where
we can measure them. SIG publishes only the 4-star value, so a cell says
met (4★) or missed (&lt;4).
3. **Our own measures.** Tests, debt, security, and accessibility and UI. We
define these bands ourselves. The table below the matrix gives each band's
basis.
4. **Gates per category.** Each row shows the gates that pass out of the gates
that apply, and the findings of all of them. Open a row to see its gates.

A cell shows the stars and the measured value. Hover a cell for the details.
An empty cell (`·`) means the app has no such code or the gate does not apply.

### SIG maintainability bands

The 4-star limits come from the SIG/TÜV NORD CERT Guidance for Producers
v18.1, dated 31 March 2026. It is the 2026 model. Volume no longer counts.

SIG publishes no other level. We derive the 5, 3 and 2-star limits ourselves.
The only public table of all levels is Alves, Correia and Visser (2011). We
take its distance between each band and the 4-star band. We apply that ratio
to the v18.1 limits. Duplication uses the band ratios of Heitlager, Kuipers
and Visser (2007). Anything worse than 2 stars is 1 star.

| Check | 5★ | 4★ (v18.1) | 3★ | 2★ |
|---|---|---|---|---|
| Unit size, code in units over 15 / 30 / 60 lines | 36.0 / 17.0 / 5.5% | 48.0 / 24.2 / 9.1% | 63.0 / 34.7 / 15.4% | 84.7 / 49.0 / 25.3% |
| Unit complexity, McCabe over 5 / 10 / 25 | 15.8 / 4.4 / 0.5% | 20.7 / 7.5 / 1.1% | 27.7 / 10.6 / 1.7% | 34.6 / 13.2 / 2.7% |
| Unit interfacing, 3+ / 5+ / 7+ parameters | 12.5 / 2.5 / 0.6% | 15.4 / 3.4 / 0.9% | 18.3 / 4.8 / 1.4% | 26.0 / 7.2 / 2.1% |
| Module coupling, fan-in over 10 / 20 / 50 | 7.4 / 3.3 / 1.2% | 9.7 / 5.3 / 1.8% | 10.7 / 5.9 / 2.3% | 13.0 / 8.0 / 3.8% |
| Duplication | 3.5% | 5.8% | 11.6% | 23% |
| Component independence | at least 97.3% | at least 94.5% | no band | no band |
| Component entanglement | at most 0.0375 | at most 0.075 | no band | no band |

The 2011 table was calibrated on a 20/20/20/20/20 spread. SIG now uses
5/30/30/30/5. So the real 5-star limits are probably stricter, and the real
2-star limits looser.

For independence and entanglement no public source gives other levels. We set
5 stars at half the 4-star exposure. A value below 4 stars shows as `<4`.
Entanglement is our estimate from the violation degree, because SIG
normalises it against benchmark figures it does not publish. It is shown, but
gate 123 does not floor it.

### Floors

A cell with a ring and a lock is floored. Gate 123 holds that check. The app
met it when its floor was recorded, and a change that loses it fails.
Re-recording a floor never lowers it. A floored check that the v18.1 limits
miss stays in the floor and shows `<4` until development meets it again.

<SigMatrix />

## What we add beyond SIG

SIG rates maintainability, security, open source health and architecture. It
does not rate tests, debt hidden in baselines, or accessibility. These are our
own measures, with the reason we add each one and the basis of its bands.

<SigBeyond />

Spec coverage counts the methods gate 16 holds in scope, with and without an
`@spec` tag. E2e coverage is gate 19's report over the scenarios that are not
excluded. PHP coverage is the floor coverage-guard records on `development`.
The gate rows come from a full-scope run of every gate on `development`.

## Floors and ratchets per app

Besides the SIG floor, each app keeps its own floors and baselines on
`development`. A baseline lists debt that may stay, so a lower count is
better. A coverage floor is a minimum, so a higher value is better. A `·`
means the app has no such file. Hover a cell for the file it came from. The
commit under each app name is the `development` head that was read on
{matrix.floors.read}.
means the app has no such file. Hover a cell for the file it came from.

<SigFloors />

## Refresh the data

The matrix reads `website/src/data/sig-matrix.json`. A script writes that file
from gate 122 reports.
The matrix reads `website/src/data/sig-matrix.json`. One script writes it.

1. Clone each app's `development` branch into one directory, named after the
app.
2. Run gate 122 in report mode on each clone:

```bash
python3 hydra-gates/scripts/lib/check_sig_maintainability.py report <clones>/<app> \
--json <report-dir>/<app>.json
```

1. Clone each app's `development` branch and run gate 122 on it in report mode.
Write one `<app>.json` per app into one directory:
3. Run all gates at full scope on each clone. The runner writes `gates.json`
into its log directory:

```bash
python3 hydra-gates/scripts/lib/check_sig_maintainability.py report <app-dir> \
--profile five-star --json <report-dir>/<app>.json
HYDRA_GATE_LOG_DIR=<gates-dir>/<app> bash hydra-gates/scripts/run-hydra-gates.sh --full <clones>/<app>
```

The file name sets the column name, so name it after the app.
2. Run the generator from the repository root:
4. Run the generator from the repository root. It runs `composer audit` and
`npm audit` on the lockfiles, so it needs the network:

```bash
python3 website/scripts/build-sig-matrix.py <report-dir> --date YYYY-MM-DD
python3 website/scripts/build-sig-matrix.py <report-dir> --clones <clones> --gates <gates-dir> --date YYYY-MM-DD
```

Leave out `--date` to use the date of the newest report. The generator then
reads each app's floors and baselines from `development` through `gh api`,
so it needs a logged-in GitHub CLI. Add `--clones DIR` to read them from
local clones in `DIR/<app>` instead, or `--no-floors` to keep the floors
already in the file.
3. Commit the new `sig-matrix.json`.
5. Commit the new `sig-matrix.json`.

The generator takes the limits from gate 122 itself, so the matrix follows any
change to the profiles. It stops when its five-star count for an app differs
from the count in the gate's own report.
The generator takes the star bands from gate 122 and the own bands from
`website/scripts/quality_measures.py`. A change to either shows up in the
matrix on the next refresh.

Back to [app health](./app-health.md).
32 changes: 24 additions & 8 deletions hydra-gates/scripts/lib/check_sig_floor.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,14 @@
Without a floor file the gate is NOT APPLICABLE (exit 4) before any tool is
loaded or any base resolved.

The 5-star checks (half of every 4-star limit, gate 122's five-star profile)
are reported next to the 4-star ones and never block.
The 5-star checks (gate 122's five-star profile, derived from the v18.1 4-star
limits) are reported next to the 4-star ones and never block.

`write` never lowers a floor. When the floor file already exists, every check
it lists stays in it, also one the current limits no longer meet (the 4-star
limits moved from v17.0 to v18.1, stricter on coupling and independence). Such
a check is printed as floored but missed, so the change of status is visible;
a check stays inherited until development meets it again.

When the head meets checks the committed floor does not list, `--proposed`
receives the raised floor file, for the workflow to offer as a pull request.
Expand Down Expand Up @@ -250,6 +256,7 @@ def floor_document(app, labels):
"Generated by check_sig_floor.py write; raised by the workflow, never lowered by hand.",
"schema": SCHEMA,
"profile": "four-star",
"limits": sig.GUIDANCE,
"tools": {"lizard": sig.LIZARD_VERSION},
"unit_reader": sig.UNIT_READER_VERSION,
"measured_at": head,
Expand Down Expand Up @@ -327,8 +334,8 @@ def cell(r):
return "n/a" if r is None else sig.pct(r["share"]) + ("" if r["ok"] else " ✗")

out = [f"## SIG score floor: {app_name}", "",
"Every 4-star check in the floor must stay met. 5-star (half of each 4-star limit) "
"is reported only and never blocks. ✗ marks a miss.", "",
f"Every 4-star check in the floor must stay met ({sig.GUIDANCE}). 5-star (derived from the "
"4-star limits) is reported only and never blocks. ✗ marks a miss.", "",
"| check | base | head | 4-star limit | floored | 5-star |", "|---|---|---|---|---|---|"]
for label in sorted(set(head4) | set(floor) | set(base4 or {})):
h, b, h5 = head4.get(label), (base4 or {}).get(label), head5.get(label)
Expand All @@ -349,7 +356,7 @@ def cell(r):
[f"- {lbl}: {why}" for lbl, why in inherited]
five_missed = sorted(lbl for lbl, r in head5.items() if not r["ok"])
if five_missed:
out += ["", f"5-star findings (not blocking): {len(five_missed)} check(s) above half the 4-star limit: "
out += ["", f"5-star findings (not blocking): {len(five_missed)} check(s) miss the 5-star limit: "
+ ", ".join(five_missed)]
if proposed:
out += ["", "The head meets checks the committed floor does not list yet: " + ", ".join(proposed)
Expand All @@ -370,13 +377,22 @@ def cmd_write(args, lizard):
print(f"{GATE}: no measured source file in this repository, so there is no floor to write")
return 4
rows = check_rows(report, "four-star")
met = [lbl for lbl, r in rows.items() if r["ok"]]
met = {lbl for lbl, r in rows.items() if r["ok"]}
path = os.path.join(args.app, args.floor)
# A floored check that left the measured scope (a sigrid.yaml change) leaves the file too.
kept = set(parse_floor(sig.read_text(path), args.floor)) & set(rows) if os.path.isfile(path) else set()
missed = sorted(kept - met)
for lbl in missed:
r = rows[lbl]
cmp = "<" if r["higher_is_better"] else ">"
print(f" floored but missed at the current limits, kept in the floor: {lbl}: "
f"{sig.pct(r['share'])} {cmp} limit {sig.pct(r['limit'])}")
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w") as fh:
json.dump(floor_document(args.app, met), fh, indent=1)
json.dump(floor_document(args.app, met | kept), fh, indent=1)
fh.write("\n")
print(f"{GATE}: wrote {args.floor}: {len(met)} of {len(rows)} 4-star checks met and floored")
print(f"{GATE}: wrote {args.floor}: {len(met)} of {len(rows)} 4-star checks met, "
f"{len(met | kept)} floored ({len(missed)} floored but missed)")
return 0


Expand Down
Loading
Loading