Replace the failed-enrolls counter with reported osquery errors

[javuto]

Replace the failed-enrolls counter with reported osquery errors

The dashboard's "Failed enrolls (24h)" tile read the audit log and filtered client-side for lines starting with failed enroll. It's replaced by "Reported errors (24h)" — status logs the fleet sent at osquery's ERROR severity — surfaced across the dashboard, the nodes table and the node detail page.

Errors only. osquery's ladder is 0=INFO, 1=WARNING, 2=ERROR, and warnings are routine enough (a table unavailable on the platform, a transient permission issue) that counting them would keep the tile permanently lit and train people to ignore it.

Counted at ingest, not at read time

The obvious implementation — COUNT(*) FROM osquery_status_data WHERE created_at > now()-24h AND severity != '0' — breaks on three things:

1. The logs may not be in the database. With logger.type set to s3, splunk, kafka, graylog or elastic, that table is empty and the tile would read a permanent zero.
2. No index supports it. OsqueryStatusData indexes only UUID; gorm.Model doesn't index CreatedAt. That's a full scan of the highest-volume table, every dashboard load, every 5 minutes.
3. Severity is stored as a string, so severity > '1' is a string comparison — an easy silent bug.

Instead osctrl-tls tallies ERROR entries per status batch and increments the existing Redis activity rollup (pkg/activity). O(1) per log, correct under every log sink, and the dashboard read becomes O(1). The count runs inside the goroutine that already calls ProcessLogs, so it stays off the request hot path.

No key migration. bitOffset is linear in the event-type index and decodeDay is bounds-checked, so appending EventStatusError at index 6 leaves every existing counter at its offset — old blobs simply decode the new type as zero. EventTypeCount now carries a comment recording why new types must be appended, never inserted.

Drill-down: which nodes are erroring

Reading every node's series to answer this would scale with fleet size, for a question whose answer is a handful of nodes. The write path also does a ZINCRBY into a per-env, per-day sorted set, so TopErrorNodes is a ZREVRANGE — cost scales with the number of erroring nodes. Only errors pay for it, and only when there are any.

GET /api/v1/stats/activity/error-nodes/{env} returns the worst 10 with hostnames resolved; a node deleted since it errored keeps its row and shows by UUID. Clicking the tile opens a dialog listing them, each linking to the node.

The tile only becomes a <button> when there's something behind it — keyboard-reachable and announced as interactive — and stays a plain <div> at zero.

Where errors show up

Surface	What was added
Dashboard tile	"Reported errors (24h)", clickable into the drill-down
Dashboard chart	A fifth line in bright red, in the existing recolourable palette
Endpoint health panel	An "Errors" row, red row wash when non-zero
All nodes table	A fifth heatmap lane, red, with its own intensity scale
Node detail heatmap	An errors row alongside status/result/query/config, plus a count badge in the header
Node detail "Endpoint last seen"	The Errors row goes red and reads "N errors in 24h"

New --error-bright token (#ff4d4f dark / #f01c1c light), deliberately hotter than --danger, which is already used for softer degraded-but-expected states. Every red surface stays neutral at zero — a permanently red panel is one nobody reads.

Errors are excluded from every "total" (activity Total series, the heatmap's totalEvents): they're a subset of the status traffic already counted, so including them would claim the node sent more events than it did.

Bugs caught during the work

• ReadSeries had a duplicated fill loop. The per-node path didn't share fillSeries with ReadEnvSeries, so per-node error counts would have silently read zero — and the drill-down is built on exactly that path. Both now share it.
• The heatmap merge was gated on config. mergeNodeActivityBuckets bounded its alignment loop by config.length; it now takes the max across all Redis series, so a payload with status_error but no config still aligns instead of reading zero.

Changes

• pkg/activity — EventStatusError, StatusError series, ErrorRankKey, TopErrorNodes
• cmd/tls/handlers — countStatusErrors, recordActivityCount, wired into the log POST path
• cmd/api/handlers/stats.go, cmd/api/main.go — EnvErrorNodesHandler + route
• frontend/ — API client, dashboard tile/chart/panel, nodes-table lane, node-detail row, --error-bright
• osctrl-api.yaml regenerated

Testing

• Go: severity filtering (int and quoted-string forms — osquery sends both), malformed-batch tolerance (returns 0 rather than failing ingestion), batch counts advancing by N, zero counts emitting no event, ranking across the UTC day boundary a 24h window straddles, env isolation, non-error events staying out of the ranking, limit truncation, and Total not double-counting.
• Frontend: tile sum and label, inert-at-zero, drill-down dialog, chart line present with no markers, heatmap lane cell count (96 → 120, commented with the lane math), merge alignment including the omitted-field case.
• The fake Redis in pkg/activity tests gained ZINCRBY/ZREVRANGE support — it's a hand-rolled RESP server that only implemented the commands used so far.
• Full suite green: go build/go vet/go test, make openapi-check, frontend 265 tests, tsc.