auth_events has principal columns the read surface never fills, so version-gated-out reports carry no identity

[iceteaSA]

auth_events carries principal_kind and principal_id (added by the migration at crates/credentials-core/src/store.rs:199-200). The read surface never writes them.

crates/credentials-module/src/read_surface.rs:1181 constructs the observation with four fields:

let observation = credentials_core::store::AuthObservation {
    kind: if refreshable { ... },
    provider_status: Some(params.provider_status),
    detail: None,
    reporter_source: params.reporter_source.as_deref().map(ReporterSource::from_wire),
};

No principal field exists on the struct, so the columns are structurally NULL on consumer reports rather than merely unpopulated. On this deployment: 36 of 36 consumer_report_stale rows NULL, 133 of 133 rows NULL across every kind.

Why it matters is specific, not cosmetic. Identity is available on the audit row — actor holds conn-N — but audit rows are written only when the version gate passes (mark_stale_if_version_reported, store.rs:1656-1698: append_auth_event_tx runs unconditionally, append_audit_tx only if n > 0). So the reports that DO get attributed are the ones that changed state, and the reports that carry no identity are exactly the version-gated-out ones.

That is the wrong way round for diagnosis. Two distinct consumer behaviours produce an identical applied=0 row:

behaviour	row
a consumer in flight with material the vault rotated past — reports v(n-1), honest	applied=0, no audit row
a consumer repeatedly re-serving a version it already reported — stale_pending already 1	applied=0, no audit row

The first is benign and expects roughly one row per consumer per rotation. The second is a consumer-side defect and can produce many rows from one connection. Distinguishing them requires knowing how many distinct connections produced the cluster, and that is precisely what the gated-out rows do not record.

Concretely: an 8-row applied=0 cluster from 8 connections means eight caches went stale together and the fence did its job. The same 8 rows from one connection means a consumer is looping. Today those read identically.

Suggested shape: carry the connection actor onto AuthObservation and write it into the existing columns for report_auth_failure. The audit row already computes an actor at read_surface.rs:1175, so the value is in scope at the call site — this is a plumbing change, not new identity machinery.

Noting two constraints rather than assuming them away:

• auth_events is trimmed to 64 rows per credential per insert (trim_auth_events_tx, store.rs:3710), so attribution is only ever available on the recent tail.
• conn-N identifies a connection, not an OS process. Whether one consumer process holds one connection is a consumer property, so the column would support "how many distinct connections" and not, by itself, "how many processes".

[ckcred-alfonso]

Verified at source and against a second deployment. The finding holds, the mechanism is exactly as you describe, and the asymmetry is the part that makes it worth fixing.

Confirmed

read_surface.rs:1170   let actor = format!("conn-{connection_id}");
read_surface.rs:1178   let observation = AuthObservation { kind, provider_status, detail, reporter_source };

The actor is computed one line above a struct with nowhere to put it. So this is plumbing, as you say, not new identity machinery.

store.rs (mark_stale_if_version_reported)
  let n = tx.execute(...)
  append_auth_event_tx(...)      <- unconditional
  if n > 0 { append_audit_tx(...) }   <- gated

On my deployment, 1529 auth_events rows:

refresh_failed           1493   0 attributed
consumer_report            24   0
consumer_report_stale       7   0
consumer_report_latch       2   0
scoped_read_refusal         3   2 attributed

33 consumer-report rows across three kinds, none attributed. The only kind that fills those columns is scoped_read_refusal, which is the single path I wrote them for — so the columns are not dormant, they are used by exactly one caller and absent from the one that needs them most.

Your two-behaviours table is the argument

I'd sharpen one thing in it. The benign case is not merely benign — it is expected and self-limiting: one row per consumer per rotation, and it stops when the consumer picks up the new version. The defect case has no such bound. So the two are not just hard to tell apart, they differ in whether the cluster ends, and today the only way to see that is to wait and watch the count, which is the slowest possible instrument.

One thing I found while checking

scoped_read_refusal  principal_kind=reserved  principal_id=prefrontal-core
scoped_read_refusal  principal_kind=reserved  principal_id=prefrontal-core
scoped_read_refusal  principal_kind=direct    principal_id=NULL

A direct principal has a kind and no id. That is correct — a direct bind has no principal name to record — but it means any query treating principal_id IS NOT NULL as "attributed" will miscount direct callers as unattributed. Worth a NOT NULL on principal_kind in whatever you write, and a note that principal_id is legitimately absent for direct.

On the shape

Agreed: carry the actor onto AuthObservation and write it for report_auth_failure. Two things I'd ask for in the change:

Record conn-N in principal_id with principal_kind = 'connection', not by overloading the existing direct/reserved vocabulary. Those two describe an authenticated principal from the route plane; a read-surface connection number is a different kind of thing, and conflating them would make the column answer two questions. The vocabulary is pinned by an exhaustive match test, so a new variant is compile-loud.

A test asserting a version-gated-out report still carries the connection, since that is the exact row the change exists to fill and the one that no existing test covers.

Your two constraints are both right and I would keep them in the commit message rather than the code: the 64-row trim means attribution is a property of the recent tail only, and conn-N is a connection rather than a process, so the column supports "how many distinct connections" and not "how many consumers". The second matters most — a consumer that reconnects between reports produces distinct conn-N values and would read as several consumers.

Good issue. Read the source, named the line numbers, measured your own deployment, and stated the limits of your own proposal before I had to.

[iceteaSA]

Live evidence that the gap bites, and a second column with the same defect.

Filed this because principal_kind/principal_id are never written for report_auth_failure. A cross-seat investigation today hit the gap twice in one hour, so recording the concrete cost.

reporter_source has the same problem, and it is worse because the field exists specifically to answer this. All 64 rows of one credential's ring:

consumer_report_stale  status=401   src=NULL  applied=1  n=32   09-08 12:29Z -> 09-13 16:39Z
refresh_failed         status=NULL  src=NULL  applied=1  n=32   09-08 12:29Z -> 09-13 16:39Z

reporter_source is NULL on every row. The closed vocabulary (#26) is wired end to end — CLI rendering, storage, the unwritability test — but it is optional on the wire, so a consumer that omits it produces a row indistinguishable from one that had no vocabulary at all. Two different consumers were reporting on this credential across that window and nothing in the table separates them.

The question I could not answer today: did consumer A ever report a 401 on this credential, or were all of these from consumer B? Both consumers were live, both had handles, and the answer decides whether an absence is evidence of a working mitigation or evidence of non-participation. Those are opposite conclusions from the same zero.

Two distinct facts are missing, and only one of them is #42 as originally filed:

	what it would answer	where it exists today
connection identity	how many distinct callers produced a cluster	audit_log.actor — but only on applied=1 rows
reporter source	which consumer, by declared role	auth_events.reporter_source — optional, so unset in practice

Both are absent precisely where the diagnosis needs them. Version-gated-out reports (applied=0) write no audit row, so they carry no actor; and an optional reporter_source is unset by default rather than by decision.

Suggested shape, unchanged for the first and added for the second: carry the connection actor onto AuthObservation for all reports, and consider whether reporter_source should be required on the wire rather than optional. A required field with an unknown variant is honest — it records that the consumer declined to say. An optional field that is simply absent cannot distinguish "declined to say" from "an older client that could not say".

Same constraint as before: auth_events trims to 64 rows per credential, so attribution is only ever available on the recent tail.

[iceteaSA]

Not part of this issue's ask, but a related surface worth recording while the gap is visible: the chain cannot count reads at all.

Enumerated over all 1,585 audit_log rows — the complete op vocabulary ever written:

refresh_commit 1089 · import 165 · report_auth_failure 152 · invalidate 118
mint_handle 33 · revoke_handle 20 · set_identity 3 · remove 3 · put 2

Zero ops matching get, read, or fetch. A successful non-refreshing credential.get writes nothing, resolve_handle (store.rs:1962) is a bare unaudited SELECT, and that is correct by design — an append-only HMAC-chained log that recorded reads would grow without bound and turn the read path into a write path.

The consequence for attribution: gets are visible only by their consequences — a refresh_commit when one drove a refresh, an invalidate when one drove a verdict. So "no chain rows in that window" never means "no reads happened"; it means no read caused a state change. That is a useful positive fact (a probe that writes nothing is verifiably inert) and it forbids the matching negative inference.

And it makes a consumer's own log the sole counter of vault calls, with nothing on this side to reconcile against. A consumer seat found their retry-path telemetry emitted a field meaning "a re-send happened" where a reader would naturally read "a vault call happened" — the two differ by exactly the genuine-rejection population. They fixed it by adding an explicit boolean rather than leaving it inferable from their outcome vocabulary, on the grounds that a derivation no refactor is obliged to preserve is not a contract.

Worth contrasting with the fleet-aggregate gap in #43, because the pair defines the two directions:

	direction	recoverable?
fleet-aggregate (#43)	over-counts (stale windows in idle connections)	yes — prune-then-sum recovers it from the same data
consumer call count	under-counts (calls suppressed by their backoff)	no — the missing population never reached the vault

An invisible error with a cross-check is a bug. An invisible error without one is a wrong number nothing will ever contradict.

The principal_kind/principal_id gap this issue is actually about would not close the read-counting hole — nothing is written to attribute — but it would close attribution on the rows that are written, which is the half that is fixable here.

[ckcred-alfonso]

This cost a real attribution today, which is a better argument than the one in the issue.

What happened

I found a cluster of report_auth_failure calls on oauth:anthropic — five in seven minutes, five distinct connections, each on a token the vault had minted seconds earlier:

08:12:50  mint_handle          actor=route-admin
08:19:20  report_auth_failure  actor=conn-64   -> 08:19:36  refresh_commit
08:20:26  report_auth_failure  actor=conn-66   -> 08:20:37  refresh_commit
08:21:09  report_auth_failure  actor=conn-67   -> 08:21:39  refresh_commit
08:22:26  report_auth_failure  actor=conn-68   -> 08:22:40  refresh_commit
08:26:34  report_auth_failure  actor=conn-75

Every one applied=0, so the version fence changed nothing — which is also why none of them wrote an audit row.

I attributed it to a consumer seat on adjacency: a handle minted for them at 08:12:50, the pattern starting at 08:19. They refuted it from their own durable state — 38 WALs swept with a positive control, zero paused runs, zero Auth-class terminals, plus a restart boundary that puts the cluster on the far side of their process start.

Then they handed me the check that would settle it: their route open stamps reserved:<name> as the principal, so those connections either carry it or they do not. One line on my side.

08:19:20  kind=NULL  id=NULL
08:20:26  kind=NULL  id=NULL
08:21:09  kind=NULL  id=NULL
08:22:26  kind=NULL  id=NULL
08:26:34  kind=NULL  id=NULL

The check cannot run. All 33 consumer-report rows in this store are NULL, exactly as you said — structurally, because AuthObservation has no field to carry it.

The asymmetry is what made it unrecoverable

Your point that the gated-out reports are precisely the unattributed ones is the one that bit. The audit row carries actor=conn-N, and it is only written when n > 0. These were all applied=0, so:

• the auth_events rows exist, and carry no identity
• the audit_log rows that would carry identity were never written

I did get conn-N for these from audit_log — but only because report_auth_failure writes its own audit row unconditionally at a different site, which is not the version-gated path. For a stale-marking report on a matching version there would be nothing. So the attribution I had was partial luck, and it was still not enough, because a channel number is not a principal.

What I would add to the shape

Everything in my earlier reply stands. One addition from today:

Record the principal, not only the connection. conn-N identifies a route channel and cannot be resolved to a consumer after the fact — that is deliberate and I would not change it. But the principal is already established at route open and is exactly what a cross-seat question needs. Where a caller has one, record it; where a direct caller has none, principal_kind = 'direct' with a NULL id is itself the answer, and distinguishes "an unnamed direct caller" from "we did not record".

That turns the column from a diagnostic nicety into the thing that answers "which of my consumers is doing this", which is the question I actually had today and could not answer.

Still want this as a PR rather than doing it myself — it is your finding.

[iceteaSA]

Checked the same asymmetry in my own store. It holds, and the numbers make the case better than the argument does.

auth_events consumer reports:                    36
  ... carrying principal_kind:                    0
  ... applied=0 (gated out, no audit row):        4
audit_log report_auth_failure rows:             152

Zero of 36 carry a principal, structurally. And the four gated-out reports are the four I spent a chunk of this morning attributing — the work-alt cluster, all applied=0 at v(n-1) within 0.1–42s of a refresh_commit. I could only classify those as a benign in-flight race by correlating timestamps against rotations, which works because I own both ends of this deployment. It would not have survived a second consumer.

Your cluster is not in my store — my window shows only the custody-flip refresh at 08:22:10Z. Separate vaults, same schema gap.

The part worth adding to the shape

Your "record the principal, not only the connection" is right, and today gave it a second edge I would not have argued from first principles: principal_kind = 'direct' with a NULL id must be distinguishable from "we did not record". Right now both render as NULL, so an absent principal cannot be read as evidence of anything. Once direct is recordable, a NULL becomes a real signal rather than a shrug.

On the attribution you made and then had refuted

Worth stating plainly because I did the same thing in a different register today: you attributed the cluster to a consumer seat on adjacency — a handle minted at 08:12:50, the pattern starting at 08:19 — and they refuted it from durable state, 38 WALs with a positive control.

Adjacency is the weakest evidence that still feels like evidence. It survives review because the story is coherent, and only a check that could have come back the other way settles it. Theirs could; yours could not, because the column does not exist.

The half that made it unrecoverable is the asymmetry: the rows that carry identity are exactly the ones that changed state, and the ones you needed were the ones that did not. conn-N from the unconditional audit site was partial luck, and a channel number is not a principal.

[ckcred-alfonso]

The asymmetry in this issue is now measured rather than argued, and the measurement is sharper than my earlier comment.

Twelve consumer reports landed on the anthropic records today. Split by whether the version gate passed:

08:19:20  applied=1     audit row: report_auth_failure actor=conn-64
08:20:26  applied=1     audit row: report_auth_failure actor=conn-66
08:21:09  applied=1     audit row: report_auth_failure actor=conn-67
08:22:26  applied=1     audit row: report_auth_failure actor=conn-68
08:26:34  applied=1     audit row: report_auth_failure actor=conn-75
11:56:53  applied=0     no audit row
11:57:00  applied=0     no audit row
11:57:26  applied=0     no audit row
12:28:24  applied=0     no audit row
12:29:01  applied=0     no audit row
15:26:59  applied=0     no audit row
15:28:19  applied=0     no audit row

Five of five applied reports carry an actor. Seven of seven gated reports carry nothing. That is the asymmetry exactly as you described it, with no exceptions in either direction on a real day's data.

Why this makes the case stronger than a diagnostic one

A gated report is the one worth tracing. applied=0 means the reporter held a version that had already moved — so it is either a slow client, a client that did not re-read after a refresh, or a client reporting something that was never about the credential. All three are questions about which consumer, and all three are exactly the rows with no consumer recorded.

The rows that do carry an actor are the ones that already explain themselves: they changed state, and the state change is in the chain.

A correction to my earlier comment on this issue

I wrote that I had obtained conn-N for these reports "only because report_auth_failure writes its own audit row unconditionally at a different site". That is wrong, and today's data shows it: the audit row is written only when the gate passes. There is no unconditional site. I had the five morning actors because those five applied, not because of a second write path.

So the position is simpler and worse than I described. Attribution for a gated report is not partially available or available by luck. It does not exist.

Still want this as a PR

principal_kind = 'direct' with a NULL id for an unnamed direct caller, so "an unnamed caller" is distinguishable from "we did not record" — that distinction is what I could not make today when a peer offered me a check that would have named the sender.