You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
improvement(tables): read row counts and versions through an append-only change log - #8653
Every row write updates row_count / rows_version on the table's single user_table_definitions row and holds that lock until its commit is acknowledged by a standby, so one slow commit makes every other writer to the same table hit lock_timeout and roll back
Phase 1 of moving those counters to an append-only change log (the buffered-counter pattern): adds user_table_row_changes, and every reader now reads stored value + unfolded tail in one statement (lib/table/row-changes.ts) — table get/list/query counts, the snapshot cache key, and the snapshot mount-safety check
rows_version is the stored version plus the count of log rows, never max(id) — ids are allocated before commit, so a late commit can carry a lower id than one already seen and the cache would serve stale bytes
New /api/cron/fold-table-row-changes (every minute, helm + docker crontab) folds each table's log into its definition row in one statement; SKIP LOCKED so it never waits on a held row and overlapping sweeps skip each other. updated_at now moves at fold time, so it trails the last insert/delete by up to a minute
Fork copy no longer overwrites row_count after copying rows — the trigger already counts them, and once writers log changes the overwrite would double-count
No behavior change on its own: nothing writes to the log yet. Phase 2 (separate PR, after this deploys) points the row-count / rows_version triggers at the log; shipping both together would let old instances read the stored column alone mid-deploy
Prod schedules crons in infra; that entry is a separate infra PR
Type of Change
Improvement
Testing
New lib/table/row-changes.integration.ts against real Postgres (push + migrate): reads include the tail, fold keeps stored + tail constant, fold skips a held definition row in <1s instead of waiting, version stays exact and non-decreasing with 8 concurrent appenders while folds run, sweep covers every table. Reverting the reads to the stored column turns the read tests red
Table integration suites on a migrated DB (155 passed), secret-provenance.integration.ts fixture gains the log table
Local E2E on the running app: logged changes by hand → tables page showed stored + tail; cron fold → stored values updated, page unchanged, log empty, updated_at in UTC
bun run test (local-only failures are .env-driven rate-limit/timeout tests untouched here, plus one flaky sandbox test that passes alone), bun run lint, check:audits, check:migrations origin/staging, block-registry check, docs-manifest:check, drizzle generate clean
Checklist
Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
[Critical risk] Adds database schema and background job for table row tracking.
The PR appears safe to merge based on the changes reviewed.
Summary
This PR adds an append-only table row-change log, reads live counts and versions as stored values plus the unfolded log, and schedules a background fold. It also removes a redundant row-count overwrite during fork copying.
The three previous findings are resolved and do not remain outstanding.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Table definition: stored count and version] --> C[Live reads: stored values plus log tail]
B[Append-only row-change log] --> C
B --> D[Scheduled fold]
D --> A
D --> E[Delete folded log rows]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Summary
row_count/rows_versionon the table's singleuser_table_definitionsrow and holds that lock until its commit is acknowledged by a standby, so one slow commit makes every other writer to the same table hitlock_timeoutand roll backuser_table_row_changes, and every reader now reads stored value + unfolded tail in one statement (lib/table/row-changes.ts) — table get/list/query counts, the snapshot cache key, and the snapshot mount-safety checkrows_versionis the stored version plus the count of log rows, nevermax(id)— ids are allocated before commit, so a late commit can carry a lower id than one already seen and the cache would serve stale bytes/api/cron/fold-table-row-changes(every minute, helm + docker crontab) folds each table's log into its definition row in one statement;SKIP LOCKEDso it never waits on a held row and overlapping sweeps skip each other.updated_atnow moves at fold time, so it trails the last insert/delete by up to a minuterow_countafter copying rows — the trigger already counts them, and once writers log changes the overwrite would double-countType of Change
Testing
lib/table/row-changes.integration.tsagainst real Postgres (push + migrate): reads include the tail, fold keeps stored + tail constant, fold skips a held definition row in <1s instead of waiting, version stays exact and non-decreasing with 8 concurrent appenders while folds run, sweep covers every table. Reverting the reads to the stored column turns the read tests redsecret-provenance.integration.tsfixture gains the log tableupdated_atin UTCbun run test(local-only failures are.env-driven rate-limit/timeout tests untouched here, plus one flaky sandbox test that passes alone),bun run lint,check:audits,check:migrations origin/staging, block-registry check,docs-manifest:check, drizzle generate cleanChecklist
test-auditauthoring gate)🤖 Generated with Claude Code
https://claude.ai/code/session_01YVFtF6hWSD8ot7omjmqF5U