Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The audit can miss index-definition drift that breaks hinted queries, while its cache retains expired successful checks and retries persistent failures on every event, causing memory growth and excess database load.
3 open issues
What changed in this PR
This PR adds grouper-side index checks ahead of event writes, preparing for collection creation on first ingestion while leaving API-side creation unchanged.
Changes:
- Defines expected indexes and creates missing collections; existing collections are audited without building indexes.
- Adds hourly per-project caching, failure logging, and timing instrumentation.
- Adds integration tests for collection creation, uniqueness enforcement, and missing-index reporting.
| File | Description |
|---|---|
workers/grouper/tests/eventIndexes.test.ts |
Tests index creation and audit-only behavior. |
workers/grouper/src/utils/eventIndexes.ts |
Defines expected indexes and implements collection checks. |
workers/grouper/src/metrics/grouperMetrics.ts |
Adds the index-check timing step. |
workers/grouper/src/index.ts |
Runs cached checks before event processing and logs results. |
Review effort: Balanced
Comment on lines
+541
to
+544
| .catch((error) => { | ||
| this.eventIndexesChecks.delete(projectId); | ||
| this.logger.error(`[ensureEventIndexes] project=${projectId} failed: ${error.message}`); | ||
| }); |
Comment on lines
+546
to
+549
| this.eventIndexesChecks.set(projectId, { | ||
| expiresAt: Date.now() + EVENT_INDEXES_CHECK_TTL, | ||
| done, | ||
| }); |
|
|
||
| for (const spec of specs) { | ||
| const found = existing.some((index) => { | ||
| return JSON.stringify(index.key) === JSON.stringify(spec.key) && !!index.unique === !!spec.unique; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grouper now checks a project's events, repetitions and dailyEvents collections before handling an event, once per hour per project. A missing collection is created with the same indexes the API sets up for a new project. For an existing collection the indexes are only compared with the expected list and the missing ones are logged, nothing is built on live data. A failed check is logged and does not stop event processing.
This is the first step towards creating event collections on the first event instead of at project creation. About 76% of projects have never received an event, and their empty collections make up more than half of all tables in the database. The API still creates collections as before, so behaviour does not change yet: the logs will show whether the grouper's index list matches production.