Skip to content

feat: PG full-text search columns (PPT-2644, 1/2) - #324

Merged
stakach merged 3 commits into
masterfrom
PPT-2644-pg-full-text-search
Aug 11, 2026
Merged

feat: PG full-text search columns (PPT-2644, 1/2)#324
stakach merged 3 commits into
masterfrom
PPT-2644-pg-full-text-search

Conversation

@camreeves

@camreeves camreeves commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

(2026-08-10: rebased onto current master — zero conflicts — and the migration renumbered 2026080610050000020260810100500000; master's #323 moved the AI-support migration onto our old number. The new ai_* tables deliberately get no search vectors: they back no rest-api search route.)


Part 1 of PPT-2644 (migrate search from Elasticsearch to PostgreSQL FTS). Additive and safe to merge first — nothing reads the new columns until the rest-api PR lands.

What

One migration (20260810100500000_add_search_vector_columns.sql):

  • a generated, STORED search_vector tsvector column + GIN index on the 25 tables whose rest-api index routes are searchable (trig deliberately excluded — trigger instances have no searchable text of their own; their search goes through the parent trigger via a query-time join)
  • two IMMUTABLE helper functions: placeos_fts_join(text[]) (arrays → text; array_to_string is only STABLE so generated columns can't call it directly) and placeos_fts_email(text) (guests-style [@._] tokenization + the raw address, so both reeves and the whole address match)
  • full Down section

Design notes (details in the plan):

  • 'simple' config on both write and query side — entity names/identifiers/emails must not be stemmed; matches the existing guests.tsv_search precedent (PPT-54 / feat: PPT-54 Add full text search capabilities to guest model #278)
  • id::text is in every vector (Backoffice sends id in every fields list)
  • secrets and encrypted content are deliberately excluded from every vector: password digests, tokens, api-key/oauth secrets, IdP credentials, and settings_string. Several of those are searchable via ES today; that ends here on purpose.
  • no setweight — the read side orders by name, not rank, so weights would be dead code

A follow-up models major will remove include Neuroplastic, the es_type:/es_subfield:/es_ignore: attribute tags and the parent: tag once rest-api is off ES (part 2).

⚠ Rollout note — prepared statements

Adding columns invalidates the result shape of SELECT * prepared statements held by already-connected services: until their connections cycle, those queries fail with PG's cached plan must not change result type (observed live on a full local stack: rest-api/auth 500s after the ALTER, gone on service restart). Roll/restart the Crystal services after this migration applies (a normal deploy restart is sufficient) rather than leaving them on old connections. This applies to any wide migration, but 25 altered tables makes it near-certain here.

Testing

  • new spec/search_vector_spec.cr (5 examples): prefix matching on names/descriptions, array columns (features/tags/keys), email tokenization + whole-address lexeme, secrets-not-indexed negative, and the modules↔driver EXISTS join pattern rest-api will use
  • full suite: 632 examples, 0 failures with the migration (2026-08-06); 627/0 baseline without it. Re-run post-rebase 2026-08-10: 632 examples with two known-flaky non-greens, both A/B-classified as pre-existing — playlist_spec.cr:170 (suite-order dependent; 13/0 standalone; fresh-DB A/B 4/4 green both ways) and user_spec.cr:137 (concurrent admin-delete advisory-lock timeout; 1-in-3 failure rate reproduced identically on a detached origin/master worktree under load)
  • CI note: the Ameba job ("use built-in formatter" on an untouched file) and one order-dependent spec flake in the crystal-latest test job fail identically on current master — pre-existing, not from this branch.

Full migration plan + research (11 reports): https://gist.github.com/camreeves/4b3cb2cffd2544e894ec34dcdae799a6

camreeves and others added 3 commits August 10, 2026 21:43
…earch (PPT-2644)

Adds a generated, stored tsvector column + GIN index to every table whose
rest-api index route is searchable, replacing the Elasticsearch write path.
'simple' text-search config on both sides (no stemming — entity names,
identifiers, emails; matches the guests.tsv_search precedent). Secrets and
encrypted content are deliberately excluded from every vector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Elasticsearch's analyzer segmented uri/file-path fields, so their parts
were individually searchable — PG 'simple' keeps '/path' as one lexeme.
Split on punctuation (keeping the raw value) for mod.uri,
driver.file_name/default_uri, repo.uri, shortener.uri and
signage_plugin.uri.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t migration

master's 9ff188d renumbered the AI-support migration to 20260806100500000,
the exact version this migration was using. Micrate versions must be
unique, and the search-vector migration must sort after everything on
master, so it moves to 20260810100500000.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the type: enhancement new feature or request label Aug 11, 2026
@stakach

stakach commented Aug 11, 2026

Copy link
Copy Markdown
Member

LGTM

@stakach
stakach merged commit 6d3a9ad into master Aug 11, 2026
8 of 9 checks passed
@stakach
stakach deleted the PPT-2644-pg-full-text-search branch August 11, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: enhancement new feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants