fix(drivers): text-operator case folding is the contract's answer, not the dialect's (#6518)

.changeset/text-operator-case-folding-is-contractual.md

@@ -0,0 +1,86 @@
+---
+"@objectstack/driver-sql": minor
+"@objectstack/driver-sqlite-wasm": minor
+"@objectstack/driver-turso": minor
+---
+
+fix(drivers): text-operator case folding is the CONTRACT's answer, not the dialect's (#6518)
+
+The `$contains` family and `$icontains` returned **different rows on different
+databases** for the same filter, because case sensitivity was decided by whatever
+`LIKE` happened to mean on the dialect underneath. Both directions **over-matched**
+— they returned rows the filter excludes, which on an ADR-0021 RLS read scope is
+over-reach rather than a loose filter (#3948):
+
+| | `$contains` / `$notContains` / `$startsWith` / `$endsWith` — case-SENSITIVE (#4706 Q2 = A) | `$icontains` — folds ASCII ONLY (#4706 Q1 = A) |
+|:--|:--|:--|
+| SQLite / turso / sqlite-wasm | ❌ `LIKE` folds ASCII | ✅ `lower()` is ASCII-only |
+| Postgres | ✅ `LIKE` is case-exact | ❌ `LOWER()` folds all of Unicode |
+| MySQL | ❌ follows the column's collation | ❌ `LOWER()` folds all of Unicode |
+
+Read across: **each dialect was already right on the half another one got wrong**,
+which is why neither half could be found from one backend alone.
+
+## What now runs
+
+The construct is chosen per dialect, in one emitter, so the escaping and the fold
+stay a single code path (an unescaped wildcard is a filter bypass, P0 — #5567):
+
+- **SQLite family → `GLOB`.** `LIKE`'s ASCII fold cannot be switched off per
+  statement (`PRAGMA case_sensitive_like` is connection-global, so one query would
+  redefine every other query on the connection), and `CAST(col AS BLOB) LIKE ?` was
+  measured to match *nothing at all*. `GLOB` is case-exact and brings its own
+  escaped class — `*`, `?`, `[` as the self-closing classes `[*]`, `[?]`, `[[]`,
+  because SQLite's grammar gives `GLOB` no `ESCAPE` clause. `$icontains` keeps
+  `lower()` on both operands, still ASCII-only.
+- **Postgres → `LIKE`, unchanged.** Only the fold moved, from `LOWER()` to an
+  explicit `translate()` over the 26 ASCII letters. Measured on a live PostgreSQL
+  16 (ICU database): `LOWER('CAFÉ')` is `'café'` — the over-fold — while the
+  `translate()` form leaves `É` alone.
+- **MySQL → `LIKE` over `CAST(… AS BINARY)`**, so the comparison is byte-wise and
+  no collation decides the case; `$icontains` folds byte-wise over the same binary
+  rendering, which is ASCII-only because UTF-8 is self-synchronising.
+- **Any other client** keeps the previous `LIKE` / `LOWER()` shape — it is the only
+  form that still runs there — and is recorded as residue rather than left to be
+  discovered.
+
+`driver-turso`'s remote transport carries the twin (it compiles filters itself and
+inherits nothing), and the two transports are now held to the same rows by a
+parity suite that runs the shared `FILTER_TEXT_CASES` on both.
+
+## Behaviour change — read this before upgrading
+
+A filter whose comparand's case did not match the stored text used to match on
+SQLite/turso/sqlite-wasm and may have matched on MySQL. It no longer does:
+
+```ts
+// rows: { id: '1', name: 'ACME Corp' }, { id: '2', name: 'acme corp' }
+{ name: { $contains: 'acme' } }   // was ['1','2'] on SQLite → now ['2'] everywhere
+{ name: { $icontains: 'acme' } }  // ['1','2'] — unchanged, and now correct on PG/MySQL too
+{ name: { $icontains: 'café' } }  // was ['3','4'] on PG/MySQL → now ['4'] everywhere
+```
+
+If you were relying on `$contains` to ignore case, **write `$icontains`** — that is
+the operator for it, and it now folds the same ASCII-only range on every backend.
+Result sets only ever get NARROWER, never wider, so a filter that was already
+correct stays correct.
+
+## Why `minor` rather than `major`
+
+No declared surface moves. `$contains` still exists, still takes the same
+comparand, and `filter.zod.ts` is untouched — the case-sensitivity this delivers
+was **already published** as the contract by #5701 (`FILTER_TEXT_CASES`, one
+release earlier in this same v17 major), and the drivers were the half that had
+not caught up. This is Prime Directive #12 applied in the direction it points:
+declared = enforced. It is graded the way its sibling #5702/#6549 was graded for
+the same operator family in the same rc cycle, and it registers nothing in the
+ADR-0087 registries because it retires no authorable key.
+
+## What is deliberately NOT in this change
+
+`driver-memory` and `driver-mongodb` still fold case on their query paths — they
+are the #5499 frozen family, so their `FILTER_TEXT_CASES` cells stay honest DEBT
+and are tracked as #6682 (case sensitivity) and #6520 (`$icontains`). The
+`service-analytics` SQL compilers were measured already compliant: they emit
+Postgres-shaped statements, where `LIKE` is case-exact, and that assumption is now
+written down and pinned rather than implied.

packages/drivers/driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts

@@ -2,44 +2,50 @@
 
 /**
  * [#5702] `$icontains` on the SQL family, and the retirement of `$regex` /
- * `$options` — the DRIVER half of the #4706 ruling (its contract half is #5701).
+ * `$options` — the DRIVER half of the #4706 ruling (its contract half is #5701)
+ * — plus [#6518] the case-sensitivity half #5702 could not deliver.
  *
- * ## What this pins, and why it is not the shared text case-set
+ * ## This cell now answers the WHOLE shared case-set
  *
- * `@objectstack/spec/data` carries a canonical text case-set whose rows this
- * file reuses (`FILTER_TEXT_ROWS` — the same nine, so a verdict here is
- * comparable to one anywhere else). It deliberately does NOT import that
- * case-set's CASES export, because `scripts/check-driver-conformance.mjs`
- * judges a cell covered by that import and this driver does not yet answer the
- * whole table: five of its cases require the `$contains` family to be
- * case-SENSITIVE (#4706 Q2 = A), which SQLite's `LIKE` is not, and which cannot
- * be fixed in this driver alone — `read-scope-sql` and `service-analytics`
- * compile the same predicate for RLS and for the analytics face, so a
- * driver-only change would give ONE permission rule two row sets (#3948). That
- * work is filed separately and the driver's DEBT row stays open for it.
+ * `FILTER_TEXT_CASES` is imported and executed below, which is what
+ * `scripts/check-driver-conformance.mjs` reads as coverage. #5702 deliberately
+ * imported only `FILTER_TEXT_ROWS` and left the CASES export alone, because
+ * five of its cases require the `$contains` family to be case-SENSITIVE (#4706
+ * Q2 = A) and SQLite's `LIKE` folds ASCII — an import then would have flipped
+ * the cell to "covered" while five cases went unanswered, which is a gate
+ * reporting success over a standard nobody runs.
  *
- * Importing the case-set here would flip the cell to "covered" while five of
- * its cases were unanswered — a gate reporting success over a standard nobody
- * runs, which is exactly the failure the gate exists to prevent.
+ * #6518 is what makes the import honest: `textMatchPredicate` emits `GLOB` on
+ * the SQLite dialects (case-exact, carrying GLOB's own `[*]` / `[?]` / `[[]`
+ * escapes because SQLite's grammar gives it no `ESCAPE` clause), keeps `LIKE`
+ * on Postgres (already case-exact) with the `$icontains` fold moved off the
+ * Unicode-folding `LOWER()` onto an ASCII-only `translate()`, and compares over
+ * `CAST(… AS BINARY)` on MySQL. The per-dialect reasoning and the measurements
+ * behind each cell live on that function.
  *
  * ## The reverse verification, direction decided BEFORE it was run
  *
  * - **Refusal face** — predicted RED, measured RED. Restoring the deleted
  *   `case '$regex':` fallthrough makes every assertion below that reads `code`
  *   / `status` fail, because the filter compiles again and nothing throws.
- * - **`$icontains` face on SQLite** — predicted red, and the prediction needed
- *   a correction that is recorded here rather than smoothed over: deleting the
- *   `case '$icontains':` arm turns these cases red LOUDLY (the operator falls to
- *   `default:` and is refused), but deleting only the `LOWER()` fold does NOT,
- *   for any comparand. SQLite's `LIKE` folds ASCII by itself, so on this
- *   dialect the fold is unobservable in rows. The compiled-SQL case at the end
- *   is what pins it, and it is the only thing here that can.
+ * - **`$icontains` face on SQLite** — #5702 recorded that deleting only the
+ *   fold changed NO row here, because `LIKE` folded ASCII by itself, so the
+ *   compiled-SQL case was the only witness it could offer. #6518 retires that
+ *   caveat, and the retirement was predicted before it was run: under `GLOB`
+ *   the fold is load-bearing in ROWS. Measured — dropping the column-side
+ *   `lower()` turns `$icontains: 'acme'` and `$icontains: 'ACME'` alike from
+ *   `['1','2']` into `['2']`, and `$icontains: 'CAFÉ'` from `['3']` into `[]`.
+ * - **`$contains` case-sensitivity** — predicted RED on exactly the five case
+ *   rows before the change, measured RED: reverting the sqlite arm of
+ *   `textMatchPredicate` to `LIKE` fails those five case rows and NO others in
+ *   the table (10 reds in this file: the five, plus the five blocks below that
+ *   name the construct directly).
  */
 
 import type { Knex } from 'knex';
 import { describe, it, expect, beforeAll, afterAll } from 'vitest';
 import type { DriverOptions, FilterCondition } from '@objectstack/spec/data';
-import { FILTER_TEXT_ROWS } from '@objectstack/spec/data';
+import { FILTER_TEXT_CASES, FILTER_TEXT_ROWS } from '@objectstack/spec/data';
 import { SqlDriver } from './sql-driver.js';
 
 /** The error a refused filter produced — never a bare `toThrow()` (see below). */
@@ -206,26 +212,105 @@ describe('[#5702] SqlDriver — $icontains, and the retired $regex/$options', ()
     await expect(driver.find('txt', { where: { name: { $regex: 'zzz' } } }, BYPASS)).rejects.toThrow();
   });
 
-  // ── The fold, where it is actually observable on SQLite ────────────────────
+  // ── [#6518] The case-SENSITIVE family (#4706 Q2 = A) ───────────────────────
 
-  it('compiles LOWER() on both operands, and $contains on neither', async () => {
-    // On SQLite this is the ONLY witness to the fold: `LIKE` already folds ASCII
-    // here, so `$contains` and `$icontains` select identical rows for every
-    // comparand and a dropped `LOWER()` changes no answer. It changes the SQL,
-    // and it changes the answer on Postgres (whose LIKE is case-exact), so the
-    // statement is what has to be pinned.
+  it('$contains is case-SENSITIVE in both directions', async () => {
+    // The defect this closes: on `origin/main` both lines answered ['1','2'],
+    // because SQLite's `LIKE` folds ASCII whatever the contract says. Over-
+    // matching, not a near miss — the caller asked for one row and got two.
+    expect(await ids({ name: { $contains: 'acme' } })).toEqual(['2']);
+    expect(await ids({ name: { $contains: 'ACME' } })).toEqual(['1']);
+  });
+
+  it('the case rule holds under negation too', async () => {
+    // Row 1 is EXCLUDED from the negation only if the positive form excluded
+    // it. A folding backend drops row 1 here and looks "stricter", which is the
+    // reading that hides the defect.
+    expect(await ids({ name: { $notContains: 'acme' } }))
+      .toEqual(['1', '3', '4', '5', '6', '7', '8', '9']);
+  });
+
+  it('$startsWith and $endsWith are case-SENSITIVE', async () => {
+    expect(await ids({ name: { $startsWith: 'ACME' } })).toEqual(['1']);
+    expect(await ids({ name: { $endsWith: 'corp' } })).toEqual(['2']);
+  });
+
+  it('$contains and $icontains no longer answer identically on SQLite', async () => {
+    // #5702 measured that they DID, for every comparand, and recorded it as the
+    // reason `$icontains` was unobservable on this dialect. That is the fact
+    // #6518 changes, so it is asserted rather than left as a comment: the two
+    // operators are now distinguishable in rows on the dialect where they were
+    // not.
+    expect(await ids({ name: { $contains: 'acme' } })).not
+      .toEqual(await ids({ name: { $icontains: 'acme' } }));
+  });
+
+  // ── [#6518] GLOB's metacharacters are escaped, exactly as LIKE's were ──────
+
+  for (const [label, comparand] of [
+    ['*', 'a*b'],
+    ['?', 'a?b'],
+    ['[', 'a[b'],
+  ] as const) {
+    it(`treats "${label}" as a literal character, not a GLOB metacharacter`, async () => {
+      // The new operator brings a NEW escaped character class, and an
+      // unescaped `*` is the same filter bypass an unescaped `%` is under LIKE:
+      // measured on this fixture, the unescaped pattern `*a*b*` returns rows
+      // 7, 8 and 9 where the escaped one returns none of them. No fixture row
+      // holds these characters, so the observable claim is that the pattern
+      // stays literal and selects nothing rather than expanding.
+      expect(await ids({ name: { $contains: comparand } })).toEqual([]);
+      expect(await ids({ name: { $icontains: comparand } })).toEqual([]);
+    });
+  }
+
+  it('compiles the case-exact SQLite construct, on both operators', async () => {
     // Identifier quoting is the dialect's (knex renders backticks on the sqlite
-    // clients), so the assertion is on the SHAPE, not on one dialect's quotes.
-    const unquote = (sql: string) => sql.replace(/[`"\[\]]/g, '');
+    // clients), so the assertion is on the SHAPE. `[` / `]` are NOT stripped
+    // here the way #5702's version stripped them: under GLOB they are the
+    // escape mechanism, so erasing them would erase what is being pinned.
+    const unquote = (sql: string) => sql.replace(/[`"]/g, '');
 
     const icontainsSql = unquote(driver.compileWhere({ name: { $icontains: 'acme' } }));
-    expect(icontainsSql).toContain('LOWER(name) LIKE LOWER(');
-    expect(icontainsSql).toContain('ESCAPE');
-    // The escaped pattern still travels as the comparand, wildcards and all.
-    expect(icontainsSql).toContain('%acme%');
+    expect(icontainsSql).toContain('lower(name) GLOB lower(');
+    expect(icontainsSql).toContain('*acme*');
+    // GLOB has no ESCAPE clause in SQLite's grammar; emitting one is a syntax
+    // error, so its absence is part of the construct rather than a detail.
+    expect(icontainsSql).not.toContain('ESCAPE');
 
     const containsSql = unquote(driver.compileWhere({ name: { $contains: 'acme' } }));
-    expect(containsSql).toContain('name LIKE');
-    expect(containsSql).not.toContain('LOWER');
+    expect(containsSql).toContain('name GLOB');
+    expect(containsSql).not.toContain('lower');
+    expect(containsSql).not.toContain('LIKE');
+  });
+
+  // ── The shared standard, executed ──────────────────────────────────────────
+
+  /**
+   * [#6518] Every case in `FILTER_TEXT_CASES`, on the live SQLite cell.
+   *
+   * The invariant this enforces is the case-set's own: a backend answers with
+   * the SAME ROW SET, or REFUSES with `INVALID_FILTER` — never a third, quieter
+   * answer. So the rejection cases assert `code` AND `status` (ADR-0112) plus
+   * the prescription the message must carry, and never a bare `toThrow()`.
+   *
+   * The blocks above are not redundant with this loop: they name the SQLite
+   * construct and its escaped class, which the shared table deliberately does
+   * not know about — it encodes the contract, not any dialect's spelling.
+   */
+  describe('FILTER_TEXT_CASES — the shared standard', () => {
+    for (const testCase of FILTER_TEXT_CASES) {
+      it(testCase.name, async () => {
+        if (testCase.expectRejection) {
+          const err = await refusalOf(testCase.filter);
+          expect(err.code).toBe(testCase.code);
+          expect(err.status).toBe(400);
+          for (const mention of testCase.mustMention) expect(err.message).toContain(mention);
+          return;
+        }
+        expect(await ids(testCase.filter), testCase.note ?? testCase.name)
+          .toEqual([...testCase.expected]);
+      });
+    }
   });
 });