You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
driver-sql: a declared operator on a multiple: true (JSON array) column silently answers wrong — $in/$eq always zero rows, $nin returns the rows it was asked to EXCLUDE #7398
A declared filter operator applied to a column the operator cannot mean anything for is compiled to SQL and executed anyway, returning a wrong answer with a 200. No error, no warning, nothing for a type checker to catch.
Concretely, on a field declared multiple: true (which driver-sql stores as a JSON text column):
$in / $eq / bare equality → always zero rows (fail-closed)
$nin → returns the rows it was asked to EXCLUDE (fail-open) ⚠️
The second one is the reason I am filing this rather than logging it as a footgun.
⚠️$nin on an array column inverts the result — this is the dangerous half
{ members: { $nin: [U1] } } returned exactly the record whose members contains U1.
The mechanism is one line of SQL: the stored column value is the text ["U1","U2"], so members not in ('U1') is true — the text genuinely is not equal to that uuid. "Exclude these"
therefore compiles to "return everything".
Why this matters more than the $in case:
$in fails closed — the caller sees fewer rows than exist. Bad, silent, but narrowing.
Environment: @objectstack/*17.0.0-rc.5, SqlDriver(better-sqlite3), single-tenant dev runtime,
queries issued over the REST list face (GET /api/v1/data/:object?filter={json}).
Fixture: one record in the table. Its multi-value lookup column members holds ["U1","U2"]
(shown here as U1/U2; they are ordinary record ids).
DATABASE_ERROR (this is the #5234 face, not this one)
Control — the same object's single-value lookup column owner
filter
HTTP
rows
verdict
{owner:{$in:[U1]}}
200
1
✅
So $in itself is fine. What is broken is $inmeeting an array column.
Raw SQL, same table, same row
Reading the column and querying it directly, with no platform in the path:
stored value: ["U1","U2"]
SELECT COUNT(*) … WHERE members IN ('U1') -> 0
SELECT COUNT(*) … WHERE members LIKE '%U1%' ESCAPE '\' -> 1
That is the whole bug in two lines. driver-sql lowers $in straight to col in (?, ?, …)
(the list('in') arm of applyNormalizedComparison) without consulting the column's type,
and a JSON text column is never equal to any single member it contains. $contains "works" only
because it lowers to LIKE '%value%' and the JSON serialization happens to contain the substring.
Why I read this as a defect and not as documented behaviour
The platform already chose "refuse" on every adjacent seam — this is the one square on the grid
with no gate:
Unknown operator → 400, and the refusal even prints the supported list. Loud. Correct.
Wrong comparand shape ($in: 'done', a scalar where an array is declared) → the unreleased collection-operator-scalar-comparand-400 changeset moves this from a 500 to a 400 INVALID_FILTER, and its wording explicitly states the filter was not applied.
So writes check the field's type against the payload; reads do not check the field's type against
the operator. Given isJsonField(type, field) { return JSON_COLUMN_TYPES.has(type) || !!field.multiple }
already exists in driver-sql, the information needed for the check is present at the point of lowering.
And the failure mode is the worst available one: 200 + an empty array is byte-identical to a
successful query that legitimately matched nothing. There is no signal for a caller to key on.
What it cost downstream (why I bothered to measure it)
In a downstream business app, a delete-guard was written as:
The rule it implements is "refuse to delete a team while any of its members is still referenced by a
plan". Because the predicate matched nothing, the guard never fired once since the feature shipped.
It threw no error, logged nothing, and type-checked. It was found only when someone happened to test
the guard's positive path by hand. A full sweep of that codebase then had to be run to prove there
was no second occurrence.
The general shape: this bug class does not produce incidents, it produces rules that quietly do not
exist. That is why "the caller should know better" is a weak answer here — there is nothing for the
caller to notice.
Asks (either would fix it; the first is the smaller change)
Refuse it. Bring "declared operator × column type it cannot apply to" under the same 400 INVALID_FILTER envelope that already covers unknown operators and malformed comparands —
naming the operator, the field, and the fact that the filter was not applied. Minimum viable
set: $in / $nin / $eq / bare equality / the ordering comparisons, against field.multiple
(and the other JSON_COLUMN_TYPES) columns.
Or give array columns a real membership operator. Today $contains is declared as a string
operator ($contains: z.ZodString, in StringOperatorSchema) and works on array columns purely
because the JSON serialization is text and LIKE '%v%' happens to hit. That is incidental, not
designed — it is substring matching over a serialization, so it is also sensitive to how the
column is serialized. There is likewise no way to express "any of these values" in one query;
the only spelling is an $or of N $contains, since FILTER_OPERATORS is a closed set of 15
with no $overlaps / $containsAny.
If the ruling is "1 only", that is completely fine for me downstream — a loud refusal is all the
downstream app needs, and $or + $contains is a perfectly workable spelling. The current state
(silently wrong, and inverted for $nin) is the only outcome that cannot be worked with.
$regex is in RETIRED_FILTER_OPERATORS per the #4706 ruling, so the error text is pointing authors
at a retired operator. Not worth its own issue — just noting it since it is emitted from the same area.
Summary
A declared filter operator applied to a column the operator cannot mean anything for is compiled to SQL and executed anyway, returning a wrong answer with a 200. No error, no warning, nothing for a type checker to catch.
Concretely, on a field declared
multiple: true(whichdriver-sqlstores as a JSON text column):$in/$eq/ bare equality → always zero rows (fail-closed)$nin→ returns the rows it was asked to EXCLUDE (fail-open)The second one is the reason I am filing this rather than logging it as a footgun.
$ninon an array column inverts the result — this is the dangerous half{ members: { $nin: [U1] } }returned exactly the record whosememberscontains U1.The mechanism is one line of SQL: the stored column value is the text
["U1","U2"], somembers not in ('U1')is true — the text genuinely is not equal to that uuid. "Exclude these"therefore compiles to "return everything".
Why this matters more than the
$incase:$infails closed — the caller sees fewer rows than exist. Bad, silent, but narrowing.$ninfails open — the caller sees rows it explicitly asked to remove. Any exclusion built onit (a row-level visibility narrowing, a de-duplication pass, an "everything except the ones already
handled" sweep) silently stops filtering, and the failure direction is widening.
That is the same class A filter with an operator outside VALID_AST_OPERATORS is silently dropped, not rejected — single-condition views return unfiltered results #3948 was rated on: a dropped/inverted predicate does not degrade a feature,
it widens a result set.
Measured
Environment:
@objectstack/*17.0.0-rc.5,SqlDriver(better-sqlite3), single-tenant dev runtime,queries issued over the REST list face (
GET /api/v1/data/:object?filter={json}).Fixture: one record in the table. Its multi-value lookup column
membersholds["U1","U2"](shown here as
U1/U2; they are ordinary record ids).Multi-value (
multiple: true) column —members{members:{$in:[U1]}}{members:{$in:[U1,U2]}}{members:{$eq:U1}}{members: U1}(bare equality){members:{$contains:U1}}{members:{$contains:U2}}{$or:[{members:{$contains:U1}},{members:{$contains:U2}}]}{members:{$nin:[U1]}}{members:{$overlaps:[U1]}}{members:{$containsAny:[U1]}}{members:{$in:[[U1,U2]]}}DATABASE_ERROR(this is the #5234 face, not this one)Control — the same object's single-value lookup column
owner{owner:{$in:[U1]}}So
$initself is fine. What is broken is$inmeeting an array column.Raw SQL, same table, same row
Reading the column and querying it directly, with no platform in the path:
That is the whole bug in two lines.
driver-sqllowers$instraight tocol in (?, ?, …)(the
list('in')arm ofapplyNormalizedComparison) without consulting the column's type,and a JSON text column is never equal to any single member it contains.
$contains"works" onlybecause it lowers to
LIKE '%value%'and the JSON serialization happens to contain the substring.Why I read this as a defect and not as documented behaviour
The platform already chose "refuse" on every adjacent seam — this is the one square on the grid
with no gate:
$in: 'done', a scalar where an array is declared) → the unreleasedcollection-operator-scalar-comparand-400changeset moves this from a 500 to a 400INVALID_FILTER, and its wording explicitly states the filter was not applied.$in/$nin的非$field对象成员,与 LIKE 族的对象比较值(String 成[object Object]) #5234, filed, same "silently zero rows" reasoning.fields, fix(objectql): 标量字段的写入载荷拒收算子对象 (#5922) #6273 refuses operator objects on scalar fields.
So writes check the field's type against the payload; reads do not check the field's type against
the operator. Given
isJsonField(type, field) { return JSON_COLUMN_TYPES.has(type) || !!field.multiple }already exists in
driver-sql, the information needed for the check is present at the point of lowering.And the failure mode is the worst available one:
200+ an empty array is byte-identical to asuccessful query that legitimately matched nothing. There is no signal for a caller to key on.
What it cost downstream (why I bothered to measure it)
In a downstream business app, a delete-guard was written as:
The rule it implements is "refuse to delete a team while any of its members is still referenced by a
plan". Because the predicate matched nothing, the guard never fired once since the feature shipped.
It threw no error, logged nothing, and type-checked. It was found only when someone happened to test
the guard's positive path by hand. A full sweep of that codebase then had to be run to prove there
was no second occurrence.
The general shape: this bug class does not produce incidents, it produces rules that quietly do not
exist. That is why "the caller should know better" is a weak answer here — there is nothing for the
caller to notice.
Asks (either would fix it; the first is the smaller change)
400 INVALID_FILTERenvelope that already covers unknown operators and malformed comparands —naming the operator, the field, and the fact that the filter was not applied. Minimum viable
set:
$in/$nin/$eq/ bare equality / the ordering comparisons, againstfield.multiple(and the other
JSON_COLUMN_TYPES) columns.$containsis declared as a stringoperator (
$contains: z.ZodString, inStringOperatorSchema) and works on array columns purelybecause the JSON serialization is text and
LIKE '%v%'happens to hit. That is incidental, notdesigned — it is substring matching over a serialization, so it is also sensitive to how the
column is serialized. There is likewise no way to express "any of these values" in one query;
the only spelling is an
$orof N$contains, sinceFILTER_OPERATORSis a closed set of 15with no
$overlaps/$containsAny.If the ruling is "1 only", that is completely fine for me downstream — a loud refusal is all the
downstream app needs, and
$or+$containsis a perfectly workable spelling. The current state(silently wrong, and inverted for
$nin) is the only outcome that cannot be worked with.Refs
$in/$nin的非$field对象成员,与 LIKE 族的对象比较值(String 成[object Object]) #5234 — closest sibling:$in/$ninlist members that are objects, also silently zero rows.Different face (comparand vs column), same family.
collection-operator-scalar-comparand-400— comparand shape; its own textnotes member typing is "another face (driver-sql:两类无意义比较对象仍编译成「静默空谓词」——
$in/$nin的非$field对象成员,与 LIKE 族的对象比较值(String 成[object Object]) #5234)". Column typing is a third face, covered by neither.filterJSON 被静默忽略 —— 返回未过滤整页(#4134/#4164 家族第三员) #4181 / fix(data): a filter the server cannot apply is rejected, not silently ignored (#4181) #4209, ObjectQL silently drops unsupported predicate keys;findOnethen returns the first row #4419,$regexon driver-sql is not a regex — it compiles to a substring LIKE, so it both over-matches and silently matches nothing #4706 — the "silently wrong filter answer" family, and the reasoningthat a widening failure outranks a narrowing one.
Minor, same code path, mentioned only so it is not lost
The refusal message for an unknown operator advertises
$regexas supported:$regexis inRETIRED_FILTER_OPERATORSper the #4706 ruling, so the error text is pointing authorsat a retired operator. Not worth its own issue — just noting it since it is emitted from the same area.