Skip to content

Support contains, startswith and endswith in $filter and database policies - #3858

Open
Richard Cadogan (arkin-nz) wants to merge 3 commits into
Azure:mainfrom
arkin-nz:feature/policy-string-functions
Open

Richard Cadogan (arkin-nz) wants to merge 3 commits into
Azure:mainfrom
arkin-nz:feature/policy-string-functions

Conversation

@arkin-nz

@arkin-nz Richard Cadogan (arkin-nz) commented Oct 13, 2026 •

Copy link
Copy Markdown

Why make this change?

What is this change?

  • SQL (ODataASTVisitor): contains, startswith and endswith are translated to column LIKE @param. The search value must resolve to a non-empty string: a constant, a bound claim, or a concat of those. It is resolved in C#, LIKE wildcards are escaped with the same rules as the GraphQL filter (the escaping is moved into a shared SqlLikePattern), and the whole pattern is bound as one parameter with the length override GraphQL uses for LIKE. No claim value reaches the SQL text.
  • Cosmos DB (ODataASTCosmosVisitor): the same functions map to the native CONTAINS/STARTSWITH/ENDSWITH(c.col, @param).
  • Claim binding (ParameterAliasRewriter): recurses into function arguments so claim aliases inside them get bound, and treats a function call as a predicate so it isn't rewritten as ... eq true.
  • Fail closed: the first argument must be a field. Other shapes (other functions such as tolower, field-to-field, a claim as the searched value, a null or empty value) throw. A non-string claim, such as a numeric ID, used in one of these functions gets a clear error saying claims must be strings. Policy failures surface as the existing 403 AuthorizationCheckFailed for both SQL and Cosmos DB; $filter failures as the existing 400 "$filter query parameter is not well formed." Comparing a function's result (startswith(title, 'A') eq true) is also rejected rather than translated, so FindTestWithUnsupportedFilterKeywords keeps passing unchanged.
  • LIKE rendering: the visitor renders LIKE through a new IQueryBuilder.BuildLikePredicate, so each dialect decides how to escape. SQL Server adds ESCAPE '\', which covers mutations as well as reads, since database policies also apply to insert, update, delete and upsert. DWSQL has no ESCAPE support yet, so it rejects these functions. MySQL and PostgreSQL use plain LIKE, where \ is already the default escape character.
  • REST field visitor: ODataASTFieldVisitor traverses function arguments, so columns referenced inside $filter functions still go through column-level authorization.

Example policy for the use case above:

"policy": {
  "database": "contains(@item.managerIds, concat(concat('|', @claims.mgr), '|'))"
}

Out of scope, noted for follow-up:

  • DWSQL support. DwSqlQueryBuilder never adds an ESCAPE clause, not even for the existing GraphQL contains, so these functions are rejected for DWSQL until it does. I couldn't test against a warehouse.
  • Converting numeric claims to strings inside these functions.
  • Moving GraphQL filter LIKE onto BuildLikePredicate, so the SQL Server read path no longer needs its regex.
  • in lists and other OData functions are still unsupported.

How was this tested?

  • Integration Tests

  • Unit Tests

  • DatabasePolicyClaimBindingUnitTests (no database needed) covers the full claim → OData AST → SQL/Cosmos path:

    • a delimited concat claim bound as one parameter (%|123456|% for SQL, |123456| for Cosmos);
    • wildcard and quote escaping for all three functions;
    • combination with and;
    • fail-closed cases (claim as the searched value, field-to-field, null or empty value, unsupported function, function compared to true), for SQL and Cosmos DB;
    • DWSQL rejection, and a clear error for Integer32 and Integer64 claims, direct and inside concat;
    • the SQL Server read path adds ESCAPE to LIKE only once.

    These fail without the change.

  • ODataASTVisitorUnitTests (SQL Server): $filter cases for contains, startswith, endswith and concat.

  • All non-database tests pass (3,280). The TestCategory=MsSql suite was run against SQL Server 2025 in Docker. Before review, it was compared with unmodified main: 1,577 passed with this change against 1,573 without. After the review changes: 1,579 passed. In every run, the failures came from the local environment (mostly ConfigurationTests and hot-reload tests) and also happen without this change.

  • Manual end-to-end check against SQL Server, with the policy above and AppService auth carrying an mgr claim:

    • manager 123456 reads exactly their rows;
    • |91234567| (a superstring) and 1234 (a substring) don't match;
    • claim 1_3456 doesn't match |1x3456|, so the wildcard is escaped;
    • a claim containing ' returns an empty result, not an error;
    • GraphQL update and REST delete succeed on own rows and return not-found on others' rows;
    • every executed LIKE carried ESCAPE '\'.
  • PostgreSQL and MySQL were not run locally. Both use \ as the default LIKE escape character, which the existing GraphQL contains already relies on.

Sample Request(s)

Policy (as above), then:

GET /api/ManagedRecord
X-MS-API-ROLE: manager
X-MS-CLIENT-PRINCIPAL: <principal with claim mgr=123456>

returns only the rows whose managerIds contains |123456|.

REST $filter:

GET /api/Book?$filter=contains(title, 'Awesome')
GET /api/Book?$filter=startswith(title, 'Awe') and not endswith(title, 'book')

…ilter

Database policies and REST $filter only supported binary comparison
operators. OData string functions parsed but failed at request time
because ODataASTVisitor and ParameterAliasRewriter did not handle
SingleValueFunctionCallNode.

- SQL: translate contains/startswith/endswith to `column LIKE @param`.
  The search value (constant, bound claim, or concat of those) is
  resolved in C#, LIKE-escaped with the GraphQL filter's rules (now a
  shared helper) and bound as a single parameter.
- Cosmos DB: map to the native CONTAINS/STARTSWITH/ENDSWITH.
- ParameterAliasRewriter binds claim aliases inside function arguments
  and treats a function call as a predicate.
- Unsupported shapes fail closed, including comparing a function's
  result (`startswith(...) eq true`).
- MsSqlQueryBuilder adds the LIKE ESCAPE clause to insert, update,
  delete and upsert predicates, not only SELECT, since policies apply
  to those operations too.
- ODataASTFieldVisitor traverses function arguments so referenced
  columns still go through column-level authorization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRDKmtDzv5kTiSFnUxh3ow
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@arkin-nz

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Arkin"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

DWSQL can evaluate authorization predicates incorrectly, and rejected Cosmos policies lack authorization-error translation.

2 open findings
What changed in this PR

Adds OData string filtering to DAB’s REST filters and database policies, partially addressing #3240 and #936.

Changes:

  • Translates contains, startswith, and endswith into parameterized SQL and Cosmos predicates.
  • Supports claim binding and constant concat, while preserving column authorization.
  • Shares LIKE escaping with GraphQL and extends SQL Server mutation escaping.
File Description
src/​Service.Tests/​UnitTests/​RelationalUpsertConcurrencyQueryBuilderTests.cs Tests upsert policy escaping.
src/​Service.Tests/​UnitTests/​ODataASTVisitorUnitTests.cs Adds string-filter translation cases.
src/​Service.Tests/​UnitTests/​DatabasePolicyClaimBindingUnitTests.cs Tests claim binding, escaping, and rejection.
src/​Core/​Resolvers/​MsSqlQueryBuilder.cs Adds escaping to mutation predicates.
src/​Core/​Parsers/​ParameterAliasRewriter.cs Binds claims inside function arguments.
src/​Core/​Parsers/​ODataStringFunctionHelper.cs Centralizes validation, folding, and escaping.
src/​Core/​Parsers/​ODataASTVisitor.cs Generates parameterized LIKE predicates.
src/​Core/​Parsers/​ODataASTFieldVisitor.cs Traverses function arguments for authorization.
src/​Core/​Parsers/​ODataASTCosmosVisitor.cs Generates native Cosmos string predicates.
src/​Core/​Models/​GraphQLFilterParsers.cs Reuses shared LIKE escaping.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Core/Parsers/ODataASTVisitor.cs Resolved
@aaronburtle aaronburtle moved this from Todo to Review In Progress in Data API builder Oct 20, 2026
@aaronburtle aaronburtle added this to the October 2026 milestone Oct 20, 2026
Comment thread src/Core/Parsers/ODataASTVisitor.cs Resolved
Comment thread src/Core/Resolvers/MsSqlQueryBuilder.cs Resolved
Comment thread src/Core/Parsers/ODataASTVisitor.cs Resolved
Comment thread src/Core/Parsers/ODataASTVisitor.cs Resolved
- Render LIKE through IQueryBuilder.BuildLikePredicate so each dialect
  decides how to escape. SQL Server adds ESCAPE, DWSQL rejects the
  functions since it has no ESCAPE support yet. This replaces the
  regex wrapping added to the mutation builders.
- Skip LIKE clauses that already have ESCAPE in the SQL Server read
  path regex.
- Move LIKE escaping to SqlLikePattern so GraphQL filters no longer
  depend on the OData helper. Validate the function name once.
- Reject empty search values, which matched every row.
- Return a clear 403 when a non-string claim is used in a string
  function or concat.
- Map Cosmos policy translation failures to AuthorizationCheckFailed
  and map Cosmos function names explicitly.
- Resolve the backing column once in the LIKE visitor.
@aaronburtle

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 6 pipeline(s).

@aaronburtle aaronburtle left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for quickly addressing the comments, looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Review In Progress

Development

Successfully merging this pull request may close these issues.

4 participants