Skip to content

feat(rest): support query auth for REST catalog tables - #534

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-520
Nov 10, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-520

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #520

A REST catalog can restrict what a query on a table with query-auth.enabled=true returns, through row filters and column masks. This PR fetches those rules from the catalog at plan time, binds them to the table location, branch and field ids they were authorized for, carries them on each split, and applies them when the split is read. Read paths that would bypass the rules are refused, and anything the rules cannot be applied to fails closed.

Catalog

  • RestCatalog implements QueryAuthCatalog::AuthTableQuery on POST /v1/{prefix}/databases/{database}/tables/{table}/auth, using the new AuthTableQueryRequest and AuthTableQueryResponse messages. External tables get no rules. LoadTableSchemaAndLocation() reads a table's schema and location from one GetTable response, so the two never come from different generations of a recreated table.
  • TableQueryAuthResult parses the rule JSON into a row filter and NULL, FIELD_REF, CONCAT, CONCAT_WS and SUBSTRING masks. It refuses malformed or unsupported rules, such as an unknown function or an out-of-range decimal type, and validates them against a schema by column name, type and field id.

Scan

  • QueryAuthTableScan requires the scan path to be the location the catalog keeps for the table and branch, so one table's rules are never applied to another table's files.
  • On every plan it fetches the rules, rechecks that location and the field ids of the selected and rule columns, and pushes down only the row filter and the predicate conjuncts on unmasked columns. Each split is wrapped in a QueryAuthSplit that records the authorized table location, branch and field ids.

Read

  • QueryAuthTableRead applies the row filter, then the masks, then the read predicate. It refuses a split authorized for another location or branch, or whose authorized columns no longer match the schema being read, including on a worker without the catalog.
  • Rows the row filter drops are removed from the batch itself, including from NextBatchWithBitmap(), not only cleared in the bitmap. Behind a row filter every dictionary, nested children included, is decoded, so a dictionary cannot keep the values of dropped rows.

Table entry points

  • TableScan::Create() and TableRead::Create() enable query auth from the catalog's schema for the branch read, which includes a branch a system table path such as t$branch_dev$files names.
  • A catalog setting that leaves query auth disabled is taken only for the table at the catalog's location, and a sys identifier is accepted only with a global system table path.
  • Search, explicit read schemas, real-time and fallback reads, and global index scans are refused on a query-auth table.

System tables

  • The $files, $manifests, $binlog, $audit_log and $ro system tables are refused on a query-auth table, because they expose raw column statistics, partition values or rows.
  • sys.partitions leaves out query-auth tables, whose partition values the rules may hide.

Shared changes

  • ScanContextBuilder::SetReadFieldNames() sets the columns the catalog is asked to authorize. The scan and read context builders carry the branch from the branch option or a system table path into the catalog identifier before requesting table credentials.
  • DataTypeJsonParser rejects a decimal precision or scale out of range as Java does, instead of aborting the process on the precision.
  • LiteralConverter reports a timestamp literal that does not fit int64 in its time unit instead of wrapping it. Comparisons then fall back to comparing Timestamps, so a cutoff such as 2500-01-01T00:00:00.000000001 is compared correctly against a TIMESTAMP(9) column.
  • In LIKE, _ matches any character, including a line terminator, as in Java.

Tests

  • Rules: TableQueryAuthResultTest.* covers rule parsing, literal types and boundaries (including timestamp cutoffs beyond epoch nanoseconds), constant folding, masks, schema and field-id validation, and malformed or unsupported rules (including DECIMAL(39, 0) field references).
  • Splits: QueryAuthSplitTest.* covers deserializing the Java-written fixtures under test/test_data/query_auth/, round trips with the authorized scope, and malformed bytes.
  • Read: QueryAuthTableReadTest.* covers dictionary and large-string decoding, closing readers when a split fails, and TestRowFilterRemovesRowsFromTheBatchWithBitmap. That test checks that a mixed-tenant batch read through both CreateReader() overloads holds only the authorized tenant's rows, and no dictionary at any depth.
  • End to end: TableTest.TestQueryAuth* plans and reads through a mock query-auth catalog. It covers branches, schema changes, pruning and limit pushdown, primary-key, deletion-vector and data-evolution tables, Parquet dictionary columns, and refused read paths. It also covers:
    • TestQueryAuthScanReadsOnlyTheLocationOfTheAuthorizedTable: another table's path, and an identifier rebound before the first plan;
    • TestQueryAuthReadRejectsSplitsAuthorizedForAnotherTable and ...AnotherBranch: splits read against another table or branch;
    • TestQueryAuthSystemTablesValidateTheBranchRead: system tables on a branch whose catalog schema alone enables query auth;
    • TestCatalogDisablesQueryAuthOnlyForTheTableAtItsLocation: a disabled setting borrowed by another path.
  • REST: RestCatalogTest.*QueryAuth*, RestCatalogTest.AuthTableQuery*, RestCatalogTest.GlobalPartitionsProtectsQueryAuthPartitionValues, RestApiTest.AuthTableQuery, RestMessagesTest.AuthTableQuery* and ResourcePathsTest.* cover the endpoint, the messages, retries, error mapping and sys identifiers.
  • Supporting changes:
    • Context builders: ScanContextTest and ReadContextTest cases TestWithCatalogResolvesTheBranch and TestSystemTablePathResolvesBranchBeforeCatalogRequest.
    • Option handling: SchemaValidationTest.TestQueryAuth, FormatTableTest.TestQueryAuthInTheSchemaCannotBeDisabledAtTheCall, CoreOptionsTest.TestDefaultValue and CoreOptionsTest.TestFromMap.
    • Predicates and literals: PredicateTest.TestLikeWildcardsMatchLineTerminators and LiteralConverterTest.TestLiteralsToArrayRejectsTimestampsOutOfRange.
    • JSON and type parsing: RapidJsonUtilTest.TestRejectUnescapedNul, TestSerializeStringWithEmbeddedNul and TestCheckNoDuplicateMembers, plus DataTypeJsonParserTest.ParseTypeRejectsEmbeddedNul and ParseTypeAtomicTypeSuccess (decimal precision and scale bounds).
  • Integration: SystemTableReadInteTest.TestReadGlobalPartitions checks that sys.partitions leaves out a query-auth table. The BlobTableInteTest cases that read a missing upstream table now expect NotExist.

API and Format

  • Public API:
    • Adds Options::QUERY_AUTH_ENABLED (query-auth.enabled), ScanContextBuilder::SetReadFieldNames(), ScanContext::GetReadFieldNames() and a ScanContext constructor that takes the read field names.
    • Updates the WithCatalog() docs of both context builders and the predicate-filter note in read_context.h.
  • Behavior:
    • A scan or read whose path is not the catalog's location for the table and branch fails with Invalid, and so does a sys identifier with a data-table path.
    • $manifests is refused on a query-auth table, and sys.partitions no longer lists query-auth tables.
    • A decimal type with a precision or scale out of range now fails to parse rather than aborting or being accepted.
  • Split serialization: a split of a query-auth table serializes as a QueryAuthSplit with the table location, branch and field ids its rules were authorized for. Other splits, and the table's data and metadata files, are unchanged.
  • Protocol: the REST catalog calls POST .../tables/{table}/auth.

Documentation

docs/source/user_guide/catalog.rst documents the option, the supported rules, how a query-auth table is planned and read, the required scan location, the refused read paths, and the limits of query-auth splits across runtimes.

Generative AI tooling

Generated-by: Claude Code (2.1.311, Claude Opus 5.5)

🤖 Generated with Claude Code

@SteNicholas
SteNicholas force-pushed the PAIMON-520 branch 4 times, most recently from b0fb5cd to 58c299b Compare November 7, 2026 06:43

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found three paths where a reader can obtain values that the catalog rules are intended to hide. Each inline comment includes a regression case.

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four P1 findings on query authorization at 46778d17.

Comment thread src/paimon/core/table/table.cpp Resolved
Comment thread src/paimon/core/table/table.cpp Resolved

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two C++ query-authorization edge cases surfaced in comparison with Java Paimon.

@SteNicholas
SteNicholas force-pushed the PAIMON-520 branch 2 times, most recently from 4d194c4 to 0878ebf Compare November 10, 2026 08:22
Enforce the row filters and column masks a REST catalog defines for
tables with `query-auth.enabled=true`.

- RestCatalog: implement `QueryAuthCatalog::AuthTableQuery` against
  `POST .../tables/{table}/auth`, with new `AuthTableQueryRequest` and
  `AuthTableQueryResponse` messages. External tables get no rules.
  `LoadTableSchemaAndLocation` reads a table's schema and location from
  one response.
- TableQueryAuthResult: parse the predicate and transform JSON into a
  row filter and NULL, FIELD_REF, CONCAT, CONCAT_WS and SUBSTRING
  masks, refuse malformed or unsupported rules, and validate them
  against a schema by column name, type and field id.
- QueryAuthTableScan: require the scan path to be the location the
  catalog returns for the table and branch, fetch and validate the
  rules and recheck that location on every plan, push down only the
  row filter and the predicate conjuncts on unmasked columns, and
  wrap each split in a `QueryAuthSplit`.
- QueryAuthTableRead: apply the row filter, then the masks, then the
  read predicate. Bind each split to its authorized table location,
  branch and field ids. Remove filtered rows from bitmap batches and
  decode dictionaries, including nested children, before filtering.
- Split: serialize `QueryAuthSplit` with the table location, branch and
  field ids its rules were authorized for.
- TableScan/TableRead: enable query auth from the catalog's schema of
  the branch read, including one a system table path names, take a
  disabled setting only for the table at the catalog's location, and
  refuse paths that would bypass the rules: search, explicit read
  schemas, real-time and fallback reads, global index scans and the
  $files, $manifests, $binlog, $audit_log and $ro system tables. Reject
  global system-table identifiers for data-table paths. `sys.partitions`
  leaves out query-auth tables.
- ScanContextBuilder: add `SetReadFieldNames`; the scan and read
  context builders carry branches from the `branch` option and
  system table paths into the catalog identifier before requesting
  table credentials.
- DataTypeJsonParser: reject a decimal precision or scale out of range
  as Java does, rather than aborting on the precision.
- LiteralConverter: report a timestamp literal that does not fit int64
  in its time unit instead of wrapping it; comparisons fall back to
  comparing `Timestamp`s.
- Like: let `_` match any character, including a line terminator.
@lucasfang

Copy link
Copy Markdown
Member

LGTM

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 7515bf6 into apache:main Nov 10, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support query auth for REST catalog

3 participants