Repository navigation
feat(rest): support query auth for REST catalog tables - #534
Merged
Merged
Conversation
SteNicholas
force-pushed
the
PAIMON-520
branch
4 times, most recently
from
November 7, 2026 06:43
b0fb5cd to
58c299b
Compare
lucasfang
reviewed
Nov 9, 2026
lucasfang
left a comment
Member
There was a problem hiding this comment.
I found three paths where a reader can obtain values that the catalog rules are intended to hide. Each inline comment includes a regression case.
SteNicholas
force-pushed
the
PAIMON-520
branch
from
November 9, 2026 06:09
58c299b to
46778d1
Compare
lucasfang
reviewed
Nov 9, 2026
lucasfang
left a comment
Member
There was a problem hiding this comment.
Four P1 findings on query authorization at 46778d17.
SteNicholas
force-pushed
the
PAIMON-520
branch
from
November 9, 2026 11:53
46778d1 to
6d629cc
Compare
SteNicholas
force-pushed
the
PAIMON-520
branch
2 times, most recently
from
November 10, 2026 03:45
3206711 to
5f40cee
Compare
lucasfang
reviewed
Nov 10, 2026
lucasfang
left a comment
Member
There was a problem hiding this comment.
Two C++ query-authorization edge cases surfaced in comparison with Java Paimon.
SteNicholas
force-pushed
the
PAIMON-520
branch
2 times, most recently
from
November 10, 2026 08:22
4d194c4 to
0878ebf
Compare
Enforce the row filters and column masks a REST catalog defines for
tables with `query-auth.enabled=true`.
- RestCatalog: implement `QueryAuthCatalog::AuthTableQuery` against
`POST .../tables/{table}/auth`, with new `AuthTableQueryRequest` and
`AuthTableQueryResponse` messages. External tables get no rules.
`LoadTableSchemaAndLocation` reads a table's schema and location from
one response.
- TableQueryAuthResult: parse the predicate and transform JSON into a
row filter and NULL, FIELD_REF, CONCAT, CONCAT_WS and SUBSTRING
masks, refuse malformed or unsupported rules, and validate them
against a schema by column name, type and field id.
- QueryAuthTableScan: require the scan path to be the location the
catalog returns for the table and branch, fetch and validate the
rules and recheck that location on every plan, push down only the
row filter and the predicate conjuncts on unmasked columns, and
wrap each split in a `QueryAuthSplit`.
- QueryAuthTableRead: apply the row filter, then the masks, then the
read predicate. Bind each split to its authorized table location,
branch and field ids. Remove filtered rows from bitmap batches and
decode dictionaries, including nested children, before filtering.
- Split: serialize `QueryAuthSplit` with the table location, branch and
field ids its rules were authorized for.
- TableScan/TableRead: enable query auth from the catalog's schema of
the branch read, including one a system table path names, take a
disabled setting only for the table at the catalog's location, and
refuse paths that would bypass the rules: search, explicit read
schemas, real-time and fallback reads, global index scans and the
$files, $manifests, $binlog, $audit_log and $ro system tables. Reject
global system-table identifiers for data-table paths. `sys.partitions`
leaves out query-auth tables.
- ScanContextBuilder: add `SetReadFieldNames`; the scan and read
context builders carry branches from the `branch` option and
system table paths into the catalog identifier before requesting
table credentials.
- DataTypeJsonParser: reject a decimal precision or scale out of range
as Java does, rather than aborting on the precision.
- LiteralConverter: report a timestamp literal that does not fit int64
in its time unit instead of wrapping it; comparisons fall back to
comparing `Timestamp`s.
- Like: let `_` match any character, including a line terminator.
SteNicholas
force-pushed
the
PAIMON-520
branch
from
November 10, 2026 09:00
0878ebf to
40c33bb
Compare
Member
|
LGTM |
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #520
A REST catalog can restrict what a query on a table with
query-auth.enabled=truereturns, through row filters and column masks. This PR fetches those rules from the catalog at plan time, binds them to the table location, branch and field ids they were authorized for, carries them on each split, and applies them when the split is read. Read paths that would bypass the rules are refused, and anything the rules cannot be applied to fails closed.Catalog
RestCatalogimplementsQueryAuthCatalog::AuthTableQueryonPOST /v1/{prefix}/databases/{database}/tables/{table}/auth, using the newAuthTableQueryRequestandAuthTableQueryResponsemessages. External tables get no rules.LoadTableSchemaAndLocation()reads a table's schema and location from oneGetTableresponse, so the two never come from different generations of a recreated table.TableQueryAuthResultparses the rule JSON into a row filter andNULL,FIELD_REF,CONCAT,CONCAT_WSandSUBSTRINGmasks. It refuses malformed or unsupported rules, such as an unknown function or an out-of-range decimal type, and validates them against a schema by column name, type and field id.Scan
QueryAuthTableScanrequires the scan path to be the location the catalog keeps for the table and branch, so one table's rules are never applied to another table's files.QueryAuthSplitthat records the authorized table location, branch and field ids.Read
QueryAuthTableReadapplies the row filter, then the masks, then the read predicate. It refuses a split authorized for another location or branch, or whose authorized columns no longer match the schema being read, including on a worker without the catalog.NextBatchWithBitmap(), not only cleared in the bitmap. Behind a row filter every dictionary, nested children included, is decoded, so a dictionary cannot keep the values of dropped rows.Table entry points
TableScan::Create()andTableRead::Create()enable query auth from the catalog's schema for the branch read, which includes a branch a system table path such ast$branch_dev$filesnames.sysidentifier is accepted only with a global system table path.System tables
$files,$manifests,$binlog,$audit_logand$rosystem tables are refused on a query-auth table, because they expose raw column statistics, partition values or rows.sys.partitionsleaves out query-auth tables, whose partition values the rules may hide.Shared changes
ScanContextBuilder::SetReadFieldNames()sets the columns the catalog is asked to authorize. The scan and read context builders carry the branch from thebranchoption or a system table path into the catalog identifier before requesting table credentials.DataTypeJsonParserrejects a decimal precision or scale out of range as Java does, instead of aborting the process on the precision.LiteralConverterreports a timestamp literal that does not fit int64 in its time unit instead of wrapping it. Comparisons then fall back to comparingTimestamps, so a cutoff such as2500-01-01T00:00:00.000000001is compared correctly against aTIMESTAMP(9)column.LIKE,_matches any character, including a line terminator, as in Java.Tests
TableQueryAuthResultTest.*covers rule parsing, literal types and boundaries (including timestamp cutoffs beyond epoch nanoseconds), constant folding, masks, schema and field-id validation, and malformed or unsupported rules (includingDECIMAL(39, 0)field references).QueryAuthSplitTest.*covers deserializing the Java-written fixtures undertest/test_data/query_auth/, round trips with the authorized scope, and malformed bytes.QueryAuthTableReadTest.*covers dictionary and large-string decoding, closing readers when a split fails, andTestRowFilterRemovesRowsFromTheBatchWithBitmap. That test checks that a mixed-tenant batch read through bothCreateReader()overloads holds only the authorized tenant's rows, and no dictionary at any depth.TableTest.TestQueryAuth*plans and reads through a mock query-auth catalog. It covers branches, schema changes, pruning and limit pushdown, primary-key, deletion-vector and data-evolution tables, Parquet dictionary columns, and refused read paths. It also covers:TestQueryAuthScanReadsOnlyTheLocationOfTheAuthorizedTable: another table's path, and an identifier rebound before the first plan;TestQueryAuthReadRejectsSplitsAuthorizedForAnotherTableand...AnotherBranch: splits read against another table or branch;TestQueryAuthSystemTablesValidateTheBranchRead: system tables on a branch whose catalog schema alone enables query auth;TestCatalogDisablesQueryAuthOnlyForTheTableAtItsLocation: a disabled setting borrowed by another path.RestCatalogTest.*QueryAuth*,RestCatalogTest.AuthTableQuery*,RestCatalogTest.GlobalPartitionsProtectsQueryAuthPartitionValues,RestApiTest.AuthTableQuery,RestMessagesTest.AuthTableQuery*andResourcePathsTest.*cover the endpoint, the messages, retries, error mapping andsysidentifiers.ScanContextTestandReadContextTestcasesTestWithCatalogResolvesTheBranchandTestSystemTablePathResolvesBranchBeforeCatalogRequest.SchemaValidationTest.TestQueryAuth,FormatTableTest.TestQueryAuthInTheSchemaCannotBeDisabledAtTheCall,CoreOptionsTest.TestDefaultValueandCoreOptionsTest.TestFromMap.PredicateTest.TestLikeWildcardsMatchLineTerminatorsandLiteralConverterTest.TestLiteralsToArrayRejectsTimestampsOutOfRange.RapidJsonUtilTest.TestRejectUnescapedNul,TestSerializeStringWithEmbeddedNulandTestCheckNoDuplicateMembers, plusDataTypeJsonParserTest.ParseTypeRejectsEmbeddedNulandParseTypeAtomicTypeSuccess(decimal precision and scale bounds).SystemTableReadInteTest.TestReadGlobalPartitionschecks thatsys.partitionsleaves out a query-auth table. TheBlobTableInteTestcases that read a missing upstream table now expectNotExist.API and Format
Options::QUERY_AUTH_ENABLED(query-auth.enabled),ScanContextBuilder::SetReadFieldNames(),ScanContext::GetReadFieldNames()and aScanContextconstructor that takes the read field names.WithCatalog()docs of both context builders and the predicate-filter note inread_context.h.Invalid, and so does asysidentifier with a data-table path.$manifestsis refused on a query-auth table, andsys.partitionsno longer lists query-auth tables.QueryAuthSplitwith the table location, branch and field ids its rules were authorized for. Other splits, and the table's data and metadata files, are unchanged.POST .../tables/{table}/auth.Documentation
docs/source/user_guide/catalog.rstdocuments the option, the supported rules, how a query-auth table is planned and read, the required scan location, the refused read paths, and the limits of query-auth splits across runtimes.Generative AI tooling
Generated-by: Claude Code (2.1.311, Claude Opus 5.5)
🤖 Generated with Claude Code