Repository navigation
Conversation
Introduce TableLoadOptions for static factories and keep operation Contexts limited to execution settings. Apply dynamic options through Table::Copy while keeping NewScan/NewRead/NewWrite/NewCommit context-only. Preserve catalog ownership, storage bindings, schema caches and query-auth behavior, and migrate callers, tests, examples and documentation. Refs apache#555 BREAKING CHANGE: Context builders no longer accept table loading parameters or option maps. Static factories take TableLoadOptions and dynamic options; loaded-table option overrides use Table::Copy.
There was a problem hiding this comment.
@lucasfang, thanks for updates. I left some comments for udpates. PTAL.
| read_options, CatalogUtils::ResolveQueryAuthReadOptions( | ||
| context->GetCatalog(), context->GetIdentifier(), context->GetOptions(), | ||
| CurrentBranch(), file_system_, root_path_, true, "scan")); | ||
| catalog_, identifier_, dynamic_options, CurrentBranch(), file_system_, |
There was a problem hiding this comment.
Use the retained catalog to resolve scan snapshots too. Writers and committers created from a catalog-loaded table now use catalog_ and can publish a snapshot only through the catalog, but NewScan() still reaches the filesystem-only GetSnapshotManager() through both managed scan implementations. Consequently, writing and committing one row through a version-managed catalog and then calling NewScan() on the same table returns an empty plan (SnapshotId() == nullopt although the catalog accepted snapshot 1); an existing filesystem snapshot instead produces stale results. Please supply the catalog-aware snapshot manager to the scan path as well, and add a commit-then-scan regression covering a snapshot that has not been published to the filesystem.
| TableSchema::CreateFromJson(*supplied_main_schema)); | ||
| schema = std::shared_ptr<TableSchema>(std::move(supplied)); | ||
| } else { | ||
| PAIMON_ASSIGN_OR_RAISE(schema, manager.Latest()); |
There was a problem hiding this comment.
Resolve the fallback schema through the bound catalog when metadata is catalog-only. The new catalog-only loading path can successfully load dev with no filesystem schema metadata, but CreateForFallbackBranch() ignores its catalog argument here and always uses manager.Latest() unless a main schema was explicitly supplied. A catalog-only load with identifier tbl$branch_dev and scan.fallback-branch=main therefore fails during TableRead::Create() with fallback schema not found, even when the catalog serves the main schema. A stale filesystem fallback schema is also used in preference to the catalog's schema. Please resolve the fallback branch's identifier and schema from the catalog while preserving the explicit supplied-main-schema precedence, and cover this combination in the catalog-only load tests.
|
|
||
| TEST(ReadContextTest, TestPrefetchMaxParallelNumZero) { | ||
| ReadContextBuilder builder("table_root_path"); | ||
| ReadContextBuilder builder; |
There was a problem hiding this comment.
Restore the assertion that calls Finish(). This test now only sets the prefetch fields and never executes the validation it is named for, so it passes even if zero prefetch parallelism is accepted. The check still exists in ReadContextBuilder::Finish() and is an execution-context concern, so separating table loading does not make this regression coverage obsolete. Please retain ASSERT_NOK_WITH_MSG(builder.Finish(), "prefetch max parallel num should be greater than 0").
| ``WithFileSystemSchemeToIdentifierMap()``, ``WithCatalog()`` and a branch. Options | ||
| given at the call still win over the ones the schema stored, as they do everywhere else. | ||
| The loaded table supplies its schema and file system. Dynamic options passed to | ||
| ``New*()`` override stored options. |
There was a problem hiding this comment.
Document CopyWithoutTimeTravel() as the way to apply dynamic options. The new NewScan/NewRead/NewWrite/NewCommit signatures accept only an operation context, and the context builders no longer expose SetOptions(), so callers cannot pass overrides to New*() as instructed here. Show copying the loaded table with CopyWithoutTimeTravel(overrides) and creating the operator from that copy. The same instruction in docs/source/api/format_table.rst needs updating, together with the paragraphs above about legacy format-table context constructors, which this PR removes.
Purpose
Linked issue: closes #555. Related to #459 and #533.
Operation Contexts still duplicate the table identity and storage configuration already bound to a loaded Table. Separate loading, table options and execution settings on the Table hierarchy merged in #527.
TableLoadContextand a shared loading path for the four static operator factories.NewScan/NewRead/NewWrite/NewCommit(context); dynamic options are applied throughTable::CopyWithoutTimeTravel(options)before creating operators.This implements the separation proposed in #533 directly on main, with option overrides owned by
Table::CopyWithoutTimeTravel.Tests
Regression coverage includes:
schemaorbranch, and distinct missing-table errors for catalog and path loads.API and Format
Breaking C++ API changes:
paimon::TableLoadContextand virtualTable::CopyWithoutTimeTravel(dynamic_options).WithBranch()and FormatTable-based constructors.Create(load_context, context, dynamic_options); pass{}explicitly when no overrides are needed.Table::NewXXX(context)remains context-only. Callers override options withCopyWithoutTimeTravel, then create operators from the returned table.Copies retain the loaded type, location, branch and storage binding; load another table to select another branch. The copy operation corresponds to Java's
copyWithoutTimeTravel: it retains the bound schema without resolving a historical schema. General branch switching and historical-schema time travel are outside this change.No storage-format or protocol changes.
Documentation
Update public API references, catalog/format-table guides, read/write/commit/realtime and cache/index examples, plus executable examples and benchmark callers. Document loading parameters separately from Arrow read projection and show option overrides through
Table::CopyWithoutTimeTravel.Generative AI tooling
Generated-by: OpenAI Codex 0.166.0 (GPT-6)