Skip to content

refactor(context): separate table loading from execution settings - #556

Open
lucasfang wants to merge 6 commits into
apache:mainfrom
lucasfang:codex/context-options-main
Open

lucasfang wants to merge 6 commits into
apache:mainfrom
lucasfang:codex/context-options-main

Conversation

@lucasfang

@lucasfang lucasfang commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: closes #555. Related to #459 and #533.

Operation Contexts still duplicate the table identity and storage configuration already bound to a loaded Table. Separate loading, table options and execution settings on the Table hierarchy merged in #527.

  • Add TableLoadContext and a shared loading path for the four static operator factories.
  • Make operation Contexts execution-only. Loaded tables expose NewScan/NewRead/NewWrite/NewCommit(context); dynamic options are applied through Table::CopyWithoutTimeTravel(options) before creating operators.
  • Keep the original table unchanged when copying, retain catalog ownership, UUID, storage routing and schema caches, and reject conflicting type or branch overrides. Apply Java-compatible immutable-option and conditional-change checks to managed tables and their audit/binlog/read-optimized views, including normalized key no-ops and chained copies. Materialize Java-compatible scan-mode defaults in copied options so chained copies retain the inferred mode unless explicitly overridden.
  • Preserve query authorization, fallback reads, external format-table directory semantics and catalog missing-table errors. Catalog-only loads retain the schema resolved by the catalog for the requested branch, even when filesystem schema metadata is missing or stale.
  • Migrate internal callers, tests, examples, benchmarks and documentation. Most changed files are call-site migrations.

This implements the separation proposed in #533 directly on main, with option overrides owned by Table::CopyWithoutTimeTravel.

Tests

Regression coverage includes:

  • Operation Context construction without table metadata I/O, invalid loading combinations, catalog path/UUID checks and explicit filesystem precedence.
  • Copy independence and chained option precedence, catalog lifetime, storage routing and warmed schema-cache reuse. Regressions cover bucket-key changes that could silently prune matching records, immutable options, conditional bucket/deletion-vector/boolean changes, normalized key no-ops, scan-mode default precedence and retention across chained copies, and all four static factories enforcing the same checks.
  • Catalog-only main and named-branch loads with absent or stale filesystem schemas, supplied main schemas used only for main/fallback, and REST catalog branch-schema loading.
  • Managed, format and system-table operators, including audit/binlog sequence-number output from copied tables.
  • Query-auth behavior, fallback schema/storage resolution, external format partitions named schema or branch, and distinct missing-table errors for catalog and path loads.
  • Migrated core, REST, read/write, Blob, index and compaction integration coverage. Tests that require particular bucket, row-tracking, data-evolution or merge-engine layouts now declare them in the loaded/initial schema instead of changing immutable runtime options; their original read, commit, recovery and conflict assertions are retained.

API and Format

Breaking C++ API changes:

  • Add exported paimon::TableLoadContext and virtual Table::CopyWithoutTimeTravel(dynamic_options).
  • Scan/Read Context builders take no table-loading arguments; Write/Commit builders take the commit user. Remove operation Context paths, table schemas, filesystems, catalogs, option maps, WithBranch() and FormatTable-based constructors.
  • Static factories become Create(load_context, context, dynamic_options); pass {} explicitly when no overrides are needed.
  • Table::NewXXX(context) remains context-only. Callers override options with CopyWithoutTimeTravel, then create operators from the returned table.

Copies retain the loaded type, location, branch and storage binding; load another table to select another branch. The copy operation corresponds to Java's copyWithoutTimeTravel: it retains the bound schema without resolving a historical schema. General branch switching and historical-schema time travel are outside this change.

No storage-format or protocol changes.

Documentation

Update public API references, catalog/format-table guides, read/write/commit/realtime and cache/index examples, plus executable examples and benchmark callers. Document loading parameters separately from Arrow read projection and show option overrides through Table::CopyWithoutTimeTravel.

Generative AI tooling

Generated-by: OpenAI Codex 0.166.0 (GPT-6)

Introduce TableLoadOptions for static factories and keep operation Contexts
limited to execution settings. Apply dynamic options through Table::Copy
while keeping NewScan/NewRead/NewWrite/NewCommit context-only.

Preserve catalog ownership, storage bindings, schema caches and query-auth
behavior, and migrate callers, tests, examples and documentation.

Refs apache#555

BREAKING CHANGE: Context builders no longer accept table loading parameters
or option maps. Static factories take TableLoadOptions and dynamic options;
loaded-table option overrides use Table::Copy.
@SteNicholas SteNicholas changed the title refactor(context)!: separate table loading from execution settings refactor(context): separate table loading from execution settings Nov 16, 2026

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucasfang, thanks for updates. I left some comments for udpates. PTAL.

read_options, CatalogUtils::ResolveQueryAuthReadOptions(
context->GetCatalog(), context->GetIdentifier(), context->GetOptions(),
CurrentBranch(), file_system_, root_path_, true, "scan"));
catalog_, identifier_, dynamic_options, CurrentBranch(), file_system_,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the retained catalog to resolve scan snapshots too. Writers and committers created from a catalog-loaded table now use catalog_ and can publish a snapshot only through the catalog, but NewScan() still reaches the filesystem-only GetSnapshotManager() through both managed scan implementations. Consequently, writing and committing one row through a version-managed catalog and then calling NewScan() on the same table returns an empty plan (SnapshotId() == nullopt although the catalog accepted snapshot 1); an existing filesystem snapshot instead produces stale results. Please supply the catalog-aware snapshot manager to the scan path as well, and add a commit-then-scan regression covering a snapshot that has not been published to the filesystem.

TableSchema::CreateFromJson(*supplied_main_schema));
schema = std::shared_ptr<TableSchema>(std::move(supplied));
} else {
PAIMON_ASSIGN_OR_RAISE(schema, manager.Latest());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolve the fallback schema through the bound catalog when metadata is catalog-only. The new catalog-only loading path can successfully load dev with no filesystem schema metadata, but CreateForFallbackBranch() ignores its catalog argument here and always uses manager.Latest() unless a main schema was explicitly supplied. A catalog-only load with identifier tbl$branch_dev and scan.fallback-branch=main therefore fails during TableRead::Create() with fallback schema not found, even when the catalog serves the main schema. A stale filesystem fallback schema is also used in preference to the catalog's schema. Please resolve the fallback branch's identifier and schema from the catalog while preserving the explicit supplied-main-schema precedence, and cover this combination in the catalog-only load tests.


TEST(ReadContextTest, TestPrefetchMaxParallelNumZero) {
ReadContextBuilder builder("table_root_path");
ReadContextBuilder builder;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restore the assertion that calls Finish(). This test now only sets the prefetch fields and never executes the validation it is named for, so it passes even if zero prefetch parallelism is accepted. The check still exists in ReadContextBuilder::Finish() and is an execution-context concern, so separating table loading does not make this regression coverage obsolete. Please retain ASSERT_NOK_WITH_MSG(builder.Finish(), "prefetch max parallel num should be greater than 0").

``WithFileSystemSchemeToIdentifierMap()``, ``WithCatalog()`` and a branch. Options
given at the call still win over the ones the schema stored, as they do everywhere else.
The loaded table supplies its schema and file system. Dynamic options passed to
``New*()`` override stored options.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document CopyWithoutTimeTravel() as the way to apply dynamic options. The new NewScan/NewRead/NewWrite/NewCommit signatures accept only an operation context, and the context builders no longer expose SetOptions(), so callers cannot pass overrides to New*() as instructed here. Show copying the loaded table with CopyWithoutTimeTravel(overrides) and creating the operator from that copy. The same instruction in docs/source/api/format_table.rst needs updating, together with the paragraphs above about legacy format-table context constructors, which this PR removes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Separate table loading and dynamic options from execution contexts

2 participants