Skip to content

[datafusion] Support persistent CREATE TABLE AS SELECT - #1448

Open
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1425
Open

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1425

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: closes #1425

Materializing a query in a persistent Paimon table currently requires an explicit schema and a separate INSERT. Add native CREATE TABLE AS SELECT to the DataFusion SQL context, deriving the schema and rows from one resolved source plan.

Brief change log

  • Infer column names, types and nullability from the analyzed query, including empty results and geospatial metadata. Reuse CREATE TABLE validation for partition keys, options and comments; reject unsupported modifiers, explicit column declarations and time outputs that would lose precision.
  • Resolve source tables and SQL functions in the session namespace independently of the destination. Preserve quoted names and qualified time-travel references, reuse the analyzed plan for execution, and normalize scalar and nested Arrow layouts for writing.
  • Skip existing native and external targets with IF NOT EXISTS before source planning. Use strict catalog creation and atomic initial-schema publication so a concurrent creator's table is never overwritten or populated.
  • Retain the table and concurrent writers' rows after population errors. Clean up owned output before commit, finish in-flight commits even if the caller cancels, and retain prepared files when publication may have succeeded.

Tests

  • CTAS integration and execution suite: 19 tests passed, covering schema inference, empty results, geospatial and nested types, time precision, quoted Unicode names, namespaces, time travel and REST SQL functions. Fault injection verifies plan reuse, catalog errors, creation races, concurrent writes, file cleanup, uncertain publication and cancellation.
  • Shared SQL regressions: 44 tests passed across time travel, SQL compatibility and REST partition operations. Existing DataFusion unit tests, SQLContext, procedures, comment DDL, read-files and spatial-index suites, plus 44 filesystem catalog tests, also passed.
  • Formatting and full-workspace Clippy passed with --all-targets --features vortex -- -D warnings using Rust 1.98.0.

API and Format

CTAS uses the existing catalog, table write and commit APIs; public Rust APIs and stored schema, snapshot and data-file formats are unchanged. Filesystem table creation now requires atomic publication of the initial schema. As with ordinary INSERT, callers must coordinate DROP, RENAME and table replacement with active writers. New CTAS destinations are native Paimon tables; OR REPLACE and external-engine CTAS remain unsupported.

Documentation

Update docs/src/sql.md and the DataFusion README with CTAS syntax, inference, namespace resolution, IF NOT EXISTS, supported types and precision, concurrent DDL requirements, and recovery after failure or cancellation.

🤖 Assisted with Codex.

Comment thread crates/paimon/src/catalog/mod.rs Resolved
Materializing a query in a persistent Paimon table requires an explicit
schema and a separate INSERT. CTAS must infer and populate the target
from one resolved source plan, preserving session name resolution and
handling creation races and uncertain publication.

- Infer native CTAS schemas from analyzed query output, including empty
  results. Preserve nullability and geospatial metadata, and validate
  partition keys, table options and comments, addressing apache#1425.
- Resolve source tables and SQL functions in the session namespace,
  preserving quoted names and qualified time-travel references. Reuse
  the analyzed plan and normalize scalar and nested Arrow layouts.
- Skip existing targets with IF NOT EXISTS before source planning. Use
  strict catalog creation and publish initial schemas atomically to
  leave a concurrent creator's table and rows untouched.
- Retain the table after population errors. Clean up owned output on
  pre-commit failures and cancellation, finish commits already started
  and retain prepared files when publication may have succeeded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support persistent CREATE TABLE AS SELECT in DataFusion

2 participants