Skip to content

Commit 0062fc4

Browse files
Awesome06Awesome06
committed
refactor: limit package root to common entry points (#1771)
Adopt the rule established in 55.0.0: the package root holds what a typical program constructs or calls on the common path, and types that are only named (enums, options classes, plan types, and catalog protocols) are imported from their submodules. - Remove 16 specialized types from datafusion.__all__ and direct imports - Add __getattr__ module-level deprecation shim with DeprecationWarning - Add __dir__ returning sorted __all__ for clean autocomplete - Update internal call sites in tests, examples, and user skill - Document package root conventions in AGENTS.md, contributor guide, and 56.0.0 upgrade guide
1 parent 878e5d3 commit 0062fc4

18 files changed

Lines changed: 281 additions & 82 deletions

‎AGENTS.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,24 @@ The `__datafusion_*__` capsule getters are one protocol with a settled
4545
convention. Before adding or changing one, read
4646
[`.ai/skills/ffi-capsule-protocol/SKILL.md`](.ai/skills/ffi-capsule-protocol/SKILL.md).
4747

48+
## Package Root Exports
49+
50+
The package root (`datafusion`) holds what a typical program constructs or calls
51+
on the common path:
52+
- Core session and environment builders (`SessionContext`, `SessionConfig`,
53+
`RuntimeEnvBuilder`, `SessionExtensionComponents`)
54+
- Core query expressions and builders (`DataFrame`, `Expr`, `col` / `column`,
55+
`lit` / `literal`, `lit_with_metadata` / `literal_with_metadata`)
56+
- Functions and decorators (`udf`, `udaf`, `udwf`, `udtf`, `Accumulator`)
57+
- File format readers (`read_avro`, `read_csv`, `read_json`, `read_parquet`)
58+
- Public submodules (`functions`, `ipc`, `object_store`, `substrait`, `unparser`)
59+
60+
Types that are only named (an enum, a protocol, an options or result type, an
61+
intermediate plan node, or advanced catalog/FFI objects) are not exported at the
62+
package root. They must be imported from their respective submodules (such as
63+
`datafusion.dataframe`, `datafusion.plan`, `datafusion.catalog`,
64+
`datafusion.options`, `datafusion.expr`, `datafusion.user_defined`).
65+
4866
## Documentation Sources
4967

5068
Search and edit `docs/source/`. `docs/temp/` is gitignored build output that

‎docs/source/contributor-guide/introduction.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,24 @@ Mostly, the `python` code is limited to pure wrappers with type hints and good d
8888
2. Simple type conversion, like from a `path` to a `string` of the path or from `number` to `lit(number)`.
8989
3. The additional code makes an API **much** more pythonic, like we do for {py:func}`~datafusion.functions.named_struct` (see [source code](https://github.com/apache/datafusion-python/blob/a0913c728f5f323c1eb4913e614c9d996083e274/python/datafusion/functions.py#L1040-L1046)).
9090

91+
## Package Root Exports
92+
93+
The package root (`datafusion`) holds what a typical program constructs or calls
94+
on the common path:
95+
- Core session and environment builders (`SessionContext`, `SessionConfig`,
96+
`RuntimeEnvBuilder`, `SessionExtensionComponents`)
97+
- Core query expressions and builders (`DataFrame`, `Expr`, `col` / `column`,
98+
`lit` / `literal`, `lit_with_metadata` / `literal_with_metadata`)
99+
- Functions and decorators (`udf`, `udaf`, `udwf`, `udtf`, `Accumulator`)
100+
- File format readers (`read_avro`, `read_csv`, `read_json`, `read_parquet`)
101+
- Public submodules (`functions`, `ipc`, `object_store`, `substrait`, `unparser`)
102+
103+
Types that are only named (an enum, a protocol, an options or result type, an
104+
intermediate plan node, or advanced catalog/FFI objects) are not exported at the
105+
package root. They must be imported from their respective submodules (such as
106+
`datafusion.dataframe`, `datafusion.plan`, `datafusion.catalog`,
107+
`datafusion.options`, `datafusion.expr`, `datafusion.user_defined`).
108+
91109
## Update Dependencies
92110

93111
To change test dependencies, change the `pyproject.toml` and run

‎docs/source/user-guide/upgrade-guides.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,38 @@
2121

2222
## DataFusion 56.0.0
2323

24+
### Package root limited to common entry points
25+
26+
To keep the top-level `datafusion` namespace predictable and focused on common
27+
workflows, types that are only named (enums, options classes, plan types, and
28+
advanced catalog protocols) are now imported from their respective submodules
29+
rather than the package root.
30+
31+
Old imports continue to work throughout DataFusion 56.0.0 via a module-level
32+
`__getattr__` deprecation shim that raises a `DeprecationWarning`, and will be
33+
removed in a future release.
34+
35+
| Deprecated root import | Canonical import location |
36+
|---|---|
37+
| `from datafusion import DataFrameWriteOptions` | `from datafusion.dataframe import DataFrameWriteOptions` |
38+
| `from datafusion import ExplainFormat` | `from datafusion.dataframe import ExplainFormat` |
39+
| `from datafusion import InsertOp` | `from datafusion.dataframe import InsertOp` |
40+
| `from datafusion import ParquetColumnOptions` | `from datafusion.dataframe import ParquetColumnOptions` |
41+
| `from datafusion import ParquetWriterOptions` | `from datafusion.dataframe import ParquetWriterOptions` |
42+
| `from datafusion import ExecutionPlan` | `from datafusion.plan import ExecutionPlan` |
43+
| `from datafusion import LogicalPlan` | `from datafusion.plan import LogicalPlan` |
44+
| `from datafusion import Metric` | `from datafusion.plan import Metric` |
45+
| `from datafusion import MetricsSet` | `from datafusion.plan import MetricsSet` |
46+
| `from datafusion import PhysicalPartitioning` | `from datafusion.plan import PhysicalPartitioning` |
47+
| `from datafusion import Catalog` | `from datafusion.catalog import Catalog` |
48+
| `from datafusion import TableProviderFactory` | `from datafusion.catalog import TableProviderFactory` |
49+
| `from datafusion import TableProviderFactoryExportable` | `from datafusion.catalog import TableProviderFactoryExportable` |
50+
| `from datafusion import DFSchema` | `from datafusion.common import DFSchema` |
51+
| `from datafusion import WindowFrame` | `from datafusion.expr import WindowFrame` |
52+
| `from datafusion import SQLOptions` | `from datafusion.context import SQLOptions` |
53+
| `from datafusion import RecordBatchStream` | `from datafusion.record_batch import RecordBatchStream` |
54+
| `from datafusion import TableFunction` | `from datafusion.user_defined import TableFunction` |
55+
2456
### `sort` and `order_by` now order `NULLS LAST` by default
2557

2658
Calling `.sort(...)` on a DataFrame or using the `.order_by(...)` function now

‎examples/datafusion-ffi-example/python/tests/_test_logical_extension_codec.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919

2020
import pyarrow as pa
2121
import pytest
22-
from datafusion import Expr, LogicalPlan, SessionContext, col, udf
22+
from datafusion import Expr, SessionContext, col, udf
23+
from datafusion.plan import LogicalPlan
2324
from datafusion_ffi_example import (
2425
MyLogicalExtensionCodec,
2526
MyTableProvider,

‎examples/datafusion-ffi-example/python/tests/_test_physical_extension_codec.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
from __future__ import annotations
1919

2020
import pyarrow as pa
21-
from datafusion import ExecutionPlan, SessionContext
21+
from datafusion import SessionContext
22+
from datafusion.plan import ExecutionPlan
2223
from datafusion_ffi_example import MyPhysicalExtensionCodec
2324

2425

‎examples/tpch/q11_important_stock_identification.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,9 @@
5757
value desc;
5858
"""
5959

60-
from datafusion import SessionContext, WindowFrame, col, lit
60+
from datafusion import SessionContext, col, lit
6161
from datafusion import functions as F
62-
from datafusion.expr import Window
62+
from datafusion.expr import Window, WindowFrame
6363
from util import get_data_path
6464

6565
NATION = "GERMANY"

‎examples/tpch/q17_small_quantity_order.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,9 @@
4848
);
4949
"""
5050

51-
from datafusion import SessionContext, WindowFrame, col, lit
51+
from datafusion import SessionContext, col, lit
5252
from datafusion import functions as F
53-
from datafusion.expr import Window
53+
from datafusion.expr import Window, WindowFrame
5454
from util import get_data_path
5555

5656
BRAND = "Brand#23"

‎examples/tpch/q22_global_sales_opportunity.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,9 @@
6666
cntrycode;
6767
"""
6868

69-
from datafusion import SessionContext, WindowFrame, col, lit
69+
from datafusion import SessionContext, col, lit
7070
from datafusion import functions as F
71-
from datafusion.expr import Window
71+
from datafusion.expr import Window, WindowFrame
7272
from util import get_data_path
7373

7474
NATION_CODES = [13, 31, 23, 29, 30, 18, 17]

‎python/datafusion/__init__.py‎

Lines changed: 83 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -57,53 +57,31 @@
5757

5858
from __future__ import annotations
5959

60+
import importlib
6061
import importlib.metadata as importlib_metadata
62+
import warnings
6163
from typing import Any
6264

6365
# Public submodules
6466
from . import functions, ipc, object_store, substrait, unparser
65-
66-
# The following imports are okay to remain as opaque to the user.
67-
from .catalog import (
68-
Catalog,
69-
Table,
70-
TableProviderFactory,
71-
TableProviderFactoryExportable,
72-
)
67+
from .catalog import Table
7368
from .col import col, column
74-
from .common import DFSchema
7569
from .context import (
7670
RuntimeEnvBuilder,
7771
SessionConfig,
7872
SessionContext,
79-
SQLOptions,
80-
)
81-
from .dataframe import (
82-
DataFrame,
83-
DataFrameWriteOptions,
84-
ExplainFormat,
85-
InsertOp,
86-
ParquetColumnOptions,
87-
ParquetWriterOptions,
8873
)
74+
from .dataframe import DataFrame
8975
from .dataframe_formatter import configure_formatter
90-
from .expr import Expr, WindowFrame
76+
from .expr import Expr
9177
from .extensions import SessionExtensionComponents
9278
from .io import read_avro, read_csv, read_json, read_parquet
9379
from .options import CsvReadOptions
94-
from .plan import (
95-
ExecutionPlan,
96-
LogicalPlan,
97-
Metric,
98-
MetricsSet,
99-
PhysicalPartitioning,
100-
)
101-
from .record_batch import RecordBatch, RecordBatchStream
80+
from .record_batch import RecordBatch
10281
from .user_defined import (
10382
Accumulator,
10483
AggregateUDF,
10584
ScalarUDF,
106-
TableFunction,
10785
WindowUDF,
10886
udaf,
10987
udf,
@@ -116,48 +94,27 @@
11694
__all__ = [
11795
"Accumulator",
11896
"AggregateUDF",
119-
"Catalog",
12097
"CsvReadOptions",
121-
"DFSchema",
12298
"DataFrame",
123-
"DataFrameWriteOptions",
124-
"ExecutionPlan",
125-
"ExplainFormat",
12699
"Expr",
127-
"InsertOp",
128-
"LogicalPlan",
129-
"Metric",
130-
"MetricsSet",
131-
"ParquetColumnOptions",
132-
"ParquetWriterOptions",
133-
"PhysicalPartitioning",
134100
"RecordBatch",
135-
"RecordBatchStream",
136101
"RuntimeEnvBuilder",
137-
"SQLOptions",
138102
"ScalarUDF",
139103
"SessionConfig",
140104
"SessionContext",
141105
"SessionExtensionComponents",
142106
"Table",
143-
"TableFunction",
144-
"TableProviderFactory",
145-
"TableProviderFactoryExportable",
146-
"WindowFrame",
147107
"WindowUDF",
148-
"catalog",
149108
"col",
150109
"column",
151-
"common",
152110
"configure_formatter",
153-
"expr",
154-
"extensions",
155111
"functions",
156112
"ipc",
157113
"lit",
114+
"lit_with_metadata",
158115
"literal",
116+
"literal_with_metadata",
159117
"object_store",
160-
"options",
161118
"read_avro",
162119
"read_csv",
163120
"read_json",
@@ -170,6 +127,81 @@
170127
"unparser",
171128
]
172129

130+
_DEPRECATED_ROOT_NAMES: dict[str, tuple[str, str]] = {
131+
# datafusion.dataframe
132+
"DataFrameWriteOptions": ("datafusion.dataframe", "DataFrameWriteOptions"),
133+
"ExplainFormat": ("datafusion.dataframe", "ExplainFormat"),
134+
"InsertOp": ("datafusion.dataframe", "InsertOp"),
135+
"ParquetColumnOptions": ("datafusion.dataframe", "ParquetColumnOptions"),
136+
"ParquetWriterOptions": ("datafusion.dataframe", "ParquetWriterOptions"),
137+
# datafusion.plan
138+
"ExecutionPlan": ("datafusion.plan", "ExecutionPlan"),
139+
"LogicalPlan": ("datafusion.plan", "LogicalPlan"),
140+
"Metric": ("datafusion.plan", "Metric"),
141+
"MetricsSet": ("datafusion.plan", "MetricsSet"),
142+
"PhysicalPartitioning": ("datafusion.plan", "PhysicalPartitioning"),
143+
# datafusion.catalog
144+
"Catalog": ("datafusion.catalog", "Catalog"),
145+
"TableProviderFactory": ("datafusion.catalog", "TableProviderFactory"),
146+
"TableProviderFactoryExportable": (
147+
"datafusion.catalog",
148+
"TableProviderFactoryExportable",
149+
),
150+
# datafusion.common
151+
"DFSchema": ("datafusion.common", "DFSchema"),
152+
# datafusion.expr
153+
"WindowFrame": ("datafusion.expr", "WindowFrame"),
154+
# datafusion.context
155+
"SQLOptions": ("datafusion.context", "SQLOptions"),
156+
# datafusion.record_batch
157+
"RecordBatchStream": ("datafusion.record_batch", "RecordBatchStream"),
158+
# datafusion.user_defined
159+
"TableFunction": ("datafusion.user_defined", "TableFunction"),
160+
}
161+
162+
_SUBMODULES: set[str] = {
163+
"catalog",
164+
"col",
165+
"common",
166+
"context",
167+
"dataframe",
168+
"dataframe_formatter",
169+
"expr",
170+
"extensions",
171+
"functions",
172+
"input",
173+
"io",
174+
"ipc",
175+
"object_store",
176+
"options",
177+
"plan",
178+
"record_batch",
179+
"substrait",
180+
"unparser",
181+
"user_defined",
182+
}
183+
184+
185+
def __getattr__(name: str) -> Any:
186+
if name in _DEPRECATED_ROOT_NAMES:
187+
module_path, target_name = _DEPRECATED_ROOT_NAMES[name]
188+
warnings.warn(
189+
f"`datafusion.{name}` is deprecated and will be removed in a future "
190+
f"release. Import `{target_name}` from `{module_path}` instead.",
191+
category=DeprecationWarning,
192+
stacklevel=2,
193+
)
194+
module = importlib.import_module(module_path)
195+
return getattr(module, target_name)
196+
if name in _SUBMODULES:
197+
return importlib.import_module(f".{name}", __name__)
198+
msg = f"module '{__name__}' has no attribute '{name}'"
199+
raise AttributeError(msg)
200+
201+
202+
def __dir__() -> list[str]:
203+
return sorted(__all__)
204+
173205

174206
def literal(value: Any) -> Expr:
175207
"""Create a literal expression."""

‎python/datafusion/context.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,6 @@
5959

6060
import pyarrow as pa
6161

62-
from datafusion import extensions as _extensions
6362
from datafusion.catalog import (
6463
Catalog,
6564
CatalogList,
@@ -78,6 +77,7 @@
7877
)
7978
from datafusion.record_batch import RecordBatchStream
8079

80+
from . import extensions as _extensions
8181
from ._internal import RuntimeEnvBuilder as RuntimeEnvBuilderInternal
8282
from ._internal import SessionConfig as SessionConfigInternal
8383
from ._internal import SessionContext as SessionContextInternal

0 commit comments

Comments
 (0)