docs: note that installing a rule drops prepared statements

Installing a physical optimizer rule rebuilds the session state, and
`SessionStateBuilder::build` starts the new state with an empty
prepared-plan map, so a session that has run PREPARE reports the
statement missing afterwards. It hits every handle sharing the session,
not just the one a call returned.

There is no fix in this repo. `SessionState` exposes `physical_optimizers`
read-only and only the builder can append to it, so the rebuild is the
only public path, and `prepared_plans` has no builder setter.

The canonical home for the claim is a new `extension_rule_rebuild`
section in the extension guide. `add_physical_optimizer_rule` and
`with_extensions` each state it in one sentence and link there. The
enumeration already on `add_physical_optimizer_rule` -- "tables, UDFs,
and catalogs are preserved" -- was incomplete and now names the
exception.

Three smaller corrections alongside it:

`test_with_extensions_declaring_no_rules_leaves_the_session_id` claimed
to pin the id surviving the rebuild, but it declares no rules, so the
rebuild is skipped and the assertion cannot reach that guarantee. Its
docstring now says it is the no-op control and names the FFI test that
does cover the rebuild.

`test_declared_rules_run_in_declaration_order` asserted
`run_order() == [0, 1]`, which ties the ordering claim to the optimizer
running exactly once per query -- a count its sibling FFI test
deliberately avoids asserting. It now checks the first pass only.

`PyPhysicalOptimizerRules` is `frozen`. Nothing mutates it between the
resolve and commit steps, so it has no reason to carry a borrow flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

crates/core/src/context.rs

@@ -1861,7 +1861,15 @@ impl PySessionContext {
 /// to carry imported rules from the resolve step to the commit step, so the
 /// import can fail before anything is written. `with_extensions` is its only
 /// producer and its only consumer.
-#[pyclass(name = "PhysicalOptimizerRules", module = "datafusion._internal")]
+///
+/// `frozen` because nothing mutates it between those two steps: the commit
+/// only reads the rules back out, so there is no reason to pay for the runtime
+/// borrow flag a mutable pyclass carries.
+#[pyclass(
+    frozen,
+    name = "PhysicalOptimizerRules",
+    module = "datafusion._internal"
+)]
 pub struct PyPhysicalOptimizerRules {
     rules: Vec<Arc<dyn PhysicalOptimizerRule + Send + Sync>>,
 }

docs/source/extension-guide/other-components.md

@@ -63,6 +63,32 @@ a bundle contributing several would clone the whole state that many times, and
 would leave the earlier ones installed if a later one failed. See
 {ref}`extension_bundles_transaction`.
 
+(extension_rule_rebuild)=
+
+### Installing a rule rebuilds the session state
+
+There is no way to append to a live `SessionState`: DataFusion exposes
+`physical_optimizers` on one read-only, and only `SessionStateBuilder` can add
+to the list. Installing a rule therefore rebuilds the state in place, and the
+rebuild carries over the tables, functions, catalogs, and session id the old
+one held.
+
+**Prepared statements are the exception.** `SessionStateBuilder::build` starts
+the new state with an empty prepared-plan map, so a session that has run
+`PREPARE` reports the statement missing once a rule is installed:
+
+```python
+ctx.sql("PREPARE p AS SELECT a FROM t").collect()
+ctx.with_extensions(MyRuleBundle())
+ctx.sql("EXECUTE p")  # ValueError: Prepared statement 'p' does not exist
+```
+
+This is not specific to bundles — `add_physical_optimizer_rule` drops them the
+same way, and both hit every handle sharing the session rather than only the
+one the call returned. Install your rules before preparing anything. Batching a
+bundle's rules into one rebuild is what keeps the cost to once per call instead
+of once per rule.
+
 ## Typed configuration
 
 **`__datafusion_extension_options__`** contributes typed configuration entries

examples/datafusion-ffi-example/python/tests/_test_session_extension.py

@@ -139,13 +139,17 @@ def test_declared_rules_run_in_declaration_order():
     Rules rewrite the plan one after another, so the order is part of what a
     bundle declares. The counters cannot show it — each rule has its own — so
     the two here append to a log they share.
+
+    Only the first pass is asserted on. How many times a query optimizes is a
+    separate claim from what order the rules run in, and pinning both here
+    would report a changed pass count as an ordering bug.
     """
     extension = MyRuleExtension()
     ctx = SessionContext().with_extensions(extension)
 
     _query(ctx)
 
-    assert extension.run_order() == [0, 1]
+    assert extension.run_order()[:2] == [0, 1]
 
 
 def test_rules_install_without_changing_the_session_id():

python/datafusion/context.py

@@ -2051,7 +2051,8 @@ def add_physical_optimizer_rule(
         PyCapsule, typically produced by a separate compiled extension. The
         underlying :class:`SessionState` is rebuilt from its current state
         with the new rule appended, so previously registered tables, UDFs,
-        and catalogs are preserved.
+        and catalogs are preserved. Prepared statements are not — see
+        :ref:`extension_rule_rebuild`.
 
         Args:
             rule: Object exposing ``__datafusion_physical_optimizer_rule__``,
@@ -2147,6 +2148,10 @@ def with_extensions(
         table, say — is not rolled back, which is why bundle objects must be
         configuration-only.
 
+        A call that installs optimizer rules rebuilds the session state, which
+        drops the session's prepared statements — see
+        :ref:`extension_rule_rebuild`.
+
         Shares its session with this context — see :py:class:`SessionContext`.
 
         See :ref:`extension_bundles` in the online documentation for why the

python/tests/test_context.py

@@ -1725,12 +1725,15 @@ def __datafusion_session_components__(self, ctx):
 
 
 def test_with_extensions_declaring_no_rules_leaves_the_session_id(ctx):
-    """Installing rules rebuilds ``SessionState``; the id has to survive it.
-
-    The rebuild mints a fresh id unless it is carried over, and a changed id
-    would break every ``TaskContext`` the session has handed out. Asserted for
-    a call declaring no rules as well, so the guarantee does not depend on
-    whether the rebuild was skipped.
+    """A call declaring no rules leaves the session id alone.
+
+    This is the control for the no-op path: with nothing to install the state
+    rebuild is skipped, so the id is untouched rather than carried over. The
+    carry-over itself is not reachable from here — the rebuild needs a real
+    rule capsule, which only a compiled extension can hand over. That half is
+    pinned by ``test_rules_install_without_changing_the_session_id`` in
+    ``datafusion-ffi-example``, where a fresh id would leave ``session_id()``
+    disagreeing with every ``TaskContext`` the session has handed out.
     """
     before = ctx.session_id()
     result = ctx.with_extensions(_FunctionExtension(udfs=(_doubler(),)))