Skip to content

fix: infer LIMIT and OFFSET placeholder parameter types#23260

Open
Probablism wants to merge 1 commit into
apache:mainfrom
Probablism:codex/issue-15978-limit-placeholder-types
Open

fix: infer LIMIT and OFFSET placeholder parameter types#23260
Probablism wants to merge 1 commit into
apache:mainfrom
Probablism:codex/issue-15978-limit-placeholder-types

Conversation

@Probablism

@Probablism Probablism commented Jun 30, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Bare placeholders used as LIMIT or OFFSET row-count expressions currently remain untyped in LogicalPlan::get_parameter_types(). DataFusion already coerces LIMIT/OFFSET row counts to Int64 during analysis, so the metadata APIs should report the same type for bare row-count placeholders.

What changes are included in this PR?

  • Infer Int64 parameter fields for bare unresolved placeholders used directly as LIMIT or OFFSET expressions.
  • Reuse existing expression placeholder inference before applying the LIMIT/OFFSET fallback, so nested expressions keep their local inference behavior.
  • Preserve existing placeholder conflict detection.
  • Add raw logical-plan and SQL/PREPARE regression tests for LIMIT/OFFSET parameter inference.

Are these changes tested?

Yes:

  • cargo fmt --all -- --check
  • cargo test -p datafusion-expr test_limit_
  • cargo test -p datafusion-sql --test sql_integration test_infer_types_from_limit
  • cargo test -p datafusion-sql --test sql_integration test_prepare_statement_infer_types_from_limit
  • cargo test -p datafusion
  • cargo test -p datafusion-cli
  • cargo test --profile=ci --test sqllogictests
  • ./dev/rust_lint.sh

Are there any user-facing changes?

Yes. LogicalPlan::get_parameter_types() and LogicalPlan::get_parameter_fields() now report Int64 for bare placeholders used directly in LIMIT and OFFSET. There are no breaking API changes.

@github-actions github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions labels Jun 30, 2026
@Probablism Probablism marked this pull request as ready for review June 30, 2026 14:40
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-expr v54.0.0 (current)
       Built [  25.372s] (current)
     Parsing datafusion-expr v54.0.0 (current)
      Parsed [   0.070s] (current)
    Building datafusion-expr v54.0.0 (baseline)
       Built [  25.592s] (baseline)
     Parsing datafusion-expr v54.0.0 (baseline)
      Parsed [   0.070s] (baseline)
    Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.578s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron

Failed in:
  ScalarUDF::is_strict, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/633fa6b836a4ab2edb8c5635672d0742ae0bfeb8/datafusion/expr/src/udf.rs:216

--- failure trait_method_missing: pub trait method removed or renamed ---

Description:
A trait method is no longer callable, and may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_missing.ron

Failed in:
  method is_strict of trait ScalarUDFImpl, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/633fa6b836a4ab2edb8c5635672d0742ae0bfeb8/datafusion/expr/src/udf.rs:714

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  55.205s] datafusion-expr
    Building datafusion-sql v54.0.0 (current)
       Built [  40.123s] (current)
     Parsing datafusion-sql v54.0.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-sql v54.0.0 (baseline)
       Built [  39.839s] (baseline)
     Parsing datafusion-sql v54.0.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-sql v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.258s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  81.651s] datafusion-sql

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 9, 2026

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Probablism
Thanks for the fix.
I found one issue that looks like it can make the new parameter metadata report the wrong type when a LIMIT or OFFSET placeholder is reused in an unconstrained expression.

(_, Some(field)) => {
param_types.insert(id.clone(), Some(Arc::clone(field)));
}
_ => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collect_parameter_fields can currently erase the Int64 type inferred from a LIMIT or OFFSET placeholder when the same parameter is later seen in an untyped context.

For example, SELECT $1 FROM person LIMIT $1 records $1 as Int64 from the row-count expression first because apply_with_subqueries visits the Limit node before its child plans. Later, the projection's bare $1 reaches this catch-all arm and overwrites that entry with None. As a result, the metadata API still reports the row-count parameter as untyped, which breaks the invariant this PR is trying to enforce when the placeholder is reused in an unconstrained expression.

Could we keep an existing Some(field) when the new occurrence is None, and only insert None if the parameter is not already known? Please also add a regression test for a LIMIT or OFFSET placeholder reused in an otherwise-untyped expression.

let mut param_types: HashMap<String, Option<FieldRef>> = HashMap::new();

self.apply_with_subqueries(|plan| {
let row_count_field = Arc::new(Field::new("", DataType::Int64, true));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small polish suggestion: consider hoisting row_count_field outside the apply_with_subqueries closure so the Int64 field is allocated once per get_parameter_fields call instead of once per visited plan node.

This is not blocking, but I think it would also make the row-count placeholder invariant a little easier to spot while reading the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change logical-expr Logical plan and expressions sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Placeholder datatype not inferred after LIMIT clause

2 participants