Skip to content

fix: Avoid casting unreachable nested List values - #24590

Open
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:fix-nested-list-unreachable-values
Open

fix: Avoid casting unreachable nested List values#24590
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:fix-nested-list-unreachable-values

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Recursive nested List casts currently cast the entire backing child array. A sliced List or a null parent row can therefore fail because an unreachable child value cannot be cast, even though that value is not part of the query result.

What changes are included in this PR?

  • Compact List and LargeList child values when slices or null parents leave unreachable values.
  • Select and remap the union of reachable ListView and LargeListView child ranges while preserving shared and overlapping values.
  • Keep a zero-copy fast path when all child values are reachable.
  • Add unit coverage for sliced, null-parent, overlapping, out-of-order, nested, and visible-invalid inputs, plus the SQL LIMIT regression from the issue.

Are these changes tested?

Yes.

  • 68 focused nested-Struct cast tests.
  • The struct.slt SQLLogicTest suite, including the LIMIT regression.
  • Repository-wide all-target, all-feature clippy with warnings denied.
  • The contributor-guide extended workspace test suite, including 505 SQLLogicTest files.
  • cargo fmt --all -- --check and git diff --check.

Are there any user-facing changes?

Nested List casts now ignore child values that are unreachable from visible, non-null parent rows. Reachable invalid child values still return an error. There are no public API or schema changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Aug 23, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.19298% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.39%. Comparing base (5134a1a) to head (1f133b0).

Files with missing lines Patch % Lines
datafusion/common/src/nested_struct.rs 97.19% 3 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24590      +/-   ##
==========================================
+ Coverage   81.38%   81.39%   +0.01%     
==========================================
  Files        1116     1116              
  Lines      397960   398238     +278     
  Branches   397960   398238     +278     
==========================================
+ Hits       323880   324162     +282     
+ Misses      55120    55110      -10     
- Partials    18960    18966       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid casting unreachable child values in nested List casts

2 participants