fix: account for empty scalar subqueries in nullability

[fornwall]

Which issue does this PR close?

• Closes Scalar subquery returning no rows is incorrectly marked non-nullable #24513.

Rationale for this change

A scalar subquery that returns zero rows evaluates to NULL, even when its projected expression is non-nullable. DataFusion currently derives Expr::ScalarSubquery nullability from the subquery's output field. This can produce a non-nullable output schema containing NULL, and can cause SimplifyExpressions to incorrectly fold predicates such as:

SELECT (SELECT 1 WHERE FALSE) IS NULL;

to false.

This change deliberately marks all scalar subqueries nullable, including those guaranteed to return exactly one row (such as an ungrouped aggregate like (SELECT count(*) FROM t)). This is a conservative trade-off that gives up some nullability precision for correctness, and is consistent with how PostgreSQL treats scalar subqueries.

A possible follow-up refinement is a LogicalPlan::min_rows() lower bound (mirroring the existing max_rows()), which would let uncorrelated scalar subqueries provably returning at least one row keep their projected field's nullability.

What changes are included in this PR?

Scalar subqueries are conservatively marked nullable in logical expression schema derivation and physical expression planning. The projected field's data type, name, and metadata are preserved.

Are these changes tested?

Yes. Unit tests cover logical schema derivation and expression simplification, and SQLLogicTests cover both zero-row execution and IS NULL correctness.

The full workspace test suite and Clippy with warnings denied pass.

Are there any user-facing changes?

Yes. Zero-row scalar subqueries with non-nullable projections now return NULL without a schema validation error, and IS NULL predicates produce the correct result.

In addition, output schemas containing scalar subqueries now always mark those fields as nullable, even for subqueries that can never produce NULL. Downstream consumers that inspect schema nullability can observe this change. No public APIs change.

---

AI usage: Created with Claude Code and Opus 5. I have reviewed the code and made modifications where it made sense.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.29%. Comparing base (5091b42) to head (c74d149).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24516      +/-   ##
==========================================
+ Coverage   81.27%   81.29%   +0.01%     
==========================================
  Files        1116     1116              
  Lines      395073   395490     +417     
  Branches   395073   395490     +417     
==========================================
+ Hits       321102   321501     +399     
- Misses      55176    55179       +3     
- Partials    18795    18810      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[neilconway]

Thanks -- this is a well-implemented PR. The root problem is clearly described and the fix is minimal and seems correct to me. Good test coverage; we could add coverage for correlated subqueries as well but I think it's probably okay as-is.

Would be great if you're interested in taking on the min_rows refinement as a followup PR.

[fornwall]

Would be great if you're interested in taking on the min_rows refinement as a followup PR.

👍 Created #24534 for that.