fix: Validate filter selectivity at SET time - #24541
Conversation
Signed-off-by: subhramit <subhramit.bb@live.in>
Signed-off-by: subhramit <subhramit.bb@live.in>
|
cc'ing @Jefffrey, @2010YOUY01, @alamb |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24541 +/- ##
==========================================
- Coverage 81.32% 81.31% -0.01%
==========================================
Files 1117 1117
Lines 396269 396307 +38
Branches 396269 396307 +38
==========================================
+ Hits 322260 322273 +13
- Misses 55186 55205 +19
- Partials 18823 18829 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| /// rejected later, when the value is actually consumed (e.g. by | ||
| /// `FilterExec`). | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] | ||
| pub struct ConfigFilterSelectivity(u8); |
There was a problem hiding this comment.
do we have any other configs that could benefit from this? that way we might rename this to something more generic
There was a problem hiding this comment.
Probably we can generalize it to a
pub struct ConfigRangeUsize {
inclusive_min: usize,
/// `None` if there is no maximum limit.
inclusive_max: Option<usize>,
}
And several existing typed config structs can reuse this one (like ConfigNonZeroUsize)
But I think this is not blocking, we can clean it up in a follow-up PR
2010YOUY01
left a comment
There was a problem hiding this comment.
Thank you for working on it!
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Which issue does this PR close?
Part of #17498
Rationale for this change
datafusion.optimizer.default_filter_selectivityis documented as a percentage in the range 0 (no selectivity) and 100 (all rows are selected), but invalid values such as 200 are currently accepted bySETand only rejected later when the value is actually consumed during physical planning / execution.So validating and rejecting out-of-range values at
SETtime to give users immediate feedback.What changes are included in this PR?
ConfigFilterSelectivitywrapper type indatafusion/common/src/config.rsoptimizer.default_filter_selectivityfromu8toConfigFilterSelectivityconfig_namespace!u8value via.get()Are these changes tested?
Yes. The change is covered by
datafusion/sqllogictest/test_files/set_variable.slt. It tests that:SET datafusion.optimizer.default_filter_selectivity = 200fails immediately0and100are acceptedRan
cargo test --profile=ci --test sqllogictests -- set_variable.sltto confirm it passes, and also rancargo test -p datafusion-cliandcargo test -p datafusionto check for any related regressions. There were none.Are there any user-facing changes?
Yes.
As described above, users now get an immediate configuration error when setting
datafusion.optimizer.default_filter_selectivityoutside0..=100, instead of seeing a later failure when the value is consumed.This also changes a public config field type from
u8toConfigFilterSelectivity, so it should likely carry theapi changelabel (I request any maintainer to add this).