What happened?
BytesDictBuilder::encode_value computes the offset of a new entry before checking whether the entry fits under max_bytes
let view = BinaryView::make_view(
val, 0,
u32::try_from(self.values.len()).vortex_expect("values length must fit in u32"),
);
let additional_bytes = /* ... */;
if self.dict_bytes() + additional_bytes > self.max_dict_bytes {
return None;
}
Because the u32::try_from runs first, the builder panics rather than returning None.
We hit this in production
values length must fit in u32:
Try from int error: out of range integral type conversion attempted
13: vortex_expect<u32, core::num::error::TryFromIntError>
at vortex-error/src/lib.rs:347:14
14: encode_value<u32>
at vortex-array/src/builders/dict/bytes.rs:110:54
15: encode_validity<u32, ...encode_varbinview::{closure_env#1}<u32>>
at vortex-array/src/builders/dict/bytes.rs:180:43
16: encode_varbinview<u32>
at vortex-array/src/builders/dict/bytes.rs:253:14
Steps to reproduce
let values = VarBinViewArray::from_iter_str(
(0..4_300_000).map(|i| format!("{i:08}{}", "x".repeat(1016))),
)
.into_array();
dict_encode(&values, &mut ctx).unwrap(); // panics
Environment
- Vortex develop @
68e2aee0a
- Rust 1.98.0
Additional context
No response
What happened?
BytesDictBuilder::encode_valuecomputes the offset of a new entry before checking whether the entry fits undermax_bytesBecause the
u32::try_fromruns first, the builder panics rather than returningNone.We hit this in production
Steps to reproduce
Environment
68e2aee0aAdditional context
No response