Repository navigation
Error when upserting and updating dataframes with dictionary encoded columns聽#3835
Description
Apache Iceberg version
0.9.1
Please describe the bug 馃悶
df = pyarrow.read_table(read_dictionary=[string_columns])
table = catalog.load_table(table_name)
table.append(df) -> 'append' casts dictionary columns into large string and table appended without any error
table.upsert(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)
table.update(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
dino-saurabh commentedon Oct 24, 2026
Hi people, I'd like to pick this up. On main, upsert no longer errors, but any append or upsert of dictionary-encoded data into a non-empty table leaves mixed dictionary/string Parquet files, and the next scan fails with
ArrowTypeError: Unable to merge.@hedger9487, are you still working on #3843? If not, I'll build on it.