Skip to content

Commit e987119

Browse files
learn: retrospective learnings
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
1 parent 7db4b87 commit e987119

1 file changed

Lines changed: 4 additions & 0 deletions

File tree

‎.claude/knowledge/learning-log.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,4 +162,8 @@ until a human merges it. The engineer author phase reads this log (see
162162
- **Context:** Engineer-bot run #39665896492 fixed GH #989 in `src/databricks/sql/utils.py::_concat_arrow_tables`, which unconditionally called `pyarrow.concat_tables(..., promote_options="default")` (fallback `promote=True`) and failed on duplicate column names.
163163
**Rule:** When concatenating pyarrow tables, only pass a `promote`/`promote_options` argument when chunk schemas actually differ — any promote option triggers Arrow schema unification, which raises `ArrowInvalid: Can't unify schema with duplicate field names` even for a single chunk; valid SQL (`SELECT 1 AS a, 2 AS a`, self-JOINs) legitimately produces duplicate column names, so short-circuit to a plain `concat_tables` when all chunk schemas equal the first, preserving promotion only for the genuine differing-schema (e.g. nullability) case.
164164

165+
### 2026-10-24: learnings since 2026-10-23T17:31:38Z
166+
- **Context:** PR #995 fixed `ArrowInvalid: Can't unify schema with duplicate field names` in `_concat_arrow_tables` (src/databricks/sql/utils.py), a regression from switching to `pyarrow.concat_tables(promote_options="default")`. Duplicate column names are valid SQL (`SELECT 1 AS a, 2 AS a`, `SELECT * FROM a JOIN b`), but Arrow's schema promotion always invokes schema unification, which rejects duplicate field names even when no promotion is actually required.
167+
**Rule:** When concatenating Arrow tables/chunks, only pass `promote_options`/`promote` when chunk schemas actually differ (guard with `all(t.schema.equals(chunks[0].schema) ...)`); unconditional promotion triggers schema unification that fails on duplicate column names even though plain `concat_tables` would succeed.
168+
165169
--- *Add new entries above this line (oldest→newest); newest sections sort to the bottom.* ---

0 commit comments

Comments
 (0)