Bug triage results: 2026-11-02 #7403

Summary

• Date: 2026-11-02
• requires-triage issues found: 65. Labeled: 63 (21 bugs, 42 enhancements). Skipped: 2. Failed to label: 0.
• Bugs by priority: priority:critical 2, priority:high 7, priority:medium 10, priority:low 2
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied and requires-triage removed from every labeled issue. Please spot-check the calls and close this issue when satisfied. Many issues already carried a type label, a priority, or both from their author; those were kept when they matched the guide.

Bugs

priority:critical

• Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently (#7330)
  • Area labels: area:expressions
  • Rationale: Native parse_url returns components or hosts where Spark returns NULL/INVALID_URL, a silent wrong result on a default-on expression.
• Native parse_url returns NULL for relative URLs (no scheme), Spark parses them (#7385)
  • Area labels: area:expressions
  • Rationale: Native parse_url silently returns NULL for scheme-less URLs where Spark returns values, a silent wrong result.

priority:high

• Native Iceberg write fails with "no more field nodes" on a struct column when its input is a Spark-to-Arrow conversion (#7239)
  • Area labels: area:writer, area:Iceberg
  • Rationale: Native Iceberg write task fails with a native exception on a struct column instead of falling back.
• Native execution fails with offset overflow when a large scalar string is expanded to the batch length (#7247)
  • Area labels: area:shuffle, area:aggregation, area:expressions
  • Rationale: An offset overflow fails the query on a native path; the author set priority:high and it matches the guide.
• Nested dictionary-encoded BINARY read as string overflows Utf8 offsets when flattened per batch (#7254)
  • Area labels: area:scan
  • Rationale: Native scan fails with an offset overflow on valid data that Spark reads, the same class as Native execution fails with offset overflow when a large scalar string is expanded to the batch length #7247.
• Executor JVM abort (NMT footer canary broken) when the in-memory cache is enabled on Spark 3.4 (#7305)
  • Area labels: area:scan
  • Rationale: Executor JVM aborts on native memory corruption (NMT footer canary), a crash.
• Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING (#7360)
  • Area labels: area:scan
  • Rationale: StringArray::new_unchecked is given invalid UTF-8, a memory-safety violation in native code (not a regression).
• GetStructField fails on a scalar struct beside a column (#7380)
  • Area labels: area:expressions
  • Rationale: CometNativeException on a scalar struct field beside a column fails the query on a native path.
• Comet 1.1.0: FileSourceScanExec.copy NoSuchMethodError on EMR Spark 4.1.1 (#7393)
  • Area labels: area:scan, spark 4.1
  • Rationale: NoSuchMethodError fails the cached query on a vendor Spark build, an unhandled exception on a supported code path.

priority:medium

• Native Iceberg write silently falls back on Iceberg 1.12: ManifestFiles.read(ManifestFile, FileIO) was removed (#7240)
  • Area labels: area:writer, area:Iceberg
  • Rationale: Native Iceberg writes silently fall back on Iceberg 1.12; the query still runs on Spark.
• Native memory requests treat an error from Spark as a refusal (#7244)
  • Area labels: area:shuffle
  • Rationale: An error from Spark during a memory request is treated as a refusal, so a task spills and continues instead of failing.
• Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF (#7245)
  • Area labels: area:expressions
  • Rationale: lpad/rpad fail on scalar string input, a broken feature with a fallback to Spark; the author-set priority:medium is kept.
• Native OR, AND, IN, concat, NULL operands and later top-K sort keys evaluate operands Spark skips, raising ANSI errors (#7246)
  • Area labels: area:expressions
  • Rationale: Native operators evaluate operands Spark skips and raise ANSI errors; the failure is visible, not silent, so the author-set priority:medium is kept.
• make_date under a codegen-dispatched parent falls back to Spark since fix: preserve make_date argument and parent evaluation #7261 (#7284)
  • Area labels: area:expressions
  • Rationale: Since fix: preserve make_date argument and parent evaluation #7261 make_date under a codegen-dispatched parent moves the whole operator back to Spark, a lost-native-coverage regression with identical results.
• JVM columnar shuffle writers use Comet defaults in map stages of jobs outside a SQL execution (#7303)
  • Area labels: area:shuffle
  • Rationale: JVM columnar shuffle writers ignore session spark.comet.* settings outside a SQL execution, a functional bug with a workaround.
• CapturedSQLConf applies a stale capture inside a SQL execution when every captured key was unset (#7312)
  • Area labels: area:shuffle
  • Rationale: A stale config capture overrides the session configs in a narrow case, a functional bug that needs a rerun map stage.
• Native Iceberg scan fails with ExpiredToken once REST-vended credentials expire (#7387)
  • Area labels: area:scan, area:Iceberg
  • Rationale: Long native Iceberg scans fail with ExpiredToken; a credential provider class is the workaround.
• The documented IcebergRESTVendedS3Provider never refreshes against a REST catalog (#7388)
  • Area labels: area:scan, area:Iceberg
  • Rationale: The documented reference provider never refreshes credentials against a REST catalog; a working provider is the workaround.
• S3 credential providers are retained until JVM shutdown when a catalog re-vends credentials (#7390)
  • Area labels: area:scan, area:Iceberg
  • Rationale: Superseded credential providers (HTTP client, auth session) are retained until JVM shutdown, a resource leak on long-lived executors.

priority:low

• ci: concurrency group is keyed on the branch name, so same-named fork branches can cancel each other's CI (#7315)
  • Area labels: area:ci
  • Rationale: CI concurrency key can cancel a same-named fork branch, a tooling problem not seen in practice.
• Delta contrib: a split classloader is reported as a Comet version mismatch (#7394)
  • Area labels: area:scan, area:Delta
  • Rationale: The scan falls back as designed but the log blames a version mismatch; the defect is a misleading message with a classpath workaround.

Enhancements

• Audit the PRs in 1.1.0 for regressions since 1.0.0 (#6399)
  • Area labels: none
  • Rationale: An audit task over the 1.1.0 PRs rather than a reported defect.
• Move the version checks for Spark patch-release fixes into SparkFixes (#7241)
  • Area labels: none
  • Rationale: Refactor that moves Spark patch-release version checks into SparkFixes.
• Run user-defined generators (Hive UDTFs, custom Generators) in the Comet pipeline (#7252)
  • Area labels: area:udf
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Offset overflow- Native shuffle writer: retry with fewer rows on offset overflow (#7269)
  • Area labels: area:shuffle
  • Rationale: Proposes retrying with fewer rows on offset overflow, a new capability tracked under [EPIC] Native operators overflow Arrow's 2 GiB offset limit on large string/binary columns #7267.
• [EPIC] Comet Native Executors: run Spark queries on native executors built on DataFusion Ballista (#7288)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• [EPIC] Iceberg work for 1.3.0: native writes on by default and scan correctness (#7293)
  • Area labels: area:Iceberg
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• perf: decode cached batches in native code instead of on the JVM (#7296)
  • Area labels: area:scan
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• perf: store dictionary-encoded columns as dictionaries in the in-memory cache (#7297)
  • Area labels: area:scan
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• perf: report vectorTypes from the Arrow cached batch serializer (#7298)
  • Area labels: area:scan
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• [EPIC] Native Spark expressions: run as many Spark built-in expressions as possible natively, by default (#7300)
  • Area labels: area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Extend sort-merge membership key reduction to anti joins and eligible conditions (#7317)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Evaluate membership key reduction across small shuffle batches (#7318)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• ci: a publish_snapshot.yml dry run still passes the Nexus secrets to the build (#7319)
  • Area labels: area:ci
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• ci: the docs deploy builds with its dependencies in a job that can write to the repository (#7320)
  • Area labels: area:ci
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Run filter lambdas natively with DataFusion's array_filter (#7321)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• perf: hll_sketch_agg merge builds a new union for every merged row (#7341)
  • Area labels: area:aggregation
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Evaluate making jemalloc the default allocator for the release build (#7344)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• docs: move existing pages to the style guide's "native" rules (#7358)
  • Area labels: none
  • Rationale: Documentation migration to the style guide.
• [EPIC] Track arrow-java bugs and limitations that Comet works around (#7362)
  • Area labels: area:ffi
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Add workflow to reject AI spam PRs (#7364)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: run LIMIT and top-K queries with a per-partition limit before the exchange (#7365)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: range-partition sorts on float, nested and expression keys (#7366)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: partition hash exchanges with Spark's Murmur3 hash (#7368)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: run a small broadcast side once rather than in every task that reads it (#7369)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: support dynamic partition pruning (#7370)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: support writes (#7371)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: cancel a query on the cluster when Spark cancels it (#7372)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: report a query's metrics in the Spark UI (#7373)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: pass scalar subquery results to the cluster without rewriting the plan (#7374)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: let the native planner reject every JVM callback, and drop the driver's list (#7375)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Comet Native Executors: reuse Flight connections in the driver's client (#7376)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Audit license and copyright texts for all native dependencies bundled in libcomet (#7383)
  • Area labels: none
  • Rationale: An audit of bundled license texts and tooling to generate them, a follow-up to fix: include licenses for bundled lz4_flex and Zstandard #7382.
• Comet Native Executors: experiment with a native client and server for Spark's shuffle block transfer protocol (#7384)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• No refreshing Iceberg credential provider on Spark 3.x builds (#7389)
  • Area labels: area:scan, area:Iceberg, spark 3.x
  • Rationale: Explicit request to ship a refreshing credential provider in Spark 3.x builds.
• Comet Native Executors: use Ballista's adaptive planning for joins inside Comet fragments (#7391)
  • Area labels: none
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Upgrade the datasketches crate to 0.6.0 and drop the HLL decode workarounds (#7395)
  • Area labels: area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Audit the HLL expressions against Spark on datasketches 0.6.0 and revisit the allowIncompatible opt-in (#7396)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• [EPIC] Native DataSketches: theta, KLL, frequent items and tuple sketches (#7397)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Native theta sketch functions (theta_*): audit the crate against Java and implement (#7398)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Native KLL sketch functions (kll_*): audit the crate against Java and implement (#7399)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Native approx_top_k functions: audit the frequent-items sketch against Java and implement (#7400)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.
• Native tuple sketch functions (tuple_*, Spark 4.2): audit the crate against Java and implement (#7401)
  • Area labels: area:aggregation, area:expressions
  • Rationale: Author-labeled request for new functionality, performance work or planning rather than a defect; type kept.

Escalations to consider

• Executor JVM abort (NMT footer canary broken) when the in-memory cache is enabled on Spark 3.4 (#7305)
  • A priority:high crash from memory corruption (a write past the end of a native block) may also produce wrong results silently before it is detected; consider priority:critical if the cause turns out to be corruption of cached data.
• Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING (#7360)
  • A priority:high memory-safety violation (invalid UTF-8 in a StringArray) could produce wrong results silently rather than a crash; consider priority:critical if so.

Skipped — needs more info

• Bug triage results: 2026-10-26 (#7230)
  • This is the previous bug triage summary issue, not a bug or enhancement; the reviewer closes it.
• TPC-DS q54/q68: local SF100 results and distributed reproduction limits (#6946)
  • A benchmark reproduction report for Investigate TPC-DS q54 and q68 slowdowns in the 1.1.0 benchmark on Spark 4.2 #6418 that says it is not a claim of a bug; it is unclear whether it is a bug, an enhancement or a record, and no defect is stated.