Skip to content

refactor(bigquery-jdbc): move Arrow batch validation out of the result-set loop - #15069

Open
keshavdandeva wants to merge 1 commit into
jdbc/arrow-support-featurefrom
refactor/jdbc-arrow-batch-validation
Open

keshavdandeva wants to merge 1 commit into
jdbc/arrow-support-featurefrom
refactor/jdbc-arrow-batch-validation

Conversation

@keshavdandeva

Copy link
Copy Markdown
Contributor

b/582676785

Follow-up to #15048, for two review comments there: the old wrapper factory and the checks in loadNextBatch(). This is a refactor: the same checks, with the same error messages.

Changes

  • BigQueryArrowBatchWrapper owns the row-count rule. checkRowCount(actualRows) holds the comparison and the error text. validate() applies it to a message without batch data, which holds no rows. Both were in BigQueryArrowResultSet.
  • ArrowDeserializer owns the vectors it decodes into and their loader, as final fields. deserializeArrowBatch checks the row count before it loads the batch, then returns the decoded rows. If it fails, the vectors hold nothing. Before, it filled a field of the result set, and the loop checked the count afterwards.
  • The result set releases a batch through the deserializer (releaseBatch()). Its vectorSchemaRoot field is now only what the getters read.
  • loadNextBatch() keeps the control flow and the two checks on the result set's own state: a batch before its schema, and the total at the end of the stream.
  • BigQueryArrowBatchWrapper.of(batch, isLast...) is replaced by ofEndOfStream(), the name BigQueryFieldValueListWrapper already uses. Main code used the old factory only for that marker. getNestedFieldValueListWrapper loses its isLast vararg, which no caller passed.
  • The result set closes its own allocator, which is now private final. Before, ArrowDeserializer.close() closed it.

Three differences in behaviour, all on a stream that is already broken:

  • A message without batch data that reports a negative row count now fails. Before, it was skipped. The reader passes on ReadRowsResponse.row_count, which is not negative.
  • A batch that is rejected is released at once. Before, it stayed in the vectors until the next next() or close().
  • A batch that reports the wrong row count and also fails while its buffers are loaded into the vectors now fails with the row-count error. Before, it failed with the decoding error. A message that Arrow can't parse still fails with the decoding error, whatever it reports.

Decisions to check

  1. validate() is called from loadNextBatch(), not from the wrapper's factory. The reader thread calls the factory, and an earlier review asked that the reader only move data.
  2. The deserializer calls checkRowCount, not validate(). The row count is in the Arrow message, which only the deserializer parses, so the wrapper can't check it by itself. The rule and its message are still in the wrapper.
  3. The result set decides when a batch is released, and the deserializer releases it. That frees the last batch's buffers when iteration ends, not at close(), as before. A batch that fails to decode never reaches the result set: the deserializer drops it.
  4. getAllocatedMemory() is a new test-only accessor on the result set. Without it, no unit test can tell whether a batch was released before close().
  5. ArrowDeserializer stays an inner class. It reads the result set's allocator and logger, and writes none of its fields.

@keshavdandeva
keshavdandeva requested review from a team as code owners November 18, 2026 16:14

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors Arrow batch handling and row-count validation in the BigQuery JDBC driver. It introduces explicit validation of reported row counts within BigQueryArrowBatchWrapper and improves memory management in BigQueryArrowResultSet by ensuring Arrow vectors are cleared and released upon stream completion, iteration limits, or decoding failures. Additionally, comprehensive unit tests have been added to verify these behaviors. Feedback on the changes suggests wrapping the arrowDeserializer.close() call in a try-finally block to guarantee that allocator.close() is always executed, ensuring exception-safe resource cleanup.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant