Repository navigation
Conversation
decodeArrowPayload returned an empty RecordBatch when a record batch arrived without a schema, so its rows were silently dropped from createQueryArrowStream results. Throw "Got an Arrow record batch before its schema." instead, matching the Python client (googleapis/google-cloud-python#18791).
There was a problem hiding this comment.
Code Review
This pull request updates the decodeArrowPayload function to throw an error if an Arrow record batch is received without its schema, preventing rows from being silently dropped, and adds a corresponding test case. The review feedback recommends checking that serializedRecordBatch is present and has a length greater than 0 to ensure consistent truthiness handling between empty strings and empty Uint8Arrays.
| if (serializedRecordBatch) { | ||
| throw new Error('Got an Arrow record batch before its schema.'); | ||
| } |
There was a problem hiding this comment.
Checking serializedRecordBatch directly for truthiness can lead to inconsistent behavior because an empty string ("") is falsy, whereas an empty Uint8Array (new Uint8Array(0)) is truthy. To ensure consistent handling of empty payloads, check that serializedRecordBatch is both present and has a length greater than 0.
| if (serializedRecordBatch) { | |
| throw new Error('Got an Arrow record batch before its schema.'); | |
| } | |
| if (serializedRecordBatch && serializedRecordBatch.length > 0) { | |
| throw new Error('Got an Arrow record batch before its schema.'); | |
| } |
There was a problem hiding this comment.
Applied in 8f1679a. An empty payload has no rows to drop, so it now takes the same path as a missing one for both string and Uint8Array, matching Python's if batch_bytes:. Added a test for an empty Uint8Array without a schema.
An empty Uint8Array batch is truthy, so decodeArrowPayload raised "before its schema" for a payload with no rows. Check its length, so an empty string and an empty Uint8Array both count as no batch.
Context & Problem
createQueryArrowStreamturned a record batch that arrived without a schema into an emptyRecordBatch, silently dropping its rows. The skip has been onmainsince #9596, so this fixes it on its own, outside the Arrow stack.Summary of Changes
decodeArrowPayloadthrowsGot an Arrow record batch before its schema.when it gets batch bytes without a schema, matching the Python client (fix(bigquery): raise when an Arrow batch arrives before its schema google-cloud-python#18791).RecordBatch.Verification
jobs.queryresponse witharrowRecordBatchbut noarrowSchema; it fails without the fix.pnpm testinhandwritten/bigquery: 551 passing.GIT_DIFF_ARG=origin/main node ./bin/linter.mjs --strict: no problems.