Skip to content

feat(bigquery): resume interrupted Arrow ReadRows streams - #9602

Open
ohmayr wants to merge 3 commits into
mainfrom
feat/bigquery-arrow-stream-resume
Open

ohmayr wants to merge 3 commits into
mainfrom
feat/bigquery-arrow-stream-resume

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Long ReadRows streams can drop on transient gRPC errors, which failed the whole read even though the Storage Read API can resume from any row. This reopens the stream at the next unread row.

  • Retries UNAVAILABLE, and INTERNAL errors caused by RST_STREAM, up to 3 times in a row. Any received batch resets the count. DEADLINE_EXCEEDED isn't retried, since it means the caller's deadline ran out.
  • An abandoned stream loses only our own listeners and keeps a no-op error listener, since google-gax emits CANCELLED after cancel().
  • Errors that aren't retryable, or persist after 3 retries, destroy the stream after the batches already yielded.

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces automatic stream resumption for BigQuery's Arrow stream reader (createQueryArrowStream) by retrying transient gRPC errors (such as UNAVAILABLE, DEADLINE_EXCEEDED, and INTERNAL with RST_STREAM) from the last successfully read offset up to a maximum of three times. Comprehensive unit tests are also added to verify this behavior. The review feedback highlights a critical issue regarding the use of removeAllListeners() on the gRPC stream during cancellation and retries, which can inadvertently strip internal library listeners and cause resource leaks. It is recommended to avoid removeAllListeners() and instead explicitly detach only the registered listeners (data, error, end) using named functions.

Comment thread handwritten/bigquery/src/arrow.ts Outdated
Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr added this pull request to stack #9597 October 13, 2026 07:08
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from e8fd2fb to 72ed966 Compare October 13, 2026 09:33
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from d4bf688 to da4e16d Compare October 13, 2026 10:42
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from a1856e7 to 663534b Compare October 13, 2026 12:00
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from da77b2d to c8290d3 Compare October 15, 2026 09:55
Comment thread handwritten/bigquery/src/arrow.ts Outdated
Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 2 times, most recently from 7614972 to 9197508 Compare October 15, 2026 18:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch 3 times, most recently from 8fe5111 to 35977c3 Compare October 16, 2026 02:19
@ohmayr
ohmayr marked this pull request as ready for review October 16, 2026 02:34
@ohmayr
ohmayr requested review from a team as code owners October 16, 2026 02:34
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 35977c3 to 9df7133 Compare October 16, 2026 21:36
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 9df7133 to 35977c3 Compare October 16, 2026 21:50
@ohmayr

ohmayr commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the #9598–#9601 changes. Same behavior and retry rules. The retry is now a short async loop in streamReadRows around a single ReadRows attempt (readRowsOnce), so each failed attempt goes through the same listener cleanup as a normal finish, and the next attempt resumes at the next unread row. The tests check that every abandoned stream keeps none of our listeners.

@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 35977c3 to ec5ff11 Compare October 20, 2026 21:48
Base automatically changed from feat/bigquery-arrow-destination-table to main October 21, 2026 19:05
Resume Storage Read API ReadRows streams at offset + rowsReceived when
interrupted by transient gRPC errors (DEADLINE_EXCEEDED, UNAVAILABLE,
or INTERNAL with RST_STREAM), up to MAX_STREAM_RETRIES consecutive
attempts, waiting 100 ms before the first reopen and doubling the wait
for each consecutive retry. Surface non-retryable errors or exhausted
retries on the stream even after earlier batches have been emitted.

Abandoned gRPC streams (on reconnect or early cancel) keep a no-op
'error' listener, because google-gax emits CANCELLED asynchronously
after cancel() and an unhandled 'error' event would crash the process.
DEADLINE_EXCEEDED on ReadRows means the caller's deadline ran out, so
pass it to the caller instead of reopening the stream, matching the
design doc and the Python reader. Route the retry backoff through
retryBackoff.wait so unit tests replace it instead of sleeping on real
timers, and check the 100/200/400 ms sequence. Mark readRowsOnce async.
Reopen ReadRows at the next unread row offset without an added
client-side timer on mid-stream UNAVAILABLE and RST_STREAM drops,
matching ReadStream in @google-cloud/bigquery-storage and Python's
ReadRowsStream. Initial connection failures when opening or reopening
a ReadRows stream are already retried with exponential backoff by
google-gax (retry-request).
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-stream-resume branch from 6742228 to ef7ca58 Compare October 21, 2026 19:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants