feat(bigquery): stream multi-page Arrow results from default job stream by ohmayr · Pull Request #9598 · googleapis/google-cloud-node

ohmayr · 2026-10-13T01:10:45Z

jobs.query returns only the first page of results inline. When rows remain, this reads them from the job's default read stream (projects/{p}/locations/{l}/jobs/{j}/streams/_default), starting right after the inline page.

Rows remain when pageRowCount < totalRows or a pageToken is set. The pageToken check covers ARROW cache hits, where jobs.query reports totalRows: "0" and omits pageRowCount; those results are read from offset 0.
One BigQueryReadClient per BigQuery instance, created on first use with the client's credentials, project, universeDomain, and a 60s keepalive. The location comes from jobReference, falling back to the client's location.
ReadRows pauses while the consumer is behind and resumes on read, so a slow consumer doesn't buffer the whole result.
Ending iteration early cancels only that ReadRows stream; the shared client stays open.
Buffers at most one decoded RecordBatch ahead of the consumer (highWaterMark: 1).
If the consumer destroys the stream while jobs.query runs, no job event or batches are emitted.

gemini-code-assist

Code Review

This pull request adds support for streaming multi-page Arrow query results from BigQuery using the BigQuery Storage API (_default stream) when results exceed the inline page size. It introduces lazy creation and reuse of a BigQueryReadClient, handles location-specific streams, supports early cancellation, and adds comprehensive unit and system tests. Regarding the feedback, overriding stream._destroy on an instantiated stream is an anti-pattern and redundant since stream.once('close', ...) is already registered for cleanup; this override should be removed.

Implements the BigQuery Storage Read API fallback for inline jobs.query calls when rows remain after the first response page (pageRowCount < totalRows, or a pageToken is set): - Lazily initializes and reuses a single BigQueryReadClient on the BigQuery instance with explicit authClient, projectId, universeDomain, and 60s/60s gRPC keepalive settings. - Emits inline RecordBatch results first, then calls ReadRows on the anonymous default job stream (projects/{p}/locations/{l}/jobs/{j}/streams/_default) at offset = Number(pageRowCount). - Cancels only the active ReadRows gRPC stream if the caller terminates iteration early without closing the shared BigQueryReadClient.

shivanee-p

Left a few comments. I'm a little apprehensive about how streams are getting manually cancelled. Can you go through and confirm that all of the stream error listeners are 1) not destroyed early so an error event isn't emitted without a listener attached and 2) correctly detached on completion/destruction? I think some unit tests covering those scenarios would be helpful. Nice work on the implementation!

shivanee-p · 2026-10-16T22:13:34Z

  // TODO: Bump minimum version (here and in package.json) once
  // @google-cloud/bigquery-storage is released with
  // @google-cloud/bigquery-storage-api >= 0.5.0.


Can you add a link to a bug or issue to track this TODO?

Addressed in #9675.

shivanee-p · 2026-10-20T03:41:30Z

+    if (!finished) {
+      finished = true;
+      grpcStream.cancel();
+    }
+  };


Calling .cancel() will internally destroy the call and emit an error event with the status code 1 (CANCELLED) on the next event loop tick. This will cause an UnhandledException to be thrown.

I think attaching a no-op listener is the best way to go:

if (!finished) { grpcStream.on('error', () => {}); grpcStream.cancel(); grpcStream = null; }

Done in c4b2070. All ReadRows cleanup now goes through one settle(), which removes every listener, adds a no-op 'error' listener before cancel(), and only cancels if the call hasn't finished. grpcStream = null isn't needed: once the listeners and ctx.onRead are cleared, nothing references the stream.

shivanee-p · 2026-10-20T03:47:07Z

+        return;
      }
+
+      if (stream._ended || stream.destroyed) {


We shouldn't be referencing private API flags like _ended. We should instead only rely on either stream.destroyed or your own private variable within the code

Removed. The query now runs in one async function, and its outcome is the only place that ends or destroys the stream, so only stream.destroyed checks remain.

Run the query in one async function whose outcome is the only place that ends or destroys the stream, so helpers no longer check the private ResourceStream _ended flag. Wrap ReadRows in a promise that settles through one settle() on end, error, decode failure, or early destroy. settle() removes every listener, keeps a no-op 'error' listener so the CANCELLED error that follows cancel() can't crash the process, and cancels the call unless it already finished.

ohmayr · 2026-10-20T18:09:32Z

Thanks! Checked every place the stream ends: normal end, gRPC error, decode error, and early destroy all go through settle(). Added unit tests asserting no listeners are left behind and that a late CANCELLED error doesn't crash. Most of the streamReadRows diff is just re-indentation from wrapping it in a Promise. I'll apply the same async approach to the polling in #9599 and the read-session helpers in #9601.

@Private

BigQueryReadClient and StorageModule are exported only so the unit tests can stub them. Document them as @Private, like the exported helper functions in arrow.ts, so they stay out of the public API docs.

shivanee-p · 2026-10-20T23:35:55Z

+          });
+          process.nextTick(() => {
+            // Emit more batches in one tick than the output stream buffers
+            // (highWaterMark 16), before the consumer can read any of them.


nit: Do we need to leave highwaterMark at 16? Passing in 1 or 2 would be more efficient and prevent high memory spikes.

Set highWaterMark: 1 in 8c53d44, so at most one decoded RecordBatch is buffered ahead of the consumer.

shivanee-p · 2026-10-20T23:38:19Z

+ *
+ * @private
+ */
+function streamReadRows(


This function needs to be explicitly defined as async

Done in 8c53d44.

shivanee-p · 2026-10-20T23:41:45Z

+    );
+  };

+  stream._read = () => {


We should avoid internal variable names beginning with _. Node provides the Simplified Construction API so callers pass a read() option

Switched to the read option in 8c53d44, passed through ResourceStream's streamOptions.

shivanee-p · 2026-10-20T23:49:52Z

+    }
+    started = true;
+    // The only place that ends or destroys the stream.
+    void (async () => {


Now that runQuery() is asynchronous, we don't need to have a try/catch within this definition. We can move the try/catch into the runQuery() definition.

Moved in 8c53d44: runQuery now owns the try/catch and is the only place that ends or destroys the stream; the query body is streamQueryResults.

shivanee-p · 2026-10-20T23:52:44Z

-        return;
-      }
+    const [job, res] = (await bq.runJobsQuery(queryReq)) as JobsQueryResponse;



We should add if (stream.destroyed) return; after await bq.runJobsQuery(queryReq) to avoid emitting events on a closed stream

Added in 8c53d44, with a test that a stream destroyed during jobs.query emits no job event or batches.

Address review feedback on the default job stream: - Return after jobs.query if the consumer destroyed the stream, so no 'job' event or batches are emitted on a closed stream. - Buffer at most one RecordBatch (highWaterMark 1) instead of 16. - Pass read() through streamOptions instead of assigning stream._read, and move the try/catch that ends or destroys the stream into runQuery. - Mark streamReadRows async. - Document calling destroy() when stopping early, and that the no-op ReadRows 'error' listener swallows every later error.

product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026

gemini-code-assist Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread handwritten/bigquery/src/arrow.ts Outdated

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from 6f50082 to 5a4b92b Compare October 13, 2026 02:11

ohmayr added this pull request to stack #9597 October 13, 2026 02:12

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from 5a4b92b to 15beb0e Compare October 13, 2026 02:41

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch 2 times, most recently from 42ce043 to 3e91b88 Compare October 13, 2026 08:41

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from 3e91b88 to 579849b Compare October 13, 2026 09:33

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from 579849b to 38ab043 Compare October 13, 2026 10:27

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch 2 times, most recently from 498668a to f9a6776 Compare October 13, 2026 11:30

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from f9a6776 to 67855b7 Compare October 13, 2026 12:00

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch 2 times, most recently from 78bf25d to ed46fbf Compare October 15, 2026 09:55

quirogas reviewed Oct 15, 2026

View reviewed changes

Comment thread handwritten/bigquery/src/arrow.ts Outdated

Comment thread handwritten/bigquery/src/arrow.ts Outdated

Comment thread handwritten/bigquery/system-test/arrow.ts

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch 4 times, most recently from 311a1db to 2948937 Compare October 16, 2026 01:12

Base automatically changed from feat/bigquery-arrow-inline to main October 16, 2026 02:19

ohmayr force-pushed the feat/bigquery-arrow-default-stream branch from 2948937 to d5bdcaf Compare October 16, 2026 02:19

ohmayr marked this pull request as ready for review October 16, 2026 02:19

ohmayr requested review from a team as code owners October 16, 2026 02:19

shivanee-p reviewed Oct 20, 2026

View reviewed changes

shivanee-p mentioned this pull request Oct 20, 2026

feat(bigquery): wait for unfinished jobs before streaming Arrow results #9599

Open

ohmayr mentioned this pull request Oct 20, 2026

feat(bigquery): support jobs.insert in createQueryArrowStream #9600

Open

docs(bigquery): mark test-only Arrow types private

d1472db

BigQueryReadClient and StorageModule are exported only so the unit tests can stub them. Document them as @Private, like the exported helper functions in arrow.ts, so they stay out of the public API docs.

ohmayr mentioned this pull request Oct 20, 2026

feat(bigquery): support explicit destination tables in createQueryArrowStream #9601

Open

Merge branch 'main' into feat/bigquery-arrow-default-stream

06ddc35

ohmayr mentioned this pull request Oct 20, 2026

feat(bigquery): resume interrupted Arrow ReadRows streams #9602

Open

shivanee-p requested changes Oct 20, 2026

View reviewed changes

Conversation

ohmayr commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shivanee-p left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ohmayr commented Oct 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ohmayr commented Oct 13, 2026 •

edited

Loading