Skip to content

feat(bigquery): support jobs.insert in createQueryArrowStream - #9600

Open
ohmayr wants to merge 4 commits into
feat/bigquery-arrow-job-waitfrom
feat/bigquery-arrow-jobs-insert
Open

feat(bigquery): support jobs.insert in createQueryArrowStream#9600
ohmayr wants to merge 4 commits into
feat/bigquery-arrow-job-waitfrom
feat/bigquery-arrow-jobs-insert

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Some query options only work with jobs.insert (for example jobId, destination, dryRun, or jobTimeoutMs), so createQueryArrowStream failed for those queries. This runs them through createQueryJob instead.

  • Used when buildQueryRequest_ returns undefined (a jobs.insert-only option, or FAST_QUERY_PATH=DISABLED). timeoutMs and jobCreationMode only apply to jobs.query, so they stay out of the job configuration; timeoutMs bounds the wait instead.
  • Sets stream.jobReference and emits the job event.
  • A dryRun emits one 0-row, 0-column RecordBatch without waiting or calling ReadRows; the job's metadata holds the dry-run statistics.
  • Otherwise waits for the job to finish and streams from streams/_default at offset = 0. Errors thrown synchronously by createQueryJob destroy the stream.
  • If the consumer destroys the stream while jobs.insert runs, the stream stops without emitting job or waiting for the job.

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 13, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a fallback to route queries that cannot be executed via jobs.query through jobs.insert (createQueryJob) in createQueryArrowStream_, and adds comprehensive unit and system tests to verify this behavior. Feedback was provided to ensure type safety by guarding against an undefined job object before calling waitForJobCompletion and to use the idiomatic stream.push(null) instead of stream.end() to signal EOF.

Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr added this pull request to stack #9597 October 13, 2026 02:55
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 2e029c1 to 86c5363 Compare October 13, 2026 04:30
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 86c5363 to 3d6c4a9 Compare October 13, 2026 08:41
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 3d6c4a9 to 799d417 Compare October 13, 2026 09:33
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 799d417 to 90cb922 Compare October 13, 2026 10:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch 2 times, most recently from 03aec32 to 9853b3a Compare October 13, 2026 11:30
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 9853b3a to 0bba2d3 Compare October 13, 2026 12:00
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch 2 times, most recently from 2410242 to d689335 Compare October 15, 2026 09:55
quirogas
quirogas reviewed Oct 15, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Outdated
Comment thread handwritten/bigquery/src/arrow.ts Outdated
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from d689335 to 97f41ff Compare October 15, 2026 18:15
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 97f41ff to 7539631 Compare October 15, 2026 18:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch 3 times, most recently from 3469bd1 to 919d8fd Compare October 16, 2026 02:19
@ohmayr
ohmayr marked this pull request as ready for review October 16, 2026 02:33
@ohmayr
ohmayr requested a review from a team as a code owner October 16, 2026 02:33
@ohmayr
ohmayr requested a review from a team as a code owner October 16, 2026 02:33
@ohmayr ohmayr mentioned this pull request Oct 16, 2026
Open
@ohmayr

ohmayr commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the #9598/#9599 changes. Same behavior, restructured for the async flow: the jobs.insert route moved into its own streamQueryJob function and uses await bq.createQueryJob(...) instead of a callback, the dry-run batch uses pushBatches, and the "wait, then read every row" step is shared with the jobs.query path in streamJobWhenComplete.

@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 919d8fd to 046a050 Compare October 20, 2026 19:55
@ohmayr
feat(bigquery): support jobs.insert in createQueryArrowStream
e514462 
Routes queries that cannot use jobs.query (!queryReq, such as jobId,
dryRun, jobTimeoutMs, or other jobs.insert-only options) through
createQueryJob:
- Keeps timeoutMs and jobCreationMode, which only apply to jobs.query,
  out of the job configuration.
- Populates stream.jobReference and emits the job event once created.
- Emits a single 0-row, 0-column RecordBatch immediately when dryRun is
  true without polling getQueryResults or calling ReadRows.
- Otherwise waits for job completion via job.getQueryResults
  (maxResults: 0), bounded by timeoutMs like BigQuery#query, and
  streams Arrow batches from the anonymous default job stream
  (streams/_default) at offset = 0.
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-jobs-insert branch from 046a050 to e514462 Compare October 20, 2026 20:45
@ohmayr ohmayr mentioned this pull request Oct 20, 2026
Open
shivanee-p
shivanee-p approved these changes Oct 21, 2026
View reviewed changes

@shivanee-p shivanee-p left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a small comment, but the rest LGTM

Comment thread handwritten/bigquery/src/arrow.ts
// Throws for an invalid request, for example a destination that isn't a
// Table, or invalid parameters.
const [job, resp] = await bq.createQueryJob(jobQuery);
if (!job) {

@shivanee-p shivanee-p Oct 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need another if statement above this one with something like if (stream.destroyed) return;

@ohmayr ohmayr Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f41ec7d, with a test that a stream destroyed during jobs.insert emits no job event and doesn't wait for the job.

ohmayr added 2 commits October 21, 2026 00:39
@ohmayr
Merge branch 'feat/bigquery-arrow-job-wait' into feat/bigquery-arrow-…
47c64f7 
…jobs-insert

# Conflicts:
#	handwritten/bigquery/test/arrow.ts
@ohmayr
fix(bigquery): stop Arrow jobs.insert path after destroy; async stubs
f41ec7d 
Return after createQueryJob if the consumer destroyed the stream, so
no 'job' event is emitted and the job isn't waited on. Stub
createQueryJob with async functions in the unit tests, matching the
awaited promise API.

Also say in the createQueryArrowStream JSDoc that timeoutMs bounds the
jobs.insert wait too and that jobs.insert queries read every row with
the Storage Read API, and add system tests for timeoutMs and dryRun on
the jobs.insert path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quirogas quirogas quirogas left review comments
@shivanee-p shivanee-p shivanee-p approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ohmayr @quirogas @shivanee-p