Skip to content

perf(bigquery): pre-size InsertAll batches and response maps - #15011

Open
lqiu96 wants to merge 1 commit into
googleapis:mainfrom
lqiu96:perf/bq-t1-10-insertall-pre-sizing
Open

lqiu96 wants to merge 1 commit into
googleapis:mainfrom
lqiu96:perf/bq-t1-10-insertall-pre-sizing

Conversation

@lqiu96

@lqiu96 lqiu96 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Reduces collection allocations and resizing in InsertAllRequest, BigQueryImpl.insertAll, and InsertAllResponse without changing lazy evaluation semantics:

  • InsertAllRequest.Builder.setRows: Stages rows in an ArrayList instead of a LinkedList, avoiding per-row Node allocations before build() copies them into an ImmutableList.
  • BigQueryImpl.insertAll: Replaces FluentIterable.from(rows).transform(...).toList() and its boolean[] wrapper with a loop into new ArrayList<>(rows.size()).
  • InsertAllResponse.toPb / fromPb: Uses ImmutableList.builderWithExpectedSize(insertErrors.size()) in toPb() and ImmutableMap.builderWithExpectedSize(errorsPb.size()).buildKeepingLast() in fromPb(), avoiding a temporary HashMap copy.

Why Lazy Evaluation Is Unaffected

  • BigQueryImpl.insertAll previously called .toList() on FluentIterable.transform(...), and InsertAllResponse.toPb() previously called ImmutableList.copyOf(...) on Iterables.transform(...). Both already materialized immediately on the same statement. The lazy wrappers only hid .size() and caused array resizing.
  • Per-row BigQueryError lists in InsertAllResponse still use lazy Lists.transform views, preserving existing behavior.

Verification

  • mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=InsertAllRequestTest,BigQueryImplTest -Dclirr.skip=true -Denforcer.skip=true -Dflatten.skip=true (InsertAllRequestTest: 21 passed, BigQueryImplTest: 133 passed)
  • mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=InsertAllResponseTest -Dclirr.skip=true -Denforcer.skip=true -Dflatten.skip=true (4 passed)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several performance and memory optimizations across the BigQuery client library. Key improvements include replacing lazy Guava transformations with eager materialization using pre-sized collections, converting a LinkedList to an ArrayList in InsertAllRequest to reduce per-row memory overhead, and simplifying a mutable single-element boolean array to a standard boolean variable in BigQueryImpl. I have no feedback to provide as there are no review comments.

@lqiu96
lqiu96 force-pushed the perf/bq-t1-10-insertall-pre-sizing branch from a551158 to 89636a1 Compare November 14, 2026 01:04
@lqiu96 lqiu96 changed the title perf(bigquery): pre-size InsertAll batches and eagerly materialize error lists perf(bigquery): pre-size InsertAll batches and response maps Nov 14, 2026
@lqiu96
lqiu96 marked this pull request as ready for review November 19, 2026 19:39
@lqiu96
lqiu96 requested review from a team as code owners November 19, 2026 19:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant