Skip to content

fix(bigquery-jdbc): harden Storage Write API writer concurrency and batch lifecycle - #15084

Open
Neenu1995 wants to merge 1 commit into
mainfrom
fix/swa-1-writer-lifecycle
Open

Neenu1995 wants to merge 1 commit into
mainfrom
fix/swa-1-writer-lifecycle

Conversation

@Neenu1995

Copy link
Copy Markdown
Contributor
  • Fix a Phaser registration race in BigQueryJdbcBulkInsertWriter.append() by registering before submitting append(...) and deregistering on synchronous failure.
  • Make BigQueryJdbcBulkInsertWriter.cleanup() interruptible (awaitAdvanceInterruptibly) and add idempotent closeQuietly() cleanup on failure.
  • Keep the connection-cached BigQueryWriteClient open across BigQueryPreparedStatement.executeBatch() calls, wrap Storage Write API failures in BigQueryJdbcException, reject null SQL in BigQueryPreparedStatement, and clear batchParameters on clearBatch(), close(), and executeBatch() completion.

Testing

  • make lint and make unittest test=BigQueryJdbcBulkInsertWriterTest,BigQueryPreparedStatementWriteApiTest,BigQueryPreparedStatementSettersTest,BigQueryCallableStatementTest (154 tests passing).

@Neenu1995
Neenu1995 requested review from a team as code owners November 19, 2026 20:50
@Neenu1995
Neenu1995 added this pull request to stack #15087 November 19, 2026 20:51

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the BigQuery JDBC driver's bulk insert mechanism using the Storage Write API to improve thread safety, resource cleanup, and error handling in BigQueryJdbcBulkInsertWriter and BigQueryPreparedStatement. Key changes include registering the phaser before appending to prevent race conditions, ensuring proper stream closure during failures, validating batch parameters, and adding comprehensive unit tests. The reviewer recommended wrapping both the append call and the addCallback registration in the try-catch block to prevent the phaser from hanging if addCallback throws an exception.

Comment on lines 93 to 104
// Register before submitting so a synchronously completed future cannot drop the party count
// to zero (which permanently terminates the Phaser) before registration occurs.
openRequestCount.register();
ApiFuture<AppendRowsResponse> future;
try {
future = jsonStreamWriter.append(data, offset);
} catch (Throwable t) {
openRequestCount.arriveAndDeregister();
throw t;
}
ApiFutures.addCallback(
future, new AppendCompleteCallback(this), MoreExecutors.directExecutor());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If ApiFutures.addCallback throws an exception (for example, if future is null or due to class loading/runtime issues), the openRequestCount phaser will remain registered. This will cause cleanup() to hang indefinitely waiting for the unarrived party.

To prevent this, wrap both the append call and the addCallback registration inside the try block so that any exception during either step safely triggers the catch block and deregisters the party.

    // Register before submitting so a synchronously completed future cannot drop the party count
    // to zero (which permanently terminates the Phaser) before registration occurs.
    openRequestCount.register();
    try {
      ApiFuture<AppendRowsResponse> future = jsonStreamWriter.append(data, offset);
      ApiFutures.addCallback(
          future, new AppendCompleteCallback(this), MoreExecutors.directExecutor());
    } catch (Throwable t) {
      openRequestCount.arriveAndDeregister();
      throw t;
    }
References
  1. Catching Throwable instead of Exception is acceptable at Future boundaries or during stream cleanup in asynchronous operations to ensure all failures are captured and propagated.

…atch lifecycle

Stack: PR 1 of 3 (base: `main`)

### Summary
- Fix a `Phaser` registration race in `BigQueryJdbcBulkInsertWriter.append()` by registering before submitting `append(...)` and idempotently deregistering on synchronous or callback failure.
- Make `BigQueryJdbcBulkInsertWriter.cleanup()` interruptible (`awaitAdvanceInterruptibly`) and add idempotent `closeQuietly()` cleanup on failure.
- Keep the connection-cached `BigQueryWriteClient` open across `BigQueryPreparedStatement.executeBatch()` calls, wrap Storage Write API failures in `BigQueryJdbcException`, reject `null` SQL in `BigQueryPreparedStatement`, and clear `batchParameters` on `clearBatch()`, `close()`, and `executeBatch()` completion.

### Testing
- `make lint`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (148 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
Neenu1995 added a commit that referenced this pull request Nov 20, 2026
…nd align columns

Stack: PR 3 of 3 (base: `fix/swa-2-unified-write-conversion`, stacked on #15084 -> #15085)

### Summary
- Fix silent expression/UDF bypass (`b/587228731`) and `AppendSerializationError` when `EnableWriteAPI=true` is used with SQL expressions, `CAST`s, or UDFs inside `INSERT ... VALUES (...)`.
- Add a zero-regex whitelist parser (`parseDirectInsertValues`) so only single-row `INSERT [INTO] <table> [(<cols>)] VALUES (?, ...) [;]` statements with bare `?` placeholders use the Storage Write API; all other `INSERT` forms fall back automatically to BigQuery SQL DML.
- Align reordered column lists (`resolveInsertSchema`), reject partial/duplicate/pseudo-column lists, honor `DefaultDataset` propagation and project matching (`BigQueryStatement.generateBigQuerySettings`, `resolveTargetProject`), skip positional dry runs in Legacy SQL mode (`getUseLegacySql`), retry unbound dry runs at most once when tables are created after `prepareStatement()`, and fall back to SQL inside active transactions or `_SESSION` temp tables.

### Testing
- `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (284 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
@Neenu1995
Neenu1995 force-pushed the fix/swa-1-writer-lifecycle branch from ef02ae7 to dda0cd5 Compare November 20, 2026 00:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant