Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors the BigQuery JDBC driver's bulk insert mechanism using the Storage Write API to improve thread safety, resource cleanup, and error handling in BigQueryJdbcBulkInsertWriter and BigQueryPreparedStatement. Key changes include registering the phaser before appending to prevent race conditions, ensuring proper stream closure during failures, validating batch parameters, and adding comprehensive unit tests. The reviewer recommended wrapping both the append call and the addCallback registration in the try-catch block to prevent the phaser from hanging if addCallback throws an exception.
| // Register before submitting so a synchronously completed future cannot drop the party count | ||
| // to zero (which permanently terminates the Phaser) before registration occurs. | ||
| openRequestCount.register(); | ||
| ApiFuture<AppendRowsResponse> future; | ||
| try { | ||
| future = jsonStreamWriter.append(data, offset); | ||
| } catch (Throwable t) { | ||
| openRequestCount.arriveAndDeregister(); | ||
| throw t; | ||
| } | ||
| ApiFutures.addCallback( | ||
| future, new AppendCompleteCallback(this), MoreExecutors.directExecutor()); |
There was a problem hiding this comment.
If ApiFutures.addCallback throws an exception (for example, if future is null or due to class loading/runtime issues), the openRequestCount phaser will remain registered. This will cause cleanup() to hang indefinitely waiting for the unarrived party.
To prevent this, wrap both the append call and the addCallback registration inside the try block so that any exception during either step safely triggers the catch block and deregisters the party.
// Register before submitting so a synchronously completed future cannot drop the party count
// to zero (which permanently terminates the Phaser) before registration occurs.
openRequestCount.register();
try {
ApiFuture<AppendRowsResponse> future = jsonStreamWriter.append(data, offset);
ApiFutures.addCallback(
future, new AppendCompleteCallback(this), MoreExecutors.directExecutor());
} catch (Throwable t) {
openRequestCount.arriveAndDeregister();
throw t;
}References
- Catching Throwable instead of Exception is acceptable at Future boundaries or during stream cleanup in asynchronous operations to ensure all failures are captured and propagated.
…atch lifecycle Stack: PR 1 of 3 (base: `main`) ### Summary - Fix a `Phaser` registration race in `BigQueryJdbcBulkInsertWriter.append()` by registering before submitting `append(...)` and idempotently deregistering on synchronous or callback failure. - Make `BigQueryJdbcBulkInsertWriter.cleanup()` interruptible (`awaitAdvanceInterruptibly`) and add idempotent `closeQuietly()` cleanup on failure. - Keep the connection-cached `BigQueryWriteClient` open across `BigQueryPreparedStatement.executeBatch()` calls, wrap Storage Write API failures in `BigQueryJdbcException`, reject `null` SQL in `BigQueryPreparedStatement`, and clear `batchParameters` on `clearBatch()`, `close()`, and `executeBatch()` completion. ### Testing - `make lint`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (148 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
…nd align columns Stack: PR 3 of 3 (base: `fix/swa-2-unified-write-conversion`, stacked on #15084 -> #15085) ### Summary - Fix silent expression/UDF bypass (`b/587228731`) and `AppendSerializationError` when `EnableWriteAPI=true` is used with SQL expressions, `CAST`s, or UDFs inside `INSERT ... VALUES (...)`. - Add a zero-regex whitelist parser (`parseDirectInsertValues`) so only single-row `INSERT [INTO] <table> [(<cols>)] VALUES (?, ...) [;]` statements with bare `?` placeholders use the Storage Write API; all other `INSERT` forms fall back automatically to BigQuery SQL DML. - Align reordered column lists (`resolveInsertSchema`), reject partial/duplicate/pseudo-column lists, honor `DefaultDataset` propagation and project matching (`BigQueryStatement.generateBigQuerySettings`, `resolveTargetProject`), skip positional dry runs in Legacy SQL mode (`getUseLegacySql`), retry unbound dry runs at most once when tables are created after `prepareStatement()`, and fall back to SQL inside active transactions or `_SESSION` temp tables. ### Testing - `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (284 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
ef02ae7 to
dda0cd5
Compare
Phaserregistration race inBigQueryJdbcBulkInsertWriter.append()by registering before submittingappend(...)and deregistering on synchronous failure.BigQueryJdbcBulkInsertWriter.cleanup()interruptible (awaitAdvanceInterruptibly) and add idempotentcloseQuietly()cleanup on failure.BigQueryWriteClientopen acrossBigQueryPreparedStatement.executeBatch()calls, wrap Storage Write API failures inBigQueryJdbcException, rejectnullSQL inBigQueryPreparedStatement, and clearbatchParametersonclearBatch(),close(), andexecuteBatch()completion.Testing
make lintandmake unittest test=BigQueryJdbcBulkInsertWriterTest,BigQueryPreparedStatementWriteApiTest,BigQueryPreparedStatementSettersTest,BigQueryCallableStatementTest(154 tests passing).