Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds support for the BigQuery Storage Write API in the JDBC driver by introducing compatibility checks, JSON row creation, and parameter formatting helpers in BigQueryParameterHandler, BigQueryPreparedStatement, and BigQueryTypeRegistry, accompanied by comprehensive unit tests. The review feedback highlights a potential NullPointerException due to a missing null check on the dry run schema, an inefficiency in serializing byte arrays that should be resolved using Base64 encoding, and a potential Denial of Service vulnerability when validating BigDecimal values with extreme scale differences.
| } else if (formatted instanceof byte[]) { | ||
| byte[] bytes = (byte[]) formatted; | ||
| JsonArray byteArray = new JsonArray(); | ||
| for (byte b : bytes) { | ||
| byteArray.add(b); | ||
| } | ||
| rowObject.add(fieldName, byteArray); |
There was a problem hiding this comment.
Converting a byte[] into a JsonArray of individual byte numbers is highly inefficient. For larger byte arrays, this allocates a JsonPrimitive object for every single byte, leading to massive GC pressure and potential OutOfMemoryError. Since the BigQuery Storage Write API's JsonToProtoMessage natively supports Base64-encoded strings for bytes fields, we should serialize byte[] directly to a Base64 string.
| } else if (formatted instanceof byte[]) { | |
| byte[] bytes = (byte[]) formatted; | |
| JsonArray byteArray = new JsonArray(); | |
| for (byte b : bytes) { | |
| byteArray.add(b); | |
| } | |
| rowObject.add(fieldName, byteArray); | |
| } else if (formatted instanceof byte[]) { | |
| rowObject.addProperty(fieldName, java.util.Base64.getEncoder().encodeToString((byte[]) formatted)); |
| JsonArray encodedBytes = row.getAsJsonArray("b_col"); | ||
| assertEquals(3, encodedBytes.size()); | ||
| assertEquals(-128, encodedBytes.get(0).getAsByte()); | ||
| assertEquals(0, encodedBytes.get(1).getAsByte()); | ||
| assertEquals(127, encodedBytes.get(2).getAsByte()); |
There was a problem hiding this comment.
Update the test assertion to match the Base64-encoded string format for the BYTES column.
| JsonArray encodedBytes = row.getAsJsonArray("b_col"); | |
| assertEquals(3, encodedBytes.size()); | |
| assertEquals(-128, encodedBytes.get(0).getAsByte()); | |
| assertEquals(0, encodedBytes.get(1).getAsByte()); | |
| assertEquals(127, encodedBytes.get(2).getAsByte()); | |
| String encodedBytes = row.get("b_col").getAsString(); | |
| assertEquals(java.util.Base64.getEncoder().encodeToString(rawBytes), encodedBytes); |
| if (columnPrecision != null) { | ||
| long effectiveScale = columnScale != null ? columnScale : 0L; | ||
| if (normalized.scale() > effectiveScale) { | ||
| return false; | ||
| } | ||
| long maxIntDigits = columnPrecision - effectiveScale; | ||
| int intDigits = | ||
| normalized.signum() == 0 ? 0 : Math.max(0, normalized.precision() - normalized.scale()); | ||
| if (intDigits > maxIntDigits) { | ||
| return false; | ||
| } | ||
| } |
There was a problem hiding this comment.
Comparing BigDecimal values with extremely large scale differences (e.g., extremely negative scales) can cause BigDecimal.compareTo to hang or throw OutOfMemoryError because it attempts to align scales by multiplying the unscaled value. To prevent this Denial of Service (DoS) risk, we should calculate and validate the number of integer digits against the maximum allowed integer digits of maxMagnitude before performing any comparison.
int intDigits =
normalized.signum() == 0 ? 0 : Math.max(0, normalized.precision() - normalized.scale());
int maxIntDigitsAllowed = maxMagnitude.precision() - maxMagnitude.scale();
if (intDigits > maxIntDigitsAllowed) {
return false;
}
if (columnPrecision != null) {
long effectiveScale = columnScale != null ? columnScale : 0L;
if (normalized.scale() > effectiveScale) {
return false;
}
long maxIntDigits = columnPrecision - effectiveScale;
if (intDigits > maxIntDigits) {
return false;
}
}Stack: PR 2 of 3 (base: `fix/swa-1-writer-lifecycle`, stacked on #15084) ### Summary - Fix locale-dependent / misquoted Storage Write API serialization (`b/587340667`) where `createJsonRow()` used default `Gson` formatting for non-string values. - Reuse `BigQueryTypeRegistry` (`isSupportedWriteApiField`, `isSupportedJavaValue`, `extractInstant`) and `BigQueryParameterHandler` (`toQueryParameterValue`, `createWriteApiJsonRow`) across both SQL parameter binding and Storage Write API row serialization (`Locale.ROOT` / UTC `DATE`, `TIME`, `DATETIME`, `TIMESTAMP(6)` / `TIMESTAMP(12)`, `byte[]` `JsonArray`, and plain `BigDecimal` strings). - Validate batch parameter compatibility (`areBatchParametersCompatibleWithWriteApi`)—including `REQUIRED` nullability, `NUMERIC(P, S)` / `BIGNUMERIC(P, S)` bounds, finite `FLOAT64`, and strict `0001..9999` calendar/timestamp ranges—falling back to standard SQL DML when SQL-side coercion is required. ### Testing - `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (77 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
…nd align columns Stack: PR 3 of 3 (base: `fix/swa-2-unified-write-conversion`, stacked on #15084 -> #15085) ### Summary - Fix silent expression/UDF bypass (`b/587228731`) and `AppendSerializationError` when `EnableWriteAPI=true` is used with SQL expressions, `CAST`s, or UDFs inside `INSERT ... VALUES (...)`. - Add a zero-regex whitelist parser (`parseDirectInsertValues`) so only single-row `INSERT [INTO] <table> [(<cols>)] VALUES (?, ...) [;]` statements with bare `?` placeholders use the Storage Write API; all other `INSERT` forms fall back automatically to BigQuery SQL DML. - Align reordered column lists (`resolveInsertSchema`), reject partial/duplicate/pseudo-column lists, honor `DefaultDataset` propagation and project matching (`BigQueryStatement.generateBigQuerySettings`, `resolveTargetProject`), skip positional dry runs in Legacy SQL mode (`getUseLegacySql`), retry unbound dry runs at most once when tables are created after `prepareStatement()`, and fall back to SQL inside active transactions or `_SESSION` temp tables. ### Testing - `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (284 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
e938733 to
76207a3
Compare
b/587340667) wherecreateJsonRow()used defaultGsonformatting for non-string values.BigQueryTypeRegistry(isSupportedWriteApiField,isSupportedJavaValue,extractInstant) andBigQueryParameterHandler(toQueryParameterValue,createWriteApiJsonRow) across both SQL parameter binding and Storage Write API row serialization (Locale.ROOT/ UTCDATE,TIME,DATETIME,TIMESTAMP(6)/TIMESTAMP(12),byte[]JsonArray, and plainBigDecimalstrings).areBatchParametersCompatibleWithWriteApi)—includingREQUIREDnullability,NUMERIC(P, S)/BIGNUMERIC(P, S)bounds, finiteFLOAT64, and strict0001..9999calendar/timestamp ranges—falling back to standard SQL DML when SQL-side coercion is required.Testing
make lintandmake unittest test=BigQueryTypeRegistryTest,BigQueryParameterHandlerTest,BigQueryPreparedStatementWriteApiTest,BigQueryPreparedStatementSettersTest(84 tests passing, including multi-locale/timezoneJsonToProtoMessageround-trips).