Skip to content

fix(bigquery-jdbc): unify Storage Write API and SQL parameter conversion - #15085

Open
Neenu1995 wants to merge 1 commit into
fix/swa-1-writer-lifecyclefrom
fix/swa-2-unified-write-conversion
Open

Neenu1995 wants to merge 1 commit into
fix/swa-1-writer-lifecyclefrom
fix/swa-2-unified-write-conversion

Conversation

@Neenu1995

Copy link
Copy Markdown
Contributor
  • Fix locale-dependent / misquoted Storage Write API serialization (b/587340667) where createJsonRow() used default Gson formatting for non-string values.
  • Reuse BigQueryTypeRegistry (isSupportedWriteApiField, isSupportedJavaValue, extractInstant) and BigQueryParameterHandler (toQueryParameterValue, createWriteApiJsonRow) across both SQL parameter binding and Storage Write API row serialization (Locale.ROOT / UTC DATE, TIME, DATETIME, TIMESTAMP(6) / TIMESTAMP(12), byte[] JsonArray, and plain BigDecimal strings).
  • Validate batch parameter compatibility (areBatchParametersCompatibleWithWriteApi)—including REQUIRED nullability, NUMERIC(P, S) / BIGNUMERIC(P, S) bounds, finite FLOAT64, and strict 0001..9999 calendar/timestamp ranges—falling back to standard SQL DML when SQL-side coercion is required.

Testing

  • make lint and make unittest test=BigQueryTypeRegistryTest,BigQueryParameterHandlerTest,BigQueryPreparedStatementWriteApiTest,BigQueryPreparedStatementSettersTest (84 tests passing, including multi-locale/timezone JsonToProtoMessage round-trips).

@Neenu1995
Neenu1995 requested review from a team as code owners November 19, 2026 20:50
@Neenu1995
Neenu1995 added this pull request to stack #15087 November 19, 2026 20:51

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the BigQuery Storage Write API in the JDBC driver by introducing compatibility checks, JSON row creation, and parameter formatting helpers in BigQueryParameterHandler, BigQueryPreparedStatement, and BigQueryTypeRegistry, accompanied by comprehensive unit tests. The review feedback highlights a potential NullPointerException due to a missing null check on the dry run schema, an inefficiency in serializing byte arrays that should be resolved using Base64 encoding, and a potential Denial of Service vulnerability when validating BigDecimal values with extreme scale differences.

Comment on lines +545 to +551
} else if (formatted instanceof byte[]) {
byte[] bytes = (byte[]) formatted;
JsonArray byteArray = new JsonArray();
for (byte b : bytes) {
byteArray.add(b);
}
rowObject.add(fieldName, byteArray);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Converting a byte[] into a JsonArray of individual byte numbers is highly inefficient. For larger byte arrays, this allocates a JsonPrimitive object for every single byte, leading to massive GC pressure and potential OutOfMemoryError. Since the BigQuery Storage Write API's JsonToProtoMessage natively supports Base64-encoded strings for bytes fields, we should serialize byte[] directly to a Base64 string.

Suggested change
} else if (formatted instanceof byte[]) {
byte[] bytes = (byte[]) formatted;
JsonArray byteArray = new JsonArray();
for (byte b : bytes) {
byteArray.add(b);
}
rowObject.add(fieldName, byteArray);
} else if (formatted instanceof byte[]) {
rowObject.addProperty(fieldName, java.util.Base64.getEncoder().encodeToString((byte[]) formatted));

Comment on lines +231 to +235
JsonArray encodedBytes = row.getAsJsonArray("b_col");
assertEquals(3, encodedBytes.size());
assertEquals(-128, encodedBytes.get(0).getAsByte());
assertEquals(0, encodedBytes.get(1).getAsByte());
assertEquals(127, encodedBytes.get(2).getAsByte());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Update the test assertion to match the Base64-encoded string format for the BYTES column.

Suggested change
JsonArray encodedBytes = row.getAsJsonArray("b_col");
assertEquals(3, encodedBytes.size());
assertEquals(-128, encodedBytes.get(0).getAsByte());
assertEquals(0, encodedBytes.get(1).getAsByte());
assertEquals(127, encodedBytes.get(2).getAsByte());
String encodedBytes = row.get("b_col").getAsString();
assertEquals(java.util.Base64.getEncoder().encodeToString(rawBytes), encodedBytes);

Comment on lines +733 to +744
if (columnPrecision != null) {
long effectiveScale = columnScale != null ? columnScale : 0L;
if (normalized.scale() > effectiveScale) {
return false;
}
long maxIntDigits = columnPrecision - effectiveScale;
int intDigits =
normalized.signum() == 0 ? 0 : Math.max(0, normalized.precision() - normalized.scale());
if (intDigits > maxIntDigits) {
return false;
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

Comparing BigDecimal values with extremely large scale differences (e.g., extremely negative scales) can cause BigDecimal.compareTo to hang or throw OutOfMemoryError because it attempts to align scales by multiplying the unscaled value. To prevent this Denial of Service (DoS) risk, we should calculate and validate the number of integer digits against the maximum allowed integer digits of maxMagnitude before performing any comparison.

    int intDigits =
        normalized.signum() == 0 ? 0 : Math.max(0, normalized.precision() - normalized.scale());
    int maxIntDigitsAllowed = maxMagnitude.precision() - maxMagnitude.scale();
    if (intDigits > maxIntDigitsAllowed) {
      return false;
    }
    if (columnPrecision != null) {
      long effectiveScale = columnScale != null ? columnScale : 0L;
      if (normalized.scale() > effectiveScale) {
        return false;
      }
      long maxIntDigits = columnPrecision - effectiveScale;
      if (intDigits > maxIntDigits) {
        return false;
      }
    }

Stack: PR 2 of 3 (base: `fix/swa-1-writer-lifecycle`, stacked on #15084)

### Summary
- Fix locale-dependent / misquoted Storage Write API serialization (`b/587340667`) where `createJsonRow()` used default `Gson` formatting for non-string values.
- Reuse `BigQueryTypeRegistry` (`isSupportedWriteApiField`, `isSupportedJavaValue`, `extractInstant`) and `BigQueryParameterHandler` (`toQueryParameterValue`, `createWriteApiJsonRow`) across both SQL parameter binding and Storage Write API row serialization (`Locale.ROOT` / UTC `DATE`, `TIME`, `DATETIME`, `TIMESTAMP(6)` / `TIMESTAMP(12)`, `byte[]` `JsonArray`, and plain `BigDecimal` strings).
- Validate batch parameter compatibility (`areBatchParametersCompatibleWithWriteApi`)—including `REQUIRED` nullability, `NUMERIC(P, S)` / `BIGNUMERIC(P, S)` bounds, finite `FLOAT64`, and strict `0001..9999` calendar/timestamp ranges—falling back to standard SQL DML when SQL-side coercion is required.

### Testing
- `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (77 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
Neenu1995 added a commit that referenced this pull request Nov 20, 2026
…nd align columns

Stack: PR 3 of 3 (base: `fix/swa-2-unified-write-conversion`, stacked on #15084 -> #15085)

### Summary
- Fix silent expression/UDF bypass (`b/587228731`) and `AppendSerializationError` when `EnableWriteAPI=true` is used with SQL expressions, `CAST`s, or UDFs inside `INSERT ... VALUES (...)`.
- Add a zero-regex whitelist parser (`parseDirectInsertValues`) so only single-row `INSERT [INTO] <table> [(<cols>)] VALUES (?, ...) [;]` statements with bare `?` placeholders use the Storage Write API; all other `INSERT` forms fall back automatically to BigQuery SQL DML.
- Align reordered column lists (`resolveInsertSchema`), reject partial/duplicate/pseudo-column lists, honor `DefaultDataset` propagation and project matching (`BigQueryStatement.generateBigQuerySettings`, `resolveTargetProject`), skip positional dry runs in Legacy SQL mode (`getUseLegacySql`), retry unbound dry runs at most once when tables are created after `prepareStatement()`, and fall back to SQL inside active transactions or `_SESSION` temp tables.

### Testing
- `make lint`, `mvn -B -ntp verify -DskipTests -DskipITs`, `mvn -q -Denforcer.skip=true -Dclirr.skip=true test-compile`, and `make unittest` (284 targeted tests passing; full suite 1305/1306 non-skipped passing, 1 pre-existing `BigQueryJdbcOAuthUtilityTest` P12 env failure; ITs require GCP credentials).
@Neenu1995
Neenu1995 force-pushed the fix/swa-2-unified-write-conversion branch from e938733 to 76207a3 Compare November 20, 2026 00:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant