From 6dc3fdf3d2e54dc74981d880592b74134acd549a Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Thu, 23 Jul 2026 13:50:57 -0400 Subject: [PATCH 1/8] docs(bigquery-jdbc): add user guide with connection property and custom endpoint reference --- java-bigquery-jdbc/USER_GUIDE.md | 992 +++++++++++++++++++++++++++++++ 1 file changed, 992 insertions(+) create mode 100644 java-bigquery-jdbc/USER_GUIDE.md diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md new file mode 100644 index 000000000000..f93f4fb49fc6 --- /dev/null +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -0,0 +1,992 @@ +# Google BigQuery JDBC Driver User Guide + +This guide provides comprehensive instructions for configuring, developing with, and optimizing the **Google BigQuery JDBC Driver** (`google-cloud-bigquery-jdbc`). + +> [!NOTE] +> This guide is aligned with and references the official [Google Cloud BigQuery JDBC Documentation](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) and [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver). + +--- + +## Table of Contents +1. [Overview & Prerequisites](#1-overview--prerequisites) +2. [Connection URL Syntax & Quickstart](#2-connection-url-syntax--quickstart) +3. [Authentication Modes & Configuration](#3-authentication-modes--configuration) +4. [Connection Properties Reference](#4-connection-properties-reference) +5. [Data Type Mapping Reference](#5-data-type-mapping-reference) +6. [JDBC Driver Architecture & Core Features](#6-jdbc-driver-architecture--core-features) + - [Transaction Management & Multi-Statement Sessions](#transaction-management--multi-statement-sessions) + - [Type Mapping & Coercion Engine](#type-mapping--coercion-engine) + - [High-Throughput Storage Read & Write APIs](#high-throughput-storage-read--write-apis) + - [Diagnostic MDC Tracing & Connection Proxying](#diagnostic-mdc-tracing--connection-proxying) +7. [Common User Journeys (CUJs) & Code Recipes](#7-common-user-journeys-cujs--code-recipes) + - [CUJ 1: Financial & Multi-Table Transactions](#cuj-1-financial--multi-table-transactions) + - [CUJ 2: High-Performance ETL Ingestion Pipeline](#cuj-2-high-performance-etl-ingestion-pipeline) + - [CUJ 3: High-Throughput Result Extraction via Storage Read API](#cuj-3-high-throughput-result-extraction-via-storage-read-api) + - [CUJ 4: Parameterized Queries with JSR-310 & Numeric Types](#cuj-4-parameterized-queries-with-jsr-310--numeric-types) + - [CUJ 5: Nested Structs & Repeated Array Operations](#cuj-5-nested-structs--repeated-array-operations) + - [CUJ 6: Service Account Impersonation & Cross-Project Billing](#cuj-6-service-account-impersonation--cross-project-billing) + - [CUJ 7: Stored Procedure Execution](#cuj-7-stored-procedure-execution) +8. [High-Throughput & Performance Tuning](#8-high-throughput--performance-tuning) +9. [Framework & BI Tool Integrations](#9-framework--bi-tool-integrations) + - [Spring Boot & HikariCP](#spring-boot--hikaricp) + - [SQL Clients (DBeaver / DataGrip)](#sql-clients-dbeaver--datagrip) +10. [Logging, Diagnostics & Troubleshooting](#10-logging-diagnostics--troubleshooting) +11. [Official Documentation References](#11-official-documentation-references) + +--- + +## 1. Overview & Prerequisites + +The BigQuery JDBC Driver enables Java applications, BI tools, and ETL pipelines to interact with Google Cloud BigQuery using the standard Java Database Connectivity (JDBC) API (JDBC 4.2 compliant). + +### Prerequisites +- **Java Runtime**: JDK 8 or higher (JDK 11, 17, or 21 recommended). +- **Google Cloud Platform**: + - An active GCP Project with BigQuery API enabled. + - Required IAM roles (e.g., `BigQuery Data Viewer`, `BigQuery Job User`). + +### Installation Coordinates + +**Maven**: +```xml + + com.google.cloud + google-cloud-bigquery-jdbc + 1.1.0 + +``` + +**Gradle**: +```groovy +implementation 'com.google.cloud:google-cloud-bigquery-jdbc:1.1.0' +``` + +--- + +## 2. Connection URL Syntax & Quickstart + +The JDBC connection string format for BigQuery is: + +``` +jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=;[DefaultDataset=;][Property1=Value1;...] +``` + +### Basic Quickstart Example + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; + +public class BigQueryQuickstart { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project-id" + + ";DefaultDataset=my_dataset" + + ";OAuthType=0"; // 0 = Application Default Credentials (ADC) + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT name, age FROM users LIMIT 10")) { + + while (rs.next()) { + System.out.printf("User: %s, Age: %d%n", rs.getString("name"), rs.getInt("age")); + } + } + } +} +``` + +--- + +## 3. Authentication Modes & Configuration + +The driver supports multiple OAuth 2.0 and identity workflows specified via the `OAuthType` connection property. + +| `OAuthType` | Authentication Strategy | Required Properties | +| :---: | :--- | :--- | +| **`0`** | **Application Default Credentials (ADC)** / gcloud CLI | None (reads `GOOGLE_APPLICATION_CREDENTIALS` or `gcloud` context) | +| **`1`** | **Service Account Key File** | `OAuthPvtKeyPath` (path to `.json` or `.p12` service account key) | +| **`1`** | **Service Account Private Key String** | `OAuthServiceAcctEmail`, `OAuthPvtKey` (raw private key contents) | +| **`2`** | **OAuth 2.0 Access Token / Refresh Token** | `OAuthRefreshToken`, `OAuthClientId`, `OAuthClientSecret` | +| **`3`** | **User Account / Service Account Email** | `OAuthServiceAcctEmail` | +| **`4`** | **Pre-generated Access Token** | `OAuthAccessToken` | +| **`5`** | **Workload Identity Federation (BYOID)** | `BYOID_AudienceUri`, `BYOID_CredentialSource`, `BYOID_SubjectTokenType` | + +### Service Account Authentication Example +```java +String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project-id" + + ";OAuthType=1" + + ";OAuthPvtKeyPath=/path/to/service-account-key.json"; +``` + +### Service Account Impersonation +To execute queries as an impersonated service account: +```java +String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project-id" + + ";OAuthType=0" + + ";ServiceAccountImpersonationEmail=target-sa@my-gcp-project.iam.gserviceaccount.com"; +``` + +--- + +## 4. Connection Properties Reference + +### Authentication & Impersonation Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `OAuthType` | `-1` | Authentication mechanism: `0` (Service Account), `1` (User Account), `2` (Pre-generated Token / Refresh Token), `3` (ADC), `4` (Workload Identity Federation / BYOID). | +| `OAuthServiceAcctEmail` | `null` | Service Account email address for Service Account authentication or token scope. | +| `OAuthPvtKeyPath` | `null` | Path to JSON or P12 private key file for Service Account authentication. | +| `OAuthPvtKey` | `null` | Raw PKCS#8 private key string content for Service Account authentication. | +| `OAuthP12Password` | `notasecret` | Password for accessing encrypted `.p12` key files. | +| `OAuthAccessToken` | `null` | Pre-generated OAuth 2.0 access token string. | +| `OAuthAccessTokenReadonly` | `false` | Indicates whether the pre-generated access token has read-only scope. | +| `OAuthRefreshToken` | `null` | Pre-generated refresh token to automatically mint access tokens. | +| `OAuthClientId` | *Google Client ID* | OAuth 2.0 Client ID for user authentication or refresh token flows. | +| `OAuthClientSecret` | *Google Secret* | OAuth 2.0 Client Secret for user authentication or refresh token flows. | +| `ServiceAccountImpersonationEmail` | `null` | Target Service Account email to impersonate for query job execution. | +| `ServiceAccountImpersonationChain` | `null` | Comma-separated list of service account emails representing the impersonation chain. | +| `ServiceAccountImpersonationScopes` | `https://www.googleapis.com/auth/bigquery` | Comma-separated OAuth 2.0 scopes for impersonated credentials. | +| `ServiceAccountImpersonationTokenLifetime` | `3600` | Lifetime (in seconds) for impersonated service account tokens. | +| `RequestGoogleDriveScope` | `0` | If `1` (or `true`), appends the Google Drive read-only scope (`https://www.googleapis.com/auth/drive.readonly`) to query external Drive tables. | + +### Workload Identity Federation (BYOID) Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `BYOID_AudienceUri` | `null` | Audience URI corresponding to the Workforce/Workload identity pool provider. | +| `BYOID_CredentialSource` | `null` | File path or JSON content defining the external subject token source. | +| `BYOID_PoolUserProject` | `null` | Google Cloud project number associated with the workforce pool. | +| `BYOID_SA_Impersonation_Uri` | `null` | Service Account impersonation URL for Workload Identity Federation. | +| `BYOID_SubjectTokenType` | `urn:ietf:params:oauth:tokentype:id_token` | Type of subject token provided (e.g., `urn:ietf:params:oauth:tokentype:jwt`). | +| `BYOID_TokenUri` | `https://sts.googleapis.com/v1/token` | Security Token Service (STS) token exchange endpoint URI. | + +### Core Query & Catalog Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `ProjectId` | *Default Project* | Google Cloud Project ID for billing and query execution. | +| `DefaultDataset` | `null` | Default dataset scope for unqualified table references in SQL queries. | +| `AdditionalProjects` | `null` | Comma-separated list of additional project IDs accessible for catalog discovery and querying. | +| `EnableProjectDiscovery` | `false` | Automatically discovers all accessible Google Cloud projects as catalog entries when `true`. | +| `FilterTablesOnDefaultDataset` | `false` | Filters `DatabaseMetaData.getTables()` and `.getColumns()` to `DefaultDataset` when wildcard patterns are used. | +| `Location` | *Auto-detected* | GCP location/region for dataset and job execution (e.g., `US`, `EU`, `asia-east1`). | +| `QueryDialect` | `SQL` | Query dialect: `SQL` (Standard SQL) or `LEGACY_SQL`. | +| `UseQueryCache` | `true` | Enables or disables BigQuery query result caching. | +| `MaximumBytesBilled` | `0` (unlimited) | Limits bytes billed per query; query fails without cost if estimated bytes exceed this limit. | +| `Labels` | `null` | Comma-separated `key=value` pairs attached to query jobs (e.g., `env=prod,dept=analytics`). | +| `QueryProperties` | `null` | Connection-level query configuration properties passed to BigQuery job execution. | +| `JobCreationMode` | `2` | Job creation strategy: `1` (Stateful session mode) or `2` (Fast stateless job creation). | +| `MaxResults` | `10000` | Maximum number of rows returned per page during standard REST result set iteration. | +| `AllowLargeResults` | `true` | Allows large query results (required when using legacy SQL destination tables). | +| `LargeResultTable` | `null` | User-specified destination table name for saving query result sets. | +| `LargeResultDataset` | `null` | User-specified destination dataset name for saving query result sets. | +| `LargeResultsDatasetExpirationTime` | `3600000` (1 hour) | Expiration time (in milliseconds) for temporary destination tables in user-specified datasets. | +| `KMSKeyName` | `null` | Cloud KMS key resource name used for encrypting query results and destination tables. | + +### Session & Transaction Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `EnableSession` | `false` | Enables multi-statement session creation and transaction support (`BEGIN`, `COMMIT`, `ROLLBACK`). | + +### High-Throughput Storage & Write API Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `EnableHighThroughputAPI` | `false` | Enables BigQuery Storage Read API (gRPC/Arrow) for fast result set retrieval. | +| `HighThroughputMinTableSize` | `10000` | Minimum query result row count threshold required to trigger the Storage Read API. | +| `HighThroughputActivationRatio` | `2` | Minimum number of result pages required before switching to the Storage Read API. | +| `UnsupportedHTAPIFallback` | `true` | Automatically falls back to standard REST API when Storage Read API is unsupported or lacks permissions. | +| `EnableWriteAPI` | `false` | Enables BigQuery Storage Write API for high-performance batch insert streams. | +| `SWA_ActivationRowCount` | `3` | Minimum row threshold in `executeBatch()` to activate Storage Write API streaming. | +| `SWA_AppendRowCount` | `1000` | Batch row size per append stream request when using Storage Write API. | + +### Network, Proxy & Endpoint Overrides + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `ProxyHost` | `null` | Hostname or IP address of the HTTP/HTTPS proxy server. | +| `ProxyPort` | `null` | Port number of the proxy server. | +| `ProxyUid` | `null` | Username for proxy server authentication. | +| `ProxyPwd` | `null` | Password for proxy server authentication. | +| `EndpointOverrides` | `null` | Semicolon/comma-separated custom service endpoints (e.g., `OAuth2=...`, `READ_API=...`, `BIGQUERY=...`, `STS=...`). | +| `PrivateServiceConnectUris` | `null` | Custom Private Service Connect (PSC) endpoint URIs. | +| `universeDomain` | `googleapis.com` | Domain name for Trusted Partner Cloud (TPC) or custom Google universe instances. | +| `SSLTrustStore` | `null` | Path to custom Java TrustStore file containing trusted server certificates for SSL. | +| `SSLTrustStorePwd` | `null` | Password for accessing the custom Java TrustStore file. | +| `HttpConnectTimeout` | `0` (system default) | HTTP socket connection timeout in milliseconds. | +| `HttpReadTimeout` | `0` (system default) | HTTP socket read timeout in milliseconds. | + +### Driver Logging, Retries & Concurrency Tuning + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `LogLevel` | `0` (OFF) | Logging verbosity level (`0` = OFF, higher integers increase logging detail). | +| `LogPath` | `""` | Directory path where log files are written when logging is enabled. | +| `Timeout` | `0` (unlimited) | Length of time (in seconds) the connector retries failed API calls before timing out. | +| `JobTimeout` | `0` (unlimited) | Job execution timeout (in seconds) after which BigQuery cancels the query job server-side. | +| `RetryInitialDelay` | `0` | Initial delay (in seconds) before executing the first retry attempt. | +| `RetryMaxDelay` | `0` | Maximum delay limit (in seconds) between retry attempts. | +| `MetaDataFetchThreadCount` | `32` | Thread pool size used to parallelize `DatabaseMetaData` catalog RPC calls. | +| `ConnectionPoolSize` | `10` | Maximum size of the internal connection pool when connection pooling is enabled. | +| `ListenerPoolSize` | `10` | Maximum size of the listener thread pool when connection pooling is enabled. | +| `PartnerToken` | `null` | Partner tracking token appended to request headers for telemetry. | +| `RequestReason` | `null` | Reason string passed in the `x-goog-request-reason` HTTP header for auditing. | + +--- + +## 5. Data Type Mapping Reference + +When running queries through the JDBC driver for BigQuery, data types map as specified in the official [BigQuery JDBC Data Type Mapping](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver): + +| BigQuery SQL Type | Java / JDBC Type | Recommended Getter / Setter | +| :--- | :--- | :--- | +| `ARRAY` | `java.sql.Array` | `rs.getArray(col)` | +| `BIGNUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | +| `BOOL` | `java.lang.Boolean` | `rs.getBoolean(col)` | +| `BYTES` | `byte[]` | `rs.getBytes(col)` | +| `DATE` | `java.sql.Date` / `java.time.LocalDate` | `rs.getDate(col)` or `rs.getObject(col, LocalDate.class)` | +| `DATETIME` | `java.sql.Timestamp` / `java.time.LocalDateTime` | `rs.getTimestamp(col)` or `rs.getObject(col, LocalDateTime.class)` | +| `FLOAT64` | `java.lang.Double` | `rs.getDouble(col)` | +| `GEOGRAPHY` | `java.lang.String` | `rs.getString(col)` | +| `INT64` | `java.lang.Long` | `rs.getLong(col)` | +| `INTERVAL` | `java.lang.String` | `rs.getString(col)` | +| `JSON` | `java.lang.String` | `rs.getString(col)` | +| `NUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | +| `STRING` | `java.lang.String` | `rs.getString(col)` | +| `STRUCT` | `java.sql.Struct` | `rs.getObject(col)` (casts to `java.sql.Struct`) | +| `TIME` | `java.sql.Time` / `java.time.LocalTime` | `rs.getTime(col)` or `rs.getObject(col, LocalTime.class)` | +| `TIMESTAMP` | `java.sql.Timestamp` / `java.time.Instant` | `rs.getTimestamp(col)` or `rs.getObject(col, Instant.class)` | + +--- + +## 6. JDBC Driver Architecture & Core Features + +### Transaction Management & Multi-Statement Sessions + +BigQuery supports **Multi-Statement Transactions** across tables using standard SQL primitives (`BEGIN TRANSACTION`, `COMMIT TRANSACTION`, `ROLLBACK TRANSACTION`). The driver bridges standard JDBC methods (`setAutoCommit`, `commit`, `rollback`) directly to BigQuery's underlying session engine. + +#### Session Lifecycle Flow: + +``` +[DriverManager.getConnection()] + │ + (EnableSession=true) + │ + ┌──────────▼──────────┐ + │ setAutoCommit(false)│ ──────► Executes: BEGIN TRANSACTION; (with setCreateSession=true) + └──────────┬──────────┘ Captures: session_id from Job statistics + │ Attaches session_id to Connection state + │ + ┌──────────▼──────────┐ + │ Execute DML & SQL │ ──────► Appends ConnectionProperty(key="session_id", value="...") + │ Statements │ Runs all queries within active BigQuery Session + └──────────┬──────────┘ + │ + ┌───────┴───────┐ + │ │ + ▼ ▼ +┌─────────┐ ┌──────────┐ +│commit() │ │rollback()│ +└────┬────┘ └────┬─────┘ + │ │ + ▼ ▼ +Executes: Executes: +COMMIT ROLLBACK +TRANSACTION; TRANSACTION; + │ │ + └───────┬───────┘ + │ + ▼ +(Auto-re-executes BEGIN TRANSACTION; if setAutoCommit is still false) +``` + +1. **Pre-requisite Check**: Calling `setAutoCommit(false)`, `commit()`, or `rollback()` requires `;EnableSession=true` in the connection URL. If disabled, an `IllegalStateException` is thrown. +2. **Session & Transaction Start**: + - `setAutoCommit(false)` issues `BEGIN TRANSACTION;` with `setCreateSession(true)`. + - BigQuery assigns a unique `session_id` in the job statistics. + - The driver attaches `session_id` to the connection state as a `ConnectionProperty`. +3. **Statement Propagation**: + - Every `Statement` or `PreparedStatement` created on the connection inherits `ConnectionProperty("session_id", sessionId)`. + - DML operations (`INSERT`, `UPDATE`, `DELETE`, `MERGE`) execute inside the session block. +4. **Commit & Rollback**: + - `commit()` executes `COMMIT TRANSACTION;` in the active session. + - `rollback()` executes `ROLLBACK TRANSACTION;` in the active session. + - If `autoCommit` remains `false`, the driver automatically starts the next transaction block by issuing `BEGIN TRANSACTION;`. +5. **Connection Close Safety**: + - If an uncommitted transaction is pending when `conn.close()` is invoked, the driver automatically performs an internal `rollbackImpl()` to abort pending modifications safely. +6. **Isolation Level & Holdability**: + - Isolation level: `Connection.TRANSACTION_SERIALIZABLE` (BigQuery multi-statement snapshot isolation). + - Holdability: `ResultSet.CLOSE_CURSORS_AT_COMMIT`. + +--- + +### Type Mapping & Coercion Engine + +The driver handles bidirectional type mapping and object coercion: + +- **Bidirectional Mapping**: Manages symmetric mapping between JDBC SQL types (`java.sql.Types`), Java types (`BigDecimal`, `byte[]`, `Date`, `Timestamp`), and BigQuery Standard SQL column types (`StandardSQLTypeName`). +- **Coercion Utilities**: Safely converts BigQuery scalar values and Arrow/JSON batch fields into target Java classes transparently during getter/setter invocations. + +--- + +### High-Throughput Storage Read & Write APIs + +For enterprise data ingestion and analytics extraction, the driver integrates with Google Cloud Storage APIs: + +- **Storage Read API (`EnableHighThroughputAPI=true`)**: + Performs high-speed result extraction over gRPC streams using Apache Arrow instead of REST JSON pagination, avoiding REST payload deserialization bottlenecks for large ResultSets. +- **Storage Write API (`EnableWriteAPI=true`)**: + Enables high-throughput streaming appends for batch operations executed via `PreparedStatement.executeBatch()`. + +--- + +### Diagnostic MDC Tracing & Connection Proxying + +The driver manages execution context and diagnostics transparently: + +- **ThreadLocal MDC Tracing**: Automatically populates `BIGQUERY_JDBC_MDC` with connection ID, catalog, dataset, and job execution metrics for application logging. +- **Automatic Resource Cleanup**: Tracks open statements and result sets to prevent unhandled memory leaks upon connection closure. + +--- + +## 7. Common User Journeys (CUJs) & Code Recipes + +### CUJ 1: Financial & Multi-Table Transactions + +**Scenario**: A financial application debits one account and credits another atomically across multiple tables using JDBC transactions. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.SQLException; + +public class FinancialTransactionCUJ { + public static void main(String[] args) { + // Note: EnableSession=true is REQUIRED for transaction support + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";DefaultDataset=finance" + + ";EnableSession=true" + + ";OAuthType=0"; + + try (Connection conn = DriverManager.getConnection(url)) { + // 1. Start Transaction Session + conn.setAutoCommit(false); + + String debitSql = "UPDATE finance.accounts SET balance = balance - ? WHERE account_id = ?"; + String creditSql = "UPDATE finance.accounts SET balance = balance + ? WHERE account_id = ?"; + String auditSql = "INSERT INTO finance.audit_log (source_id, target_id, amount, transfer_time) VALUES (?, ?, ?, CURRENT_TIMESTAMP())"; + + try (PreparedStatement debitStmt = conn.prepareStatement(debitSql); + PreparedStatement creditStmt = conn.prepareStatement(creditSql); + PreparedStatement auditStmt = conn.prepareStatement(auditSql)) { + + double amount = 500.00; + long sourceAccount = 1001L; + long targetAccount = 2002L; + + // Debit source account + debitStmt.setDouble(1, amount); + debitStmt.setLong(2, sourceAccount); + debitStmt.executeUpdate(); + + // Credit target account + creditStmt.setDouble(1, amount); + creditStmt.setLong(2, targetAccount); + creditStmt.executeUpdate(); + + // Log audit record + auditStmt.setLong(1, sourceAccount); + auditStmt.setLong(2, targetAccount); + auditStmt.setDouble(3, amount); + auditStmt.executeUpdate(); + + // 2. Commit transaction atomically + conn.commit(); + System.out.println("Transaction committed successfully."); + + } catch (SQLException e) { + // 3. Rollback transaction on failure + System.err.println("Transaction failed. Rolling back changes: " + e.getMessage()); + conn.rollback(); + } + } catch (SQLException e) { + e.printStackTrace(); + } + } +} +``` + +--- + +### CUJ 2: High-Performance ETL Ingestion Pipeline + +**Scenario**: An ETL ingestion pipeline loads batches of records into BigQuery using `executeBatch()` with parameter binding and high-throughput write APIs. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.SQLException; +import java.time.Instant; +import java.util.UUID; + +public class EtlBatchIngestionCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";DefaultDataset=telemetry" + + ";EnableWriteAPI=true" // High-throughput write streaming + + ";OAuthType=0"; + + String insertSql = "INSERT INTO telemetry.sensor_readings (reading_id, sensor_id, temperature, is_valid, record_time) VALUES (?, ?, ?, ?, ?)"; + + try (Connection conn = DriverManager.getConnection(url); + PreparedStatement pstmt = conn.prepareStatement(insertSql)) { + + int batchSize = 1000; + for (int i = 1; i <= batchSize; i++) { + pstmt.setString(1, UUID.randomUUID().toString()); + pstmt.setInt(2, 100 + (i % 10)); // Sensor IDs 100-109 + pstmt.setDouble(3, 20.0 + (Math.random() * 15.0)); // Temp 20-35 C + pstmt.setBoolean(4, true); + pstmt.setObject(5, java.sql.Timestamp.from(Instant.now())); + pstmt.addBatch(); + + if (i % 500 == 0) { + int[] results = pstmt.executeBatch(); + System.out.printf("Flushed batch of %d records.%n", results.length); + } + } + } + } +} +``` + +--- + +### CUJ 3: High-Throughput Result Extraction via Storage Read API + +**Scenario**: An analytical application extracts large query result sets using standard JDBC ResultSet APIs accelerated transparently by BigQuery Storage Read API. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; + +public class HighThroughputExtractionCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";DefaultDataset=analytics" + + ";EnableHighThroughputAPI=true" // Stream via gRPC Storage Read API + + ";OAuthType=0"; + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT user_id, event_name, event_timestamp FROM analytics.user_events WHERE event_date >= '2026-01-01'")) { + + long rowCount = 0; + while (rs.next()) { + rowCount++; + String userId = rs.getString("user_id"); + String eventName = rs.getString("event_name"); + java.sql.Timestamp eventTime = rs.getTimestamp("event_timestamp"); + } + System.out.println("Extracted " + rowCount + " rows via Storage Read API."); + } + } +} +``` + +--- + +### CUJ 4: Parameterized Queries with JSR-310 & Numeric Types + +**Scenario**: Binding precise decimals (`BigDecimal`), temporal types (`java.sql.Date`, `Timestamp`), and binary payloads (`byte[]`). + +```java +import java.math.BigDecimal; +import java.sql.Connection; +import java.sql.Date; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.Timestamp; +import java.time.Instant; +import java.time.LocalDate; + +public class ParameterizedTypesCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + + String sql = "SELECT order_id FROM sales.orders " + + "WHERE order_date = ? " + + " AND created_timestamp <= ? " + + " AND total_amount >= ? " + + " AND security_token = ?"; + + try (Connection conn = DriverManager.getConnection(url); + PreparedStatement pstmt = conn.prepareStatement(sql)) { + + // Temporal parameter binding + pstmt.setDate(1, Date.valueOf(LocalDate.of(2026, 7, 22))); + pstmt.setTimestamp(2, Timestamp.from(Instant.now())); + + // High-precision numeric binding + pstmt.setBigDecimal(3, new BigDecimal("12499.99")); + + // Binary payload binding + byte[] tokenBytes = new byte[] {0x12, 0x34, 0x56, 0x78}; + pstmt.setBytes(4, tokenBytes); + + try (ResultSet rs = pstmt.executeQuery()) { + while (rs.next()) { + System.out.println("Matched Order ID: " + rs.getLong("order_id")); + } + } + } + } +} +``` + +--- + +### CUJ 5: Nested Structs & Repeated Array Operations + +**Scenario**: Processing e-commerce orders containing nested address `STRUCT` objects and line-item `ARRAY` objects. + +```java +import java.sql.Array; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.sql.Struct; + +public class ComplexTypesCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + + String query = "SELECT " + + " order_id, " + + " STRUCT('123 Main St' AS street, 'Seattle' AS city) AS shipping_address, " + + " [STRUCT('Item A' AS sku, 2 AS qty), STRUCT('Item B' AS sku, 1 AS qty)] AS line_items, " + + " ['tag1', 'tag2', 'tag3'] AS tags"; + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery(query)) { + + while (rs.next()) { + long orderId = rs.getLong("order_id"); + + // 1. Reading Struct + Struct addressStruct = (Struct) rs.getObject("shipping_address"); + Object[] addressAttrs = addressStruct.getAttributes(); + System.out.printf("Order %d Address: %s, %s%n", orderId, addressAttrs[0], addressAttrs[1]); + + // 2. Reading Array of Structs + Array itemsArray = rs.getArray("line_items"); + Struct[] items = (Struct[]) itemsArray.getArray(); + for (Struct item : items) { + Object[] itemAttrs = item.getAttributes(); + System.out.printf(" Item -> SKU: %s, Qty: %s%n", itemAttrs[0], itemAttrs[1]); + } + + // 3. Reading Primitive Array + Array tagsArray = rs.getArray("tags"); + String[] tags = (String[]) tagsArray.getArray(); + System.out.println(" Tags: " + String.join(", ", tags)); + } + } + } +} +``` + +--- + +### CUJ 6: Service Account Impersonation & Cross-Project Billing + +**Scenario**: Running a query where billing occurs under Project A (`billing-project`), while querying data located in Project B (`data-project`) via Service Account Impersonation. + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; + +public class CrossProjectImpersonationCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=billing-project-id" // Project billed for the query + + ";AdditionalProjects=data-project-id" // Cross-project dataset access + + ";DefaultDataset=shared_dataset" + + ";OAuthType=0" // Use ADC context for initial authentication + + ";ServiceAccountImpersonationEmail=analytics-executor@billing-project-id.iam.gserviceaccount.com"; + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT count(*) FROM `data-project-id.shared_dataset.global_logs`")) { + + if (rs.next()) { + System.out.println("Cross-project log count: " + rs.getLong(1)); + } + } + } +} +``` + +--- + +### CUJ 7: Stored Procedure Execution + +**Scenario**: Calling a BigQuery stored procedure with input and output parameters. + +```java +import java.math.BigDecimal; +import java.sql.CallableStatement; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.Types; + +public class StoredProcedureCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + + String procSql = "CALL my_dataset.calculate_tax(?, ?)"; + + try (Connection conn = DriverManager.getConnection(url); + CallableStatement cstmt = conn.prepareCall(procSql)) { + + cstmt.setBigDecimal(1, new BigDecimal("1000.00")); // Input subtotal + cstmt.registerOutParameter(2, Types.DECIMAL); // Output calculated tax + + cstmt.execute(); + + BigDecimal taxAmount = cstmt.getBigDecimal(2); + System.out.println("Calculated Tax Amount: $" + taxAmount); + } + } +} +``` + +--- + +## 8. High-Throughput & Performance Tuning + +For large dataset extractions and high-concurrency ingestion, tune the following connection options: + +### High-Throughput Storage Read API (HTAPI) Deep Dive + +The driver can switch query result extraction from standard REST JSON pagination to high-speed gRPC streams using the **BigQuery Storage Read API** (Apache Arrow over gRPC). + +#### HTAPI Property Relationship & Activation Matrix + +The decision to activate the Read API for a query is governed by four interconnected connection properties: + +| Property Name | Connection Parameter | Default Value | Functional Role | +| :--- | :--- | :---: | :--- | +| **`EnableHighThroughputAPI`** | `EnableHighThroughputAPI=true` | `false` | **Master Toggle**: Must be `true` to evaluate Read API switching. | +| **`HighThroughputMinTableSize`** | `HighThroughputMinTableSize=10000` | `10000` | **Minimum Row Threshold**: Minimum total rows (`totalRows`) required. | +| **`HighThroughputActivationRatio`** | `HighThroughputActivationRatio=2` | `2` | **Page Ratio Threshold**: `totalRows / pageSize` must exceed this ratio. | +| **`MaxResults`** | `MaxResults=1000` | `1000` (or server default) | **Page Size (`pageSize`)**: Controls rows per page in standard REST calls. | + +#### Activation Formula & Internal Safeguards + +The driver evaluates Storage Read API activation using the following rules: + +$$\text{Storage Read API Active} = \text{EnableHighThroughputAPI} \land \text{meetsReadRatio}$$ + +Where `meetsReadRatio` requires **ALL** of the following conditions to pass: + +1. `EnableHighThroughputAPI == true` +2. `totalRows >= HighThroughputMinTableSize` (default: $\ge 10,000$ rows) +3. `results.hasNextPage() == true` (**Crucial Safeguard**): If all rows fit in the first response page, the driver will **never** switch to Read API, avoiding redundant gRPC calls and memory re-allocations. +4. **Page Ratio Test**: $\frac{\text{totalRows}}{\text{pageSize}} > \text{HighThroughputActivationRatio}$ + +#### How to Manipulate Properties to Control Read API Activation + +##### 1. Force Read API for Smaller Query Results ($\ge 100$ rows) +To trigger the Read API even for small analytical datasets: +``` +jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0;EnableHighThroughputAPI=true;HighThroughputMinTableSize=100;HighThroughputActivationRatio=0;MaxResults=50 +``` +*Why this works*: `totalRows` ($100$) $\ge$ `MinTableSize` ($100$), `pageSize` ($50$) forces `hasNextPage() = true`, and $\frac{100}{50} = 2 > 0$ (`ActivationRatio`). + +##### 2. High-Volume Production Defaults ($\ge 100,000$ rows) +With standard defaults (`EnableHighThroughputAPI=true`): +- For a $100,000$ row result set with `pageSize = 1,000`: $\frac{100,000}{1,000} = 100 > 2$ (`ActivationRatio`). +- The driver transparently switches to the gRPC Storage Read API stream. + +##### 3. Automatic Fallback on Permission Failure +If `EnableHighThroughputAPI=true` is set but the connecting service account lacks the `BigQuery Read Session User` IAM role, the driver catches the `PERMISSION_DENIED` status and **automatically falls back to standard REST JSON pagination** without failing the application's query. + +### Storage Write API (SWA) Configuration Property Matrix + +The **BigQuery Storage Write API** provides high-performance gRPC streaming ingestion for batch `PreparedStatement.executeBatch()` operations. The behavior and streaming thresholds are governed by three connection properties: + +#### 1. SWA Property Reference + +| Property Name | Connection Parameter | Default Value | Functional Role | +| :--- | :--- | :---: | :--- | +| **`EnableWriteAPI`** | `EnableWriteAPI=true` | `false` | **Master Toggle**: Must be set to `true` to enable gRPC Storage Write API stream ingestion. | +| **`SWA_ActivationRowCount`** | `SWA_ActivationRowCount=3` | `3` | **Activation Threshold**: Minimum number of row batches added via `addBatch()` required to trigger SWA streaming. | +| **`SWA_AppendRowCount`** | `SWA_AppendRowCount=1000` | `1000` | **Stream Chunk Size**: Maximum number of rows packed into a single gRPC append stream payload before flushing. | + +#### 2. SWA Activation Formula & Fallback Safeguard + +The driver evaluates Storage Write API activation during `PreparedStatement.executeBatch()` as follows: + +$$\text{Storage Write API Active} = \text{EnableWriteAPI} \land (\text{batchParameters.size()} \ge \text{SWA\_ActivationRowCount})$$ + +- **Above Threshold ($\ge \text{SWA\_ActivationRowCount}$)**: The driver opens a gRPC Storage Write API stream, converts batch parameters to JSON rows, appends rows in chunks of `SWA_AppendRowCount`, and commits the write stream. +- **Below Threshold ($< \text{SWA\_ActivationRowCount}$)**: The driver falls back to standard REST DML query concatenation (`INSERT INTO ...; INSERT INTO ...;`), preventing unnecessary gRPC channel overhead for small batches. + +#### 3. SWA Configuration Scenarios Matrix + +| Workload Scenario | `EnableWriteAPI` | `SWA_ActivationRowCount` | `SWA_AppendRowCount` | Executed Ingestion Mechanism | Enterprise Use Case | +| :--- | :---: | :---: | :---: | :--- | :--- | +| **Standard DML (Default)** | `false` | `3` (ignored) | `1000` (ignored) | Concatenated REST SQL DML (`INSERT INTO ...;`) | Small transactional DML; standard SQL compatibility. | +| **Default High-Throughput ETL** | `true` | `3` | `1000` | gRPC SWA stream for batches $\ge 3$, flushes every 1,000 rows | Standard batch loader applications (Spring Batch, Spark). | +| **Real-Time Micro-Batching** | `true` | `1` | `100` | gRPC SWA stream for any batch $\ge 1$, flushes every 100 rows | High-frequency streaming events (Kafka/Flink consumers). | +| **High-Volume Bulk Ingestion** | `true` | `100` | `5000` | gRPC SWA stream for batches $\ge 100$, flushes in 5,000 row chunks | Large nightly bulk ETL loading millions of records. | +| **Small Batch Fallback** | `true` | `100` | `1000` | Concatenated REST SQL DML (for batches $< 100$ rows) | Adaptive pipelines using REST for small batches, gRPC for large ones. | + +--- + +### Additional Performance Tuning Options + +- **Metadata Thread Pooling**: For applications fetching schema catalog metadata across multiple datasets, set `;MetaDataFetchThreadCount=64` to parallelize catalog RPC calls. +- **Internal Queue Buffering**: The driver maintains an internal asynchronous row buffer (default 10,000 rows) to decouple network fetch RPCs from client `ResultSet.next()` iteration. + +--- + +### Custom Endpoints, Private Service Connect (PSC) & Universe Domains + +For enterprise environments using **VPC Service Controls**, **Private Service Connect (PSC)**, **Regional Endpoints**, or **Trusted Partner Cloud (TPC)** instances, the driver supports full endpoint overriding and universe domain configuration. + +#### 1. Endpoint Overrides (`EndpointOverrides`) + +The `EndpointOverrides` property allows redirecting traffic for individual Google Cloud services to private IP endpoints, regional gateways, or corporate proxy targets. + +**Format**: +``` +EndpointOverrides=SERVICE_KEY_1=URI_1,SERVICE_KEY_2=URI_2; +``` + +**Supported Service Keys**: + +| Service Key | Target Service | Default Endpoint URI | Typical Override Use Case | +| :--- | :--- | :--- | :--- | +| **`BIGQUERY`** | BigQuery Core REST API | `https://www.googleapis.com/bigquery/v2/` | Regional endpoints (e.g. `us-east4`) or PSC private VIPs. | +| **`READ_API`** | Storage Read API (HTAPI) | `bigquerystorage.googleapis.com:443` | Private gRPC endpoints for high-throughput extractions. | +| **`OAUTH2`** | Google OAuth2 Token Server | `https://oauth2.googleapis.com/token` | Private OAuth token exchange gateways. | +| **`STS`** | Security Token Service | `https://sts.googleapis.com` | Workload Identity / BYOID token exchange. | + +**Example - Private Service Connect (PSC) Endpoint**: +``` +jdbc:bigquery://https://bigquery-privateendpoint.p.googleapis.com:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;EndpointOverrides=BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com,OAUTH2=https://oauth2-private.p.googleapis.com/token +``` + +**Example - Regional BigQuery Endpoint Override**: +``` +jdbc:bigquery://https://us-east4-bigquery.googleapis.com:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;EndpointOverrides=BIGQUERY=https://us-east4-bigquery.googleapis.com +``` + +#### 2. Custom Endpoint Requirements per Authentication Type (`OAuthType`) + +The interaction between custom endpoint properties (`EndpointOverrides`, `PrivateServiceConnectUris`, `universeDomain`) and authentication flows (`OAuthType`) behaves as follows: + +| Authentication Mode (`OAuthType`) | Applicable Custom Endpoint Keys | Endpoint Override Behavior & Actions | Primary Enterprise Use Case | +| :--- | :--- | :--- | :--- | +| **`OAuthType=0`
(Service Account)** | • `OAUTH2`
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Overrides OAuth token exchange server URI via `OAUTH2`
• Sets custom universe domain for TPC instances | Private service account key token exchange via internal proxy; regional or PSC endpoints. | +| **`OAuthType=1`
(User Account)** | • `OAUTH2`
• `BIGQUERY`
• `READ_API` | • Overrides 3-legged browser authorization token server via `OAUTH2` | Desktop browser 3-legged OAuth code exchange via private corporate authorization server. | +| **`OAuthType=2`
(Pre-generated Token)** | • `OAUTH2` (Refresh Token)
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Overrides token refresh endpoint via `OAUTH2` for refresh tokens
• Bypasses token server for static access tokens | Custom token refresh server for long-running batch jobs; direct static access token routing. | +| **`OAuthType=3`
(ADC)** | • `BIGQUERY`
• `READ_API`
• `PrivateServiceConnectUris` | • Routes BigQuery REST query calls and Storage Read gRPC streams via PSC VIPs or regional endpoints | On-premise or GKE workload running ADC routed strictly over Private Service Connect (PSC). | +| **`OAuthType=4`
(External Account / BYOID)** | • `STS`
• `BYOIDTokenUri`
• `BYOIDSAImpersonationUri`
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Configures Security Token Service URL (`token_url`) via `STS`/`BYOIDTokenUri`
• Configures custom universe domain for external account config | Multi-cloud Workload Identity Federation (AWS/Azure/OIDC) exchanging tokens via custom STS VIP. | +| **Impersonation
(`ServiceAccountImpersonationEmail`)** | • Applies across ALL `OAuthType` modes | • Routes IAM Credentials API impersonation calls over configured BigQuery and OAuth endpoints | Cross-project or elevated role execution via Service Account Impersonation. | + +--- + +#### 3. Endpoint Configuration Code Recipes by `OAuthType` + +##### Recipe A: Service Account (`OAuthType=0`) with Private OAuth & BigQuery PSC +```java +// Service Account authentication routed entirely over Private Service Connect (PSC) VIPs +String url = "jdbc:bigquery://https://bigquery-privateendpoint.p.googleapis.com:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=0" + + ";OAuthServiceAcctEmail=sa-data-reader@my-gcp-project.iam.gserviceaccount.com" + + ";OAuthPvtKeyPath=/var/secrets/bq-sa-key.json" + + ";EndpointOverrides=" + + "BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com," + + "OAUTH2=https://oauth2-private.p.googleapis.com/token," + + "READ_API=https://storage-privateendpoint.p.googleapis.com"; + +Connection conn = DriverManager.getConnection(url); +``` + +##### Recipe B: Pre-Generated Refresh Token (`OAuthType=2`) with Custom Token Server +```java +// Refresh token flow pointing to an internal corporate OAuth token refresh server +String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=2" + + ";OAuthRefreshToken=1//04_example_refresh_token" + + ";OAuthClientId=my-client-id.apps.googleusercontent.com" + + ";OAuthClientSecret=my-client-secret" + + ";EndpointOverrides=OAUTH2=https://oauth-proxy.internal.corp.com/token"; + +Connection conn = DriverManager.getConnection(url); +``` + +##### Recipe C: Workload Identity Federation (`OAuthType=4` / BYOID) with Custom STS Endpoint +```java +// AWS / OIDC Workload Identity exchanging tokens via custom Security Token Service (STS) endpoint +String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=4" + + ";BYOIDAudienceUri=//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider" + + ";BYOIDSubjectTokenType=urn:ietf:params:aws:token-type:aws-sigv4" + + ";BYOIDCredentialSource={\"environment_id\":\"aws-1\",\"region_url\":\"http://169.254.169.254/latest/meta-data/placement/availability-zone\"}" + + ";BYOIDTokenUri=https://sts-privateendpoint.p.googleapis.com/v1/token" + + ";EndpointOverrides=STS=https://sts-privateendpoint.p.googleapis.com/v1/token"; + +Connection conn = DriverManager.getConnection(url); +``` + +##### Recipe D: Regional Endpoint Override with Application Default Credentials (`OAuthType=3`) +```java +// ADC authentication targeting a regional BigQuery REST endpoint (us-east4) +String url = "jdbc:bigquery://https://us-east4-bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=3" + + ";EndpointOverrides=BIGQUERY=https://us-east4-bigquery.googleapis.com"; + +Connection conn = DriverManager.getConnection(url); +``` + +--- + +#### 4. Private Service Connect URIs (`PrivateServiceConnectUris`) + +For VPC setups using explicit Private Service Connect URI aliases: +``` +jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;PrivateServiceConnectUris=BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com +``` + +#### 5. Custom Universe Domains (`universeDomain`) + +To target Google Cloud Trusted Partner Cloud (TPC) or non-default Universe Domain instances, specify `universeDomain`: + +``` +jdbc:bigquery://https://www.my-universe.cloud:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;universeDomain=my-universe.cloud +``` + +--- + +## 9. Framework Integration & Deployment + +### Spring Boot (`application.yml`) +```yaml +spring: + datasource: + url: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-gcp-project;DefaultDataset=my_dataset;OAuthType=0;OAuthServiceAcctEmail=sa@my-project.iam.gserviceaccount.com;OAuthPvtKeyPath=/path/to/key.json + driver-class-name: com.google.cloud.bigquery.jdbc.BigQueryDriver + hikari: + maximum-pool-size: 10 + minimum-idle: 2 + idle-timeout: 300000 + connection-test-query: SELECT 1 +``` + +### Key Deployment Guidelines +1. **Driver Class**: `com.google.cloud.bigquery.jdbc.BigQueryDriver` +2. **Connection Pooling**: BigQuery connection objects are lightweight wrappers around thread-safe Google API clients. Standard connection pools like HikariCP can be safely used. +3. **IAM Permissions**: Ensure the principal has `BigQuery Data Viewer` and `BigQuery Job User` roles on the target project and dataset. + +--- + +## 10. Logging, Diagnostics & Troubleshooting + +### Configuring Connection-Level Logging + +The driver includes a built-in logging subsystem built on top of `java.util.logging`. Logging can be configured directly via JDBC URL connection properties or system environment variables: + +#### Connection URL Parameters: +```java +String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=0" + + ";LogLevel=5" // Set log level (1=SEVERE to 7=FINEST) + + ";LogPath=/var/log/bigquery-jdbc"; // Output directory for log files +``` + +#### Log Level Reference: + +| Integer Value | String Constant | `java.util.logging` Level | Description | +| :---: | :---: | :---: | :--- | +| `0` | `OFF` | `Level.OFF` | Logging completely disabled (default). | +| `1` | `SEVERE` | `Level.SEVERE` | Critical errors, connection failures, unrecoverable exceptions. | +| `2` | `WARNING` | `Level.WARNING` | Non-fatal warnings, API fallbacks (e.g., Read API permission denied fallback to REST). | +| `3` | `INFO` | `Level.INFO` | High-level driver events, connection establishment, job submission. | +| `4` | `CONFIG` | `Level.CONFIG` | Driver configuration initialization and property resolution details. | +| `5` | `FINE` | `Level.FINE` | SQL query execution statements, parameter binding, row counts. | +| `6` | `FINER` | `Level.FINER` | Internal method entry (`++enter++`) and exit (`++exit++`) tracing. | +| `7` | `FINEST` | `Level.FINEST` | Maximum diagnostic verbosity, raw batch payloads, Arrow stream packet details. | + +#### Log File Naming & Format + +Log files are generated inside the specified `LogPath` directory with per-connection log isolation. + +**Standard Log Record Format**: +```text +2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] c.g.c.b.j.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM analytics.orders +``` + +- **Timestamp**: `yyyy-MM-dd HH:mm:ss.SSS` in local time. +- **Connection ID (`MDC`)**: Unique identifier `[conn-bq-XXXX]` isolating statements per connection context. +- **Level**: Padded log level (`INFO`, `FINE`, `WARNING`, etc.). +- **Process ID & Thread**: OS Process ID and centered Thread Name. +- **Class & Method**: Abbreviated source class and method name originating the log message. + +--- + +### Exception Hierarchy + +All driver exceptions inherit from `java.sql.SQLException`: + +- **`BigQueryJdbcException`**: Base exception class wrapping underlying Google Cloud API and network RPC errors. +- **`BigQueryJdbcSqlSyntaxErrorException`**: Thrown when standard SQL query validation or parsing fails on the server. +- **`BigQueryConversionException`**: Thrown on data type coercion failures or unparseable target representations. +- **`BigQueryJdbcSqlFeatureNotSupportedException`**: Thrown when an unsupported JDBC API method is invoked. + +--- + +## 11. Official Documentation References + +- 🌐 [Official Google Cloud BigQuery JDBC Driver Overview](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) +- ⚙️ [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver) +- 📊 [BigQuery Quotas & Limits](https://cloud.google.com/bigquery/quotas) From 4ff5c6ef9f84ed2b58ab0d689bd7804b39b9544c Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Thu, 23 Jul 2026 15:26:56 -0400 Subject: [PATCH 2/8] docs(bigquery-jdbc): fix OAuthType numerical mappings in USER_GUIDE.md --- java-bigquery-jdbc/USER_GUIDE.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index f93f4fb49fc6..b2922e23f330 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -84,7 +84,7 @@ public class BigQueryQuickstart { String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + ";ProjectId=my-gcp-project-id" + ";DefaultDataset=my_dataset" - + ";OAuthType=0"; // 0 = Application Default Credentials (ADC) + + ";OAuthType=3"; // 3 = Application Default Credentials (ADC) try (Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement(); @@ -106,19 +106,17 @@ The driver supports multiple OAuth 2.0 and identity workflows specified via the | `OAuthType` | Authentication Strategy | Required Properties | | :---: | :--- | :--- | -| **`0`** | **Application Default Credentials (ADC)** / gcloud CLI | None (reads `GOOGLE_APPLICATION_CREDENTIALS` or `gcloud` context) | -| **`1`** | **Service Account Key File** | `OAuthPvtKeyPath` (path to `.json` or `.p12` service account key) | -| **`1`** | **Service Account Private Key String** | `OAuthServiceAcctEmail`, `OAuthPvtKey` (raw private key contents) | -| **`2`** | **OAuth 2.0 Access Token / Refresh Token** | `OAuthRefreshToken`, `OAuthClientId`, `OAuthClientSecret` | -| **`3`** | **User Account / Service Account Email** | `OAuthServiceAcctEmail` | -| **`4`** | **Pre-generated Access Token** | `OAuthAccessToken` | -| **`5`** | **Workload Identity Federation (BYOID)** | `BYOID_AudienceUri`, `BYOID_CredentialSource`, `BYOID_SubjectTokenType` | +| **`0`** | **Service Account Key File / Key String** | `OAuthPvtKeyPath` (path to `.json`/`.p12` key) or `OAuthPvtKey` & `OAuthServiceAcctEmail` | +| **`1`** | **User Account (Interactive 3-Legged OAuth)** | `OAuthClientId`, `OAuthClientSecret` | +| **`2`** | **Pre-generated Access Token or Refresh Token** | `OAuthAccessToken` or `OAuthRefreshToken` (`OAuthClientId`, `OAuthClientSecret`) | +| **`3`** | **Application Default Credentials (ADC)** / gcloud CLI | None (reads `GOOGLE_APPLICATION_CREDENTIALS` or `gcloud` context) | +| **`4`** | **Workload Identity Federation (BYOID)** | `BYOID_AudienceUri`, `BYOID_CredentialSource`, `BYOID_SubjectTokenType` | ### Service Account Authentication Example ```java String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + ";ProjectId=my-gcp-project-id" - + ";OAuthType=1" + + ";OAuthType=0" + ";OAuthPvtKeyPath=/path/to/service-account-key.json"; ``` @@ -127,7 +125,7 @@ To execute queries as an impersonated service account: ```java String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + ";ProjectId=my-gcp-project-id" - + ";OAuthType=0" + + ";OAuthType=3" + ";ServiceAccountImpersonationEmail=target-sa@my-gcp-project.iam.gserviceaccount.com"; ``` From a2a15f595abdddd7f530b6242766057142dfd4b2 Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Thu, 23 Jul 2026 15:35:04 -0400 Subject: [PATCH 3/8] docs(bigquery-jdbc): harden code snippets against NPEs and fix BYOID property names --- java-bigquery-jdbc/USER_GUIDE.md | 59 +++++++++++++++++++------------- 1 file changed, 36 insertions(+), 23 deletions(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index b2922e23f330..d39c7a4c72af 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -373,7 +373,7 @@ public class FinancialTransactionCUJ { + ";ProjectId=my-gcp-project" + ";DefaultDataset=finance" + ";EnableSession=true" - + ";OAuthType=0"; + + ";OAuthType=3"; // 3 = Application Default Credentials (ADC) try (Connection conn = DriverManager.getConnection(url)) { // 1. Start Transaction Session @@ -443,7 +443,7 @@ public class EtlBatchIngestionCUJ { + ";ProjectId=my-gcp-project" + ";DefaultDataset=telemetry" + ";EnableWriteAPI=true" // High-throughput write streaming - + ";OAuthType=0"; + + ";OAuthType=3"; String insertSql = "INSERT INTO telemetry.sensor_readings (reading_id, sensor_id, temperature, is_valid, record_time) VALUES (?, ?, ?, ?, ?)"; @@ -487,7 +487,7 @@ public class HighThroughputExtractionCUJ { + ";ProjectId=my-gcp-project" + ";DefaultDataset=analytics" + ";EnableHighThroughputAPI=true" // Stream via gRPC Storage Read API - + ";OAuthType=0"; + + ";OAuthType=3"; try (Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement(); @@ -525,7 +525,7 @@ import java.time.LocalDate; public class ParameterizedTypesCUJ { public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; String sql = "SELECT order_id FROM sales.orders " + "WHERE order_date = ? " @@ -561,7 +561,7 @@ public class ParameterizedTypesCUJ { ### CUJ 5: Nested Structs & Repeated Array Operations -**Scenario**: Processing e-commerce orders containing nested address `STRUCT` objects and line-item `ARRAY` objects. +**Scenario**: Processing e-commerce orders containing nested address `STRUCT` objects and line-item `ARRAY` objects with null-safe guardrails. ```java import java.sql.Array; @@ -573,7 +573,7 @@ import java.sql.Struct; public class ComplexTypesCUJ { public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; String query = "SELECT " + " order_id, " @@ -588,23 +588,36 @@ public class ComplexTypesCUJ { while (rs.next()) { long orderId = rs.getLong("order_id"); - // 1. Reading Struct + // 1. Reading Struct with Null Safety Struct addressStruct = (Struct) rs.getObject("shipping_address"); - Object[] addressAttrs = addressStruct.getAttributes(); - System.out.printf("Order %d Address: %s, %s%n", orderId, addressAttrs[0], addressAttrs[1]); + if (addressStruct != null) { + Object[] addressAttrs = addressStruct.getAttributes(); + if (addressAttrs != null && addressAttrs.length >= 2) { + System.out.printf("Order %d Address: %s, %s%n", orderId, addressAttrs[0], addressAttrs[1]); + } + } - // 2. Reading Array of Structs + // 2. Reading Array of Structs with Null Safety Array itemsArray = rs.getArray("line_items"); - Struct[] items = (Struct[]) itemsArray.getArray(); - for (Struct item : items) { - Object[] itemAttrs = item.getAttributes(); - System.out.printf(" Item -> SKU: %s, Qty: %s%n", itemAttrs[0], itemAttrs[1]); + if (itemsArray != null && itemsArray.getArray() != null) { + Object[] itemsObj = (Object[]) itemsArray.getArray(); + for (Object itemObj : itemsObj) { + if (itemObj instanceof Struct) { + Struct item = (Struct) itemObj; + Object[] itemAttrs = item.getAttributes(); + if (itemAttrs != null && itemAttrs.length >= 2) { + System.out.printf(" Item -> SKU: %s, Qty: %s%n", itemAttrs[0], itemAttrs[1]); + } + } + } } - // 3. Reading Primitive Array + // 3. Reading Primitive Array with Null Safety Array tagsArray = rs.getArray("tags"); - String[] tags = (String[]) tagsArray.getArray(); - System.out.println(" Tags: " + String.join(", ", tags)); + if (tagsArray != null && tagsArray.getArray() != null) { + String[] tags = (String[]) tagsArray.getArray(); + System.out.println(" Tags: " + String.join(", ", tags)); + } } } } @@ -629,7 +642,7 @@ public class CrossProjectImpersonationCUJ { + ";ProjectId=billing-project-id" // Project billed for the query + ";AdditionalProjects=data-project-id" // Cross-project dataset access + ";DefaultDataset=shared_dataset" - + ";OAuthType=0" // Use ADC context for initial authentication + + ";OAuthType=3" // Use ADC context for initial authentication + ";ServiceAccountImpersonationEmail=analytics-executor@billing-project-id.iam.gserviceaccount.com"; try (Connection conn = DriverManager.getConnection(url); @@ -659,7 +672,7 @@ import java.sql.Types; public class StoredProcedureCUJ { public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0"; + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; String procSql = "CALL my_dataset.calculate_tax(?, ?)"; @@ -860,10 +873,10 @@ Connection conn = DriverManager.getConnection(url); String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" + ";ProjectId=my-gcp-project" + ";OAuthType=4" - + ";BYOIDAudienceUri=//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider" - + ";BYOIDSubjectTokenType=urn:ietf:params:aws:token-type:aws-sigv4" - + ";BYOIDCredentialSource={\"environment_id\":\"aws-1\",\"region_url\":\"http://169.254.169.254/latest/meta-data/placement/availability-zone\"}" - + ";BYOIDTokenUri=https://sts-privateendpoint.p.googleapis.com/v1/token" + + ";BYOID_AudienceUri=//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider" + + ";BYOID_SubjectTokenType=urn:ietf:params:aws:token-type:aws-sigv4" + + ";BYOID_CredentialSource={\"environment_id\":\"aws-1\",\"region_url\":\"http://169.254.169.254/latest/meta-data/placement/availability-zone\"}" + + ";BYOID_TokenUri=https://sts-privateendpoint.p.googleapis.com/v1/token" + ";EndpointOverrides=STS=https://sts-privateendpoint.p.googleapis.com/v1/token"; Connection conn = DriverManager.getConnection(url); From 9c1ae3a57164cb69ca3eb64dadffde2a3746368d Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Fri, 24 Jul 2026 12:52:21 -0400 Subject: [PATCH 4/8] fix code --- java-bigquery-jdbc/USER_GUIDE.md | 121 +++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 38 deletions(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index d39c7a4c72af..8658b3dfa394 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -24,8 +24,8 @@ This guide provides comprehensive instructions for configuring, developing with, - [CUJ 3: High-Throughput Result Extraction via Storage Read API](#cuj-3-high-throughput-result-extraction-via-storage-read-api) - [CUJ 4: Parameterized Queries with JSR-310 & Numeric Types](#cuj-4-parameterized-queries-with-jsr-310--numeric-types) - [CUJ 5: Nested Structs & Repeated Array Operations](#cuj-5-nested-structs--repeated-array-operations) - - [CUJ 6: Service Account Impersonation & Cross-Project Billing](#cuj-6-service-account-impersonation--cross-project-billing) - - [CUJ 7: Stored Procedure Execution](#cuj-7-stored-procedure-execution) + - [CUJ 6: Service Account Impersonation](#cuj-6-service-account-impersonation) + - [CUJ 7: Stored Procedure Execution & Procedural Results](#cuj-7-stored-procedure-execution--procedural-results) 8. [High-Throughput & Performance Tuning](#8-high-throughput--performance-tuning) 9. [Framework & BI Tool Integrations](#9-framework--bi-tool-integrations) - [Spring Boot & HikariCP](#spring-boot--hikaricp) @@ -88,7 +88,7 @@ public class BigQueryQuickstart { try (Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT name, age FROM users LIMIT 10")) { + ResultSet rs = stmt.executeQuery("SELECT name, age FROM `my-gcp-project-id.my_dataset.users` LIMIT 10")) { while (rs.next()) { System.out.printf("User: %s, Age: %d%n", rs.getString("name"), rs.getInt("age")); @@ -305,7 +305,7 @@ TRANSACTION; TRANSACTION; (Auto-re-executes BEGIN TRANSACTION; if setAutoCommit is still false) ``` -1. **Pre-requisite Check**: Calling `setAutoCommit(false)`, `commit()`, or `rollback()` requires `;EnableSession=true` in the connection URL. If disabled, an `IllegalStateException` is thrown. +1. **Pre-requisite Check**: Calling `setAutoCommit(false)`, `commit()`, or `rollback()` requires `;EnableSession=true` in the connection URL. If disabled or invoked without an active transaction, an `IllegalStateException` is thrown by the driver. 2. **Session & Transaction Start**: - `setAutoCommit(false)` issues `BEGIN TRANSACTION;` with `setCreateSession(true)`. - BigQuery assigns a unique `session_id` in the job statistics. @@ -379,9 +379,9 @@ public class FinancialTransactionCUJ { // 1. Start Transaction Session conn.setAutoCommit(false); - String debitSql = "UPDATE finance.accounts SET balance = balance - ? WHERE account_id = ?"; - String creditSql = "UPDATE finance.accounts SET balance = balance + ? WHERE account_id = ?"; - String auditSql = "INSERT INTO finance.audit_log (source_id, target_id, amount, transfer_time) VALUES (?, ?, ?, CURRENT_TIMESTAMP())"; + String debitSql = "UPDATE `my-gcp-project.finance.accounts` SET balance = balance - ? WHERE account_id = ?"; + String creditSql = "UPDATE `my-gcp-project.finance.accounts` SET balance = balance + ? WHERE account_id = ?"; + String auditSql = "INSERT INTO `my-gcp-project.finance.audit_log` (source_id, target_id, amount, transfer_time) VALUES (?, ?, ?, CURRENT_TIMESTAMP())"; try (PreparedStatement debitStmt = conn.prepareStatement(debitSql); PreparedStatement creditStmt = conn.prepareStatement(creditSql); @@ -445,7 +445,7 @@ public class EtlBatchIngestionCUJ { + ";EnableWriteAPI=true" // High-throughput write streaming + ";OAuthType=3"; - String insertSql = "INSERT INTO telemetry.sensor_readings (reading_id, sensor_id, temperature, is_valid, record_time) VALUES (?, ?, ?, ?, ?)"; + String insertSql = "INSERT INTO `my-gcp-project.telemetry.sensor_readings` (reading_id, sensor_id, temperature, is_valid, record_time) VALUES (?, ?, ?, ?, ?)"; try (Connection conn = DriverManager.getConnection(url); PreparedStatement pstmt = conn.prepareStatement(insertSql)) { @@ -491,7 +491,7 @@ public class HighThroughputExtractionCUJ { try (Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT user_id, event_name, event_timestamp FROM analytics.user_events WHERE event_date >= '2026-01-01'")) { + ResultSet rs = stmt.executeQuery("SELECT user_id, event_name, event_timestamp FROM `my-gcp-project.analytics.user_events` WHERE event_date >= '2026-01-01'")) { long rowCount = 0; while (rs.next()) { @@ -527,7 +527,7 @@ public class ParameterizedTypesCUJ { public static void main(String[] args) throws Exception { String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - String sql = "SELECT order_id FROM sales.orders " + String sql = "SELECT order_id FROM `my-project.sales.orders` " + "WHERE order_date = ? " + " AND created_timestamp <= ? " + " AND total_amount >= ? " @@ -576,7 +576,7 @@ public class ComplexTypesCUJ { String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; String query = "SELECT " - + " order_id, " + + " 1001 AS order_id, " + " STRUCT('123 Main St' AS street, 'Seattle' AS city) AS shipping_address, " + " [STRUCT('Item A' AS sku, 2 AS qty), STRUCT('Item B' AS sku, 1 AS qty)] AS line_items, " + " ['tag1', 'tag2', 'tag3'] AS tags"; @@ -588,7 +588,7 @@ public class ComplexTypesCUJ { while (rs.next()) { long orderId = rs.getLong("order_id"); - // 1. Reading Struct with Null Safety + // 1. Reading Struct Struct addressStruct = (Struct) rs.getObject("shipping_address"); if (addressStruct != null) { Object[] addressAttrs = addressStruct.getAttributes(); @@ -597,7 +597,7 @@ public class ComplexTypesCUJ { } } - // 2. Reading Array of Structs with Null Safety + // 2. Reading Array of Structs Array itemsArray = rs.getArray("line_items"); if (itemsArray != null && itemsArray.getArray() != null) { Object[] itemsObj = (Object[]) itemsArray.getArray(); @@ -612,7 +612,7 @@ public class ComplexTypesCUJ { } } - // 3. Reading Primitive Array with Null Safety + // 3. Reading Primitive Array Array tagsArray = rs.getArray("tags"); if (tagsArray != null && tagsArray.getArray() != null) { String[] tags = (String[]) tagsArray.getArray(); @@ -626,9 +626,14 @@ public class ComplexTypesCUJ { --- -### CUJ 6: Service Account Impersonation & Cross-Project Billing +### CUJ 6: Service Account Impersonation -**Scenario**: Running a query where billing occurs under Project A (`billing-project`), while querying data located in Project B (`data-project`) via Service Account Impersonation. +**Scenario**: An enterprise microservice running with Application Default Credentials (ADC) impersonates a central Service Account (`analytics-executor@my-gcp-project.iam.gserviceaccount.com`) to execute query jobs with elevated IAM permissions without storing or managing static private key files. + +> [!NOTE] +> **IAM Roles Required for Impersonation**: +> 1. **Caller Principal (ADC)**: Must have the `roles/iam.serviceAccountTokenCreator` role on the target Service Account. +> 2. **Impersonated Service Account**: Must have `roles/bigquery.jobUser` (to submit query jobs) and `roles/bigquery.dataViewer` (to read target datasets). ```java import java.sql.Connection; @@ -636,21 +641,20 @@ import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.Statement; -public class CrossProjectImpersonationCUJ { +public class ServiceAccountImpersonationCUJ { public static void main(String[] args) throws Exception { String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=billing-project-id" // Project billed for the query - + ";AdditionalProjects=data-project-id" // Cross-project dataset access - + ";DefaultDataset=shared_dataset" - + ";OAuthType=3" // Use ADC context for initial authentication - + ";ServiceAccountImpersonationEmail=analytics-executor@billing-project-id.iam.gserviceaccount.com"; + + ";ProjectId=my-gcp-project" + + ";DefaultDataset=analytics" + + ";OAuthType=3" // 3 = Application Default Credentials (ADC) context + + ";ServiceAccountImpersonationEmail=analytics-executor@my-gcp-project.iam.gserviceaccount.com"; try (Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT count(*) FROM `data-project-id.shared_dataset.global_logs`")) { + ResultSet rs = stmt.executeQuery("SELECT count(*) AS total_records FROM `my-gcp-project.analytics.user_events`")) { if (rs.next()) { - System.out.println("Cross-project log count: " + rs.getLong(1)); + System.out.println("Total events recorded via impersonated SA: " + rs.getLong("total_records")); } } } @@ -659,33 +663,74 @@ public class CrossProjectImpersonationCUJ { --- -### CUJ 7: Stored Procedure Execution +### CUJ 7: Stored Procedure Execution & Procedural Results + +**Scenario**: Calling a BigQuery stored procedure or procedural script and extracting output results. + +> [!IMPORTANT] +> **BigQuery Stored Procedure OUT Parameter Support**: +> Traditional JDBC `CallableStatement.registerOutParameter(...)` and `cstmt.getXXX(index)` methods are **not supported** for capturing `OUT` parameters. Because BigQuery's REST API returns query execution results as tabular data (`TableResult`) rather than parameter bind response buffers, `cstmt.getXXX()` will return `null`. +> +> To retrieve output variables or procedural results from BigQuery stored procedures, use one of the two supported patterns below via standard `ResultSet` extraction. + +#### Pattern A: Executing a Procedure that Returns a ResultSet +If a stored procedure executes `SELECT` statements in its body, call it directly using `Statement.executeQuery()`: + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; + +public class StoredProcedureResultSetCUJ { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("CALL `my-project.my_dataset.get_top_customers`()")) { + + while (rs.next()) { + System.out.printf("Customer: %s, Total Spent: $%.2f%n", + rs.getString("customer_name"), rs.getDouble("total_spent")); + } + } + } +} +``` -**Scenario**: Calling a BigQuery stored procedure with input and output parameters. +#### Pattern B: Retrieving Procedure OUT Parameters via Multi-Statement Script +To retrieve `OUT` parameters from a procedure, execute a multi-statement procedural script that declares an output variable, passes it into the `CALL` statement, and selects the variable to return it as a `ResultSet`: ```java import java.math.BigDecimal; -import java.sql.CallableStatement; import java.sql.Connection; import java.sql.DriverManager; -import java.sql.Types; +import java.sql.PreparedStatement; +import java.sql.ResultSet; -public class StoredProcedureCUJ { +public class StoredProcedureOutParameterCUJ { public static void main(String[] args) throws Exception { String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - String procSql = "CALL my_dataset.calculate_tax(?, ?)"; + // Multi-statement script declaring an output variable and selecting it as a ResultSet + String scriptSql = "DECLARE tax_out NUMERIC; " + + "CALL `my-project.my_dataset.calculate_tax`(?, tax_out); " + + "SELECT tax_out AS calculated_tax;"; try (Connection conn = DriverManager.getConnection(url); - CallableStatement cstmt = conn.prepareCall(procSql)) { + PreparedStatement pstmt = conn.prepareStatement(scriptSql)) { - cstmt.setBigDecimal(1, new BigDecimal("1000.00")); // Input subtotal - cstmt.registerOutParameter(2, Types.DECIMAL); // Output calculated tax + // Bind input parameter ($1 = subtotal) + pstmt.setBigDecimal(1, new BigDecimal("1000.00")); - cstmt.execute(); - - BigDecimal taxAmount = cstmt.getBigDecimal(2); - System.out.println("Calculated Tax Amount: $" + taxAmount); + // Execute script and read OUT variable from the returned ResultSet + try (ResultSet rs = pstmt.executeQuery()) { + if (rs.next()) { + BigDecimal taxAmount = rs.getBigDecimal("calculated_tax"); + System.out.println("Calculated Tax Amount: $" + taxAmount); + } + } } } } @@ -974,7 +1019,7 @@ Log files are generated inside the specified `LogPath` directory with per-connec **Standard Log Record Format**: ```text -2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] c.g.c.b.j.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM analytics.orders +2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] c.g.c.b.j.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM `my-project.analytics.orders` ``` - **Timestamp**: `yyyy-MM-dd HH:mm:ss.SSS` in local time. From d8a4896a46ae04ae2b5db6c5b5dc77535804feab Mon Sep 17 00:00:00 2001 From: Neenu Shaji Date: Fri, 24 Jul 2026 12:59:27 -0400 Subject: [PATCH 5/8] Update java-bigquery-jdbc/USER_GUIDE.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- java-bigquery-jdbc/USER_GUIDE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index 8658b3dfa394..dd1eba3b0c1f 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -27,9 +27,9 @@ This guide provides comprehensive instructions for configuring, developing with, - [CUJ 6: Service Account Impersonation](#cuj-6-service-account-impersonation) - [CUJ 7: Stored Procedure Execution & Procedural Results](#cuj-7-stored-procedure-execution--procedural-results) 8. [High-Throughput & Performance Tuning](#8-high-throughput--performance-tuning) -9. [Framework & BI Tool Integrations](#9-framework--bi-tool-integrations) - - [Spring Boot & HikariCP](#spring-boot--hikaricp) - - [SQL Clients (DBeaver / DataGrip)](#sql-clients-dbeaver--datagrip) +9. [Framework Integration & Deployment](#9-framework-integration--deployment) + - [Spring Boot (application.yml)](#spring-boot-applicationyml) + - [Key Deployment Guidelines](#key-deployment-guidelines) 10. [Logging, Diagnostics & Troubleshooting](#10-logging-diagnostics--troubleshooting) 11. [Official Documentation References](#11-official-documentation-references) From bc241cfcd014dbfb2cd3de5c86aac1679bf748c3 Mon Sep 17 00:00:00 2001 From: Neenu Shaji Date: Fri, 24 Jul 2026 13:00:21 -0400 Subject: [PATCH 6/8] Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- java-bigquery-jdbc/USER_GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index dd1eba3b0c1f..1d0c92728aef 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -969,7 +969,7 @@ jdbc:bigquery://https://www.my-universe.cloud:443 ```yaml spring: datasource: - url: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-gcp-project;DefaultDataset=my_dataset;OAuthType=0;OAuthServiceAcctEmail=sa@my-project.iam.gserviceaccount.com;OAuthPvtKeyPath=/path/to/key.json + url: "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-gcp-project;DefaultDataset=my_dataset;OAuthType=0;OAuthServiceAcctEmail=sa@my-project.iam.gserviceaccount.com;OAuthPvtKeyPath=/path/to/key.json" driver-class-name: com.google.cloud.bigquery.jdbc.BigQueryDriver hikari: maximum-pool-size: 10 From 2bd26b843ecd0116400fcfd150886574e8da356d Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Fri, 24 Jul 2026 13:02:58 -0400 Subject: [PATCH 7/8] corrcet log --- java-bigquery-jdbc/USER_GUIDE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md index 8658b3dfa394..3d33dd986e67 100644 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ b/java-bigquery-jdbc/USER_GUIDE.md @@ -1019,14 +1019,14 @@ Log files are generated inside the specified `LogPath` directory with per-connec **Standard Log Record Format**: ```text -2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] c.g.c.b.j.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM `my-project.analytics.orders` +2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] com.google.cloud.bigquery.jdbc.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM `my-project.analytics.orders` ``` - **Timestamp**: `yyyy-MM-dd HH:mm:ss.SSS` in local time. - **Connection ID (`MDC`)**: Unique identifier `[conn-bq-XXXX]` isolating statements per connection context. - **Level**: Padded log level (`INFO`, `FINE`, `WARNING`, etc.). - **Process ID & Thread**: OS Process ID and centered Thread Name. -- **Class & Method**: Abbreviated source class and method name originating the log message. +- **Class & Method**: Fully qualified source class name and method name originating the log message. --- From 74785eff069c6c4977478b9d602939acdd47b3f1 Mon Sep 17 00:00:00 2001 From: Neenu1995 Date: Tue, 28 Jul 2026 15:17:36 -0400 Subject: [PATCH 8/8] docs: address comments --- java-bigquery-jdbc/README.MD | 2 + java-bigquery-jdbc/USER_GUIDE.md | 1048 ----------------------- java-bigquery-jdbc/docs/STORAGE_APIS.md | 69 ++ java-bigquery-jdbc/docs/USER_GUIDE.md | 765 +++++++++++++++++ 4 files changed, 836 insertions(+), 1048 deletions(-) delete mode 100644 java-bigquery-jdbc/USER_GUIDE.md create mode 100644 java-bigquery-jdbc/docs/STORAGE_APIS.md create mode 100644 java-bigquery-jdbc/docs/USER_GUIDE.md diff --git a/java-bigquery-jdbc/README.MD b/java-bigquery-jdbc/README.MD index 834e47a24b99..3b2d085ed91c 100644 --- a/java-bigquery-jdbc/README.MD +++ b/java-bigquery-jdbc/README.MD @@ -7,6 +7,8 @@ Java idiomatic client for [BigQuery JDBC][product-docs]. - [Product Documentation][product-docs] - [Client Library Documentation][javadocs] +- [Driver User Guide](docs/USER_GUIDE.md) +- [Storage APIs Deep-Dive Guide](docs/STORAGE_APIS.md) ## Quickstart diff --git a/java-bigquery-jdbc/USER_GUIDE.md b/java-bigquery-jdbc/USER_GUIDE.md deleted file mode 100644 index 2d9fb607677f..000000000000 --- a/java-bigquery-jdbc/USER_GUIDE.md +++ /dev/null @@ -1,1048 +0,0 @@ -# Google BigQuery JDBC Driver User Guide - -This guide provides comprehensive instructions for configuring, developing with, and optimizing the **Google BigQuery JDBC Driver** (`google-cloud-bigquery-jdbc`). - -> [!NOTE] -> This guide is aligned with and references the official [Google Cloud BigQuery JDBC Documentation](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) and [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver). - ---- - -## Table of Contents -1. [Overview & Prerequisites](#1-overview--prerequisites) -2. [Connection URL Syntax & Quickstart](#2-connection-url-syntax--quickstart) -3. [Authentication Modes & Configuration](#3-authentication-modes--configuration) -4. [Connection Properties Reference](#4-connection-properties-reference) -5. [Data Type Mapping Reference](#5-data-type-mapping-reference) -6. [JDBC Driver Architecture & Core Features](#6-jdbc-driver-architecture--core-features) - - [Transaction Management & Multi-Statement Sessions](#transaction-management--multi-statement-sessions) - - [Type Mapping & Coercion Engine](#type-mapping--coercion-engine) - - [High-Throughput Storage Read & Write APIs](#high-throughput-storage-read--write-apis) - - [Diagnostic MDC Tracing & Connection Proxying](#diagnostic-mdc-tracing--connection-proxying) -7. [Common User Journeys (CUJs) & Code Recipes](#7-common-user-journeys-cujs--code-recipes) - - [CUJ 1: Financial & Multi-Table Transactions](#cuj-1-financial--multi-table-transactions) - - [CUJ 2: High-Performance ETL Ingestion Pipeline](#cuj-2-high-performance-etl-ingestion-pipeline) - - [CUJ 3: High-Throughput Result Extraction via Storage Read API](#cuj-3-high-throughput-result-extraction-via-storage-read-api) - - [CUJ 4: Parameterized Queries with JSR-310 & Numeric Types](#cuj-4-parameterized-queries-with-jsr-310--numeric-types) - - [CUJ 5: Nested Structs & Repeated Array Operations](#cuj-5-nested-structs--repeated-array-operations) - - [CUJ 6: Service Account Impersonation](#cuj-6-service-account-impersonation) - - [CUJ 7: Stored Procedure Execution & Procedural Results](#cuj-7-stored-procedure-execution--procedural-results) -8. [High-Throughput & Performance Tuning](#8-high-throughput--performance-tuning) -9. [Framework Integration & Deployment](#9-framework-integration--deployment) - - [Spring Boot (application.yml)](#spring-boot-applicationyml) - - [Key Deployment Guidelines](#key-deployment-guidelines) -10. [Logging, Diagnostics & Troubleshooting](#10-logging-diagnostics--troubleshooting) -11. [Official Documentation References](#11-official-documentation-references) - ---- - -## 1. Overview & Prerequisites - -The BigQuery JDBC Driver enables Java applications, BI tools, and ETL pipelines to interact with Google Cloud BigQuery using the standard Java Database Connectivity (JDBC) API (JDBC 4.2 compliant). - -### Prerequisites -- **Java Runtime**: JDK 8 or higher (JDK 11, 17, or 21 recommended). -- **Google Cloud Platform**: - - An active GCP Project with BigQuery API enabled. - - Required IAM roles (e.g., `BigQuery Data Viewer`, `BigQuery Job User`). - -### Installation Coordinates - -**Maven**: -```xml - - com.google.cloud - google-cloud-bigquery-jdbc - 1.1.0 - -``` - -**Gradle**: -```groovy -implementation 'com.google.cloud:google-cloud-bigquery-jdbc:1.1.0' -``` - ---- - -## 2. Connection URL Syntax & Quickstart - -The JDBC connection string format for BigQuery is: - -``` -jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=;[DefaultDataset=;][Property1=Value1;...] -``` - -### Basic Quickstart Example - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; -import java.sql.Statement; - -public class BigQueryQuickstart { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project-id" - + ";DefaultDataset=my_dataset" - + ";OAuthType=3"; // 3 = Application Default Credentials (ADC) - - try (Connection conn = DriverManager.getConnection(url); - Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT name, age FROM `my-gcp-project-id.my_dataset.users` LIMIT 10")) { - - while (rs.next()) { - System.out.printf("User: %s, Age: %d%n", rs.getString("name"), rs.getInt("age")); - } - } - } -} -``` - ---- - -## 3. Authentication Modes & Configuration - -The driver supports multiple OAuth 2.0 and identity workflows specified via the `OAuthType` connection property. - -| `OAuthType` | Authentication Strategy | Required Properties | -| :---: | :--- | :--- | -| **`0`** | **Service Account Key File / Key String** | `OAuthPvtKeyPath` (path to `.json`/`.p12` key) or `OAuthPvtKey` & `OAuthServiceAcctEmail` | -| **`1`** | **User Account (Interactive 3-Legged OAuth)** | `OAuthClientId`, `OAuthClientSecret` | -| **`2`** | **Pre-generated Access Token or Refresh Token** | `OAuthAccessToken` or `OAuthRefreshToken` (`OAuthClientId`, `OAuthClientSecret`) | -| **`3`** | **Application Default Credentials (ADC)** / gcloud CLI | None (reads `GOOGLE_APPLICATION_CREDENTIALS` or `gcloud` context) | -| **`4`** | **Workload Identity Federation (BYOID)** | `BYOID_AudienceUri`, `BYOID_CredentialSource`, `BYOID_SubjectTokenType` | - -### Service Account Authentication Example -```java -String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project-id" - + ";OAuthType=0" - + ";OAuthPvtKeyPath=/path/to/service-account-key.json"; -``` - -### Service Account Impersonation -To execute queries as an impersonated service account: -```java -String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project-id" - + ";OAuthType=3" - + ";ServiceAccountImpersonationEmail=target-sa@my-gcp-project.iam.gserviceaccount.com"; -``` - ---- - -## 4. Connection Properties Reference - -### Authentication & Impersonation Properties - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `OAuthType` | `-1` | Authentication mechanism: `0` (Service Account), `1` (User Account), `2` (Pre-generated Token / Refresh Token), `3` (ADC), `4` (Workload Identity Federation / BYOID). | -| `OAuthServiceAcctEmail` | `null` | Service Account email address for Service Account authentication or token scope. | -| `OAuthPvtKeyPath` | `null` | Path to JSON or P12 private key file for Service Account authentication. | -| `OAuthPvtKey` | `null` | Raw PKCS#8 private key string content for Service Account authentication. | -| `OAuthP12Password` | `notasecret` | Password for accessing encrypted `.p12` key files. | -| `OAuthAccessToken` | `null` | Pre-generated OAuth 2.0 access token string. | -| `OAuthAccessTokenReadonly` | `false` | Indicates whether the pre-generated access token has read-only scope. | -| `OAuthRefreshToken` | `null` | Pre-generated refresh token to automatically mint access tokens. | -| `OAuthClientId` | *Google Client ID* | OAuth 2.0 Client ID for user authentication or refresh token flows. | -| `OAuthClientSecret` | *Google Secret* | OAuth 2.0 Client Secret for user authentication or refresh token flows. | -| `ServiceAccountImpersonationEmail` | `null` | Target Service Account email to impersonate for query job execution. | -| `ServiceAccountImpersonationChain` | `null` | Comma-separated list of service account emails representing the impersonation chain. | -| `ServiceAccountImpersonationScopes` | `https://www.googleapis.com/auth/bigquery` | Comma-separated OAuth 2.0 scopes for impersonated credentials. | -| `ServiceAccountImpersonationTokenLifetime` | `3600` | Lifetime (in seconds) for impersonated service account tokens. | -| `RequestGoogleDriveScope` | `0` | If `1` (or `true`), appends the Google Drive read-only scope (`https://www.googleapis.com/auth/drive.readonly`) to query external Drive tables. | - -### Workload Identity Federation (BYOID) Properties - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `BYOID_AudienceUri` | `null` | Audience URI corresponding to the Workforce/Workload identity pool provider. | -| `BYOID_CredentialSource` | `null` | File path or JSON content defining the external subject token source. | -| `BYOID_PoolUserProject` | `null` | Google Cloud project number associated with the workforce pool. | -| `BYOID_SA_Impersonation_Uri` | `null` | Service Account impersonation URL for Workload Identity Federation. | -| `BYOID_SubjectTokenType` | `urn:ietf:params:oauth:tokentype:id_token` | Type of subject token provided (e.g., `urn:ietf:params:oauth:tokentype:jwt`). | -| `BYOID_TokenUri` | `https://sts.googleapis.com/v1/token` | Security Token Service (STS) token exchange endpoint URI. | - -### Core Query & Catalog Properties - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `ProjectId` | *Default Project* | Google Cloud Project ID for billing and query execution. | -| `DefaultDataset` | `null` | Default dataset scope for unqualified table references in SQL queries. | -| `AdditionalProjects` | `null` | Comma-separated list of additional project IDs accessible for catalog discovery and querying. | -| `EnableProjectDiscovery` | `false` | Automatically discovers all accessible Google Cloud projects as catalog entries when `true`. | -| `FilterTablesOnDefaultDataset` | `false` | Filters `DatabaseMetaData.getTables()` and `.getColumns()` to `DefaultDataset` when wildcard patterns are used. | -| `Location` | *Auto-detected* | GCP location/region for dataset and job execution (e.g., `US`, `EU`, `asia-east1`). | -| `QueryDialect` | `SQL` | Query dialect: `SQL` (Standard SQL) or `LEGACY_SQL`. | -| `UseQueryCache` | `true` | Enables or disables BigQuery query result caching. | -| `MaximumBytesBilled` | `0` (unlimited) | Limits bytes billed per query; query fails without cost if estimated bytes exceed this limit. | -| `Labels` | `null` | Comma-separated `key=value` pairs attached to query jobs (e.g., `env=prod,dept=analytics`). | -| `QueryProperties` | `null` | Connection-level query configuration properties passed to BigQuery job execution. | -| `JobCreationMode` | `2` | Job creation strategy: `1` (Stateful session mode) or `2` (Fast stateless job creation). | -| `MaxResults` | `10000` | Maximum number of rows returned per page during standard REST result set iteration. | -| `AllowLargeResults` | `true` | Allows large query results (required when using legacy SQL destination tables). | -| `LargeResultTable` | `null` | User-specified destination table name for saving query result sets. | -| `LargeResultDataset` | `null` | User-specified destination dataset name for saving query result sets. | -| `LargeResultsDatasetExpirationTime` | `3600000` (1 hour) | Expiration time (in milliseconds) for temporary destination tables in user-specified datasets. | -| `KMSKeyName` | `null` | Cloud KMS key resource name used for encrypting query results and destination tables. | - -### Session & Transaction Properties - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `EnableSession` | `false` | Enables multi-statement session creation and transaction support (`BEGIN`, `COMMIT`, `ROLLBACK`). | - -### High-Throughput Storage & Write API Properties - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `EnableHighThroughputAPI` | `false` | Enables BigQuery Storage Read API (gRPC/Arrow) for fast result set retrieval. | -| `HighThroughputMinTableSize` | `10000` | Minimum query result row count threshold required to trigger the Storage Read API. | -| `HighThroughputActivationRatio` | `2` | Minimum number of result pages required before switching to the Storage Read API. | -| `UnsupportedHTAPIFallback` | `true` | Automatically falls back to standard REST API when Storage Read API is unsupported or lacks permissions. | -| `EnableWriteAPI` | `false` | Enables BigQuery Storage Write API for high-performance batch insert streams. | -| `SWA_ActivationRowCount` | `3` | Minimum row threshold in `executeBatch()` to activate Storage Write API streaming. | -| `SWA_AppendRowCount` | `1000` | Batch row size per append stream request when using Storage Write API. | - -### Network, Proxy & Endpoint Overrides - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `ProxyHost` | `null` | Hostname or IP address of the HTTP/HTTPS proxy server. | -| `ProxyPort` | `null` | Port number of the proxy server. | -| `ProxyUid` | `null` | Username for proxy server authentication. | -| `ProxyPwd` | `null` | Password for proxy server authentication. | -| `EndpointOverrides` | `null` | Semicolon/comma-separated custom service endpoints (e.g., `OAuth2=...`, `READ_API=...`, `BIGQUERY=...`, `STS=...`). | -| `PrivateServiceConnectUris` | `null` | Custom Private Service Connect (PSC) endpoint URIs. | -| `universeDomain` | `googleapis.com` | Domain name for Trusted Partner Cloud (TPC) or custom Google universe instances. | -| `SSLTrustStore` | `null` | Path to custom Java TrustStore file containing trusted server certificates for SSL. | -| `SSLTrustStorePwd` | `null` | Password for accessing the custom Java TrustStore file. | -| `HttpConnectTimeout` | `0` (system default) | HTTP socket connection timeout in milliseconds. | -| `HttpReadTimeout` | `0` (system default) | HTTP socket read timeout in milliseconds. | - -### Driver Logging, Retries & Concurrency Tuning - -| Property Name | Default Value | Description | -| :--- | :---: | :--- | -| `LogLevel` | `0` (OFF) | Logging verbosity level (`0` = OFF, higher integers increase logging detail). | -| `LogPath` | `""` | Directory path where log files are written when logging is enabled. | -| `Timeout` | `0` (unlimited) | Length of time (in seconds) the connector retries failed API calls before timing out. | -| `JobTimeout` | `0` (unlimited) | Job execution timeout (in seconds) after which BigQuery cancels the query job server-side. | -| `RetryInitialDelay` | `0` | Initial delay (in seconds) before executing the first retry attempt. | -| `RetryMaxDelay` | `0` | Maximum delay limit (in seconds) between retry attempts. | -| `MetaDataFetchThreadCount` | `32` | Thread pool size used to parallelize `DatabaseMetaData` catalog RPC calls. | -| `ConnectionPoolSize` | `10` | Maximum size of the internal connection pool when connection pooling is enabled. | -| `ListenerPoolSize` | `10` | Maximum size of the listener thread pool when connection pooling is enabled. | -| `PartnerToken` | `null` | Partner tracking token appended to request headers for telemetry. | -| `RequestReason` | `null` | Reason string passed in the `x-goog-request-reason` HTTP header for auditing. | - ---- - -## 5. Data Type Mapping Reference - -When running queries through the JDBC driver for BigQuery, data types map as specified in the official [BigQuery JDBC Data Type Mapping](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver): - -| BigQuery SQL Type | Java / JDBC Type | Recommended Getter / Setter | -| :--- | :--- | :--- | -| `ARRAY` | `java.sql.Array` | `rs.getArray(col)` | -| `BIGNUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | -| `BOOL` | `java.lang.Boolean` | `rs.getBoolean(col)` | -| `BYTES` | `byte[]` | `rs.getBytes(col)` | -| `DATE` | `java.sql.Date` / `java.time.LocalDate` | `rs.getDate(col)` or `rs.getObject(col, LocalDate.class)` | -| `DATETIME` | `java.sql.Timestamp` / `java.time.LocalDateTime` | `rs.getTimestamp(col)` or `rs.getObject(col, LocalDateTime.class)` | -| `FLOAT64` | `java.lang.Double` | `rs.getDouble(col)` | -| `GEOGRAPHY` | `java.lang.String` | `rs.getString(col)` | -| `INT64` | `java.lang.Long` | `rs.getLong(col)` | -| `INTERVAL` | `java.lang.String` | `rs.getString(col)` | -| `JSON` | `java.lang.String` | `rs.getString(col)` | -| `NUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | -| `STRING` | `java.lang.String` | `rs.getString(col)` | -| `STRUCT` | `java.sql.Struct` | `rs.getObject(col)` (casts to `java.sql.Struct`) | -| `TIME` | `java.sql.Time` / `java.time.LocalTime` | `rs.getTime(col)` or `rs.getObject(col, LocalTime.class)` | -| `TIMESTAMP` | `java.sql.Timestamp` / `java.time.Instant` | `rs.getTimestamp(col)` or `rs.getObject(col, Instant.class)` | - ---- - -## 6. JDBC Driver Architecture & Core Features - -### Transaction Management & Multi-Statement Sessions - -BigQuery supports **Multi-Statement Transactions** across tables using standard SQL primitives (`BEGIN TRANSACTION`, `COMMIT TRANSACTION`, `ROLLBACK TRANSACTION`). The driver bridges standard JDBC methods (`setAutoCommit`, `commit`, `rollback`) directly to BigQuery's underlying session engine. - -#### Session Lifecycle Flow: - -``` -[DriverManager.getConnection()] - │ - (EnableSession=true) - │ - ┌──────────▼──────────┐ - │ setAutoCommit(false)│ ──────► Executes: BEGIN TRANSACTION; (with setCreateSession=true) - └──────────┬──────────┘ Captures: session_id from Job statistics - │ Attaches session_id to Connection state - │ - ┌──────────▼──────────┐ - │ Execute DML & SQL │ ──────► Appends ConnectionProperty(key="session_id", value="...") - │ Statements │ Runs all queries within active BigQuery Session - └──────────┬──────────┘ - │ - ┌───────┴───────┐ - │ │ - ▼ ▼ -┌─────────┐ ┌──────────┐ -│commit() │ │rollback()│ -└────┬────┘ └────┬─────┘ - │ │ - ▼ ▼ -Executes: Executes: -COMMIT ROLLBACK -TRANSACTION; TRANSACTION; - │ │ - └───────┬───────┘ - │ - ▼ -(Auto-re-executes BEGIN TRANSACTION; if setAutoCommit is still false) -``` - -1. **Pre-requisite Check**: Calling `setAutoCommit(false)`, `commit()`, or `rollback()` requires `;EnableSession=true` in the connection URL. If disabled or invoked without an active transaction, an `IllegalStateException` is thrown by the driver. -2. **Session & Transaction Start**: - - `setAutoCommit(false)` issues `BEGIN TRANSACTION;` with `setCreateSession(true)`. - - BigQuery assigns a unique `session_id` in the job statistics. - - The driver attaches `session_id` to the connection state as a `ConnectionProperty`. -3. **Statement Propagation**: - - Every `Statement` or `PreparedStatement` created on the connection inherits `ConnectionProperty("session_id", sessionId)`. - - DML operations (`INSERT`, `UPDATE`, `DELETE`, `MERGE`) execute inside the session block. -4. **Commit & Rollback**: - - `commit()` executes `COMMIT TRANSACTION;` in the active session. - - `rollback()` executes `ROLLBACK TRANSACTION;` in the active session. - - If `autoCommit` remains `false`, the driver automatically starts the next transaction block by issuing `BEGIN TRANSACTION;`. -5. **Connection Close Safety**: - - If an uncommitted transaction is pending when `conn.close()` is invoked, the driver automatically performs an internal `rollbackImpl()` to abort pending modifications safely. -6. **Isolation Level & Holdability**: - - Isolation level: `Connection.TRANSACTION_SERIALIZABLE` (BigQuery multi-statement snapshot isolation). - - Holdability: `ResultSet.CLOSE_CURSORS_AT_COMMIT`. - ---- - -### Type Mapping & Coercion Engine - -The driver handles bidirectional type mapping and object coercion: - -- **Bidirectional Mapping**: Manages symmetric mapping between JDBC SQL types (`java.sql.Types`), Java types (`BigDecimal`, `byte[]`, `Date`, `Timestamp`), and BigQuery Standard SQL column types (`StandardSQLTypeName`). -- **Coercion Utilities**: Safely converts BigQuery scalar values and Arrow/JSON batch fields into target Java classes transparently during getter/setter invocations. - ---- - -### High-Throughput Storage Read & Write APIs - -For enterprise data ingestion and analytics extraction, the driver integrates with Google Cloud Storage APIs: - -- **Storage Read API (`EnableHighThroughputAPI=true`)**: - Performs high-speed result extraction over gRPC streams using Apache Arrow instead of REST JSON pagination, avoiding REST payload deserialization bottlenecks for large ResultSets. -- **Storage Write API (`EnableWriteAPI=true`)**: - Enables high-throughput streaming appends for batch operations executed via `PreparedStatement.executeBatch()`. - ---- - -### Diagnostic MDC Tracing & Connection Proxying - -The driver manages execution context and diagnostics transparently: - -- **ThreadLocal MDC Tracing**: Automatically populates `BIGQUERY_JDBC_MDC` with connection ID, catalog, dataset, and job execution metrics for application logging. -- **Automatic Resource Cleanup**: Tracks open statements and result sets to prevent unhandled memory leaks upon connection closure. - ---- - -## 7. Common User Journeys (CUJs) & Code Recipes - -### CUJ 1: Financial & Multi-Table Transactions - -**Scenario**: A financial application debits one account and credits another atomically across multiple tables using JDBC transactions. - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.SQLException; - -public class FinancialTransactionCUJ { - public static void main(String[] args) { - // Note: EnableSession=true is REQUIRED for transaction support - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";DefaultDataset=finance" - + ";EnableSession=true" - + ";OAuthType=3"; // 3 = Application Default Credentials (ADC) - - try (Connection conn = DriverManager.getConnection(url)) { - // 1. Start Transaction Session - conn.setAutoCommit(false); - - String debitSql = "UPDATE `my-gcp-project.finance.accounts` SET balance = balance - ? WHERE account_id = ?"; - String creditSql = "UPDATE `my-gcp-project.finance.accounts` SET balance = balance + ? WHERE account_id = ?"; - String auditSql = "INSERT INTO `my-gcp-project.finance.audit_log` (source_id, target_id, amount, transfer_time) VALUES (?, ?, ?, CURRENT_TIMESTAMP())"; - - try (PreparedStatement debitStmt = conn.prepareStatement(debitSql); - PreparedStatement creditStmt = conn.prepareStatement(creditSql); - PreparedStatement auditStmt = conn.prepareStatement(auditSql)) { - - double amount = 500.00; - long sourceAccount = 1001L; - long targetAccount = 2002L; - - // Debit source account - debitStmt.setDouble(1, amount); - debitStmt.setLong(2, sourceAccount); - debitStmt.executeUpdate(); - - // Credit target account - creditStmt.setDouble(1, amount); - creditStmt.setLong(2, targetAccount); - creditStmt.executeUpdate(); - - // Log audit record - auditStmt.setLong(1, sourceAccount); - auditStmt.setLong(2, targetAccount); - auditStmt.setDouble(3, amount); - auditStmt.executeUpdate(); - - // 2. Commit transaction atomically - conn.commit(); - System.out.println("Transaction committed successfully."); - - } catch (SQLException e) { - // 3. Rollback transaction on failure - System.err.println("Transaction failed. Rolling back changes: " + e.getMessage()); - conn.rollback(); - } - } catch (SQLException e) { - e.printStackTrace(); - } - } -} -``` - ---- - -### CUJ 2: High-Performance ETL Ingestion Pipeline - -**Scenario**: An ETL ingestion pipeline loads batches of records into BigQuery using `executeBatch()` with parameter binding and high-throughput write APIs. - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.SQLException; -import java.time.Instant; -import java.util.UUID; - -public class EtlBatchIngestionCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";DefaultDataset=telemetry" - + ";EnableWriteAPI=true" // High-throughput write streaming - + ";OAuthType=3"; - - String insertSql = "INSERT INTO `my-gcp-project.telemetry.sensor_readings` (reading_id, sensor_id, temperature, is_valid, record_time) VALUES (?, ?, ?, ?, ?)"; - - try (Connection conn = DriverManager.getConnection(url); - PreparedStatement pstmt = conn.prepareStatement(insertSql)) { - - int batchSize = 1000; - for (int i = 1; i <= batchSize; i++) { - pstmt.setString(1, UUID.randomUUID().toString()); - pstmt.setInt(2, 100 + (i % 10)); // Sensor IDs 100-109 - pstmt.setDouble(3, 20.0 + (Math.random() * 15.0)); // Temp 20-35 C - pstmt.setBoolean(4, true); - pstmt.setObject(5, java.sql.Timestamp.from(Instant.now())); - pstmt.addBatch(); - - if (i % 500 == 0) { - int[] results = pstmt.executeBatch(); - System.out.printf("Flushed batch of %d records.%n", results.length); - } - } - } - } -} -``` - ---- - -### CUJ 3: High-Throughput Result Extraction via Storage Read API - -**Scenario**: An analytical application extracts large query result sets using standard JDBC ResultSet APIs accelerated transparently by BigQuery Storage Read API. - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; -import java.sql.Statement; - -public class HighThroughputExtractionCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";DefaultDataset=analytics" - + ";EnableHighThroughputAPI=true" // Stream via gRPC Storage Read API - + ";OAuthType=3"; - - try (Connection conn = DriverManager.getConnection(url); - Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT user_id, event_name, event_timestamp FROM `my-gcp-project.analytics.user_events` WHERE event_date >= '2026-01-01'")) { - - long rowCount = 0; - while (rs.next()) { - rowCount++; - String userId = rs.getString("user_id"); - String eventName = rs.getString("event_name"); - java.sql.Timestamp eventTime = rs.getTimestamp("event_timestamp"); - } - System.out.println("Extracted " + rowCount + " rows via Storage Read API."); - } - } -} -``` - ---- - -### CUJ 4: Parameterized Queries with JSR-310 & Numeric Types - -**Scenario**: Binding precise decimals (`BigDecimal`), temporal types (`java.sql.Date`, `Timestamp`), and binary payloads (`byte[]`). - -```java -import java.math.BigDecimal; -import java.sql.Connection; -import java.sql.Date; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.Timestamp; -import java.time.Instant; -import java.time.LocalDate; - -public class ParameterizedTypesCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - - String sql = "SELECT order_id FROM `my-project.sales.orders` " - + "WHERE order_date = ? " - + " AND created_timestamp <= ? " - + " AND total_amount >= ? " - + " AND security_token = ?"; - - try (Connection conn = DriverManager.getConnection(url); - PreparedStatement pstmt = conn.prepareStatement(sql)) { - - // Temporal parameter binding - pstmt.setDate(1, Date.valueOf(LocalDate.of(2026, 7, 22))); - pstmt.setTimestamp(2, Timestamp.from(Instant.now())); - - // High-precision numeric binding - pstmt.setBigDecimal(3, new BigDecimal("12499.99")); - - // Binary payload binding - byte[] tokenBytes = new byte[] {0x12, 0x34, 0x56, 0x78}; - pstmt.setBytes(4, tokenBytes); - - try (ResultSet rs = pstmt.executeQuery()) { - while (rs.next()) { - System.out.println("Matched Order ID: " + rs.getLong("order_id")); - } - } - } - } -} -``` - ---- - -### CUJ 5: Nested Structs & Repeated Array Operations - -**Scenario**: Processing e-commerce orders containing nested address `STRUCT` objects and line-item `ARRAY` objects with null-safe guardrails. - -```java -import java.sql.Array; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; -import java.sql.Statement; -import java.sql.Struct; - -public class ComplexTypesCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - - String query = "SELECT " - + " 1001 AS order_id, " - + " STRUCT('123 Main St' AS street, 'Seattle' AS city) AS shipping_address, " - + " [STRUCT('Item A' AS sku, 2 AS qty), STRUCT('Item B' AS sku, 1 AS qty)] AS line_items, " - + " ['tag1', 'tag2', 'tag3'] AS tags"; - - try (Connection conn = DriverManager.getConnection(url); - Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery(query)) { - - while (rs.next()) { - long orderId = rs.getLong("order_id"); - - // 1. Reading Struct - Struct addressStruct = (Struct) rs.getObject("shipping_address"); - if (addressStruct != null) { - Object[] addressAttrs = addressStruct.getAttributes(); - if (addressAttrs != null && addressAttrs.length >= 2) { - System.out.printf("Order %d Address: %s, %s%n", orderId, addressAttrs[0], addressAttrs[1]); - } - } - - // 2. Reading Array of Structs - Array itemsArray = rs.getArray("line_items"); - if (itemsArray != null && itemsArray.getArray() != null) { - Object[] itemsObj = (Object[]) itemsArray.getArray(); - for (Object itemObj : itemsObj) { - if (itemObj instanceof Struct) { - Struct item = (Struct) itemObj; - Object[] itemAttrs = item.getAttributes(); - if (itemAttrs != null && itemAttrs.length >= 2) { - System.out.printf(" Item -> SKU: %s, Qty: %s%n", itemAttrs[0], itemAttrs[1]); - } - } - } - } - - // 3. Reading Primitive Array - Array tagsArray = rs.getArray("tags"); - if (tagsArray != null && tagsArray.getArray() != null) { - String[] tags = (String[]) tagsArray.getArray(); - System.out.println(" Tags: " + String.join(", ", tags)); - } - } - } - } -} -``` - ---- - -### CUJ 6: Service Account Impersonation - -**Scenario**: An enterprise microservice running with Application Default Credentials (ADC) impersonates a central Service Account (`analytics-executor@my-gcp-project.iam.gserviceaccount.com`) to execute query jobs with elevated IAM permissions without storing or managing static private key files. - -> [!NOTE] -> **IAM Roles Required for Impersonation**: -> 1. **Caller Principal (ADC)**: Must have the `roles/iam.serviceAccountTokenCreator` role on the target Service Account. -> 2. **Impersonated Service Account**: Must have `roles/bigquery.jobUser` (to submit query jobs) and `roles/bigquery.dataViewer` (to read target datasets). - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; -import java.sql.Statement; - -public class ServiceAccountImpersonationCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";DefaultDataset=analytics" - + ";OAuthType=3" // 3 = Application Default Credentials (ADC) context - + ";ServiceAccountImpersonationEmail=analytics-executor@my-gcp-project.iam.gserviceaccount.com"; - - try (Connection conn = DriverManager.getConnection(url); - Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("SELECT count(*) AS total_records FROM `my-gcp-project.analytics.user_events`")) { - - if (rs.next()) { - System.out.println("Total events recorded via impersonated SA: " + rs.getLong("total_records")); - } - } - } -} -``` - ---- - -### CUJ 7: Stored Procedure Execution & Procedural Results - -**Scenario**: Calling a BigQuery stored procedure or procedural script and extracting output results. - -> [!IMPORTANT] -> **BigQuery Stored Procedure OUT Parameter Support**: -> Traditional JDBC `CallableStatement.registerOutParameter(...)` and `cstmt.getXXX(index)` methods are **not supported** for capturing `OUT` parameters. Because BigQuery's REST API returns query execution results as tabular data (`TableResult`) rather than parameter bind response buffers, `cstmt.getXXX()` will return `null`. -> -> To retrieve output variables or procedural results from BigQuery stored procedures, use one of the two supported patterns below via standard `ResultSet` extraction. - -#### Pattern A: Executing a Procedure that Returns a ResultSet -If a stored procedure executes `SELECT` statements in its body, call it directly using `Statement.executeQuery()`: - -```java -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; -import java.sql.Statement; - -public class StoredProcedureResultSetCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - - try (Connection conn = DriverManager.getConnection(url); - Statement stmt = conn.createStatement(); - ResultSet rs = stmt.executeQuery("CALL `my-project.my_dataset.get_top_customers`()")) { - - while (rs.next()) { - System.out.printf("Customer: %s, Total Spent: $%.2f%n", - rs.getString("customer_name"), rs.getDouble("total_spent")); - } - } - } -} -``` - -#### Pattern B: Retrieving Procedure OUT Parameters via Multi-Statement Script -To retrieve `OUT` parameters from a procedure, execute a multi-statement procedural script that declares an output variable, passes it into the `CALL` statement, and selects the variable to return it as a `ResultSet`: - -```java -import java.math.BigDecimal; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.ResultSet; - -public class StoredProcedureOutParameterCUJ { - public static void main(String[] args) throws Exception { - String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=3"; - - // Multi-statement script declaring an output variable and selecting it as a ResultSet - String scriptSql = "DECLARE tax_out NUMERIC; " - + "CALL `my-project.my_dataset.calculate_tax`(?, tax_out); " - + "SELECT tax_out AS calculated_tax;"; - - try (Connection conn = DriverManager.getConnection(url); - PreparedStatement pstmt = conn.prepareStatement(scriptSql)) { - - // Bind input parameter ($1 = subtotal) - pstmt.setBigDecimal(1, new BigDecimal("1000.00")); - - // Execute script and read OUT variable from the returned ResultSet - try (ResultSet rs = pstmt.executeQuery()) { - if (rs.next()) { - BigDecimal taxAmount = rs.getBigDecimal("calculated_tax"); - System.out.println("Calculated Tax Amount: $" + taxAmount); - } - } - } - } -} -``` - ---- - -## 8. High-Throughput & Performance Tuning - -For large dataset extractions and high-concurrency ingestion, tune the following connection options: - -### High-Throughput Storage Read API (HTAPI) Deep Dive - -The driver can switch query result extraction from standard REST JSON pagination to high-speed gRPC streams using the **BigQuery Storage Read API** (Apache Arrow over gRPC). - -#### HTAPI Property Relationship & Activation Matrix - -The decision to activate the Read API for a query is governed by four interconnected connection properties: - -| Property Name | Connection Parameter | Default Value | Functional Role | -| :--- | :--- | :---: | :--- | -| **`EnableHighThroughputAPI`** | `EnableHighThroughputAPI=true` | `false` | **Master Toggle**: Must be `true` to evaluate Read API switching. | -| **`HighThroughputMinTableSize`** | `HighThroughputMinTableSize=10000` | `10000` | **Minimum Row Threshold**: Minimum total rows (`totalRows`) required. | -| **`HighThroughputActivationRatio`** | `HighThroughputActivationRatio=2` | `2` | **Page Ratio Threshold**: `totalRows / pageSize` must exceed this ratio. | -| **`MaxResults`** | `MaxResults=1000` | `1000` (or server default) | **Page Size (`pageSize`)**: Controls rows per page in standard REST calls. | - -#### Activation Formula & Internal Safeguards - -The driver evaluates Storage Read API activation using the following rules: - -$$\text{Storage Read API Active} = \text{EnableHighThroughputAPI} \land \text{meetsReadRatio}$$ - -Where `meetsReadRatio` requires **ALL** of the following conditions to pass: - -1. `EnableHighThroughputAPI == true` -2. `totalRows >= HighThroughputMinTableSize` (default: $\ge 10,000$ rows) -3. `results.hasNextPage() == true` (**Crucial Safeguard**): If all rows fit in the first response page, the driver will **never** switch to Read API, avoiding redundant gRPC calls and memory re-allocations. -4. **Page Ratio Test**: $\frac{\text{totalRows}}{\text{pageSize}} > \text{HighThroughputActivationRatio}$ - -#### How to Manipulate Properties to Control Read API Activation - -##### 1. Force Read API for Smaller Query Results ($\ge 100$ rows) -To trigger the Read API even for small analytical datasets: -``` -jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-project;OAuthType=0;EnableHighThroughputAPI=true;HighThroughputMinTableSize=100;HighThroughputActivationRatio=0;MaxResults=50 -``` -*Why this works*: `totalRows` ($100$) $\ge$ `MinTableSize` ($100$), `pageSize` ($50$) forces `hasNextPage() = true`, and $\frac{100}{50} = 2 > 0$ (`ActivationRatio`). - -##### 2. High-Volume Production Defaults ($\ge 100,000$ rows) -With standard defaults (`EnableHighThroughputAPI=true`): -- For a $100,000$ row result set with `pageSize = 1,000`: $\frac{100,000}{1,000} = 100 > 2$ (`ActivationRatio`). -- The driver transparently switches to the gRPC Storage Read API stream. - -##### 3. Automatic Fallback on Permission Failure -If `EnableHighThroughputAPI=true` is set but the connecting service account lacks the `BigQuery Read Session User` IAM role, the driver catches the `PERMISSION_DENIED` status and **automatically falls back to standard REST JSON pagination** without failing the application's query. - -### Storage Write API (SWA) Configuration Property Matrix - -The **BigQuery Storage Write API** provides high-performance gRPC streaming ingestion for batch `PreparedStatement.executeBatch()` operations. The behavior and streaming thresholds are governed by three connection properties: - -#### 1. SWA Property Reference - -| Property Name | Connection Parameter | Default Value | Functional Role | -| :--- | :--- | :---: | :--- | -| **`EnableWriteAPI`** | `EnableWriteAPI=true` | `false` | **Master Toggle**: Must be set to `true` to enable gRPC Storage Write API stream ingestion. | -| **`SWA_ActivationRowCount`** | `SWA_ActivationRowCount=3` | `3` | **Activation Threshold**: Minimum number of row batches added via `addBatch()` required to trigger SWA streaming. | -| **`SWA_AppendRowCount`** | `SWA_AppendRowCount=1000` | `1000` | **Stream Chunk Size**: Maximum number of rows packed into a single gRPC append stream payload before flushing. | - -#### 2. SWA Activation Formula & Fallback Safeguard - -The driver evaluates Storage Write API activation during `PreparedStatement.executeBatch()` as follows: - -$$\text{Storage Write API Active} = \text{EnableWriteAPI} \land (\text{batchParameters.size()} \ge \text{SWA\_ActivationRowCount})$$ - -- **Above Threshold ($\ge \text{SWA\_ActivationRowCount}$)**: The driver opens a gRPC Storage Write API stream, converts batch parameters to JSON rows, appends rows in chunks of `SWA_AppendRowCount`, and commits the write stream. -- **Below Threshold ($< \text{SWA\_ActivationRowCount}$)**: The driver falls back to standard REST DML query concatenation (`INSERT INTO ...; INSERT INTO ...;`), preventing unnecessary gRPC channel overhead for small batches. - -#### 3. SWA Configuration Scenarios Matrix - -| Workload Scenario | `EnableWriteAPI` | `SWA_ActivationRowCount` | `SWA_AppendRowCount` | Executed Ingestion Mechanism | Enterprise Use Case | -| :--- | :---: | :---: | :---: | :--- | :--- | -| **Standard DML (Default)** | `false` | `3` (ignored) | `1000` (ignored) | Concatenated REST SQL DML (`INSERT INTO ...;`) | Small transactional DML; standard SQL compatibility. | -| **Default High-Throughput ETL** | `true` | `3` | `1000` | gRPC SWA stream for batches $\ge 3$, flushes every 1,000 rows | Standard batch loader applications (Spring Batch, Spark). | -| **Real-Time Micro-Batching** | `true` | `1` | `100` | gRPC SWA stream for any batch $\ge 1$, flushes every 100 rows | High-frequency streaming events (Kafka/Flink consumers). | -| **High-Volume Bulk Ingestion** | `true` | `100` | `5000` | gRPC SWA stream for batches $\ge 100$, flushes in 5,000 row chunks | Large nightly bulk ETL loading millions of records. | -| **Small Batch Fallback** | `true` | `100` | `1000` | Concatenated REST SQL DML (for batches $< 100$ rows) | Adaptive pipelines using REST for small batches, gRPC for large ones. | - ---- - -### Additional Performance Tuning Options - -- **Metadata Thread Pooling**: For applications fetching schema catalog metadata across multiple datasets, set `;MetaDataFetchThreadCount=64` to parallelize catalog RPC calls. -- **Internal Queue Buffering**: The driver maintains an internal asynchronous row buffer (default 10,000 rows) to decouple network fetch RPCs from client `ResultSet.next()` iteration. - ---- - -### Custom Endpoints, Private Service Connect (PSC) & Universe Domains - -For enterprise environments using **VPC Service Controls**, **Private Service Connect (PSC)**, **Regional Endpoints**, or **Trusted Partner Cloud (TPC)** instances, the driver supports full endpoint overriding and universe domain configuration. - -#### 1. Endpoint Overrides (`EndpointOverrides`) - -The `EndpointOverrides` property allows redirecting traffic for individual Google Cloud services to private IP endpoints, regional gateways, or corporate proxy targets. - -**Format**: -``` -EndpointOverrides=SERVICE_KEY_1=URI_1,SERVICE_KEY_2=URI_2; -``` - -**Supported Service Keys**: - -| Service Key | Target Service | Default Endpoint URI | Typical Override Use Case | -| :--- | :--- | :--- | :--- | -| **`BIGQUERY`** | BigQuery Core REST API | `https://www.googleapis.com/bigquery/v2/` | Regional endpoints (e.g. `us-east4`) or PSC private VIPs. | -| **`READ_API`** | Storage Read API (HTAPI) | `bigquerystorage.googleapis.com:443` | Private gRPC endpoints for high-throughput extractions. | -| **`OAUTH2`** | Google OAuth2 Token Server | `https://oauth2.googleapis.com/token` | Private OAuth token exchange gateways. | -| **`STS`** | Security Token Service | `https://sts.googleapis.com` | Workload Identity / BYOID token exchange. | - -**Example - Private Service Connect (PSC) Endpoint**: -``` -jdbc:bigquery://https://bigquery-privateendpoint.p.googleapis.com:443 - ;ProjectId=my-gcp-project - ;OAuthType=0 - ;EndpointOverrides=BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com,OAUTH2=https://oauth2-private.p.googleapis.com/token -``` - -**Example - Regional BigQuery Endpoint Override**: -``` -jdbc:bigquery://https://us-east4-bigquery.googleapis.com:443 - ;ProjectId=my-gcp-project - ;OAuthType=0 - ;EndpointOverrides=BIGQUERY=https://us-east4-bigquery.googleapis.com -``` - -#### 2. Custom Endpoint Requirements per Authentication Type (`OAuthType`) - -The interaction between custom endpoint properties (`EndpointOverrides`, `PrivateServiceConnectUris`, `universeDomain`) and authentication flows (`OAuthType`) behaves as follows: - -| Authentication Mode (`OAuthType`) | Applicable Custom Endpoint Keys | Endpoint Override Behavior & Actions | Primary Enterprise Use Case | -| :--- | :--- | :--- | :--- | -| **`OAuthType=0`
(Service Account)** | • `OAUTH2`
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Overrides OAuth token exchange server URI via `OAUTH2`
• Sets custom universe domain for TPC instances | Private service account key token exchange via internal proxy; regional or PSC endpoints. | -| **`OAuthType=1`
(User Account)** | • `OAUTH2`
• `BIGQUERY`
• `READ_API` | • Overrides 3-legged browser authorization token server via `OAUTH2` | Desktop browser 3-legged OAuth code exchange via private corporate authorization server. | -| **`OAuthType=2`
(Pre-generated Token)** | • `OAUTH2` (Refresh Token)
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Overrides token refresh endpoint via `OAUTH2` for refresh tokens
• Bypasses token server for static access tokens | Custom token refresh server for long-running batch jobs; direct static access token routing. | -| **`OAuthType=3`
(ADC)** | • `BIGQUERY`
• `READ_API`
• `PrivateServiceConnectUris` | • Routes BigQuery REST query calls and Storage Read gRPC streams via PSC VIPs or regional endpoints | On-premise or GKE workload running ADC routed strictly over Private Service Connect (PSC). | -| **`OAuthType=4`
(External Account / BYOID)** | • `STS`
• `BYOIDTokenUri`
• `BYOIDSAImpersonationUri`
• `BIGQUERY`
• `READ_API`
• `universeDomain` | • Configures Security Token Service URL (`token_url`) via `STS`/`BYOIDTokenUri`
• Configures custom universe domain for external account config | Multi-cloud Workload Identity Federation (AWS/Azure/OIDC) exchanging tokens via custom STS VIP. | -| **Impersonation
(`ServiceAccountImpersonationEmail`)** | • Applies across ALL `OAuthType` modes | • Routes IAM Credentials API impersonation calls over configured BigQuery and OAuth endpoints | Cross-project or elevated role execution via Service Account Impersonation. | - ---- - -#### 3. Endpoint Configuration Code Recipes by `OAuthType` - -##### Recipe A: Service Account (`OAuthType=0`) with Private OAuth & BigQuery PSC -```java -// Service Account authentication routed entirely over Private Service Connect (PSC) VIPs -String url = "jdbc:bigquery://https://bigquery-privateendpoint.p.googleapis.com:443" - + ";ProjectId=my-gcp-project" - + ";OAuthType=0" - + ";OAuthServiceAcctEmail=sa-data-reader@my-gcp-project.iam.gserviceaccount.com" - + ";OAuthPvtKeyPath=/var/secrets/bq-sa-key.json" - + ";EndpointOverrides=" - + "BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com," - + "OAUTH2=https://oauth2-private.p.googleapis.com/token," - + "READ_API=https://storage-privateendpoint.p.googleapis.com"; - -Connection conn = DriverManager.getConnection(url); -``` - -##### Recipe B: Pre-Generated Refresh Token (`OAuthType=2`) with Custom Token Server -```java -// Refresh token flow pointing to an internal corporate OAuth token refresh server -String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";OAuthType=2" - + ";OAuthRefreshToken=1//04_example_refresh_token" - + ";OAuthClientId=my-client-id.apps.googleusercontent.com" - + ";OAuthClientSecret=my-client-secret" - + ";EndpointOverrides=OAUTH2=https://oauth-proxy.internal.corp.com/token"; - -Connection conn = DriverManager.getConnection(url); -``` - -##### Recipe C: Workload Identity Federation (`OAuthType=4` / BYOID) with Custom STS Endpoint -```java -// AWS / OIDC Workload Identity exchanging tokens via custom Security Token Service (STS) endpoint -String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";OAuthType=4" - + ";BYOID_AudienceUri=//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider" - + ";BYOID_SubjectTokenType=urn:ietf:params:aws:token-type:aws-sigv4" - + ";BYOID_CredentialSource={\"environment_id\":\"aws-1\",\"region_url\":\"http://169.254.169.254/latest/meta-data/placement/availability-zone\"}" - + ";BYOID_TokenUri=https://sts-privateendpoint.p.googleapis.com/v1/token" - + ";EndpointOverrides=STS=https://sts-privateendpoint.p.googleapis.com/v1/token"; - -Connection conn = DriverManager.getConnection(url); -``` - -##### Recipe D: Regional Endpoint Override with Application Default Credentials (`OAuthType=3`) -```java -// ADC authentication targeting a regional BigQuery REST endpoint (us-east4) -String url = "jdbc:bigquery://https://us-east4-bigquery.googleapis.com:443" - + ";ProjectId=my-gcp-project" - + ";OAuthType=3" - + ";EndpointOverrides=BIGQUERY=https://us-east4-bigquery.googleapis.com"; - -Connection conn = DriverManager.getConnection(url); -``` - ---- - -#### 4. Private Service Connect URIs (`PrivateServiceConnectUris`) - -For VPC setups using explicit Private Service Connect URI aliases: -``` -jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443 - ;ProjectId=my-gcp-project - ;OAuthType=0 - ;PrivateServiceConnectUris=BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com -``` - -#### 5. Custom Universe Domains (`universeDomain`) - -To target Google Cloud Trusted Partner Cloud (TPC) or non-default Universe Domain instances, specify `universeDomain`: - -``` -jdbc:bigquery://https://www.my-universe.cloud:443 - ;ProjectId=my-gcp-project - ;OAuthType=0 - ;universeDomain=my-universe.cloud -``` - ---- - -## 9. Framework Integration & Deployment - -### Spring Boot (`application.yml`) -```yaml -spring: - datasource: - url: "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=my-gcp-project;DefaultDataset=my_dataset;OAuthType=0;OAuthServiceAcctEmail=sa@my-project.iam.gserviceaccount.com;OAuthPvtKeyPath=/path/to/key.json" - driver-class-name: com.google.cloud.bigquery.jdbc.BigQueryDriver - hikari: - maximum-pool-size: 10 - minimum-idle: 2 - idle-timeout: 300000 - connection-test-query: SELECT 1 -``` - -### Key Deployment Guidelines -1. **Driver Class**: `com.google.cloud.bigquery.jdbc.BigQueryDriver` -2. **Connection Pooling**: BigQuery connection objects are lightweight wrappers around thread-safe Google API clients. Standard connection pools like HikariCP can be safely used. -3. **IAM Permissions**: Ensure the principal has `BigQuery Data Viewer` and `BigQuery Job User` roles on the target project and dataset. - ---- - -## 10. Logging, Diagnostics & Troubleshooting - -### Configuring Connection-Level Logging - -The driver includes a built-in logging subsystem built on top of `java.util.logging`. Logging can be configured directly via JDBC URL connection properties or system environment variables: - -#### Connection URL Parameters: -```java -String url = "jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443" - + ";ProjectId=my-gcp-project" - + ";OAuthType=0" - + ";LogLevel=5" // Set log level (1=SEVERE to 7=FINEST) - + ";LogPath=/var/log/bigquery-jdbc"; // Output directory for log files -``` - -#### Log Level Reference: - -| Integer Value | String Constant | `java.util.logging` Level | Description | -| :---: | :---: | :---: | :--- | -| `0` | `OFF` | `Level.OFF` | Logging completely disabled (default). | -| `1` | `SEVERE` | `Level.SEVERE` | Critical errors, connection failures, unrecoverable exceptions. | -| `2` | `WARNING` | `Level.WARNING` | Non-fatal warnings, API fallbacks (e.g., Read API permission denied fallback to REST). | -| `3` | `INFO` | `Level.INFO` | High-level driver events, connection establishment, job submission. | -| `4` | `CONFIG` | `Level.CONFIG` | Driver configuration initialization and property resolution details. | -| `5` | `FINE` | `Level.FINE` | SQL query execution statements, parameter binding, row counts. | -| `6` | `FINER` | `Level.FINER` | Internal method entry (`++enter++`) and exit (`++exit++`) tracing. | -| `7` | `FINEST` | `Level.FINEST` | Maximum diagnostic verbosity, raw batch payloads, Arrow stream packet details. | - -#### Log File Naming & Format - -Log files are generated inside the specified `LogPath` directory with per-connection log isolation. - -**Standard Log Record Format**: -```text -2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] com.google.cloud.bigquery.jdbc.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM `my-project.analytics.orders` -``` - -- **Timestamp**: `yyyy-MM-dd HH:mm:ss.SSS` in local time. -- **Connection ID (`MDC`)**: Unique identifier `[conn-bq-XXXX]` isolating statements per connection context. -- **Level**: Padded log level (`INFO`, `FINE`, `WARNING`, etc.). -- **Process ID & Thread**: OS Process ID and centered Thread Name. -- **Class & Method**: Fully qualified source class name and method name originating the log message. - ---- - -### Exception Hierarchy - -All driver exceptions inherit from `java.sql.SQLException`: - -- **`BigQueryJdbcException`**: Base exception class wrapping underlying Google Cloud API and network RPC errors. -- **`BigQueryJdbcSqlSyntaxErrorException`**: Thrown when standard SQL query validation or parsing fails on the server. -- **`BigQueryConversionException`**: Thrown on data type coercion failures or unparseable target representations. -- **`BigQueryJdbcSqlFeatureNotSupportedException`**: Thrown when an unsupported JDBC API method is invoked. - ---- - -## 11. Official Documentation References - -- 🌐 [Official Google Cloud BigQuery JDBC Driver Overview](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) -- ⚙️ [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver) -- 📊 [BigQuery Quotas & Limits](https://cloud.google.com/bigquery/quotas) diff --git a/java-bigquery-jdbc/docs/STORAGE_APIS.md b/java-bigquery-jdbc/docs/STORAGE_APIS.md new file mode 100644 index 000000000000..1c5fbf8dd12b --- /dev/null +++ b/java-bigquery-jdbc/docs/STORAGE_APIS.md @@ -0,0 +1,69 @@ +# BigQuery Storage APIs Deep Dive & Tuning Guide + +This document provides architectural details, property matrices, activation criteria, and workload tuning scenarios for the **BigQuery Storage Read API** and **BigQuery Storage Write API** integrated into the BigQuery JDBC Driver. + +--- + +## 1. High-Throughput Storage Read API (HTAPI) + +The Storage Read API streams query result sets over high-speed gRPC channels using Apache Arrow format, bypassing standard REST JSON serialization for large datasets. + +### Property Reference + +| Property Name | Connection Parameter | Default Value | Functional Role | +| :--- | :--- | :---: | :--- | +| **`EnableHighThroughputAPI`** | `EnableHighThroughputAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Read API evaluation. | +| **`HighThroughputMinTableSize`** | `HighThroughputMinTableSize=10000` | `10000` | **Minimum Row Threshold**: Minimum total rows (`totalRows`) required. | +| **`HighThroughputActivationRatio`** | `HighThroughputActivationRatio=2` | `2` | **Page Ratio Threshold**: `totalRows / MaxResults` ratio required. | +| **`MaxResults`** | `MaxResults=10000` | `10000` | **Page Size**: Controls rows per page in standard REST calls. | + +### Activation Criteria & Fallback Mechanics + +When `EnableHighThroughputAPI=true` is set, the driver transparently switches to the Storage Read API if all of the following conditions are met: + +1. **Master Toggle**: `EnableHighThroughputAPI=true` is set. +2. **Minimum Row Threshold**: The query returns at least `HighThroughputMinTableSize` rows (default: $\ge 10,000$ rows). +3. **Multiple Response Pages**: The result set spans more than one page (total rows exceed `MaxResults`). If all rows fit on page 1, standard REST is used to avoid unnecessary gRPC stream setup. +4. **Activation Ratio Test**: The ratio of total rows to page size ($\frac{\text{totalRows}}{\text{MaxResults}}$) exceeds `HighThroughputActivationRatio` (default: $> 2$). + +> [!NOTE] +> **Automatic Permission Fallback**: If `EnableHighThroughputAPI=true` is set but the connecting principal lacks the `BigQuery Read Session User` IAM role, the driver catches the `PERMISSION_DENIED` status and automatically falls back to standard REST JSON pagination. + +### Workload Scenarios Matrix + +| Workload Scenario | `EnableHighThroughputAPI` | `HighThroughputMinTableSize` | `HighThroughputActivationRatio` | `MaxResults` | Execution Mechanism | Use Case | +| :--- | :---: | :---: | :---: | :---: | :--- | :--- | +| **Standard REST (Default)** | `false` | `10000` (ignored) | `2` (ignored) | `10000` | REST JSON Pagination | Small/medium queries; standard REST security policies. | +| **Default Production Extractions** | `true` | `10000` | `2` | `10000` | gRPC Storage Read API (for results $> 20,000$ rows) | Standard analytical reports and ETL extracts. | +| **Aggressive Streaming** | `true` | `100` | `0` | `50` | gRPC Storage Read API (for results $\ge 100$ rows) | High-speed streaming for smaller analytical datasets. | +| **Bulk ETL Analytics** | `true` | `50000` | `5` | `10000` | gRPC Storage Read API (for results $> 50,000$ rows) | Large multi-gigabyte dataset extractions. | + +--- + +## 2. Storage Write API (SWA) + +The Storage Write API streams high-throughput bulk insertions over gRPC channels for `PreparedStatement.executeBatch()` calls. + +### Property Reference + +| Property Name | Connection Parameter | Default Value | Functional Role | +| :--- | :--- | :---: | :--- | +| **`EnableWriteAPI`** | `EnableWriteAPI=true` | `false` | **Master Toggle**: Must be `true` to enable Storage Write API streaming. | +| **`SWA_ActivationRowCount`** | `SWA_ActivationRowCount=3` | `3` | **Activation Threshold**: Minimum batch size added via `addBatch()` required to trigger SWA. | +| **`SWA_AppendRowCount`** | `SWA_AppendRowCount=1000` | `1000` | **Chunk Size**: Maximum rows per gRPC append payload before flushing. | + +### Activation Criteria & Fallback Mechanics + +When `EnableWriteAPI=true` is set, the driver evaluates the batch size during `PreparedStatement.executeBatch()`: + +- **At or Above Threshold ($\ge \text{SWA\_ActivationRowCount}$)**: The driver opens a gRPC Storage Write stream and appends batch records in payload chunks governed by `SWA_AppendRowCount`. +- **Below Threshold ($< \text{SWA\_ActivationRowCount}$)**: The driver uses standard SQL DML (`INSERT INTO ...`) to avoid gRPC stream overhead for tiny batches. + +### Workload Scenarios Matrix + +| Workload Scenario | `EnableWriteAPI` | `SWA_ActivationRowCount` | `SWA_AppendRowCount` | Execution Mechanism | Use Case | +| :--- | :---: | :---: | :---: | :--- | :--- | +| **Standard SQL DML (Default)** | `false` | `3` (ignored) | `1000` (ignored) | Concatenated REST SQL DML | Small transactional DML; standard SQL compatibility. | +| **Default High-Throughput ETL** | `true` | `3` | `1000` | gRPC SWA stream (batches $\ge 3$, flushes per 1,000 rows) | Standard batch loader applications (Spring Batch, Spark). | +| **Real-Time Micro-Batching** | `true` | `1` | `100` | gRPC SWA stream (batches $\ge 1$, flushes per 100 rows) | High-frequency streaming events (Kafka/Flink consumers). | +| **High-Volume Bulk Ingestion** | `true` | `100` | `5000` | gRPC SWA stream (batches $\ge 100$, flushes per 5,000 rows) | Large nightly bulk ETL loading millions of records. | diff --git a/java-bigquery-jdbc/docs/USER_GUIDE.md b/java-bigquery-jdbc/docs/USER_GUIDE.md new file mode 100644 index 000000000000..0f38e0fe252e --- /dev/null +++ b/java-bigquery-jdbc/docs/USER_GUIDE.md @@ -0,0 +1,765 @@ +# Google BigQuery JDBC Driver User Guide + +This guide provides comprehensive instructions for configuring, developing with, and optimizing the **Google BigQuery JDBC Driver** (`google-cloud-bigquery-jdbc`). + +> [!NOTE] +> This guide is aligned with and references the official [Google Cloud BigQuery JDBC Documentation](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) and [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver). + +--- + +## Table of Contents +1. [Overview & Prerequisites](#1-overview--prerequisites) +2. [Connection URL Syntax & Quickstart](#2-connection-url-syntax--quickstart) +3. [Authentication Modes & Configuration](#3-authentication-modes--configuration) +4. [Connection Properties Reference](#4-connection-properties-reference) +5. [Data Type Mapping Reference](#5-data-type-mapping-reference) +6. [JDBC Driver Architecture & Core Features](#6-jdbc-driver-architecture--core-features) + - [Transaction Management & Multi-Statement Sessions](#transaction-management--multi-statement-sessions) + - [High-Throughput Storage Read & Write APIs](#high-throughput-storage-read--write-apis) +7. [Feature Examples & Code Snippets](#7-feature-examples--code-snippets) + - [Transactions (Manual Commit & Rollback)](#transactions-manual-commit--rollback) + - [Prepared Statements & Parameter Binding](#prepared-statements--parameter-binding) + - [Callable Statements & Stored Procedures](#callable-statements--stored-procedures) + - [Batch Ingestion with Storage Write API](#batch-ingestion-with-storage-write-api) + - [High-Throughput Storage Read API](#high-throughput-storage-read-api) + - [Struct & Array Column Operations](#struct--array-column-operations) + - [Service Account Impersonation](#service-account-impersonation) +8. [High-Throughput & Performance Tuning](#8-high-throughput--performance-tuning) +9. [Framework Integration & Deployment](#9-framework-integration--deployment) + - [Spring Boot (application.yml)](#spring-boot-applicationyml) + - [Key Deployment Guidelines](#key-deployment-guidelines) +10. [Logging, Diagnostics & Troubleshooting](#10-logging-diagnostics--troubleshooting) +11. [Official Documentation References](#11-official-documentation-references) + +--- + +## 1. Overview & Prerequisites + +The BigQuery JDBC Driver enables Java applications, BI tools, and ETL pipelines to interact with Google Cloud BigQuery using the standard Java Database Connectivity (JDBC) API (JDBC 4.2 compliant). + +### Prerequisites +- **Java Runtime**: JDK 8 or higher (JDK 11, 17, or 21 recommended). +- **Google Cloud Platform**: + - An active GCP Project with BigQuery API enabled. + - Required IAM roles (e.g., `BigQuery Data Viewer`, `BigQuery Job User`). + +### Installation Coordinates + +**Maven**: +```xml + + com.google.cloud + google-cloud-bigquery-jdbc + 1.1.0 + +``` + +**Gradle**: +```groovy +implementation 'com.google.cloud:google-cloud-bigquery-jdbc:1.1.0' +``` + +--- + +## 2. Connection URL Syntax & Quickstart + +The JDBC connection string format for BigQuery is: + +``` +jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=;[DefaultDataset=;][Property1=Value1;...] +``` + +### Basic Quickstart Example + +```java +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; + +public class BigQueryQuickstart { + public static void main(String[] args) throws Exception { + String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project-id" + + ";DefaultDataset=my_dataset" + + ";OAuthType=3"; // 3 = Application Default Credentials (ADC) + + try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT name, age FROM `my-gcp-project-id.my_dataset.users` LIMIT 10")) { + + while (rs.next()) { + System.out.printf("User: %s, Age: %d%n", rs.getString("name"), rs.getInt("age")); + } + } + } +} +``` + +--- + +## 3. Authentication Modes & Configuration + +The driver supports multiple OAuth 2.0 and identity workflows specified via the `OAuthType` connection property. + +| `OAuthType` | Authentication Strategy / Variant | Required Properties | Supported / Optional Properties | Notes | +| :---: | :--- | :--- | :--- | :--- | +| **`0`** | **Service Account Key File (JSON)** | `OAuthPvtKeyPath` | `OAuthServiceAcctEmail` | Path to service account `.json` key file. Service account email and private key are automatically extracted from the file. | +| **`0`** | **Service Account Key File (P12)** | `OAuthPvtKeyPath`, `OAuthServiceAcctEmail` | `OAuthP12Password` | Path to `.p12` key file. Requires explicit `OAuthServiceAcctEmail` as `.p12` files do not contain email metadata. `OAuthP12Password` defaults to `notasecret`. | +| **`0`** | **Service Account Key String (JSON)** | `OAuthPvtKey` | `OAuthServiceAcctEmail` | Raw JSON key file string content in URL. Service account email and private key are automatically extracted from the JSON string. | +| **`0`** | **Service Account Key String (PKCS#8 / P12 Bytes)** | `OAuthPvtKey`, `OAuthServiceAcctEmail` | `OAuthP12Password` | Raw PKCS#8 or P12 bytes key string content in URL. Requires explicit `OAuthServiceAcctEmail`. | +| **`1`** | **User Account (Interactive 3-Legged OAuth)** | None (for built-in client app) | `OAuthClientId`, `OAuthClientSecret` | Launches local browser tab for interactive user login on desktop. Built-in client ID/secret used if omitted. | +| **`2`** | **Pre-generated Access Token** | `OAuthAccessToken` | `OAuthAccessTokenReadonly` | Uses short-lived bearer token directly. `OAuthAccessTokenReadonly` defaults to `false`. | +| **`2`** | **Pre-generated Refresh Token** | `OAuthRefreshToken`, `OAuthClientId`, `OAuthClientSecret` | None | Uses refresh token with client credentials to automatically mint access tokens. | +| **`3`** | **Application Default Credentials (ADC)** | None | None | Uses GCP standard credential resolution (`GOOGLE_APPLICATION_CREDENTIALS` env var or `gcloud` CLI context). | +| **`4`** | **Workload / Workforce Identity Federation (BYOID File)** | `OAuthPvtKeyPath` | None | Path to external account credential JSON configuration file. | +| **`4`** | **Workload / Workforce Identity Federation (BYOID Content)** | `OAuthPvtKey` | None | Raw JSON string content of external account credentials. | +| **`4`** | **Workload / Workforce Identity Federation (BYOID Properties)** | `BYOID_AudienceUri`, `BYOID_CredentialSource`, `BYOID_SubjectTokenType` | `BYOID_PoolUserProject`, `BYOID_SA_Impersonation_Uri`, `BYOID_TokenUri` | Inline BYOID connection properties configured directly in connection URL string. | + +> [!NOTE] +> **Global Authentication Properties**: +> The following properties are supported across **all** `OAuthType` authentication modes: +> - `universeDomain`: Custom universe domain name for Google Cloud Dedicated (GCD) or non-default Google Cloud environments (defaults to `googleapis.com`). +> - `ServiceAccountImpersonationEmail`, `ServiceAccountImpersonationChain`, `ServiceAccountImpersonationScopes`, `ServiceAccountImpersonationTokenLifetime`: Service account impersonation properties. + +### Service Account Authentication Example +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project-id" + + ";OAuthType=0" + + ";OAuthPvtKeyPath=/path/to/service-account-key.json"; +``` + +### Service Account Impersonation +To execute queries as an impersonated service account: +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project-id" + + ";OAuthType=3" + + ";ServiceAccountImpersonationEmail=target-sa@my-gcp-project.iam.gserviceaccount.com"; +``` + +--- + +## 4. Connection Properties Reference + +> [!TIP] +> **Boolean Connection Properties**: +> For boolean connection properties (such as `OAuthAccessTokenReadonly`, `RequestGoogleDriveScope`, `EnableHighThroughputAPI`, etc.), prefer using boolean values (`true` / `false`) when possible (e.g., `;OAuthAccessTokenReadonly=true`). Numeric representation (`1` / `0`) is also accepted for backwards compatibility. + +### Authentication & Impersonation Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `OAuthType` | **Required** (Default: `-1`) | **Required.** Specifies the authentication mechanism: `0` (Service Account), `1` (User Account), `2` (Pre-generated Token / Refresh Token), `3` (ADC), `4` (Workload & Workforce Identity Federation / BYOID). | +| `OAuthServiceAcctEmail` | `null` | Service Account email address for Service Account authentication or token scope. | +| `OAuthPvtKeyPath` | `null` | Path to JSON or P12 private key file for Service Account authentication. | +| `OAuthPvtKey` | `null` | Raw PKCS#8 private key string content for Service Account authentication. | +| `OAuthP12Password` | `notasecret` | Password for accessing encrypted `.p12` key files. | +| `OAuthAccessToken` | `null` | Pre-generated OAuth 2.0 access token string. | +| `OAuthAccessTokenReadonly` | `false` | Indicates whether the pre-generated access token has read-only scope. | +| `OAuthRefreshToken` | `null` | Pre-generated refresh token to automatically mint access tokens. | +| `OAuthClientId` | *Google Client ID* | OAuth 2.0 Client ID for user authentication or refresh token flows. | +| `OAuthClientSecret` | *Google Secret* | OAuth 2.0 Client Secret for user authentication or refresh token flows. | +| `ServiceAccountImpersonationEmail` | `null` | Target Service Account email to impersonate for query job execution. | +| `ServiceAccountImpersonationChain` | `null` | Comma-separated list of service account emails representing the impersonation chain. | +| `ServiceAccountImpersonationScopes` | `https://www.googleapis.com/auth/bigquery` | Comma-separated OAuth 2.0 scopes for impersonated credentials. | +| `ServiceAccountImpersonationTokenLifetime` | `3600` | Lifetime (in seconds) for impersonated service account tokens. | +| `RequestGoogleDriveScope` | `0` | If `1` (or `true`), appends the Google Drive read-only scope (`https://www.googleapis.com/auth/drive.readonly`) to query external Drive tables. | + +### Workload & Workforce Identity Federation (BYOID) Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `BYOID_AudienceUri` | `null` | Audience URI corresponding to the Workforce/Workload identity pool provider. | +| `BYOID_CredentialSource` | `null` | File path or JSON content defining the external subject token source. | +| `BYOID_PoolUserProject` | `null` | Google Cloud project number associated with the workforce pool. | +| `BYOID_SA_Impersonation_Uri` | `null` | Service Account impersonation URL for Workload Identity Federation. | +| `BYOID_SubjectTokenType` | `urn:ietf:params:oauth:tokentype:id_token` | Type of subject token provided (e.g., `urn:ietf:params:oauth:tokentype:jwt`). | +| `BYOID_TokenUri` | `https://sts.googleapis.com/v1/token` | Security Token Service (STS) token exchange endpoint URI. | + +### Core Query & Catalog Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `ProjectId` | *Default Project* | Google Cloud Project ID for billing and query execution. | +| `DefaultDataset` | `null` | Default dataset scope for unqualified table references in SQL queries and metadata catalog filtering (when `FilterTablesOnDefaultDataset=true`). | +| `AdditionalProjects` | `null` | Comma-separated list of additional project IDs accessible for catalog discovery and querying. | +| `EnableProjectDiscovery` | `false` | Automatically discovers all accessible Google Cloud projects as catalog entries when `true`. | +| `FilterTablesOnDefaultDataset` | `false` | When `true`, restricts `DatabaseMetaData` catalog calls (such as `getTables()` and `getColumns()`) to `DefaultDataset` whenever catalog/schema parameters are omitted or set to wildcard (`%`), preventing full project scans. | +| `Location` | *Auto-detected* | GCP location/region for dataset and job execution (e.g., `US`, `EU`, `asia-east1`). | +| `QueryDialect` | `SQL` | Query dialect: `SQL` (Standard SQL) or `LEGACY_SQL`. | +| `UseQueryCache` | `true` | Enables or disables BigQuery query result caching. | +| `MaximumBytesBilled` | `0` (unlimited) | Limits bytes billed per query; query fails without cost if estimated bytes exceed this limit. | +| `Labels` | `null` | Comma-separated `key=value` pairs attached to query jobs (e.g., `env=prod,dept=analytics`). | +| `QueryProperties` | `null` | Connection-level query configuration properties passed to BigQuery job execution. | +| `JobCreationMode` | `2` | Job creation strategy: `1` (`JOB_CREATION_REQUIRED` - forces explicit query job creation for every query) or `2` (`JOB_CREATION_OPTIONAL` - default, allows queries to execute directly without creating explicit jobs when possible). | +| `MaxResults` | `10000` | Maximum number of rows returned per page during standard REST result set iteration. | +| `AllowLargeResults` | `true` | Allows large query results (required when using legacy SQL destination tables). | +| `LargeResultTable` | `null` | Destination table name for query result sets. If omitted when `LargeResultDataset` is set, the driver automatically generates a temporary table name (`_jdbc_tmp_`). | +| `LargeResultDataset` | `null` | Destination dataset name for query result sets. If the specified dataset does not exist, the driver automatically creates it. Under Legacy SQL mode, defaults to creating/using `_jdbc_tmp`. | +| `LargeResultsDatasetExpirationTime` | `3600000` (1 hour) | Expiration time (in milliseconds) for temporary destination tables in user-specified datasets. | +| `KMSKeyName` | `null` | Cloud KMS key resource name used for encrypting query results and destination tables. | + +### Session & Transaction Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `EnableSession` | `false` | Enables multi-statement session creation and transaction support (`BEGIN`, `COMMIT`, `ROLLBACK`). | + +### High-Throughput Storage & Write API Properties + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `EnableHighThroughputAPI` | `false` | Enables BigQuery Storage Read API (gRPC/Arrow) for faster result retrieval. Requires `BigQuery Read Session User` (`roles/bigquery.readSessionUser`) IAM permission. | +| `HighThroughputMinTableSize` | `10000` | Minimum query result row count threshold required to trigger the Storage Read API. | +| `HighThroughputActivationRatio` | `2` | Minimum number of result pages required before switching to the Storage Read API. | +| `UnsupportedHTAPIFallback` | `true` | Automatically falls back to standard REST API when Storage Read API is unsupported or lacks permissions. | +| `EnableWriteAPI` | `false` | Enables BigQuery Storage Write API for high-performance batch insert streams. | +| `SWA_ActivationRowCount` | `3` | Minimum row threshold in `executeBatch()` to activate Storage Write API streaming. | +| `SWA_AppendRowCount` | `1000` | Batch row size per append stream request when using Storage Write API. | + +### Network, Proxy & Endpoint Overrides + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `ProxyHost` | `null` | Hostname or IP address of the HTTP/HTTPS proxy server. | +| `ProxyPort` | `null` | Port number of the proxy server. | +| `ProxyUid` | `null` | Username for proxy server authentication. | +| `ProxyPwd` | `null` | Password for proxy server authentication. | +| `EndpointOverrides` | `null` | Semicolon or comma-separated list of custom service endpoint overrides. Supported keys: `BIGQUERY`, `READ_API`, `OAUTH2`, `STS`, `ACCOUNTS`. | +| `PrivateServiceConnectUris` | `null` | Alias for `EndpointOverrides`. Accepts the same custom service endpoint override format and supported keys. | +| `universeDomain` | `googleapis.com` | Domain name for Google Cloud Dedicated (GCD) or custom Google universe instances. | +| `SSLTrustStore` | `null` | Path to custom Java TrustStore file containing trusted server certificates for SSL. | +| `SSLTrustStorePwd` | `null` | Password for accessing the custom Java TrustStore file. | +| `SSLTrustStoreType` | *System Default* (`JKS`/`PKCS12`) | Type of the custom Java TrustStore file. | +| `SSLTrustStoreProvider` | `null` | Security provider name for the custom Java TrustStore. | +| `HttpConnectTimeout` | `0` (system default) | HTTP socket connection timeout in milliseconds. | +| `HttpReadTimeout` | `0` (system default) | HTTP socket read timeout in milliseconds. | + +### Driver Logging, Retries & Concurrency Tuning + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `LogLevel` | `0` (OFF) | Logging verbosity level (`0` = OFF to `7` = FINEST). Controls internal `java.util.logging` detail. See [Logging, Diagnostics & Troubleshooting](#10-logging-diagnostics--troubleshooting) for level definitions and details. | +| `LogPath` | `""` | Directory path where log files are written when logging is enabled. | +| `Timeout` | `0` (unlimited) | Length of time (in seconds) the connector retries failed API calls before timing out. | +| `JobTimeout` | `0` (unlimited) | Job execution timeout (in seconds) after which BigQuery cancels the query job server-side. | +| `RetryInitialDelay` | `0` | Initial delay (in seconds) before executing the first retry attempt. | +| `RetryMaxDelay` | `0` | Maximum delay limit (in seconds) between retry attempts. | +| `MetaDataFetchThreadCount` | `32` | Thread pool size used to parallelize `DatabaseMetaData` catalog RPC calls. | +| `ConnectionPoolSize` | `10` | Maximum size of the internal connection pool when connection pooling is enabled. | +| `ListenerPoolSize` | `10` | Maximum size of the listener thread pool when connection pooling is enabled. | +| `RequestReason` | `null` | Reason string passed in the `x-goog-request-reason` HTTP header for auditing. | + +### OpenTelemetry & Cloud Observability Exporters + +| Property Name | Default Value | Description | +| :--- | :---: | :--- | +| `enableGcpTraceExporter` | `false` | Enables direct exporting of driver OpenTelemetry trace spans to Google Cloud Trace. | +| `enableGcpLogExporter` | `false` | Enables direct exporting of driver OpenTelemetry logs to Google Cloud Logging. | +| `gcpTelemetryProjectId` | `null` | GCP Project ID target for OpenTelemetry log and trace telemetry export. Defaults to query `ProjectId` if omitted. | +| `gcpTelemetryCredentials` | `null` | File path or raw JSON credentials string for OpenTelemetry GCP exporters. | +| `useGlobalOpenTelemetry` | `false` | Instructs the driver to register with the global OpenTelemetry instance (`GlobalOpenTelemetry`) in the JVM environment. | + +--- + +## 5. Data Type Mapping Reference + +When running queries through the JDBC driver for BigQuery, data types map as specified in the official [BigQuery JDBC Data Type Mapping](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver): + +| BigQuery SQL Type | Java / JDBC Type | Recommended Getter / Setter | +| :--- | :--- | :--- | +| `ARRAY` | `java.sql.Array` | `rs.getArray(col)` | +| `BIGNUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | +| `BOOL` | `java.lang.Boolean` | `rs.getBoolean(col)` | +| `BYTES` | `byte[]` | `rs.getBytes(col)` | +| `DATE` | `java.sql.Date` / `java.time.LocalDate` | `rs.getDate(col)` or `rs.getObject(col, LocalDate.class)` | +| `DATETIME` | `java.sql.Timestamp` / `java.time.LocalDateTime` | `rs.getTimestamp(col)` or `rs.getObject(col, LocalDateTime.class)` | +| `FLOAT64` | `java.lang.Double` | `rs.getDouble(col)` | +| `GEOGRAPHY` | `java.lang.String` | `rs.getString(col)` | +| `INT64` | `java.lang.Long` | `rs.getLong(col)` | +| `INTERVAL` | `java.lang.String` | `rs.getString(col)` | +| `JSON` | `java.lang.String` | `rs.getString(col)` | +| `NUMERIC` | `java.math.BigDecimal` | `rs.getBigDecimal(col)` | +| `STRING` | `java.lang.String` | `rs.getString(col)` | +| `STRUCT` | `java.sql.Struct` | `rs.getObject(col)` (casts to `java.sql.Struct`) | +| `TIME` | `java.sql.Time` / `java.time.LocalTime` | `rs.getTime(col)` or `rs.getObject(col, LocalTime.class)` | +| `TIMESTAMP` | `java.sql.Timestamp` / `java.time.Instant` | `rs.getTimestamp(col)` or `rs.getObject(col, Instant.class)` | + +--- + +## 6. JDBC Driver Architecture & Core Features + +### Transaction Management & Multi-Statement Sessions + +BigQuery supports **Multi-Statement Transactions** across tables using standard SQL primitives (`BEGIN TRANSACTION`, `COMMIT TRANSACTION`, `ROLLBACK TRANSACTION`). The driver bridges standard JDBC methods (`setAutoCommit`, `commit`, `rollback`) directly to BigQuery's underlying session engine. + +#### Session Lifecycle Flow: + +``` +[DriverManager.getConnection()] + │ + (EnableSession=true) + │ + ┌──────────▼──────────┐ + │ setAutoCommit(false)│ ──────► Begins transaction block in session + └──────────┬──────────┘ + │ + ┌──────────▼──────────┐ + │ Execute DML & SQL │ ──────► Runs queries within active session + │ Statements │ + └──────────┬──────────┘ + │ + ┌───────┴───────┐ + │ │ + ▼ ▼ +┌─────────┐ ┌──────────┐ +│commit() │ │rollback()│ +└────┬────┘ └────┬─────┘ + │ │ + ▼ ▼ +Executes: Executes: +COMMIT ROLLBACK +TRANSACTION; TRANSACTION; + │ │ + └───────┬───────┘ + │ + ▼ +(Auto-re-executes BEGIN TRANSACTION; if setAutoCommit remains false) +``` + +1. **Pre-requisite Check**: Calling `setAutoCommit(false)`, `commit()`, or `rollback()` requires `;EnableSession=true` in the connection URL. If disabled or invoked without an active transaction, an exception is thrown by the driver. +2. **Session & Transaction Start**: + - `setAutoCommit(false)` initiates a multi-statement transaction session in BigQuery. +3. **Statement Propagation**: + - All `Statement` or `PreparedStatement` instances created on the connection execute within the scope of the active session. +4. **Commit & Rollback**: + - `commit()` executes `COMMIT TRANSACTION;` to commit changes. + - `rollback()` executes `ROLLBACK TRANSACTION;` to discard changes. + - If `autoCommit` remains `false`, the driver automatically starts the next transaction block. +5. **Connection Close Safety**: + - If an uncommitted transaction is pending when `conn.close()` is invoked, the driver automatically rolls back the transaction to prevent uncommitted changes from persisting. +6. **Isolation Level & Holdability**: + - Isolation level: `Connection.TRANSACTION_SERIALIZABLE` (BigQuery multi-statement snapshot isolation). + - Holdability: `ResultSet.CLOSE_CURSORS_AT_COMMIT`. + +--- + +### High-Throughput Storage Read & Write APIs + +For enterprise data ingestion and analytics extraction, the driver integrates with BigQuery Storage APIs: + +- **Storage Read API (`EnableHighThroughputAPI=true`)**: + Performs high-speed result extraction over gRPC streams using Apache Arrow instead of REST JSON pagination for large ResultSets. +- **Storage Write API (`EnableWriteAPI=true`)**: + Enables high-throughput streaming appends for batch operations executed via `PreparedStatement.executeBatch()`. + +--- + +## 7. Feature Examples & Code Snippets + +### Transactions (Manual Commit & Rollback) +Transactions require `;EnableSession=true` in the connection URL to enable multi-statement sessions in BigQuery. + +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;EnableSession=true;OAuthType=3"; + +try (Connection conn = DriverManager.getConnection(url)) { + conn.setAutoCommit(false); // Enable manual transaction control + + try (PreparedStatement debit = conn.prepareStatement("UPDATE finance.accounts SET balance = balance - ? WHERE account_id = ?"); + PreparedStatement credit = conn.prepareStatement("UPDATE finance.accounts SET balance = balance + ? WHERE account_id = ?")) { + + debit.setDouble(1, 500.00); + debit.setLong(2, 1001L); + debit.executeUpdate(); + + credit.setDouble(1, 500.00); + credit.setLong(2, 2002L); + credit.executeUpdate(); + + conn.commit(); // Commit transaction atomically + } catch (SQLException e) { + conn.rollback(); // Rollback on error + throw e; + } +} +``` + +--- + +### Prepared Statements & Parameter Binding +Use `PreparedStatement` to safely bind parameters including primitive types, decimals (`BigDecimal`), temporal values (`Date`, `Timestamp`), and byte arrays (`byte[]`). + +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;OAuthType=3"; +String sql = "SELECT order_id FROM sales.orders WHERE order_date = ? AND total_amount >= ? AND status_code = ?"; + +try (Connection conn = DriverManager.getConnection(url); + PreparedStatement pstmt = conn.prepareStatement(sql)) { + + pstmt.setDate(1, java.sql.Date.valueOf(LocalDate.of(2026, 7, 22))); + pstmt.setBigDecimal(2, new BigDecimal("12499.99")); + pstmt.setString(3, "COMPLETED"); + + try (ResultSet rs = pstmt.executeQuery()) { + while (rs.next()) { + long orderId = rs.getLong("order_id"); + } + } +} +``` + +--- + +### Callable Statements & Stored Procedures +BigQuery stored procedure outputs are returned as standard result sets (`TableResult`) rather than JDBC `OUT` parameters. + +#### Pattern A: Executing a Stored Procedure Returning a ResultSet +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;OAuthType=3"; + +try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("CALL my_dataset.get_top_customers()")) { + + while (rs.next()) { + String customer = rs.getString("customer_name"); + double spent = rs.getDouble("total_spent"); + } +} +``` + +#### Pattern B: Retrieving Procedure Output Variables via Multi-Statement Script +To retrieve `OUT` parameters from a procedure, execute a procedural script that declares an output variable, calls the procedure, and returns the variable in a final `SELECT` statement: + +```java +String scriptSql = "DECLARE tax_out NUMERIC; " + + "CALL my_dataset.calculate_tax(?, tax_out); " + + "SELECT tax_out AS calculated_tax;"; + +try (Connection conn = DriverManager.getConnection(url); + PreparedStatement pstmt = conn.prepareStatement(scriptSql)) { + + pstmt.setBigDecimal(1, new BigDecimal("1000.00")); + + try (ResultSet rs = pstmt.executeQuery()) { + if (rs.next()) { + BigDecimal taxAmount = rs.getBigDecimal("calculated_tax"); + } + } +} +``` + +--- + +### Batch Ingestion with Storage Write API +Enable `;EnableWriteAPI=true` in the connection URL to stream bulk batches via `executeBatch()`. + +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;EnableWriteAPI=true;OAuthType=3"; +String sql = "INSERT INTO telemetry.sensor_readings (sensor_id, temperature, is_valid) VALUES (?, ?, ?)"; + +try (Connection conn = DriverManager.getConnection(url); + PreparedStatement pstmt = conn.prepareStatement(sql)) { + + for (int i = 0; i < 1000; i++) { + pstmt.setInt(1, 100 + (i % 10)); + pstmt.setDouble(2, 25.5); + pstmt.setBoolean(3, true); + pstmt.addBatch(); + + if (i % 500 == 0) { + pstmt.executeBatch(); // Flushes batch using Storage Write API + } + } + pstmt.executeBatch(); +} +``` + +--- + +### High-Throughput Storage Read API +Enable `;EnableHighThroughputAPI=true` to stream large query result sets over high-speed gRPC streams using Apache Arrow format. + +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;EnableHighThroughputAPI=true;OAuthType=3"; + +try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT user_id, event_name FROM analytics.user_events WHERE event_date >= '2026-01-01'")) { + + while (rs.next()) { + String userId = rs.getString("user_id"); + String event = rs.getString("event_name"); + } +} +``` + +--- + +### Struct & Array Column Operations +Extract nested BigQuery `STRUCT` columns (using `java.sql.Struct`) and `ARRAY` columns (using `java.sql.Array`). + +```java +String query = "SELECT STRUCT('123 Main St' AS street, 'Seattle' AS city) AS address, ['tag1', 'tag2'] AS tags"; + +try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery(query)) { + + if (rs.next()) { + // Read STRUCT attributes + Struct address = (Struct) rs.getObject("address"); + Object[] attrs = address.getAttributes(); // [123 Main St, Seattle] + + // Read ARRAY elements + Array tagsArray = rs.getArray("tags"); + String[] tags = (String[]) tagsArray.getArray(); // [tag1, tag2] + } +} +``` + +--- + +### Service Account Impersonation +Impersonate a service account using Application Default Credentials (ADC) without managing static service account key files. + +> [!NOTE] +> **IAM Roles Required for Impersonation**: +> 1. **Caller Principal (ADC)**: Must have the `roles/iam.serviceAccountTokenCreator` role on the target Service Account. +> 2. **Impersonated Service Account**: Must have `roles/bigquery.jobUser` (to submit query jobs) and `roles/bigquery.dataViewer` (to read target datasets). + +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-project" + + ";OAuthType=3" // Application Default Credentials (ADC) + + ";ServiceAccountImpersonationEmail=analytics-executor@my-project.iam.gserviceaccount.com"; + +try (Connection conn = DriverManager.getConnection(url); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT COUNT(*) FROM analytics.user_events")) { + + if (rs.next()) { + long count = rs.getLong(1); + } +} +``` + +--- + +## 8. High-Throughput & Performance Tuning + +For large dataset extractions and high-concurrency ingestion, tune the following connection options: + +### Storage Read & Write APIs Summary + +The BigQuery JDBC driver supports gRPC-accelerated streaming APIs for both reading large result sets and appending batch records: + +- **Storage Read API (`EnableHighThroughputAPI=true`)**: + Streams query result sets over high-speed gRPC channels using Apache Arrow format instead of standard REST JSON pagination. Automatically evaluates minimum table size and page count thresholds before activating. +- **Storage Write API (`EnableWriteAPI=true`)**: + Streams bulk batch insertions (`PreparedStatement.executeBatch()`) over gRPC append streams instead of concatenated SQL DML statements. + +> [!TIP] +> For complete property matrices, activation criteria, and workload tuning scenario matrices for both Storage Read and Write APIs, see the [Storage APIs Guide](STORAGE_APIS.md). + +--- + +### Additional Performance Tuning Options + +- **Metadata Thread Pooling**: For applications fetching schema catalog metadata across multiple datasets, set `;MetaDataFetchThreadCount=64` to parallelize catalog RPC calls. + +--- + +### Custom Endpoints, Private Service Connect (PSC) & Universe Domains + +For enterprise environments using **VPC Service Controls**, **Private Service Connect (PSC)**, **Regional Endpoints**, or **Google Cloud Dedicated (GCD)** instances, the driver supports full endpoint overriding and universe domain configuration. + +#### 1. Endpoint Overrides (`EndpointOverrides`) + +The `EndpointOverrides` property allows redirecting traffic for individual Google Cloud services to private IP endpoints, regional gateways, or corporate proxy targets. + +**Format**: +``` +EndpointOverrides=BIGQUERY=https://bigquery.googleapis.com,OAUTH2=https://oauth2.googleapis.com/token; +``` + +**Supported Service Keys**: + +| Service Key | Target Service / Auth Flow | Default Endpoint URI | When to Use & Applicable Auth Modes | +| :--- | :--- | :--- | :--- | +| **`BIGQUERY`** | BigQuery Core REST API | `https://bigquery.googleapis.com` | Overrides BigQuery REST query execution endpoints for regional gateways or PSC VIPs (**Applies to all `OAuthType` modes**). | +| **`READ_API`** | Storage API (Read & Write) | `https://bigquerystorage.googleapis.com` | Overrides gRPC stream endpoints for Storage Read (HTAPI) and Storage Write (SWA) (**Applies to all `OAuthType` modes**). | +| **`OAUTH2`** | Google OAuth 2.0 Token Server | `https://oauth2.googleapis.com/token` | Overrides OAuth token exchange endpoints for **Service Account (`OAuthType=0`)**, **User Account (`OAuthType=1`)**, and **Refresh Token (`OAuthType=2`)**. | +| **`STS`** | Security Token Service | `https://sts.googleapis.com` | Overrides Security Token Service (`token_url`) endpoints for **Workload Identity / BYOID (`OAuthType=4`)**. | + +> [!NOTE] +> **Credentials File Endpoint Resolution**: +> Workload credentials files (such as Workload Identity / External Account JSON files or Application Default Credentials) may define their own endpoint URLs (`token_url`, `credential_source`, or `universe_domain`). Connection parameters (such as `EndpointOverrides`, `BYOIDTokenUri`, or `universeDomain`) take precedence when explicitly specified in the JDBC URL, allowing applications to override values configured inside credential files for private routing or custom proxies. + +--- + +#### 2. Private Service Connect URIs (`PrivateServiceConnectUris`) + +For VPC setups using explicit Private Service Connect URI aliases: +``` +jdbc:bigquery://https://bigquery.googleapis.com:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;PrivateServiceConnectUris=BIGQUERY=https://bigquery-privateendpoint.p.googleapis.com +``` + +#### 3. Custom Universe Domains (`universeDomain`) + +To target Google Cloud Dedicated (GCD) or non-default Universe Domain instances, specify `universeDomain`: + +``` +jdbc:bigquery://https://www.my-universe.cloud:443 + ;ProjectId=my-gcp-project + ;OAuthType=0 + ;universeDomain=my-universe.cloud +``` + +--- + +## 9. Framework Integration & Deployment + +### Spring Boot (`application.yml`) +```yaml +spring: + datasource: + url: "jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-gcp-project;DefaultDataset=my_dataset;OAuthType=0;OAuthPvtKeyPath=/path/to/key.json" + driver-class-name: com.google.cloud.bigquery.jdbc.BigQueryDriver + hikari: + maximum-pool-size: 10 + minimum-idle: 2 + idle-timeout: 300000 + connection-test-query: SELECT 1 +``` + +### Key Deployment Guidelines +1. **Driver Class**: `com.google.cloud.bigquery.jdbc.BigQueryDriver` +2. **Connection Pooling**: BigQuery connection objects are lightweight wrappers around thread-safe Google API clients. Standard connection pools like HikariCP can be safely used. +3. **IAM Permissions**: Ensure the principal has `BigQuery Data Viewer` and `BigQuery Job User` roles on the target project and dataset. + +--- + +## 10. Logging, Diagnostics & Troubleshooting + +### Configuring Connection-Level Logging + +The driver includes a built-in logging subsystem built on top of `java.util.logging`. Logging can be configured directly via JDBC URL connection properties or system environment variables: + +#### Connection URL Parameters: +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=0" + + ";LogLevel=5" // Set log level (1=SEVERE to 7=FINEST) + + ";LogPath=/var/log/bigquery-jdbc"; // Output directory for log files +``` + +#### Environment Variables for Logging: + +Logging can also be configured globally across all driver connections using environment variables: + +| Environment Variable | Equivalent Connection Property | Description | Example | +| :--- | :--- | :--- | :--- | +| **`BIGQUERY_JDBC_LOG_LEVEL`** | `LogLevel` | Global default log verbosity level (`0`–`7` or `OFF`–`FINEST`). Overridden by `LogLevel` URL parameter. | `export BIGQUERY_JDBC_LOG_LEVEL=5` | +| **`BIGQUERY_JDBC_LOG_PATH`** | `LogPath` | Global output directory for driver log files. Overridden by `LogPath` URL parameter. | `export BIGQUERY_JDBC_LOG_PATH=/var/log/bigquery-jdbc` | + +#### Log Level Reference: + +| Integer Value | String Constant | `java.util.logging` Level | Description | +| :---: | :---: | :---: | :--- | +| `0` | `OFF` | `Level.OFF` | Logging completely disabled (default). | +| `1` | `SEVERE` | `Level.SEVERE` | Critical errors, connection failures, unrecoverable exceptions. | +| `2` | `WARNING` | `Level.WARNING` | Non-fatal warnings, API fallbacks (e.g., Read API permission denied fallback to REST). | +| `3` | `INFO` | `Level.INFO` | High-level driver events, connection establishment, job submission. | +| `4` | `CONFIG` | `Level.CONFIG` | Driver configuration initialization and property resolution details. | +| `5` | `FINE` | `Level.FINE` | SQL query execution statements, parameter binding, row counts. | +| `6` | `FINER` | `Level.FINER` | Internal method entry (`++enter++`) and exit (`++exit++`) tracing. | +| `7` | `FINEST` | `Level.FINEST` | Maximum diagnostic verbosity, raw batch payloads, Arrow stream packet details. | + +#### Log File Naming & Format + +Log files are generated inside the specified `LogPath` directory with per-connection log isolation. + +**Standard Log Record Format**: +```text +2026-07-22 14:25:10.123 [conn-bq-8f3a] INFO 12345 --- [ main ] com.google.cloud.bigquery.jdbc.BigQueryStatement executeQuery : Executing query: SELECT COUNT(*) FROM `my-project.analytics.orders` +``` + +- **Timestamp**: `yyyy-MM-dd HH:mm:ss.SSS` in local time. +- **Connection ID (`MDC`)**: Unique identifier `[conn-bq-XXXX]` isolating statements per connection context. +- **Level**: Padded log level (`INFO`, `FINE`, `WARNING`, etc.). +- **Process ID & Thread**: OS Process ID and centered Thread Name. +- **Class & Method**: Fully qualified source class name and method name originating the log message. + +--- + +### OpenTelemetry & Cloud Observability + +The BigQuery JDBC driver features native OpenTelemetry (OTel) instrumentation for distributed tracing and Cloud Logging correlation across SQL query executions, prepared statement batching, and gRPC storage streams. + +#### Telemetry Connection Properties: + +| Connection Property | Type | Default | Description | +| :--- | :---: | :---: | :--- | +| **`enableGcpTraceExporter`** | `Boolean` | `false` | Enables automatic exporting of OpenTelemetry trace spans directly to Google Cloud Trace over OTLP gRPC (`https://telemetry.googleapis.com:443`). | +| **`enableGcpLogExporter`** | `Boolean` | `false` | Enables exporting driver log events directly to Google Cloud Logging with correlated `traceId` and `spanId` context. | +| **`useGlobalOpenTelemetry`** | `Boolean` | `false` | Directs the driver to use the application's global `GlobalOpenTelemetry` instance registered in the JVM context. | +| **`gcpTelemetryProjectId`** | `String` | `null` | Specifies a custom GCP Project ID for telemetry export when different from the main query connection project catalog. | +| **`gcpTelemetryCredentials`** | `String` | `null` | Path or raw JSON string for dedicated service account credentials used for telemetry export. | + +#### OpenTelemetry Span Attributes: + +Spans generated by the driver automatically capture standard OpenTelemetry database semantic conventions: +- `db.system`: Always set to `bigquery`. +- `db.connection_id`: Unique connection identifier (e.g., `conn-bq-8f3a`). +- `db.statement`: Executed SQL statement text (safely truncated at 32KB to avoid trace payload overflow). +- `db.application`: Application client name (defaults to `Google-BigQuery-JDBC-Driver` or custom partner token). + +#### Enabling GCP Cloud Trace & Cloud Logging via JDBC URL: +```java +String url = "jdbc:bigquery://https://bigquery.googleapis.com:443" + + ";ProjectId=my-gcp-project" + + ";OAuthType=0" + + ";OAuthPvtKeyPath=/var/secrets/bq-key.json" + + ";enableGcpTraceExporter=true" + + ";enableGcpLogExporter=true"; + +Connection conn = DriverManager.getConnection(url); +``` + +#### Programmatic Custom OpenTelemetry Injection: +Applications with an existing `OpenTelemetry` SDK instance can pass it programmatically via connection `Properties`: +```java +OpenTelemetry customOtel = OpenTelemetrySdk.builder()...build(); + +Properties props = new Properties(); +props.put("customOpenTelemetry", customOtel); + +Connection conn = DriverManager.getConnection("jdbc:bigquery://https://bigquery.googleapis.com:443;ProjectId=my-project;OAuthType=3", props); +``` + +--- + +### Exception Hierarchy + +All driver exceptions inherit from `java.sql.SQLException`: + +- **`BigQueryJdbcException`**: Base exception class wrapping underlying Google Cloud API and network RPC errors. +- **`BigQueryJdbcSqlSyntaxErrorException`**: Thrown when standard SQL query validation or parsing fails on the server. +- **`BigQueryConversionException`**: Thrown on data type coercion failures or unparseable target representations. +- **`BigQueryJdbcSqlFeatureNotSupportedException`**: Thrown when an unsupported JDBC API method is invoked. + +--- + +## 11. Official Documentation References + +- 🌐 [Official Google Cloud BigQuery JDBC Driver Overview](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery) +- ⚙️ [Running Queries with the JDBC Driver](https://cloud.google.com/bigquery/docs/jdbc-for-bigquery#run_queries_with_the_driver) +- 📊 [BigQuery Quotas & Limits](https://cloud.google.com/bigquery/quotas)