From c139eecb5c4ddc632a07835a5866c272246def77 Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Sun, 6 Sep 2026 15:01:54 -0700 Subject: [PATCH 1/8] Updated integration-jdbc with error handling and some additional information --- docs/integration-jdbc.md | 185 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) diff --git a/docs/integration-jdbc.md b/docs/integration-jdbc.md index 4d35fa885..f24751f8e 100644 --- a/docs/integration-jdbc.md +++ b/docs/integration-jdbc.md @@ -28,6 +28,49 @@ Two distributions are published under the same artifact: --- +## Development Environment + +### Java Version + +The ClickHouse JDBC driver requires **Java 8 or newer**. All mainline development and testing are validated on recent LTS versions (Java 8, 11, 17, and 21 as available). We recommend using up-to-date LTS releases, as feature coverage, TLS support, and performance are best with newer runtimes. + +### Local ClickHouse for Development + +For local development and rapid prototyping, you can run ClickHouse in a container with minimal setup: + +```sh +docker run --rm -d -p 8123:8123 --name clickhouse-server clickhouse/clickhouse-server:latest +``` + +This launches a fresh ClickHouse server, with the HTTP API available at `localhost:8123`. The default database is `default` and no password is required for the default user (`default`). This is ideal for smoke tests, schema exploration, and validating basic integration. + +You can also use any recent [official ClickHouse binary](https://clickhouse.com/docs/en/getting-started/install/) or use Docker Compose for more complex setups. + +> **Note:** Local environments are best for initial integration, experiments, and CI pipelines — not for realistic scale, performance, or operational requirements. + +### ClickHouse Cloud (Recommended for Production-like Integration) + +While local ClickHouse is excellent for early development, **we strongly recommend using [ClickHouse Cloud](https://clickhouse.com/cloud/)** for end-to-end integration and before considering your application production ready. + +ClickHouse Cloud environments mirror real-world production settings: + +- Managed authentication, scaling, secure TLS endpoints +- Production-like network performance and stability +- Long-running, stateful databases without the churn or "works on my machine" surprises + +Using ClickHouse Cloud during integration helps uncover configuration or authentication nuances, avoid brittle local-only assumptions, and validates your app against true cloud operations. All features described in this guide are supported equally, and connection examples are identical — just provide your cloud hostname, port, and TLS-enabled URL in the JDBC connection string. + +> **Best practice:** Validate your integration against ClickHouse Cloud regularly, especially before feature launches and release cutoffs. + +### Summary + +- **Local ClickHouse**: Fast, simple; best for initial trials and interactive development. +- **ClickHouse Cloud**: Closest to real-world production, surfaces integration and operational issues early, and ensures all JDBC (and client) features work as expected. + +For more, see the [ClickHouse Cloud docs](https://clickhouse.com/docs/en/cloud/) and the [official integrations guide](https://clickhouse.com/docs/en/integrations/java/). + + + ## Integration path at a glance Work through these steps in order. The "Common Pitfalls" notes tell you what breaks if you skip one. @@ -48,6 +91,14 @@ Work through these steps in order. The "Common Pitfalls" notes tell you what bre **Goal:** decide the lifecycle of a JDBC `Connection` and how you pool connections. +This is important to mention about user provided configuration when system is transparent about connection properties. Build an adoption layer between connector and JDBC driver properties to +decouple your system from changes in JDBC driver. Having such adoption layer helps also with migration to new version and resolving compatibility issues. Always separate your application or +connector configuration from JDBC driver. It may be very problematic in future to resolve name collisions and handle upgrades. It is very important to document how JDBC driver +can be configured by user and where to find driver configuration references. + +**Security Note** Always validate input for configuration. JDBC verifies it own configuration but some values can be semantically invalid. For example, do not let to reconfigure client name. + + ### What the JDBC objects are | Object | Class | Role | @@ -391,6 +442,8 @@ public boolean checkConnectionHealth(Connection conn, int timeoutSeconds) throws } ``` + + ### Common Pitfalls @@ -603,6 +656,41 @@ public Connection createRowBinaryInsertConnection() throws SQLException { | HTTP compression | `client.use_http_compression=true` | Content-Encoding on the HTTP layer | | Async insert | `async_insert=1` (server setting) | Server-side insert buffering | +When using the RowBinary writer for inserts, you can combine it with async inserts (`async_insert=1` and optionally `wait_async_insert=1`) to achieve very high-throughput, low-latency ingestion. The RowBinary format is efficient for bulk data transfer, while async insert lets the server buffer and process inserts in the background, enabling the client to proceed without blocking on disk writes. + +**How this helps:** +- **Lower client-side latency:** The JDBC driver streams RowBinary-encoded data directly to the server; with async insert enabled, the server acknowledges receipt quickly, decoupling the client from storage latency. +- **Higher throughput:** The combination is ideal for ingest-heavy workloads—batches or streaming ETL—because the client isn't forced to wait for each insert to be durably stored before proceeding. +- **Optimized network and server utilization:** RowBinary reduces payload size and overhead, async insert handles bursty/high-rate loads by buffering, and both together reduce insert round-trip costs. + +**Example: Batched, streaming async insert using RowBinary** + +```java +public void asyncRowBinaryInsert(Connection conn, List events) throws SQLException { + if (events.isEmpty()) { + return; + } + try (PreparedStatement ps = conn.prepareStatement( + "INSERT INTO events (id, name, created_at) " + + "SETTINGS async_insert=1, wait_async_insert=1 VALUES (?, ?, ?)")) { + for (Event event : events) { + ps.setLong(1, event.getId()); + ps.setString(2, event.getName()); + ps.setObject(3, event.getCreatedAt()); + ps.addBatch(); + } + ps.executeBatch(); + } +} +``` + +This leverages the JDBC driver's RowBinary streaming (when `beta.row_binary_for_simple_insert=true` is set) *plus* async server-side processing, so your application can ingest large volumes with minimal response-time overhead. + +> **Tip:** For the best results on very high-volume pipelines, adjust both server async insert settings (see [ClickHouse docs](https://clickhouse.com/docs/en/operations/settings/settings#async_insert)) and the client's connection/HTTP pool size and streaming/chunking properties. +> +> **Caution:** Remember that with async insert, success response means "data accepted for processing" — not yet "written." Use `wait_async_insert=1` to wait for commit, and deduplication tokens if retrying inserts to avoid duplicates. + + ### Idempotency — deduplication token JDBC does not expose `insert_deduplication_token` as a first-class API. Three ways to use it: @@ -623,6 +711,103 @@ public void insertWithPerStatementDedup(Connection conn, String token) throws SQ See [integration-client.md — deduplication token](integration-client.md#idempotency--deduplication-token) for semantics and requirements. + +**Async inserts** +ClickHouse supports "async inserts", which allow inserts to buffer on the server side (`async_insert=1`), returning control to the client before the data is fully written to disk. **Caveat:** The insert operation may report success even though the data is not yet persisted, and you *cannot* reliably check row count in statistics to confirm completion. + +**To ensure the insert is actually complete and data is committed, the only supported option is to use `wait_async_insert=1` in the SQL SETTINGS clause.** This makes the server wait for the buffered insert to finish before acknowledging the operation: + +```java +public void safeAsyncInsert(Connection conn) throws SQLException { + try (Statement stmt = conn.createStatement()) { + stmt.executeUpdate( + "INSERT INTO events SETTINGS async_insert=1, wait_async_insert=1 VALUES (1, 'a')"); + } +} +``` + +> **Note:** Inserts using async mode may interact with deduplication. If your workflow is retry-prone or idempotency is required, always set an explicit deduplication token (see prior section) in `insert_deduplication_token`. This avoids duplicate inserts if the first async INSERT is still being processed when a retry occurs. + +**Summary:** +- Never rely on server statistics/row count to determine async insert completion. +- Use `wait_async_insert=1` to guarantee data is committed before continuing. +- For deduplication, combine `insert_deduplication_token` with async inserts as needed (`SETTINGS async_insert=1, wait_async_insert=1, insert_deduplication_token = 'your_token'`). + +### Failure Handling + +#### Retriable Operations + +Most operations retried by the JDBC driver are **network-level failures** — for example, timeouts, connection drops, and intermediate transport errors. +The underlying Java Client (`client-v2`) already implements robust automatic retry logic for such network and transient issues, so in almost all cases **explicit retry logic in application code is unnecessary**. + +Some server-side errors, however, may also be transient and suitable for retry, depending on the use case: + +- **Execution timeout errors**: If a query fails due to a timeout (e.g., server-side `max_execution_time` or too aggressive timeout settings), the operation may be retried with adjusted settings or after a backoff delay. +- **Schema mismatch errors** (see below): When the table schema changes between requests (causing errors such as "UNABLE TO READ ALL DATA"), retrying the operation after refreshing the schema can resolve the problem. +- **Server overloads or resource spikes**: Very occasionally, a server might return a retryable error indicating temporary overload; review the exception chain for details. + +> **Best Practice:** +> - Trust the client's network retry strategy for network or I/O failures. +> - For retrying queries after **server-side errors**, examine the exception code and context to determine whether a retry makes sense — especially for timeouts, schema mismatches, or temporary overloads. +> - Avoid blanket retries of all `SQLException`s, since some are non-retriable (invalid SQL, authentication failure, etc.), and indiscriminate retrying can produce duplicate writes if idempotency is not ensured. + +See below for how to detect and handle schema mismatch errors specifically. + +#### Schema mismatch + +When working with a ClickHouse cluster, you may encounter errors if the table schema changes unexpectedly between queries or ingestions. While most JDBC applications escape this because generated SQL typically matches the current schema, dynamic ingestion workflows or custom binary writers (like RowBinary insert streams) are particularly sensitive. + +**Typical error:** +A schema change — such as adding, removing, or altering a column — can result in ingestion failures with errors like: + +- **ClickHouse Error Code 33 — "UNABLE TO READ ALL DATA"**: + This error appears when the data sent from the client does not match the expected table schema on the server (column count, order, or types are incompatible). + - For example, if you use a RowBinary writer and the schema is out of sync, ClickHouse cannot parse the incoming data block, resulting in this error. + - Code 33 may also be reported as a generic SQL exception by the JDBC layer, and the actual error code is only accessible if the exception's cause is a `ServerException` from the ClickHouse client. + +**Detection & Handling:** +- **Detection:** + Error Code 33 signals that ClickHouse could not read all required data — most often due to a schema drift or column mismatch between the client and ClickHouse table. + However, in the JDBC pathway, the underlying ClickHouse error code is accessible *only if* the root cause of the `SQLException` is a `com.clickhouse.client.ServerException`. You should check if `SQLException.getCause()` is a `ServerException` and inspect the code/message. +- **Best practices for handling:** + 1. **Catch and examine SQL exceptions.** + - If `ex.getCause()` is a `ServerException` with code 33 (or message indicating unable to read all data), assume a schema mismatch. + 2. **Proactively refresh schema.** + - Fetch the latest table schema using JDBC's `DatabaseMetaData` or a `DESCRIBE TABLE` query. + 3. **Regenerate SQL or update your ingestion/mapping logic.** + - Make sure your SQL, `PreparedStatement`, or RowBinary column mapping exactly matches the current table schema. + 4. **Retry the operation.** + +**Example: Handling Error Code 33 (ServerException only)** + +```java +import com.clickhouse.client.ServerException; + +try { + // ... ingestion code ... +} catch (SQLException ex) { + Throwable cause = ex.getCause(); + if (cause instanceof ServerException) { + ServerException serverEx = (ServerException) cause; + // ClickHouse code 33: UNABLE TO READ ALL DATA (schema mismatch) + if (serverEx.getErrorCode() == 33 ) { + // Possible table schema mismatch + // 1. Reload schema via DatabaseMetaData or DESCRIBE TABLE + // 2. Update SQL or RowBinary mapping as needed + // 3. Retry with updated logic + } else { + throw ex; // not a schema mismatch + } + } else { + throw ex; // Cannot determine server error code + } +} +``` + +> **Note:** Always ensure your application's data shape matches the target table exactly when using RowBinary or similar binary ingestion APIs. If your code holds a schema cache, invalidate it and fetch the up-to-date schema on a Code 33 error before retrying. + +See [ClickHouse error codes documentation](https://clickhouse.com/docs/en/operations/error-codes/) for more on Code 33 and related scenarios. + ### Best practices From dd2cf91ee8f2972edc034311a092577d2785cb95 Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Sat, 26 Sep 2026 00:55:22 -0700 Subject: [PATCH 2/8] updated integration-jdbc: corrections, restructured some sections. more todo --- docs/integration-index.md | 43 ++++++++ docs/integration-jdbc.md | 222 +++++++++++++++++--------------------- 2 files changed, 142 insertions(+), 123 deletions(-) diff --git a/docs/integration-index.md b/docs/integration-index.md index 047476b53..adc63ec8e 100644 --- a/docs/integration-index.md +++ b/docs/integration-index.md @@ -88,6 +88,49 @@ The JDBC driver is a thin adaptation layer that uses the Java Client underneath --- +## Development Environment + +### Java Version + +The ClickHouse JDBC driver and Client require **Java 8 or newer**. All mainstream development and testing are validated on recent LTS versions (Java 8, 11, 17, and 21 as available). We recommend using up-to-date LTS releases, as feature coverage, TLS support, and performance are best with newer runtimes. + +### Local ClickHouse for Development + +For local development and rapid prototyping, you can run ClickHouse in a container with minimal setup: + +```sh +docker run --rm -d -p 8123:8123 --name clickhouse-server clickhouse/clickhouse-server:latest +``` + +This launches a fresh ClickHouse server, with the HTTP API available at `localhost:8123`. The default database is `default` and no password is required for the default user (`default`). This is ideal for smoke tests, schema exploration, and validating basic integration. + +You can also use any recent [official ClickHouse binary](https://clickhouse.com/docs/en/getting-started/install/) or use Docker Compose for more complex setups. + +> **Note:** Local environments are best for initial integration, experiments, and CI pipelines — not for realistic scale, performance, or operational requirements. + +### ClickHouse Cloud (Recommended for Production-like Integration) + +While local ClickHouse is excellent for early development, **we strongly recommend using [ClickHouse Cloud](https://clickhouse.com/cloud/)** for end-to-end integration and before considering your application production ready. + +ClickHouse Cloud environments mirror real-world production settings: + +- Managed authentication, scaling, secure TLS endpoints +- Production-like network performance and stability +- Long-running, stateful databases without the churn or "works on my machine" surprises + +Using ClickHouse Cloud during integration helps uncover configuration or authentication nuances, avoid brittle local-only assumptions, and validates your app against true cloud operations. All features described in this guide are supported equally, and connection examples are identical — just provide your cloud hostname, port, and TLS-enabled URL in the JDBC connection string. + +> **Best practice:** Validate your integration against ClickHouse Cloud regularly, especially before feature launches and release cutoffs. + +### Summary + +- **Local ClickHouse**: Fast, simple; best for initial trials and interactive development. +- **ClickHouse Cloud**: Closest to real-world production, surfaces integration and operational issues early, and ensures all JDBC (and client) features work as expected. + +For more, see the [ClickHouse Cloud docs](https://clickhouse.com/docs/en/cloud/) and the [official integrations guide](https://clickhouse.com/docs/en/integrations/java/). + +--- + ## Choosing Between the Java Client and JDBC Making the right choice between the client and driver is crucial for application architecture and long-term maintainability. diff --git a/docs/integration-jdbc.md b/docs/integration-jdbc.md index ae7d848de..34b95b813 100644 --- a/docs/integration-jdbc.md +++ b/docs/integration-jdbc.md @@ -1,89 +1,52 @@ # ClickHouse JDBC Integration Guide -This guide is a **step-by-step integration path** for the **JDBC Driver V2** (`jdbc-v2`, published as `com.clickhouse:clickhouse-jdbc`). It is written to be used as context for building an application or a downstream integration spec: each step states the decisions you must make, how to configure them, and the common pitfalls to avoid. +## Preface -**Prerequisites:** Read [integration-common.md](integration-common.md) to understand the JDBC trade-offs before committing to this path. +This document describes usage of **JDBC Driver V2** (`jdbc-v2`, published as `com.clickhouse:clickhouse-jdbc`). Here you will find information about different aspects of ClickHouse JDBC Driver integration (authentication, configuration, operations, etc) and about internal logic and mechanics of the driver. Treat this document as a guide of how to use driver for different use cases. Library documentation, however, stays single source of truth. -> **Architecture in one line.** Every JDBC `Connection` wraps a `client-v2` [`Client`](../client-v2/src/main/java/com/clickhouse/client/api/Client.java) internally. JDBC is not a separate protocol stack — it is a `java.sql.*` façade over the Java Client. -> -> ``` -> Application → ConnectionImpl → Client → HTTP pool → ClickHouse -> ``` +**Important:** Please read [integration-index.md](integration-index.md) to get over overview of `clickhouse-java` project as library and its components. -> **Configuration philosophy.** This guide names only the properties relevant to each step. The exhaustive lists live in [`DriverProperties`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/DriverProperties.java), [`ClientConfigProperties`](../client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java), and the official docs. Configuration splits into two groups: -> - **Init configuration** — set once via the JDBC URL or `Properties`: endpoint, connection pool size, authentication, TLS. Covered in Steps 1–3. -> - **Operation configuration** — set per statement or as connection defaults: fetch size, timeouts, batch behavior, dedup tokens. Covered in Steps 4–6. -> -> Property routing (see [`DriverProperties`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/DriverProperties.java)): if a property is a JDBC-specific driver property it is handled by the driver; **all other properties are forwarded to `ClientConfigProperties`**. +### Architecture in one line -## Artifacts +Every JDBC `Connection` wraps a `client-v2` [`Client`](../client-v2/src/main/java/com/clickhouse/client/api/Client.java) internally. JDBC is not a separate protocol stack — it is a `java.sql.*` facade over the Java Client. -The driver is published to Maven Central as **`com.clickhouse:clickhouse-jdbc`**. - -Two distributions are published under the same artifact: - -- **Standard artifact** (default, no classifier) — the driver together with its dependencies declared as ordinary transitive Maven dependencies. Recommended for managed builds in which the application controls the dependency tree. -- **Shaded artifact** (`all` classifier) — a single self-contained archive that bundles and **relocates** most third-party dependencies. Recommended when transitive dependencies cannot be managed. - ---- - -## Development Environment - -### Java Version - -The ClickHouse JDBC driver requires **Java 8 or newer**. All mainline development and testing are validated on recent LTS versions (Java 8, 11, 17, and 21 as available). We recommend using up-to-date LTS releases, as feature coverage, TLS support, and performance are best with newer runtimes. - -### Local ClickHouse for Development - -For local development and rapid prototyping, you can run ClickHouse in a container with minimal setup: - -```sh -docker run --rm -d -p 8123:8123 --name clickhouse-server clickhouse/clickhouse-server:latest +``` + Application -> ConnectionImpl -> (Client + HTTP pool) -> ClickHouse ``` -This launches a fresh ClickHouse server, with the HTTP API available at `localhost:8123`. The default database is `default` and no password is required for the default user (`default`). This is ideal for smoke tests, schema exploration, and validating basic integration. - -You can also use any recent [official ClickHouse binary](https://clickhouse.com/docs/en/getting-started/install/) or use Docker Compose for more complex setups. - -> **Note:** Local environments are best for initial integration, experiments, and CI pipelines — not for realistic scale, performance, or operational requirements. - -### ClickHouse Cloud (Recommended for Production-like Integration) - -While local ClickHouse is excellent for early development, **we strongly recommend using [ClickHouse Cloud](https://clickhouse.com/cloud/)** for end-to-end integration and before considering your application production ready. - -ClickHouse Cloud environments mirror real-world production settings: - -- Managed authentication, scaling, secure TLS endpoints -- Production-like network performance and stability -- Long-running, stateful databases without the churn or "works on my machine" surprises +### Configuration References -Using ClickHouse Cloud during integration helps uncover configuration or authentication nuances, avoid brittle local-only assumptions, and validates your app against true cloud operations. All features described in this guide are supported equally, and connection examples are identical — just provide your cloud hostname, port, and TLS-enabled URL in the JDBC connection string. +This guide names only the properties relevant to each step. The exhaustive lists live in [`DriverProperties`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/DriverProperties.java), [`ClientConfigProperties`](../client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java), and the official docs. Configuration splits into two groups: +- **Init configuration** — set once via the JDBC URL or `Properties`: endpoint, connection pool size, authentication, TLS. Covered in Steps 1–3. +- **Operation configuration** — set per statement or as connection defaults: fetch size, timeouts, batch behavior, dedup tokens. Covered in Steps 4–6. -> **Best practice:** Validate your integration against ClickHouse Cloud regularly, especially before feature launches and release cutoffs. +- Property routing (see [`DriverProperties`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/DriverProperties.java)): if a property is a JDBC-specific driver property it is handled by the driver; **all other properties are forwarded to `ClientConfigProperties`**. -### Summary +### Maven Artifacts -- **Local ClickHouse**: Fast, simple; best for initial trials and interactive development. -- **ClickHouse Cloud**: Closest to real-world production, surfaces integration and operational issues early, and ensures all JDBC (and client) features work as expected. +The driver is published to Maven Central as [**`com.clickhouse:clickhouse-jdbc`**](https://mvnrepository.com/artifact/com.clickhouse/clickhouse-jdbc). -For more, see the [ClickHouse Cloud docs](https://clickhouse.com/docs/en/cloud/) and the [official integrations guide](https://clickhouse.com/docs/en/integrations/java/). +Two distributions are published under the same artifact: +- **Standard artifact** (default, no classifier) — the driver together with its dependencies declared as ordinary transitive Maven dependencies. Recommended for managed builds in which the application controls the dependency tree. +- **Shaded artifact** (`all` classifier) — a single self-contained archive that bundles and **relocates** most third-party dependencies. Recommended when transitive dependencies cannot be managed. +--- ## Integration path at a glance -Work through these steps in order. The "Common Pitfalls" notes tell you what breaks if you skip one. +Integration can be done in several step. Please read through all of them to get full information about library capabilities and specific behavior. -| # | Milestone | Core decision | -|---|-----------|---------------| -| 1 | [Instantiation strategy](#step-1--instantiation-strategy) | Connection lifecycle, pooling, and workload identification | -| 2 | [Authentication](#step-2--authentication) | Which auth mechanism and how to configure it via URL/Properties | +| # | Step | Core decision | +|---|----------------------------------------------------------------------------------|---------------| +| 1 | [Instantiation strategy](#step-1--instantiation-strategy) | Connection lifecycle, pooling, and workload identification | +| 2 | [Authentication](#step-2--authentication) | Which auth mechanism and how to configure it via URL/Properties | | 3 | [Transport & connectivity](#step-3--transport--connectivity-tls-proxies-timeouts) | TLS/mTLS, proxies, timeouts, health checks | -| 4 | [Formats under the hood](#step-4--formats-under-the-hood) | What the driver does internally; when JDBC is not enough | -| 5 | [Read operations & tuning](#step-5--read-operations--tuning) | `ResultSet` streaming; type mapping; heavy-read tuning | -| 6 | [Write operations & tuning](#step-6--write-operations--tuning) | Batch vs RowBinary beta; heavy-ingest tuning; idempotency | -| 7 | [Metadata & schema discovery](#step-7--metadata--schema-discovery) | `DatabaseMetaData`, `ResultSetMetaData`, type mapping | +| 4 | [Formats under the hood](#step-4--formats-under-the-hood) | What the driver does internally; when JDBC is not enough | +| 5 | [Read operations & tuning](#step-5--read-operations--tuning) | `ResultSet` streaming; type mapping; heavy-read tuning | +| 6 | [Write operations & tuning](#step-6--write-operations--tuning) | Batch vs RowBinary beta; heavy-ingest tuning; idempotency | +| 7 | [Metadata & schema discovery](#step-7--metadata--schema-discovery) | `DatabaseMetaData`, `ResultSetMetaData`, type mapping | --- @@ -91,14 +54,6 @@ Work through these steps in order. The "Common Pitfalls" notes tell you what bre **Goal:** decide the lifecycle of a JDBC `Connection` and how you pool connections. -This is important to mention about user provided configuration when system is transparent about connection properties. Build an adoption layer between connector and JDBC driver properties to -decouple your system from changes in JDBC driver. Having such adoption layer helps also with migration to new version and resolving compatibility issues. Always separate your application or -connector configuration from JDBC driver. It may be very problematic in future to resolve name collisions and handle upgrades. It is very important to document how JDBC driver -can be configured by user and where to find driver configuration references. - -**Security Note** Always validate input for configuration. JDBC verifies it own configuration but some values can be semantically invalid. For example, do not let to reconfigure client name. - - ### What the JDBC objects are | Object | Class | Role | @@ -110,17 +65,11 @@ can be configured by user and where to find driver configuration references. | `ResultSet` | [`ResultSetImpl`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/ResultSetImpl.java) | Row-by-row streaming of query results | | `Driver` | [`Driver`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/Driver.java) | Registers `jdbc:clickhouse:` and `jdbc:ch:` URLs | -### Decisions +### Configuration -| Question | Recommended answer | -|----------|--------------------| -| How many connections? | **One per concurrent thread of work**, obtained from a pool. | -| Short- or long-lived? | Let a **connection pool** (HikariCP, DBCP, container-managed) manage lifetime; borrow and return. | -| Thread-safe? | **No** — `Connection` is *not* thread-safe. **CONSTRAINT:** Never share one across threads. | -| Own pool needed? | **Yes** — **CONSTRAINT:** Use a standard JDBC connection pool. Each `Connection` still owns an HTTP pool via its internal `Client`. | - -### JDBC URL format +JDBC interactions start with creating a `Connection` object via `Driver.connect()` and there are only two ways to configure connection and other components: pass a `Properties` or add to connection URL (see format below). +**JDBC URL format**: ``` jdbc:clickhouse://[host][:port][/[path/]database][?param=value&...] jdbc:clickhouse:https://host:8443/mydb?ssl=true @@ -139,6 +88,36 @@ public Connection createConnection() throws SQLException { Prefer `Properties` over embedding credentials in the URL. URL/Properties parsing is handled by [`JdbcConfiguration`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/JdbcConfiguration.java); programmatic setup is available via [`DataSourceImpl`](../jdbc-v2/src/main/java/com/clickhouse/jdbc/DataSourceImpl.java). +#### User configuration + +When application expects external user to provide connection configuration then building an adoption layer between application and JDBC driver properties is needed to decouple your system from changes in JDBC driver. Having such layer helps also with migration to new version and resolving compatibility issues. Always separate your application or connector configuration from JDBC driver. It may be very problematic in future to resolve name collisions and handle upgrades. It is very important to document how JDBC driver can be configured by user and where to find driver configuration references. + +**Security Note** Always validate input for configuration. JDBC verifies it own configuration but some values can be semantically invalid. For example, do not let to reconfigure client name. + +### Server and Client Settings + +Both forward through URL/Properties: + +```java +public Connection createConfiguredConnection() throws SQLException { + Properties props = new Properties(); + props.setProperty("user", "default"); + props.setProperty(com.clickhouse.jdbc.DriverProperties.serverSetting("max_execution_time"), "60"); // server setting + props.setProperty("max_open_connections", "20"); // client setting + return DriverManager.getConnection( + "jdbc:clickhouse://localhost:8123/default", props); +} +``` + +Client expect that server settings has prefix `clickhouse_setting_` in name and +handles them separately. Client (therefore JDBC Driver) will throw an exception then unknown parameter is passed. + +Per-statement server settings: +- Available via `Statement` method like `Statement.setQueryTimeout(...)`. +- Can be in SQL `SETTINGS` clause (makes SQL not portable). +- Can be set by unwrapping `Statement` to library class and using `com.clickhouse.jdbc.StatementImpl.getLocalSettings`. + + ### Init configuration — pool sizing and threading Because each `Connection` wraps a `Client` with its own HTTP pool, the client-level `max_open_connections` (default 10) forwards through. Two layers of pooling exist: @@ -148,7 +127,7 @@ Because each `Connection` wraps a `Client` with its own HTTP pool, the client-le ### Workload identification & client name -In production environments, a single ClickHouse cluster is often shared across diverse workloads: user-facing web services, streaming ingestion pipelines, ETL batch jobs, BI reporting dashboards (e.g., Superset, Tableau, Grafana), and ad-hoc analytics. When queries fail, time out, or consume excessive memory (`MEMORY_LIMIT_EXCEEDED`), identifying the originating application or workload is essential for fast troubleshooting, root-cause analysis, and resource attribution. +In production environments, a single ClickHouse cluster is often shared across diverse workloads: user-facing web services, streaming ingestion pipelines, ETL batch jobs, BI reporting dashboards (e.g., Superset, Tableau, Grafana), and ad-hoc analytics. When queries fail, time out, or consume excessive memory (`MEMORY_LIMIT_EXCEEDED`), identifying the originating application or workload is essential for fast troubleshooting, root-cause analysis, and resource attribution. Besides it gives very good statistics about usage. #### Setting client name @@ -312,6 +291,16 @@ LIMIT 100; See [integration-testing.md — Connecting](integration-testing.md#connecting) for tests that verify pool sizing (`max_open_connections`) and connection release under load. + +### Decisions + +| Question | Recommended answer | +|----------|--------------------| +| How many connections? | **One per concurrent thread of work**, obtained from a pool. | +| Short- or long-lived? | Let a **connection pool** (HikariCP, DBCP, container-managed) manage lifetime; borrow and return. | +| Thread-safe? | **No** — `Connection` is *not* thread-safe. **CONSTRAINT:** Never share one across threads. | +| Own pool needed? | **Yes** — **CONSTRAINT:** Use a standard JDBC connection pool. Each `Connection` still owns an HTTP pool via its internal `Client`. | + --- ## Step 2 — Authentication @@ -390,14 +379,6 @@ public Connection createCustomHeadersConnection() throws SQLException { } ``` -### Decisions - -| Question | Guidance | -|----------|----------| -| Which mechanism? | Password for most deployments; `access_token`/`bearer_token` for gateway-fronted or cloud setups; mTLS for certificate-based zero-trust; custom headers for API gateways. | -| Credentials in URL or Properties? | **Properties** — keeps secrets out of URLs and logs. | -| Behind an auth proxy? | Prefer token auth (Option B) or custom headers via `http_header_` (Option D). | - ### Common Pitfalls @@ -408,6 +389,15 @@ public Connection createCustomHeadersConnection() throws SQLException { See [integration-testing.md — Configuration](integration-testing.md#configuration) for what to verify when testing authentication end-to-end (boundary/invalid values, certificate acceptance). + +### Decisions + +| Question | Guidance | +|----------|----------| +| Which mechanism? | Password for most deployments; `access_token`/`bearer_token` for gateway-fronted or cloud setups; mTLS for certificate-based zero-trust; custom headers for API gateways. | +| Credentials in URL or Properties? | **Properties** — keeps secrets out of URLs and logs. | +| Behind an auth proxy? | Prefer token auth (Option B) or custom headers via `http_header_` (Option D). | + --- ## Step 3 — Transport & connectivity (TLS, proxies, timeouts) @@ -416,33 +406,16 @@ See [integration-testing.md — Configuration](integration-testing.md#configurat ### TLS / mTLS / proxies -| Scenario | Property / URL parameter | -|----------|--------------------------| -| Enable HTTPS | `ssl=true` + port 8443 | -| Self-signed server cert | `sslrootcert=/path/to/ca.crt` | -| Client certificate (mTLS) | `sslcert`, `ssl_key`, `ssl_authentication=true` | -| Trust store | `trust_store`, `key_store_password`, `key_store_type` | +| Scenario | Property / URL parameter | +|----------|------------------------------------------------------------------------------| +| Enable HTTPS | `ssl=true` + port 8443 or use `https` in connection URL | +| Self-signed server cert | `sslrootcert=/path/to/ca.crt` | +| Client certificate (mTLS) | `sslcert`, `ssl_key`, `ssl_authentication=true` | +| Trust store | `trust_store`, `key_store_password`, `key_store_type` | | HTTP proxy | `proxy_type=http`, `proxy_host`, `proxy_port`, `proxy_user`, `proxy_password` | See [examples/jdbc SSLExamples](../examples/jdbc/src/main/java/com/clickhouse/examples/jdbc/SSLExamples.java) and [authentication.md](authentication.md). See also [integration-testing.md — Test Environment](integration-testing.md#test-environment) for testing across protocols, hosts, and ClickHouse versions. -### Init configuration — server vs client settings - -Both forward through URL/Properties: - -```java -public Connection createConfiguredConnection() throws SQLException { - Properties props = new Properties(); - props.setProperty("user", "default"); - props.setProperty("max_execution_time", "60"); // server setting - props.setProperty("max_open_connections", "20"); // client setting - return DriverManager.getConnection( - "jdbc:clickhouse://localhost:8123/default", props); -} -``` - -Per-statement server settings: use `Statement.setQueryTimeout(...)` or an SQL `SETTINGS` clause. - ### Health check ```java @@ -455,7 +428,8 @@ public boolean checkConnectionHealth(Connection conn, int timeoutSeconds) throws } ``` - +**Note** +This check can be used to wake up idle ClickHouse Cloud instance. ### Common Pitfalls @@ -490,7 +464,7 @@ The response format can be configured using the `format` connection property (`C - On ClickHouse 26.8+, the request format header sent by the driver (`X-ClickHouse-Format`) takes priority over a `FORMAT` clause written in the SQL query string. - By default, the driver sends `format=RowBinaryWithNamesAndTypes`. - To read JSON in JDBC, the recommended approach is setting `format=JSONEachRow` in connection properties along with `jdbc_json_parser_factory`. -- Setting `format=` (empty string) or `null` is an **expert-only setting**: +- Setting `format=` (empty string) or `null` to do not send anything: - Setting `format=` omits the `X-ClickHouse-Format` request header, allowing explicit SQL `FORMAT` clauses written in query strings to take effect. - **Caveat:** For any statement without an explicit SQL `FORMAT` clause, the server falls back to its `default_format` (`TabSeparated`). Because JDBC `ResultSet` only consumes `RowBinaryWithNamesAndTypes` and `JSONEachRow`, such queries fail with a `SQLException`. - `DatabaseMetaData` operations (e.g. `getTables()`, `getColumns()`) are not affected by the `format` property: they pin `RowBinaryWithNamesAndTypes` on the statements they run internally. @@ -535,6 +509,8 @@ public void readJsonEachRowResultSet(Connection conn) throws Exception { } ``` +See also example [application in repo](../examples/jdbc-v2-json-processors). + ### When JDBC's format contract is not enough | Goal | JDBC approach | Better alternative | @@ -631,14 +607,14 @@ public void readSpecialTypes(ResultSet rs) throws SQLException { ### Operation configuration — tuning heavy reads -| Setting | Property / method | Notes | -|---------|-------------------|-------| -| Response compression | `compress=true` (default) | Server-side LZ4 | -| Max execution time | `max_execution_time` | Server-side query timeout | -| Max result rows | `jdbc_use_max_result_rows=true` | Enforce server `max_result_rows` | -| Query timeout | `Statement.setQueryTimeout(seconds)` | JDBC-level timeout | +| Setting | Property / method | Notes | +|---------|-------------------|------------------------------------| +| Response compression | `compress=true` (default) | Server-side LZ4 / ZSTD | +| Max execution time | `max_execution_time` | Server-side query timeout | +| Max result rows | `jdbc_use_max_result_rows=true` | Enforce server `max_result_rows` | +| Query timeout | `Statement.setQueryTimeout(seconds)` | JDBC-level timeout | | Result-set auto-close | `jdbc_resultset_auto_close=true` (default) | Close previous result on new query | -| Fetch size | `Statement.setFetchSize(n)` | Streaming batch-size hint | +| Fetch size | `Statement.setFetchSize(n)` | Streaming batch-size hint | **Tip:** enforce row limits in SQL (`LIMIT n`). With `jdbc_use_max_result_rows` disabled, the driver stops reading at the limit but the server may still send remaining data. @@ -827,7 +803,7 @@ See below for how to detect and handle schema mismatch errors specifically. #### Schema mismatch -When working with a ClickHouse cluster, you may encounter errors if the table schema changes unexpectedly between queries or ingestions. While most JDBC applications escape this because generated SQL typically matches the current schema, dynamic ingestion workflows or custom binary writers (like RowBinary insert streams) are particularly sensitive. +When working with a ClickHouse cluster, you may encounter errors if the table schema changes unexpectedly between queries or ingestion. While most JDBC applications escape this because generated SQL typically matches the current schema, dynamic ingestion workflows or custom binary writers (like RowBinary insert streams) are particularly sensitive. **Typical error:** A schema change — such as adding, removing, or altering a column — can result in ingestion failures with errors like: From 6b56c3c57ad023ce815f1c887f06a531f06837fa Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Sat, 26 Sep 2026 22:23:26 -0700 Subject: [PATCH 3/8] Added documentation for FINAL and sequential consistency. Fixed some formatting issues --- docs/integration-jdbc.md | 69 +++++++++++++++++++++++++--------------- 1 file changed, 43 insertions(+), 26 deletions(-) diff --git a/docs/integration-jdbc.md b/docs/integration-jdbc.md index 34b95b813..c55a8754f 100644 --- a/docs/integration-jdbc.md +++ b/docs/integration-jdbc.md @@ -618,6 +618,19 @@ public void readSpecialTypes(ResultSet rs) throws SQLException { **Tip:** enforce row limits in SQL (`LIMIT n`). With `jdbc_use_max_result_rows` disabled, the driver stops reading at the limit but the server may still send remaining data. +### Read Specific Features + +#### `FINAL` + +There is a need sometime to use [`FINAL`](https://clickhouse.com/docs/reference/statements/select/from#final-modifier) modifier for select a query without modifying this query (common for tools accepting raw SQL from user). In this case server setting `final=1` should be set via connection configuration or before executing statement (see [Server and Client Settings](#Server-and-Client-Settings) + +#### Sequential Consistency + +When reading data from cluster it may happen that `SELECT` reaches replica that has not replicated recently written data. In such cases queries can have setting `select_sequential_consistency = 1` . For more information see [documentation](https://clickhouse.com/docs/get-started/migrate/postgres/appendix#sequential-consistency). + +**Warning** +Avoid using this option until absolutely needed. This setting puts pressure on inter-server communication and will slow down other operations. + ### Best practices @@ -627,6 +640,7 @@ public void readSpecialTypes(ResultSet rs) throws SQLException { - **Prefer `PreparedStatement`** for repeated / parameterized queries. - **Map large integers**: `jdbc_type_mappings=UInt64=java.math.BigInteger`. - **Use `getObject(column, Class)`** for `java.time` types instead of legacy getters. +- **Off load data processing from read** when reading big chunks of data. Server may timeout on write if read is too slow. ### Common Pitfalls @@ -738,12 +752,10 @@ This leverages the JDBC driver's RowBinary streaming (when `beta.row_binary_for_ > > **Caution:** Remember that with async insert, success response means "data accepted for processing" — not yet "written." Use `wait_async_insert=1` to wait for commit, and deduplication tokens if retrying inserts to avoid duplicates. - ### Idempotency — deduplication token JDBC does not expose `insert_deduplication_token` as a first-class API. Three ways to use it: - **SQL `SETTINGS` clause** (per statement): ```java @@ -831,28 +843,30 @@ A schema change — such as adding, removing, or altering a column — can resul ```java import com.clickhouse.client.ServerException; -try { - // ... ingestion code ... -} catch (SQLException ex) { - Throwable cause = ex.getCause(); - if (cause instanceof ServerException) { - ServerException serverEx = (ServerException) cause; - // ClickHouse code 33: UNABLE TO READ ALL DATA (schema mismatch) - if (serverEx.getErrorCode() == 33 ) { - // Possible table schema mismatch - // 1. Reload schema via DatabaseMetaData or DESCRIBE TABLE - // 2. Update SQL or RowBinary mapping as needed - // 3. Retry with updated logic +void writeData(List data, TableSchema schema) { + try { + // ... ingestion code ... using schema + } catch (SQLException ex) { + Throwable cause = ex.getCause(); + if (cause instanceof ServerException) { + ServerException serverEx = (ServerException) cause; + // ClickHouse code 33: UNABLE TO READ ALL DATA (schema mismatch) + if (serverEx.getErrorCode() == 33) { + // Possible table schema mismatch + // 1. Reload schema via DatabaseMetaData or DESCRIBE TABLE + // 2. Update SQL or RowBinary mapping as needed + // 3. Retry with updated logic + } else { + throw ex; // not a schema mismatch + } } else { - throw ex; // not a schema mismatch + throw ex; // Cannot determine server error code } - } else { - throw ex; // Cannot determine server error code } } ``` -> **Note:** Always ensure your application's data shape matches the target table exactly when using RowBinary or similar binary ingestion APIs. If your code holds a schema cache, invalidate it and fetch the up-to-date schema on a Code 33 error before retrying. +> **Note:** Always ensure your application's data schema matches the target table one exactly when using RowBinary or similar binary ingestion APIs. If your code holds a schema cache, invalidate it and fetch the up-to-date schema on a Code 33 error before retrying. See [ClickHouse error codes documentation](https://clickhouse.com/docs/en/operations/error-codes/) for more on Code 33 and related scenarios. @@ -874,8 +888,8 @@ See [ClickHouse error codes documentation](https://clickhouse.com/docs/en/operat - **Batching complex INSERT shapes** (`INSERT SELECT`, multi-table) is unsupported — use `Statement`. - **Retrying without a dedup token** on MergeTree can create duplicates. - **Maximum ingest throughput** is not JDBC's strength — the [Java Client](integration-client.md) stream insert is faster. - + --- ## Step 7 — Metadata & schema discovery @@ -965,8 +979,8 @@ For schema-driven POJO binding and binary format writers, use the [Java Client i - **JDBC metadata may not reflect every ClickHouse type nuance** — use `getColumnTypeName()` for the native string. - **Default type mappings** may not match your expectations for large integers or complex types — override with `jdbc_type_mappings`. - + --- ## JDBC-specific features @@ -1030,14 +1044,17 @@ You can enable Client V2 operational metrics (durations, counts, retries) for JD ```java // Configure via JDBC URL parameter -String url = "jdbc:clickhouse://localhost:8123/default?jdbc_metrics_recorder=com.clickhouse.client.api.observability.micrometer.MicrometerMetricsRecorder"; -// Or set via java.util.Properties -Properties properties = new Properties(); -properties.setProperty("jdbc_metrics_recorder", "com.clickhouse.client.api.observability.micrometer.MicrometerMetricsRecorder"); +import java.sql.Connection; +import java.sql.SQLException; + +Connection createConnection(String host) throws SQLException { + final String url = "jdbc:clickhouse://" + host + "/default?jdbc_metrics_recorder=com.clickhouse.client.api.observability.micrometer.MicrometerMetricsRecorder"; -try (Connection conn = DriverManager.getConnection(url, properties)) { - // JDBC operations report operation metrics to Micrometer's globalRegistry + // Or set via java.util.Properties + Properties properties = new Properties(); + properties.setProperty("jdbc_metrics_recorder", "com.clickhouse.client.api.observability.micrometer.MicrometerMetricsRecorder"); + return DriverManager.getConnection(url, properties); } ``` From 487648248bdfc7f2301f058f6e5bd8e47ea40065 Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Tue, 29 Sep 2026 23:18:08 -0700 Subject: [PATCH 4/8] Updated part of integration-client.md --- docs/integration-client.md | 145 +++++++++++++++++++++++++++++-------- 1 file changed, 115 insertions(+), 30 deletions(-) diff --git a/docs/integration-client.md b/docs/integration-client.md index d118912d7..dd64c6c18 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -1,14 +1,16 @@ # ClickHouse Java Client Integration Guide -This guide is a **step-by-step, end-to-end integration path** for the **Java Client V2** (`client-v2`). It is written to be used as context for building an application or a downstream integration spec: each step states the decisions you must make, how to configure them, and the common pitfalls to avoid. It is self-contained — you can work through it from the empty project to a running read/write path without other prerequisites. +This guide is a **end-to-end integration path** for the **Java Client V2** (`client-v2`). It is written to be used as context for building an application or a downstream integration spec: each step states the decisions you must make, how to configure them, and the common pitfalls to avoid. It is self-contained — you can work through it from the empty project to a running read/write path without other prerequisites. -> **Configuration philosophy.** This guide names only the properties relevant to each step. It does not repeat the exhaustive property list — that lives in [`ClientConfigProperties`](../client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java) and the official docs. Configuration splits into two groups: -> - **Init configuration** — set once when the client is built: endpoint, connection pool size, async mode, authentication. Covered in Steps 1–4. -> - **Operation configuration** — set per request or as client defaults: formats, buffer sizes, timeouts, retries, dedup tokens. Covered in Steps 5–7. +### Configuration References -## Artifacts +This guide names only the properties relevant to each step. The exhaustive lists live in [`ClientConfigProperties`](../client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java), and the official docs. Configuration splits into two groups: +- **Init configuration** — set once using [Client.Builder](/client-v2/src/main/java/com/clickhouse/client/api/Client.java) methods: endpoint, connection pool size, authentication, TLS. Covered in Steps 1–4. +- **Operation configuration** — set via `com.clickhouse.client.api.Client` method like `com.clickhouse.client.api.Client.Builder.setAccessToken`. -The client is published to Maven Central as **`com.clickhouse:client-v2`**. Browse versions and copy a ready-made dependency snippet for any build system (Maven, Gradle, sbt, Ivy, ...) from the [Maven Central page](https://central.sonatype.com/artifact/com.clickhouse/client-v2). +### Maven Artifacts + +The client is published to Maven Central as `com.clickhouse:client-v2`: https://central.sonatype.com/artifact/com.clickhouse/client-v2. Two distributions are published under the same artifact: @@ -45,16 +47,15 @@ Work through these steps in order. Each one is a decision point; the "Common Pit **Goal:** decide how many `Client` instances exist, how long they live, and how the internal connection pool is sized. -### What a `Client` is - -The [`Client`](https://javadoc.io/doc/com.clickhouse/client-v2/latest/com/clickhouse/client/api/Client.html) is the single entry point for all operations. It owns: -- An HTTP connection pool (Apache HttpClient) -- Endpoint configuration and retry policy -- A table schema cache -- The POJO serialization/deserialization registry -- Optional client-wide session and settings defaults +The [com.clickhouse.client.api.Client](/client-v2/src/main/java/com/clickhouse/client/api/Client.java) is library interface object for most interactions with it: +- Read and Write operations +- Getting a table schema +- Registering POJO for use in insert operations +- Manipulating active session +- Updating user credentials +This object is constructed using a `Client.Builder`: ```java import com.clickhouse.client.api.Client; @@ -76,13 +77,20 @@ public Client createAnalyticsDBClient(Client.Builder baseClient) { } ``` -### Instantiation Strategy +### Instantiation and Life Cycle Strategies + +- **Shared single instance:** to make request in the same domain and security context. Usually long living. For example, data sync job. +- **Short-lived instance:** to make multiple operations within one application request for a single user. + +> In ClickHouse Cloud instances can enter suspended mode when idle. Make sure to have a "wake-up" call using `Client#ping()`. Using suspended mode useful to reduce cost +> by not running instance when it is almost idle. -- **Share a single instance:** A single, shared `Client` instance suits most use cases. It is thread-safe and designed to be reused across your application. -- **Long-lived lifecycle:** Build the client once at startup and close it once at shutdown. Creating a new client per request or operation is an anti-pattern because initialization takes time to set up internal structures (like the connection pool and schema cache), which adds latency to your requests. -- **Serverless functions:** For serverless environments (like AWS Lambda), initialize the client outside the function handler so it can be reused across invocations. -- **Warm-up (optional):** Calling `client.ping()` at startup can help initialize the connectivity part and verify the endpoint before serving live traffic, though it is not strictly required. It may also require to wakeup cloud instance. -- **Caching:** The application is responsible for holding the reference to the `Client` instance (e.g., via dependency injection or a singleton). The library does not provide a global static cache. +Application should take care of: +- `Client` instances caching if needed. Library doesn't hold reference to instantiated `Client` objects +- Closing client when it is not needed. +- Separating `Client` instances of different tenants. + +> Even client init is a lightweight operation avoid make it in hot path. For example, prefer caching instances if working in multi-tenant environment ### Workload identification & client name @@ -220,9 +228,7 @@ LIMIT 100; ## Step 2 — Authentication -**Goal:** configure the authentication mechanism the ClickHouse deployment requires. The mechanism is dictated by the server and any fronting infrastructure, not chosen freely; the task is to identify it and configure it correctly. - -> **CONSTRAINT:** Configure exactly one mechanism. Different method cannot be mixed to avoid configuration errors. +**Goal:** choose exactly one primary authentication mechanism and pass it through the JDBC URL or `Properties`. The driver forwards these to the underlying client, whose `CredentialsManager` rejects mixed mechanisms. ### Option A — Basic (username + password) @@ -248,8 +254,7 @@ void updateClientCredentials(AppConfiguration appConf) { } ``` -**Note**: realtime credentials update would work well with runtime configuration update but would not work for multi-tenant setup. Multi tenant application should organize exclusive access to client -while handling tenant operation to avoid cross-talk problem. Separate client instance per tenant must be used when each tenant has own database. +**Note**: realtime credentials update would work well with runtime configuration update but would not work for multi-tenant setup. Multi tenant application should organize exclusive access to client while handling tenant operation to avoid cross-talk problem. Separate client instance per tenant must be used when each tenant has own database. ### Option B — Token / bearer @@ -353,7 +358,7 @@ public Client createAnalyticsClient(Client.Builder baseClient) { > **Note on Client Name:** How the client name surfaces in `system.query_log` depends on the protocol used. For HTTP connections (used by Java Client V2), it appears in the `http_user_agent` column. The `client_name` column in `system.query_log` is populated only for native TCP connections. See [Workload identification & client name](#workload-identification--client-name) for full details and query log troubleshooting queries. -### Identifying the required mechanism +### Decisions The mechanism follows from how the server and any fronting infrastructure are configured: @@ -387,14 +392,13 @@ See [integration-testing.md — Configuration](integration-testing.md#configurat See [SSLExamples](../examples/client-v2/src/main/java/com/clickhouse/examples/client_v2/SSLExamples.java) for a runnable walkthrough and [authentication.md](authentication.md) for full details. See also [integration-testing.md — Test Environment](integration-testing.md#test-environment) for testing across protocols, hosts, and ClickHouse versions. -### Init configuration — timeouts +### Operation Timeouts -Timeouts are critical parameters that directly impact application stability under load and over long distances: +Timeouts are critical parameters that directly impact application stability under load and over long run time: -- **Connection timeout** (`.setConnectTimeout()`): The TCP connect timeout. Setting this value too low can cause failures when the application and server are in different geographical regions. Additionally, connection timeouts are closely tied to the connection pool: if the application issues concurrent requests that exceed the available pool size, it may manifest as a connection timeout because no free connections are present. +- **Connection timeout** (`.setConnectTimeout()`): timeout for establishing connection. Setting this value too low can cause failures when the application and server are in different geographical regions. Additionally, connection timeouts are closely tied to the connection pool: if the application issues concurrent requests that exceed the available pool size, it may manifest as a connection timeout because no free connections are present. - **Socket timeout** (`.setSocketTimeout()`): The timeout for underlying socket read/write operations. While it applies strictly to socket activity, it is vital because it dictates how long the client will wait for long-running queries to return data. If your workload involves heavy analytical queries, you may need a very long socket timeout. However, the trade-off of a long socket timeout is the increased risk of encountering stale or silently dropped connections. - **TCP keepalive**: Can be enabled to mitigate stale connections, though the host operating system's settings may ultimately override it. System-level TCP keepalive defaults are often several hours; configuring a shorter keepalive period makes sense for long-running operations. Keep in mind that executing extremely long operations over the public internet remains inherently risky. -- **Socket buffers** (`.setSocketRcvbuf()`, `.setSocketSndbuf()`): Not set by default, so the operating system sizes the socket buffers and auto-tunes them for the connection. Setting a fixed size turns that auto-tuning off and is additionally capped by the operating system limits, so a large value may have no effect. Configure these options only when a measurement shows a benefit for your workload. > **Note on runtime configuration:** You can optionally override the default network timeout on a per-operation basis using `QuerySettings.setNetworkTimeout(long timeout, ChronoUnit unit)`. This allows you to set stricter boundaries on specific queries without altering the client-wide defaults. @@ -431,6 +435,10 @@ See [integration-testing.md — Connecting](integration-testing.md#connecting) f In the Java Client a "connection" is an **HTTP connection borrowed from the internal pool**, not a long-lived database session. Each operation borrows a connection, sends a request, streams the response, and returns the connection to the pool. +### Sockets Configuration + +- **Socket buffers** (`.setSocketRcvbuf()`, `.setSocketSndbuf()`): Not set by default, so the operating system sizes the socket buffers and auto-tunes them for the connection. Setting a fixed size turns that auto-tuning off and is additionally capped by the operating system limits, so a large value may have no effect. Configure these options only when a measurement shows a benefit for your workload. + ### Connection limit (`max_open_connections`) The pool size depends on your workload — specifically on its **concurrency**, not on how much data it moves. What matters is **how many operations run at the same time**, not the number of rows or bytes any single operation transfers. A pool of 20 connections serves at most 20 simultaneous operations regardless of whether each returns one row or a million. This is the single setting you actually tune. The table below will help to estimate rough number. Having slightly bigger number than actualy needed is not a problem because unused connections will be garbage collected. It is recommended to perform a load testing with one application instance to detect if estimated number works — see [integration-testing.md — Load Testing](integration-testing.md#load-testing) for how to design that test. @@ -497,6 +505,83 @@ public Client.Builder createBaseClient() { See [integration-testing.md — Connecting](integration-testing.md#connecting) for tests that verify pool sizing (`max_open_connections`) and connection release under load. --- +### Retry Policy + +The `Client` has built-in support for retrying requests that fail due to *transient* conditions, such as network timeouts or server unavailability. The retry strategy is controlled by configuration properties, which can be set at the client or per-request level. + +**Stale Connection:** when connection in the pool is not used for a long time server can close it and next request that uses such a connection will fail. Usually it is `NoHttpResponse` error. Use `ClientFaultCause.NoHttpResponse` to retry on such errors. Usually TCP keepalive doesn't solve the problem because TCP will not check connection in hours and OS configuration would be needed to change this time period. + +**Configuring Retries:** + +| Property (Builder method) | Purpose | Default | +|---------------------------|---------|---------| +| `retry` (`.setRetryOnFailure(int)`) | Maximum number of automatic retries for retriable failures (not user/SQL errors). | `3` | +| `client_retry_on_failures` (`.setRetryOnFailureFor(List)`) | Fine-grained control of which failure types to retry. See below. | `NoHttpResponse`, `ConnectTimeout`, `ConnectionRequestTimeout`, `ServerRetryable` | + +Example: + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.ClientFaultCause; + +public Client.Builder createResilientClient() { + return new Client.Builder() + .addEndpoint("http://localhost:8123") + .setRetryOnFailure(5) + .setRetryOnFailureFor( + ClientFaultCause.NoHttpResponse, // stale connection problem + ClientFaultCause.ConnectTimeout, // server did not accept connection + ClientFaultCause.ServerRetryable // transient HTTP 5xx errors, server overload + ); +} +``` + +#### Handling ServerException and Retriable Errors + +Failed requests (e.g., a rejected query or a server-side error) throw a `ServerException`. Not all failures are safe to retry—retries make sense for *transient* problems, but not for syntax errors, quota violations, or authorization failures. + +To determine whether an error is retriable, use the `isRetriable()` method on `ServerException`: + +```java +import com.clickhouse.client.api.ServerException; + +void doSync() { + try { + // ... client request ... + } catch (ServerException e) { + if (e.isRetriable()) { + // Retry logic: safe to reattempt + // (Most users should use the client's built-in retry mechanism instead) + } else { + // Not retriable: fail/abort or escalate + } + } +} +``` + +**What is considered retriable?** + +- Server-side errors classified as *transient*: HTTP 502/503/504 responses, server overload, shutdown, resource limits that may be cleared after delay. +- Connection-level issues: `NoHttpResponse` (dropped/stale pooled connection), `ConnectTimeout`, etc. + +See [`ClientFaultCause`](../client-v2/src/main/java/com/clickhouse/client/api/ClientFaultCause.java) for the full set. + +> **Note:** Retries are never triggered for application errors (bad SQL, type mismatch), authentication failures, or client-side value validation errors. + +#### Tuning retry logic + +- Use a higher retry count in long-running ingestion or ETL jobs, but stick to fewer retries for user-facing APIs (to avoid delays). +- You may override `retry` and `client_retry_on_failures` **per query** via `setOption("retry", ...)` and `setOption("client_retry_on_failures", ...)`. + +#### Negative test: Making sure unrecoverable errors are not retried + +If you want to test that application errors *do not* trigger a retry, deliberately send a bad SQL and check that only a single attempt is made. + +--- + +For further details and advanced configuration, see [ClientConfigProperties.java](../client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java). + + ## Step 5 — Data formats, readers & writers All formats are defined in [`ClickHouseFormat`](../clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseFormat.java); each declares whether it supports input, output, binary encoding, headers, and row layout. From 8f781f5be3f4655919b387d47c342b77e18ebe96 Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Wed, 14 Oct 2026 14:15:07 -0700 Subject: [PATCH 5/8] Made general cleanup of integration spec for client --- docs/integration-client.md | 108 ++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 62 deletions(-) diff --git a/docs/integration-client.md b/docs/integration-client.md index de742586a..ad5823d7c 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -94,7 +94,7 @@ Application should take care of: ### Workload identification & client name -In production environments, a single ClickHouse cluster often handles diverse workloads from multiple services: real-time user-facing APIs, event ingestion pipelines (e.g., Kafka consumers, CDC streams), scheduled ETL batch jobs, BI reporting dashboards (e.g., Superset, Tableau, Grafana), and ad-hoc analytical queries. When queries fail, time out, or cause memory spikes (`MEMORY_LIMIT_EXCEEDED`), identifying the originating application or workload is essential for fast troubleshooting, root-cause analysis, and resource attribution. +In production environments, a single ClickHouse cluster often handles diverse workloads from multiple services: real-time user-facing APIs, event ingestion pipelines (e.g., Kafka consumers, CDC streams), scheduled ETL batch jobs, BI reporting dashboards (e.g., Superset, Tableau, Grafana), and ad-hoc analytical queries. When queries fail, time out, or cause memory spikes (`MEMORY_LIMIT_EXCEEDED`), identifying the originating application or workload is essential for fast troubleshooting and resource attribution. #### Setting client name @@ -225,6 +225,22 @@ ORDER BY event_time DESC LIMIT 100; ``` +### Default Database + +If application works with specific database then default database can be set on client level: + +```java +import com.clickhouse.client.api.Client; + +public Client createAnalyticsClient(Client.Builder baseClient) { + return baseClient + .setDefaultDatabase("analytics") // Default database for queries + .build(); +} +``` + +**Note**: when database is set on request level then it is recommended to set non-existing database as default one. It +will cause exception and will not let any data be written to `default` schema. ## Step 2 — Authentication @@ -341,23 +357,6 @@ public Client.Builder createBaseClient() { } ``` -### Client Identity & Default Database - -Regardless of the authentication mechanism, you can also configure the client's identity and default database: - -```java -import com.clickhouse.client.api.Client; - -public Client createAnalyticsClient(Client.Builder baseClient) { - return baseClient - .setClientName("my-analytics-app/1.0") - .setDefaultDatabase("analytics") // Default database for queries - .build(); -} -``` - -> **Note on Client Name:** How the client name surfaces in `system.query_log` depends on the protocol used. For HTTP connections (used by Java Client V2), it appears in the `http_user_agent` column. The `client_name` column in `system.query_log` is populated only for native TCP connections. See [Workload identification & client name](#workload-identification--client-name) for full details and query log troubleshooting queries. - ### Decisions The mechanism follows from how the server and any fronting infrastructure are configured: @@ -382,13 +381,13 @@ See [integration-testing.md — Configuration](integration-testing.md#configurat ### TLS / mTLS / proxies -| Scenario | Builder methods / properties | -|----------|------------------------------| -| HTTPS with public CA | `addEndpoint("https://host:8443")` | -| Self-signed server cert | `setRootCertificate("/path/to/ca.crt")` | -| mTLS (client certificate) | `useSSLAuthentication(true)`, `setClientCertificate(...)`, `setClientKey(...)` | -| Trust store (JKS/PKCS12) | `setSSLTrustStore(...)`, `setSSLTrustStorePassword(...)` | -| HTTP proxy | `setProxy(ProxyType.HTTP, host, port)`, `setProxyCredentials(user, password)` | +| Scenario | Property(ies) | Builder methods | +|-----------------------------|------------------------------------------------------|------------------------------------------------------| +| HTTPS with public CA | - | `addEndpoint("https://host:8443")` | +| Self-signed server cert | `root_certificate` | `setRootCertificate("/path/to/ca.crt")` | +| mTLS (client certificate) | `ssl_auth`, `client_certificate`, `client_key` | `useSSLAuthentication(true)`, `setClientCertificate(...)`, `setClientKey(...)` | +| Trust store (JKS/PKCS12) | `ssl_trust_store`, `ssl_trust_store_password` | `setSSLTrustStore(...)`, `setSSLTrustStorePassword(...)` | +| HTTP proxy | `proxy_type`, `proxy_host`, `proxy_port`, `proxy_user`, `proxy_password` | `setProxy(ProxyType.HTTP, host, port)`, `setProxyCredentials(user, password)` | See [SSLExamples](../examples/client-v2/src/main/java/com/clickhouse/examples/client_v2/SSLExamples.java) for a runnable walkthrough and [authentication.md](authentication.md) for full details. See also [integration-testing.md — Test Environment](integration-testing.md#test-environment) for testing across protocols, hosts, and ClickHouse versions. @@ -457,40 +456,21 @@ The reasoning is only valid for **short-lived read operations**, where connectio The `Client` uses an internal Apache HttpClient 5 pool. Key pool settings exposed by client configuration are listed below: -| Property (Builder method) | Purpose | Default | -|---------------------------|---------|---------| -| `max_open_connections` (`.setMaxConnections()`) | Maximum open HTTP connections per server endpoint. Size set from parallelism guidance above. | `10` | -| `connection_ttl` (`.setConnectionTTL()`) | Time-to-live after which an active connection is closed and recreated. | `-1` (disabled) | -| `http_keep_alive_timeout` (`.setKeepAliveTimeout()`) | HTTP Keep-Alive duration for idle pooled connections. | Server default | -| `connection_request_timeout` (`.setConnectionRequestTimeout()`) | Maximum time a thread blocks waiting for an available connection from the pool. | `10000ms` | -| `connection_reuse_strategy` (`.setConnectionReuseStrategy()`) | Connection pool allocation strategy (`FIFO` or `LIFO`). | `FIFO` | -| `connection_pool_enabled` (`.enableConnectionPool()`) | Enable/disable HTTP connection pooling (keep enabled). | `true` | - -#### Recommended Pool Configuration - -```java -import com.clickhouse.client.api.Client; -import com.clickhouse.client.api.enums.ConnectionReuseStrategy; - -import java.util.concurrent.TimeUnit; +| Property | Builder method | Purpose | +|-------------------------------|-------------------------------------|-------------------------------------------------------------------------------------------| +| `max_open_connections` | `.setMaxConnections()` | Maximum open HTTP connections per server endpoint. Size set from parallelism guidance above.| +| `connection_ttl` | `.setConnectionTTL()` | Time-to-live after which an active connection is closed and recreated. | +| `http_keep_alive_timeout` | `.setKeepAliveTimeout()` | HTTP Keep-Alive duration for idle pooled connections. | +| `connection_request_timeout` | `.setConnectionRequestTimeout()` | Maximum time a thread blocks waiting for an available connection from the pool. | +| `connection_reuse_strategy` | `.setConnectionReuseStrategy()` | Connection pool allocation strategy (`FIFO` or `LIFO`). | +| `connection_pool_enabled` | `.enableConnectionPool()` | Enable/disable HTTP connection pooling (keep enabled). | -public Client.Builder createBaseClient() { - return new Client.Builder() - .addEndpoint("http://localhost:8123") - .setUsername("default") - .setPassword("secret") - .setMaxConnections(50) - .setConnectionRequestTimeout(5, TimeUnit.SECONDS) - .setKeepAliveTimeout(60, TimeUnit.SECONDS) - .setConnectionReuseStrategy(ConnectionReuseStrategy.FIFO); -} -``` -**`setKeepAliveTimeout` and stale connections.** If your application encounters `NoHttpResponseException`, it usually indicates stale pooled connections that were closed by the server or an intermediate proxy/load balancer due to idle timeouts. Setting `setKeepAliveTimeout` to a duration shorter than the server's Keep-Alive timeout ensures idle connections are refreshed before sending a request. +- **`http_keep_alive_timeout` and stale connections.** If your application encounters `NoHttpResponseException`, it usually indicates stale pooled connections that were closed by the server or an intermediate proxy/load balancer due to idle timeouts. Setting `setKeepAliveTimeout` to a duration shorter than the server's Keep-Alive timeout ensures idle connections are refreshed before sending a request. -**`connection_ttl` against ClickHouse Cloud.** Keep it **relatively small** when the endpoint is a Cloud (or otherwise load-balanced) deployment. A short TTL forces connections to be retired and re-established frequently, so new connections keep going through the load balancer, which lets it **redistribute traffic across nodes** instead of pinning long-lived connections to whichever node they first landed on. +- **`connection_ttl` against ClickHouse Cloud.** Keep it **relatively small** when the endpoint is a Cloud (or otherwise load-balanced) deployment. A short TTL forces connections to be retired and re-established frequently, so new connections keep going through the load balancer, which lets it **redistribute traffic across nodes** instead of pinning long-lived connections to whichever node they first landed on. -**`connection_pool_enabled` is not recommended to change.** Leave pooling **enabled** (the default). The only reason to disable it is a workload with extreme concurrency where the internal Apache HttpClient connection pool itself becomes a contention point — under very high parallel request rates the pool's own bookkeeping can serialize threads. Disabling pooling avoids that bottleneck at the cost of a fresh connection per operation (higher latency, more sockets), so treat it as a last resort after measuring, not a default. +- **`connection_pool_enabled` is not recommended to change.** Leave pooling **enabled** (the default). The only reason to disable it is a workload with extreme concurrency where the internal Apache HttpClient connection pool itself becomes a contention point — under very high parallel request rates the pool's own bookkeeping can serialize threads. Disabling pooling avoids that bottleneck at the cost of a fresh connection per operation (higher latency, more sockets), so treat it as a last resort after measuring, not a default. ### Common Pitfalls @@ -503,9 +483,13 @@ public Client.Builder createBaseClient() { > **CONSTRAINT:** Always `close()` the client at shutdown to avoid leaking the pool and its threads. See [integration-testing.md — Connecting](integration-testing.md#connecting) for tests that verify pool sizing (`max_open_connections`) and connection release under load. + + --- -### Retry Policy +### Handling Errors & Retry Policy + +#### Retry Policy The `Client` has built-in support for retrying requests that fail due to *transient* conditions, such as network timeouts or server unavailability. The retry strategy is controlled by configuration properties, which can be set at the client or per-request level. @@ -513,10 +497,10 @@ The `Client` has built-in support for retrying requests that fail due to *transi **Configuring Retries:** -| Property (Builder method) | Purpose | Default | -|---------------------------|---------|---------| -| `retry` (`.setRetryOnFailure(int)`) | Maximum number of automatic retries for retriable failures (not user/SQL errors). | `3` | -| `client_retry_on_failures` (`.setRetryOnFailureFor(List)`) | Fine-grained control of which failure types to retry. See below. | `NoHttpResponse`, `ConnectTimeout`, `ConnectionRequestTimeout`, `ServerRetryable` | +| Property | Builder Method | Purpose | +|---------------------------|----------------------------------------|------------------------------------------------------------| +| `retry` | `.setRetryOnFailure(int)` | Max automatic retries for retriable (not user/SQL) errors. | +| `client_retry_on_failures`| `.setRetryOnFailureFor(ClientFaultCause...)` | Select which failure types will be retried. | Example: @@ -654,7 +638,7 @@ public ClickHouseFormat inspectQueryFormat(Client client, String sql) throws Exc } ``` -## Step 6 — Read operations & tuning +## Step 6 — Read operations **Goal:** read results efficiently and configure the operation-level settings for heavy analytical reads. @@ -807,7 +791,7 @@ See the [Error model](#error-model) for the exception hierarchy and how to unwra See [integration-testing.md — Fetching Data](integration-testing.md#fetching-data) for read-path test scenarios to cover (data types, formats, failure handling). --- -## Step 7 — Write operations & tuning +## Step 7 — Write operations **Goal:** choose an insert pattern, tune it for bulk ingest, and make retries idempotent. From 7751b3fbdfe13e516dd7f9bb3d665158e6bccf58 Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Thu, 15 Oct 2026 10:46:04 -0700 Subject: [PATCH 6/8] Added type mapping for integration spec --- docs/integration-client.md | 80 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/integration-client.md b/docs/integration-client.md index ad5823d7c..b5a52a849 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -638,6 +638,86 @@ public ClickHouseFormat inspectQueryFormat(Client client, String sql) throws Exc } ``` +### Data Types + +A clear type system is essential for integrating application components with ClickHouse. Decisions made at this stage carry over into the future evolution of your application, so review the type mappings carefully before finalizing your design. + +Start by reviewing the [ClickHouse data types](https://clickhouse.com/docs/reference/data-types/index) (new types are added regularly), then check the client documentation for the list of [supported data types](https://clickhouse.com/docs/integrations/language-clients/java#supported-data-types). If you need support for a type that is not yet covered, feel free to raise an issue in the Java client repository. + +Database data types are mapped to Java types by the reader or writer and generally follow similar rules across formats (though custom or format-specific readers and writers may use different mappings). + + +**ClickHouse / Java Type Mapping** + +The following table summarizes typical mappings between ClickHouse database and Java types. Conversion depends on data +format and reader/writer implementation. Following table represents supported conversions when used with ClickHouse binary formats +like `RowBinary` and `Native`. + +Reader Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader` +Writer Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatWriter` + +| ClickHouse Type | Java Type (Reader) | Java Type (Writer) | +|--------------------------------------|---------------------------------------------------------------|-------------------------------------------------------------------------------------| +| `Bool` | `boolean` | `boolean` | +| `Int8` | `byte`, `short`, `int`, `long` | `byte`, `short`, `int`, `long` | +| `UInt8` | `short`, `int`, `long` | `byte`, `short`, `int`, `long` | +| `Int16` | `short`, `int`, `long` | `short`, `int`, `long` | +| `UInt16` | `int`, `long` | `short`, `int`, `long` | +| `Int32` | `int`, `long` | `int`, `long` | +| `UInt32` | `long` | `int`, `long` | +| `Int64` | `long`, `BigInteger` | `long`, `BigInteger` | +| `UInt64` | `BigInteger`, `long` | `long`, `BigInteger` | +| `Int128`, `UInt128` | `BigInteger` | `BigInteger`, `long` | +| `Int256`, `UInt256` | `BigInteger` | `BigInteger`, `long` | +| `BFloat16` | `float`, `double` | `float` | +| `Float32` | `float`, `double` | `float`, `double` | +| `Float64` | `double` | `float`, `double` | +| `Decimal`, `Decimal32`..`Decimal256` | `BigDecimal` | `BigDecimal`, `String` | +| `String`, `FixedString` | `String`, `byte[]` | `String`, `byte[]` | +| `Date` | `LocalDate` | `LocalDate` | +| `Date32` | `LocalDate` | `LocalDate` | +| `DateTime`, `DateTime32` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | +| `DateTime64` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | +| `Time`, `Time64` | `LocalTime`, `LocalDateTime`, `Duration`, `TemporalAmount` | `int` (`Time`), `long`, `BigInteger`, `Instant` (`Time64`) | +| `Interval*` | `TemporalAmount` (`Period` / `Duration`), `Duration` | `Period` (date intervals), `Duration` (time/day intervals) | +| `Enum8`, `Enum16` | `String`, `byte` (`Enum8`), `short` (`Enum16`), `int` | `String`, `byte`, `short`, `int` | +| `UUID` | `UUID`, `String` | `UUID`, `String` | +| `IPv4`, `IPv6` | `Inet4Address`, `Inet6Address`, `String` | `Inet4Address` (`IPv4`), `Inet6Address` (`IPv6`) | +| `Nullable(T)` | `T` or `null` | `T` or `null` | +| `LowCardinality(T)` | `T` | `T` | +| `Array(T)` | `List`, primitive `T[]`, `String[]`, `Object[]` | `List`, `T[]` | +| `Tuple` | `Object[]`, `List` (POJO) | `Object[]`, `List` | +| `Map(K, V)` | `Map` | `Map` | +| `Nested` | `List`, `Object[]` | `List`, `Object[][]` | +| `Point` | `double[]` | `double[]` | +| `Ring`, `LineString`, `MultiPoint` | `double[][]` | `double[][]` | +| `Polygon`, `MultiLineString` | `double[][][]` | `double[][][]` | +| `MultiPolygon` | `double[][][][]` | `double[][][][]` | +| `Geometry` | `double[]` … `double[][][][]` | `double[]` … `double[][][][]` | +| `QBit(T, N)` | `float[]`, `double[]`, `List` | `float[]`, `double[]`, `List` (length `N`) | +| `Variant`, `Dynamic` | `Object` (resolved runtime value) | `Object` (supported scalar/array/map types) | +| `JSON` | `Map`, `String` | `String` | +| `SimpleAggregateFunction(f, T)` | `T` | `T` | + +**Notes:** +- **Signed vs. unsigned integers:** Unsigned types (`UInt8`, `UInt16`, `UInt32`) are widened to the next larger signed Java primitive (`short`, `int`, `long`) on read so values above the signed max do not overflow; `UInt64`, `Int128`, `UInt128`, `Int256`, and `UInt256` use `BigInteger`. Narrower getters (e.g., `getByte` on `UInt8 > 127` or `getLong` on `UInt64 > Long.MAX_VALUE`) throw `ArithmeticException`. +- **Binary strings (`String`, `FixedString`):** By default, strings are decoded as UTF-8 `String` on read. To read raw binary bytes via `getByteArray(...)` (or POJO `byte[]` fields) without UTF-8 transcoding, enable `Client.Builder#binaryStringSupport(true)` (or per-query `binary_string_support`). +- **Dates (`Date`, `Date32`):** Use `java.time.LocalDate` (timezone-agnostic calendar date). Note that `java.sql.Date` is not supported by the `RowBinary` writer. +- **Timestamps (`DateTime`, `DateTime64`):** **Prefer `ZonedDateTime`, `OffsetDateTime`, or `Instant`** over `LocalDateTime`. ClickHouse stores `DateTime` / `DateTime64` as UTC epoch values. Because `LocalDateTime` carries no timezone offset, using it can easily cause timezone confusion: + - On **write**, a `LocalDateTime` is interpreted in the column's declared timezone (e.g., `DateTime('Asia/Tokyo')`), or in **`UTC`** if the column has no timezone parameter — *not* the JVM default timezone. + - On **read**, the reader decodes the timestamp into a `ZonedDateTime` using the column's timezone (or the client's configured timezone), and `getLocalDateTime(...)` simply strips that zone (`toLocalDateTime()`), which may not match the application's local wall clock. + - Using `Instant` or `ZonedDateTime` / `OffsetDateTime` keeps the exact point in time unambiguous across client, server, and column timezones. +- **Nullability:** Reading a `NULL` value through a primitive getter (`getInt`, `getLong`, `getDouble`, `getBoolean`, `getEnum8`, etc.) or into a primitive POJO field throws `NullValueException`. Check `hasValue(...)` first or use boxed/object types. +- **`LowCardinality(T)`:** Treated as a transparent storage modifier by the client. A `LowCardinality(T)` (or `LowCardinality(Nullable(T))`) column is unwrapped to its underlying type `T` and follows the exact same Java type conversion rules on both read and write. +- **Containers (`Array`, `Tuple`, `Map`, `Nested`):** `getList(...)` reads any `Array` as `List`, primitive array getters (`getIntArray`, `getDoubleArray`, etc.) read 1D primitive arrays, and `getObjectArray(...)` reads any 1D or multidimensional array as `Object[]`. `Tuple` is read via `getTuple(...)` as `Object[]` (and accepts `Object[]` or `List` on write and in POJO mapping). +- **Enums:** `Enum8` and `Enum16` can be read/written by label (`String`) or numeric value (`getEnum8`/`getEnum16`/`getInteger`). +- **Decimals:** `Decimal` and `Decimal32`/`64`/`128`/`256` map to `BigDecimal` to preserve scale and precision. + +When reading or writing data programmatically, favor the canonical Java type above for portability. + + + + ## Step 6 — Read operations **Goal:** read results efficiently and configure the operation-level settings for heavy analytical reads. From 01a159717a06aca9587002c6916007637e8f9e6e Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Thu, 15 Oct 2026 14:24:27 -0700 Subject: [PATCH 7/8] Added Optimization section and words about compression --- docs/clickhouse-docs/client.mdx | 100 ++++++++++++++ docs/integration-client.md | 226 +++++++++++++++++++++----------- 2 files changed, 250 insertions(+), 76 deletions(-) diff --git a/docs/clickhouse-docs/client.mdx b/docs/clickhouse-docs/client.mdx index b408ed64a..9fff52654 100644 --- a/docs/clickhouse-docs/client.mdx +++ b/docs/clickhouse-docs/client.mdx @@ -523,6 +523,79 @@ This client version supports: | [Markdown](/reference/formats/Markdown) | - | raw | | [Form](/reference/formats/Form) | raw | - | +## Data Types {#data-types} + +Database data types are mapped to Java types by the reader or writer and generally follow similar rules across formats (though custom or format-specific readers and writers may use different mappings). + +**ClickHouse / Java Type Mapping** + +The following table summarizes typical mappings between ClickHouse database and Java types. Conversion depends on data +format and reader/writer implementation. Following table represents supported conversions when used with ClickHouse binary formats +like `RowBinary` and `Native`. + +Reader Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader` +Writer Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatWriter` + +| ClickHouse Type | Java Type (Reader) | Java Type (Writer) | +|--------------------------------------|---------------------------------------------------------------|-------------------------------------------------------------------------------------| +| `Bool` | `boolean` | `boolean` | +| `Int8` | `byte`, `short`, `int`, `long` | `byte`, `short`, `int`, `long` | +| `UInt8` | `short`, `int`, `long` | `byte`, `short`, `int`, `long` | +| `Int16` | `short`, `int`, `long` | `short`, `int`, `long` | +| `UInt16` | `int`, `long` | `short`, `int`, `long` | +| `Int32` | `int`, `long` | `int`, `long` | +| `UInt32` | `long` | `int`, `long` | +| `Int64` | `long`, `BigInteger` | `long`, `BigInteger` | +| `UInt64` | `BigInteger`, `long` | `long`, `BigInteger` | +| `Int128`, `UInt128` | `BigInteger` | `BigInteger`, `long` | +| `Int256`, `UInt256` | `BigInteger` | `BigInteger`, `long` | +| `BFloat16` | `float`, `double` | `float` | +| `Float32` | `float`, `double` | `float`, `double` | +| `Float64` | `double` | `float`, `double` | +| `Decimal`, `Decimal32`..`Decimal256` | `BigDecimal` | `BigDecimal`, `String` | +| `String`, `FixedString` | `String`, `byte[]` | `String`, `byte[]` | +| `Date` | `LocalDate` | `LocalDate` | +| `Date32` | `LocalDate` | `LocalDate` | +| `DateTime`, `DateTime32` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | +| `DateTime64` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | +| `Time`, `Time64` | `LocalTime`, `LocalDateTime`, `Duration`, `TemporalAmount` | `int` (`Time`), `long`, `BigInteger`, `Instant` (`Time64`) | +| `Interval*` | `TemporalAmount` (`Period` / `Duration`), `Duration` | `Period` (date intervals), `Duration` (time/day intervals) | +| `Enum8`, `Enum16` | `String`, `byte` (`Enum8`), `short` (`Enum16`), `int` | `String`, `byte`, `short`, `int` | +| `UUID` | `UUID`, `String` | `UUID`, `String` | +| `IPv4`, `IPv6` | `Inet4Address`, `Inet6Address`, `String` | `Inet4Address` (`IPv4`), `Inet6Address` (`IPv6`) | +| `Nullable(T)` | `T` or `null` | `T` or `null` | +| `LowCardinality(T)` | `T` | `T` | +| `Array(T)` | `List`, primitive `T[]`, `String[]`, `Object[]` | `List`, `T[]` | +| `Tuple` | `Object[]`, `List` (POJO) | `Object[]`, `List` | +| `Map(K, V)` | `Map` | `Map` | +| `Nested` | `List`, `Object[]` | `List`, `Object[][]` | +| `Point` | `double[]` | `double[]` | +| `Ring`, `LineString`, `MultiPoint` | `double[][]` | `double[][]` | +| `Polygon`, `MultiLineString` | `double[][][]` | `double[][][]` | +| `MultiPolygon` | `double[][][][]` | `double[][][][]` | +| `Geometry` | `double[]` … `double[][][][]` | `double[]` … `double[][][][]` | +| `QBit(T, N)` | `float[]`, `double[]`, `List` | `float[]`, `double[]`, `List` (length `N`) | +| `Variant`, `Dynamic` | `Object` (resolved runtime value) | `Object` (supported scalar/array/map types) | +| `JSON` | `Map`, `String` | `String` | +| `SimpleAggregateFunction(f, T)` | `T` | `T` | + +**Notes:** +- **Signed vs. unsigned integers:** Unsigned types (`UInt8`, `UInt16`, `UInt32`) are widened to the next larger signed Java primitive (`short`, `int`, `long`) on read so values above the signed max do not overflow; `UInt64`, `Int128`, `UInt128`, `Int256`, and `UInt256` use `BigInteger`. Narrower getters (e.g., `getByte` on `UInt8 > 127` or `getLong` on `UInt64 > Long.MAX_VALUE`) throw `ArithmeticException`. +- **Binary strings (`String`, `FixedString`):** By default, strings are decoded as UTF-8 `String` on read. To read raw binary bytes via `getByteArray(...)` (or POJO `byte[]` fields) without UTF-8 transcoding, enable `Client.Builder#binaryStringSupport(true)` (or per-query `binary_string_support`). +- **Dates (`Date`, `Date32`):** Use `java.time.LocalDate` (timezone-agnostic calendar date). Note that `java.sql.Date` is not supported by the `RowBinary` writer. +- **Timestamps (`DateTime`, `DateTime64`):** **Prefer `ZonedDateTime`, `OffsetDateTime`, or `Instant`** over `LocalDateTime`. ClickHouse stores `DateTime` / `DateTime64` as UTC epoch values. Because `LocalDateTime` carries no timezone offset, using it can easily cause timezone confusion: + - On **write**, a `LocalDateTime` is interpreted in the column's declared timezone (e.g., `DateTime('Asia/Tokyo')`), or in **`UTC`** if the column has no timezone parameter — *not* the JVM default timezone. + - On **read**, the reader decodes the timestamp into a `ZonedDateTime` using the column's timezone (or the client's configured timezone), and `getLocalDateTime(...)` simply strips that zone (`toLocalDateTime()`), which may not match the application's local wall clock. + - Using `Instant` or `ZonedDateTime` / `OffsetDateTime` keeps the exact point in time unambiguous across client, server, and column timezones. +- **Nullability:** Reading a `NULL` value through a primitive getter (`getInt`, `getLong`, `getDouble`, `getBoolean`, `getEnum8`, etc.) or into a primitive POJO field throws `NullValueException`. Check `hasValue(...)` first or use boxed/object types. +- **`LowCardinality(T)`:** Treated as a transparent storage modifier by the client. A `LowCardinality(T)` (or `LowCardinality(Nullable(T))`) column is unwrapped to its underlying type `T` and follows the exact same Java type conversion rules on both read and write. +- **Containers (`Array`, `Tuple`, `Map`, `Nested`):** `getList(...)` reads any `Array` as `List`, primitive array getters (`getIntArray`, `getDoubleArray`, etc.) read 1D primitive arrays, and `getObjectArray(...)` reads any 1D or multidimensional array as `Object[]`. `Tuple` is read via `getTuple(...)` as `Object[]` (and accepts `Object[]` or `List` on write and in POJO mapping). +- **Enums:** `Enum8` and `Enum16` can be read/written by label (`String`) or numeric value (`getEnum8`/`getEnum16`/`getInteger`). +- **Decimals:** `Decimal` and `Decimal32`/`64`/`128`/`256` map to `BigDecimal` to preserve scale and precision. + +When reading or writing data programmatically, favor the canonical Java type above for portability. + + ## Insert API {#insert-api} ### insert(String tableName, InputStream data, ClickHouseFormat format) {#insertstring-tablename-inputstream-data-clickhouseformat-format} @@ -1028,6 +1101,33 @@ public Client.Builder configureGlobalTelemetry(Client.Builder builder) { } ``` +### Optimization {#v2-optimization} + +#### Compression modes & supported algorithms {#v2-compression} + +The client supports two wire-level compression mechanisms and an application passthrough mode: + +| Mode | How it works on the wire | Supported algorithms | Read (Server → Client) | Write (Client → Server) | +|------|--------------------------|----------------------|------------------------|-------------------------| +| **ClickHouse Native Block Compression** *(default when `useHttpCompression` is `false`)* | Uses ClickHouse's native 25-byte framed block format with `CityHash128` checksums via query parameters `compress=1` (read) and `decompress=1` (write). | `LZ4`, `ZSTD` ([`CompressionMethod`](../client-v2/src/main/java/com/clickhouse/client/api/CompressionMethod.java)) | **Enabled by default** (`compress=true`). Automatically decompresses both `LZ4` (`0x82`) and `ZSTD` (`0x90`) blocks returned by the server. | **Disabled by default** (`decompress=false`). When enabled, compresses blocks using `ZSTD` by default (configurable to `LZ4` via `compressionMethod(...)`). | +| **HTTP Compression** *(`useHttpCompression(true)`)* | Uses standard HTTP headers (`Accept-Encoding` for reads, `Content-Encoding` for writes) with query parameter `enable_http_compression=1`. | `lz4` *(default)*, `zstd`, `gzip`, `deflate` | Sends `Accept-Encoding: lz4` by default; override per-client or per-query with `httpHeader("Accept-Encoding", "")`. | Sends `Content-Encoding: lz4` by default; override per-client or per-insert with `httpHeader("Content-Encoding", "")`. | +| **Application-Compressed Data** *(`appCompressedData`)* | Streams pre-compressed payloads directly to the server with `enable_http_compression=1` and `Content-Encoding: ` without re-compressing on the client. | `gzip`, `lz4`, `zstd`, `deflate`, `br`, `xz`, `bz2`, `snappy` *(any format supported by the ClickHouse server)* | N/A | Enabled via `InsertSettings.appCompressedData(true, "")`. | + +#### Configuration properties reference + +Compression settings can be configured globally on `Client.Builder` or overridden per operation via `QuerySettings` and `InsertSettings`: + +| Property key (`ClientConfigProperties`) | `Client.Builder` method | Operation-level (`QuerySettings` / `InsertSettings`) | Default | Applies to | Description | +|-----------------------------------------|-------------------------|------------------------------------------------------|---------|------------|-------------| +| `compress` (`COMPRESS_SERVER_RESPONSE`) | `compressServerResponse(boolean)` | `QuerySettings.setOption("compress", boolean)` | `true` | Read | Requests the server to compress query responses. | +| `decompress` (`COMPRESS_CLIENT_REQUEST`) | `compressClientRequest(boolean)` | `InsertSettings.compressClientRequest(boolean)` | `false` | Write | Compresses client request payloads before sending to the server. | +| `compression.method` (`COMPRESSION_METHOD`) | `compressionMethod(CompressionMethod)` | `InsertSettings.setOption("compression.method", CompressionMethod)` | `CompressionMethod.ZSTD` | Write (Native) | Selects `LZ4` or `ZSTD` for native block compression of client requests. | +| `client.use_http_compression` (`USE_HTTP_COMPRESSION`) | `useHttpCompression(boolean)` | `InsertSettings.useHttpCompression(boolean)` / `QuerySettings.setOption("client.use_http_compression", boolean)` | `false` | Read & Write | Switches from ClickHouse native block compression to standard HTTP header-based compression (`enable_http_compression=1`). | +| `app_compressed_data` (`APP_COMPRESSED_DATA`) | `appCompressedData(boolean)` | `InsertSettings.appCompressedData(boolean, String)` | `false` | Write | Indicates the insert stream is already compressed by the application; sets `useHttpCompression(true)` and `Content-Encoding`. | +| `compression.lz4.uncompressed_buffer_size` (`COMPRESSION_LZ4_UNCOMPRESSED_BUF_SIZE`) | `setLZ4UncompressedBufferSize(int)` | `setOption("compression.lz4.uncompressed_buffer_size", int)` | `65536` (64 KB) | Read & Write | Uncompressed block buffer size (in bytes) for native block compression and decompression. | +| `disable_native_compression` (`DISABLE_NATIVE_COMPRESSION`) | `disableNativeCompression(boolean)` | Client-level only | `false` | Read & Write | Disables JNI/native `lz4-java` bindings (`LZ4Factory.fastestInstance()`) and falls back to the pure Java implementation (`LZ4Factory.fastestJavaInstance()`). | + + ### Metrics Reported {#v2-o11y-metrics} #### Operational Metrics diff --git a/docs/integration-client.md b/docs/integration-client.md index b5a52a849..2e49c77d5 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -644,79 +644,10 @@ A clear type system is essential for integrating application components with Cli Start by reviewing the [ClickHouse data types](https://clickhouse.com/docs/reference/data-types/index) (new types are added regularly), then check the client documentation for the list of [supported data types](https://clickhouse.com/docs/integrations/language-clients/java#supported-data-types). If you need support for a type that is not yet covered, feel free to raise an issue in the Java client repository. -Database data types are mapped to Java types by the reader or writer and generally follow similar rules across formats (though custom or format-specific readers and writers may use different mappings). - - -**ClickHouse / Java Type Mapping** - -The following table summarizes typical mappings between ClickHouse database and Java types. Conversion depends on data -format and reader/writer implementation. Following table represents supported conversions when used with ClickHouse binary formats -like `RowBinary` and `Native`. - -Reader Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader` -Writer Interface: `com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatWriter` - -| ClickHouse Type | Java Type (Reader) | Java Type (Writer) | -|--------------------------------------|---------------------------------------------------------------|-------------------------------------------------------------------------------------| -| `Bool` | `boolean` | `boolean` | -| `Int8` | `byte`, `short`, `int`, `long` | `byte`, `short`, `int`, `long` | -| `UInt8` | `short`, `int`, `long` | `byte`, `short`, `int`, `long` | -| `Int16` | `short`, `int`, `long` | `short`, `int`, `long` | -| `UInt16` | `int`, `long` | `short`, `int`, `long` | -| `Int32` | `int`, `long` | `int`, `long` | -| `UInt32` | `long` | `int`, `long` | -| `Int64` | `long`, `BigInteger` | `long`, `BigInteger` | -| `UInt64` | `BigInteger`, `long` | `long`, `BigInteger` | -| `Int128`, `UInt128` | `BigInteger` | `BigInteger`, `long` | -| `Int256`, `UInt256` | `BigInteger` | `BigInteger`, `long` | -| `BFloat16` | `float`, `double` | `float` | -| `Float32` | `float`, `double` | `float`, `double` | -| `Float64` | `double` | `float`, `double` | -| `Decimal`, `Decimal32`..`Decimal256` | `BigDecimal` | `BigDecimal`, `String` | -| `String`, `FixedString` | `String`, `byte[]` | `String`, `byte[]` | -| `Date` | `LocalDate` | `LocalDate` | -| `Date32` | `LocalDate` | `LocalDate` | -| `DateTime`, `DateTime32` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | -| `DateTime64` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime` | `ZonedDateTime`, `OffsetDateTime`, `Instant`, `LocalDateTime`, `java.sql.Timestamp` | -| `Time`, `Time64` | `LocalTime`, `LocalDateTime`, `Duration`, `TemporalAmount` | `int` (`Time`), `long`, `BigInteger`, `Instant` (`Time64`) | -| `Interval*` | `TemporalAmount` (`Period` / `Duration`), `Duration` | `Period` (date intervals), `Duration` (time/day intervals) | -| `Enum8`, `Enum16` | `String`, `byte` (`Enum8`), `short` (`Enum16`), `int` | `String`, `byte`, `short`, `int` | -| `UUID` | `UUID`, `String` | `UUID`, `String` | -| `IPv4`, `IPv6` | `Inet4Address`, `Inet6Address`, `String` | `Inet4Address` (`IPv4`), `Inet6Address` (`IPv6`) | -| `Nullable(T)` | `T` or `null` | `T` or `null` | -| `LowCardinality(T)` | `T` | `T` | -| `Array(T)` | `List`, primitive `T[]`, `String[]`, `Object[]` | `List`, `T[]` | -| `Tuple` | `Object[]`, `List` (POJO) | `Object[]`, `List` | -| `Map(K, V)` | `Map` | `Map` | -| `Nested` | `List`, `Object[]` | `List`, `Object[][]` | -| `Point` | `double[]` | `double[]` | -| `Ring`, `LineString`, `MultiPoint` | `double[][]` | `double[][]` | -| `Polygon`, `MultiLineString` | `double[][][]` | `double[][][]` | -| `MultiPolygon` | `double[][][][]` | `double[][][][]` | -| `Geometry` | `double[]` … `double[][][][]` | `double[]` … `double[][][][]` | -| `QBit(T, N)` | `float[]`, `double[]`, `List` | `float[]`, `double[]`, `List` (length `N`) | -| `Variant`, `Dynamic` | `Object` (resolved runtime value) | `Object` (supported scalar/array/map types) | -| `JSON` | `Map`, `String` | `String` | -| `SimpleAggregateFunction(f, T)` | `T` | `T` | - -**Notes:** -- **Signed vs. unsigned integers:** Unsigned types (`UInt8`, `UInt16`, `UInt32`) are widened to the next larger signed Java primitive (`short`, `int`, `long`) on read so values above the signed max do not overflow; `UInt64`, `Int128`, `UInt128`, `Int256`, and `UInt256` use `BigInteger`. Narrower getters (e.g., `getByte` on `UInt8 > 127` or `getLong` on `UInt64 > Long.MAX_VALUE`) throw `ArithmeticException`. -- **Binary strings (`String`, `FixedString`):** By default, strings are decoded as UTF-8 `String` on read. To read raw binary bytes via `getByteArray(...)` (or POJO `byte[]` fields) without UTF-8 transcoding, enable `Client.Builder#binaryStringSupport(true)` (or per-query `binary_string_support`). -- **Dates (`Date`, `Date32`):** Use `java.time.LocalDate` (timezone-agnostic calendar date). Note that `java.sql.Date` is not supported by the `RowBinary` writer. -- **Timestamps (`DateTime`, `DateTime64`):** **Prefer `ZonedDateTime`, `OffsetDateTime`, or `Instant`** over `LocalDateTime`. ClickHouse stores `DateTime` / `DateTime64` as UTC epoch values. Because `LocalDateTime` carries no timezone offset, using it can easily cause timezone confusion: - - On **write**, a `LocalDateTime` is interpreted in the column's declared timezone (e.g., `DateTime('Asia/Tokyo')`), or in **`UTC`** if the column has no timezone parameter — *not* the JVM default timezone. - - On **read**, the reader decodes the timestamp into a `ZonedDateTime` using the column's timezone (or the client's configured timezone), and `getLocalDateTime(...)` simply strips that zone (`toLocalDateTime()`), which may not match the application's local wall clock. - - Using `Instant` or `ZonedDateTime` / `OffsetDateTime` keeps the exact point in time unambiguous across client, server, and column timezones. -- **Nullability:** Reading a `NULL` value through a primitive getter (`getInt`, `getLong`, `getDouble`, `getBoolean`, `getEnum8`, etc.) or into a primitive POJO field throws `NullValueException`. Check `hasValue(...)` first or use boxed/object types. -- **`LowCardinality(T)`:** Treated as a transparent storage modifier by the client. A `LowCardinality(T)` (or `LowCardinality(Nullable(T))`) column is unwrapped to its underlying type `T` and follows the exact same Java type conversion rules on both read and write. -- **Containers (`Array`, `Tuple`, `Map`, `Nested`):** `getList(...)` reads any `Array` as `List`, primitive array getters (`getIntArray`, `getDoubleArray`, etc.) read 1D primitive arrays, and `getObjectArray(...)` reads any 1D or multidimensional array as `Object[]`. `Tuple` is read via `getTuple(...)` as `Object[]` (and accepts `Object[]` or `List` on write and in POJO mapping). -- **Enums:** `Enum8` and `Enum16` can be read/written by label (`String`) or numeric value (`getEnum8`/`getEnum16`/`getInteger`). -- **Decimals:** `Decimal` and `Decimal32`/`64`/`128`/`256` map to `BigDecimal` to preserve scale and precision. - -When reading or writing data programmatically, favor the canonical Java type above for portability. - - +Database data types are mapped to Java types by the reader or writer and generally follow similar rules across formats (though custom or format-specific readers and writers may use different mappings). +Next read the official client documentation regarding [data types mapping](/docs/clickhouse-docs/client.mdx#data-types). Client API is built around JDK native data types to make +application code more portable. Any additional traits can be implemented with different adaptors. ## Step 6 — Read operations @@ -810,9 +741,9 @@ public void streamEvents(Client client) throws Exception { ### Direct InputStream access -The `QueryResponse` object also provides direct access to the underlying input stream via `response.getInputStream()`. This is incredibly useful when you want to read text formats using third-party libraries (like Jackson for JSON or OpenCSV for CSV). - -Because of this direct stream access, it is also possible to stream columnar formats directly into memory structures like **Apache Arrow**. (See our [examples directory](../examples/src/main/java/com/clickhouse/examples) for reference implementations). +Sometimes it is more efficient to read data without materializing to any of the structures. For example, backend may request data in `JSONEachRow` format and stream it back to backend client (query should be well-designed to avoid +creating a tight coupling on database schema). The `QueryResponse` object also provides direct access to the underlying input stream via `response.getInputStream()`. Such approach is useful to integrate with other libraries like +Jackson to leverage its JSON/CSV parsing capabilities or to stream columnar formats directly into memory structures like **Apache Arrow**. (See our [examples directory](../examples/src/main/java/com/clickhouse/examples) for reference implementations). ### Parameterized queries @@ -842,7 +773,7 @@ public void queryEventsWithParam(Client client, long minId, QuerySettings settin | [`GenericRecord`](../client-v2/src/main/java/com/clickhouse/client/api/query/GenericRecord.java) | Column access by name or index | | [`OperationMetrics`](../client-v2/src/main/java/com/clickhouse/client/api/metrics/OperationMetrics.java) | Server timing and row counts via `QueryResponse.getMetrics()` | -### Operation configuration — tuning heavy reads +### Operation configuration The most useful **client settings** for reads are configured on `QuerySettings`: @@ -869,6 +800,7 @@ See the [Error model](#error-model) for the exception hierarchy and how to unwra | Connection dropped **mid-stream** (after you began iterating) | `DataTransferException` while reading | You cannot resume from the middle — some rows were already consumed. Close the `QueryResponse` and re-run the entire query. Make consumers tolerant of re-reading from the start. | See [integration-testing.md — Fetching Data](integration-testing.md#fetching-data) for read-path test scenarios to cover (data types, formats, failure handling). + --- ## Step 7 — Write operations @@ -1043,6 +975,148 @@ See the [Error model](#error-model) for the exception hierarchy and how to unwra - If retries are not safe for your source, disable them (`retry=0`) and handle re-submission at the application level with a dedup token. --- +## Optimization + +Generally we want to optimize space and time. Memory is the main space in our case. If application reads big data chunks it will cost +memory first of all. Time depends on how much memory we used for request - it unfolds into memory allocation, data processing, +work of GC. All this defines application throughput. + +First we need to picture client operations stages. All starts with request to the client to execute request, and it should find or allocate connection +to DBs. When done client can send/receive data - request send first that actual data comes. Commonly data is processed on application side while +reading operation and on server while write. Each stage can be optimized in own way. + +### Compression + +Network is relatively slow and unstable compare to memory. Reducing number of bytes sent via network reduces number of +expensive interactions with operating system. Compression is the best way to reduce network traffic and request time. + +Here we need to note that historically there are two ways to compress data if using HTTP: +- Data compression - data is compressed by server logic while writing response or reading data from client. There are two algorithms - `LZ4` and +`ZSTD`. More details in [ClickHouse Compression Modes](https://clickhouse.com/docs/guides/clickhouse/data-modelling/compression/compression-modes). +- HTTP compression - available only for HTTP interface (java client support only HTTP right now). [ClickHouse HTTP Interface — Compression](https://clickhouse.com/docs/interfaces/http#compression) — server-side behavior for `compress`, `decompress`, and `enable_http_compression`. + +Java client can use both methods. See [Java Client V2 Configuration — Client configuration](https://clickhouse.com/docs/integrations/language-clients/java/client#client-configuration) and [Compression](https://clickhouse.com/docs/integrations/language-clients/java/client#v2-compression) . By default, **read compression (`compressServerResponse`) is enabled** and **write compression (`compressClientRequest`) is disabled**. You can enable write compression and select the native block algorithm (`LZ4` or `ZSTD`) at client initialization or per operation: + + +#### Configuration examples + + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.ClientConfigProperties; +import com.clickhouse.client.api.CompressionMethod; +import com.clickhouse.client.api.insert.InsertResponse; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.query.QueryResponse; +import com.clickhouse.client.api.query.QuerySettings; +import com.clickhouse.data.ClickHouseFormat; + +public Client.Builder configureNativeCompression(Client.Builder baseClient) { + return baseClient + .compressServerResponse(true) // enabled by default for reads + .compressClientRequest(true) // enable native block compression for writes + .compressionMethod(CompressionMethod.LZ4) // use LZ4 (default is ZSTD) + .setLZ4UncompressedBufferSize(65536); +} + +public void insertWithClientCompression(Client client, InputStream dataStream) throws Exception { + // Enable or override write compression for a single insert operation + InsertSettings settings = new InsertSettings() + .compressClientRequest(true) + .setOption(ClientConfigProperties.COMPRESSION_METHOD.getKey(), CompressionMethod.ZSTD); + + try (InsertResponse response = client.insert("events", dataStream, ClickHouseFormat.RowBinary, settings).get()) { + // handle response metrics + } +} + +public void queryWithoutCompression(Client client) throws Exception { + // Disable server response compression for a specific small query if needed + QuerySettings settings = new QuerySettings() + .setOption(ClientConfigProperties.COMPRESS_SERVER_RESPONSE.getKey(), false); + + try (QueryResponse response = client.query("SELECT * FROM events LIMIT 10", settings).get()) { + // process response + } +} +``` + +**Enabling HTTP compression** + +Enable HTTP compression when routing traffic through HTTP proxies/load balancers that inspect standard HTTP encodings or when you want to use `gzip`, `deflate`, `zstd`, or framed `lz4`. + +When `useHttpCompression(true)` is set, the client defaults to `lz4` (`Accept-Encoding: lz4` for reads and `Content-Encoding: lz4` for writes). To use another algorithm (`gzip`, `zstd`, or `deflate`), set the `Accept-Encoding` or `Content-Encoding` HTTP header: + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.insert.InsertResponse; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.client.api.query.QueryResponse; +import com.clickhouse.client.api.query.QuerySettings; +import com.clickhouse.data.ClickHouseFormat; + +public Client.Builder configureHttpCompression(Client.Builder baseClient) { + return baseClient + .useHttpCompression(true) + .compressServerResponse(true) // sends Accept-Encoding: lz4 by default + .compressClientRequest(true); // sends Content-Encoding: lz4 by default +} + +public void queryWithHttpCompression(Client client, String compressionAlgo) throws Exception { + // compressionAlgo can be "lz4", "zstd", "gzip", or "deflate" + QuerySettings settings = new QuerySettings() + .httpHeader("Accept-Encoding", compressionAlgo); + + try (QueryResponse response = client.query("SELECT * FROM events", settings).get()) { + // response stream is automatically decompressed by the client + } +} + +public void insertWithHttpCompression(Client client, InputStream dataStream, String compressionAlgo) throws Exception { + // compressionAlgo can be "lz4", "zstd", "gzip", or "deflate" + InsertSettings settings = new InsertSettings() + .useHttpCompression(true) + .compressClientRequest(true) + .httpHeader("Content-Encoding", compressionAlgo); + + try (InsertResponse response = client.insert("events", dataStream, ClickHouseFormat.JSONEachRow, settings).get()) { + // handle response metrics + } +} +``` + +**Writing pre-compressed application data** + +If your application already receives or stores compressed files/streams (for example, `.gz` or `.lz4` files in object storage), use `InsertSettings.appCompressedData(true, "")`. The client skips compressing the stream itself and sends the raw compressed bytes directly to ClickHouse with `enable_http_compression=1` and `Content-Encoding: `: + +```java +import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.insert.InsertResponse; +import com.clickhouse.client.api.insert.InsertSettings; +import com.clickhouse.data.ClickHouseFormat; + +public void insertPreCompressedStream(Client client, InputStream gzippedJsonStream) throws Exception { + InsertSettings settings = new InsertSettings() + .appCompressedData(true, "gzip"); + + try (InsertResponse response = client.insert("events", gzippedJsonStream, ClickHouseFormat.JSONEachRow, settings).get()) { + // handle response metrics + } +} +``` + +> **Important notes & constraints:** +> - **Parameterized queries with form data (`useHttpFormDataForQuery(true)`):** When `client.http.use_form_request_for_query` is enabled, query parameters are sent as `multipart/form-data` in the request body. Client-side request compression (`decompress`) is automatically disabled for multipart query requests, while server response compression (`compress`) remains active. +> - **Dependencies in standard vs. shaded jar:** Compression relies on `org.lz4:lz4-pure-java`, `com.github.luben:zstd-jni`, and `org.apache.commons:commons-compress`. In restricted environments where native JNI libraries cannot be loaded, call `disableNativeCompression(true)` on `Client.Builder`. + + +### Connection Pool + +Connection pools is mostly useful when application handles many concurrent request because pool can utilize each connection more (creating a new one +is expensive because of SSL handshake and initialization in the OS). In this case client should have been configured to have enough max number of connections. When limit is reached then all new request will line up and it will significantly increase operation time what will cause timeouts. However making max number of connections may result on uncontrolled pool grow what can overload client. + +See [Client - Connection Pooling](https://clickhouse.com/docs/integrations/language-clients/java/client#connection-pooling) for details about configuration. + ## Step 8 — Metadata & schema discovery **Goal:** obtain table and query schemas using the client API. From 727a60a0f05d51b6007bd3efa1156bc50215dd8c Mon Sep 17 00:00:00 2001 From: Sergey Chernov Date: Thu, 15 Oct 2026 16:01:37 -0700 Subject: [PATCH 8/8] Added about JVM and memory --- docs/integration-client.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/integration-client.md b/docs/integration-client.md index 2e49c77d5..4c6c248bf 100644 --- a/docs/integration-client.md +++ b/docs/integration-client.md @@ -1117,6 +1117,17 @@ is expensive because of SSL handshake and initialization in the OS). In this cas See [Client - Connection Pooling](https://clickhouse.com/docs/integrations/language-clients/java/client#connection-pooling) for details about configuration. + +### JVM Memory and GC + +Data processing consumes memory, and it creates different problems down the road. Java application commonly uses objects to store data and when +reading a lot of data many objects are created on the heap. After processing all these objects should be GC'ed. Usually it happens on the edge of two requests (one finished and one starting). +The new request needs memory for its object and GC has to work hard to free it what may result in long world pause. This creates second problem network gets more data queued for client. + +Selection of proper GC would help and here is [official guide](https://docs.oracle.com/en/java/javase/27/gctuning/introduction-garbage-collection-tuning.html). +Allocating adequate amount of memory for JVM will reduce GC pressure but costs so we recommend load testing to find optimal values. +Using objects pool reduces number of memory allocations however considered unsafe in some cases (to avoid cross-talk, for example, when data of one customers present in objects of another customer). + ## Step 8 — Metadata & schema discovery **Goal:** obtain table and query schemas using the client API.