Flaky test: client-v2 cluster legs — testOperationCustomRoles / testClientCustomRoles / testPasswordAuthentication use a new user or role before it reaches every replica #3237

Description

Three client-v2 integration tests fail intermittently on the client-v2 + CH cluster * CI legs. Each test creates a user or role, and then uses it in the next request. In the cluster setup, the next request can go to a different replica, and that replica does not yet know the new user or role.

Affected tests (line numbers on main at 7f2cb57):

• ClientTests.testOperationCustomRoles: GRANT ... TO some_user at ClientTests.java:449, which runs immediately after CREATE USER some_user (line 448).
• ClientTests.testClientCustomRoles: the same sequence at lines 480-481.
• HttpTransportTests.testPasswordAuthentication: CREATE USER some_user (as dba), then a new client logs in as some_user.

Mechanism. The cluster test environment (ClusterClickHouseContainerStarter) starts three replicas behind nginx:

• containers/clickhouse-cluster/config.d/custom_config.xml stores SQL-created users and roles in <user_directories><replicated> (Keeper). The replica that runs CREATE USER / CREATE ROLE / GRANT has the change at once. The other replicas get it later, asynchronously, through a Keeper watch.
• containers/nginx/nginx.conf selects the upstream with hash $http_x_clickhouse_replica_tag consistent;. These tests do not send X-ClickHouse-Replica-Tag, so the hash key is empty and nginx uses round-robin. nginx selects the upstream for each request, so requests on one pooled connection also go to different replicas.

As a result, CREATE USER some_user can run on replica A and the next statement can run on replica B before B has the new user:

• GRANT ... TO some_user on B fails with UNKNOWN_ROLE (the grantee is not found).
• A login as some_user on B fails with AUTHENTICATION_FAILED (the test reports "Failed to authenticate").
• SELECT with role=ROL1&role=ROL2 on B can fail with SET_NON_GRANTED_ROLE (B has the user, but not the grant yet).

The client is not at fault: the server answers correctly for its local state. The defect is in the tests, which assume read-after-write consistency for access entities across replicas. Commit 3103197 ("fixed tests to work with cluster env") already added retry loops for the same lag to ClientTests.testRuntimeCredentialChange. These three tests did not get a change.

The same pattern exists in jdbc-v2. StatementTest.testSettingRole (CREATE USER some_user / GRANT ... TO some_user at lines 569-570) failed with the same message on the jdbc-v2 + ClickHouse Cloud leg, which also has several replicas behind a proxy. That run had many other failures, so it is supporting evidence only: run 39484738955, StatementTest.testSettingRole:570 » SQL Code: 511 ... There is no role 'some_user' ... (version 26.6.1.2326).

Evidence of non-determinism (CI)

The failures are on two PRs whose changes do not touch authentication, roles, or the HTTP routing path: #3191 (JDBC metadata class names for IPv4/IPv6) and #3213 (client-v2 User-Agent).

On the same heads, the other two cluster legs were green (for example, cluster 25.8 and latest on a1d074f8f1, 3abea18227 and d7be3dadb0). The 25.8 leg was green on #3191 head 0e432ce49 (run 39471587436), then red on the next push, which changed only docs.

Steps to reproduce

The failure needs the cluster environment, so I reproduced it outside Testcontainers. I used the repo's own container files from main (keeper, three replicas with clickhouse-cluster/config.d/custom_config.xml + users.d/cluster_quorum.xml, and nginx/nginx.conf) in a plain Docker Compose stack. Then I replayed each test's statements in order through the nginx HTTP port, on one keep-alive connection, without a replica tag.

1. Start keeper, clickhouse-1..3, and nginx with the files above.
2. For 200 iterations, send the statements of testOperationCustomRoles (or the setup of testPasswordAuthentication), then the user query, through nginx.
3. Repeat with the replicas CPU-limited (docker update --cpus 0.25) to simulate a loaded CI runner. Then repeat with every request pinned to one replica (X-ClickHouse-Replica-Tag: pinned-session).

Results (failed iterations / 200):

The local error at the GRANT step matches CI word for word. Pinning the session to one replica removes all failures under the same load, which confirms the routing mechanism.

Error Log or Exception StackTrace

[ERROR]   ClientTests.testOperationCustomRoles:449 » Server Code: 511. DB::Exception: There is no role `some_user` in `user directories`. (UNKNOWN_ROLE) (version 25.8.33.6 (official build))  (queryId= c8e1c27a-076b-4290-af85-607a4004cdc1)
[ERROR]   ClientTests.testClientCustomRoles:481 » Server Code: 511. DB::Exception: There is no role `some_user` in `user directories`. (UNKNOWN_ROLE) (version 26.3.56.2 (official build))
[ERROR]   HttpTransportTests.testPasswordAuthentication:1043 Failed to authenticate

Local equivalents:

Code: 511. DB::Exception: There is no role `some_user` in `user directories`. (UNKNOWN_ROLE) (version 25.8.33.6 (official build))
Code: 516. DB::Exception: some_user: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED)
Code: 512. DB::Exception: Role ROL1 should be granted to set as a current. (SET_NON_GRANTED_ROLE) (version 25.8.33.6 (official build))

Expected Behaviour

The tests pass on every run in the cluster environment. A test that creates a user or role must use it only on a replica that already has it.

Code Example

The statement sequence that fails (from ClientTests.testOperationCustomRoles). Each execute can go to a different replica:

client.execute("CREATE USER some_user IDENTIFIED BY '" + password + "'").get().close(); // replica A
client.execute("GRANT " + rolesList + " TO some_user").get().close();                     // replica B -> UNKNOWN_ROLE

Suggested fix

Make the tests deterministic. Do not add sleeps or blind retries.

1. Pin each affected test to one replica. Use one X-ClickHouse-Replica-Tag value for every client in the test (admin client and user client), for example newClient().httpHeader(ClickHouseHttpProto.HEADER_REPLICA_TAG, tag) with a per-test random tag. The replica that runs the DDL has the entity at once, so the next request on the same replica sees it. HttpTransportTests.testStickyReplica shows that this routing works in the cluster setup. testPasswordAuthentication builds its clients directly with new Client.Builder(), so it needs the header there too.
2. Or wait until the entity is visible on all replicas before using it. For example, poll SELECT count() FROM clusterAllReplicas('default', system.users) WHERE name = 'some_user' until it equals the replica count (use system.role_grants for the GRANT).

Option 1 also lets testRuntimeCredentialChange drop its retry loops. Unique user names per test (instead of the shared some_user) also remove the ACCESS_ENTITY_ALREADY_EXISTS shape, where DROP USER runs on one replica and CREATE USER on another. That alone does not fix the visibility race.

Configuration

Client Configuration

// Test defaults: newClient() / new Client.Builder() against the nginx endpoint, no custom headers.

Environment

• Cloud
• Client version: main at 7f2cb57 (0.12.0-rc1-SNAPSHOT)
• Language version: Java 17 (CI temurin)
• OS: ubuntu-latest (CI); Linux + Docker for the local reproduction

ClickHouse Server

• ClickHouse Server version: 25.8.33.6 and 26.3.56.2 (CI); 25.8.33.6 and 26.3.28.5 (local)
• ClickHouse Server non-default settings, if any: the repo's cluster test configuration (TEST_CLICKHOUSE_CONTAINER=cluster): 3 replicas, Keeper, user_directories = users_xml + replicated, nginx in front.
• CREATE TABLE statements for tables involved: none (access entities only)

Found by Polyglot AI's PR monitor, which saw these failures on unrelated PRs while it monitored our own PRs. I verified the mechanism against live servers (the repo's own cluster configuration, server builds that match CI). This is not a client bug and it does not affect users. It is test debt that makes the cluster legs red on unrelated PRs.