Skip to content

Retry newly provisioned Query API endpoints until ready - #504

Closed
sdairs wants to merge 6 commits into
issue-451-query-provision-singleflightfrom
issue-453-query-endpoint-readiness
Closed

Retry newly provisioned Query API endpoints until ready#504
sdairs wants to merge 6 commits into
issue-451-query-provision-singleflightfrom
issue-453-query-endpoint-readiness

Conversation

@sdairs

@sdairs sdairs commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • retry a newly provisioned Query API endpoint for up to 120 seconds with exponential backoff capped at 5 seconds
  • retry only non-SQL 401, 403, and generic 404 readiness responses; stopped services, SQL errors, rate limits, transport failures, and 5xx responses remain terminal
  • reuse the single key and endpoint upsert from the inherited locked provisioning flow
  • add subprocess wiremock coverage for a 401 -> 403 -> 404 -> success sequence, asserting one key creation and one endpoint upsert

Closes #453

Tests

  • cargo test -p clickhousectl (748 passed)
  • cargo test -p clickhousectl --test cli_request_shape_test service_query_retries_new_endpoint_readiness_without_reprovisioning -- --exact (1 passed)
  • cargo fmt --all --check
  • cargo clippy -p clickhousectl --all-targets -- -D warnings

No live Cloud resources or credentials were used.

Stack

This is the child of issue-451-query-provision-singleflight (PR #493). It targets that parent branch; PR #493 remains unchanged and should merge first.

@sdairs sdairs changed the title Retry newly provisioned query endpoints Retry newly provisioned Query API endpoints until ready Aug 24, 2026
Comment thread crates/clickhousectl/src/cloud/services.rs Outdated
Comment thread crates/clickhousectl/src/cloud/services.rs Outdated
@sdairs
sdairs force-pushed the issue-453-query-endpoint-readiness branch from 75e8a6c to f970de0 Compare August 25, 2026 12:11

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f970de0. Configure here.

Comment thread crates/clickhousectl/src/cloud/services.rs Outdated
#[allow(clippy::too_many_arguments)]
async fn run_basic_service_query(
client: &CloudClient,
client: &clickhouse_cloud_api::Client,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this necessary?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessed this type change specifically: the raw API client was not required. Query credentials are still passed explicitly to run_query, and readiness/idle handling continues to return the raw query response/error for the existing conversion and streaming boundary. The raw type only made .with_query_host(...) convenient in the direct wiremock test.

Commit 6d78b4f restores &CloudClient for both query helpers, keeps the established internal client.api().run_query(...) call, removes client.api() from production call sites, and uses a test-only query-host override to preserve deterministic coverage. Readiness timing, 408 classification, idle/wake behavior, and user-query execution are unchanged.

Verified with cargo test -p clickhousectl query_readiness -- --nocapture, cargo test -p clickhousectl service_query -- --nocapture (including all affected request-shape tests), cargo fmt --all --check, cargo check -p clickhousectl, and cargo clippy -p clickhousectl --all-targets -- -D warnings.

Comment thread crates/clickhousectl/src/cloud/services.rs Outdated

@iskakaushik iskakaushik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sdairs

sdairs commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #537 in stack #563.

@sdairs sdairs closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry newly provisioned Query API endpoints until ready

2 participants