Retry newly provisioned Query API endpoints until ready

[sdairs]

Summary

• wait for newly provisioned Query API endpoints with a harmless SELECT 1 probe, bounded exponential backoff, and a user-visible status message
• apply the remaining 120-second readiness deadline to every probe and report a generic readiness timeout on exhaustion
• run user SQL once after readiness while preserving idle wake handling and immediate stopped-service failures

Tests

• cargo test -p clickhousectl
• cargo fmt --all --check
• cargo clippy -p clickhousectl --all-targets -- -D warnings

Closes #453

[sdairs]

Carry-over review from PR #504 (old PR for #453). This PR meets the letter of the acceptance criteria and is less invasive (no client.rs test hook; the stopped-service subprocess test is a genuine addition over #504). However it regresses three concerns that were raised and fixed during #504's review (fixes confirmed by sdairs on that PR). Required before merge:

1. Per-attempt deadline (regressed). On Retry newly provisioned Query API endpoints until ready #504, sdairs confirmed: "Fixed in 4938448. Each readiness attempt is now wrapped in tokio::time::timeout with the remaining overall deadline; expiry returns a clear, non-retryable readiness-timeout error." Here (crates/clickhousectl/src/cloud/services.rs, run_just_provisioned_service_query), the 120s budget is only checked between attempts (if now + backoff > deadline { return Err(error); }). An endpoint that accepts the connection but stalls leaves the CLI hung indefinitely — the budget is illusory for a single stuck attempt. Wrap each attempt in tokio::time::timeout(remaining, ...).

2. Non-auth error on budget exhaustion (regressed). On Retry newly provisioned Query API endpoints until ready #504, sdairs confirmed: "Completed retryable readiness probes that consume the overall deadline now return the same explicit 408 timeout…" Here, exhaustion returns the last raw 401/403/404, which convert_error maps to CloudErrorKind::Auth → exit code 4, reporting 2 minutes of propagation delay as "bad credentials". Produce a synthetic non-auth readiness-timeout error ("Query API endpoint did not become ready within …") on expiry and add a test pinning the exhaustion path (there is currently no test covering it).

3. Probe vs user SQL (partially regressed). On Retry newly provisioned Query API endpoints until ready #504, sdairs confirmed: "Newly provisioned endpoints now retry a bounded SELECT 1 readiness probe. ServiceIdle marks the endpoint ready, and the user query runs once with wake handling outside the readiness deadline." Here the loop re-sends the user's SQL on every retry. That is strictly weaker than a harmless probe (and the subprocess test uses SELECT 1 as the user query, so it cannot detect user-SQL repetition). Either retry with a SELECT 1 probe and run the user query once after readiness, or add a test proving the user SQL is not re-executed.

4. Minor: no user-visible "waiting for endpoint" indication; the backoff "test" is arithmetic on constants and never executes the loop.

If these are ported, prefer this PR. Otherwise PR #504 is already approved with all of the above fixed and would need only a trivial stopped-service subprocess test added.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 13dbebe. Configure here.}