From 37a030a6afc275aa7de2718ac1725a7c23d711c4 Mon Sep 17 00:00:00 2001 From: Andrew Goldis Date: Sat, 14 Nov 2026 23:19:53 -0700 Subject: [PATCH 1/4] feat: add settings to turn on ClickHouse test steps storage STEPS_CLICKHOUSE_DUAL_WRITE_FROM, STEPS_CLICKHOUSE_ONLY_FROM and STEPS_RETENTION_DAYS go in .env, which every service reads, so director and write-worker get the same times. They are commented out in .env.example, so nothing changes until an operator sets them. ai-queue-worker reads test steps from ClickHouse but had no CLICKHOUSE_PASSWORD; it now gets it from CLICKHOUSE_CURRENTS_PASSWORD, as api and write-worker do. The three compose files are regenerated. The configuration reference gets the order to turn it on. It needs Currents images that include currents-dev/currents#5007. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd --- docs/configuration.md | 34 ++++++++++++++++++++++++++ on-prem/.env.example | 20 +++++++++++++++ on-prem/CHANGELOG.md | 6 +++++ on-prem/docker-compose.cache.yml | 2 ++ on-prem/docker-compose.database.yml | 2 ++ on-prem/docker-compose.full.yml | 2 ++ on-prem/templates/compose.currents.yml | 2 ++ 7 files changed, 68 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 1b2574a..015d08d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -148,6 +148,40 @@ To turn the worker off, remove `ai` from `COMPOSE_PROFILES` and run `docker comp | `AI_QUEUE_CONCURRENCY` | int | `5` | AI tasks run at the same time | | `USE_MOCK_AI` | bool | `false` | Return mock answers without calling an LLM (testing) | +### Test Steps in ClickHouse (Optional) + +Currents can keep the steps of each test attempt in ClickHouse instead of object storage. It is off by default: until you set a time below, steps stay in object storage. Add the variables to `.env`; every service reads it, so they all get the same values. + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write their steps to ClickHouse and to object storage. The dashboard and AI read them from object storage once Redis no longer has them. Read by `director` and `write-worker` | +| `STEPS_CLICKHOUSE_ONLY_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write steps to ClickHouse only, and read them from there. It wins over `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` when both match. Read by `director` and `write-worker` | +| `STEPS_RETENTION_DAYS` | int | `21` | Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values round up. Applies to steps inserted after `write-worker` restarts. Read by `write-worker` | + +Both times are ISO 8601 with a timezone, such as `2026-12-01T00:00:00Z`. The services read them at startup, so set a time that has not come yet, and recreate the containers before it comes. A service still on the old value chooses the storage differently for tests that start after the new time. Tests that started before the time you set keep their steps in object storage. Steps in ClickHouse are deleted after the retention period. + +There is no separate test-record ingestion service on Compose: `director` does that work. + +To turn it on: + +1. Upgrade to images that include this feature (see [Upgrading](./upgrading.md)). `scheduler` creates the `test_attempt_steps` table in ClickHouse when it starts, and the table stays empty until you set a time. +2. Check that `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. They get `CLICKHOUSE_URL` and `CLICKHOUSE_USERNAME` from `.env`, and `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`. `api` and `ai-queue-worker` read steps from ClickHouse, and `write-worker` inserts them: + + ```bash + docker compose exec write-worker printenv | grep CLICKHOUSE_ + docker compose exec api printenv | grep CLICKHOUSE_ + docker compose exec ai-queue-worker printenv | grep CLICKHOUSE_ + ``` + + `ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers. +3. Set `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` to a future time in `.env`, and `STEPS_RETENTION_DAYS` if 21 days is not right. Run `docker compose up -d` to recreate the containers. +4. After that time, run tests. Open one in the dashboard more than 2 hours after its last update, when Redis has dropped it, and check that its steps show. Also check that ClickHouse has rows: `SELECT count() FROM test_attempt_steps`. +5. Set `STEPS_CLICKHOUSE_ONLY_FROM` to a future time, and run `docker compose up -d` again. + +Do not set `STEPS_CLICKHOUSE_ONLY_FROM` unless `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. Without it `write-worker` cannot insert steps and nothing else holds them, so the steps of those tests are lost. The dashboard and AI show no steps when `api` or `ai-queue-worker` cannot read them. + +To go back, empty both times in `.env` and run `docker compose up -d`. New tests then write steps to object storage. Tests already stored in ClickHouse are still read from there. + ### SAML SSO (Optional) Enable SAML single sign-on by setting the two required variables and mounting your IdP metadata file into `./data/sso`. See [Enable SAML SSO](sso-saml.md) for the full walkthrough. SSO turns on automatically once both required variables are set — there is no separate enable flag. diff --git a/on-prem/.env.example b/on-prem/.env.example index dcefb6c..4d53411 100644 --- a/on-prem/.env.example +++ b/on-prem/.env.example @@ -207,6 +207,26 @@ OAUTH_DYNAMIC_CLIENT_REGISTRATION_ENABLED=true # EXPLANATION_TRACE_TOOLS= # AI_QUEUE_CONCURRENCY=5 # AI tasks run at the same time +# ============================================================================= +# Test steps in ClickHouse (Optional) +# ============================================================================= +# Store the steps of each test attempt in ClickHouse. Both times are off while +# unset, and steps stay in object storage. ISO 8601 with a timezone. Every service +# reads this file, so the values reach all of them. Order and checks: +# docs/configuration.md#test-steps-in-clickhouse-optional. +# +# Tests that start at or after this time write steps to ClickHouse and to object +# storage. Set it to a time that has not come yet. +# STEPS_CLICKHOUSE_DUAL_WRITE_FROM=2026-12-01T00:00:00Z +# +# Tests that start at or after this time write steps to ClickHouse only. Needs the +# write-worker, api and ai-queue-worker to reach ClickHouse, or steps are lost. +# STEPS_CLICKHOUSE_ONLY_FROM=2026-12-08T00:00:00Z +# +# Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values +# round up. +# STEPS_RETENTION_DAYS=21 + # ============================================================================= # SAML SSO (Optional) # ============================================================================= diff --git a/on-prem/CHANGELOG.md b/on-prem/CHANGELOG.md index 4ccf907..cba9aff 100644 --- a/on-prem/CHANGELOG.md +++ b/on-prem/CHANGELOG.md @@ -6,6 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Compose File Changes +- `ai-queue-worker` gets `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`, as `api` and `write-worker` do, so it can read test steps from ClickHouse. Regenerate your compose file if you use custom templates. + +### Added +- **Test steps in ClickHouse, off by default.** `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` and `STEPS_CLICKHOUSE_ONLY_FROM` in `.env` choose, by the time a test starts, whether its steps go to ClickHouse as well as object storage, or to ClickHouse only. `STEPS_RETENTION_DAYS` sets how long `write-worker` keeps them (default 21). Nothing changes until you set a time. Needs images that include the feature; see Test Steps in ClickHouse in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse-optional). + ## [2026-11-10-003] - 2026-11-13 Feature release: the API serves trace links, so agents can read a failed test's Playwright trace, and AI tasks move to an optional `ai-queue-worker` service. It changes the compose files: pull the new images first, then `docker compose up -d`, as in [Updates with Compose File Changes](../docs/upgrading.md#updates-with-compose-file-changes). From `2026-07-26-00x`, follow the `2026-10-14-001` steps below as well. diff --git a/on-prem/docker-compose.cache.yml b/on-prem/docker-compose.cache.yml index fd5240c..1977c20 100644 --- a/on-prem/docker-compose.cache.yml +++ b/on-prem/docker-compose.cache.yml @@ -161,6 +161,8 @@ services: - .env environment: CURRENTS_ENV: onprem + # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} depends_on: redis: condition: service_started diff --git a/on-prem/docker-compose.database.yml b/on-prem/docker-compose.database.yml index b7c7b1c..b52f752 100644 --- a/on-prem/docker-compose.database.yml +++ b/on-prem/docker-compose.database.yml @@ -189,6 +189,8 @@ services: - .env environment: CURRENTS_ENV: onprem + # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} depends_on: mongodb: condition: service_healthy diff --git a/on-prem/docker-compose.full.yml b/on-prem/docker-compose.full.yml index 7188a1e..a0097a8 100644 --- a/on-prem/docker-compose.full.yml +++ b/on-prem/docker-compose.full.yml @@ -195,6 +195,8 @@ services: - .env environment: CURRENTS_ENV: onprem + # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} FILE_STORAGE_FORCE_PATH_STYLE: "true" depends_on: mongodb: diff --git a/on-prem/templates/compose.currents.yml b/on-prem/templates/compose.currents.yml index 7cff7fb..30b3531 100644 --- a/on-prem/templates/compose.currents.yml +++ b/on-prem/templates/compose.currents.yml @@ -130,6 +130,8 @@ services: - .env environment: CURRENTS_ENV: onprem + # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} networks: default: null From 09688f16110ed320f8428a0595ab3c0256bcb90b Mon Sep 17 00:00:00 2001 From: Andrew Goldis Date: Sat, 14 Nov 2026 23:21:37 -0700 Subject: [PATCH 2/4] docs: check ClickHouse directly before setting the ClickHouse-only time The dashboard reads object storage for tests written in dual-write, so it cannot show whether ClickHouse has their steps. Step 4 now queries test_attempt_steps and compares step_count with the object storage file. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd --- docs/configuration.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 015d08d..220bc8c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -175,7 +175,17 @@ To turn it on: `ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers. 3. Set `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` to a future time in `.env`, and `STEPS_RETENTION_DAYS` if 21 days is not right. Run `docker compose up -d` to recreate the containers. -4. After that time, run tests. Open one in the dashboard more than 2 hours after its last update, when Redis has dropped it, and check that its steps show. Also check that ClickHouse has rows: `SELECT count() FROM test_attempt_steps`. +4. After that time, run tests, then query `test_attempt_steps` in ClickHouse directly for tests that started after it: + + ```sql + SELECT record_id, attempt, step_count + FROM default.test_attempt_steps + WHERE started_at >= toDateTime64('2026-12-01 00:00:00', 3, 'UTC') + ORDER BY started_at DESC + LIMIT 20 + ``` + + Every completed attempt that has steps must have a row, and its `step_count` must match the steps in its object storage file, `/steps///.json.gz`. The dashboard reads object storage for these tests, so it does not show whether ClickHouse has them. 5. Set `STEPS_CLICKHOUSE_ONLY_FROM` to a future time, and run `docker compose up -d` again. Do not set `STEPS_CLICKHOUSE_ONLY_FROM` unless `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. Without it `write-worker` cannot insert steps and nothing else holds them, so the steps of those tests are lost. The dashboard and AI show no steps when `api` or `ai-queue-worker` cannot read them. From ccded6987226d01a85bdf4492f206e2ba1f42765 Mon Sep 17 00:00:00 2001 From: Andrew Goldis Date: Sat, 14 Nov 2026 23:22:20 -0700 Subject: [PATCH 3/4] docs: say how to change the steps retention without two expiry dates The writer looks up a record's stored expiry and inserts in separate queries, so two writers on different retention values can put one test in two partitions. Changing the value needs all writers stopped first. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd --- docs/configuration.md | 14 +++++++++++++- on-prem/.env.example | 3 ++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 220bc8c..60ccc3e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -156,7 +156,7 @@ Currents can keep the steps of each test attempt in ClickHouse instead of object |----------|------|---------|-------------| | `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write their steps to ClickHouse and to object storage. The dashboard and AI read them from object storage once Redis no longer has them. Read by `director` and `write-worker` | | `STEPS_CLICKHOUSE_ONLY_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write steps to ClickHouse only, and read them from there. It wins over `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` when both match. Read by `director` and `write-worker` | -| `STEPS_RETENTION_DAYS` | int | `21` | Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values round up. Applies to steps inserted after `write-worker` restarts. Read by `write-worker` | +| `STEPS_RETENTION_DAYS` | int | `21` | Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values round up. Applies to steps inserted after `write-worker` restarts. To change it, see [Changing the retention](#changing-the-retention). Read by `write-worker` | Both times are ISO 8601 with a timezone, such as `2026-12-01T00:00:00Z`. The services read them at startup, so set a time that has not come yet, and recreate the containers before it comes. A service still on the old value chooses the storage differently for tests that start after the new time. Tests that started before the time you set keep their steps in object storage. Steps in ClickHouse are deleted after the retention period. @@ -190,6 +190,18 @@ To turn it on: Do not set `STEPS_CLICKHOUSE_ONLY_FROM` unless `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. Without it `write-worker` cannot insert steps and nothing else holds them, so the steps of those tests are lost. The dashboard and AI show no steps when `api` or `ai-queue-worker` cannot read them. +#### Changing the retention + +To change `STEPS_RETENTION_DAYS`, stop `write-worker`, change `.env`, then start it. If you run more than one `write-worker`, stop all of them first. Writers restarted one at a time can store two versions of one test with different expiry dates. + +```bash +docker compose stop write-worker +# edit STEPS_RETENTION_DAYS in .env +docker compose up -d write-worker +``` + +#### Going back + To go back, empty both times in `.env` and run `docker compose up -d`. New tests then write steps to object storage. Tests already stored in ClickHouse are still read from there. ### SAML SSO (Optional) diff --git a/on-prem/.env.example b/on-prem/.env.example index 4d53411..876c24d 100644 --- a/on-prem/.env.example +++ b/on-prem/.env.example @@ -224,7 +224,8 @@ OAUTH_DYNAMIC_CLIENT_REGISTRATION_ENABLED=true # STEPS_CLICKHOUSE_ONLY_FROM=2026-12-08T00:00:00Z # # Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values -# round up. +# round up. To change it, stop write-worker, edit this file, then start it; two +# writers on different values can store one test with different expiry dates. # STEPS_RETENTION_DAYS=21 # ============================================================================= From 6be62dd8fbf68143487e287d706fc4bd19bd6ea7 Mon Sep 17 00:00:00 2001 From: Andrew Goldis Date: Sat, 14 Nov 2026 23:22:54 -0700 Subject: [PATCH 4/4] docs: wait for the writers to exit before starting them on a new retention Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd --- docs/configuration.md | 1 + on-prem/.env.example | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 60ccc3e..cd1dff8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -196,6 +196,7 @@ To change `STEPS_RETENTION_DAYS`, stop `write-worker`, change `.env`, then start ```bash docker compose stop write-worker +# Wait until docker compose ps -a shows write-worker as exited, so in-flight inserts finish # edit STEPS_RETENTION_DAYS in .env docker compose up -d write-worker ``` diff --git a/on-prem/.env.example b/on-prem/.env.example index 876c24d..c1e66b6 100644 --- a/on-prem/.env.example +++ b/on-prem/.env.example @@ -224,7 +224,8 @@ OAUTH_DYNAMIC_CLIENT_REGISTRATION_ENABLED=true # STEPS_CLICKHOUSE_ONLY_FROM=2026-12-08T00:00:00Z # # Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values -# round up. To change it, stop write-worker, edit this file, then start it; two +# round up. To change it, stop write-worker, wait until it has exited, edit this +# file, then start it; two # writers on different values can store one test with different expiry dates. # STEPS_RETENTION_DAYS=21