diff --git a/docs/configuration.md b/docs/configuration.md index cd1dff8..31573e9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -148,47 +148,30 @@ To turn the worker off, remove `ai` from `COMPOSE_PROFILES` and run `docker comp | `AI_QUEUE_CONCURRENCY` | int | `5` | AI tasks run at the same time | | `USE_MOCK_AI` | bool | `false` | Return mock answers without calling an LLM (testing) | -### Test Steps in ClickHouse (Optional) +### Test Steps in ClickHouse -Currents can keep the steps of each test attempt in ClickHouse instead of object storage. It is off by default: until you set a time below, steps stay in object storage. Add the variables to `.env`; every service reads it, so they all get the same values. +Currents stores the steps of each test attempt in the ClickHouse table `test_attempt_steps`. `scheduler` creates the table when it starts. `write-worker` inserts the steps, and `api` and `ai-queue-worker` read them for the dashboard, AI and MCP. | Variable | Type | Default | Description | |----------|------|---------|-------------| -| `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write their steps to ClickHouse and to object storage. The dashboard and AI read them from object storage once Redis no longer has them. Read by `director` and `write-worker` | -| `STEPS_CLICKHOUSE_ONLY_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write steps to ClickHouse only, and read them from there. It wins over `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` when both match. Read by `director` and `write-worker` | | `STEPS_RETENTION_DAYS` | int | `21` | Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values round up. Applies to steps inserted after `write-worker` restarts. To change it, see [Changing the retention](#changing-the-retention). Read by `write-worker` | -Both times are ISO 8601 with a timezone, such as `2026-12-01T00:00:00Z`. The services read them at startup, so set a time that has not come yet, and recreate the containers before it comes. A service still on the old value chooses the storage differently for tests that start after the new time. Tests that started before the time you set keep their steps in object storage. Steps in ClickHouse are deleted after the retention period. +`write-worker`, `api` and `ai-queue-worker` must reach ClickHouse. They get `CLICKHOUSE_URL` and `CLICKHOUSE_USERNAME` from `.env`, and `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`. To check: -There is no separate test-record ingestion service on Compose: `director` does that work. - -To turn it on: - -1. Upgrade to images that include this feature (see [Upgrading](./upgrading.md)). `scheduler` creates the `test_attempt_steps` table in ClickHouse when it starts, and the table stays empty until you set a time. -2. Check that `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. They get `CLICKHOUSE_URL` and `CLICKHOUSE_USERNAME` from `.env`, and `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`. `api` and `ai-queue-worker` read steps from ClickHouse, and `write-worker` inserts them: - - ```bash - docker compose exec write-worker printenv | grep CLICKHOUSE_ - docker compose exec api printenv | grep CLICKHOUSE_ - docker compose exec ai-queue-worker printenv | grep CLICKHOUSE_ - ``` +```bash +docker compose exec write-worker printenv | grep CLICKHOUSE_ +docker compose exec api printenv | grep CLICKHOUSE_ +docker compose exec ai-queue-worker printenv | grep CLICKHOUSE_ +``` - `ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers. -3. Set `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` to a future time in `.env`, and `STEPS_RETENTION_DAYS` if 21 days is not right. Run `docker compose up -d` to recreate the containers. -4. After that time, run tests, then query `test_attempt_steps` in ClickHouse directly for tests that started after it: +`ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers. - ```sql - SELECT record_id, attempt, step_count - FROM default.test_attempt_steps - WHERE started_at >= toDateTime64('2026-12-01 00:00:00', 3, 'UTC') - ORDER BY started_at DESC - LIMIT 20 - ``` +- If `write-worker` cannot reach ClickHouse, the steps of new tests are lost. Nothing else stores them. +- If `api` or `ai-queue-worker` cannot reach ClickHouse, the dashboard, AI and MCP show no steps once Redis no longer has them, after about 2 hours. - Every completed attempt that has steps must have a row, and its `step_count` must match the steps in its object storage file, `/steps///.json.gz`. The dashboard reads object storage for these tests, so it does not show whether ClickHouse has them. -5. Set `STEPS_CLICKHOUSE_ONLY_FROM` to a future time, and run `docker compose up -d` again. +Steps are deleted from ClickHouse after the retention period. -Do not set `STEPS_CLICKHOUSE_ONLY_FROM` unless `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. Without it `write-worker` cannot insert steps and nothing else holds them, so the steps of those tests are lost. The dashboard and AI show no steps when `api` or `ai-queue-worker` cannot read them. +Currents does not read steps from object storage or MongoDB. Step files in the bucket under `/steps///.json.gz` do not show in the dashboard, AI or MCP, so tests whose steps are only there show no steps. You can delete the `/steps/` folders from the bucket. #### Changing the retention @@ -201,10 +184,6 @@ docker compose stop write-worker docker compose up -d write-worker ``` -#### Going back - -To go back, empty both times in `.env` and run `docker compose up -d`. New tests then write steps to object storage. Tests already stored in ClickHouse are still read from there. - ### SAML SSO (Optional) Enable SAML single sign-on by setting the two required variables and mounting your IdP metadata file into `./data/sso`. See [Enable SAML SSO](sso-saml.md) for the full walkthrough. SSO turns on automatically once both required variables are set — there is no separate enable flag. diff --git a/on-prem/.env.example b/on-prem/.env.example index c1e66b6..999a6bf 100644 --- a/on-prem/.env.example +++ b/on-prem/.env.example @@ -208,20 +208,11 @@ OAUTH_DYNAMIC_CLIENT_REGISTRATION_ENABLED=true # AI_QUEUE_CONCURRENCY=5 # AI tasks run at the same time # ============================================================================= -# Test steps in ClickHouse (Optional) +# Test steps in ClickHouse # ============================================================================= -# Store the steps of each test attempt in ClickHouse. Both times are off while -# unset, and steps stay in object storage. ISO 8601 with a timezone. Every service -# reads this file, so the values reach all of them. Order and checks: -# docs/configuration.md#test-steps-in-clickhouse-optional. -# -# Tests that start at or after this time write steps to ClickHouse and to object -# storage. Set it to a time that has not come yet. -# STEPS_CLICKHOUSE_DUAL_WRITE_FROM=2026-12-01T00:00:00Z -# -# Tests that start at or after this time write steps to ClickHouse only. Needs the -# write-worker, api and ai-queue-worker to reach ClickHouse, or steps are lost. -# STEPS_CLICKHOUSE_ONLY_FROM=2026-12-08T00:00:00Z +# Steps of each test attempt are stored in ClickHouse. write-worker inserts them; +# api and ai-queue-worker read them. See +# docs/configuration.md#test-steps-in-clickhouse. # # Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values # round up. To change it, stop write-worker, wait until it has exited, edit this diff --git a/on-prem/CHANGELOG.md b/on-prem/CHANGELOG.md index cba9aff..7960556 100644 --- a/on-prem/CHANGELOG.md +++ b/on-prem/CHANGELOG.md @@ -6,11 +6,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Breaking Changes +- Test steps are stored in ClickHouse only. `write-worker` inserts the steps of each test attempt into the ClickHouse table `test_attempt_steps`, and `api` and `ai-queue-worker` read them from there. Currents does not write or read steps in object storage or MongoDB. Tests recorded before the upgrade show no steps in the dashboard, AI or MCP. Their step files under `/steps/` in the bucket are not read, and you can delete them. See Test Steps in ClickHouse in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse). + +### Upgrade Notes +- Before upgrading, check that `write-worker`, `api` and `ai-queue-worker` reach ClickHouse with `CLICKHOUSE_URL`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`. If `write-worker` cannot, the steps of new tests are lost. If `api` or `ai-queue-worker` cannot, no steps show once Redis no longer has them, after about 2 hours. The check commands are in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse). +- `scheduler` creates the `test_attempt_steps` table when it starts. + ### Compose File Changes - `ai-queue-worker` gets `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`, as `api` and `write-worker` do, so it can read test steps from ClickHouse. Regenerate your compose file if you use custom templates. -### Added -- **Test steps in ClickHouse, off by default.** `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` and `STEPS_CLICKHOUSE_ONLY_FROM` in `.env` choose, by the time a test starts, whether its steps go to ClickHouse as well as object storage, or to ClickHouse only. `STEPS_RETENTION_DAYS` sets how long `write-worker` keeps them (default 21). Nothing changes until you set a time. Needs images that include the feature; see Test Steps in ClickHouse in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse-optional). +### New Environment Variables +- `STEPS_RETENTION_DAYS` (default `21`): days steps stay in ClickHouse. Read by `write-worker`. + +### Removed Environment Variables +- `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` and `STEPS_CLICKHOUSE_ONLY_FROM`. The images do not read them. If you set them in `.env`, remove them. ## [2026-11-10-003] - 2026-11-13 diff --git a/on-prem/docker-compose.cache.yml b/on-prem/docker-compose.cache.yml index 1977c20..a63a673 100644 --- a/on-prem/docker-compose.cache.yml +++ b/on-prem/docker-compose.cache.yml @@ -161,7 +161,7 @@ services: - .env environment: CURRENTS_ENV: onprem - # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + # Reads test steps from ClickHouse. CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} depends_on: redis: diff --git a/on-prem/docker-compose.database.yml b/on-prem/docker-compose.database.yml index b52f752..6eb5158 100644 --- a/on-prem/docker-compose.database.yml +++ b/on-prem/docker-compose.database.yml @@ -189,7 +189,7 @@ services: - .env environment: CURRENTS_ENV: onprem - # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + # Reads test steps from ClickHouse. CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} depends_on: mongodb: diff --git a/on-prem/docker-compose.full.yml b/on-prem/docker-compose.full.yml index a0097a8..1593693 100644 --- a/on-prem/docker-compose.full.yml +++ b/on-prem/docker-compose.full.yml @@ -195,7 +195,7 @@ services: - .env environment: CURRENTS_ENV: onprem - # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + # Reads test steps from ClickHouse. CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} FILE_STORAGE_FORCE_PATH_STYLE: "true" depends_on: diff --git a/on-prem/templates/compose.currents.yml b/on-prem/templates/compose.currents.yml index 30b3531..954f879 100644 --- a/on-prem/templates/compose.currents.yml +++ b/on-prem/templates/compose.currents.yml @@ -130,7 +130,7 @@ services: - .env environment: CURRENTS_ENV: onprem - # Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set. + # Reads test steps from ClickHouse. CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} networks: