Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 13 additions & 34 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,47 +148,30 @@ To turn the worker off, remove `ai` from `COMPOSE_PROFILES` and run `docker comp
| `AI_QUEUE_CONCURRENCY` | int | `5` | AI tasks run at the same time |
| `USE_MOCK_AI` | bool | `false` | Return mock answers without calling an LLM (testing) |

### Test Steps in ClickHouse (Optional)
### Test Steps in ClickHouse

Currents can keep the steps of each test attempt in ClickHouse instead of object storage. It is off by default: until you set a time below, steps stay in object storage. Add the variables to `.env`; every service reads it, so they all get the same values.
Currents stores the steps of each test attempt in the ClickHouse table `test_attempt_steps`. `scheduler` creates the table when it starts. `write-worker` inserts the steps, and `api` and `ai-queue-worker` read them for the dashboard, AI and MCP.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write their steps to ClickHouse and to object storage. The dashboard and AI read them from object storage once Redis no longer has them. Read by `director` and `write-worker` |
| `STEPS_CLICKHOUSE_ONLY_FROM` | timestamp | _(empty)_ | Tests that start at or after this time write steps to ClickHouse only, and read them from there. It wins over `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` when both match. Read by `director` and `write-worker` |
| `STEPS_RETENTION_DAYS` | int | `21` | Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values round up. Applies to steps inserted after `write-worker` restarts. To change it, see [Changing the retention](#changing-the-retention). Read by `write-worker` |

Both times are ISO 8601 with a timezone, such as `2026-12-01T00:00:00Z`. The services read them at startup, so set a time that has not come yet, and recreate the containers before it comes. A service still on the old value chooses the storage differently for tests that start after the new time. Tests that started before the time you set keep their steps in object storage. Steps in ClickHouse are deleted after the retention period.
`write-worker`, `api` and `ai-queue-worker` must reach ClickHouse. They get `CLICKHOUSE_URL` and `CLICKHOUSE_USERNAME` from `.env`, and `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`. To check:

There is no separate test-record ingestion service on Compose: `director` does that work.

To turn it on:

1. Upgrade to images that include this feature (see [Upgrading](./upgrading.md)). `scheduler` creates the `test_attempt_steps` table in ClickHouse when it starts, and the table stays empty until you set a time.
2. Check that `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. They get `CLICKHOUSE_URL` and `CLICKHOUSE_USERNAME` from `.env`, and `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`. `api` and `ai-queue-worker` read steps from ClickHouse, and `write-worker` inserts them:

```bash
docker compose exec write-worker printenv | grep CLICKHOUSE_
docker compose exec api printenv | grep CLICKHOUSE_
docker compose exec ai-queue-worker printenv | grep CLICKHOUSE_
```
```bash
docker compose exec write-worker printenv | grep CLICKHOUSE_
docker compose exec api printenv | grep CLICKHOUSE_
docker compose exec ai-queue-worker printenv | grep CLICKHOUSE_
```

`ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers.
3. Set `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` to a future time in `.env`, and `STEPS_RETENTION_DAYS` if 21 days is not right. Run `docker compose up -d` to recreate the containers.
4. After that time, run tests, then query `test_attempt_steps` in ClickHouse directly for tests that started after it:
`ai-queue-worker` runs only with the `ai` profile (see [AI Explanations and Chat](#ai-explanations-and-chat-optional)). With your own ClickHouse, `CLICKHOUSE_URL` must be reachable from all three containers.

```sql
SELECT record_id, attempt, step_count
FROM default.test_attempt_steps
WHERE started_at >= toDateTime64('2026-12-01 00:00:00', 3, 'UTC')
ORDER BY started_at DESC
LIMIT 20
```
- If `write-worker` cannot reach ClickHouse, the steps of new tests are lost. Nothing else stores them.
- If `api` or `ai-queue-worker` cannot reach ClickHouse, the dashboard, AI and MCP show no steps once Redis no longer has them, after about 2 hours.

Every completed attempt that has steps must have a row, and its `step_count` must match the steps in its object storage file, `<orgId>/steps/<runId>/<recordId>/<attempt>.json.gz`. The dashboard reads object storage for these tests, so it does not show whether ClickHouse has them.
5. Set `STEPS_CLICKHOUSE_ONLY_FROM` to a future time, and run `docker compose up -d` again.
Steps are deleted from ClickHouse after the retention period.

Do not set `STEPS_CLICKHOUSE_ONLY_FROM` unless `write-worker`, `api` and `ai-queue-worker` reach ClickHouse. Without it `write-worker` cannot insert steps and nothing else holds them, so the steps of those tests are lost. The dashboard and AI show no steps when `api` or `ai-queue-worker` cannot read them.
Currents does not read steps from object storage or MongoDB. Step files in the bucket under `<orgId>/steps/<runId>/<recordId>/<attempt>.json.gz` do not show in the dashboard, AI or MCP, so tests whose steps are only there show no steps. You can delete the `<orgId>/steps/` folders from the bucket.

#### Changing the retention

Expand All @@ -201,10 +184,6 @@ docker compose stop write-worker
docker compose up -d write-worker
```

#### Going back

To go back, empty both times in `.env` and run `docker compose up -d`. New tests then write steps to object storage. Tests already stored in ClickHouse are still read from there.

### SAML SSO (Optional)

Enable SAML single sign-on by setting the two required variables and mounting your IdP metadata file into `./data/sso`. See [Enable SAML SSO](sso-saml.md) for the full walkthrough. SSO turns on automatically once both required variables are set — there is no separate enable flag.
Expand Down
17 changes: 4 additions & 13 deletions on-prem/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -208,20 +208,11 @@ OAUTH_DYNAMIC_CLIENT_REGISTRATION_ENABLED=true
# AI_QUEUE_CONCURRENCY=5 # AI tasks run at the same time

# =============================================================================
# Test steps in ClickHouse (Optional)
# Test steps in ClickHouse
# =============================================================================
# Store the steps of each test attempt in ClickHouse. Both times are off while
# unset, and steps stay in object storage. ISO 8601 with a timezone. Every service
# reads this file, so the values reach all of them. Order and checks:
# docs/configuration.md#test-steps-in-clickhouse-optional.
#
# Tests that start at or after this time write steps to ClickHouse and to object
# storage. Set it to a time that has not come yet.
# STEPS_CLICKHOUSE_DUAL_WRITE_FROM=2026-12-01T00:00:00Z
#
# Tests that start at or after this time write steps to ClickHouse only. Needs the
# write-worker, api and ai-queue-worker to reach ClickHouse, or steps are lost.
# STEPS_CLICKHOUSE_ONLY_FROM=2026-12-08T00:00:00Z
# Steps of each test attempt are stored in ClickHouse. write-worker inserts them;
# api and ai-queue-worker read them. See
# docs/configuration.md#test-steps-in-clickhouse.
#
# Days steps stay in ClickHouse: 7, 14, 21, 30, 60, 90, 180 or 365. Other values
# round up. To change it, stop write-worker, wait until it has exited, edit this
Expand Down
14 changes: 12 additions & 2 deletions on-prem/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Breaking Changes
- Test steps are stored in ClickHouse only. `write-worker` inserts the steps of each test attempt into the ClickHouse table `test_attempt_steps`, and `api` and `ai-queue-worker` read them from there. Currents does not write or read steps in object storage or MongoDB. Tests recorded before the upgrade show no steps in the dashboard, AI or MCP. Their step files under `<orgId>/steps/` in the bucket are not read, and you can delete them. See Test Steps in ClickHouse in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse).

### Upgrade Notes
- Before upgrading, check that `write-worker`, `api` and `ai-queue-worker` reach ClickHouse with `CLICKHOUSE_URL`, `CLICKHOUSE_USERNAME` and `CLICKHOUSE_PASSWORD`. If `write-worker` cannot, the steps of new tests are lost. If `api` or `ai-queue-worker` cannot, no steps show once Redis no longer has them, after about 2 hours. The check commands are in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse).
- `scheduler` creates the `test_attempt_steps` table when it starts.

### Compose File Changes
- `ai-queue-worker` gets `CLICKHOUSE_PASSWORD` from `CLICKHOUSE_CURRENTS_PASSWORD`, as `api` and `write-worker` do, so it can read test steps from ClickHouse. Regenerate your compose file if you use custom templates.

### Added
- **Test steps in ClickHouse, off by default.** `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` and `STEPS_CLICKHOUSE_ONLY_FROM` in `.env` choose, by the time a test starts, whether its steps go to ClickHouse as well as object storage, or to ClickHouse only. `STEPS_RETENTION_DAYS` sets how long `write-worker` keeps them (default 21). Nothing changes until you set a time. Needs images that include the feature; see Test Steps in ClickHouse in the [Configuration Reference](../docs/configuration.md#test-steps-in-clickhouse-optional).
### New Environment Variables
- `STEPS_RETENTION_DAYS` (default `21`): days steps stay in ClickHouse. Read by `write-worker`.

### Removed Environment Variables
- `STEPS_CLICKHOUSE_DUAL_WRITE_FROM` and `STEPS_CLICKHOUSE_ONLY_FROM`. The images do not read them. If you set them in `.env`, remove them.

## [2026-11-10-003] - 2026-11-13

Expand Down
2 changes: 1 addition & 1 deletion on-prem/docker-compose.cache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ services:
- .env
environment:
CURRENTS_ENV: onprem
# Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set.
# Reads test steps from ClickHouse.
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
depends_on:
redis:
Expand Down
2 changes: 1 addition & 1 deletion on-prem/docker-compose.database.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ services:
- .env
environment:
CURRENTS_ENV: onprem
# Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set.
# Reads test steps from ClickHouse.
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
depends_on:
mongodb:
Expand Down
2 changes: 1 addition & 1 deletion on-prem/docker-compose.full.yml
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ services:
- .env
environment:
CURRENTS_ENV: onprem
# Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set.
# Reads test steps from ClickHouse.
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}
FILE_STORAGE_FORCE_PATH_STYLE: "true"
depends_on:
Expand Down
2 changes: 1 addition & 1 deletion on-prem/templates/compose.currents.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ services:
- .env
environment:
CURRENTS_ENV: onprem
# Reads test steps from ClickHouse when STEPS_CLICKHOUSE_* is set.
# Reads test steps from ClickHouse.
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD}

networks:
Expand Down