Skip to content

feat: add settings to turn on ClickHouse test steps storage - #45

Merged
twk3 merged 4 commits into
mainfrom
agoldis/steps-clickhouse
Nov 17, 2026
Merged

twk3 merged 4 commits into
mainfrom
agoldis/steps-clickhouse

Conversation

@agoldis

@agoldis agoldis commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Owner: @agoldis
Linear: ENG-2238

Do not release before https://github.com/currents-dev/currents/pull/5007 (allow ClickHouse test steps on-prem) ships in a Currents release. Sibling PR for Helm: currents-dev/helm-charts#81.

What and why

Lets a Compose install turn on test steps in ClickHouse. The variables are commented out in .env.example, so nothing changes until an operator sets them.

  • STEPS_CLICKHOUSE_DUAL_WRITE_FROM, STEPS_CLICKHOUSE_ONLY_FROM and STEPS_RETENTION_DAYS go in .env. Every service has env_file: .env, so director and write-worker get the same values from one place. No per-service entries are needed.
  • ai-queue-worker gets CLICKHOUSE_PASSWORD: ${CLICKHOUSE_CURRENTS_PASSWORD} (see below). The three compose files are regenerated with sync-templates.sh.
  • docs/configuration.md has a new section, "Test Steps in ClickHouse (Optional)", with what each setting does and the order to turn it on. .env.example and on-prem/CHANGELOG.md (Unreleased) are updated.
  • VERSION and the image tag are not changed; the release PR does that.

Service mapping

Name in #5007 This repo (service / image)
director director / director
test-record ingestion service none. Compose has no such service. director runs the test record code itself.
writer write-worker / writer
API api / api
AI queue worker ai-queue-worker / ai-queue-worker (profile ai)

ClickHouse access

  • api and write-worker already get CLICKHOUSE_URL, CLICKHOUSE_USERNAME (from .env) and CLICKHOUSE_PASSWORD.
  • ai-queue-worker got CLICKHOUSE_URL and CLICKHOUSE_USERNAME from .env but no CLICKHOUSE_PASSWORD: the template sets it only on api, changestreams-worker, write-worker and scheduler, and .env holds the password only as CLICKHOUSE_CURRENTS_PASSWORD. Without it the worker could not authenticate to read steps. This PR adds it. This is the one change that applies to existing installs: for those with the ai profile on, the worker's environment gains a password it did not have.

Checks

  • docker compose -f docker-compose.{full,database,cache}.yml config --quiet (Compose v5.6.1): all three valid.
  • on-prem/scripts/sync-templates.sh run; the compose files differ from main only by the new ai-queue-worker line.
  • on-prem/scripts/check-env.sh against a copy of .env.example: no missing variables.
  • Not run: the smoke test and Podman validation (CI does them).

🤖 Generated with Claude Code

https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd

STEPS_CLICKHOUSE_DUAL_WRITE_FROM, STEPS_CLICKHOUSE_ONLY_FROM and
STEPS_RETENTION_DAYS go in .env, which every service reads, so director
and write-worker get the same times. They are commented out in
.env.example, so nothing changes until an operator sets them.

ai-queue-worker reads test steps from ClickHouse but had no
CLICKHOUSE_PASSWORD; it now gets it from CLICKHOUSE_CURRENTS_PASSWORD,
as api and write-worker do. The three compose files are regenerated.

The configuration reference gets the order to turn it on. It needs
Currents images that include currents-dev/currents#5007.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
@agoldis agoldis self-assigned this Nov 15, 2026
@baz-reviewer

baz-reviewer Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Review this PR on Baz

Baz Summary

Enable Compose operators to configure ClickHouse storage for test steps through shared .env settings, including dual-write, ClickHouse-only storage, and retention controls. Update ai-queue-worker credentials, Compose templates, configuration guidance, and release notes so director, write-worker, api, and AI workers can use the feature safely.

Topics

TopicDetails
ClickHouse Step Storage Document and expose opt-in test-step storage settings, rollout order, retention management, validation, and rollback behavior through .env.example and operator guidance.
Modified files (3)
  • docs/configuration.md
  • on-prem/.env.example
  • on-prem/CHANGELOG.md
Latest Contributors(2)
UserCommitDate
agoldis@users.noreply....docs: wait for the wri...November 15, 2026
dj@currents.devrelease: on-prem 2026-...November 13, 2026
Compose Credentials Provide ai-queue-worker with CLICKHOUSE_PASSWORD from CLICKHOUSE_CURRENTS_PASSWORD across generated Compose variants so AI step reads authenticate correctly.
Modified files (4)
  • on-prem/docker-compose.cache.yml
  • on-prem/docker-compose.database.yml
  • on-prem/docker-compose.full.yml
  • on-prem/templates/compose.currents.yml
Latest Contributors(2)
UserCommitDate
agoldis@users.noreply....feat: add settings to ...November 15, 2026
dj@currents.devrelease: on-prem 2026-...November 13, 2026

Merger   ✅ Ready to Merge

MERGE: The PR delivers the documented ClickHouse test-step settings and required worker password wiring. CI ran successfully, and there are no unresolved discussions.



Skills  Activate Skill Maintainer to keep your skills up to date

Planner  This PR would have been improved with Baz Planner - Try it now

agoldis and others added 2 commits November 14, 2026 23:21
The dashboard reads object storage for tests written in dual-write, so
it cannot show whether ClickHouse has their steps. Step 4 now queries
test_attempt_steps and compares step_count with the object storage file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
The writer looks up a record's stored expiry and inserts in separate
queries, so two writers on different retention values can put one test
in two partitions. Changing the value needs all writers stopped first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
@agoldis
agoldis requested a review from twk3 November 15, 2026 06:22
@twk3
twk3 merged commit eaa3221 into main Nov 17, 2026
5 checks passed
@twk3
twk3 deleted the agoldis/steps-clickhouse branch November 17, 2026 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants