Skip to content

feat: add values to turn on ClickHouse test steps storage - #81

Merged
twk3 merged 5 commits into
mainfrom
agoldis/steps-clickhouse
Nov 17, 2026
Merged

twk3 merged 5 commits into
mainfrom
agoldis/steps-clickhouse

Conversation

@agoldis

@agoldis agoldis commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Owner: @agoldis
Linear: ENG-2238

Do not release before https://github.com/currents-dev/currents/pull/5007 (allow ClickHouse test steps on-prem) ships in a Currents release. Sibling PR for Compose: currents-dev/docker#45.

What and why

Lets a Helm install turn on test steps in ClickHouse. Everything is empty by default and renders nothing: the rendered manifests with default values are identical to main.

  • currents.testSteps.dualWriteFrom and currents.testSteps.onlyFrom render STEPS_CLICKHOUSE_DUAL_WRITE_FROM and STEPS_CLICKHOUSE_ONLY_FROM. One helper (currents.testStepsEnv) adds them to the director and the writer, so both always get the same values.
  • currents.testSteps.retentionDays renders STEPS_RETENTION_DAYS on the writer (7, 14, 21, 30, 60, 90, 180 or 365; rounded up; default 21 in the app).
  • docs/eks/quickstart.md has a new section, "Test Steps in ClickHouse (Optional)", with what each value does and the order to turn it on. The values table in docs/configuration.md is regenerated.
  • CHANGELOG.md has an Unreleased entry. The chart version is not bumped; the release PR does that.

Service mapping

Name in #5007 This chart
director director
test-record ingestion service none. The chart has no such deployment (images are api, change-streams, director, scheduler, writer, webhooks, ai-queue-worker). The director runs the test record code itself.
writer writer
API server
AI queue worker ai-queue-worker

ClickHouse access

No template change was needed. currents.connectionConfigEnv gives every deployment CLICKHOUSE_URL, CLICKHOUSE_USERNAME and CLICKHOUSE_PASSWORD (or the access token) from currents.clickhouse. The server, ai-queue-worker and writer already have them. They can differ only if an install overrides them in global.env or a component's env; the docs tell the operator to check.

Checks

  • helm lint charts/currents -f charts/currents/ci/lint-values.yaml (helm 3.19.0): passed.
  • helm template with default values: output identical to main.
  • helm template with dualWriteFrom, onlyFrom and retentionDays set (by --set and by a values file with an unquoted timestamp): both STEPS_CLICKHOUSE_* appear on director and writer only; STEPS_RETENTION_DAYS on writer only.
  • helm-docs: docs/configuration.md regenerated and committed.
  • Not run: ct install (CI does it).

🤖 Generated with Claude Code

https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd

currents.testSteps.dualWriteFrom and onlyFrom render
STEPS_CLICKHOUSE_DUAL_WRITE_FROM and STEPS_CLICKHOUSE_ONLY_FROM on the
director and the writer from one helper, so both get the same times.
currents.testSteps.retentionDays renders STEPS_RETENTION_DAYS on the
writer. All three are empty by default and render nothing.

The quickstart gets the order to turn it on. It needs Currents images
that include currents-dev/currents#5007.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
@agoldis agoldis self-assigned this Nov 15, 2026
@baz-reviewer

baz-reviewer Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Review this PR on Baz

Baz Summary

Enable Helm users to configure ClickHouse storage for test steps through currents.testSteps, wiring timestamp and retention settings into the director and writer. Document the rollout, ClickHouse access requirements, retention behavior, and rollback process while keeping the feature disabled by default.

Topics

TopicDetails
Document rollout Explain how to enable, validate, retain, and roll back ClickHouse test-step storage, and record the new configuration in the changelog and generated values reference.
Modified files (3)
  • CHANGELOG.md
  • docs/configuration.md
  • docs/eks/quickstart.md
Latest Contributors(2)
UserCommitDate
agoldis@users.noreply....docs: find the writer ...November 15, 2026
dj@currents.devdocs: show how an agen...November 13, 2026
Enable test steps Configure optional ClickHouse test-step storage by propagating dualWriteFrom and onlyFrom to the director and writer, and retentionDays to the writer. Keep defaults empty so existing installations remain unchanged.
Modified files (4)
  • charts/currents/templates/_common.tpl
  • charts/currents/templates/director/deployment.yaml
  • charts/currents/templates/writer/deployment.yaml
  • charts/currents/values.yaml
Latest Contributors(2)
UserCommitDate
agoldis@users.noreply....feat: add values to tu...November 15, 2026
dj@currents.devfeat: serve trace link...November 13, 2026

Merger   ✅ Ready to Merge

MERGE: the chart values, environment wiring, documentation, and changelog deliver the requested ClickHouse test-steps configuration, and the review discussion is settled. CI ran successfully.



Skills  Activate Skill Maintainer to keep your skills up to date

Planner  This PR would have been improved with Baz Planner - Try it now

agoldis and others added 2 commits November 14, 2026 23:21
The dashboard reads object storage for tests written in dual-write, so
it cannot show whether ClickHouse has their steps. Step 4 now queries
test_attempt_steps and compares step_count with the object storage file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
The writer looks up a record's stored expiry and inserts in separate
queries, so two writers on different retention values can put one test
in two partitions. Changing the value needs all writers stopped first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
@agoldis
agoldis requested a review from twk3 November 15, 2026 06:22
Comment thread docs/eks/quickstart.md Resolved
…writer`

The deployment is named after the release, so deploy/currents-writer
does not exist for a release called anything else, and the scale to 0
would silently do nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011baD3jVnrAJyGsQvfW3Bvd
@twk3
twk3 merged commit fdc847c into main Nov 17, 2026
3 checks passed
@twk3
twk3 deleted the agoldis/steps-clickhouse branch November 17, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants