feat(webapp): route ClickHouse reads to an optional read replica#4081
feat(webapp): route ClickHouse reads to an optional read replica#4081nicktrn wants to merge 4 commits into
Conversation
…reads Runs list reads (dashboard list, runs list API, live reload, child-status counts) went through the shared standard client (CLICKHOUSE_URL). Add a dedicated runsList client type backed by RUNS_LIST_CLICKHOUSE_URL so this high-traffic read path can target a read replica without moving ingest or replication writes off CLICKHOUSE_URL. Falls back to CLICKHOUSE_URL when unset, so it is a no-op unless configured.
Read-only clients (logs, query, admin, runsList, engine, realtime) now fall back X_CLICKHOUSE_URL ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL, and the events client uses the reader/writer split so trace/span/log reads hit the replica while event + log inserts stay on the writer. Set CLICKHOUSE_READER_URL once to move all reads off the primary. Writes (events, replication, sessions_replication, standard) always stay on CLICKHOUSE_URL. No-op when CLICKHOUSE_READER_URL is unset.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The run detail page loads a small prev/next navigation list via NextRunListPresenter; point it at the runsList client too so all runs list reads use the same dedicated client.
The events client both inserts events and reads traces/spans/logs through one instance. Give it a dedicated reader URL so reads can move to a replica while inserts stay on EVENTS_CLICKHOUSE_URL. Unlike the read-only clients, it does not fall back to CLICKHOUSE_READER_URL: a write-capable client only moves reads on an explicit opt-in.
| // Optional read replica endpoint. Read-only clients (logs, query, admin, runsList, | ||
| // engine, realtime) and the events client's READ path default to this when their own | ||
| // URL is unset; writes always stay on CLICKHOUSE_URL. Set once to move all reads to a | ||
| // replica. Must share storage with the CLICKHOUSE_URL warehouse. |
There was a problem hiding this comment.
🚩 CLICKHOUSE_READER_URL comment incorrectly claims events reads fall back to it
The comment on CLICKHOUSE_READER_URL (apps/webapp/app/env.server.ts:1630-1633) states: "the events client's READ path default to this when their own URL is unset." However, EVENTS_READER_CLICKHOUSE_URL at apps/webapp/app/env.server.ts:1693 is defined as z.string().optional() with no CLICKHOUSE_READER_URL fallback. The comment on that line (apps/webapp/app/env.server.ts:1692) explicitly says: "No CLICKHOUSE_READER_URL fallback by design." These two comments directly contradict each other. An operator reading the CLICKHOUSE_READER_URL description might believe setting it alone moves all reads (including events/traces/spans/logs) to a replica, but events reads will remain on the primary unless EVENTS_READER_CLICKHOUSE_URL is also explicitly set.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
Adds optional configuration to send ClickHouse read traffic to a separate instance (for example a read replica) while writes stay on the primary
CLICKHOUSE_URL. This lets operators offload read load (runs list, traces, logs, queries) from the cluster that handles inserts. Fully backwards compatible: with nothing new set, every client resolves toCLICKHOUSE_URLexactly as before.What it adds
CLICKHOUSE_READER_URL(optional): a single reader endpoint that the read-only clients fall back to. Read clients resolve<own URL> ?? CLICKHOUSE_READER_URL ?? CLICKHOUSE_URL. The task-events client (which both inserts events and reads traces, spans, and logs) is built as a reader/writer pair so queries use the reader while inserts stay onCLICKHOUSE_URL.RUNS_LIST_CLICKHOUSE_URL(optional): a dedicated client for the runs list (dashboard list, runs list API, live reload, child-status counts), so the highest-traffic read path can target its own instance.Safety
Only read-only clients fall back to the reader: logs, query, admin, runs list, the pending-version lookup, and the realtime run-id resolver. The query page is constrained to read-only (the TSQL parser rejects anything that is not a
SELECT, and areadonlysetting is applied). The task-events client routes inserts to the writer and queries to the reader per method, so a write can never reach the reader. Pure-write clients (event inserts, replication) always useCLICKHOUSE_URL.Note: this PR targets a baseline branch rather than
mainso the diff stays scoped to the read-replica changes. It will be retargeted tomainbefore merge.