From bddbd3c1838c41c80152e91f12c91ac96f434447 Mon Sep 17 00:00:00 2001 From: Tyler Hartwig Date: Thu, 17 Sep 2026 08:37:00 -0400 Subject: [PATCH 1/3] docs: add a load testing guide for serving workloads Explains the read critical path (balancerd -> environmentd/coordinator -> cluster replica), which levers move throughput and latency (fast path eligibility, prepared statements, isolation level, concurrency knee, cluster/environmentd/balancerd sizing), how to build a load generator that doesn't bottleneck the test (CPU throttling, single-process caps, tunnels), server-side latency cross-checks via mz_recent_activity_log, and write-side/CDC test guidance (hydration, freshness, memory headroom). Written primarily for self-managed deployments but applicable to Cloud (the environmentd/balancerd sections are marked self-managed-only). Distills findings from a field load-testing engagement where every initial bottleneck turned out to be in the harness or proxy layer, not in query execution. Co-Authored-By: Claude Fable 5 --- .../content/serve-results/load-testing.md | 351 ++++++++++++++++++ 1 file changed, 351 insertions(+) create mode 100644 doc/user/content/serve-results/load-testing.md diff --git a/doc/user/content/serve-results/load-testing.md b/doc/user/content/serve-results/load-testing.md new file mode 100644 index 0000000000000..967e36ce68612 --- /dev/null +++ b/doc/user/content/serve-results/load-testing.md @@ -0,0 +1,351 @@ +--- +title: "Load testing" +description: "How to load test Materialize: what sits in the read critical path, how to build a trustworthy load generator, and how to reach the best throughput and latency." +menu: + main: + parent: serve-results + weight: 47 + name: "Load testing" +--- + +This guide explains how to design a load test that measures what Materialize +can actually do: what sits in the critical path of a query, which knobs move +throughput and latency, and — just as importantly — how to build a load +generator that doesn't become the bottleneck itself. + +In our experience, the majority of disappointing load-test results are not +produced by Materialize. They are produced by the test harness: an +under-provisioned load generator, an unindexed query compiling a new dataflow +on every request, or a client re-parsing the same statement millions of times. +This guide helps you rule all of that out, so that whatever number you end up +with is a number you can trust. + +## The read critical path + +Every `SELECT` travels this path: + +``` +client / driver + │ (network, TLS) + ▼ +balancerd ── pgwire proxy, TLS termination + │ + ▼ +environmentd ── parse & plan, timestamp selection, + │ peek dispatch (the "coordinator") + ▼ +cluster replica ── reads the result out of an + │ index (arrangement) in memory + ▼ +back through environmentd → balancerd → client +``` + +Two properties of this path shape everything else in this guide: + +1. **For well-indexed queries, per-query cost is dominated by coordination, + not computation.** A *fast path* query (see below) is answered by looking + up a key in an index the cluster already maintains — microseconds of + replica work. Most of the remaining per-query cost is protocol work in the + coordinator: parsing, planning, and timestamp selection. This is why the + biggest wins in this guide are protocol-level (prepared statements, query + shape), not hardware-level. + +2. **The coordinator sequences all queries in the environment.** Throughput + eventually plateaus at the rate the coordinator can sequence statements, + no matter how many serving clusters or client connections you add. Past + that plateau, additional concurrency only adds queueing latency. + +### What you control + +| Lever | Materialize Cloud | Self-managed | +|---|---|---| +| Query shape / indexes (fast path) | ✔ | ✔ | +| Prepared statements | ✔ | ✔ | +| Isolation level | ✔ | ✔ | +| Connection count / pooling | ✔ | ✔ | +| Cluster (replica) sizes | ✔ | ✔ | +| Load generator design | ✔ | ✔ | +| `environmentd` resources | managed for you | ✔ | +| `balancerd` resources | managed for you | ✔ | + +Everything in the top section of the table applies identically to Cloud and +self-managed deployments — and those levers are where most of the performance +lives. + +## Make every serving query a fast path query + +Materialize executes a `SELECT` in one of two ways: + +- **Fast path**: the query can be answered by an equality lookup against an + existing [index](/fundamentals/concepts/indexes/). No dataflow is built; the coordinator + sends a "peek" for the key to the serving replica, which reads the rows out + of the in-memory arrangement. +- **Standard path**: the query cannot be served from an existing index as + written, so Materialize compiles a **temporary dataflow** on the cluster, + runs it, returns the result, and tears it down. This is correct but orders + of magnitude more expensive per query. + +A serving workload that is 90% fast path and 10% standard path will spend +most of its cluster CPU on the 10%. Before load testing, verify **every** +query in your serving mix: + +```mzsql +EXPLAIN SELECT * FROM winning_bids WHERE item_id = 42; +``` + +Fast path queries say so explicitly: + +``` +Explained Query (fast path): + ... + →Index Lookup on materialize.public.winning_bids ... +``` + +If a query you expect to be cheap is *not* fast path, reshape the maintained +objects so that it is: + +- **Index the exact equality key the query filters on.** A filter on + `customer_id` needs an index on `customer_id` — an index on a different + column, or on an expression, will not be used for the lookup. +- **Push work into the view, not the query.** A query that unions, joins, or + aggregates at `SELECT` time compiles a dataflow. Define a view that + produces the final shape, index it on the lookup key, and query that view + with a simple equality filter. For example, a "find all edges touching this + node, in either direction" query written as `... WHERE src = $1 UNION ALL + ... WHERE dst = $1` is standard path; a view that pre-unions both + directions keyed by a single `node_id` column, indexed on `node_id`, makes + the same question a fast path lookup. +- **Isolate unavoidable ad-hoc queries.** If your workload genuinely includes + ad-hoc joins or traversals, run them on a separate cluster so their + dataflow cost cannot interfere with the latency of the fast path serving + traffic. + +## Use prepared statements + +For fast path workloads, **parsing and planning is typically the single +largest per-query cost** — and it is pure overhead when the application sends +the same parameterized statement over and over, as API and GraphQL resolver +layers do. + +[Prepared statements](https://www.postgresql.org/docs/current/sql-prepare.html) +let the client parse and plan once per connection, then execute repeatedly +with different parameters. In our load testing, switching a keyed-lookup +workload from per-query parsing to prepared statements increased sustainable +throughput by more than half and reduced median latency by an order of +magnitude, on identical infrastructure. + +Most PostgreSQL drivers support prepared statements natively (often +automatically for repeated statements — check your driver's documentation). +Two things to verify: + +- **Your benchmark client uses them.** Many load-testing harnesses issue each + query as a fresh parse+bind+execute. If your production application will + use prepared statements, a harness that doesn't is understating what your + deployment can do. +- **Your connection pooler preserves them.** Poolers in transaction-pooling + mode may not support named prepared statements across transactions. See + [Connection pooling](/serve-results/connection-pooling/) for configuration + guidance. + +## Choose the isolation level deliberately + +Materialize defaults to [strict +serializable](/serve-results/isolation-level/) isolation, which guarantees +linearizability at the cost of extra coordination on every read. Many serving +workloads — dashboards, entity lookups, API reads that don't need to +read-their-own-writes across sessions — are well served by `serializable`, +which is cheaper on the read path: + +```mzsql +SET transaction_isolation = 'serializable'; +``` + +Whichever level you choose, **pin it explicitly in the load test and record +it with your results**. Isolation level materially affects both latency and +throughput, and an unstated default makes results impossible to compare. + +## Understand concurrency: more connections ≠ more throughput + +A closed-loop load test (every user waits for a response before sending the +next query) obeys Little's law: + +``` +throughput ≈ concurrent connections ÷ average latency +``` + +Once you saturate the environment's statement pipeline, throughput stops +rising — and every additional connection simply waits longer. The signature +is unmistakable: doubling users leaves throughput flat while median latency +doubles, and *every* query type inflates by the same amount regardless of its +cost (because the wait happens in a shared queue before execution). + +Sweep concurrency to find the knee: run the same fixed-duration test at +increasing connection counts and plot throughput and p50/p99. The best +operating point is the lowest concurrency that reaches the plateau — +typically a modest number (low hundreds of connections, not thousands). +Beyond the knee you are purchasing latency, not throughput. + +## Size the infrastructure for serving + +### Serving clusters: size for state, not for QPS + +Fast path lookups barely use the serving cluster — the arrangements are +already maintained, and a peek is a keyed read. In our testing, the same fast +path workload measured **identical throughput on clusters four times apart in +size**, with the larger replica nearly idle. Cluster size matters for: + +- the **memory** to hold your indexed views and indexes (watch sustained + memory utilization — see the operational notes below), +- **standard path** queries and the maintenance of the dataflows themselves, +- **hydration** (rebuilding state after a restart or resize). + +Do not scale a serving cluster up in the hope of raising fast path QPS; the +plateau lives in the coordinator, not the replica. Conversely, per-peek +coordination overhead grows slightly with worker count, so an oversized +serving cluster can even cost a little throughput on peek-heavy workloads. + +### Self-managed: `environmentd` and `balancerd` + +{{< note >}} +This section applies to self-managed deployments only. In Materialize Cloud, +these components are sized and managed by Materialize. +{{< /note >}} + +- **`environmentd`** hosts the coordinator, and its CPU is where fast path + throughput saturates. Watch `environmentd` container CPU during the test: + if it is pinned near its allocation while serving clusters idle, it is the + ceiling. Give it dedicated headroom (several full cores; production serving + deployments commonly run 8–16 CPUs) and note that coordinator scaling is + sublinear — at high statement rates, reducing per-statement work (prepared + statements, fast path shapes) buys more than adding cores. +- **`balancerd`** terminates TLS and proxies every byte of every session. + Give it a real CPU allocation (at least 1 CPU per replica, 2+ replicas) + and **do not set CPU limits** on it. Its memory scales with connection + count — budget for your peak connection count, and check the pods' + restart counts after load tests: a periodically OOM-killed proxy + manufactures latency and connection errors that look like server problems. + +{{< warning >}} +A CPU-throttled container **looks idle on usage graphs**. CPU usage can never +exceed the configured limit, so a starved `balancerd` (or load generator) +shows a calm, flat line while requests queue. To detect throttling, look at +the CFS throttling counters (`container_cpu_cfs_throttled_periods_total` in +cAdvisor/Prometheus metrics), not at CPU usage. +{{< /warning >}} + +## Build a load generator you can trust + +The load generator is the least glamorous part of the test and the most +common source of wrong conclusions. The failure mode is always the same: the +generator saturates, requests queue *inside the client*, and the queueing is +reported as "database latency." + +- **Never CPU-limit the generator.** Kubernetes CPU limits and + container-platform vCPU allocations (for example, Azure Container Apps + enforces its vCPU setting as a hard limit, defaulting to 0.5 vCPU) throttle + the client in ~100 ms scheduling quanta. The symptom is a hard throughput + plateau with uniformly inflated latency across all query types — while the + client's own CPU graph looks unremarkable (see the warning above). +- **Distribute across processes.** Python-based tools such as Locust are + effectively single-core per process; a single process saturates at a few + hundred to a couple thousand requests per second depending on the driver + and query mix. Run multiple worker processes (Locust's distributed mode, or + simply several independent processes) and keep each process's user count + modest. Locust prints `CPU usage above 90%` when it is the bottleneck — + treat that warning as invalidating the run. +- **Ramp gradually and warm up.** Establishing hundreds of TLS connections + at once is itself a load spike (and on a throttled client can starve the + test entirely). Ramp connections over tens of seconds, and exclude the + ramp/warmup window from the measured results. +- **Keep the network path honest.** Run the generator in the same region and + network as Materialize. Never measure through developer tunnels + (`kubectl port-forward`, SSH tunnels): a single forwarded TCP stream caps + out far below what the deployment can do. +- **Load shared test data once.** If each simulated user bootstraps its own + key pool with a table scan, a large ramp becomes a self-inflicted load + spike that pollutes the measurement window. Load pools once per process + and share them. + +## Measure server-side, not just client-side + +The single most valuable cross-check in any load test: compare the latency +the **server** measured against the latency the **client** reported, during +the same window. + +```mzsql +-- Statement latency as measured inside Materialize (last 60s). +SELECT + count(*) AS statements, + round(avg(extract(epoch FROM finished_at - began_at)) * 1000, 1) AS avg_ms, + round(max(extract(epoch FROM finished_at - began_at)) * 1000, 1) AS max_ms +FROM mz_internal.mz_recent_activity_log +WHERE began_at > now() - INTERVAL '60 seconds' + AND sql LIKE 'SELECT%' + AND finished_status = 'success'; +``` + +If the server reports single-digit milliseconds while your load tool reports +hundreds, the time is being spent in the client, the network path, or a +throttled proxy — not in query execution. (Note that statement logging is +sampled; the sample is ample for a latency cross-check.) The +[Console](/developer-tools/console/)'s Query History provides the same information +interactively. + +Alongside client metrics, record during every run: + +- `environmentd` CPU (self-managed) — the fast path ceiling indicator, +- serving cluster CPU and memory (`mz_internal.mz_cluster_replica_utilization`), +- `balancerd` CPU, memory, and restart count (self-managed), +- the generator's own CPU and any throttling counters. + +## Load testing the write side (CDC / ingestion) + +Read QPS is half the story; the other half is whether results stay fresh +while the source database is busy. When testing combined read + write load: + +- **Hydrate fully before measuring.** A cluster that is still hydrating (or + a source still snapshotting) is not in steady state. Check + `mz_internal.mz_hydration_statuses` and your source's snapshot progress + before starting the measurement window. +- **Measure freshness, not just throughput.** Track + `mz_internal.mz_wallclock_global_lag` for the objects you serve from, + and report it alongside read latency. Bounded single-digit-second lag + under sustained write load is the healthy pattern; lag that climbs without + recovering means the ingestion or transformation clusters need attention. +- **Watch transformation-cluster memory, not just CPU.** Sustained memory + above ~90% on a cluster is the primary early-warning signal — query latency + and freshness often look healthy right up until an out-of-memory restart. + Size for headroom before pushing write rates. +- **Mind source-side retention windows.** For CDC sources (e.g. SQL Server), + ensure the retention window on the source comfortably exceeds any planned + pause or outage during a multi-day test, or the source may become + unrecoverable and require a re-snapshot. +- **Expect the source database to be a bottleneck too.** At high write rates, + the upstream database's own CPU, log throughput, and engine settings often + cap the test before Materialize does. Record source-side metrics with the + same care as Materialize-side ones. + +## Checklist + +Before trusting a load-test number: + +- [ ] Every serving query shows `Explained Query (fast path)` under `EXPLAIN`. +- [ ] The client uses prepared statements (and the pooler preserves them). +- [ ] Isolation level is set explicitly and recorded. +- [ ] Concurrency was swept; you're reporting the knee, not an arbitrary point. +- [ ] The generator is distributed, un-throttled, in-region, and never + printed a CPU warning. +- [ ] Server-side latency (`mz_recent_activity_log`) was compared against + client-reported latency. +- [ ] (Self-managed) `environmentd` CPU and `balancerd` + CPU/memory/restarts were captured; no CPU limits on `balancerd`. +- [ ] (With writes) clusters were fully hydrated first; freshness lag and + transformation-cluster memory were recorded. + +## Related pages + +- [Indexes](/fundamentals/concepts/indexes/) +- [Isolation levels](/serve-results/isolation-level/) +- [Connection pooling](/serve-results/connection-pooling/) +- [Optimize query results serving](/transform-data/optimization/) +- [Observability](/observability/) From 0fa27671e60660ffabe9e4c52b05de1765d292d0 Mon Sep 17 00:00:00 2001 From: Tyler Hartwig Date: Thu, 17 Sep 2026 08:42:03 -0400 Subject: [PATCH 2/3] docs: drop prepared-statement guidance; scope cluster sizing by execution path Prepared statements don't move the needle much on Materialize's fast path (the earlier internal benchmark conflated the driver/harness change with statement preparation), so remove that section rather than overstate it. Rework the serving-cluster sizing guidance to depend on execution path: fast path workloads should size for state (cluster size doesn't raise peek QPS), while standard path workloads scale QPS directly with cluster compute. Co-Authored-By: Claude Fable 5 --- .../content/serve-results/load-testing.md | 91 ++++++++----------- 1 file changed, 36 insertions(+), 55 deletions(-) diff --git a/doc/user/content/serve-results/load-testing.md b/doc/user/content/serve-results/load-testing.md index 967e36ce68612..235fc70f1ff96 100644 --- a/doc/user/content/serve-results/load-testing.md +++ b/doc/user/content/serve-results/load-testing.md @@ -15,8 +15,8 @@ generator that doesn't become the bottleneck itself. In our experience, the majority of disappointing load-test results are not produced by Materialize. They are produced by the test harness: an -under-provisioned load generator, an unindexed query compiling a new dataflow -on every request, or a client re-parsing the same statement millions of times. +under-provisioned load generator, a CPU-throttled proxy, or an unindexed query +compiling a new dataflow on every request. This guide helps you rule all of that out, so that whatever number you end up with is a number you can trust. @@ -47,20 +47,21 @@ Two properties of this path shape everything else in this guide: up a key in an index the cluster already maintains — microseconds of replica work. Most of the remaining per-query cost is protocol work in the coordinator: parsing, planning, and timestamp selection. This is why the - biggest wins in this guide are protocol-level (prepared statements, query - shape), not hardware-level. + biggest wins in this guide come from query and workload shape, not from + hardware. -2. **The coordinator sequences all queries in the environment.** Throughput - eventually plateaus at the rate the coordinator can sequence statements, - no matter how many serving clusters or client connections you add. Past - that plateau, additional concurrency only adds queueing latency. +2. **The coordinator sequences all queries in the environment.** For fast + path workloads, throughput eventually plateaus at the rate the coordinator + can sequence statements, no matter how many serving clusters or client + connections you add. Past that plateau, additional concurrency only adds + queueing latency. (Standard path workloads usually saturate the serving + cluster first — see the sizing section below.) ### What you control | Lever | Materialize Cloud | Self-managed | |---|---|---| | Query shape / indexes (fast path) | ✔ | ✔ | -| Prepared statements | ✔ | ✔ | | Isolation level | ✔ | ✔ | | Connection count / pooling | ✔ | ✔ | | Cluster (replica) sizes | ✔ | ✔ | @@ -120,33 +121,6 @@ objects so that it is: dataflow cost cannot interfere with the latency of the fast path serving traffic. -## Use prepared statements - -For fast path workloads, **parsing and planning is typically the single -largest per-query cost** — and it is pure overhead when the application sends -the same parameterized statement over and over, as API and GraphQL resolver -layers do. - -[Prepared statements](https://www.postgresql.org/docs/current/sql-prepare.html) -let the client parse and plan once per connection, then execute repeatedly -with different parameters. In our load testing, switching a keyed-lookup -workload from per-query parsing to prepared statements increased sustainable -throughput by more than half and reduced median latency by an order of -magnitude, on identical infrastructure. - -Most PostgreSQL drivers support prepared statements natively (often -automatically for repeated statements — check your driver's documentation). -Two things to verify: - -- **Your benchmark client uses them.** Many load-testing harnesses issue each - query as a fresh parse+bind+execute. If your production application will - use prepared statements, a harness that doesn't is understating what your - deployment can do. -- **Your connection pooler preserves them.** Poolers in transaction-pooling - mode may not support named prepared statements across transactions. See - [Connection pooling](/serve-results/connection-pooling/) for configuration - guidance. - ## Choose the isolation level deliberately Materialize defaults to [strict @@ -187,22 +161,30 @@ Beyond the knee you are purchasing latency, not throughput. ## Size the infrastructure for serving -### Serving clusters: size for state, not for QPS - -Fast path lookups barely use the serving cluster — the arrangements are -already maintained, and a peek is a keyed read. In our testing, the same fast -path workload measured **identical throughput on clusters four times apart in -size**, with the larger replica nearly idle. Cluster size matters for: - -- the **memory** to hold your indexed views and indexes (watch sustained - memory utilization — see the operational notes below), -- **standard path** queries and the maintenance of the dataflows themselves, -- **hydration** (rebuilding state after a restart or resize). - -Do not scale a serving cluster up in the hope of raising fast path QPS; the -plateau lives in the coordinator, not the replica. Conversely, per-peek -coordination overhead grows slightly with worker count, so an oversized -serving cluster can even cost a little throughput on peek-heavy workloads. +### Serving clusters: size for the work your queries actually do + +How cluster size relates to QPS depends entirely on which execution path your +workload takes: + +- **Fast path workloads: size for state, not QPS.** Fast path lookups barely + use the serving cluster — the arrangements are already maintained, and a + peek is a keyed read. In our testing, the same fast path workload measured + **identical throughput on clusters four times apart in size**, with the + larger replica nearly idle; the throughput plateau lived in the coordinator. + For these workloads, size the cluster for the **memory** to hold your + indexed views (watch sustained memory utilization) and for **hydration** + (rebuilding state after a restart or resize) — not for query throughput. + Per-peek coordination overhead even grows slightly with worker count, so an + oversized cluster can cost a little throughput on peek-heavy workloads. +- **Standard path workloads: cluster CPU is the throughput.** Every standard + path query compiles and runs a dataflow on the cluster, so QPS scales with + the cluster's compute — here, scaling the cluster up (or isolating these + queries on their own cluster) directly raises throughput and protects the + latency of any fast path traffic sharing the environment. + +If your mix contains both, measure them separately: the standard path fraction +will dominate cluster CPU, and its ceiling responds to cluster sizing, while +the fast path fraction's ceiling does not. ### Self-managed: `environmentd` and `balancerd` @@ -216,8 +198,8 @@ these components are sized and managed by Materialize. if it is pinned near its allocation while serving clusters idle, it is the ceiling. Give it dedicated headroom (several full cores; production serving deployments commonly run 8–16 CPUs) and note that coordinator scaling is - sublinear — at high statement rates, reducing per-statement work (prepared - statements, fast path shapes) buys more than adding cores. + sublinear — at high statement rates, reducing per-statement work (fast path + query shapes) often buys more than adding cores. - **`balancerd`** terminates TLS and proxies every byte of every session. Give it a real CPU allocation (at least 1 CPU per replica, 2+ replicas) and **do not set CPU limits** on it. Its memory scales with connection @@ -330,7 +312,6 @@ while the source database is busy. When testing combined read + write load: Before trusting a load-test number: - [ ] Every serving query shows `Explained Query (fast path)` under `EXPLAIN`. -- [ ] The client uses prepared statements (and the pooler preserves them). - [ ] Isolation level is set explicitly and recorded. - [ ] Concurrency was swept; you're reporting the knee, not an arbitrary point. - [ ] The generator is distributed, un-throttled, in-region, and never From 9cdf757d3014220082499fcd8d2645bce2cec198 Mon Sep 17 00:00:00 2001 From: Tyler Hartwig Date: Thu, 17 Sep 2026 09:05:02 -0400 Subject: [PATCH 3/3] docs: reframe load testing guide as a diagnostic guide Remove assumptions about where slowdowns usually live (harness-blame framing, environmentd throughput-plateau claims, 'ceiling' language). Each section now describes what to measure and how to interpret the measurement, rather than presuming which component is the bottleneck. Co-Authored-By: Claude Fable 5 --- .../content/serve-results/load-testing.md | 100 ++++++++---------- 1 file changed, 43 insertions(+), 57 deletions(-) diff --git a/doc/user/content/serve-results/load-testing.md b/doc/user/content/serve-results/load-testing.md index 235fc70f1ff96..4c995da039feb 100644 --- a/doc/user/content/serve-results/load-testing.md +++ b/doc/user/content/serve-results/load-testing.md @@ -1,6 +1,6 @@ --- title: "Load testing" -description: "How to load test Materialize: what sits in the read critical path, how to build a trustworthy load generator, and how to reach the best throughput and latency." +description: "How to load test Materialize: what sits in the read critical path, how to measure each component so latency is attributed correctly, and which levers to adjust based on what you find." menu: main: parent: serve-results @@ -8,17 +8,15 @@ menu: name: "Load testing" --- -This guide explains how to design a load test that measures what Materialize -can actually do: what sits in the critical path of a query, which knobs move -throughput and latency, and — just as importantly — how to build a load -generator that doesn't become the bottleneck itself. +This guide explains how to design a load test whose results you can trust: +what sits in the critical path of a query, how to measure each hop so you can +attribute latency to the component actually producing it, and which levers to +adjust once the data tells you where the limit is. -In our experience, the majority of disappointing load-test results are not -produced by Materialize. They are produced by the test harness: an -under-provisioned load generator, a CPU-throttled proxy, or an unindexed query -compiling a new dataflow on every request. -This guide helps you rule all of that out, so that whatever number you end up -with is a number you can trust. +A load-test result is a claim about a whole system — the client, the network, +the proxy, the coordinator, and the cluster all sit between "query sent" and +"result received". Rather than assuming any one of them is the bottleneck, +instrument all of them and let the measurements decide. ## The read critical path @@ -40,22 +38,13 @@ cluster replica ── reads the result out of an back through environmentd → balancerd → client ``` -Two properties of this path shape everything else in this guide: - -1. **For well-indexed queries, per-query cost is dominated by coordination, - not computation.** A *fast path* query (see below) is answered by looking - up a key in an index the cluster already maintains — microseconds of - replica work. Most of the remaining per-query cost is protocol work in the - coordinator: parsing, planning, and timestamp selection. This is why the - biggest wins in this guide come from query and workload shape, not from - hardware. - -2. **The coordinator sequences all queries in the environment.** For fast - path workloads, throughput eventually plateaus at the rate the coordinator - can sequence statements, no matter how many serving clusters or client - connections you add. Past that plateau, additional concurrency only adds - queueing latency. (Standard path workloads usually saturate the serving - cluster first — see the sizing section below.) +Every hop on this path can be the limiting factor, and each one leaves a +different signature. The sections below cover, hop by hop, what to check and +what the measurements mean. One property of the path is worth internalizing +first: for a well-indexed (*fast path*) query, the replica's share of the work +is a keyed lookup against state it already maintains — so when latency +inflates under load, the added time is spent *somewhere* on this path, and +the job of the load test is to find out where, not to guess. ### What you control @@ -70,8 +59,7 @@ Two properties of this path shape everything else in this guide: | `balancerd` resources | managed for you | ✔ | Everything in the top section of the table applies identically to Cloud and -self-managed deployments — and those levers are where most of the performance -lives. +self-managed deployments. ## Make every serving query a fast path query @@ -86,8 +74,8 @@ Materialize executes a `SELECT` in one of two ways: runs it, returns the result, and tears it down. This is correct but orders of magnitude more expensive per query. -A serving workload that is 90% fast path and 10% standard path will spend -most of its cluster CPU on the 10%. Before load testing, verify **every** +A serving workload that is 90% fast path and 10% standard path can easily +spend most of its cluster CPU on the 10%. Before load testing, verify **every** query in your serving mix: ```mzsql @@ -147,9 +135,8 @@ next query) obeys Little's law: throughput ≈ concurrent connections ÷ average latency ``` -Once you saturate the environment's statement pipeline, throughput stops -rising — and every additional connection simply waits longer. The signature -is unmistakable: doubling users leaves throughput flat while median latency +Once any component on the path saturates, throughput stops rising — and +every additional connection simply waits longer. The signature: doubling users leaves throughput flat while median latency doubles, and *every* query type inflates by the same amount regardless of its cost (because the wait happens in a shared queue before execution). @@ -168,23 +155,23 @@ workload takes: - **Fast path workloads: size for state, not QPS.** Fast path lookups barely use the serving cluster — the arrangements are already maintained, and a - peek is a keyed read. In our testing, the same fast path workload measured - **identical throughput on clusters four times apart in size**, with the - larger replica nearly idle; the throughput plateau lived in the coordinator. - For these workloads, size the cluster for the **memory** to hold your - indexed views (watch sustained memory utilization) and for **hydration** - (rebuilding state after a restart or resize) — not for query throughput. - Per-peek coordination overhead even grows slightly with worker count, so an - oversized cluster can cost a little throughput on peek-heavy workloads. + peek is a keyed read. The diagnostic is replica utilization + (`mz_internal.mz_cluster_replica_utilization`): if throughput has plateaued + while the serving replica sits at low CPU, cluster size is not the limiting + factor, and scaling it will not help. For these workloads, size the cluster + for the **memory** to hold your indexed views (watch sustained memory + utilization) and for **hydration** (rebuilding state after a restart or + resize) — not for query throughput. - **Standard path workloads: cluster CPU is the throughput.** Every standard path query compiles and runs a dataflow on the cluster, so QPS scales with the cluster's compute — here, scaling the cluster up (or isolating these queries on their own cluster) directly raises throughput and protects the latency of any fast path traffic sharing the environment. -If your mix contains both, measure them separately: the standard path fraction -will dominate cluster CPU, and its ceiling responds to cluster sizing, while -the fast path fraction's ceiling does not. +If your mix contains both, measure them separately: the two fractions respond +to different levers, and a blended number hides which one is limiting you. Use +the replica-utilization check above to tell whether the cluster is the +component under pressure. ### Self-managed: `environmentd` and `balancerd` @@ -193,13 +180,12 @@ This section applies to self-managed deployments only. In Materialize Cloud, these components are sized and managed by Materialize. {{< /note >}} -- **`environmentd`** hosts the coordinator, and its CPU is where fast path - throughput saturates. Watch `environmentd` container CPU during the test: - if it is pinned near its allocation while serving clusters idle, it is the - ceiling. Give it dedicated headroom (several full cores; production serving - deployments commonly run 8–16 CPUs) and note that coordinator scaling is - sublinear — at high statement rates, reducing per-statement work (fast path - query shapes) often buys more than adding cores. +- **`environmentd`** hosts the coordinator, which every statement passes + through. Watch `environmentd` container CPU during the test: if it is + pinned near its allocation while serving clusters idle, it is the limiting + factor for that run — give it more headroom and re-measure. If it has + headroom while throughput has stopped rising, look elsewhere (client, + proxy, network) before adding cores. - **`balancerd`** terminates TLS and proxies every byte of every session. Give it a real CPU allocation (at least 1 CPU per replica, 2+ replicas) and **do not set CPU limits** on it. Its memory scales with connection @@ -217,10 +203,10 @@ cAdvisor/Prometheus metrics), not at CPU usage. ## Build a load generator you can trust -The load generator is the least glamorous part of the test and the most -common source of wrong conclusions. The failure mode is always the same: the -generator saturates, requests queue *inside the client*, and the queueing is -reported as "database latency." +The load generator is part of the system under test, whether you intend it +to be or not. When a generator saturates, requests queue *inside the client* +and the queueing is reported as "database latency" — so before attributing a +number to Materialize, establish that the generator itself had headroom. - **Never CPU-limit the generator.** Kubernetes CPU limits and container-platform vCPU allocations (for example, Azure Container Apps @@ -275,7 +261,7 @@ interactively. Alongside client metrics, record during every run: -- `environmentd` CPU (self-managed) — the fast path ceiling indicator, +- `environmentd` CPU (self-managed), - serving cluster CPU and memory (`mz_internal.mz_cluster_replica_utilization`), - `balancerd` CPU, memory, and restart count (self-managed), - the generator's own CPU and any throttling counters.