Skip to content

docs: load testing guide for serving workloads - #38913

Open
tylerhartwig wants to merge 3 commits into
mainfrom
docs/load-testing-guide
Open

tylerhartwig wants to merge 3 commits into
mainfrom
docs/load-testing-guide

Conversation

@tylerhartwig

@tylerhartwig tylerhartwig commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Motivation

There is currently no doc that walks a customer through load testing Materialize: what sits in the read critical path, what they control (Cloud vs self-managed), how to attribute latency to the component actually producing it, and how to build a harness whose numbers can be trusted. A recent field engagement showed how easily load-test latency gets mis-attributed when the client, proxy, and server aren't measured independently.

Changes

Adds doc/user/content/serve-results/load-testing.md, written as a diagnostic guide — each section describes what to measure and how to interpret it, rather than presuming where the bottleneck is:

  • The read critical path — client → balancerd → environmentd (coordinator) → cluster replica, with a Cloud vs self-managed "what you control" table so the page serves both audiences.
  • Fast path eligibilityEXPLAIN verification, indexing the exact equality key, pushing unions/joins into indexed views, isolating ad-hoc queries.
  • Isolation level — pick deliberately, pin explicitly, record with results.
  • Concurrency — Little's law, sweeping to find the knee, the queueing signature (flat throughput + uniformly inflating latency).
  • Sizing, split by execution path — fast path workloads size clusters for state (with replica utilization as the diagnostic for whether the cluster is the limiter), standard path workloads scale QPS with cluster compute; plus self-managed-only diagnostics for environmentd and balancerd, including the "throttled containers look idle on CPU graphs" trap.
  • Building a trustworthy load generator — CPU limits/platform vCPU caps, single-process client ceilings, ramping, tunnels, shared test data.
  • Server-side cross-checkmz_internal.mz_recent_activity_log latency vs client-reported latency during the same window.
  • Write-side (CDC) testing — hydration before measurement, freshness lag, transformation-cluster memory headroom, source retention windows.
  • A final checklist.

Tips for reviewer

  • Placement: serve-results/, alongside connection-pooling and isolation-levels (weight 47). Happy to move it if there's a better home (e.g. a future "operational guidelines" section).
  • The guide deliberately avoids claims about where bottlenecks usually live and avoids absolute throughput/ceiling statements — it teaches attribution instead. Performance observations that remain are qualitative and tied to a described measurement.
  • Self-managed-only sections are marked with a note shortcode; everything else applies to Cloud as well.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (docs-only)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design. (n/a)
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label. (n/a)
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label. (n/a)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post. (docs-only)

🤖 Generated with Claude Code

Explains the read critical path (balancerd -> environmentd/coordinator ->
cluster replica), which levers move throughput and latency (fast path
eligibility, prepared statements, isolation level, concurrency knee,
cluster/environmentd/balancerd sizing), how to build a load generator
that doesn't bottleneck the test (CPU throttling, single-process caps,
tunnels), server-side latency cross-checks via mz_recent_activity_log,
and write-side/CDC test guidance (hydration, freshness, memory headroom).

Written primarily for self-managed deployments but applicable to Cloud
(the environmentd/balancerd sections are marked self-managed-only).
Distills findings from a field load-testing engagement where every
initial bottleneck turned out to be in the harness or proxy layer, not
in query execution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tylerhartwig
tylerhartwig requested a review from a team as a code owner September 17, 2026 12:37
tylerhartwig and others added 2 commits September 17, 2026 08:42
…tion path

Prepared statements don't move the needle much on Materialize's fast path
(the earlier internal benchmark conflated the driver/harness change with
statement preparation), so remove that section rather than overstate it.
Rework the serving-cluster sizing guidance to depend on execution path:
fast path workloads should size for state (cluster size doesn't raise
peek QPS), while standard path workloads scale QPS directly with cluster
compute.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove assumptions about where slowdowns usually live (harness-blame
framing, environmentd throughput-plateau claims, 'ceiling' language).
Each section now describes what to measure and how to interpret the
measurement, rather than presuming which component is the bottleneck.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Sep 17, 2026

Copy link
Copy Markdown

EDU-92

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant