docs: load testing guide for serving workloads - #38913
Open
tylerhartwig wants to merge 3 commits into
Open
tylerhartwig wants to merge 3 commits into
tylerhartwig wants to merge 3 commits into
Conversation
Explains the read critical path (balancerd -> environmentd/coordinator -> cluster replica), which levers move throughput and latency (fast path eligibility, prepared statements, isolation level, concurrency knee, cluster/environmentd/balancerd sizing), how to build a load generator that doesn't bottleneck the test (CPU throttling, single-process caps, tunnels), server-side latency cross-checks via mz_recent_activity_log, and write-side/CDC test guidance (hydration, freshness, memory headroom). Written primarily for self-managed deployments but applicable to Cloud (the environmentd/balancerd sections are marked self-managed-only). Distills findings from a field load-testing engagement where every initial bottleneck turned out to be in the harness or proxy layer, not in query execution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion path Prepared statements don't move the needle much on Materialize's fast path (the earlier internal benchmark conflated the driver/harness change with statement preparation), so remove that section rather than overstate it. Rework the serving-cluster sizing guidance to depend on execution path: fast path workloads should size for state (cluster size doesn't raise peek QPS), while standard path workloads scale QPS directly with cluster compute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove assumptions about where slowdowns usually live (harness-blame framing, environmentd throughput-plateau claims, 'ceiling' language). Each section now describes what to measure and how to interpret the measurement, rather than presuming which component is the bottleneck. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
There is currently no doc that walks a customer through load testing Materialize: what sits in the read critical path, what they control (Cloud vs self-managed), how to attribute latency to the component actually producing it, and how to build a harness whose numbers can be trusted. A recent field engagement showed how easily load-test latency gets mis-attributed when the client, proxy, and server aren't measured independently.
Changes
Adds
doc/user/content/serve-results/load-testing.md, written as a diagnostic guide — each section describes what to measure and how to interpret it, rather than presuming where the bottleneck is:EXPLAINverification, indexing the exact equality key, pushing unions/joins into indexed views, isolating ad-hoc queries.mz_internal.mz_recent_activity_loglatency vs client-reported latency during the same window.Tips for reviewer
serve-results/, alongside connection-pooling and isolation-levels (weight 47). Happy to move it if there's a better home (e.g. a future "operational guidelines" section).Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel. (n/a)🤖 Generated with Claude Code