benchmarking: add Nighthawk router-capacity benchmark - #998
benchmarking: add Nighthawk router-capacity benchmark#998Chuang Wang (chuangw6) wants to merge 1 commit into
Conversation
6af5e9e to
0d302da
Compare
Measures the max RPS atenet-router sustains at a given Envoy CPU limit, under a tail-latency SLO, using Nighthawk's adaptive load controller in open-loop mode against the real routing path (Host-header routing via ext_proc to warmed glutton actors). - benchmarking/nighthawk/: runner Job that creates and warms the actor fleet, drives nighthawk_service + nighthawk_adaptive_load_client with Host rotation across actors, and uploads JSON/JSONL results to GCS. - Search converges on three thresholds — tail latency (measured mean+2stdev must stay under tailLatencySloMs), success-rate, and send-rate — and records which one bounded the run. The client is oversized (fixed event loops, large pools) so the harness is never the ceiling. - orchestrator.py: new `type: nighthawk` tests.yaml entries; pins the router (cpu requests=limits, envoy --concurrency) before each run. Validated end to end on a dev GKE cluster: ~8.9k RPS at 2 Envoy CPUs under a 25ms tail-latency SLO.
0d302da to
78a25c6
Compare
|
|
||
| def parse_output_textproto(text: str, desc_path: str) -> dict: | ||
| # Deferred imports: the transforms below must work without protobuf. | ||
| from google.protobuf import json_format |
There was a problem hiding this comment.
what does this mean?
|
we might want to name this to make it clear this is measuring the Ingress side. We also need a separate benchmark for Actor Egress. |
Bowei Du (bowei)
left a comment
Bowei Du (bowei)
left a comment
There was a problem hiding this comment.
Comments, mostly for clarification.
| runs `hack/install-microvm-deps.sh --install` between the two, which | ||
| stages kata + cloud-hypervisor + virtiofsd assets to the cluster's object | ||
| store bucket and applies the cluster-wide `microvm` SandboxConfig. | ||
| For a `nighthawk` test the orchestrator additionally patches the |
There was a problem hiding this comment.
why does the test run have to patch the Deployment?
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Generates nighthawk.desc: one FileDescriptorSet covering Nighthawk's |
There was a problem hiding this comment.
Probably change this to
"Generate nighthawk.desc to avoid putting a large number of
dependencies into the repository for running the benchmark"
| ATEAPI_HOST = "api.ate-system.svc.cluster.local:443" | ||
| TOKEN_FILE = "/run/ateapi-token/token" | ||
|
|
||
| # Shared atespace (locust-harness convention); sb-<uuid> names can't collide. |
There was a problem hiding this comment.
I don't understand this comment. Is the atespace named with sb- or just called benchmark
There was a problem hiding this comment.
FWIW the "sb-" convention was just taken from counter demo back in the day. Actors can have whatever name makes sense.
| TOKEN_FILE = "/run/ateapi-token/token" | ||
|
|
||
| # Shared atespace (locust-harness convention); sb-<uuid> names can't collide. | ||
| ATESPACE = "benchmark" |
There was a problem hiding this comment.
Should benchmarks use uniquely named atespaces?
There was a problem hiding this comment.
I think this is an interesting discussion. From what I remember from the summit, atespaces were expected to be fairly low cardinality. Unsure if that is still the intent, or what the performance implications of violating that guideline are.
|
|
||
|
|
||
| def _warm_one( | ||
| stub, router_url: str, name: str, atespace: str, deadline: float, log |
There was a problem hiding this comment.
put a docstring describing what this is meant to do
I think I know what "_warm_one" is intended to do, but can be unclear.
There was a problem hiding this comment.
probably should be called warm_actor?
| SANDBOX_CLASSES = ("gvisor", "microvm") | ||
|
|
||
|
|
||
| def validate_and_normalize_tests(tests: list[dict[str, Any]]) -> None: |
There was a problem hiding this comment.
I get the feeling nighthawk won't be the last time we have different types of performance tests. Can you split nighthawk into its own file and we can try to make these things more pluggable?
I'm thinking:
- A registry function that adds subcommands and flags for those subcommands (think
kubectl ate get actors) - A lifecycle object that provides hooks for the test to hook into (pre-test, test, post-test)
- Try to find common library functions.
Though we'd need to see how well that holds up to scrutiny.
Keeping it in a separate file we can start to tease out the primitives with the goal of moving to separate packages per test-type.
I don't want to put you on the hook for refactoring orchestrator though, so a separate file seems like a happy medium.
3 participants
Fixes #665
Measures the max RPS atenet-router sustains at a given Envoy CPU limit, under a tail-latency SLO, using Nighthawk's adaptive load controller in open-loop mode against the real routing path (Host-header routing via ext_proc to warmed glutton actors). See
benchmarking/nighthawk/README.mdfor design, knobs, and usage.benchmarking/nighthawk/: runner Job that creates and warms the actor fleet, drivesnighthawk_service+nighthawk_adaptive_load_clientwith Host rotation across actors, and uploads JSON/JSONL results to GCS.tailLatencySloMs), success-rate, and send-rate — and records which one bounded the run. The client is oversized (fixed event loops, large pools) so the harness is never the ceiling.orchestrator.py: newtype: nighthawktests.yaml entries (defaultlocust, back-compatible); pins the router (cpu requests=limits, envoy--concurrency) before each run.run-dev.sh: one-command dev runs (~8 min) against a prepared cluster.Validated end to end on a dev GKE cluster: ~9.4k RPS at 2 Envoy CPUs under a 25 ms tail-latency SLO (latency mean+2σ ≤ 25 ms, ≈p95; Nighthawk's search cannot gate on true percentiles).