Repository navigation
[TENT] Integrate TENT 4 receiver credit on current upstream main - #55
Draft
Connor-Matthew wants to merge 24 commits into
Draft
Connor-Matthew wants to merge 24 commits into
Connor-Matthew wants to merge 24 commits into
Conversation
13 of 21 tasks
github-actions
Bot
added
documentation
ControlClient::sendData/recvData gained receiver-credit session/epoch parameters. tcp_regression_test calls them through function pointers, where default arguments do not apply, so the calls failed to compile. Pass explicit zero fences for the non-credit paths, and build the malformed-length probe from the legacy header layout so the server rejects it as an invalid transfer request size.
ControlClient::recvData replaced the server's rejection text with a generic 'invalid or rejected legacy response' string for every short reply, so a denied READ lost 'permission denied' and the upstream tcp_regression permission test started failing. Propagate the server message when the reply cannot be payload, keeping a descriptive fallback for empty replies; apply the same treatment to the v2 reply path so fence rejections stay diagnosable.
DfsOnlyBatchFailureOnlyRevokesFailedKey leaves the persistent write fault injection armed, so the retry Put it asserts on can never succeed. Disarm the injection after the rollback assertions and before the retry. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolves the tebench.md --xport_type conflict by keeping upstream's MLU wording plus the TENT transport additions (ub, sunrise_link). Restores a mergeable PR so the pull_request CI trigger keeps working.
Conflicts: - tent/src/runtime/control_plane.cpp: keep both <limits> and <optional>. - tent/src/runtime/transfer_engine_impl.cpp: adopt main's progressLockFor(batch_id) scope in submitTransfer while keeping the branch's owner_kind argument to prepareSubmit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A staged owner holds the receiver credit for all of its chunks, and the chunks themselves are credit-exempt. If one chunk fails while a sibling's cross stage is still writing into the receiver, returning the credit at that point would let the receiver grant the same capacity to another sender while the old write is still landing. Upstream kvcache-ai#4720 made ProxyManager keep the parent PENDING until in-flight siblings drain. Add an end-to-end regression through the runtime queue and receiver-credit gate: the staged owner stays PENDING and a second credit-gated write never reaches the transport until the held cross-stage write drains; only then does the owner fail, return the credit, and let the second write complete. Reverting the upstream drain makes the test fail on all three checks. CI hosts have no device memory, so add a test-only setStagingPolicyForTest hook that forces host staging for selected requests.
… per poll On two UB nodes, tebench throughput collapsed ~1000x (6 GB/s -> 0.004 GB/s, multi-second latency) whenever concurrent demand exceeded the receiver's data_bytes credit, and recovered as soon as capacity covered the demand. A deferred owner is retried on every refill, and refill runs on every status poll. Each retry re-sent the demand exchange, a synchronous RPC to the receiver made under progress_mutex_. With credit exhausted, the waiting threads issued thousands of refused RPCs per second and kept the lock busy, so the in-flight owners could not be polled to completion and never returned the credit they held. A local receiver did not show it because a local exchange is a plain function call. Remember a refused (or failed) exchange per credit key and requeue the owner without another RPC until this sender releases credit for the key, asks for less, or a capped exponential backoff (1 ms .. 32 ms) expires. A local release retries immediately; the backoff bounds the wait for capacity that another sender returned, which this sender is not told about. A cross-engine reproduction of the tebench workload (4 threads x 8 x 1 MiB, 1 ms simulated completion) goes from 4 to ~1500 batches/s at 16 MiB of credit. New tests check that a refused owner exchanges at most a handful of times while busy-polled (686 times in 50 ms without the fix) and is dispatched immediately on a local release, and that a refused sender recovers when another sender releases.
16 of 36 tasks
…he-ai#5062) The HTTP metadata server refuses an unauthenticated non-loopback bind (kvcache-ai#4472), so `python3 -m mooncake.http_metadata_server --port N` exited right after startup. The test only checked that Popen returned, waited 30s for a readiness probe that could never pass, then continued anyway, so the failure surfaced much later as "Failed to setup Mooncake store, error code: -1" and every case errored in setUpClass. - pass --host 127.0.0.1 and launch with sys.executable - inherit the service output instead of an unread pipe - fail fast when a service exits, instead of continuing anyway - probe the metadata endpoint with a key (any HTTP status means the listener is up) and the master by TCP connect, replacing the fixed 30s spin and the sleep-based master check - report already-exited services instead of noisy "No such process" Verified with tone run-single test_hicache_storage_mooncake_backend.sh (artifact 12849421510): 4 passed, previously 4 errors.
…s fallback (kvcache-ai#5050) We found a production performance issue on MI308X + bnxt_re (Thor2), kernel 6.6.117-45.14.tl4: the kernel has CONFIG_PCI_P2PDMA disabled, but isKernelDmabufSupported() failed to catch this and let the HIP dmabuf MR path through. amdgpu then silently migrated the registered VRAM buffer to GTT/system memory, collapsing GPU-side access bandwidth from ~1838 GB/s to 115 GB/s (attention kernels 5-50x slower, and no recovery after MR deregistration). Host evidence: # cat /boot/config-6.6.117-45.14.tl4.x86_64 | grep -i P2p # CONFIG_PCI_P2PDMA is not set # cat /proc/kallsyms | grep pci_p2pdma ffffffffae1d0c70 T __pfx_dma_pci_p2pdma_supported ffffffffae1d0c80 T dma_pci_p2pdma_supported ffffffffaf625fe8 r __ksymtab_dma_pci_p2pdma_supported ffffffffc0954210 t nvme_pci_supports_pci_p2pdma [nvme] ffffffffc0954200 t __pfx_nvme_pci_supports_pci_p2pdma [nvme] The kallsyms fallback matched the substring "pci_p2pdma" against the whole line. dma_pci_p2pdma_supported() (and the nvme helper above) are compiled in regardless of CONFIG_PCI_P2PDMA, so the check always passed. Inside containers the config-file candidates are usually not visible either, which makes this fallback the deciding probe. Fix: parse the symbol-name field of each kallsyms line and require an exact match against symbols that only exist when CONFIG_PCI_P2PDMA=y: pci_p2pdma_add_resource, pci_p2pdma_distance_many and pci_p2pdma_enable_store (from drivers/pci/p2pdma.c; the =n variants are static inline and never appear in kallsyms).
…ai#4935) * [TE] hp_tcp: assign each operation to the least-loaded lane hp_tcp picked a lane from a per-peer submission sequence (lane_sequence % connections_per_peer), regardless of which lanes were busy. Each lane runs one operation at a time, so under a closed-loop client (a fixed number of transfers in flight, one submitted per completion) the lane that just finished received its next operation only after the sequence came back around, i.e. after connections_per_peer more completions. Lane utilization then settles at whatever fraction it starts at (busy fraction A/N is self-consistent for any number A of active lanes), and with deeper windows the equal round-robin split piles the window onto the slowest connections. Measured on two VMs over a 200 Gb/s pod network (16 MB writes, host memory, 16 lanes): every lane received a request every ~30 ms and transferred for ~15 ms; 7-9 of 16 lanes were sending at any time; the gap from a lane's response to its next request header was 14-17 ms while the client resubmitted within 0.6 ms. Throughput was 5.6-8.9 GB/s with occasional 14-15 GB/s runs. Keep an outstanding-operation count per (target, lane) and give each single-slice operation to the lane with the fewest outstanding operations, ties broken from the round-robin position so the original spreading is kept. The count is released on completion and on both cancel paths. Multi-slice (rail-striped) operations are unchanged. 1 pair, 16 lanes: 7.9-9.2 -> 12.9-14.9 GB/s, run-to-run spread < 5%. 1 pair, 48 lanes, 32 workers: 19.8 GB/s. 2 process pairs: 10.9 -> 19.2 GB/s (classic tcp 19.8, iperf3 21.7). * [TENT] hp_tcp: release lane counts on every path The per-lane outstanding count was taken when an operation was planned and given back only on completion or cancellation. Planned commands that never run leaked it: a batch whose later request fails to plan drops the commands planned for earlier requests, and tryCommitBatch() can reject (admission backpressure, TooManyRequests) and destroy the commands without calling cancel; retries have the same pattern. Each leak left +1 on a lane for good, so the lane was avoided from then on. The count is now held by a LaneSlot shared by the planned command, the queued operation and its completion callback. It is released explicitly on completion and on both cancel paths (so the next submission sees the lane free at once), and otherwise when the last holder is destroyed, at most once. Every path that drops a planned command, current or future, gives the count back. Test: LaneLoadReturnsToZeroOnRejectedSubmits checks that the counts return to zero after an admission rejection, after a mid-batch planning failure, and after a completed transfer. It fails on the previous revision (all three checks) and passes with this change; ctest -R hp_tcp: 9/9.
…vcache-ai#4885) * [Conductor] Index vLLM CPU events as local with shared precedence Enable Conductor to index vLLM/vLLM-Ascend `CPU` KV events as `cpu_local` while preserving Mooncake Store's `cpu_share` ownership. * [Conductor] fix ut description fix ut description
…cache-ai#5068) * [TENT] Wait for in-flight io_uring I/O before freeing a sub-batch freeSubBatch() destroys the ring of a sub-batch with io_uring_queue_exit(), which does not wait for I/O still running, and then frees the tasks, whose destructors release their staging buffers. A read still running then completes into a freed staging buffer. Before destroying the ring, wait for the completions of every submitted request that has not completed. SQEs still in the SQ ring were never submitted and are not waited for. A sub-batch whose tasks have all completed is freed without waiting, as before. Add tests that keep a read in flight on a FIFO: freeSubBatch() stays blocked until the read has been fed, and when the wait fails it leaves the staging slot out of the pool, and the read still lands in it. Signed-off-by: Zhan Xue <zhan.xue@intel.com> * [TENT] Keep draining io_uring completions when the wait fails Poll with io_uring_peek_cqe() until every in-flight request has completed, then free the sub-batch. freeSubBatch() never returns with I/O in flight then nothing leaks. Signed-off-by: Zhan Xue <zhan.xue@intel.com> --------- Signed-off-by: Zhan Xue <zhan.xue@intel.com>
…ache-ai#5064) * [TE] Add bounded RDMA endpoint posting Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Add Classic RDMA congestion window adapter Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Connect RDMA workers to congestion control Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Verify installed congestion control consumers Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [Docs] Describe Classic RDMA congestion control integration Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Exercise rail recovery wrappers in shared builds Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Grow adaptive windows on clean completions Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Coalesce active-route RDMA replenishment Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [Docs] Describe error-driven Classic RDMA control Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> * [TE] Preserve installed congestion control ABI Generate and install the TE feature configuration so public Slice layouts match the library without consumer compile definitions. Exercise installed consumers in both ON and OFF builds. Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com> --------- Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Every NPU wheel build since Oct 26 (v0.3.14-pre and its re-runs, 13 legs, both arches) died mid-compile with 'The runner has received a shutdown signal' or froze silently - always inside the mooncake-store compile, never with a compile error, while CUDA/non-CUDA legs on the same runners completed normally. Root cause: the mooncake-store translation units pull in the CANN headers and reach ~3-4GB per g++ at -O2. At -j$(nproc)=4 that peaks at 12-16GB on a 16GB runner (3GB swap) - right at the edge, which is why earlier builds sometimes survived. Cap the mooncake build at -j2: peak drops to ~7GB; the compile step takes roughly 2x longer (~4 -> ~8 min per leg). Validated end-to-end on a fork: the same pipeline went from 0/13 legs to green, wheels attached to https://github.com/MooncakeHixl/Mooncake/releases/tag/v0.3.99-pre7 Co-authored-by: tangjie66 <tangjie66@huawei.com>
The PG device library was built for 80-real/90-real(/103-real) only, so GPUs outside that list (sm_100, sm_120, ...) had no kernel image to load. Use the same list as mooncake-ep, which also embeds PTX for each entry. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces #54 after renaming the source branch from
codex/tent4-integrationtoub/tent4-integration. The code commits are unchanged.Migrate only original TENT 4 commit
d78e20dd3f9568c8223fbdbb3e98e00f6ed34c67onto officialkvcache-ai/Mooncakemain at1a6d00d0f4dc7f54c573c5fcd4ff5f80bd1f2813, after TENT 3 landed upstream in kvcache-ai#4605.This adds receiver-credit authority/exchange, runtime dispatch gating, transport session fences, staging propagation, and the native UB validation guide. Conflict resolution preserves current-main cancellation, safe TCP retry, payload limits, registered-memory permissions, copy-error propagation, runtime policy snapshots and submit-stage failover. Required-credit scatter submissions are rejected explicitly because scatter bypasses the credit-owning runtime queue; a regression test covers the rejection before transport handoff. A separate migration compatibility fix validates the receiver session/epoch against the pinned HP TCP metadata before initial or retry handoff, with missing/stale/version/flags rejection and matching-session success coverage. HP TCP was introduced upstream after the original TENT 4 snapshot.
Related design discussions: kvcache-ai#2849 (receiver-advertised credits, closed historical RFC) and kvcache-ai#3093 (native UB/URMA). This is TENT 4 follow-up work, distinct from the existing TENT 3 integration PR #53 and native UB/bonding PR #40. No duplicate TENT 4 PR was open when checked.
Base note: the fork's
mainhas been synced to officialkvcache-ai/Mooncakemain, and official main was merged into this branch (latest93bc4ff5). The commits on top of official main are the migrated TENT 4 change, the HP TCP compatibility fixes, and the staging/receiver-credit regression test.Validation PR: do not merge. Keep draft until human review and the owner's two-node UB traffic/data-integrity validation finish.
Resolved: staged failure released receiver credit early
Static review found that original TENT 4 could release an outer receiver-credit reservation after a staged parent reported failure while credit-exempt sibling chunks were still writing into the receiver. That would let the receiver grant the same capacity to another sender while the old write was still landing.
Official main fixed the underlying lifetime in kvcache-ai#4720:
ProxyManagernow keeps the staged parentPENDINGuntil in-flight sibling copies and remote operations are terminal, so the queued owner (and its credit) is not finished early. This PR adds an end-to-end regression,RuntimeQueueDispatch.StagedFailureKeepsReceiverCreditUntilChunksDrain, through the runtime queue and receiver-credit gate:PENDING, and a second credit-gated write is deferred and never reaches the transport;Reverting the kvcache-ai#4720 drain makes all three checks fail. The test passes 400/400 under 8-way concurrent repetition. CI hosts have no device memory, so a test-only
setStagingPolicyForTesthook forces host staging for selected requests.Known limitation: credit held by senders that exit uncleanly is not reclaimed
ReceiverCreditAuthoritylowers a sender's outstanding credit only when that sender reports a release. If a sender exits uncleanly (crash, kill, or engine shutdown with committed in-flight work), the credit it held at that moment stays outstanding, and the sender keeps amax_sendersslot, until the receiver process restarts and starts a new session.This is a deliberate, conservative choice: RDMA/UB writes are one-sided, and the receiver cannot prove that a silent sender's old DMA is gone. Reclaiming on silence alone could hand capacity to another sender while a stale write is still landing. The impact is bounded:
receiver_credit.enable = false);Possible follow-up (not in this PR): when a grant is denied for capacity, probe senders that hold outstanding credit and reclaim only on positive proof that the instance is gone: connection refused (process exited, so its QPs/Jettys are torn down) or a different instance ID (process restarted). Unreachable senders would stay unreclaimed by default.
Suggested Merge Priority
Module
mooncake-transfer-engine)Type of Change
How Has This Been Tested?
Test commands:
Test results: 26 receiver-credit + 50 admission-queue + 11 request-merge tests pass (87 total). Before migration, the official baseline passed its 14 receiver-credit + 43 admission-queue tests. Documentation HTML builds successfully and the changed pages/navigation were inspected. All applicable pre-commit hooks pass, including clang-format 20.1.8 on changed lines and codespell.
tent-ciran 66 TENT tests, 0 failed)tent_runtime_queue_dispatch_test,tent_hp_tcp_transport_test)Checklist
./scripts/code_format.shvia the pre-commit hookAI Assistance Disclosure
Codex migrated the original commit, resolved conflicts against current main, added the scatter compatibility guard/regression, and ran the available local checks. Independent Codex agents reviewed migration risks. An AI coding assistant also wrote the staged-failure receiver-credit regression and the test-only staging hook, verified that reverting kvcache-ai#4720 makes it fail, and analyzed the unclean-exit credit limitation documented above. The human submitter's full review and hardware validation remain pending; this PR must stay unmerged.