tensor-parallel

A light llama-like llm inference framework based on the triton kernel.

GLM-5.3-Flash EXL3 (320B MoE) on 2x NVIDIA DGX Spark — production serving kit, 1M context, 97%+ multi-session prefix caching, DFlash2 spec decode

Reproducible recipe: serve abliterated Gemma-4-12B (gemma4_unified) at 50-118 tok/s on no-NVLink Blackwell (SM120) via vLLM nightly + ModelOpt FP8/NVFP4 + MTP spec-decode.

Reproducible kit to deploy DeepSeek-V4-Flash-DSpark on a 2× NVIDIA DGX Spark (GB10) cluster: vLLM TP=2 over QSFP 200GbE, NVFP4 KV, DSpark speculative decoding, 1M context, systemd self-heal. Apache-2.0.

Reproducible LLM inference on four NVIDIA DGX Spark (GB10) nodes wired as a direct switchless ConnectX ring — serving profiles, ring launchers, pinned ARM64 runtime images and measured results, newest tested model first.

Minimal SGLang patch for Qwen3.8-27B + DFlash2 on dual RTX 3080: TP-sharded Draft fc and static per-head FP8 KV.

Reproducible llama.cpp kernel and runtime optimization lab for dual NVIDIA Tesla V100 GPUs (SM70)

Patches + recipe to deploy festr2/MiMo-V2.5-Pro-NVFP4-MXFP8-attn-TP8 on 8-node DGX Spark sm_121 (Ray + vLLM, TP=8). Fixes the fused-qkv loader bug that mis-slotted Q values as K/V on 7 of 8 ranks.

Field-tested guide: multi-GPU vLLM tensor-parallel (TP=2/TP=4) on Intel Arc Pro B70 (Battlemage BMG-G31, Xe2) on Linux. Driver setup (xe force_probe=e223), bare-metal vLLM + oneAPI 2025.3, the compute-runtime multi-root USM + triton-xpu init_devices fixes, FP8/int4-AutoRound quant, root-cause error reports. AI-agent readable (AGENTS.md).

Measured LLM benchmarks for NVIDIA DGX Spark (GB10): DeepSeek-V4-Flash 284B MoE on a TP=2 pair over 200G RoCE — tok/s by profile and concurrency, 1M-token context curve, the MoE backend flag, monitoring traps. Every number links to raw runs.

Measured serving recipe: DeepSeek-V4.1-Flash on 4x NVIDIA DGX Spark (GB10) with 1M context, CUDA graphs, vision and tool calling. vLLM TP4 over RoCE. EN + 中文.

SM121-optimized vLLM serving of nvidia/GLM-5.3-Flash-NVFP4 on 2x GB10 (TP=2, DFlash2 K=7): patches, click-run container, serve profiles, sanitized evidence

Run zai-org/GLM-5.3-Flash (NVFP4) on 3x NVIDIA DGX Spark with vLLM, TP=3 + EP, DFlash2 speculative decoding, CUDA graphs. Full recipe with rationale, benchmarks, patches, and what we tried.

Runbook + benchmarks: Qwen3.8-Flash-Next-ABLITERATED NVFP4 on 4× Tesla V100-32GB (reflashed SXM2→PCIe, 2+2 NVLink + PLX). 1Cat-vLLM 1.5.0, TP4 — 262,144-token context validated, 46 tok/s decode, 122 tok/s aggregate at 4 concurrent streams. Full E0–E17 optimization log with measured evidence.

GLM-5.3-Flash NVFP4 vLLM TP=2 serving cluster on 2x NVIDIA DGX Spark (GB10) — one compose, one entrypoint, stacks as pure config

A ~800-line PyTorch implementation of Megatron-LM's TP + PP + DP + AMP. 1.6-2x faster than Megatron-Core on 125M models.

Qwen3.8-Flash-Next (NVIDIA NVFP4 weights) served as W4A16 on 2x DGX Spark GB10 with vLLM, TP=2. Pinned engine build, 4 start-time patches, measured results, and the levers that were tried and rejected.

Verified four-GB10 DeepSeek-V4-Flash-0731 TP4 DSpark Graph-8 deployment and benchmarks

Serve an abliterated Gemma-4-12B at high speeds on Blackwell GPUs without NVLink using vLLM, FP8 quantization, and MTP speculative decoding.

Deploy DeepSeek-V4-Flash-0731 on dual NVIDIA RTX PRO 6000 Blackwell GPUs with vLLM PR #41834 (jasl fork) and DSpark speculative decoding, achieving ~200-227 tok/s in no-overseas-network environments.