Repository navigation
Conversation
sufubao
force-pushed
the
refactor/attention-backend-plans
branch
3 times, most recently
from
November 10, 2026 15:12
09fab80 to
011b909
Compare
sufubao
force-pushed
the
refactor/attention-backend-plans
branch
from
November 11, 2026 00:44
011b909 to
592cbca
Compare
sufubao
force-pushed
the
refactor/attention-backend-plans
branch
from
November 11, 2026 02:55
690e372 to
ab254cd
Compare
sufubao
force-pushed
the
refactor/attention-backend-plans
branch
from
November 12, 2026 05:24
167964f to
b9618da
Compare
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
将 attention 计算、批次上下文和 KV/Conv/SSM 持久状态分开。模型按 LayerSpec 分组,LayerAttentionBinding 保存本层参数与共享 backend,通过 attention.context(infer_state) 取得当前 context,再显式调用 forward。context 按 allocate_context → prepare → update → forward 执行;State 继续负责存储、缓存恢复和释放。保留 group_id。无持久 KV 的视觉/音频路径使用 VarlenAttentionOp,保留 --vit_att_backend。
本轮收敛了共享 backend 的状态归属:MTP cursor 放入层参数,GDN/KDA native/replay 共用 decode layout,按 State 分别取得 accepted count;backend 不再浅拷贝注入 spec。context 使用 slots dataclass,workspace requirement 和 replay State 接口有明确类型;BatchBuilder 集中借用 InferState 的输入 tensor。selector 显式接收 specs、设备能力和执行约束,不再反向查询 basemodel。Dense/MLA FA3 共用 decode planning。out 明确由调用方拥有,支持 provider 返回 padded/strided buffer 的有效 view,也允许临时结果复制;不承诺零分配。import 按标准库、第三方、lightllm、相对导入分组,模型调用及中英文文档同步更新。
已合并 upstream/main 9640f76,包含 ReplaySSM #1690、KDA prefill 融合 #1705、对角求解并行化 #1703 和 decode 文件句柄修复 #1707。新 KDA kernel 除 import 外计算 AST 与上游一致。
本机 H200 验证(实验命令、源码哈希和失败记录在 /data6/exp_logs/runs):
之前扩展测试中的旧native MTP test_mtp_verify_path[1]仍有torch.equal最大差值0.000977;合并前测试/kernel覆盖对照可复现(180025),未放宽断言,本轮不计为通过。此前ReplaySSM完整624例结果见历史实验,本轮重跑的是上述受影响路径。跨节点PD、完整多模态模型、全部MTP组合及性能未验收。
仍保留共享AttentionBatch字段名和现有候选表;没有引入统一Registry或强制所有provider直接写out。Optional字段用于尚未prepare的context,泛型尚未经过全仓静态类型检查。无关工作区修改未纳入提交。