Repository navigation
[WIP] support PPO - #2132
Closed
PengchengShi00 wants to merge 15 commits into
Closed
[WIP] support PPO#2132PengchengShi00 wants to merge 15 commits into
PengchengShi00 wants to merge 15 commits into
Conversation
…troller - AgentLoop.canonicalize_train_fields (base + localhost/sandbox overrides) builds the unified full-sequence train fields (input_ids/labels/logprobs) at generation time; semantic holes are baked into labels by the loops. - RolloutState drops response_mask; labels become the only supervision carrier. agent_loop_type records the producing loop class name and AGENTIC_AGENT_LOOP_TYPES discriminates agentic full-sequence samples. - TrainingController.fit accepts list[list[RolloutState]] and absorbs validation, session-clustered advantages, shift/tensorization, seq_ctx, teacher fields and data_info; BaseRLTrainer._prepare_train_data is gone. - calculate_group_effective_response_masks now bakes token staleness into labels in place (monotone, convergent) and its agentic exclusion uses agent_loop_type instead of the input_ids/labels presence heuristic.
…ut conversion - response_ids now denotes the contiguous suffix of input_ids after the prompt (env/tool tokens included) in every loop; localhost/sandbox export input_ids[len(prompt_ids):] so response_model_steps stay aligned with the full response region. - Token staleness baking becomes branch-free: effective mask = semantic mask (labels != -100 on the suffix) * per-token staleness mask; the zero-prompt suffix state becomes eligible and agent_loop_type plus AGENTIC_AGENT_LOOP_TYPES are removed. - data_info stats: prompt_len reports the original prompt length and response_len the supervised (LLM-generated) token count; env/tool injected tokens count in neither. - _rollout_groups_to_colate_items is phase-split into session reward clustering, group advantage estimation, per-state ColateItem conversion and data_info summarization.
matrix72c
added a commit
to matrix72c/xtuner
that referenced
this pull request
Oct 26, 2026
…zation Depends on upstream PR InternLM#2132. Preserve validated ownership semantics without prior merge history.
Collaborator
Author
|
see: #2186 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
preview pr