Repository navigation
Conversation
Denny991
changed the title
Denny991
changed the title
added 10 commits
October 30, 2026 15:44
Field-tested the skill end to end by aligning an older feature branch against the current main snapshot and the reference framework. All indexed anchors resolved (failure mode was default/signature drift, not missing symbols); the run exposed gaps now fixed: - anchors: point the per-update order at Trainer.fit (logging precedes scheduler.step); index model.scale_and_reduce_grad inside clip_grad_norm as the unscale/replica-mean step and note reduce-group coverage for Replicate params not managed by FSDP; cross-check CE gradient accumulation precision, not just forward options; scope the aux-loss example to VLM/diffusion recipes; add the model registration entry (get_model_config_from_hf) and a same-framework version-anchor arm recipe (function-level diff + one-variable transplant). - pitfalls: P1.3 record identity for .git-less snapshots; P2.1 optimizer param-group defaults are version-sensitive; P4.4 record the dump's step/lr-phase mapping; new P4.6 replay adapters and config files are pinned to a code version (import + pack-order + Config.fromfile smoke). - SKILL.md: experiment cards must state reused-baseline contract-match evidence; document compare_step_metrics CSV columns.
Prevent incomplete or non-finite metric comparisons from reporting false passes, and clarify cross-model evidence requirements and safe preprocessing-cache reuse.
A field run stopped at reporting the failure instead of continuing localization until the user supplied a later fix PR. State that a failed gate blocks the next stage, not the investigation, and require the next observation, a follow-up experiment card when GPU time is needed, and CPU/source analysis while waiting. Also label every claimed cause as found independently or verified after a known fix, so later verification is never reported as an independent find or a blind-test pass.
Field runs showed gradient direction can collapse while scalar loss and magnitudes stay close, and that blaming the nearest kernel (attention, recompute, compilation) before boundary evidence leads to retracted attributions. Add a routing row for direction-only divergence, a diagnostics recipe for hooking per-layer outputs to find the first divergent backward boundary and bisecting within that layer, and the generic hook guards learned from failed probes: hook outside compiled regions, idempotent collection under reentrant recomputation, skip no-grad tensors and None gradients, light single-rank sampling payloads, and model-specific output resolution instead of hard-coded key names. Also state the shard-reassembly self-check: calibrate per-placement reduction factors until per-parameter squares reconstruct the official norm squared, which catches duplicate-replica and wrong-factor errors before elementwise comparison.
A field session drifted toward treating an upstream fix-PR list as the goal of diagnosis: experiments were proposed to match known fixes rather than to close observed symptoms, and fixes for conditions the current scope never triggers risked being chased as missing finds. State the convergence criterion explicitly: a symptom is closed by a single cause confirmed with a one-variable A/B plus a quantified or attributed residual. A known fix list is only a post-hoc coverage check, and out-of-scope fixes stay "not reproduced in this scope".
Retrospective on three bugs that were only found after seeing upstream fix PRs, although the evidence already existed in our own artifacts. Add the missing actions, all model-agnostic: - Runtime submodule hyperparameter reconciliation: dump built modules' eps/dtype via named_modules() and check against the checkpoint config when forward loss differs on equal inputs; constructor defaults can hide drift that config dumps never show (P1.10, diagnostics section 2). - Objective-level config differences must be adjudicated against a third-party authority before comparing; "known different" explains a gap but never justifies comparing across it (preparation gate). - Close every decomposed factor: a probe that splits a discrepancy into N factors owes N single-variable confirmations; shrinking the headline metric with the dominant fix does not retire the rest (discipline). - Zero-GPU same-side perturbation discriminator to separate amplified forward noise from structural backward differences using saved gradient snapshots (diagnostics section 4).
A second field session overturned a "structural kernel difference" verdict: three ineffective precision toggles prompted the call, but an uninherited upstream fix was the real cause, and its exclusion argument had already been invalidated by newer data that was never rechecked. Also, near-alignment was treated as terminal without a final fix-list sweep or a quantitative noise floor. Add: structural or environment-level residuals require exhausting the inheritable fix list and one's own decomposed factors, and any exclusion whose premise new data broke must be re-run (discipline section 1); the same-side perturbation floor is the stopping rule — call a residual kernel noise only when it sits at that floor after a final fix-list sweep, because "mostly aligned" is not terminal (diagnostics section 4); and integer selection tensors get their own check category (diagnostics section 7) — exact sets, sentinel layout, and forward/backward consumption consistency, since prefix- reading and mask-gathering kernels disagree in only one direction, with operator-input descent when a component matches but the layer does not.
An independent review of the five field-run commits found real defects, including one rule our own later experiment had already falsified but that was never revised. Fixes, all still model-agnostic: - Derive shard-reduction factors from mesh and placement semantics and use the norm-squared identity only as a cross-check; fitting factors to the official total is under-determined (diagnostics section 3). - Downgrade the same-side perturbation contrast to screening evidence: a compilation toggle perturbs forward and backward, and either outcome admits other explanations, so drop the claim that elementwise parity requires a matched stack (diagnostics section 4). - A same-side noise floor is one supporting input, not a stopping rule; judge against predeclared bounds and keep out-of-bound residuals open regardless of the floor (diagnostics section 4). - Localization closes on material factors and interactions, not "a single cause"; a decomposed factor must be confirmed, ruled out, or recorded open, with no mechanical N-probes rule (discipline 1-2). - Structural verdicts require checking every applicable fix and material factor, recording why anything unchecked was skipped. - Mark register_hook guards as PyTorch-specific and conditional. - A failed gate requires continued investigation within scope, budget and declared stops, not an open-ended obligation (SKILL stage 1). - Objective differences block parity verdicts, not labeled diagnostic contrasts that intentionally vary the objective (SKILL preparation). - Integer selection tensors compare by consumer semantics (ordered, multiset, set); mismatches can affect forward, backward, or both (diagnostics section 7). - Merge the two overlapping gradient-direction routing rows and widen provenance to independent / externally informed / mixed / unknown, keeping the field mandatory (SKILL, report template).
A second review pass caught two spots where the previous commit claimed full application but stale text remained: the report checklist still used the old two-way provenance labels (contradicting the four-way labels in the diagnose section and report template) and still demanded provenance for open items with no candidate cause; and P1.10 kept the old absolute wording (named_modules-only discovery, checkpoint-config- only comparison) that diagnostics section 2 now links to, resurrecting the superseded rule through the back door. Also clarify that an independent find keeps its provenance when later verification confirms it — the prohibition targets relabeling externally informed or mixed work as independent.
Denny991
force-pushed
the
fix/text-alignment-field-hardening
branch
from
October 30, 2026 07:45
9e28147 to
cdd90e2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
基于两次 GLM-5.3-Flash 实地对齐实验及两轮独立交叉模型评审，加固
align-automodel-xtuner-textskill：同时加固
compare_step_metrics.py：窗口化阈值检查（声明 START:END 则要求两侧完整覆盖）、严格整数解析（拒绝截断小数）、非有限值拒绝、rank/regex 一致性校验、JSON 先验证再写文件。Test plan
python3 .claude/skills/align-automodel-xtuner-text/scripts/test_compare_step_metrics.py— 21 项测试全部通过