Repository navigation
Conversation
An input can now be fed on-chip from another operator's bound output (Operand.feed): the consumer's cores read the fifos the producer's cores write, and neither operand has a host argument, handle or transfer. OperatorRuntime splits into take/preamble/issue/await_queues so MergedRuntime.fuse can issue every operator's transfers in one task group, finished once (mlir-aie's Runtime.merge(run=...)); finishing per operator would free a fill whose completion only a later operator's drain establishes. BuiltArray takes `fed` to feed an input before its build. Elementwise feeds a non-held input. scripts/proto_stream_fusion.py builds silu(gate) * up as SiLU chained into ElementwiseMul: on npu1 the intermediate lowers to shared memory between adjacent cores, the fused module passes aie-resource-stages, and it compiles to an xclbin, with and without per-call `valid` values. Not yet run on hardware. Co-Authored-By: Claude <noreply@anthropic.com>
proto_stream_fusion.py --run compiles the fused design, dispatches it and holds y to silu(gate) * up, the two operators' references chained, at 3 bf16 ulps: SiLU's 1-ulp contract becomes at most 2 ulps of the product, plus half an ulp for each of the device's and the reference's rounding. It also holds y bit-equal to the two operators run unfused on the device, and times both. A per-call run writes both operators' `valid` (one count, S6); --valid bounds it, and nothing past the bounded call's last round of tiles may be written. A full ELF writes only the words its parameter table holds, and is built with --get-scratchpad-parameters. On xcoradaie220 (Strix Halo, npu2), with mlir-aie's merged-sequence sync fix: xclbin and full ELF, each with and without per-call values, bounded and not, pass at 0 ulps and bit-equal at 2048 x 1 column, 2048 x 8 columns and 8 Mi x 8 columns. At 8 Mi the fused call takes ~690 us against ~1160 us for SiLU then ElementwiseMul. Co-Authored-By: Claude <noreply@anthropic.com>
An Image takes `chains`: each `Chain(producer, output, input)` streams step `producer`'s output on-chip into the next step's input. Composition builds the pair as one device configuration, the consumer's array fed from the producer's (`BuiltArray(fed=)`), with one `MergedRuntime.fuse` runtime that `main` calls once with both steps' host buffers; the intermediate makes no host transfer. The pair's device is content-named apart from either member's own design, and Composition's identity covers it. An image without chains composes as before. Chains need a full ELF; PerStep refuses them. iron/tests/infrastructure/test_chain.py: SiLU -> ElementwiseMul at 1 and 8 columns places and composes one device fewer, the refusals, and on xcoradaie220 (Strix Halo, npu2) the chained image holds y to silu(gate) * up at 3 bf16 ulps and bit-equal to the unchained image, also with both operators' `valid` sharing one scratchpad word. Co-Authored-By: Claude <noreply@anthropic.com>
A profile's optional "chains" entries name a producer and a consumer by class and dims, matched as tunable entries are, and the consumer input the producer's output streams into (`Profile.add_chain`, `Profile.chained`); a profile with none saves byte-identically. `iron.common.graph.chain`: `candidates` lists every edge a recording can run on-chip, overlapping ones included, for a tuner to choose among; `chains` keeps those the profile names, a step in at most one, first in step order. Graph.compile passes them to the Image of a full-ELF version not compiled with `check`. For Llama 3.2 1B the candidates are SiLU.y -> ElementwiseMul.a, one per layer, in the decode step and the prompt chunk. Co-Authored-By: Claude <noreply@anthropic.com>
A build takes a streamed operand's lanes from `Operand.fifos()`, which a fed input answers with its source's fifo ends, so a consumer needs no logic of its own to be fed; `bind()` refuses a fed input. A sequence override gets its task groups from `rt.task_group()` and finishes them with `rt.finish(tg)`, which inside a fused sequence are its one group, finished once, so `MergedRuntime.fuse` takes operators with their own sequence. An on-chip operand's `fill`/`drain` emit nothing. `feed()` refuses a pinned source and, per lane, host access orders that differ, comparing flat element order (`compare_access_orders`): GEMV.C's `[1, 2048]` lanes feed ElementwiseAdd.b's `[2048]` ones. `candidates()` admits any edge whose pair builds fed, in place of Elementwise alone. Migrated: Elementwise, WeightedRowwise (x, y), GEMV, GEMM (sequence), Clamp, RoPE, Merge, Emit, DepthwiseConv1d. A lane several cores read (a held weight) keeps its own fifo and a fresh consumer end per core. Every catalog case of these, and the elementwise family's, builds a byte-identical module (107 compared). Llama 3.2 1B's candidates: the decode step's SiLU -> ElementwiseMul and both residual adds (GEMV.C -> ElementwiseAdd.b) per layer; the prompt chunk's SiLU -> ElementwiseMul per layer and Copy -> RMSNorm. On xcoradaie220 (Strix Halo, npu2): the non-extensive npu2 catalog (161) and the touched operators' extensive cases (28) pass; a chained GEMV (2048 x 2048) -> ElementwiseAdd is bit-equal to the unchained pair and within GEMV's tolerance of numpy, but takes 1646 us against 903 us apart (medians, 20 interleaved rounds). Co-Authored-By: Claude <noreply@anthropic.com>
… the runtime Copy, Softmax, Sample, Resample, ResampleTaps and flm.DequantBFP take each plain per-lane fifo from `Operand.fifos()`; every sequence override here (also MHA's and flm.LMHead's) takes its task groups from `rt.task_group()` and finishes them with `rt.finish(tg)`. Left as they are, to be fed by a staged kind later: lanes placed on given tiles (MHA, flm.LMHead, flm.ProjectionGEMM), pinned (Gather.y), forwarded or split through a memtile (Copy.y, PatchPositions), typed other than the operand (Sample.logits), or read by several cores (Sample.draws, Resample.image and taps_w). Every catalog case of these, and hand-built Gather and GatherWords, builds a byte-identical module (67 compared). On xcoradaie220 (npu2) their catalog cases and own tests pass. Co-Authored-By: Claude <noreply@anthropic.com>
`measure_rewrite` times a set of steps as recorded against the same steps rewritten, over one dataflow, interleaved so rounds pair, as the excess of many over one per repeat, and refuses a rewrite whose output is not bit-identical; `measure_chain` is it for a chain, and the cost cache keeps chain figures apart from operator figures. `price_chains` groups candidate edges by producer shape, consumer shape and input, weighted over each call's points, and `select_rewrites` takes those that gain beyond the noise, cheapest first, skipping one that shares a step with one taken, as the tuning-fusion branch decided its folds. `python -m iron.lm.tune` prices chains after tunables, re-recorded under the new profile; `--chains-only` prices them against the profile as it stands. `Profile.chains()` lists the chain entries. A gaining chain's entry is refused where its dims cannot be written to JSON (Copy's access patterns). test_autotune.py set a profile through a `compile()` keyword it does not take; it sets `Graph.profile` now. On xcoradaie220 (npu2): SiLU -> ElementwiseMul at 2^22 elements is priced, named, and compiled with the profile runs bit-identically. Co-Authored-By: Claude <noreply@anthropic.com>
`Fusion(kind, steps, edges)` replaces `Chain`: steps that build and dispatch as one device configuration, each `Edge` an on-chip hand-off from a producer's output to a consumer's input. `FusionKind.STREAM` builds today; staged, fold, pack and merge are named for what follows. A stream fusion runs any number of consecutive steps, each array built fed from the ones before it, one fused runtime called once with every step's host buffers. An edge skips no step outside the fusion, runs forward, and feeds an input once. One output feeding several consumers is representable; Composition refuses it until a lane can serve several readers. Profiles name fusions under "fusions": each a kind, its steps' classes and dims, and its edges between them; a dim JSON cannot hold (Copy's access patterns) is left out of the entry, which then matches on the rest. `candidates` proposes every run of two or more consecutive admissible steps, overlapping ones included; `fusions` keeps those a profile names, a step in at most one. The tuner prices any of them (`measure_fusion`, `price_fusions`); `--fusions-only` replaces `--chains-only`. On xcoradaie220 (npu2): SiLU -> ElementwiseMul fused matches numpy and the unfused pair, also with a shared per-call value, and a gaining one is priced, named and runs bit-identically under its profile. The GEMV fusion tests (GEMV -> ElementwiseAdd; GEMV -> SiLU -> ElementwiseMul) run once the operand and GEMV work in progress lands. Co-Authored-By: Claude <noreply@anthropic.com>
`Operand.fifos` takes `readers=` (consumer ends per lane, for a lane several cores read), `tiles=` (each lane's host end placed on a given shim tile; returns the fifos for a build to split, join or forward), and `obj_type=` (an element type reinterpreting the operand's). A fed input answers `readers=` with the end bound for its source's drain plus more ends of the source's own fifo, kept in `lane_fifos`, and `tiles=` with the source's fifos. `feed()` refuses a held (`replicate`) input, which acquires once a call, where a producer streams a tile an iteration. Migrated: Elementwise's held inputs and WeightedRowwise.weight, DepthwiseConv1d.weight, Sample.draws, Resample.image and taps_w (several readers); MHA's Q, K, V and O, flm.LMHead's x, w and y, flm.ProjectionGEMM's C (placed). Every catalog case of these and flm.ProjectionGEMM's own shapes build byte-identical modules (89 compared). On xcoradaie220 (npu2): their catalog cases, extensive ones too, and own tests pass; Emit's image fed on-chip into Sample.draws, read by every select core and the combine core, matches numpy and the unfused pair bit for bit. Co-Authored-By: Claude <noreply@anthropic.com>
GEMV takes B riding at the head of each lane's A stream, copied out per batch into a core buffer, which no producer can feed. `feed_b=True` (a tunable, default False) streams B instead: one fifo, a consumer end per core, which a fed B takes from its producer. B's obj_shape is `Select(feed_b, (K,), None)` (mlir-aie's Select picking no stream), so at the default B does not stream, every module is byte-identical (18 catalog cases compared) and `shim_columns` charges no channel for it: it leaves out a stream whose obj_shape Select picks none under the flags given. A fed B with `feed_b` False is refused. Unfused, streamed B times within noise of the head tile at Llama 3.2 1B's decode shapes (M x K: 2048x2048, 512x2048, 8192x2048, 2048x8192, 128256x2048). On xcoradaie220 (npu2), fed on-chip and bit-equal to the unfused pair: RMSNorm (rows=1) -> GEMV(2048, 2048).B 698 against 791 us, ElementwiseMul -> GEMV(2048, 4096).B 1289 against 1376 us (medians), each held to numpy. Feeding B from a producer of several lanes, each core needing the whole vector, is a memtile gather and broadcast, still to come. Needs mlir-aie's "Let an obj_shape Select pick no stream". Co-Authored-By: Claude <noreply@anthropic.com>
… end to end Repeats of a fused group are one design back to back, configured once, where each repeat of the steps apart reconfigures twice; in a model's image another design runs between, so the fused group configures once on entry. The excess of many repeats over one credited every fusion one configure (~165 us here). Each repeat on both sides is now followed by a spacer, a cheap ElementwiseAdd of a design neither side has, so the fused group configures on entry as in an image; the spacer's cost is the same on both sides and cancels in the difference. A command per repeat would also configure each, but its dispatch cost lands in the figure. `price_fusions` then confirms the fusions it takes: each call's version is compiled with and without them, its outputs must match bit for bit, and the two are timed interleaved; while a version gains no more than the noise, the fusion predicted to gain least is dropped. On xcoradaie220 (npu2): Llama 3.2 1B decode's SiLU -> ElementwiseMul (8192) prices at -63 +- 5 us, from -216, against ~-50 us a pair end to end; its prompt chunk's (16 Mi) at -891 +- 3 us, from -1042. Co-Authored-By: Claude <noreply@anthropic.com>
A per-step image (an xclbin per design, each step its own dispatch, as every graph on NPU1 is) now takes fusions: a fusion's steps link as one `FusedDesign`, its members' arrays fed by its edges in one configuration, compiled as one more link in the xclbin chain and dispatched once with every member's host buffers; its per-call values are that design's dispatch-time scalars. Graph.compile passes a profile's fusions to a per-step version too; only `check=True`, which runs each step against its reference, still refuses them. A per-step image now finds a step's link by its index, not its operator: one operator object a fusion member at one step and standalone at another was dispatched through the fused layout, and wrote zeros. test_fusion.py's GEMV -> ElementwiseAdd timing runs each image over its own rounds (`time_interleaved`), where alternating them every call timed hardware-context swaps. On xcoradaie220 (npu2, per-step): SiLU -> ElementwiseMul (1 and 4 columns, with a bound per-call value), GEMV -> ElementwiseAdd and a small graph compiled with a fusion profile match numpy and the unfused steps bit for bit; a fusion saves a dispatch (GEMV -> ElementwiseAdd 254 us against 1390 us, minimums). Those not tied to npu2 dispatch on whichever device is attached; on NPU1, device-free, a fusion builds an xclbin per fused step and Llama 3.2 1B finds one SiLU -> ElementwiseMul per layer. Run on a Phoenix host: pytest iron/tests/infrastructure/test_fusion_perstep.py -v Co-Authored-By: Claude <noreply@anthropic.com>
GEMV(2048, 2048) on one column streams C as one lane of 2048 elements; the three-step fusion tests gave SiLU and ElementwiseMul 1024-element tiles, which `feed()` refuses, so they failed full-ELF and skipped per-step. They take 2048-element tiles now. The per-step tests no longer turn a GEMV that does not build into a skip: they fail. On xcoradaie220 (npu2): the three-step fusion matches numpy and the steps unfused, full ELF and per-step; on NPU1, device-free, both GEMV fusions build an xclbin per fused step. Co-Authored-By: Claude <noreply@anthropic.com>
The fixed-frame test prices each fusion group the small Llama's decode step admits (SiLU -> ElementwiseMul and both GEMV -> ElementwiseAdd residual shapes) standalone and in the step (`confirm_fusions`), and holds the two within three standard errors or 80 us, half the configure the old frame credited. A GEMV pair's two figures differ by 11-34 us, past their noise: its placement is not pinned, so a probe alone and the same shape among the step's others need not place alike. On xcoradaie220 (npu2), GEMV(2048, 2048) -> ElementwiseAdd re-prices at -53 +- 0.3 us, from -210, against ~-50 us a pair end to end. Co-Authored-By: Claude <noreply@anthropic.com>
…ation Two more fusion kinds build. PACK: consecutive steps none of which reads what another writes, co-resident in one configuration and issued in one task group, so they run at once (Llama's q, k and v projections). MERGE: consecutive steps, data still through host memory, in one configuration run back to back with no configure between them. Neither takes edges; a pack whose members depend on each other is refused where its buffers are known, in Composition. `candidates(recording, kinds=)` also proposes packs (maximal runs of independent steps and their sub-runs) and merges (runs of up to four steps whose designs place together, `fits()`, cached by design); `fusions` considers every kind a profile names. On xcoradaie220 (npu2), each matching numpy and the steps apart bit for bit: Llama decode's q/k/v GEMVs (M 2048/512/512, K 2048) packed, and RMSNorm -> GEMV(2048, 2048) -> ElementwiseAdd merged. Measured at columns narrowed to co-reside (q/k/v 4/2/2; 1/4/2), before GEMV's B stopped costing a shim channel: packed 437 against 557 us, merged 266 against 470 us (medians); q/k/v now co-reside at the profile's full width. Co-Authored-By: Claude <noreply@anthropic.com>
A fold runs a consumer step's kernel inside its producer's cores, on the producer's output tile before it leaves L1, the consumer's other input streaming in beside it: no cores of the consumer's own and no DMA hop for the tile. `Operator.fold(consumer, at)` returns the producer carrying a `finish` step (Elementwise and GEMV fold; the base folds nothing), with the consumer's other input as a new `finish0` stream on the producer's spare input channel. `iron.common.graph.fold`: `fold_candidates` lists each adjacent pair whose tile only the consumer reads, overlapping ones included; a profile names folds as fusion entries of kind "fold"; `Graph.compile` applies the named ones before planning, so other fusions are found over the folded steps. Refused: no free input channel, a tile a graph output or read twice, a per-call bound, GEMV with repeat above 1, more than one finish step. Default builds are byte-identical. GEMV gives every core its own reader of B (`fifos(1, readers=)`), so a fed B reaches all of them; the feed_b module is byte-identical. Ported from the tuning-fusion branch's folds. On xcoradaie220 (npu2), each bit-equal to the steps apart and within the composed tolerance of numpy, timed the tuner's way, folded / apart / stream fusion: SiLU -> ElementwiseMul at 8192 89.5 / 289.9 / 86.8 us and at 2 Mi 253.1 / 555.2 / 248.2 us; GEMV(2048, 2048) -> ElementwiseAdd 243.3 / 450.9 / 241.2 us; GEMV(2048, 8192) -> ElementwiseAdd 691.7 / 898.4 / 687.9 us. Co-Authored-By: Claude <noreply@anthropic.com>
…sumers STAGED: an edge whose producer and consumer lanes differ in count, tile or order goes through memtiles. Each side's lanes, from its own `transfers()`, become flat element streams; a consumer lane joins whole producer lanes at offsets in a memtile object, over as many rounds as still reproduce its order element for element (a group of one is a forward, which covers tile mismatches), else one producer object a round is split into consumer slots. A join wider than a memtile's inputs is a join of joins, each part's output feeding the next link. Refused, naming the edge: a per-call bound on either side, sides that are not one buffer, an order neither reproduces exactly, a split wider than a memtile's outputs, a round larger than a memtile. Fan-out: `Operand.feeds` is a list and `feed(source, staged=)` stages only where a direct feed is refused; `Operand.reader` gives the first reader the end bound for the source's drain and the rest new ends. An array with an on-chip operand is never shared between designs, which had left a second consumer's ends unread. Composition and per-step images build fusions through one `fused_configuration`, also behind `fits()`. `candidates` proposes runs of up to eight steps whose shared buffers all hand off inside them, staged where direct is refused (defaults STREAM and STAGED); fan-out and staged runs must place. Needs mlir-aie's "Let a fifo chain two ObjectFifoLinks" and "Charge a core's broadcast one producer channel in the placer". On xcoradaie220 (npu2), each matching numpy and the steps apart bit for bit, full ELF and per-step, medians full ELF fused / apart: ElementwiseAdd (8 lanes) -> RMSNorm (4) 89.6 / 258.9 us; RMSNorm -> ElementwiseMul (4), a split, 78.0 / 199.1 us; SiLU (8 lanes, a join of joins) -> GEMV(2048, 8192, feed_b).B broadcast to 8 cores 691.9 / 901.2 us; RMSNorm (rows 1) -> q/k/v GEMVs (8/4/2 columns, one core to 14) 370.5 / 644.5 us; SiLU -> two ElementwiseMuls. ElementwiseMul (8 lanes) -> GEMV.B and q/k/v at 8/8/8 columns need more than npu2's 16 shim input channels. Co-Authored-By: Claude <noreply@anthropic.com>
A pack or merge often co-resides only with members narrower than their standalone best, and a fed fan-out or gather only within the shim channels a narrower producer leaves. For each candidate group the tuner now searches the members' widths for the fused form (each member's settings collapsed to one per column count, the Pareto-maximal combinations, at most 32, widest first), trying the recorded widths first, and prices the fastest that builds against the members apart at their own settings. A taken group writes its fusion entry and the members' widths; a narrowed shape that also runs unfused is narrowed only where the gain outweighs its cost there, and two groups wanting different widths for one shape keep the cheaper. `confirm_fusions` compiles the versions with the widths being written. `autotune.replacing` writes entries as `autotune` does. A repeat keeps its group's edges whatever its kind; the rewrite is held bit-equal on every step's host-visible outputs, so a fan-out's several final steps are all checked. A point's per-call values are layered over the call's own (a member's value a point leaves out raised KeyError). On xcoradaie220 (npu2), each compiled bit-identical to unfused: Llama decode's q/k/v pack taken at full width; RMSNorm -> GEMV(2048, 2048) -> ElementwiseAdd merged with the add narrowed; of a stream and a merge overlapping at the GEMV, the cheaper stream taken. Co-Authored-By: Claude <noreply@anthropic.com>
A producer's `finish` is a chain of any length: each further consumer folds onto the last, at most one of them binary (its other input takes the producer cores' one free input channel, counted over streamed inputs only, so GEMV's head-tile B costs none). The cores run the chain over the output tile with two scratch lines in turn, the second allocated only for a chain longer than one. `fold_candidates` walks forward from each step and proposes every depth reached, overlapping ones included; `apply_folds` collapses a run in one pass. A per-call bound no longer refuses a fold where producer and consumer are bounded by the same graph value, as the prompt chunk's rows bound both halves of its SwiGLU gate. On xcoradaie220 (npu2), each matching numpy and the steps apart bit for bit: SiLU -> ElementwiseMul -> GELU, SiLU -> GELU -> ElementwiseMul, GEMV -> ElementwiseAdd -> SiLU; SiLU -> ElementwiseMul over x[:n], one image called at n = 1024, 2048 and 512. Default builds are byte-identical. Co-Authored-By: Claude <noreply@anthropic.com>
Fusion matches and stages an edge from each operand's per-lane host order (`Operand.transfers()`), derived from the declaration, which an operator writing its own `sequence(rt)` does not follow, so every edge touching one was refused. `Operator.lane_transfers(operand)` describes it in `transfers()`'s own tuples, None where not described; `transfers()` takes it before deriving. Described: GEMM's A, B and C and MHA's Q and O (their DMA moves every row whatever the bound), MHA's K and V (shortened by `kv_valid`), Softmax's x, y and addend, Copy's plain per-channel x and y, and flm.ProjectionGEMM's A and C where its B is one slab, one M chunk, no idle column and no per-call bound. No build changes. iron/tests/infrastructure/test_lane_patterns.py holds each description to the shim tasks the generated module issues, element order lane by lane, for every catalog case of GEMM, MHA, Softmax and Copy and ten ProjectionGEMM shapes, and the refusals. Llama 3.2 1B's prompt-chunk projection edges are now described; they wait on staging a per-call bound. Co-Authored-By: Claude <noreply@anthropic.com>
A pack or merge of two steps of one design, each with a per-call value (Llama decode's K- and V-cache Copy writes, RoPE per layer), bound the design's one DispatchTime parameter twice in the fused xclbin's sequence. In a per-step image each member's dispatch-time values are now parameters of their own (`<symbol>_<position>`), which the callable writes from the host symbol each stands for (`Link.symbols`); a full ELF keeps its one shared scratchpad word. `BuiltArray.reads` is empty for an xclbin design, which has no scratchpad, rather than resolving the design alone, which bound its parameters once more. A fusion member sharing another operator's xclbin (flm.ProjectionGEMM, flm.DequantBFP) builds its own array, as it does in a full ELF; only a shipped or exported design is refused. Llama 3.2 1B now compiles per-step with its candidate fusions (98 in the decode step, 83 in the prompt chunk). On xcoradaie220 (npu2), a per-step pack of two same-design Copies, each writing its own row into its own cache, matches the writes apart exactly. Co-Authored-By: Claude <noreply@anthropic.com>
…wn widths A profile written to JSON and read back stopped matching every entry naming a tuple-valued dim: JSON has no tuples, so Elementwise's and GEMV's `finish=()` came back as a list and `() == []` is false. Every fusion over a GEMV or Elementwise-family step was inert once a tuned profile was reloaded (only 33 of Llama 3.2 1B's 98 decode fusions were live, no stream fusion). `Profile.matches` compares values normalized (lists and tuples as tuples, recursively), which covers tunable entries, fusion entries and their per-step tunables alike. A fusion entry may give each step tunables of its own, applied only where the step runs inside that fusion (`iron.common.graph.fusion_tunables`, before planning), so a shape can run narrowed fused and at its own width apart; the tuner writes a taken group's widths there instead of narrowing a shape everywhere. The joint search tries `feed_b` both ways for a member an edge feeds through B. A fusion probe repeats on buffers of its own only below the size `measure()` uses, which kept a prompt-sized pack's probe over aiebu's 4 GiB addend limit; only a fusion with edges needs distinct buffers. Values the tuner writes must round-trip equal. Co-Authored-By: Claude <noreply@anthropic.com>
A pack of three steps of one design (Llama decode's k and v projections, and q alongside) shared one array, and hung the NPU: the fused sequence writes every member's preamble before issuing, so the shared cores' words were rewritten and their barrier set three times back to back; the set is absolute, so a pass was lost, the cores ran two members and the third's drain waited forever. Two members on one array survived by timing. A fused configuration (`fuse`) now gives every step its own array; a merge, which awaits each step before the next, and an unfused configuration still share one array per identity. A full-width q/k/v pack no longer places (24 shim channels of 16) and must be narrowed to co-reside, where it had run its members one after another. On xcoradaie220 (npu2): three same-design ElementwiseAdds packed get three arrays, each barrier set once, and match numpy and the adds apart bit for bit (in a child process a hang cannot outlive). Co-Authored-By: Claude <noreply@anthropic.com>
A bounded edge stages where both sides are bounded by one graph value, move the same elements a round, take the same whole rounds for every value of it (each side's rows a round in proportion, compared exactly) and their slots tile each lane's tile into whole rounds: every call then moves the same whole rounds on both sides, and the static memtile link stays in step for any count. Refused, naming the edge: one side bounded, no graph value, other round sizes or slopes, tiles not whole rounds. `candidates` no longer gates a stream, staged or merge run on placing at the recorded widths, which hid fusions that place only narrowed; the tuner narrows them or drops one that does not build. It also proposes an edge into an input that streams only with a reader's bool tunable set (GEMV's B under `feed_b`), leaving the recorded operators as they are. Llama 3.2 1B's decode step now has 430 candidates (40 distinct by design and wiring), among them RMSNorm -> q/k/v and gate/up fan-outs and ElementwiseMul -> down-projection gathers; the prompt chunk has 19. On xcoradaie220 (npu2), bit-equal to the steps apart and matching the graph's reference on [:n]: a bounded SiLU (8 lanes) -> ElementwiseMul (4 lanes) staged, one image called at n = 131072, 16384, 2048 and 49664; the decode MLP window GEMV(gate) + GEMV(up) + SiLU + ElementwiseMul fused at the profile's widths, full ELF and per-step. Co-Authored-By: Claude <noreply@anthropic.com>
The tuner's group-fit check composed a pack as an unfused configuration, whose members share arrays by identity, so it passed full-width q/k/v GEMVs that the fused configuration (each member its own array) cannot place. It now builds the fused module the image would (the group's kind and edges) and keys its cached verdict on that module's text and the toolchain's artifact hashes, so any change in how fusions compose misses a stale verdict by construction. Llama decode's RMSNorm -> q/k/v fan-out is now refused at 8/8/8 and 8/4/4 columns and places at 8/4/2. Co-Authored-By: Claude <noreply@anthropic.com>
…atches A device timeout (the driver's ERT_CMD_STATE_TIMEOUT) while pricing a group refuses it outright, never retried; one while confirming a version end to end drops every group active in it. After each group the fusions taken so far are written atomically beside the profile (`<profile>.progress.json`). `--max-price N` prices at most N groups and stops, so a sweep can run as many short device-locked calls, the lock released between them, cached groups re-read cheaply. A member an edge feeds through GEMV's B is searched with `feed_b=True` only: the head-tile form cannot be fed, and its combinations crowded the fed ones out of the search's budget. Co-Authored-By: Claude <noreply@anthropic.com>
How a profile names fusions, the five kinds, how Graph.compile applies them in a full ELF and per step, and how `iron.lm.tune --fusions-only` prices them; the limits: a staged edge under a per-call bound needs equal whole rounds on both sides, a staged lane visits each element once (so flm.ProjectionGEMM's A, re-read per column block, is not fed on-chip), and fused members never share an array. An xclbin image dispatches a fusion once, and the tuning bullet says how a fusion is priced. FusionKind's docstring no longer calls FOLD unbuilt. Co-Authored-By: Claude <noreply@anthropic.com>
ProjectionGEMM's `lane_transfers` for C permuted its row units wrongly where a column holds more than one (M=512 K=512 N=1024 on npu2, M=1024), describing an element order its sequence does not move. No fusion read it: every edge through ProjectionGEMM.C was refused as bounded. It now matches the generated module's shim tasks, with two- and four-unit regression cases. Gather.y describes its lanes, unbounded and bounded by `valid_rows`, held to the shim tasks at several counts. Co-Authored-By: Claude <noreply@anthropic.com>
An unbounded producer feeding a Copy of a bounded view (`Copy.step(y[:n])`) has a consumer whose lanes move a per-call length (Copy's `src_valid`) while neither operand reads as bounded, Copy declaring no `per_call(bounds=)`. Staging planned that edge as unbounded: a static memtile link moving the producer's whole buffer every call while the Copy drains `n`, so the two fall out of step. Staging now refuses a side whose lanes carry a per-call length that no bound of its operator names, and `candidates` no longer proposes the edge. Co-Authored-By: Claude <noreply@anthropic.com>
…idths in full `--lock-in-process` has the tuner take the NPU's coordination lock itself, once per measurement, from its first dispatch until its images and their hardware contexts are released; a long sweep then interleaves with other jobs on the NPU instead of holding it throughout. Holding the lock only around dispatches let another process take it while this one's contexts were still alive, and that process failed to create its own (CREATE_HWCTX -22). The lock re-enters, so a measurement holding it calls the timing helpers that take it too. A refusal now says why: which outputs differed from the steps apart, how many elements, by how much. The width search pruned dominated combinations for every group, but only a pack or merge has a fit check behind that pruning; for a group with edges the widest combination dominated all the others, so it alone was tried. Llama decode's RMSNorm -> q/k/v fan-out, which places at 8/4/2 columns with GEMV's B fed, was refused at every width; it is now searched in full. Co-Authored-By: Claude <noreply@anthropic.com>
`confirm_fusions` compiled each version with and without the taken fusions and let a compile error end the sweep (a Llama prompt chunk whose fused version read more scratchpad parameters than a command holds). A version that does not build now drops the group predicted to gain least among those active in it, logs why, and tries again, as for a version that does not gain. Errors after the build, from dispatching or timing, still raise. Co-Authored-By: Claude <noreply@anthropic.com>
…omposed steps plan() read each fusion's members' words together as one command span rather than splitting inside a pack or chain, since a fusion's steps run as one call. Composition mapped a plan's step index to its composed (fusion-collapsed) position, so a split past a fusion lands on the right composed step instead of the recording's original index. confirm_fusions still planned the fused build against the unpatched recording's command split; Graph.compile now passes the fusions a version takes to plan(), so a narrowed design's own words decide where it splits. Repro (scratch/fusion-staged-review/scratchpad_repro.py): the prompt chunk's unpatched plan put 41 words in one command (over the 32-word scratchpad); with fusions passed through, it splits into three commands of 32, 28, and 8 words. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ording's confirm_fusions reused chosen, the plan() computed once from the UNPATCHED recording, to build fused_cg for every combination of active groups it tries. patched's steps (after fusion_tunables_apply narrows their widths) read different words than the plain recording's, and plan()'s split points are specific to one recording's words; reusing chosen could under-split a patched version into a command over the scratchpad's 32 words, or land a split inside a fusion's span (now refused since the prior commit). confirm_fusions now plans patched with its own fusions before building fused_cg, inside the same try that already treats a build failure as this version's own and drops its worst-priced group. Verified against the real prompt-chunk repro (scratch/fusion-staged-review/scratchpad_repro.py) that crashed the sweep: planning the unpatched recording puts all 226 composed steps in one 41-word command (over the limit); planning patched with its fusions splits it into three commands of 32, 28, and 8 words. The device-free confirm_fusions suite (iron/tests/infrastructure/test_tune_fusions.py, -m "not supported_devices and not extensive") still passes, 37/37, including the build-failure drop this change's try/except still covers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… bytes The real sweep reached, for the first time past the scratchpad-overflow fix, confirm_fusions actually dispatching the prompt chunk's plain_cg: _random_tensor fills every recording.inputs handle with arbitrary small bytes, fine where an end-to-end confirmation only compares bit for bit against the same recording's unfused run and the content need not be representative, but the embedding's ids input feeds a Gather, whose control_words refuses an id past its table (correctly: a wrong id would read outside it). Random bytes read as int32 routinely land in the tens of millions, past Llama's 128256-row table, crashing confirm_fusions with an IndexError instead of a RuntimeError its build try/except and device-timeout handling already cover. _gather_table_rows names each input-role gather id (Recording.steps' own step.gather, set where Recorder._gather records one) by its table's row count; confirm_fusions draws those within range instead of through _random_tensor. Keyed by buffer name: the handle a step's gather holds is its own view, a different object from recording.inputs' of the same buffer (Handle compares by identity, not value). test_gather_table_rows_names_an_input_roles_ids_handle (device-free) calls the real Gather.control_words the crash site is: bounded ids pass, raw int64 bytes in [0, 2**31) still raise. Confirmed against the actual failure too: the real sweep's log had it crash on exactly this path (iron/operators/copy.py's IndexError, "Gather: gathering [...] from 128256 rows"); full device-free iron/tests/infrastructure/test_tune_fusions.py (not supported_devices/extensive) passes, 38/38. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Where a binary consumer folds into a producer whose channels are already full (BinaryElementwise with uses_extra, GEMV's feed_b), the consumer's other input has no channel of its own left. Elementwise's _can_ride (no per-call bound, one tile a lane) and GEMV's own A-riding let it ride the producer's own channel instead, one more transfer a call on the same stream: BinaryElementwise.sequence() queues finish0's own transfer right after a's, and GEMV's A reads the finish row an extra ``reads`` times to the same effect. Elementwise's core_fn_chain had a double-acquire bug on the ridden fifo: `a`'s own element stayed acquired (held for the whole iteration's release loop) while the chain step's own `fifos[ride_index].acquire(1)` tried to take the fifo's second transfer, but an objectfifo's acquire is cumulative on what is still outstanding, not independent per call, so the second acquire aliased `a`'s slot instead of reaching finish0's. Fixed by releasing `a`'s acquire right after `kernel_fn` consumes it, before the chain's own acquire of the same fifo, and skipping it in the end-of-iteration release loop (it is already released by then). GEMV's own A-riding build needed no such fix; both are now verified on the device: bit-equal to both a numpy reference and the pair run unfused (iron/tests/infrastructure/test_fusion_fold.py's test_binary_elementwise_rides_matches_reference_and_unfused_on_device, test_gemv_rides_a_matches_reference_unfused_and_stream_on_device). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fusion_tunables.apply rebuilds a step a fusion entry narrows as a new operator, a design of its own that reads its own value symbols, but left the recording's bindings on the operator it replaced. No call wrote the narrowed design's row counts, so a bounded member kept whatever the scratchpad held. apply now moves each binding to the narrowed operator, its member looked up by name, keeping it on the original only where a step still runs that one. Found running the tuned Llama 3.2 1B npu2 profile end to end on real weights: its prompt-chunk merges narrow a bounded ElementwiseAdd and RMSNorm, and the 3262-token prompt's second chunk runs 1214 of 2048 rows. Bit identity failed at every decode step, mean KL against the golden was 3.53 and decode ran at 0.48 tok/s. With the fix it is bit-identical to stock, mean KL 0.000846 (stock's own), TTFT 1.539 -> 1.507 s and decode 14.58 -> 14.77 tok/s. confirm_fusions passed the profile because it checks a full chunk only. Co-Authored-By: Claude <noreply@anthropic.com>
Section 9 walks a graph from recording to a fused version through the public path: candidates() lists what a recording admits, a profile's add_fusion() entry names one, and Graph.compile builds it as one device configuration. Its example, swiglu_fusion.py, is Llama 3.2 1B's decode MLP up to the down projection (gate and up GEMVs at 8192x2048, SiLU, the multiply), fused as one stream. Both versions are held to graph.reference() and to each other bit for bit, then timed interleaved on their images. iron now exports FusionKind, the kind add_fusion takes. The example carries forward the measurements of #78 and #79, which fused this MLP by hand-written kernels and showed it pays: here the graph fuser builds it from the library's own operators. On xcoradaie211 (Strix, NPU2), 8 rounds of 50 calls, NPU time over three runs: unfused 1679.8-1699.5 us min / 1719.2-1727.5 us median, fused 1290.8-1296.0 us min / 1322.9-1344.1 us median (1.30x at the median). Fused and unfused outputs are bit-identical. iron/tests/docs/test_programming_guide.py (22) and iron/tests/common/test_fusion_profile.py (24) pass, on the device. Co-Authored-By: Claude <noreply@anthropic.com>
CI's Lint job runs black --check over the repo; two long lines on this branch wrapped. Formatting only. Co-Authored-By: Claude <noreply@anthropic.com>
… call confirm_fusions checked bit identity at the call's own values alone, and a prompt chunk's call runs every bound at its largest (rows=C). A design reading its rows wrongly agrees there, so the tuned Llama 3.2 1B npu2 profile passed confirmation although each narrowed prompt merge's row counts went unwritten, wrong on any partial chunk. _bound_points gives the call's values with each per-call value that bounds an extent halved, one at a time, where the recording's limits allow it; confirm_fusions holds fused to unfused at each, then at the call's own values last, which timing then dispatches. The prompt chunk is checked at half its rows, the decode step at half its position. On npu2, a merge narrowing two bounded ReLUs is refused at n=2048 with the previous commit's fix reverted, and confirmed with it. Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relies on Xilinx/mlir-aie#3995
Generalizes the SiLU → ElementwiseMul on-chip chaining prototype into a kind-tagged
Fusionthe tuner finds, prices, and confirms on its own:stream(a producer's output feeds a consumer directly, lanes meeting on-chip),staged(as stream, joined/split through memtiles where lane counts differ),pack(independent steps issued concurrently),merge(consecutive steps in one configuration), andfold(a consumer's kernel runs in its producer's own cores, chains of any length, bounded per call).fusion_groupsfinds candidates a recorded graph admits; each is priced alone, searching member widths jointly so they can co-reside;select_rewritestakes the cheapest non-overlapping set;confirm_fusionsrebuilds the whole version with them applied, checks bit-identity against the unfused run, times it in context, and drops anything that doesn't actually gain — with hang refusal, resumable checkpointing, and a per-measurement NPU lock so a multi-hour sweep never holds the device past what it's using.Three tooling bugs turned up running a real multi-hour tuning sweep against Llama 3.2 1B and are fixed here, each with its own device-free test: a command could split inside a fusion's span instead of at its first member;
confirm_fusionsplanned a fused build against the unpatched recording's command split instead of its own; andconfirm_fusions's random inputs fed an embeddingGatherunbounded bytes instead of ids within its table.Scope
This is the fusion mechanism and tuner support only. Llama's shipped
profiles/npu2.jsonis unchanged — a fuller tuned profile (14 fusions across decode and prompt) came out of this work but is still pending review before it replaces the current one. mlir-aie's pin also needs to move to a wheel built fromehunhoff/fusion-supportbefore any of this is reachable outside a localbuild-fusion.Measured
Llama 3.2 1B, npu2, a greedy (not yet tuner-selected) pass: bit-identical output, KL unchanged, decode 14.55 → 15.28 tok/s (+5%), TTFT 1.539 → 1.524s (~-1%) on a 3262-token prompt. Fusion's own ceiling is real but modest here — the dominant cost (GEMV, ProjectionGEMM, MHA) mostly isn't reachable by fusion.