gather: deterministic context-pack tool (aft_gather)

[iceteaSA]

gather: deterministic context-pack tool (aft_gather)

What

One new tool — aft_gather — assembles a bounded "context pack" (ranked, deduped, budgeted verbatim code evidence) in a single call. It replaces the multi-turn search → outline → zoom → callgraph read chain an agent otherwise runs to build context around a question or a symbol.

Two modes (mutually exclusive):

• question: "how does X work?" — seeds from handle_semantic_search (same pipeline as aft_search, all lanes/fallbacks)
• symbol + filePath — seeds from the callgraph (impact depth-1 callers + call_tree depth-1 callees)

Seeds expand one hop through the callgraph, dedupe by canonicalized (file, symbol) with seeds winning, and render via render_symbol_within_budget until a hard line budget (default 400, cap 800) is spent. Everything past the cut appears as one-line stubs under ## Beyond budget (zoom to expand) — nothing is silently dropped.

Why

Agents burn serial turns assembling context: search, then outline the hits, then zoom the symbols, then chase callers. Each turn round-trips through the model. A pack returns evidence (verbatim bodies with file:line headers), not conclusions — the agent reasons over it directly, ready to attach to a subagent dispatch.

Measured on a real config repo (3 questions, one call each vs. the manual chain):

• "how does the lane-verdict cache flow end-to-end" → complete 4-file chain (cache shapes, verdict logic, write/read, plugin deny hook) in one pack, used=283/400. Manual baseline: 5-6 tool calls.
• "how does score-tap decide when to write an executor row" → the full decision chain (event filter → dedup → context sources → verdict parse → DB insert) in one pack at budget=200.
• An unrehearsed cross-file question → correct core symbols plus their docstrings, first try.

Independently reproduced on the aft codebase itself: question: "how does bash output compression dispatch pick a compressor" → seeds=15, used=226/400, one pack assembling the full dispatch chain (compress → gate → compress_with_registry_exit_code 20-compressor array → Compressor trait → install path → subc mirror) that otherwise takes a 4–5-call search→zoom chain.

Honest degradation

The pack never lies about its own quality:

• While the semantic index is building, long NL queries degrade to lexical-only file-level hits. The pack renders them as visible file:line (no containing symbol) stubs and flags the header with degraded=semantic-index-building (partial results — retry when index ready) — detected via the response's semantic_status field, cleared as soon as one real seed resolves. No blocking or retry inside the tool.
• Grep-fallback hits ({file, line_text, line}, no symbol name) resolve to their containing symbol by line containment — definitions, call sites, and comment hits all upgrade to the enclosing symbol. Hits with no containing symbol stay visible as stubs.
• Unresolved external/stdlib callees collapse to one summary line ((N unresolved external calls omitted)) instead of drowning the stub list; unresolved seeds and callers are never suppressed.

Implementation

• crates/aft/src/commands/gather.rs — Rust-side composition: calls handle_semantic_search / impact_result / call_tree_result / render_symbol_within_budget directly (shared &AppContext, no bridge round-trips, no parallel reimplementation of search).
• Wiring: main.rs dispatch arm, subc_translate.rs mapping, TS factory packages/opencode-plugin/src/tools/gather.ts + registration (same tier as aft_callgraph — depends on the callgraph store).
• No new dependencies, no config surface beyond the tool args, no LLM calls, no caching.
• Follows the tri-state honest-reporting convention (protocol.rs Response doc-comment): success:false+code for un-performable calls (e.g. invalid_request on a bad mode combo), success:true with a visible degraded/stub pack for partial results — never a bare empty success.

Tests

22 unit tests in gather.rs, including red-checked regressions (each confirmed to fail against pre-fix code): mid-codepoint truncation panic, duplicate-symbol line-anchored resolution, abs/rel path dedupe, containing-symbol resolution via a real TreeSitterProvider, callee-only stub suppression driven through the production build_pack path, and degradation-flag presence/absence/mixed cases.

Limitations (deliberate scope)

• 1-hop expansion only — multi-hop was deliberately excluded: the budget math and ranking get harder and the packs get noisier.
• Callgraph-dependent by design: neighbor expansion quality follows the callgraph store's freshness, same as aft_callgraph.
• Budget counted in lines, not tokens — matches aft's existing budget idiom across zoom/outline.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds aft_gather, a single-call context-pack builder that returns ranked, deduped, verbatim code within a fixed line budget, replacing the search→outline→zoom→callgraph chain. Supports question-seeded or symbol+filePath callgraph-seeded packs with 1‑hop neighbors and clear degradation/availability notices.

• New Features

  • Two modes: question (semantic seeds) or symbol+filePath (impact callers + call-tree callees); 1‑hop expansion; dedupe by (file, symbol) with seeds winning.
  • Hard line budget (default 400, max 800) via render_symbol_within_budget; overflow appears under “Beyond budget”; unresolved external callees collapse to a single summary line; seeds/callers remain visible stubs.
  • Grep-fallback hits resolve to containing symbols; no-containing-symbol hits stay as stubs.
  • Header flags semantic-index-building degradation and skipped neighbors when the callgraph store is unavailable.
  • Exposed as gather in Rust dispatch with translate_gather; added aft_gather tool in @opencode (tier: “all”). No new dependencies. 22 tests.

• Migration

  • Call with either { question: "how does X work?" } or { symbol: "foo", filePath: "src/foo.rs" }; optional budget 1–800 (default 400).
  • Neighbor expansion uses the callgraph store; required in symbol mode. In question mode, neighbors are skipped with a header notice if unavailable.

^{Written for commit cf09b9b. Summary will update on new commits.}

Greptile Summary

This PR introduces aft_gather, a one-call context-pack builder that replaces the multi-turn search → outline → zoom → callgraph chain by assembling ranked, deduped, verbatim code evidence within a configurable line budget. The implementation is a new Rust command (gather.rs), wired into the dispatch table and subc_translate, plus a TypeScript tool definition in the opencode-plugin package.

• Core algorithm: Two modes — question (semantic search → 1-hop callgraph expansion) and symbol+filePath (impact callers + call-tree callees) — feed into a shared build_pack path that deduplicates by (file, name), renders via render_symbol_within_budget, and lists overflow candidates as stubs. The callgraph store correctly normalizes relative paths via its own project_root, so seeds from semantic search (normalized to repo-relative form) are handled correctly.
• Honest degradation: The header flags semantic-index-building state, callgraph unavailability, and file-not-found/unresolved-symbol stubs — nothing is silently dropped. Previous review feedback on the separator line-counting bug and the used= post-processing splice has been addressed in this revision.
• Tests: 22 unit tests covering truncation panic, duplicate-symbol line-anchored resolution, abs/rel path dedupe, containing-symbol resolution, callee suppression, and degradation flag cases.

Confidence Score: 5/5

Safe to merge — no correctness defects on any call path; all findings are non-blocking style and dead-code observations.

The callgraph store's nodes_for correctly normalizes relative paths via its own project_root, so the seed paths from semantic search (normalized to repo-relative form in collect_callgraph_neighbors) resolve correctly. Budget accounting, deduplication, degradation flagging, and stub suppression are all correct and backed by 22 unit tests including regression cases. The only issues are an unreachable neighbor loop, a misleading comment on an unused field, and a missing lower bound on the Rust-side budget — none affect runtime behavior.

No files require special attention. gather.rs warrants a second look only for the dead orphan-neighbor loop at lines 479–486.

Important Files Changed

Filename	Overview
crates/aft/src/commands/gather.rs	New 1460-line file implementing the full gather pipeline. Core logic is sound and well-tested; two minor dead-code paths exist (orphan-neighbor loop and unused hop_distance field with misleading comment).
packages/opencode-plugin/src/tools/gather.ts	TypeScript tool definition with correct mode validation, Zod schema with min(1)/max(800) budget guard, and straightforward delegation to callToolCall. Clean implementation.
crates/aft/src/subc_translate.rs	Adds translate_gather with correct path resolution via resolve_path_from_project_root and full mode validation mirroring the Rust-side checks.
packages/opencode-plugin/src/index.ts	Correctly registers aft_gather in the ALL_ONLY_TOOLS set and spreads gatherTools(ctx) into the tool map.
crates/aft/src/main.rs	Single-line dispatch arm addition for "gather" — straightforward and correct.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant gather.ts
    participant handle_gather (Rust)
    participant handle_semantic_search
    participant CallGraphStore
    participant render_symbol_within_budget

    Agent->>gather.ts: "aft_gather({ question | symbol+filePath, budget })"
    gather.ts->>handle_gather (Rust): dispatch "gather"

    alt question mode
        handle_gather (Rust)->>handle_semantic_search: top_k=15 semantic query
        handle_semantic_search-->>handle_gather (Rust): results[] + semantic_status
        handle_gather (Rust)->>handle_gather (Rust): resolve containing symbols (grep-fallback hits)
        handle_gather (Rust)->>CallGraphStore: impact_result per seed (1-hop callers)
        handle_gather (Rust)->>CallGraphStore: call_tree_result per seed (1-hop callees)
    else symbol mode
        handle_gather (Rust)->>CallGraphStore: impact_result (callers of symbol)
        handle_gather (Rust)->>CallGraphStore: call_tree_result (callees of symbol)
    end

    handle_gather (Rust)->>handle_gather (Rust): dedupe by (file, name), seeds win
    loop each candidate (budget remaining)
        handle_gather (Rust)->>render_symbol_within_budget: render within per_symbol_budget
        render_symbol_within_budget-->>handle_gather (Rust): content + status (Complete|Truncated|Menu)
    end

    handle_gather (Rust)->>handle_gather (Rust): build header (used=N), append stubs section
    handle_gather (Rust)-->>gather.ts: { text: pack }
    gather.ts-->>Agent: verbatim pack with file:line headers

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant gather.ts
    participant handle_gather (Rust)
    participant handle_semantic_search
    participant CallGraphStore
    participant render_symbol_within_budget

    Agent->>gather.ts: "aft_gather({ question | symbol+filePath, budget })"
    gather.ts->>handle_gather (Rust): dispatch "gather"

    alt question mode
        handle_gather (Rust)->>handle_semantic_search: top_k=15 semantic query
        handle_semantic_search-->>handle_gather (Rust): results[] + semantic_status
        handle_gather (Rust)->>handle_gather (Rust): resolve containing symbols (grep-fallback hits)
        handle_gather (Rust)->>CallGraphStore: impact_result per seed (1-hop callers)
        handle_gather (Rust)->>CallGraphStore: call_tree_result per seed (1-hop callees)
    else symbol mode
        handle_gather (Rust)->>CallGraphStore: impact_result (callers of symbol)
        handle_gather (Rust)->>CallGraphStore: call_tree_result (callees of symbol)
    end

    handle_gather (Rust)->>handle_gather (Rust): dedupe by (file, name), seeds win
    loop each candidate (budget remaining)
        handle_gather (Rust)->>render_symbol_within_budget: render within per_symbol_budget
        render_symbol_within_budget-->>handle_gather (Rust): content + status (Complete|Truncated|Menu)
    end

    handle_gather (Rust)->>handle_gather (Rust): build header (used=N), append stubs section
    handle_gather (Rust)-->>gather.ts: { text: pack }
    gather.ts-->>Agent: verbatim pack with file:line headers

Loading

_{Reviews (3): Last reviewed commit: "gather: deterministic context-pack tool ..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[iceteaSA]

Follow-up on the two maintainability notes from the Greptile summary (they weren't separate review threads, so noting here) — both addressed in cf09b9b9:

• degraded expression flagged as always-false in the non-empty-seeds path — intentional: degraded only applies when zero symbol-level seeds resolve (a building index yields no seeds; a ready index with seeds is never degraded). Logic unchanged; added a comment on the why so it doesn't read as dead code.
• callee-suppression string-match coupling — extracted UNRESOLVED_MARKER ("(symbol not resolved)") and CALLEE_PROVENANCE_PREFIX ("callee-of-") consts, referenced by both the producer (render_symbol_section err arm / collect_callees_for_seed) and the consumer (build_pack suppression guard) so the match can't silently drift. Only test-site literals remain, intentionally — a test pointing at the const couldn't catch const drift.

Matched/rendered text is byte-identical; 23/23 gather tests green.