Skip to content

Release 0.6.1: reliable search, honest review outcomes, and improved defaults - #34

Merged
pkieltyka merged 2 commits into
masterfrom
update
Sep 24, 2026
Merged

pkieltyka merged 2 commits into
masterfrom
update

Conversation

@pkieltyka

@pkieltyka pkieltyka commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Reviews could silently miss search matches, count normal text lookup as degraded, or imply a clean result after required work failed. This 0.6.1 release improves evidence retrieval and makes failures, incomplete work, and unresolved questions explicit while preserving completed findings.

Changes

  • Reliable repository search: use a shared validated glob dialect and tracked revision paths, with bounded literal Git pathspec batches. Invalid globs and query syntax return actionable errors instead of empty matches. Pack complete search entries against each consumer's allowance, preserve symbol identities, and disclose omitted evidence. Oversized matching lines no longer discard useful later matches; an all-omitted response is an explicit limitation.
  • Honest review outcomes: derive report health from execution and coverage outcomes on the host. Reports, saved artifacts, CLI exits, and Action finalization distinguish failures, incomplete work, unresolved questions, and clean completion. Terminal exceptions retain the actual stage and underlying redacted reason. Unknown coverage is marked unavailable instead of showing placeholder zeros; completed findings remain available.
  • Evidence reconciliation: include original packet questions and successful complete source reads with provenance in the existing bounded composition inventory. Select concerns independently of the display cap, deduplicate evidence, and prioritize relevant sources. Resolving a concern still requires explicit supporting references; rejection alone does not resolve uncertainty.
  • Missing-test guidance: require an established consequential requirement and evidence of an unprotected regression boundary before promoting a testing finding. Preserve actionable coverage gaps without treating every uncovered branch as a defect.
  • Review defaults: enable composition reasoning step-down by default, with explicit false and --no-composition-reasoning-step-down overrides. Skip SVG files by default through review.skipSvgReview, with positive/negative CLI flags and eval overrides. SVG evidence remains searchable. Investigation and verification retain their configured reasoning.
  • Text-mode accounting: unsupported languages/formats use normal text metadata. Configured-parser fallbacks and actual retrieval limitations remain visible; mixed results do not claim wholly syntax-verified evidence.
  • Provider and CLI experience: route OpenRouter DeepSeek models through the configured DeepSeek/Fireworks/Together allowlist with allow_fallbacks: false; retain focused automatic submit-tool handling and strict validation. Add codegenie use <model>[:<level>] as shorthand for provider use.
  • Release configuration: package version and Action examples use 0.6.1. Repository configuration enables composition step-down and excludes specs, eval data, and test fixtures from self-review. Plan 123 records the implementation and validation.

Validation

  • pnpm test: 1,362 tests passed across 62 files, including workflow checks.
  • pnpm run typecheck, pnpm run build, and git diff --check passed.
  • Regression coverage includes glob/query failures, bounded search packing, oversized lines, cache delivery limits, failure-report/Action outcomes, evidence provenance, SVG overrides, and planning-stage provider usage limits/outages.
  • Read-only OMSX reproductions retained expected scoped matches, returned 17 complete matches around an oversized SVG line, and admitted all 15 recorded concerns within the existing 16,000-character reconciliation inventory.
  • Fresh built-CLI eval 129 on the trails-api case used Astra via OpenAI Codex, medium reasoning, composition step-down, no local model cache, and prompt p10.17. It passed with 10/10 hunks, one consolidated finding, 33 successful calls, no repairs, timeouts, or tool-budget refusals, 2m 34s, and $3.0385 recorded review cost. The separate recommendation judge graded both the remedy and regression test correct ($0.03045).

Remaining follow-up

The Astra smoke run had no unresolved questions, so it does not independently establish live reconciliation quality. Other model runs still show investigation timeouts, withheld unverified advice, and occasional inconsistencies between related recommendations. These remain focused follow-up work; this release preserves their limitations rather than weakening validation. Overall budgets and repair-attempt limits are unchanged by Plan 123.

Enable compositionReasoningStepDown by default while preserving explicit
false values from configuration and the existing CLI overrides. Update the
README and configuration/CLI regression coverage for the new default.

Treat unsupported formats as normal text-mode operation instead of syntax
fallback degradation. Apply this consistently to outlines, symbol reads,
definition discovery, symbol mentions, packet accounting, and high-risk
context warnings. Continue reporting fallback from configured parsers as
degraded, preserve text/heuristic precision, and avoid claiming syntax
verification for empty mention results or mixed text/syntax definitions.

Cover RIDL, JSON, YAML, Markdown, SQL, unknown extensions, extensionless
files, mixed search results, unavailable configured grammars, and generic
packet context. Preserve real retrieval errors and context-limit reporting.

Update OpenRouter DeepSeek routing to use deepseek, fireworks, and together
in that order for both only/order, retaining allow_fallbacks=false and the
focused automatic submit-tool compatibility path. Update payload tests and
documentation to match the routing configuration.

Add repository review configuration that excludes specs and eval/test
fixture data. Bump the package to 0.6.1 and synchronize documented GitHub
Action references with that version.

Add reviewed Plan 123 and its index entry. Document reproduced OMSX search
scope and result-packing defects, honest failure/completion reporting,
evidence-backed question reconciliation, and consequential missing-test
criteria. Plan 123 remains unimplemented for the next iteration.

Validation: all 1,309 tests in 60 files, workflow checks, TypeScript
checking, package build, and git diff --check pass. No live inference calls
were needed for validation.
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

🧞 Codegenie review failed (llm_call_failed).

HTTP 400: You have reached your specified API usage limits. You will regain access on 2026-10-01 at 00:00 UTC.

— View Workflow Job

Implement Plan 123 and the follow-ups from the OMSX and trails-api traces.

Search and evidence delivery:
- Share validated repository glob semantics across discovery and search;
  resolve tracked revision paths into bounded literal Git pathspec batches.
- Keep invalid globs/queries and backend failures distinct from empty matches.
- Pack complete search entries after cache lookup for each consumer budget,
  preserving symbol identity and disclosing truncated or omitted evidence.
- Skip oversized matching lines while retaining subsequent useful matches;
  reject all-omitted delivery rather than claiming an exhaustive zero result.

Review outcomes and reconciliation:
- Derive failed, incomplete, unresolved, and completed review health on the
  host, and use it consistently in reports, artifacts, CLI and Action outcomes.
- Preserve completed findings and redacted failure reasons; prohibit clean
  conclusions when required work failed or material questions remain open.
- Retain the actual pipeline stage on terminal failures and mark unmeasured
  coverage unavailable instead of presenting placeholder zero counts.
- Preserve health diagnostics during artifact serialization and retain stage
  timing snapshots through the telemetry observer.
- Reconcile original packet questions as well as verifier concerns using
  explicit supporting references, including successful complete source reads.
- Admit concerns within the existing character budget independently of the
  display cap; deduplicate evidence and prioritize question-relevant sources.
- Distinguish deliberate exclusions from incomplete work and require a
  material behavioral requirement before promoting missing-test findings.

Defaults and CLI:
- Skip SVG review by default with review.skipSvgReview and positive/negative
  CLI overrides; retain SVG availability for repository evidence searches.
- Add codegenie use as shorthand for provider use, including help routing.
- Update documentation, prompt/schema versions, and implementation records.

Validation:
- pnpm test: 1,362 tests passed across 62 files, including workflow checks.
- pnpm run typecheck, pnpm run build, and git diff --check passed.
- Fresh built-CLI Astra eval 129 passed: medium reasoning, composition step-down,
  prompt p10.17, 10/10 hunks, one finding, 33 successful calls, no repairs,
  timeouts or tool-budget refusals, 153.971s, and $3.038528 review cost.
- Separate recommendation judge graded both fix and test correct.

The smoke eval emitted no unresolved questions; wider model comparisons and
cross-finding recommendation consistency remain follow-up work.
@pkieltyka pkieltyka changed the title Improve review defaults, routing, and text-mode accounting for 0.6.1 Release 0.6.1: reliable search, honest review outcomes, and improved defaults Sep 24, 2026
@pkieltyka
pkieltyka merged commit cc63b0c into master Sep 24, 2026
7 of 8 checks passed
@pkieltyka
pkieltyka deleted the update branch September 24, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant