Skip to content

fix: recognize verification commands only in command positions - #20

Open
imxyanua wants to merge 10 commits into
EvoMap:mainfrom
imxyanua:codex/fix-verification-evidence
Open

imxyanua wants to merge 10 commits into
EvoMap:mainfrom
imxyanua:codex/fix-verification-evidence

Conversation

@imxyanua

@imxyanua imxyanua commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Printing a check name, such as echo "npm test", currently creates a passing check, a distillation candidate and L1 reuse evidence. Recognize checks in command positions instead of matching names anywhere in shell text.

tool result → direct command / known wrapper → check
            → quoted text / argument / comment → no check

Preserve supported direct checks, chains, pipelines, environment assignments, npx wrappers and heredoc handling. Dynamic substitutions are conservatively excluded; this does not evaluate arbitrary shell control flow.

Evidence

  • Before: a public turn lifecycle regression records a check for echo "npm test".

  • After: the same turn has no checks, no candidate and tier-none reuse evidence. Targeted verification/lifecycle tests: 36 passed on Windows Node 24.21.0.

  • Linux Node 22.23.4 and 26.13.0: each clean npm ci, full npm test 647/647, and packing passed. GitHub CI also passed for aa57d06.

  • Native Claude Code 2.1.316 with isolated profiles and loopback model/Proxy: a real PowerShell echo produced no candidate/L1; a real node --test fixture produced one passing test and retained candidate/L1. No external provider/account/Hub calls.

  • Wider Windows adapter tests exposed POSIX fixture/path limitations; those failed logs remain separate from Linux/native results. No workload, timeout or assertion was relaxed.

  • Follow-up head 94edc6a fixes the reviewed executable-path regression: ./gradlew, local .bin tools and absolute known-tool paths are recognized only as executable tokens, including through npx. New RED→GREEN regression plus verification/lifecycle/checks/reuse 67/67 and packing passed. The earlier 647/native results above predate this follow-up; CI reruns the final commit.

Merge Danger

Door: two-way

No state schema or migration. Unsupported command forms may stop contributing verification evidence.

Blast Radius: shared-runtime

The classifier is shared by native host adapters, candidate selection and reuse reporting. Real model/Proxy/Hub acceptance is separate from fixture tests.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 1 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="aa57d06ee9862b1d0cf299f9ebb0f497d071adae",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 aa57d06ee986

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 1 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 路径形式的构建工具不再被识别 — 新正则要求工具名位于字符串开头,但实际项目通常通过相对或绝对路径启动 wrapper。旧正�...(truncated) · lib/verification.js · advisory_0e55617f340b8837bacde8d4

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 aa57d06ee986

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 2 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="94edc6aa390952fe028181464524854c5d77ed34",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 94edc6aa3909

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 2 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 带空格的引号可执行路径无法识别 — SHELL_DATA 在命令扫描前会把所有单引号、双引号内容替换为 __data__;因此引号中的可执行...(truncated) · lib/verification.js · advisory_4261497063435b8ea2b18627
  • [P3] 绝对路径 env 包装的验证命令不会写入检查状态 — ENVIRONMENT 在去除可执行文件路径之前运行,因此 /usr/bin/env 不会匹配 env 包装规则;随��...(truncated) · lib/verification.js · advisory_d36de2661dba887c5573783f

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 94edc6aa3909

@forrestlinfeng forrestlinfeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review — changes requested

Right direction, and it removes a real false positive (echo "npm test", quoted separators, heredocs, (cd app && npm test) all now behave). But it introduces new false negatives compared with main.

Major

  1. # anywhere is treated as a comment — SHELL_DATA in lib/verification.js:36 matches #[^\r\n]*. In shell, # only starts a comment at the start of a word, so these now return false: curl http://x/#f && npm test, git log --format=%h#%s; npm test, echo a#b; npm test. Match # only at line start or after whitespace/separator.
  2. Any $( rejects the whole command — lib/verification.js:44. cd $(git rev-parse --show-toplevel) && npm test was true on main, now false; agents write this often. Blank the $(...) span as data instead of returning false.
  3. Common wrappers/runners not recognised (only env and npx are unwrapped): timeout 60 npm test, sudo …, time …, nohup …, env -i …, yarn vitest, bunx vitest, python -m pytest, uv run pytest, poetry run pytest, { npm test; }, if …; then npm test; fi, bash -c "npm test". The cost is silent (no candidate, no L1 evidence). Suggest one table of known wrappers with a test for each.

Minor

  • Windows paths: C:\tools\make.exe is no longer recognised (backslash treated as escape, .exe not stripped). Either handle it or state it is out of scope.
  • The __data__ placeholder is a magic string a later rule could match by accident; as a regex tokenizer without nesting state this will need a single table of supported cases, tested in both directions, to avoid rotting.

Note: combined with #27 (fewer passes counted), this reduces L1 evidence more than either PR alone — please run the full suite on the combined branch.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 3 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="61b5499fa7cec180372f824c483bdc48c97ede95",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 61b5499fa7ce

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 3 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 引号内实际执行的验证命令被当作普通数据丢弃 — withoutShellData() 无条件把单引号和双引号内容替换为 \0。这能排除 echo "npm test",但也会�...(truncated) · lib/verification.js · advisory_61a428f17604815097199fa7
  • [P3] Windows .cmd / .bat 构建工具无法识别 — withoutExecutablePath 只移除 .exe 后缀;Windows 常见的 gradlew.bat、npm.cmd、pnpm.cmd、yarn.cmd ...(truncated) · lib/verification.js · advisory_e1fec2c7236ee7934e8c7700
  • [P3] 带选项的命令包装器会丢失实际验证结果 — 新解析器只支持少量无选项包装器。它会先剥离 sudo ,但将剩余的 -u root npm test 当作普��...(truncated) · lib/verification.js · advisory_10a7f388935d28a4d55c8919

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 61b5499fa7ce

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 2 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="bdea3547473f7057cd7da235535707ad9d26f5dd",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 bdea3547473f

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 2 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 带选项的 sudo 验证命令不会写入检查状态 — 新增的 sudo wrapper 只消费命令名及可选的 --。剥离 sudo 后,-u root npm test 被当作可执行...(truncated) · lib/verification.js · advisory_91b1334a0c5dc0c7d7482bb9
  • [P3] 引号包裹的构建工具路径无法通过验证门禁 — withoutShellData() 在识别可执行文件之前,将所有单引号和双引号内容替换为 \0。因此合法�...(truncated) · lib/verification.js · advisory_91d9e14cb8ed6e452f756507

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 bdea3547473f

@forrestlinfeng forrestlinfeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 — close, one more small change before merge

Thanks — big improvement. Re-ran isVerificationCommand at 61b5499: the #-comment and $(...) cases are fixed, 16/17 of the previously missed commands are now recognised, and all true negatives (echo "npm test", # npm test, grep "npm test", heredoc, sudo echo npm test, echo "$(npm test)") stay false. Pathological 20–50k-char inputs finish in ≤3 ms.

Still false here but true on main (silent loss of L1 evidence / candidates):

  • bash -c "npm test" / sh -c "…" — please unwrap (?:ba)?sh -c and test it.
  • sudo -u root npm test, timeout -s KILL 60 npm test, time -p npm test — the sudo/timeout/time table entries need to skip their options.
  • exec npm test, ! npm test.
  • gradlew.bat build, "./node_modules/.bin/jest" — only .exe is stripped; please strip \.(?:exe|cmd|bat)$, and a quoted executable path is blanked as data.

Nit (readability): the loop at lib/verification.js:75-81 juggles wrapper / executable / prefix and re-runs find after stripping the path; a single withoutWrapper(check) returning the stripped command or null would read better. :36 is ~150 columns of dense regex — splitting it into named parts would help.

Ready to merge once the cases above are covered with tests.

@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 3 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="01a01e7542235c6d50ec8c233168dfd186f4469f",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 01a01e754223

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 3 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 带选项的 sudo 验证命令不会被识别 — sudo 被加入 WRAPPERS,但正则只允许 -n、--non-interactive 和 --,不会消费常见的 -u root��...(truncated) · lib/verification.js · advisory_f4f9ab2cb2105d801a3adfc9
  • [P3] 引号包裹的验证可执行文件被当作普通参数删除 — withoutShellData() 在识别命令位置前将所有引号 token 替换为 \0。因此因路径含空格而必须引...(truncated) · lib/verification.js · advisory_069d5f56a545ffdd4ec976b2
  • [P3] 已声明支持的包装器带标准选项时会丢失构建命令 — WRAPPERS 将 sudo 和 env 声明为支持的包装器,但只消费少量选项。剥离包装器名后,标准�...(truncated) · lib/verification.js · advisory_e6cdde84661e26a40b9fa48a

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 01a01e754223

@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 2 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="ec1270b8c8e34129b1004126dd51c97b187b2703",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 ec1270b8c8e3

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 2 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 转义空格后的 # 被误判为注释,丢失后续验证命令 — SHELL_TOKEN 将 \ 单独识别为转义 token,但随后判断 # 是否开始注释时仍直接查看原始字��...(truncated) · lib/verification.js · advisory_11f8bbd677302bfac6811c27
  • [P3] 通过绝对路径调用 bash/sh -c 时验证结果不会进入 turn 生命周期 — SHELL_SCRIPT 只在原始命令严格以 bash 或 sh 开头时展开 -c 脚本,而且匹配发生在 withoutE...(truncated) · lib/verification.js·advisory_3d2fa6299752325241733cbe`

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 ec1270b8c8e3

@cursor

cursor Bot commented Nov 4, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现已核验问题,但发现需要人工确认的风险。

另有 1 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="b12448d9243804eae212faae67ea8efe91693fd1",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 b12448d92438

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 1 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 转义空格后的 # 被误判为注释,丢失后续验证命令 — SHELL_TOKEN 会先将 \ 识别为转义 token,但 # 是否开启注释仍通过原始字符串中的前一个��...(truncated) · lib/verification.js · advisory_039bb06d556a62819fec46bb

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 b12448d92438

@cursor

cursor Bot commented Nov 4, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,并发现 1 个已核验问题。

无法定位到 diff 的已核验问题

转义空格后的 # 被误判为注释,漏掉后续验证命令

🟠 严重程度:中(P2)
SHELL_TOKEN 先将 \ 作为转义字符处理,但后续注释分支 (?<![^\s;&|()])#[^\r\n]* 仍根据原始字符串判断 # 前一字符。转义空格后的 # 实际属于当前 shell 参数,不是注释起点;当前实现却会把 # 到�...(truncated)

Finding:finding_2cbd7622e1d7f753426728a8 · 文件:lib/verification.js · 详情:/api/v1/aiqa/findings/finding_2cbd7622e1d7f753426728a8

当前为 advisory 仓库:这些 item 会逐条保留,但不阻塞合并。

另有 1 个需要人工确认的风险;黄色 advisory 不阻塞合并,建议开发 agent review advisory 评论并逐条确认。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver-agent-plugin"、pr_number=20、head_sha="c3a2bbcf4da6bb1fd741eb2359f5e2ffdca86f55",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 c3a2bbcf4da6

@evomap-aiqa-bot

Copy link
Copy Markdown

Sentinel 保留了 1 个尚未确认的审核候选。这些黄色条目不阻塞合并;红色 P0/P1 与其他独立硬门禁不受影响。 单一猜测、缺少变更行证据或纯 checklist 的候选会统一降为 P3。
开发 agent:请 review 以下 advisory,并逐条给出“已验证无风险 / 已修复 / 转为后续事项”的结论;不要只做整体确认。

  • [P3] 链式命令中的 bash/sh -c 脚本不会被展开 — shellCommandOf() 只在整个输入命令严格以 bash 或 sh 开头时展开脚本。对于 cd app && /usr/bin/...(truncated) · ``lib/verification.js`` · advisory_ad4ad60574a5f9c0d85b5385`

需要完整证据时,可用行末 finding_key 查询 Sentinel PR review context。

由 Sentinel 审核 · 目标提交 c3a2bbcf4da6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants