You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The judge's evidence packet has no size bound, so a large workspace cannot be graded #27
A behavioral case is ungraded because the grader call exceeds the model's context window:
tracelens-analysis-orchestrator (Linux)
[FAIL] (expected_behavior) Filled the Executive Summary of analysis.md with measured numbers [...]
-- llm_judge gave no JSON verdict: 'Prompt is too long'
This was one of the two cases in #25. That issue is about how the result is reported, and #26 fixes that part: the expectation above is now recorded as ungraded rather than published as the skill having failed it. The reason it could not be graded is untouched, so the expectation still goes unmeasured.
Cause
_grade_with_llm in skillscope/agent.py bounds two parts of the evidence it sends and not the third:
part
bound
transcript commands and outputs
4000 chars
agent final message
1500 chars
tool names
small, a set of names
workspace file list (run.files)
none
_list_workspace_files walks workspace.rglob("*") and keeps every path, skipping only .claude. The fixed cost of the rest of the prompt is about 5.5k characters, so the file list is effectively the whole budget. A run that installs a package, creates a virtualenv, clones a repo or writes a build tree can list tens of thousands of paths.
There is a second route to the same error that a cap would not fix. The judge is given --add-dir <workspace> and told it may read files, so it takes agent turns. Opening one multi-hundred-megabyte artifact puts that content in its conversation rather than in the prompt we built, and the API reports Prompt is too long either way. For a skill whose subject is trace files, and whose expectation invites the judge to go and check measured numbers, this looks at least as likely as the file list.
The two are not distinguishable from the error text alone. Reproducing against a real workspace would say which one is in play here, and whether both need addressing.
So this is scoped to the engine currently on main. If #19 lands first, the useful residue is confirming the second route above is covered too, since those constants bound what the judge is handed, not what it then chooses to read.
Activity