fix(evaluation): remove public llm-grader assertion

.agents/conventions.md

@@ -141,6 +141,7 @@ Grader types use kebab-case everywhere.
 - Internal TypeScript may keep shared LLM grader implementation names, but new authored evals should not use `llm-grader` as a public type.
 - Output `scores[].type`: use the authored grader type when available, such as `"llm-rubric"`, `"g-eval"`, or `"is-json"`.
 - Registry keys: `registry.register('llm-rubric', ...)` for the authored rubric surface, with `llm-grader` retained as internal/shared implementation plumbing.
+- Public SDK assertion helpers and authored YAML schemas must not expose `llm-grader`; use `llmRubricGrader()` / `type: llm-rubric` or `type: agent-rubric`.
 
 Source of truth: `GRADER_KIND_VALUES` in `packages/core/src/evaluation/types.ts`.
 
@@ -149,10 +150,10 @@ Backward compatibility:
 - Snake_case is accepted in YAML by `normalizeGraderType()` in `grader-parser.ts`, for example `llm_rubric` -> `llm-rubric`.
 - Single-word types such as `contains`, `equals`, `regex`, `latency`, and `cost` are unchanged.
 
-Two type definitions exist and must stay in sync:
+Public and internal type definitions intentionally differ for internal-only grader plumbing:
 
-- `EvaluatorKind` in `packages/core/src/evaluation/types.ts`
-- `AssertionType` in `packages/sdk/src/assertion.ts`
+- `GraderKind` in `packages/core/src/evaluation/types.ts` includes internal/runtime keys such as `llm-grader`.
+- `AssertionType` in `packages/sdk/src/assertion.ts` includes public authored assertion types only.
 
 ## Python Scripts
 

apps/cli/test/commands/eval/pipeline/fixtures/input-test.eval.yaml

@@ -12,8 +12,8 @@ tests:
         command: node grader-score-1.js
         weight: 1
       - metric: relevance
-        type: llm-grader
-        prompt: Did the response echo the input?
+        type: llm-rubric
+        value: Did the response echo the input?
         weight: 2
     vars:
       input: hello world

apps/web/src/content/docs/docs/next/graders/llm-graders.mdx

@@ -50,8 +50,8 @@ Use `provider:` when you want different `llm-rubric` entries in the same eval to
 
 Use `type: agent-rubric` when a Promptfoo-style rubric needs an agentic grader
 that can inspect the workspace instead of only judging the final answer text.
-AgentV routes `agent-rubric` through the same `llm-rubric` / `llm-grader`
-scoring path, so results still appear as normal `EvaluationScore` entries.
+AgentV routes `agent-rubric` through the same shared LLM rubric scoring path,
+so results still appear as normal `EvaluationScore` entries.
 
 ```yaml
 assert:

apps/web/src/content/docs/docs/v4.42.4/evaluation/eval-cases.mdx

@@ -86,7 +86,7 @@ tests:
       target: gpt4_target
     assertions:
       - name: depth_check
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/depth.md
 ```
 
@@ -111,7 +111,7 @@ tests:
       skip_defaults: true
     assertions:
       - name: custom_eval
-        type: llm-grader
+        type: llm-rubric
     # Does NOT get latency_check
 ```
 
@@ -330,14 +330,14 @@ The `criteria` field is a **data field** that describes what the response should
 
 ### No `assertions` — implicit LLM grader
 
-When a test has no `assertions` field, a default `llm-grader` grader runs automatically and uses `criteria` as the evaluation prompt:
+When a test has no `assertions` field, a default `llm-rubric` grader runs automatically and uses `criteria` as the evaluation prompt:
 
 ```yaml
 tests:
   - id: simple-eval
     criteria: Assistant correctly explains the bug and proposes a fix
     input: "Debug this function..."
-    # No assertions → default llm-grader evaluates against criteria
+    # No assertions → default llm-rubric evaluates against criteria
 ```
 
 Suite-level `preprocessors` also apply to this implicit grader. That matters when the agent output is a `ContentFile` block rather than plain text:
@@ -355,13 +355,13 @@ tests:
 
 ### `assertions` present — explicit graders only
 
-When `assertions` is defined, only the declared graders run. No implicit grader is added. Graders that are declared (such as `llm-grader`, `code-grader`, or `rubrics`) receive `criteria` as input automatically.
+When `assertions` is defined, only the declared graders run. No implicit grader is added. Graders that are declared (such as `llm-rubric`, `script`, or `rubrics`) receive `criteria` as input automatically.
 
 If `assertions` contains only deterministic graders (like `contains` or `regex`), the `criteria` field is not evaluated and a warning is emitted:
 
 ```
 Warning: Test 'my-test': criteria is defined but no grader in assertions
-will evaluate it. Add 'type: llm-grader' to assertions, or remove criteria
+will evaluate it. Add 'type: llm-rubric' to assertions, or remove criteria
 if it is documentation-only.
 ```
 
@@ -373,7 +373,7 @@ tests:
     criteria: Response is helpful and mentions the fix
     input: "Debug this function..."
     assertions:
-      - type: llm-grader        # explicit — receives criteria automatically
+      - type: llm-rubric        # explicit — receives criteria automatically
       - type: contains
         value: "fix"
 ```
@@ -390,7 +390,7 @@ tests:
     criteria: Response is helpful and mentions the fix
     input: "Debug this function..."
     assertions:
-      - type: llm-grader
+      - type: llm-rubric
         preprocessors:
           - type: xlsx
             command: ["bun", "run", "scripts/preprocessors/xlsx-to-json.ts"]

apps/web/src/content/docs/docs/v4.42.4/evaluation/examples.mdx

@@ -91,7 +91,7 @@ tests:
         command: [uv, run, validate_json.py]
         cwd: ./graders
       - name: content_evaluator
-        type: llm-grader
+        type: llm-rubric
         prompt: ./graders/semantic_correctness.md
 
     input: |-
@@ -108,7 +108,7 @@ tests:
 
 ## File Output Preprocessing
 
-Convert a binary file output into text before the `llm-grader` sees it:
+Convert a binary file output into text before the `llm-rubric` sees it:
 
 ```yaml
 description: Grade spreadsheet output via a preprocessor
@@ -183,15 +183,15 @@ assertions:
     threshold: 0.6
     assertions:
       - name: grader-gpt-5-mini
-        type: llm-grader
+        type: llm-rubric
         target: grader_gpt_5_mini
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-claude-haiku
-        type: llm-grader
+        type: llm-rubric
         target: grader_claude_haiku
         prompt: ../prompts/grader-pass-fail-v1.md
       - name: grader-gemini-flash
-        type: llm-grader
+        type: llm-rubric
         target: grader_gemini_flash
         prompt: ../prompts/grader-pass-fail-v1.md
 ```

apps/web/src/content/docs/docs/v4.42.4/graders/execution-metrics.mdx

@@ -129,7 +129,7 @@ tests:
     assertions:
       # Semantic quality
       - name: quality
-        type: llm-grader
+        type: llm-rubric
         prompt: ./prompts/code-quality.md
 
       # Efficiency constraints

apps/web/src/content/docs/docs/v4.42.4/graders/llm-graders.mdx

@@ -12,14 +12,14 @@ LLM graders use a language model to evaluate agent responses against custom crit
 
 ## Default Grader
 
-When a test defines `criteria` but has **no `assertions` field**, a default `llm-grader` runs automatically. The built-in prompt evaluates the response against your `criteria` and `expected_output`:
+When a test defines `criteria` but has **no `assertions` field**, a default `llm-rubric` runs automatically. The built-in prompt evaluates the response against your `criteria` and `expected_output`:
 
 ```yaml
 tests:
   - id: simple-eval
     criteria: Correctly explains the bug and proposes a fix
     input: "Debug this function..."
-    # No assertions needed — default llm-grader evaluates against criteria
+    # No assertions needed — default llm-rubric evaluates against criteria
 ```
 
 When `assertions` **is** present, no default grader is added. To use an LLM grader alongside other graders, declare it explicitly. See [How criteria and assertions interact](/docs/v4.42.4/evaluation/eval-cases/#how-criteria-and-assertions-interact).
@@ -31,12 +31,12 @@ Reference an LLM grader in your eval file:
 ```yaml
 assertions:
   - name: semantic_check
-    type: llm-grader
+    type: llm-rubric
     prompt: file://graders/correctness.md
     target: grader_gpt_5_mini   # optional: route this grader to a named LLM target
 ```
 
-Use `target:` when you want different `llm-grader` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.
+Use `target:` when you want different `llm-rubric` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.
 
 ## Prompt Files
 
@@ -100,26 +100,26 @@ tests:
       row: 1
     assertions:
       - name: dexter_semantic
-        type: llm-grader
+        type: llm-rubric
         prompt: file://prompts/dexter-grader.md
-        rubrics:
+        value:
           - operator: correctness
-            criteria: Uses the provided ticker and company.
+            outcome: Uses the provided ticker and company.
 ```
 
 ## Per-Grader Target
 
-By default, an `llm-grader` uses `defaults.grader` from the resolved config graph.
+By default, an `llm-rubric` uses `defaults.grader` from the resolved config graph.
 Override it per grader when you need multiple grader models in one run:
 
 ```yaml
 assertions:
   - name: grader-gpt
-    type: llm-grader
+    type: llm-rubric
     target: grader_gpt_5_mini
     prompt: ./prompts/pass-fail.md
   - name: grader-haiku
-    type: llm-grader
+    type: llm-rubric
     target: grader_claude_haiku
     prompt: ./prompts/pass-fail.md
 ```
@@ -178,7 +178,7 @@ When using TypeScript templates, configure them in YAML with optional `config` d
 ```yaml
 assertions:
   - name: custom-eval
-    type: llm-grader
+    type: llm-rubric
     prompt:
       command: [bun, run, ../prompts/custom-grader.ts]
       config:
@@ -190,7 +190,7 @@ The `config` object is available as `ctx.config` inside the template function.
 
 ## Preprocessing File Outputs
 
-If an agent returns a `ContentFile` block instead of plain text, you can preprocess that file into text before `llm-grader` builds the candidate prompt.
+If an agent returns a `ContentFile` block instead of plain text, you can preprocess that file into text before `llm-rubric` builds the candidate prompt.
 
 AgentV always tries a default UTF-8 text read first. That is enough for text-based formats such as CSV, JSON, SQL, Markdown, YAML, HTML, XML, and plain text. For binary formats such as `.xlsx`, `.pdf`, or `.docx`, add a preprocessor command:
 
@@ -205,7 +205,7 @@ tests:
     input: Generate the spreadsheet report
     assertions:
       - name: spreadsheet-check
-        type: llm-grader
+        type: llm-rubric
         prompt: |
           Check whether the transformed spreadsheet text contains the revenue rows:
 
@@ -220,7 +220,7 @@ Resolution order:
 - if no preprocessor matches, AgentV falls back to a UTF-8 text read
 - if the fallback read looks binary or invalid, the grader receives a warning note instead of failing the test run
 
-The implicit default `llm-grader` also inherits suite-level `preprocessors`, so you can omit `assertions` and still preprocess file outputs before grading.
+The implicit default `llm-rubric` also inherits suite-level `preprocessors`, so you can omit `assertions` and still preprocess file outputs before grading.
 
 See [`examples/features/preprocessors/`](../../../../../examples/features/preprocessors/) for a runnable example with a file-producing target and a custom preprocessor script.
 

apps/web/src/content/docs/docs/v4.42.4/guides/benchmark-provenance.mdx

@@ -212,7 +212,7 @@ description: Generated finance research cases with row-level provenance.
 
 assertions:
   - name: answer-quality
-    type: llm-grader
+    type: llm-rubric
     prompt: ./graders/finance-answer.md
     required: true
 

apps/web/src/content/docs/docs/v4.42.4/guides/evaluation-types.mdx

@@ -67,7 +67,7 @@ AgentV's eval tooling is designed for **execution quality**:
 - **`EVAL.yaml`** — define test cases with inputs, expected outputs, and assertions
 - **`evals.json`** — lightweight skill evaluation format (prompt/expected-output pairs)
 - **`agentv eval`** — execute evaluations and collect results
-- **Graders** — `llm-grader`, `code-grader`, `tool-trajectory`, `rubrics`, `contains`, `regex`, and others all measure execution behavior
+- **Graders** — `llm-rubric`, `script`, `tool-trajectory`, `rubrics`, `contains`, `regex`, and others all measure execution behavior
 
 These tools assume the skill is already loaded and invoked. They measure what happens *after* routing, not the routing decision itself.
 

apps/web/src/content/docs/docs/v4.42.4/guides/human-review.mdx

@@ -96,7 +96,7 @@ The `feedback.json` file is a structured annotation of a single eval run. It rec
       "notes": "Missing coverage of multi-document queries.",
       "evaluator_overrides": {
         "code-grader:format-check": "Too strict — penalized valid output with trailing newline",
-        "llm-grader:quality": "Score 0.6 seems fair, answer was incomplete"
+        "llm-rubric:quality": "Score 0.6 seems fair, answer was incomplete"
       },
       "workspace_notes": "Workspace had stale cached files from previous run — may have affected retrieval results."
     },
@@ -148,7 +148,7 @@ For workspace evaluations with multiple graders (code graders, LLM graders, tool
   "verdict": "needs_improvement",
   "evaluator_overrides": {
     "code-grader:test-pass": "Tests pass but the refactored code has a subtle race condition the tests don't cover",
-    "llm-grader:quality": "Score 0.9 is too high — the agent left dead code behind",
+    "llm-rubric:quality": "Score 0.9 is too high — the agent left dead code behind",
     "tool-trajectory:efficiency": "Used 12 tool calls where 5 would suffice, but the result is correct"
   },
   "workspace_notes": "Agent cloned the repo correctly but didn't clean up temp files."

apps/web/src/content/docs/docs/v4.42.4/guides/skill-improvement-workflow.mdx

@@ -234,15 +234,15 @@ tests:
         content: "Review this Python function for bugs:..."
     assertions:
       - name: assertion-1
-        type: llm-grader
+        type: llm-rubric
         prompt: "Identifies the division by zero risk"
       # Replace with type: contains for deterministic checks:
       # - type: contains
       #   value: "ZeroDivisionError"
 ```
 
 After converting, you can:
-- Replace `llm-grader` assertions with faster deterministic graders (`contains`, `regex`, `equals`)
+- Replace `llm-rubric` assertions with faster deterministic graders (`contains`, `regex`, `equals`)
 - Add `workspace` configuration for file-system isolation
 - Use `code-grader` for custom scoring logic
 - Define `tool-trajectory` assertions to check tool usage patterns

apps/web/src/content/docs/docs/v4.42.4/integrations/agent-skills-evals.mdx

@@ -59,7 +59,7 @@ When AgentV loads `evals.json`, it promotes fields to its internal representatio
 |---|---|---|
 | `prompt` | `input` | Wrapped as `[{role: "user", content: prompt}]` |
 | `expected_output` | `expected_output` + `criteria` | Used as reference answer and evaluation criteria |
-| `assertions[]` | `assertions[]` | Each string becomes `{type: llm-grader, prompt: text}` |
+| `assertions[]` | `assertions[]` | Each string becomes `{type: llm-rubric, value: text}` |
 | `files[]` | `file_paths` | Resolved relative to evals.json, copied into workspace |
 | `skill_name` | `metadata.skill_name` | Carried as metadata |
 | `id` (number) | `id` (string) | Converted via `String(id)` |
@@ -171,7 +171,7 @@ tests:
     # Replace with type: is_json, contains, or regex for deterministic checks
     assertions:
       - name: assertion-1
-        type: llm-grader
+        type: llm-rubric
         prompt: "Output identifies November as the highest revenue month"
 ```
 
@@ -236,17 +236,17 @@ tests:
       An empathetic response that offers to track the order and provides next steps.
     assertions:
       - name: acknowledges-frustration
-        type: llm-grader
+        type: llm-rubric
         prompt: Response acknowledges the customer's frustration
       - name: looks-up-order
         type: contains
         value: "12345"
       - name: has-next-steps
-        type: llm-grader
+        type: llm-rubric
         prompt: Response provides clear next steps
 ```
 
-Notice how the EVAL.yaml version can mix `llm-grader` (for subjective checks) with `contains` (for deterministic checks) — the order number check is now instant and free.
+Notice how the EVAL.yaml version can mix `llm-rubric` (for subjective checks) with `contains` (for deterministic checks) — the order number check is now instant and free.
 
 ## References
 

apps/web/src/content/docs/docs/v4.42.4/integrations/phoenix.mdx

@@ -52,7 +52,7 @@ bun packages/phoenix-adapter/src/cli.ts run \
 | `regex` | Converts to deterministic Phoenix evaluator logic |
 | `equals` | Converts to deterministic Phoenix evaluator logic |
 | `is-json` | Converts to deterministic Phoenix evaluator logic |
-| `llm-grader`, rubrics, `code-grader`, `tool-trajectory`, assert-set, metrics, and custom families | Reported as unsupported in the adapter report |
+| `llm-rubric`, rubrics, `code-grader`, `tool-trajectory`, assert-set, metrics, and custom families | Reported as unsupported in the adapter report |
 
 Unsupported families do not fail conversion by default. Add
 `--fail-on-unsupported` when a parity report should fail CI if any suite needs a

apps/web/src/content/docs/docs/v4.42.4/tools/convert.mdx

@@ -38,7 +38,7 @@ Converts an [Agent Skills `evals.json`](/docs/v4.42.4/integrations/agent-skills-
 
 - Maps `prompt` → `input` message array
 - Maps `expected_output` → `expected_output`
-- Maps `assertions` → `assertions` graders (llm-grader)
+- Maps `assertions` → `assertions` graders (`llm-rubric`)
 - Resolves `files[]` paths relative to the evals.json directory
 - Adds TODO comments for AgentV-specific features (workspace setup, code graders, rubrics)
 

apps/web/src/content/docs/docs/v4.42.4/tools/import.mdx

@@ -51,7 +51,7 @@ Default output: `EVAL.yaml` beside the promptfoo config file.
 - per-test `vars`, `description`, `threshold`, `metadata`, prompt filters, and provider filters
 - simple prompt templates are preserved as AgentV `{{var}}` input templates instead of being eagerly flattened
 - deterministic assertions that map directly to AgentV: `equals`, `contains`, `icontains`, `regex`, `starts-with`, `ends-with`, `contains-any`, `contains-all`, `icontains-any`, `icontains-all`, `is-json`, `latency`, `cost`
-- rubric-style assertions mapped to `llm-grader`: `llm-rubric`, `g-eval`, `factuality`, `context-faithfulness`, `context-recall`
+- rubric-style assertions mapped to `llm-rubric`: `llm-rubric`, `g-eval`, `factuality`, `context-faithfulness`, `context-recall`
 
 ### What still needs manual migration
 

packages/core/src/evaluation/evaluate.ts

@@ -136,7 +136,7 @@ export interface ConversationTurnInput {
  * Matches the YAML `assert` block structure.
  */
 export interface EvalAssertionInput {
-  /** Assertion type (e.g., 'contains', 'llm-grader', 'script') */
+  /** Assertion type (e.g., 'contains', 'llm-rubric', 'script') */
   readonly type: string;
   /** Score/check metric name */
   readonly metric?: string;
@@ -148,7 +148,7 @@ export interface EvalAssertionInput {
   readonly required?: boolean;
   /** Minimum score (0-1) for this evaluator to pass. Independent of `required` gate. */
   readonly min_score?: number;
-  /** Prompt file for llm-grader */
+  /** Prompt file for LLM rubric assertions */
   readonly prompt?: string;
   /** Command for script grader */
   readonly command?: string | readonly string[];