Repository navigation
feat: tutorial-level AI grading context (#3090) - #3119
Draft
jung-thomas wants to merge 13 commits into
Draft
jung-thomas wants to merge 13 commits into
jung-thomas wants to merge 13 commits into
Conversation
…#3090) - publishValidateAnswerSpecs: destructure `gradingContext` from payload; byte-cap reject (400 too_long) before any DB write; inside cds.tx() REPLACE TutorialGradingContext row (DELETE then conditional INSERT); absent/empty → DELETE only (clears stale); TutorialGradingContext entity absent (srv-qa) → skip without throwing. - fetch-tutorials.ts: add `gradingContext` to parseRulesVrEnriched destructure; write sidecar when aiGradedSpecs present OR gradingContext present; include gradingContext in sidecar payload when non-empty. - publish-validate-answer.js: update comment to document new payload shape { slug, specs, gradingContext? }; raw-bytes send already carries it. - Test: 9 cases covering write, replace, clear, byte-cap, unknown-slug.
- validate-answer-prompt: PROMPT_VERSION v4→v5; buildUserMessage gains optional gradingContext inserted after videoContext, before expected- answer; system prompt adds grading-context domain-guidance line - code-check-prompt: PROMPT_VERSION v2→v3; buildUserMessage gains optional gradingContext inserted before referenceSolution; system prompt adds grading-context domain-guidance line (rule 10) - Both: absent/empty gradingContext → byte-identical to prior version - Tests: 4 new cases per module (present, positioning, baseline, sys prompt)
- srv/admin-service.cds: add @readonly TutorialGradingContext projection; add Tutorials.gradingContext to-one association mirroring codeCheckSpecs - app/admin-annotations.cds: field annotations + FieldGroup#GradingContext for TutorialGradingContext; add AI Grading Context facet to Tutorials OP between Code-Check Specs and AI-Author Requests - app/admin/joule/webapp/view/Settings.view.xml: add gradingContextEnabled Switch in the AI Grading panel, mirroring codeCheckEnabled layout - app/admin/joule/webapp/controller/Settings.controller.js: wire gradingContextEnabled into JSONModel default, _loadSettings, and onSave - app/admin/joule/webapp/i18n/i18n.properties: add gradingContextEnabled and gradingContextEnabledHelp keys - srv/__tests__/admin-grading-context-projection.test.js: TDD test (4/4 pass) verifying GET, POST/PATCH rejection, and gradingContext $expand
New reference doc docs/developers/reference/grading-context.md covers: - [GRADING_CONTEXT] block syntax (###Domain/###GradingCriteria/###CommonMistakes) - Tutorial-wide scope, all three graders (validate-answer, code-check, challenge-grade) - Published via existing validate-answer-specs publish — no new endpoint - Runtime: grading-context-loader.js reads TutorialGradingContext, injected via buildUserMessage(); gated by ChatSettings.gradingContextEnabled (default OFF, fail-open) - Admin-UI read-only display on Tutorial object page srv-qa cp-list audit: grading-context-loader.js is NOT reachable from content-store.js (imported only by server.js) — no mta.yaml change needed. Cross-reference line added to docs/developers/architecture/free-text-grader.md.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Implements Option A of #3090 — tutorial-level AI grading context. Authors can now give the AI graders domain focus (e.g. n8n workflows in Joule Studio) so verdicts are accurate for specialized content instead of relying on generic LLM reasoning.
Spec:
docs/superpowers/specs/2026-11-12-grading-context-design.mdPlan:
docs/superpowers/plans/2026-11-12-grading-context.mdHow it works
[GRADING_CONTEXT]block inrules.vrwith light sub-sections (###Domain,###GradingCriteria,###CommonMistakes).validate-answer-specspublish into a new server-onlyTutorialGradingContextentity (one row per tutorial).grading-context-loader.jsand inject it into the prompt as a "domain guidance, do not quote" section (same pattern asvideoContext).ChatSettings.gradingContextEnabledflag — default OFF, grounding-only (never gates the grade), fail-open.Scope / non-goals (v1)
No deterministic code validators, no live tools/MCP, no per-step overrides, no author-editable Admin. Context is not routed through the leak-redaction guard (by design — it's domain guidance, kept as an authoring guideline).
Tests
10 feature test files, 101 tests green (parser, entity, publish, loader, both prompt modules, all three graders, admin projection). Each grader verified: flag-OFF still grades, loader-error fails open, absent context → byte-identical prompt. srv-qa entity-absent paths guarded in publish handler, loader, and the new challenge-grade ChatSettings read.
Built via spec → plan → subagent-driven implementation (per-task review + whole-branch review, all clean).
Docs:
docs/developers/reference/grading-context.md.Closes #3090.