Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions app/admin-annotations.cds
Original file line number Diff line number Diff line change
Expand Up @@ -967,6 +967,23 @@ annotate AdminService.CodeCheckSpecs with @UI: {
}
};

// TutorialGradingContext — tutorial-level grading context persisted at
// publish time and consumed by the challenge grader to ground LLM calls.
// Read-only; the admin facet surfaces the context text for visibility.
// Joined via the `gradingContext` to-one association on the Tutorials projection.
annotate AdminService.TutorialGradingContext with {
context @Common.Label: 'Grading Context' @UI.MultiLineText;
};

annotate AdminService.TutorialGradingContext with @UI: {
FieldGroup #GradingContext: {
$Type: 'UI.FieldGroupType',
Data: [
{ Value: context, Label: 'Grading Context' }
]
}
};

// AuthorAiRequests — per-tutorial AI-author telemetry (OS variant
// generation today; future flows extend). Joined via `aiRequests`
// association (PR-3 added the tutorial FK).
Expand Down Expand Up @@ -1033,6 +1050,8 @@ annotate AdminService.Tutorials with @UI: {
Target: 'validationRules/@UI.LineItem' },
{ $Type: 'UI.ReferenceFacet', Label: 'Code-Check Specs', ID: 'CodeCheckSpecsFacet',
Target: 'codeCheckSpecs/@UI.LineItem' },
{ $Type: 'UI.ReferenceFacet', Label: 'AI Grading Context', ID: 'GradingContextFacet',
Target: 'gradingContext/@UI.FieldGroup#GradingContext' },
{ $Type: 'UI.ReferenceFacet', Label: 'AI-Author Requests', ID: 'AiRequestsFacet',
Target: 'aiRequests/@UI.LineItem' },
{ $Type: 'UI.ReferenceFacet', Label: 'Freshness Reports', ID: 'FreshnessReportsFacet', Target: 'freshnessReports/@UI.PresentationVariant' },
Expand Down
8 changes: 8 additions & 0 deletions app/admin/joule/webapp/controller/Settings.controller.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ sap.ui.define([
// in the view makes the cost trade-off explicit.
validateAnswerEnabled: false,
codeCheckEnabled: false,
// Grading context flag (#3090). Default false to mirror schema default
// (db/schema.cds). When on, all three AI graders (validate-answer,
// code-check, challenge-grade) read each tutorial's TutorialGradingContext
// row to ground the LLM call with domain-specific context about what
// correct answers look like. No extra token cost beyond what the active graders consume.
gradingContextEnabled: false,
// Issue #172 pilot master flag. Default false to mirror the schema
// default (db/schema.cds:530). When off, the public /api/ChatConfig
// singleton reports branchingEnabled=false and the Learning
Expand Down Expand Up @@ -108,6 +114,7 @@ sap.ui.define([
embeddingMinScore: data.embeddingMinScore != null ? data.embeddingMinScore : 0.25,
validateAnswerEnabled: !!data.validateAnswerEnabled,
codeCheckEnabled: !!data.codeCheckEnabled,
gradingContextEnabled: !!data.gradingContextEnabled,
branchingEnabled: !!data.branchingEnabled,
communityPeersEnabled: !!data.communityPeersEnabled,
communityLabelLlmBudgetPerDay: data.communityLabelLlmBudgetPerDay != null ? data.communityLabelLlmBudgetPerDay : 50,
Expand Down Expand Up @@ -154,6 +161,7 @@ sap.ui.define([
embeddingMinScore: data.embeddingMinScore === "" || data.embeddingMinScore == null ? null : Number(data.embeddingMinScore),
validateAnswerEnabled: !!data.validateAnswerEnabled,
codeCheckEnabled: !!data.codeCheckEnabled,
gradingContextEnabled: !!data.gradingContextEnabled,
branchingEnabled: !!data.branchingEnabled,
communityPeersEnabled: !!data.communityPeersEnabled,
communityLabelLlmBudgetPerDay: parseInt(data.communityLabelLlmBudgetPerDay, 10) || 50,
Expand Down
2 changes: 2 additions & 0 deletions app/admin/joule/webapp/i18n/i18n.properties
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ validateAnswerEnabled=Free-text answer grading
validateAnswerEnabledHelp=When on, [VALIDATE_N] free-text questions are graded by the LLM. When off, the inline widget shows "Answer checking is temporarily unavailable" and the /api/validate-answer endpoint returns 503. Default off — flipping this on starts consuming AI Core tokens per submission.
codeCheckEnabled=AI code-check
codeCheckEnabledHelp=When on, [CODECHECK_N] blocks call the LLM grader and the checkCode tool is wired into the chat. When off, the /api/codecheck endpoint returns 503. Default off — flipping this on starts consuming AI Core tokens per submission.
gradingContextEnabled=Grading context enrichment
gradingContextEnabledHelp=When on, each tutorial's stored grading context (published via the content pipeline) is injected into all AI graders (validate-answer, code-check, challenge-grade) as domain-specific guidance about correct answers. No extra AI Core tokens beyond what the active graders already consume. Default off.

branchingSection=Tutorial Branching Pilot (issue #172)
branchingEnabled=Enable adaptive branching
Expand Down
6 changes: 6 additions & 0 deletions app/admin/joule/webapp/view/Settings.view.xml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,12 @@

<Label text="" />
<Text text="{i18n>codeCheckEnabledHelp}" wrapping="true" />

<Label text="{i18n>gradingContextEnabled}" />
<Switch state="{settings>/gradingContextEnabled}" />

<Label text="" />
<Text text="{i18n>gradingContextEnabledHelp}" wrapping="true" />
</f:SimpleForm>
</Panel>

Expand Down
7 changes: 7 additions & 0 deletions db/last-dev/csn.json
Original file line number Diff line number Diff line change
Expand Up @@ -3115,6 +3115,13 @@
},
"@cds.persistence.name": "VALIDATEANSWERENABLED"
},
"gradingContextEnabled": {
"type": "cds.Boolean",
"default": {
"val": false
},
"@cds.persistence.name": "GRADINGCONTEXTENABLED"
},
"branchingEnabled": {
"type": "cds.Boolean",
"default": {
Expand Down
14 changes: 14 additions & 0 deletions db/schema.cds
Original file line number Diff line number Diff line change
Expand Up @@ -797,6 +797,11 @@ entity ChatSettings : cuid, managed {
// and the dispatch short-circuits without calling the LLM.
validateAnswerEnabled : Boolean default false;

// Tutorial-level AI grading context (issue #3090). When true, the grading
// subsystem uses context from TutorialGradingContext to enhance LLM-based
// grading for AI-evaluated quiz questions and challenges.
gradingContextEnabled : Boolean default false;

// Branching paths runtime master flag (issue #172). When false:
// - /api/branches/decide → 404
// - /build/mission/<slug> omits `recommendation`
Expand Down Expand Up @@ -1032,6 +1037,15 @@ entity ChallengeAnswers : managed {
reference : LargeString @mandatory;
}

// Tutorial-level AI grading context (issue #3090). Provides semantic/background
// information for a tutorial to enhance LLM-based grading. Server-only — context
// is never shipped to clients and is read by the grading subsystem at request time.
// One row per tutorial.
entity TutorialGradingContext : managed {
key tutorial : Association to Tutorials;
context : LargeString;
}

// Full parsed rules.vr rule set for a tutorial, persisted at publish time.
// Unlike ValidateAnswerSpecs (AI-graded only), this holds ALL rule types so
// the admin Validation Questions facet can display the complete rule set.
Expand Down
7 changes: 6 additions & 1 deletion db/src/com.sap.developers.ims.ChatSettings.hdbmigrationtable
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
== version=19
== version=20
COLUMN TABLE com_sap_developers_ims_ChatSettings (
ID NVARCHAR(36) NOT NULL,
createdAt TIMESTAMP,
Expand All @@ -19,6 +19,7 @@ COLUMN TABLE com_sap_developers_ims_ChatSettings (
semanticSearchEnabled BOOLEAN DEFAULT FALSE,
codeCheckEnabled BOOLEAN DEFAULT FALSE,
validateAnswerEnabled BOOLEAN DEFAULT FALSE,
gradingContextEnabled BOOLEAN DEFAULT FALSE,
branchingEnabled BOOLEAN DEFAULT FALSE,
kgPathBetweenEnabled BOOLEAN DEFAULT FALSE,
kgSearchExpansionEnabled BOOLEAN DEFAULT TRUE,
Expand All @@ -43,6 +44,10 @@ COLUMN TABLE com_sap_developers_ims_ChatSettings (
PRIMARY KEY(ID)
)

== migration=20
-- generated by cds-compiler version 7.0.1
ALTER TABLE com_sap_developers_ims_ChatSettings ADD (gradingContextEnabled BOOLEAN DEFAULT FALSE);

== migration=19
-- generated by cds-compiler version 7.0.1
ALTER TABLE com_sap_developers_ims_ChatSettings ADD (semanticSearchEnabled BOOLEAN DEFAULT FALSE);
Expand Down
20 changes: 20 additions & 0 deletions docs/authors/writing-tutorials.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,26 @@ You can also have the platform **generate** quiz questions for you at build time

A hand-authored `[VALIDATE_N]` question for a step always wins over any `[AUTOAUTHOR_*]` directive for that step — the generated questions are opt-in *extra* coverage, never an override. If a video has no captions the generated quiz is simply skipped; your build still succeeds.

#### Focusing the AI grader on your tutorial's domain

When your tutorial covers specialized material (for example an n8n workflow in Joule Studio), the AI grader can judge submissions more accurately if you tell it what the domain is and what a good answer looks like. Add **one** `[GRADING_CONTEXT]` block to `rules.vr` — it applies to **every** AI-graded question, code-check, and challenge in the whole tutorial:

```
[GRADING_CONTEXT]
###Domain
This tutorial teaches building n8n workflows in Joule Studio. Submissions are
the exported workflow JSON.
###GradingCriteria
A correct workflow starts with a Trigger node, wires nodes through the
connections map (not just placing them), and ends with the action the step
asks for. Node naming is free-form; wiring is not.
###CommonMistakes
Learners often paste disconnected nodes, omit the trigger, or submit a
screenshot's text instead of the exported JSON.
```

All three sub-sections are optional. The text is **guidance for the grader**, not an answer key — do not put the expected answer here (it is not hidden from the model's reasoning and is not covered by answer-leak protection). Grounding only takes effect once a platform admin enables it; until then your block is parsed and stored but has no effect on grading.

---

## 3. Anatomy of a tutorial
Expand Down
1 change: 1 addition & 0 deletions docs/developers/architecture/free-text-grader.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,5 +206,6 @@ the design sketch.
- Sibling docs:
- [Validation widget](./validation-widget.md) (PR #226 — local-first widget this extends)
- AI code-check spike: `docs/superpowers/specs/2026-06-02-ai-code-check-spike-design.md` (PR #205 — same prompt redaction + rate-limit pattern)
- [Tutorial-level grading context](../reference/grading-context.md) (#3090 — `[GRADING_CONTEXT]` block in `rules.vr` for domain grounding; applies to all three graders)
- Spec: `docs/superpowers/specs/2026-06-04-209-free-text-grader-design.md`
- Tracking: [sap-tutorials/tutorials-ims#209](https://github.com/sap-tutorials/tutorials-ims/issues/209)
124 changes: 124 additions & 0 deletions docs/developers/reference/grading-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Tutorial-level AI Grading Context (`[GRADING_CONTEXT]`)

**Issue:** [#3090](https://github.com/sap-tutorials/tutorials-ims/issues/3090)
**Design:** [`docs/superpowers/specs/2026-11-12-grading-context-design.md`](../../superpowers/specs/2026-11-12-grading-context-design.md)

## What it is

A tutorial-wide block in a `-Contribution` `rules.vr` file that supplies
domain grounding to the AI graders. Without it, graders reason generically
about a learner's answer. With it, the grader knows the tutorial's domain,
what a correct submission looks like in that context, and what mistakes
learners typically make.

The block is **tutorial-wide** — one block per `rules.vr`, not one per
step. It applies to **all three** AI graders for that tutorial:
`validate-answer`, `code-check`, and `challenge-grade`.

## Authoring syntax

Add a `[GRADING_CONTEXT]` block anywhere in the tutorial's `rules.vr`
file (conventionally at the top, before any `[VALIDATE_N]` or
`[CODECHECK_N]` blocks):

```
[GRADING_CONTEXT]
###Domain
<Describe the tutorial's subject area — what technology/tool/concept a
correct submission is working with.>
###GradingCriteria
<Describe what a correct submission looks like in this domain. Focus on
structural or semantic properties the grader should verify, not on
reproducing the expected answer.>
###CommonMistakes
<Describe typical learner errors in this domain that should influence the
verdict.>
```

All three sub-sections are **optional**. An empty `[GRADING_CONTEXT]`
block, or omitting the block entirely, gives today's behavior: grading
without domain grounding.

### Example (n8n workflows)

```
[GRADING_CONTEXT]
###Domain
This tutorial teaches building n8n workflows in Joule Studio. Submissions
are n8n workflow JSON exported from the canvas.
###GradingCriteria
A correct workflow starts with a Trigger node, connects nodes via the
"connections" map (not just placing them), and ends with an action that
produces the tutorial's stated output. Node naming is free-form; wiring
is not.
###CommonMistakes
Learners often paste disconnected nodes, omit the trigger, or submit the
canvas screenshot's text instead of the exported JSON.
```

## How it is published

The parser (`scripts/parsers/rules.ts`) assembles the sub-sections into
one context string (order: Domain → GradingCriteria → CommonMistakes,
empties omitted) at parse time. The assembled string is stored in the
server-only `TutorialGradingContext` entity (one row per tutorial,
upserted on slug) as part of the existing `validate-answer-specs` publish
call. There is no separate publish endpoint.

## Runtime injection

At grading time, `srv/lib/grading-context-loader.js`
(`defaultLoadGradingContext(slug)`) reads the assembled string and passes
it to each grader's `buildUserMessage()` as a fenced section labeled
*"Tutorial grading context (domain guidance for your judgment only — do
not quote)"*, inserted before the per-step reference/expected answer.

This injection is **gated by `ChatSettings.gradingContextEnabled`**
(default **OFF**). To enable on a given environment:

```bash
# In a hybrid CAP shell or via the Admin-UI Joule Settings tile
await UPDATE('com.sap.developers.ims.ChatSettings').set({ gradingContextEnabled: true });
```

The Admin-UI Settings tile (`/admin-ui/#joule-display`) exposes the
toggle next to the `validateAnswerEnabled` / `codeCheckEnabled` toggles.

## Fail-open contract

A missing context row, a DB read error, or a `gradingContextEnabled:
false` flag all degrade silently to today's behavior: the grade proceeds
without domain grounding. The grading call is never blocked or failed
by a missing context.

## Anti-goals (authoring guidelines)

- **Do not embed per-question reference answers** in `[GRADING_CONTEXT]`.
The block is domain guidance for the grader's judgment, not a shortcut
to supply per-step answers. The existing `redactReferenceLeaks` guard
does **not** scan the context block, so embedding answers there would
bypass that protection.
- **Treat the block as guidance for the LLM's judgment**, not as
instructions for the LLM to follow. The system prompt reinforces this.

## Admin-UI read-only display

The assembled context string is visible (read-only) on the Tutorial
object page in the Admin-UI, in the AI-Grading facet group. There is no
write path in the UI — authoring is always through `rules.vr` and the
content publish pipeline.

## Reference

- Parser: [`scripts/parsers/rules.ts`](../../../scripts/parsers/rules.ts)
(look for `GRADING_CONTEXT` section handling)
- Loader: [`srv/lib/grading-context-loader.js`](../../../srv/lib/grading-context-loader.js)
- DB entity: [`db/schema.cds` — `TutorialGradingContext`](../../../db/schema.cds)
- Grader prompts:
- [`srv/lib/validate-answer-prompt.js`](../../../srv/lib/validate-answer-prompt.js)
(`buildUserMessage` — also used by `challenge-grade`)
- [`srv/lib/code-check-llm.js`](../../../srv/lib/code-check-llm.js) (`buildUserMessage`)
- Publish integration: [`srv/lib/validate-answer-spec-publish.js`](../../../srv/lib/validate-answer-spec-publish.js)
- Sibling docs:
- [Free-text grader](../architecture/free-text-grader.md) — `validate-answer` AI grading
- [Validation widget](../architecture/validation-widget.md) — the base widget this extends
Loading
Loading