Skip to content

test(tutor-quality): run Tutor Quality scenarios with the real planner only (TutorQuality §5) - #157

Merged
ttbombadil merged 1 commit into
mainfrom
fix/tutor-quality-scenarios-real-planner
Oct 3, 2026
Merged

ttbombadil merged 1 commit into
mainfrom
fix/tutor-quality-scenarios-real-planner

Conversation

@ttbombadil

Copy link
Copy Markdown
Collaborator

Purpose

Conformance finding from the Tutor SSOT audit. TutorQuality SSOT §5: "The runner MUST use TutorService and, where Course Content is supplied, CurriculumTutorAdapter. It MUST fake only the external provider." The Stage-1 plan says the same ("with only the provider faked").

Finding

Since the runner was created (33b87cc9), it accepted a planning override (scenario.planning ?? new CurriculumTutorAdapter()). Five scenarios used it to script planning outcomes:

Scenario Faked
retains a repaired transition/blocked × exact/near duplicate question (4) planFollowup returning hand-written transition/blocked metadata
lets an application-owned TutorPlan replace provider question and metadata planInitial returning a hand-written plan
TQ-REG-001 › repairs repetition … coherent transition metadata the whole planner while Course Content was supplied, mutating state to an invented pwm-output Topic

So §3 items 6/7 and the TQ-REG-001 clause "result/state metadata must remain coherent through transition or blocked planning outcomes" were checked against invented outcomes, not the real planner.

Change (tests only)

  • The runner has no planning override. Course Content is always planned by the real adapter; without Course Content no planner exists.
  • The same outcomes now come from Course Content states:
    • transition: a strong answer completes LEARN mastery of a Topic that has no further LEARN question;
    • blocked: a DEEPEN answer with no further post-mastery question ends in content-exhausted;
    • plan authority: the real initial plan replaces the provider's question and forged metadata;
    • TQ-REG-001: a minimal PWM test Topic (activates on type-used: int, the only fact the extractor yields for the PWM sketch) is mastered by one strong answer. The real transition serves the turn under LEARN (R-PH-1), and the state commits DEEPEN on pwm-output; the near-repeat is still repaired. The old fake claimed learningPhase: DEEPEN on the served turn, which the real planner does not produce.
  • No product code changes. No Course Content outside test fixtures.

Evidence

  • RED: removing the override makes the three faking call sites fail to type-check. They are the only uses.
  • Mutation: when the repaired question is dropped for transition/blocked outcomes in TutorService.applyDialogFollowUp, all five rewritten scenarios fail; reverted.
  • npm run check 0 errors, npm run test:tutor-quality 407 passed, ESLint clean, pre-push (unit + Sonar) passed.

🤖 Generated with Claude Code

…r only (TutorQuality §5)

TutorQuality SSOT §5: the scenario runner "MUST use TutorService and, where
Course Content is supplied, CurriculumTutorAdapter. It MUST fake only the
external provider." The runner accepted a `planning` override, and five
scenarios used it to script transition, blocked, and plan outcomes; the
TQ-REG-001 progression scenario even faked the planner while Course Content
was supplied. Those scenarios therefore checked TutorService against invented
planning outcomes, not the real planner.

The runner no longer takes a planning override. The scenarios now reach the
same outcomes through Course Content states:

- transition: a strong answer completes LEARN mastery of a Topic with no
  further LEARN question;
- blocked: a DEEPEN answer with no further post-mastery question ends in
  content-exhausted;
- plan authority: the real initial plan replaces the provider question;
- TQ-REG-001 progression: a minimal PWM test Topic, mastered by one strong
  answer, transitions LEARN -> DEEPEN; the answer is served under LEARN
  (R-PH-1) and the state commits DEEPEN.

A mutation that drops the repaired question for transition/blocked outcomes
fails all five rewritten scenarios.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ttbombadil
ttbombadil merged commit ced308b into main Oct 3, 2026
5 checks passed
@ttbombadil
ttbombadil deleted the fix/tutor-quality-scenarios-real-planner branch October 3, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant