test(tutor-quality): run Tutor Quality scenarios with the real planner only (TutorQuality §5) - #157
Merged
Conversation
…r only (TutorQuality §5) TutorQuality SSOT §5: the scenario runner "MUST use TutorService and, where Course Content is supplied, CurriculumTutorAdapter. It MUST fake only the external provider." The runner accepted a `planning` override, and five scenarios used it to script transition, blocked, and plan outcomes; the TQ-REG-001 progression scenario even faked the planner while Course Content was supplied. Those scenarios therefore checked TutorService against invented planning outcomes, not the real planner. The runner no longer takes a planning override. The scenarios now reach the same outcomes through Course Content states: - transition: a strong answer completes LEARN mastery of a Topic with no further LEARN question; - blocked: a DEEPEN answer with no further post-mastery question ends in content-exhausted; - plan authority: the real initial plan replaces the provider question; - TQ-REG-001 progression: a minimal PWM test Topic, mastered by one strong answer, transitions LEARN -> DEEPEN; the answer is served under LEARN (R-PH-1) and the state commits DEEPEN. A mutation that drops the repaired question for transition/blocked outcomes fails all five rewritten scenarios. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Conformance finding from the Tutor SSOT audit. TutorQuality SSOT §5: "The runner MUST use
TutorServiceand, where Course Content is supplied,CurriculumTutorAdapter. It MUST fake only the external provider." The Stage-1 plan says the same ("with only the provider faked").Finding
Since the runner was created (
33b87cc9), it accepted aplanningoverride (scenario.planning ?? new CurriculumTutorAdapter()). Five scenarios used it to script planning outcomes:retains a repaired transition/blocked × exact/near duplicate question(4)planFollowupreturning hand-written transition/blocked metadatalets an application-owned TutorPlan replace provider question and metadataplanInitialreturning a hand-written planTQ-REG-001 › repairs repetition … coherent transition metadatapwm-outputTopicSo §3 items 6/7 and the TQ-REG-001 clause "result/state metadata must remain coherent through transition or blocked planning outcomes" were checked against invented outcomes, not the real planner.
Change (tests only)
content-exhausted;type-used: int, the only fact the extractor yields for the PWM sketch) is mastered by one strong answer. The real transition serves the turn underLEARN(R-PH-1), and the state commitsDEEPENonpwm-output; the near-repeat is still repaired. The old fake claimedlearningPhase: DEEPENon the served turn, which the real planner does not produce.Evidence
TutorService.applyDialogFollowUp, all five rewritten scenarios fail; reverted.npm run check0 errors,npm run test:tutor-quality407 passed, ESLint clean, pre-push (unit + Sonar) passed.🤖 Generated with Claude Code