use-cases/AbhishekSharma55: a weekly digest that is edited not regenerated, and a multi-document GDD editor - #53
Open
AbhishekSharma55 wants to merge 67 commits into
Conversation
…n-level diff The layer both builds sit on. Four things it guarantees: - no operation spent without a durable ledger line (budget.py) - no key in a log, fixture or commit (secrets.py redaction) - tests run with no key and no network (transport.py record/replay) - 'only the intended section changed' proved from bytes, not from the API's own summary (docdiff.py residue_map handles nested chunk ids)
…res, repro In-flight work committed out-of-band after the agent stalled mid-run. Includes the multi-document roster repro (two docs return one, both doc_primary).
Raw measurement output (md/html/diff/ledger/failed-attempts) committed as grading requires raw data on disk. In-flight when the agent was torn down.
The default 'core' tier applied 1 of 5 instructed steps, four weeks running, and reported success. Same instruction, same document, pro/deep: all thirteen invariants pass first attempt. Both bill one operation, so the fix costs latency, not money, and the cost comparison is unaffected.
Each document gets its own session because the shared multi-document session returns a one-entry roster with every document labelled doc_primary and a different document on each identical GET (bug B10). Cross-document consistency becomes this code's job, so it is verified rather than assumed.
…till counts SuperDocs answers 'add a new section' with <div data-chunk-id><h2>..</h2>..</div>. The old top-level-chunk walk saw no new section; the section was plainly there and my model was wrong. Driving off heading positions fixes that case and every sibling of it - heading as a chunk, inside a chunk, or nested three deep.
…shed weeks
Two measured failures fixed. 'Replace ALL bullets' was being read as append, and
a severity cell came back 'Medium / Low' - both old and new value. The wording
now says what the section must contain AFTER the edit and that one cell holds
one value. Verified on the week that failed: bullets and risk rows now exact.
The archive coverage invariant was demanding {1..N-1}, which a single rolled-back
week made unsatisfiable forever after. It now expects the weeks that were really
published - the check measures the present, not a past that never happened.
B10 verified end-to-end by repro/B10-multidoc-roster.sh on a fresh session: three documents opened, roster of one, every document_id the constant string doc_primary, and identical successive GETs returning different documents.
Two are built from documents the live API really returned - the table it destroyed and the section it wrapped in a div - so they test the actual failures rather than my idea of them.
… bullets Three measured fixes, each from a failure I could point at: - visible_text now unescapes: SuperDocs returns 'Week 6 .' and I was comparing markup to prose, so a correct status line failed. My bug. - the archive block restated every row in full; by week 6 it was most of a 5KB instruction and the model did the archive then dropped the four steps after it. Naming rows in order carries the same information in a third the size. - 'replace all bullets' read as append; 'delete the list, then write a new one' does not. One declared tolerance: a single trailing full stop, documented in verify.py rather than folded in silently.
Adds an over-delivery check with a repair: the model answered 'add one section' by also inventing a table nobody asked for. 'Change nothing else' has to include 'and add nothing else', so the scenario detects it and repairs it in a follow-up operation that is counted like any other.
I used the real key as one of the 'credential-shaped strings' in the redaction test. Caught by the pre-PR grep in PR-DRAFT.md. Fixture replaced with an obviously fake constant, the four affected commits rewritten, refs/original and the stash deleted, reflog expired, objects pruned. Verified zero occurrences across all git objects. Nothing was ever pushed. Logged in ASSUMPTIONS.md A7 with the rotation command, because a secret that touched disk should be rotated even when the exposure was local.
Measured, not preferred. Describing the target ('replace all bullets with
exactly these four') was read as append often enough to matter - week 7 came
back with the five old bullets kept and four new ones added. Three rewordings
reduced the rate without removing it.
So the model is no longer asked to author, re-type or reason about a fact; it
splices a known fragment into a formatted document, which is what a document
editor is for. It still writes every archive summary and rollup from what the
document says. Facts are transported, prose is generated - now literally.
Verified on the week that produced nine bullets: bullets, next, risk rows all
exact, alternating shading intact.
…epair The earlier fix for the dropped-steps bug was a bigger model. The better fix was a less ambiguous instruction: with the factual sections handed over as literal HTML, the DEFAULT tier gets every step right and does it 4.5x faster (43s vs 195s a week) on the exact document that had failed four times. So escalation moves to where it belongs - the repair path. Paying for the strongest model on all 52 weeks to cover a low single-digit failure rate is the wrong trade, and tier does not change what SuperDocs bills either way.
…dget On the default tier, regenerating week 8 returned a 1.2KB document holding one of the four sections. That is a real cost of regenerating and it belongs in the comparison - but only if arm B is also allowed to retry, exactly as arm A is. Same invariants, same escalation, same accounting, both arms.
…spend' My own bug, and it cost five real operations to find. build_transport treated live=None as 'decide for me', found the key in .env, and went live - so a command a reader would reasonably read as a dry run spent money. The expensive path now has to be asked for by name. Also adds --out so a replay writes elsewhere and can never overwrite the committed artefacts of the run it is replaying, and softens the replay claim in the READMEs to exactly what is verified.
Across weeks 4-7 the default tier failed first-attempt verification about 60% of the time on the same checks. The single-week test that convinced me to switch was not enough data, and the code comment and both docs now say so. The literal-HTML change stays and was the real fix for content correctness; it just did not remove the need for the stronger model. Tier does not change what SuperDocs bills, so this is latency, not money.
…ulti-doc surface A dead-code sweep found three helpers nothing calls any more. The multi-document session methods also go uncalled, but they stay - they are how bug B10 was established from this client rather than only from the curl script, and they are the surface build superdocsapp#2 should move back to if it is fixed. The comment says so.
…its keep
Removed what nothing reaches: docdiff._TAG_NAME and chunk_order (both left
behind by the heading-driven sections() rewrite, along with the parser's
order list that only chunk_order read), client.export_to, client.blank_document,
corpus._H3. Checked each against every caller, test, Makefile target and
scenario script first - _ChunkParser.handle_starttag and its siblings look
just as unreferenced and are called by HTMLParser itself.
The session/multi-document methods stayed, but the comment claiming they kept
bug B10 "reproducible from this client" was not true of code nothing called.
So `gdd probe-multidoc --live` now drives them: upload three documents, open
all three into one session, read the roster four times, and print what came
back. Zero billable operations, its own ledger, and a cap of zero so a call
that turned out to be billable would raise rather than spend.
Run three times just now, B10 reproduces exactly as written - roster of one,
document_id "doc_primary" for every document, and the document changes between
identical reads. The probe also tests B10's suggested workaround, which is how
it found that focus() rejects the durable id it was just opened with
("Document '...' is not open in this session") while focus("doc_primary")
succeeds and leaves the roster still arbitrary.
…und claim
The header said "three further mistakes" over four, the closing line said
"twelve bugs" over fourteen, and the summary table stopped at B7 while the file
ran to B14. Counts in a bug log are a claim like any other.
B10's suggested workaround - focus + one document at a time - was written
without being run. Running it: focus rejects the durable id the session was
just opened with (404 "not open in this session"), and focus("doc_primary")
returns 200 and leaves the roster arbitrary. There is no id that addresses a
specific open document, which makes the bug worse than the entry claimed and
the one-session-per-document design the only option rather than the cautious one.
Found by reading the live run's artefacts rather than its verdict. Week 2 came back with the archive label written as "1" instead of "Week 1". parse_existing drops rows it does not recognise - right for planning, silently wrong for verifying - so the coverage invariant could not see that row at all. The repair turn then added a second row with the correct label, and the week PASSED "archive covers every past week exactly once" while the reader's document carried both rows. The invariant did not catch the drift it is named after. Two changes, one root cause each: * verify.py gains a fourteenth invariant: every row in the archive body must be a row this system can read. That closes the class - a mangled label, a merged row, a stray sentence - not just the label that surfaced it. Regression test covers the mangled row alone and next to the correct one, which is the exact document the live run produced. * archive.py now says the first cell must include the word "Week". The column is headed "Week", which invites the bare number; the instruction previously quoted 'Week 1' and left the rest to inference. Same lesson as ASSUMPTIONS A9 - remove the ambiguity before reaching for a bigger model. 62 tests, still no key and no network.
…al drift verify_surgical defaults to allow_new=True, and untouched_report never overrode it. So "only the Concept art section changed" was satisfied by a turn that appended a SECOND <h2>Concept art</h2> and hung the images off that - every image present, section name right, document quietly carrying a duplicate heading. `gdd sections` shows it in the committed artefacts. The check said 18/18 over a document that had drifted, which is the exact failure this build is an argument against. Three changes: * untouched_report now decides per document whether additions are legal: a name in `allowed` that is not a section of the SNAPSHOT means this act may create it; if every allowed name already existed, a new chunk is unintended. ACT 3 and the image act name only existing sections, so they now forbid additions - and both still pass on the committed snapshots, so the check got stricter without being bent to fit. * untouched_report takes `created`, so a document that appears is checked against the list of documents the act was allowed to create rather than failing byte-identity against a snapshot that predates it. That is why act-2's report said "ok": false while the headline said 18/18. * the scenario meets the duplicate heading with a targeted repair and counts the operation, the same way ACT 2 already meets an invented table. Logged as bugs.md B15. The scenario re-runs live next; its committed artefacts and this code have to agree.
…ocuments B15 - "add these images in the 'Concept art' section only" appended a second section with the same heading. Every surface check an integrator would write says it worked. M5 and M6 are mine, and they belong together: both are checks that certified a document they should have failed. M5, the archive parser skipping the row it could not read, so a duplicate row passed "covers every past week exactly once". M6, the surgical check waving new chunks through, so "only the Concept art section changed" passed a turn that added a section. Counts corrected everywhere they appear: fourteen invariants, not thirteen; 62 tests, not 61.
…o CLIs M2 was "omitting --live spent five real operations", and the fix went into both CLIs by coercing live to a bool. build_transport itself kept the three-valued behaviour, so anyone importing the kit directly still had the trap. Now the network is only reachable when live is truthy: spending has to be asked for by name, everywhere. Makefile: demo diffs week 14 (a rollup week - standing sections turn over AND four archive rows fold into one, in one operation) instead of falling back between 30 and 3, and replay asks for the number of weeks that were actually recorded. Ending a demo command in a ReplayMiss teaches the wrong lesson about a transport that is behaving correctly.
Tightening untouched_report to forbid additions was right and too blunt: the image act legitimately creates chunks - two <img> elements inside 'Concept art' cannot exist in the before document - and the live re-run duly failed a correct edit. A check with a false positive gets switched off, which would have cost the true positive with it. verify_surgical now resolves the named sections in BOTH documents. section_named returns the FIRST section with a given name, so content under the existing heading is intended and content under a SECOND heading of the same name is not. That is exactly the line between the edit and bugs.md B15, and the regression test asserts both sides of it. 63 tests.
multi-document story plainly A10 still described attempt 1 running on the default tier; the code went back to pro/deep weeks ago and only the READMEs had been updated. A stale assumption is worse than no assumption - it reads as a claim about the running system. A11 logs what a second pair of eyes found, including the two checks of mine that certified documents they should have failed. The top-level README now separates what multi-document means here - four linked documents, a fifth created at runtime, per-document edits, byte-identical proof for the documents nobody named - from what is honestly absent: SuperDocs' own multi-document session, because of B10, with two zero-operation reproductions and the cost of the workaround stated before anyone asks.
The archive files one row per week: advancing to week N writes a row for week N-1 summarising what that week shipped. The instruction said "summarise the bullets currently under 'What changed'" - true when the archive step runs, false by the end of the same instruction, because step 3 replaces those bullets. Measured over seven live weeks, three of seven rows summarised the incoming week instead of the outgoing one. The row labelled "Week 5" described what week 6 shipped. Every structural invariant passed, because not one of them reads the sentence the model wrote - and the archive is the document's historical record. Found by reading a committed week-over-week diff by eye, then scoring each row's similarity against its own week versus the next one. 3/7. Fixed by handing the outgoing week's bullets and risks over literally, in quotes, and telling the model to summarise the quoted text and nothing else in the document. The model still writes the sentence - that is the part only a model can do - it just no longer has to find the source in a document being rewritten underneath it. Test asserts the instruction contains the outgoing week's content and not the incoming week's. Logged as M7. Arm A restarted from week 1 so the committed artefacts come from this code.
superdocs_kit promises, in its own docstring, that an operation is never spent without a durable ledger line. True everywhere except the human-gated path. chat_async submits the turn - which is when SuperDocs bills it - and returns a job envelope with no usage on it; the usage lands on the finished job and nothing read it. `gdd review` printed "0 operation(s)" while the account's monthly_used went up by one. Found by comparing the ledger against GET /v1/agents/whoami after the demo. review.settle() now hands the finished job to budget.record(), which already knew how to read result.usage.ops_charged. The demo now reports 1 operation, and a denied change is visibly charged - the thing a reviewer most needs to know before building a UI that proposes speculatively. Also: B13 did NOT reproduce on two later runs of the same instruction against the same document - both times the model targeted Dash and Overheat pulse correctly. Marked intermittent in the entry rather than left reading as deterministic. The argument for the gate is unchanged; a failure that fires occasionally and silently is the one you most need a human for.
…hecks that exist because the file was wrong before
…oftest part of the measurement named Every figure in the walkthrough's summary table now comes from something on disk: 20/20 and 6 operations from the scenario report, 64 tests and 0.04s from pytest, 15+8 bugs from the log, 18 archive rows at week 52 from the offline simulation. Where a number is generated - the digest's per-week cost and run length - the table points at MEASUREMENT.md and says that file wins. METHOD no longer hardcodes "20 weeks". It states the stopping rule and defers the week count to the generated Results section, because a method that asserts its own sample size is one restart away from being wrong. It also names what the measurement is NOT: no invariant reads the sentence the model wrote, which is how M7 hid for seven weeks. B13's "he will ask whether it always does that" answer added: it does not, twice not reproduced.
…and the archive story matches the code
Live arm-A digest run halted at 30/32 ops, 13 weeks (the run resumes). GDD graded scenario passing 20/20 in 6 ops under the stricter checks. Committed on stopping for the night so nothing sits uncommitted.
…have The old table put arm A's "52-week figure" as "22 ops (measured, not extrapolated)". Twenty-two operations is what eighteen weeks cost. Presenting it in a column headed "52-week" made a projection look like a measurement, in the one document whose whole job is to keep those apart. Both arms now get the same treatment - measured mean times 51, labelled a projection - and the section leads with why the projection is weak: arm A's fit has R2 0.005, arm B's 0.170 with a negative slope, so neither supports a trend and extrapolating along either line would be inventing precision. The headline also now carries arm B's validity, because it is the actual result: arm A 18/18 weeks valid at 1.22 ops/week, arm B 0/8 valid at 1.88. The cost gap is 1.44x measured like-for-like over the 7 overlapping weeks - real but modest. The quality gap is total, and a cost comparison between a valid artefact and an invalid one is the wrong comparison. Saying that is worth more than a bigger multiple would have been.
make replay now byte-compares every replayed week against the committed live run and exits non-zero if any week differs. On the committed run: 19 weeks byte-identical, 0 differing, 22 operations reproduced exactly. It also counts --weeks from the artefacts on disk instead of a hardcoded 20, which is what made the command end in a ReplayMiss on a fresh clone: the live run stopped at 19, the Makefile still asked for 20, and a correct transport refusing to invent week 20 looked like a broken demo. The README claimed the weaker thing - "does not prove a byte-identical re-run". For this run it does, and now the command says so out loud. The honest caveat about what happens on divergence stays, because that is a property of the transport rather than a result.
M7 - the archive row for week N-1 describing week N - could not become an invariant, because grading prose needs a rubric and a second model call. So it became a cheap offline check with its method printed above its result: take each row's distinctive content words and count how many belong to the week it names versus how many belong only to the week after it. A row about the right week scores zero on the second count. On the committed run: 0 of 18 rows describe the wrong week. Before the fix it was 3 of 7. Both numbers now come from the same command rather than from me. Its limit is in the command's own docstring and in the README: it catches the off-by-one that actually happened, not whether the summary is true. Wired into make demo so a reader meets it without being told to look. Also caught while verifying this: a similarity-ratio version of the check called week 18's row a misattribution. Reading the row proved it correct - the corpus repeats bullets across weeks, which is realistic and which string similarity cannot see. The committed check counts words that belong ONLY to the next week, which does not have that failure mode. Nearly reported a defect that was not there.
"One billable operation per week" was the design, and the measurement says 1.22 - sixteen weeks at one operation, two at three, none rolled back. The round number was the more impressive one and the measured number is the one with a table behind it, so every place that claimed the design now states the measurement and points at the per-week table. The digest README also now leads its cost section with what the control arm actually showed: 1.88 operations a week against 1.22, which is a real gap and a modest one - and a document that failed verification in every week it ran. The cost multiple is the weaker half of the result and is no longer presented as the strong half. Also: 0.05 s -> 0.04 s for the suite, which is what pytest prints.
…s for my own bugs
…ces fixed Arm B now runs the same 18 weeks as arm A, so the comparison no longer needs a caveat about unequal lengths: arm A 18 weeks 22 ops 1.22/week 18/18 weeks valid arm B 18 weeks 35 ops 1.94/week 0/18 weeks valid like for like over the 17 overlapping weeks: 21 vs 33 ops (1.57x) Two sentences the generator was producing had stopped being true when the data grew, which is the failure mode of prose written around a number: * "arm B's slope is even negative" - it was, at n=8; at n=18 it is +0.0052. The sentence now derives both slopes and their signs from the fit instead of describing a run I happened to have seen. * falsification criterion 3 was reported using arm A's validity alone. It now prints both arms and names the direction: the cheaper arm is the valid one, so the cost comparison understates the difference rather than flattering it.
Every place that described arm B as a bounded prefix or an extrapolation now says what it is: 18 weeks against arm A's 18, bounded by the same rule, because a control that stops earlier than the treatment is not a control. Any 52-week number for either arm is a projection and is labelled as one for both. The digest README's replay paragraph still claimed the weaker property after the Makefile started proving the stronger one. It now matches the top-level README and the command: 19 weeks byte-identical, 0 differing, 22 operations reproduced. What 18 weeks cannot rule out - a degradation that first appears at week 40 - is now stated in both READMEs and the walkthrough rather than left for a reader.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Abhishek Sharma
Two projects in
use-cases/AbhishekSharma55/, built on the REST API.weekly-digest/— one standingdocument revised in place week after week instead of a fresh file each week, so
the reader always opens the same familiar document. Each week is a
single billable operation covering the status line, three standing sections, and
an archive that folds old weeks into rolled-up rows so it stays navigable (18
rows at week 52 rather than 51, with 11 rollups). Fourteen invariants are
re-derived from the document's own bytes after every week; a week that fails is
repaired by decomposing it into two simpler turns, or rolled back so the document
is never left corrupt, and every repair operation is counted. A control arm
regenerates the same digest from the same inputs so the cost claim has a number
behind it — the method, including what would falsify it, is committed in
METHOD.mdbefore the results.Measured over 18 live weeks per arm: editing 1.22 operations/week, 18/18
weeks passing all fourteen invariants; regenerating 1.94/week, 0/18 passing.
The cost gap is real and modest (1.57x like-for-like over the 17 overlapping
weeks); the validity gap is the actual result, and
MEASUREMENT.mdsays sorather than quoting the flattering multiple.
make replayre-runs all 19 savedweeks from committed fixtures with no key and no network and byte-compares them
against the live run — 19 byte-identical, 0 differing.
gdd-editor/— a chat-driveneditor for long game design documents split across linked sub-documents
(mechanics, narrative, art bible), with image support for concept art. The
scenario it is built to pass — a reviewer asks for a new GDD section plus one
linked sub-document, then requests a revision to one mechanic, and only the
correct section changes across both documents — is executable as
gdd scenario --liveand passes 20/20 checks in 6 operations. The proof isre-derived from each document's bytes, not read from the API's change summary.
Both run on a shared client layer (
superdocs_kit/) that keeps a durable ledgerof every operation spent, records every live call as a fixture so the 64 tests
run with no key and no network in 0.04 s, and proves "nothing else changed" from
the document rather than from a success message.
bugs.mdhas fifteen SuperDocs entries found while building — the highest-valueone being that
changes_summaryreported"1 section edited, rest untouched"for a turn that had destroyed a table. Two of them are runnable for zero
operations:
repro/B10-multidoc-roster.shandgdd probe-multidoc --livebothreproduce a multi-document session returning a one-entry roster with every
document sharing
document_id: "doc_primary". Eight further entries are my ownmistakes and are in the same file, four of them cases where my own checking said
a document was fine when it was not.
Corpora are synthetic — the studio, the game and the people are fictional.
Built with Claude Code.