docs: design for image input — paste a screenshot, ask about it by ndemianc · Pull Request #89 · levelcodeai/levelcode

ndemianc · 2026-08-18T03:08:57Z

Design only — no feature code. Seven slices, I1–I7.

The target is what Cursor and the Claude Code console already do: screenshot, ⌘V into the composer, ask why it looks wrong. That part is easy. This documents what breaks in our code when the first image goes through, and the cost model that decides how the bytes should be shaped before they leave the webview.

Four things that break quietly

Verified against develop, not recalled. Three of the four fail silently, which is the reason to write them down before writing feature code.

where	what	why it matters
`extension.js:1752`	`content` is a string — `blocks.join('\n\n') + text`	no shape an image can take; this is the structural change
`providers/translate.js:99`	silently drops unrecognised blocks	on OpenAI-compatible providers the image vanishes between composer and wire. No error, no log. The model answers confidently about text it never saw
`agentMemory.js:40`	estimator is `JSON.stringify(m).length / 4`	a 1 MB screenshot books ~333,000 phantom tokens. It drives `findCompactionCut`, so one paste evicts real history
`providers/catalog.js`	`vision: true` is already there, nothing reads it	there is `supportsToolsForModel` and no vision equivalent

The third would have shipped as a mystery report: "long conversations forget things after I paste a screenshot."

The numbers changed a decision

I checked the vision API rather than trusting my prior, and the prior was wrong — cost is not w × h / 750, it is ⌈w/28⌉ × ⌈h/28⌉ over 28px patches, capped per tier (2576px/4784 tokens on 4.7+, 1568/1568 below).

That changed the design. Token cost is already capped server-side. Sending a 12 MB PNG does not buy more than 4784 tokens of fidelity — it buys latency. So client-side downscaling is not a defence against a token blowup, as I would otherwise have written it; it is a deliberate fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire.

I reimplemented the resize rule and checked it against every worked example in the docs: token count matches on all twelve, dimensions on eleven — one standard-tier row lands a single pixel off, and the doc says so rather than claiming exactness.

Slices

I1 is worth merging regardless of images — the silent block drop is a live bug today.


I1	Fail loudly at the translator	ships alone; predates images
I2	`supportsVisionForModel`	reads the flag already in the catalog
I3	The normalizer, pure and tested	geometry only, no canvas
I4	Store and account	content-addressed bytes, estimator learns images
I5	Paste	the end-to-end path on one input
I6	Drop, picker, thumbnail	remaining inputs + rendering
I7	The refusals	each names the actual constraint

Every exit criterion is written to be bypass-verifiable.

Deferred, with reasons

Files API upload (Anthropic-direct only; wins on repeat turns, not the first), image output (Claude does not generate images), PDF blocks, coordinates, client-side OCR.

Also

A readable version of this plan, with a live visual-token calculator built from the verified formula: https://claude.ai/code/artifact/a31690f7-79f1-4f6e-9902-3c32fd7f91de

The interaction being copied — ⌘V a screenshot into the composer, no upload step — is the easy part. This documents the four things in our code that break quietly when the first image goes through, and the cost model that decides how bytes should be shaped before they leave the webview. The four, each verified against develop rather than recalled: 1. conversation carries content as a STRING (extension.js:1752). There is no shape an image can take. This is the structural change; everything else follows. 2. translate.js SILENTLY DROPS any block it doesn't recognise. On OpenAI-compatible providers an attached image would vanish between composer and wire, and the model would answer confidently about text it never saw — no error, no log line. A user would reasonably conclude the model hallucinates. Worth fixing on its own merits, before images, which is why it's slice I1. 3. `vision: true` is ALREADY in the catalog, per model, and nothing reads it. There's a supportsToolsForModel and no supportsVisionForModel. Half the gate exists. 4. estimateMsgTokens is JSON.stringify(m).length / 4 — sound for text, catastrophic for base64: a 1MB screenshot books ~333,000 phantom tokens, more than most context windows. The same estimate drives findCompactionCut, so pasting one screenshot would evict real conversation history. This is the bug that would have shipped as "long conversations forget things after I paste an image". ON THE NUMBERS. I checked the vision API rather than trusting my prior, and the prior was wrong: cost is not w×h/750, it is ⌈w/28⌉ × ⌈h/28⌉ visual tokens over 28px patches, with a per-tier cap (2576px/4784 tokens on 4.7+, 1568/1568 below). I reimplemented the resize rule and reproduced the documented figures exactly — 1092² → 1521, 1000² → 1296, 1920×1080 → 2691 — so the cost table in §1 is arithmetic, not estimate. That verification changed a decision. Token cost is ALREADY capped server-side, so client-side downscaling is not a defence against a token blowup — it is a deliberate fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire. The doc says so rather than implying downscaling is load-bearing for cost safety. Also recorded: writing §D3 I ran a 1160×480 capture through `sips -Z 1568` and it GREW, 40KB → 89KB, because the tool scaled it up to meet the cap. Never upscale — the rule is min(1, cap/longEdge), and a factor of 1 means pass the original bytes through untouched, which also avoids stacking compression artifacts on screenshots of text. Seven slices, I1–I7, each independently shippable with bypass-verifiable exit criteria. Deferred with reasons: Files API upload (Anthropic-only, wins on repeat turns), PDFs, coordinates, client-side OCR.

Building a live cost calculator from the doc's numbers caught two things the prose had glossed. The resize rule is not an iterative shrink. My first implementation stepped the scale down by 1% until the patch grid fit, which gets the TOKEN COUNT right every time but misreports the sent dimensions — 1447x814 where the docs say 1456x819. Replaced with a binary search for the largest scale whose grid fits the cap, which is what the rule actually is. Checked against every worked example in the vision docs: token count matches on all twelve, dimensions on eleven. The twelfth is one standard-tier row a single pixel wide of the reference (1270 vs 1269, same 1564 tokens) — a rounding convention I could not derive from six data points, and the doc now says so rather than claiming the rule was reproduced "exactly". Two rows in the cost table were computed with the stepping version and are corrected: the macOS retina grab is 2380x1546 (not 2377x1544) and the 12 MP photo is 2212x1659 at 4740 tokens (not 2193x1645 / 4661). Both were mine, neither came from the docs. None of this moves a decision — the 1568 default and the "server already caps the cost" argument rest on the 4K row, which was right. But a plan whose own arithmetic disagrees with its live calculator is worse than one with no calculator.

Copilot

Pull request overview

Adds a design document outlining how LevelCode’s AI chat should support image input (e.g., pasting screenshots), focusing on the current codebase constraints, a verified vision token cost model, and a staged implementation plan (I1–I7) to avoid silent failures and incorrect token accounting.

Changes:

Introduces docs/IMAGES.md describing the end-to-end image pipeline (webview normalization → host storage → provider request shaping).
Documents four current “quiet breakages” (string-only content, silent block drops, unused vision gating, and naive token estimation) and their implications.
Specifies concrete design decisions (block widening, image-first ordering, normalization rules, content-addressed storage, and capability gating) plus slice-by-slice exit criteria.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ndemianc · 2026-08-23T21:58:46Z

+
+This is the most dangerous thing in the list. On any OpenAI-compatible provider — which is most of them through the gateway — an attached image would **vanish between the composer and the wire**, and the model would answer confidently about text it never saw. No error, no warning, no log line. A user would reasonably conclude the model is hallucinating.
+
+**The vision capability is already modelled, and nothing reads it.** `providers/catalog.js` carries `vision: true` per model and has done since the multi-provider work:


Correct — describeCaps reads c.vision (catalog.js:162) to render "vision" in the model picker's detail line, and getModelChoices passes it through at :192. "Nothing reads it" was wrong; nothing gates on it is the actual gap, and that is what the doc says now.

Fixed here, and the gate itself now exists — supportsVisionForModel landed in the implementation branch.

ndemianc · 2026-08-23T21:58:48Z

+}
+```
+
+Sound for text. For a base64 image it charges roughly **one third of the byte count as tokens** — a 1 MB screenshot books ~333,000 phantom tokens, which is larger than most context windows. `findCompactionCut` would fire on the first screenshot and evict real conversation history to make room for an image that actually costs ~4,800. This is not a rounding error; it is the meter reading the wrong quantity entirely.


You are right, and this was the one worth catching. I asserted a causal chain without opening the function I named.

findCompactionCut cuts on message count and goal boundaries — len - keepRecent, then isGoalBoundary — and never reads a token number. compactAgentMemory calls estimateMsgTokens only for beforeMsgTokens/afterMsgTokens, which land in a dbg line and the return value. The function's own comment says it: used only for the UI meter.

So the live consequence is narrower than I wrote: the meter reads wildly high the moment an image is attached, telling someone to start a new chat when they are nowhere near full. Still worth fixing — and it becomes a correctness bug rather than a display one the day any auto-compaction policy keys off that number. The doc says that now.

I also corrected the same overstatement where it had already reached shipped code: the comment on imageBlockTokens in imageCost.js and its test both repeated it. A false causal claim in the source outlives one in a design note, because the next person reads the comment and not this thread.

ndemianc · 2026-08-23T21:58:50Z

+| High-resolution | Claude 4.7 and later | 2576 px | 4784 |
+| Standard | everything else | 1568 px | 1568 |
+
+Images above either limit are **downscaled server-side, preserving aspect ratio**. I reimplemented the rule and checked it against every worked example in the documentation: the **token count matches on all twelve** (1092² → 1521, 1000² → 1296, 1920×1080 → 2691, 3840×2160 → 2576×1449 at 4784), and the sent dimensions match on eleven — one standard-tier row lands a single pixel off (1270 vs 1269 wide, same 1564 tokens), a rounding convention I could not derive from six data points. Cost is exact; geometry is exact to a pixel:


Fair — the paragraph describes a one-pixel difference and then says "exact to a pixel", which reads as exact equality. Now: Cost is exact; geometry is correct to within a pixel.

Three review comments on #89, all correct. THE ONE THAT MATTERS. The doc claimed a bad token estimate would make findCompactionCut evict real conversation history on the first pasted screenshot. That is false. findCompactionCut cuts on message count and goal boundaries and never reads a token number; compactAgentMemory uses estimateMsgTokens only for its before/after report. The function's own comment says so — "used only for the UI meter". I asserted a causal chain without opening the function it named. The live consequence is narrower and the doc now says so: the context meter reads wildly high the moment an image is attached, telling someone to start a new chat when they are nowhere near full. Still worth fixing, and it becomes a correctness bug rather than a display one the day anything automatic keys off that number. Also corrected: - "nothing reads it" about the vision flag — describeCaps DOES read it to render "vision" in the model picker. Nothing GATES on it, which is the actual gap. - "geometry is exact to a pixel" read as exact equality when the paragraph had just described a one-pixel difference. Now "correct to within a pixel".

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

docs/IMAGES.md:239

In the Budget table, the last row is labeled “the compaction bug” and the following sentence repeats the earlier (now corrected) claim that pasting a screenshot makes long conversations forget history. Earlier in §0 you explicitly note compaction does not currently key off estimateMsgTokens (agentMemory.js:37-41; extension.js:1542-1591), so this section should describe the token meter / UX misread (and the future risk), not compaction today.

| Token-meter error | ~333,000 phantom tokens per MB | 0 | the compaction bug |

The last row is the one that would have shipped as a mystery bug report: *"long conversations forget things after I paste a screenshot."*

The doc still described the plan, not the build. It named a 1568 cap eleven times, said PNG-in-PNG-out, claimed images were "deleted with the session", and had no mention of the core patch, the media sweep, multi-image labelling or the in-flight race. Every place the implementation diverged now says so IN PLACE, with the reason: D3 cap 1568 -> 2000, and PNG -> WebP when resizing. Both overturned by measuring Claude Code's own transcripts rather than arguing from the docs: 24 images, every re-encoded one exactly 2000px, originals passed through untouched in their source format. D4 "deleted with the session" was never true — sessions are append-only and trash() only writes a lifecycle event. Documents the sweep and why it needs an age floor. D7 the gate is provider AND model, and it is re-checked at send because a model can be switched between attaching and pressing enter. D8 drag-and-drop needed a core patch. Records both traps: the rewritten mainThreadWebview- viewType that made the patch inert, and that Shift is a different code path — so "drag and drop + shift works" was never evidence the patch worked. D10 NEW — several images are introduced by name, per the vision guidance. The cost table and the budget are recomputed at the shipped cap, and the budget now says the token saving is SMALLER than the plan promised (1.6x, not 2.7x) and why that is the right trade: the server caps at 4784 either way, so the extra 432px buys legibility for tokens already being spent. The bytes are where the win actually is, and they moved further than planned because resizing re-encodes to WebP. Also recorded: the S3-versus-local decision and the reasoning behind rejecting S3, since that question will be asked again. And a new §8, "What this document got wrong" — eight rows, kept deliberately. A design note that records where it was wrong is worth more than one quietly rewritten to match the code; half of those entries were caught by reviewers rather than by me, and that is the useful part.

…rder D9 before D10

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

docs/IMAGES.md:237

This repeats the claim that I1–I7 are “all shipped” in #90, but #90 is not a shipped/merged baseline and (per its own description) does not cover all slices. Reword this section to avoid presenting slice status as completed/shipped in the design doc.

**Status: I1–I7 all shipped** in [#90](https://github.com/levelcodeai/levelcode/pull/90), plus the core patch D8 turned out to need. What follows is the plan as written; where the implementation diverged, the decision above it says so.

docs/IMAGES.md:130

This section is written as a post-implementation correction (“Corrected during implementation”, “Format policy as shipped”), but this PR is explicitly design-only and the repository baseline does not contain an image pipeline yet. Consider rephrasing this as a design decision backed by external evidence (Claude Code transcripts) rather than implying the behavior already shipped here.

**⚠️ Corrected during implementation.** Claude Code's own transcripts are on disk, so rather than reason about the right cap I read what Anthropic's client actually ships: **24 images, and every re-encoded one is exactly 2000 px on the long edge**. That is the threshold the vision docs name for staying clear of the stricter per-image dimension limit above 20 images per request — the largest size that is never unsafe. It also sits above both model tiers' own caps, so the server does the final downscale and we never discard fidelity it would have kept. The 1568 argument traded legibility for a saving the server was going to make anyway.

The same transcripts confirmed the pass-through rule above, which had been derived rather than observed: images under the cap go through **untouched, in their original format** (their PNGs stay PNG, their JPEGs stay JPEG), and only oversize ones are resized and re-encoded — to **WebP**, not PNG. That is the second correction: the plan said PNG-in-PNG-out, and WebP at q0.92 is materially smaller for the same screenshot with no visible loss (a 4K PNG grab: 764 KB → 115 KB).

Format policy as shipped: pass through PNG / JPEG / GIF / WebP untouched under the cap; re-encode to WebP only when resizing. Never JPEG a screenshot of text.

docs/IMAGES.md:150

This paragraph asserts a specific storage implementation (“bounded by an explicit sweep (imageStore.sweep → sessions.sweepMedia)”) as if it already exists, but there’s no corresponding code in this branch (those symbols only appear in this doc). In a design-only document, this should be phrased as a requirement/proposal (what should be built), not as a correction against shipped behavior.

**⚠️ Corrected during implementation.** An earlier version of this section said images live beside the session "so they are deleted with it". That was never true: sessions are append-only and `trash()` only writes a lifecycle event, so nothing removed a stored image, ever. Storage is **project-scoped**, and it is bounded by an explicit sweep (`imageStore.sweep` → `sessions.sweepMedia`) that runs on session seal and deletes media no session refers to any more.

The sweep has an **age floor**, which is not incidental: a normal (non-agent) chat writes media whose refs are never persisted to any session file, so an unreferenced-means-delete rule would delete files belonging to a conversation that is still open. A week is long past the point a conversation is live, and it bounds the growth — which was the actual problem.

ndemianc added 2 commits August 17, 2026 20:35

Copilot AI lite review requested due to automatic review settings August 18, 2026 03:08

Copilot started reviewing on behalf of ndemianc August 18, 2026 03:09 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

ndemianc mentioned this pull request Aug 23, 2026

feat(images): paste a screenshot, ask about it #90

Merged

ndemianc requested a lite review from Copilot August 26, 2026 19:31

Copilot started reviewing on behalf of ndemianc August 26, 2026 19:32 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

ndemianc added 2 commits August 26, 2026 20:12

docs(images): header said 'design, not built' on a shipped feature; o…

e0669e2

…rder D9 before D10

ndemianc requested a lite review from Copilot August 27, 2026 00:19

Copilot started reviewing on behalf of ndemianc August 27, 2026 00:19 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Comment thread docs/IMAGES.md

ndemianc merged commit 3172bed into develop Aug 27, 2026
2 checks passed

ndemianc deleted the feat/image-input branch August 27, 2026 01:31

Navigation Menu

Uh oh!

docs: design for image input — paste a screenshot, ask about it - #89

docs: design for image input — paste a screenshot, ask about it#89
ndemianc merged 5 commits into
developfrom
feat/image-input

ndemianc commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ndemianc Aug 23, 2026

Uh oh!

ndemianc Aug 23, 2026

Uh oh!

ndemianc Aug 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		This is the most dangerous thing in the list. On any OpenAI-compatible provider — which is most of them through the gateway — an attached image would vanish between the composer and the wire, and the model would answer confidently about text it never saw. No error, no warning, no log line. A user would reasonably conclude the model is hallucinating.

		The vision capability is already modelled, and nothing reads it. `providers/catalog.js` carries `vision: true` per model and has done since the multi-provider work:

Uh oh!

Conversation

ndemianc commented Aug 18, 2026

Four things that break quietly

The numbers changed a decision

Slices

Deferred, with reasons

Also

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ndemianc Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

ndemianc Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

ndemianc Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants