Warn when changing model/options mid-session breaks the prompt cache#323594
Open
bhavyaus wants to merge 1 commit into
Open
Warn when changing model/options mid-session breaks the prompt cache#323594bhavyaus wants to merge 1 commit into
bhavyaus wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds infrastructure and UI affordances in the workbench chat experience to warn users when switching models or model options mid-session will invalidate the warm prompt cache (potentially increasing cost and latency).
Changes:
- Introduces
IChatCacheBreakServiceto track cache-relevant request configuration per chat session and to compute whether a candidate selection would break cache. - Records the baseline configuration from the shared chat request send path (skipping system-initiated/background turns).
- Updates the model picker and model-options picker headers to conditionally show a “cache-break” cost hint, and adds unit tests for cache-break detection and tool-key normalization.
Show a summary per file
| File | Description |
|---|---|
| src/vs/workbench/contrib/chat/common/chatCacheBreakService.ts | New singleton service to record per-session cache baselines and compare candidate selections by dimension. |
| src/vs/workbench/contrib/chat/test/common/chatCacheBreakService.test.ts | New unit tests covering baseline behavior, per-dimension detection, clearSession, and tool-key ordering. |
| src/vs/workbench/contrib/chat/common/chatService/chatServiceImpl.ts | Records the cache-relevant configuration on non-system-initiated sends. |
| src/vs/workbench/contrib/chat/browser/widget/input/chatInputPart.ts | Plumbs a selectionBreaksCache delegate from the input to the picker surfaces via IChatCacheBreakService. |
| src/vs/workbench/contrib/chat/browser/widget/input/chatModelPicker.ts | Shows conditional cache-break hints in the model picker and options picker headers. |
| src/vs/workbench/contrib/chat/browser/widget/input/modelPickerActionItem.ts | Extends the picker delegate interface with an optional selectionBreaksCache callback. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Low
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
from
e2d9b2d to
dd23eeb
Compare
Contributor
|
Base:
|
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
2 times, most recently
from
47b6179 to
c530a5b
Compare
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
2 times, most recently
from
283b30c to
2c22cb0
Compare
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
2 times, most recently
from
a38d735 to
16c2110
Compare
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
2 times, most recently
from
1b3d3e3 to
340ff43
Compare
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
from
340ff43 to
ff7dbbf
Compare
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
3 times, most recently
from
5656080 to
729b8e0
Compare
…cache The model and options pickers surface a cache-break cost hint when the chat session's prompt cache is warm — switching the model or changing options mid-session resets that warm cache and may increase cost. The hint includes a "Learn more" link (rendered via the shared Link widget through a new optional headerLink on the action list header banner) pointing at the Copilot docs on optimizing AI usage. Warmth is derived directly from the session's request history at the picker rather than tracked in a parallel in-memory map: getRequests() length for the default chat, and session status leaving Untitled for agent-host sessions. Both warm as soon as the first request is sent, so the signal is drift-free and consistent across surfaces — it covers reloaded/restored sessions, every request path, and the agents window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bhavyaus
force-pushed
the
bhavyaus/cache-break-warning
branch
from
729b8e0 to
7832a7c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First pass at #319750
Surfaces a cache-break cost hint in the chat model/options pickers so users know when switching their selection mid-session would reset the warm prompt cache (costing more and running slower).
The prompt cache is keyed on the request prefix, so changing the model, reasoning effort, context size, mode, or enabled tool set between turns of the same session invalidates the cache the previous turn warmed up. Previously there was no signal to the user that a change carried this cost.
Setting
The hint is gated by the new
chat.cacheBreakHint.enabledsetting (application-scoped, taggedexperimental). It is off by default — users must opt in to see the hint. Users can also dismiss the hint inline via the "×" on the banner, which suppresses it persistently.