[xvcezrqpbxtpzblrt]

Problem and result

With --serve and no --conversation-cache-mib, the conversation-cache budget is 40% of the free RAM, capped at 8192 MiB. At long contexts one parked snapshot is larger than that, so every park is skipped and the next request reads its whole prompt again. The 0.1.50 Harness replays a compaction as a prefix of the conversation (282,474 tokens = 282,077 reused + 397 read in the run below); when the conversation cannot stay parked, that prefix is lost and the whole prompt is read again (the 331K-342K reads of the 0.1.48 log in #3998; that older log also re-rendered the top of the compaction prompt, 6,127 reused of 337,603, which needs #3584 and is not what this PR fixes).

Approach

auto_conversation_cache_mib() now takes max_context and never returns less than one snapshot of it (18 KiB per context token: the measured snapshots were 5,138,736,824 B / 282,108 tokens = 17.79 KiB and 6,318,446,860 B / 359,207 = 17.18 KiB with int8; q4_0 measured 11.3 KiB) while the RAM above the parking floor allows it. The 40% rule stays for shorter contexts; max_context 0 keeps the old rule alone. The physical-RAM admission is unchanged, so a snapshot the machine cannot hold is still refused, and a 4 GiB-free machine still turns the cache off.

Validation

Passed:

• cmake -S . -B build-local -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_BUILD_TESTS=ON -DSTRATA_BUILD_CONVERSATION_TESTS=ON and cmake --build build-local --target strata conversation_memory_test -j 8 at 14c455b5 on Windows 11, MSVC 14.44, CUDA 13.4, sm_86.
• conversation_memory_test: 40 checks passed (33 before, 7 new). New cases: 512K at 9 GiB free -> 6656 MiB, at 20 GiB -> 9216 MiB, at 40 GiB -> 9216 MiB; the reported machine's 12,160 MiB free -> 9216 MiB (the old rule gave 3,840 MiB, the value in [Engine] Compaction re-reads 331K-342K tokens five times (19.5-20.1 min each) because checkpoints match only from token 0 #3998's log); 32K context -> the 40% rule unchanged; 4 GiB free -> cache off as before. Output line: conversation_memory_test: 40 checks passed.
• Engine run, built from this branch (14c455b5), RTX 3080 Ti Laptop 16 GB / 64 GB RAM, IQ3_XXS 512K int8, --kv-grow --vram-reserve-mib 1024 --conversation-cache-min-free-mib 1024, no --conversation-cache-mib. Log %TEMP%\kv512-bench-auto-int8-grow.log:
  • conversation cache sized from the free RAM: 9216 MiB for up to 4 parked conversations
  • a 359,207-token prompt read cold in 331,216 ms (1,084.5 tok/s);
  • parked 359254 tokens in 1971.0 ms (6,318,446,860 B = 5.88 GiB);
  • restored 359254 tokens (live) in 906.4 ms;
  • the follow-up after a side conversation: prompt 359272 tokens = 359254 reused + 18 read in 1776 ms (1.9 s wall).
• Same prompt without room for the snapshot: prompt 359283 tokens = 0 reused + 359283 read in 1,105.8 s (%TEMP%\kv512-bench-int8-grow.log).
• Real /compact through the app's Harness, engine 0.1.50 with the explicit 16,384 MiB budget and 1,024 MiB floor (log C:\Users\123\AppData\Local\Strata\install\strata-swift-iq3_xxs.log, line 29,736): compaction request 282,474 tokens, reused 282,077 (99.86%), read 397, prefill 3,823 ms, summary 5,227 tokens in 137,393 ms. The post-summary turn: 11,810 tokens, 6,127 reused + 5,683 read, 8,485 ms. This shows the prefix reuse; the parked/restore path itself is shown by the 359K run above (park, side request, restore, follow-up).

Failed: none.

Note: the commit message quotes the earlier 0.1.50 run with the explicit 16 GiB budget (park 2.76 s, restore 0.94 s, follow-up 2.3 s); the validation above is the built 0.1.51 run at 14c455b5.

Not run:

• HIP and SYCL builds: no toolchain on this machine. The change is in backend-independent code (conversation_memory.hpp and one call in generate.cpp).
• macOS Metal: compiled and linked by the Metal build CI job on this head (metal-link), not executed on a Mac.

Limits

• The 18 KiB per token estimate is calibrated on int8 (17.18-17.79 KiB measured) and is a ceiling, not a reservation: the physical-RAM gate still refuses a snapshot the machine cannot hold. --kv fp16 is about twice int8's K/V and is not covered by the constant; at 512K that snapshot is about 16 GiB, so a 64 GB machine would still refuse it at the RAM gate. A follow-up can pass the KV mode's per-token size (q4_0 12 / int8 18 / fp16 36 KiB) from the caller.
• At 512K with --kv int8 on this 64 GB machine the snapshot is about 8 GiB and the free RAM beside the default 2560 MiB floor is about 8.2 GiB, so such a conversation still cannot park; that is a RAM limit, not a budget limit. --conversation-cache-min-free-mib moves that boundary (the runs above used 1024).
• The first turn after a summary still cannot reuse the retained tail, because the summary is new text and matching is prefix-only. In the 0.1.50 Harness that cost is 5,683 tokens / 8,485 ms in the run above; reusing it needs the segment-addressed cache ([Engine] suffix-reuse: conversation KV prefix mount, quant-safe splices, state checkpoint ladder #3584).

[xvcezrqpbxtpzblrt]

Superseded by #4606, which carries this change with the rest of the 0.2.7 branch. Closing so the review has one place.