[MannXo]

Summary

Adds a streaming_hints plugin. It watches the model's reasoning as it is generated and injects a hint from a memory bank when the model hesitates ("wait", "not sure", "let me check") or moves to a new step. This is the inference half of #206. The RL-trained retriever is left for a follow-up, and the README in the plugin folder says so.

Changes

• optillm/plugins/streaming_hints/
  • memory_bank.py holds Hint and HintMemoryBank. Retrieval is TF-IDF cosine over hint text plus keywords (keywords count double), in pure Python, so there is no new dependency. search(query, k, exclude) is the interface an embedding-backed bank could implement later.
  • retriever.py holds StreamingHintRetriever. It consumes reasoning incrementally, sentence by sentence, across chunk boundaries. A sentence is an injection point when min_gap_chars have passed since the last hint and it carries an uncertainty signal or ends a paragraph. At that point it queries the bank with the problem plus the last 600 characters of reasoning and returns the best unused hint above hint_threshold.
  • streaming_hints.py is the driver. It generates in segments of segment_tokens, feeds each segment to the retriever, cuts the segment at the injection point, appends <hint>...</hint> and continues from the cut. It stops when the model finishes, at max_segments, or when max_tokens is spent across segments.
  • hints.json is a bundled bank of 20 general hints (math, coding, reasoning). hints_path points to a custom one.
  • README.md covers how it works, config, the custom hint format and future work.
• optillm/plugins/streaming_hints_plugin.py, the SLUG = "streaming_hints" entry point.
• pyproject.toml ships hints.json as package data (checked in a built wheel).
• README plugin table row, tests/test_plugins.py lists, and the new test file added to the unit-test step in test.yml.

Usage follows the issue:

client.chat.completions.create(
    model="streaming_hints-gpt-4o-mini",
    messages=[...],
    extra_body={"hint_threshold": 0.25, "show_hints": True},
)

By default the response is the model text only. show_hints: true returns it with the <hint> tags.

Testing

tests/test_streaming_hints.py adds 11 tests with a fake client. They cover ranking in the bundled bank, exclusion, injection only at an uncertainty signal after the gap, a sentence split across two chunks, the threshold and max_hints, cutting and continuing (the discarded tail never reaches the next prompt), show_hints, continuation after a length cut-off, the max_tokens budget across segments, and the plugin entry point.

The unit-test step from test.yml, run locally on Python 3.12:

tests/test_ci_quick.py              All CI quick tests completed
tests/test_plugins.py               22 passed
tests/test_proxy_plugin.py          9 passed
tests/test_streaming_hints.py       11 passed
tests/test_approaches.py            All tests completed

test_reasoning_simple.py and test_batching.py failed locally on main as well, with pyarrow has no attribute PyExtensionType from an old datasets that my resolver picked. CI on main is green, so this is my environment.

End to end through the server with local inference (OPTILLM_API_KEY=optillm, Qwen/Qwen2.5-0.5B-Instruct, segment_tokens=100, min_gap_chars=80, max_tokens=300, show_hints=true), on "A train travels 120 km in 1.5 hours. What is its speed in meters per second?":

To find the speed of a train in meters per second, we need to follow these steps:


<hint>Write down the units of every quantity and check that both sides of each equation have the same units.</hint>
First, let's convert kilometers (km) into meters (m). One kilometer is equivalent to 1000 meters.
...
INFO - Segment 1: injected hint math-units (score 0.357)
INFO - Streaming hints done: 1 hints injected, 300 completion tokens

I have no benchmark numbers. That needs an API budget and is listed as future work.

Notes for reviewers

• Plugin instead of a core approach. The issue suggests optillm/streaming_hints/ like autothink/. autothink is a local-decoding approach, while this one only needs the OpenAI client, so a plugin (with its package under plugins/ like deepthink/) works with every provider and needs no change to server.py. Happy to move it if you prefer the core layout.
• Segments instead of a token stream. The local inference server raises on stream=True, so the driver generates in segments, and up to one segment of text after an injection point is generated and then discarded. Feeding token deltas from a real stream and cancelling it would avoid that for providers that stream. The retriever already takes arbitrary deltas, so only the driver would change.
• Truncation detection. The local server always reports finish_reason: "stop", so a segment that used its whole token budget is also treated as cut off.
• Continuation goes through the user turn. The reasoning so far is passed inside <reasoning_so_far> and the model is asked to continue. That works on every provider, but where a segment was cut by the token limit mid-sentence, a small model sometimes starts a new sentence instead of finishing it (visible as "120,Next" in the live run). Assistant-prefill continuation would be smoother on providers that support it.
• Open questions for the RL follow-up. What should the reward be (per-problem accuracy with versus without hints, or a learned value of each hint)? And should the policy pick the timing, the hint, or both? Hint.uses is counted as a first usage statistic.

Fixes #206

[CLAassistant]

All committers have signed the CLA.