Repository navigation
Python: fix(redis): skip malformed history entries instead of raising on read #9909
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,6 +9,7 @@ | |
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import logging | ||
| import warnings | ||
| from collections.abc import Awaitable, Sequence | ||
| from inspect import isawaitable | ||
|
|
@@ -23,6 +24,8 @@ | |
|
|
||
| from ._feature_usage import FeatureIndex | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| _T = TypeVar("_T") | ||
|
|
||
|
|
||
|
|
@@ -271,7 +274,18 @@ async def get_messages( | |
| redis_messages = cast("list[str]", await _redis_result(cast("Any", self._redis_client).lrange(key, 0, -1))) | ||
| messages: list[Message] = [] | ||
| for serialized in redis_messages: | ||
| messages.append(Message.from_dict(self._deserialize_json(serialized))) | ||
| try: | ||
| payload = self._deserialize_json(serialized) | ||
| if not isinstance(payload, dict): | ||
| raise ValueError(f"Expected a JSON object, got {type(payload).__name__}") | ||
| messages.append(Message.from_dict(payload)) | ||
| except (ValueError, TypeError, AttributeError, json.JSONDecodeError) as exc: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Skipping one record here can break function-call/result pairing: dropping a malformed tool-result record leaves its assistant call without a terminal result, while dropping a malformed call record can leave an orphaned tool result. Those lossy histories can survive serialization and make the next provider request invalid. Could we recover or reject complete tool exchanges atomically rather than individual records, and add corrupted call/result regressions consistent with the function-calling loop contract? |
||
| # Log the exception class, not its message: ``Message.from_dict`` | ||
| # embeds the payload's ``type`` value in mismatch errors, so logging | ||
| # ``str(exc)`` could copy untrusted history data into application logs. | ||
| logger.warning( | ||
| "Skipping malformed Redis history entry (%s)", type(exc).__name__ | ||
| ) | ||
| return messages | ||
|
|
||
| async def save_messages( | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lisry-1 Skipping a malformed middle record can duplicate the valid history on the next save — For stored history
[A, malformed, C],get_messages()now returns[A, C]. If the caller then saves the full transcript[A, B, C, D], the current contiguous-overlap check can append all four messages and persist[A, C, A, B, C, D]. Please make save-time deduplication account for skipped records and add this replay regression.