Skip to content

Python: fix(redis): skip malformed history entries instead of raising on read - #9909

Open
lisry-1 wants to merge 3 commits into
microsoft:mainfrom
lisry-1:fix/redis-history-skip-malformed-entries
Open

lisry-1 wants to merge 3 commits into
microsoft:mainfrom
lisry-1:fix/redis-history-skip-malformed-entries

Conversation

@lisry-1

@lisry-1 lisry-1 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

RedisHistoryProvider.get_messages deserialized every Redis list entry without per-entry error handling. A single empty string or corrupt JSON entry would raise JSONDecodeError / ValueError and abort the entire read, even when the remaining entries were valid.

CosmosHistoryProvider.get_messages already handles this case by skipping malformed entries with a warning (see _history_provider.py in azure-cosmos).

Change

Wrap each entry in try/except (ValueError, json.JSONDecodeError), log a warning, and continue, matching the Cosmos behavior. Added a regression test that feeds ["", "not-json", <valid>] and asserts the valid message is still returned.

get_messages deserialized every Redis list entry without per-entry error
handling, so a single empty string or corrupt JSON entry would raise
JSONDecodeError / ValueError and abort the whole read. CosmosHistoryProvider
already skips such entries with a warning.

Wrap each entry in try/except (ValueError, json.JSONDecodeError), log a
warning, and continue, matching the Cosmos behavior.
Copilot AI balanced review requested due to automatic review settings October 31, 2026 07:19
@lisry-1
lisry-1 temporarily deployed to github-app-auth October 31, 2026 07:19 — with GitHub Actions Inactive
@lisry-1
lisry-1 temporarily deployed to github-app-auth October 31, 2026 07:20 — with GitHub Actions Inactive
@agent-framework-automation agent-framework-automation Bot added the python Usage: [Issues, PRs], Target: Python label Oct 31, 2026
@github-actions github-actions Bot changed the title fix(redis): skip malformed history entries instead of raising on read Python: fix(redis): skip malformed history entries instead of raising on read Oct 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid JSON entries with malformed message shapes can raise uncaught exceptions that abort the entire history read, preventing later valid messages from being returned.

1 open issue
What changed in this PR

This PR makes the Python Redis history provider more resilient by skipping entries that raise JSON decoding or value errors instead of aborting the entire history read.

Changes:

  • Add per-entry error handling and warning logs to get_messages.
  • Add a regression test confirming valid messages remain readable after empty or invalid JSON entries.
File Description
python/​packages/​redis/​tests/​test_providers.py Tests recovery from empty and invalid JSON history entries.
python/​packages/​redis/​agent_framework_redis/​_history_provider.py Catches deserialization value errors and logs skipped entries.

Review effort: Balanced

Comment on lines +277 to +280
try:
messages.append(Message.from_dict(self._deserialize_json(serialized)))
except (ValueError, json.JSONDecodeError) as exc:
logger.warning("Skipping malformed Redis history entry: %s", exc)
Valid JSON that deserializes to a non-dict (null, []) or a dict missing
required fields ({}) previously raised AttributeError/TypeError from
Message.from_dict, which was not caught and aborted the entire read.

Guard the decoded payload with an isinstance-dict check and broaden the
except clause to (ValueError, TypeError, AttributeError, JSONDecodeError).
@lisry-1
lisry-1 temporarily deployed to github-app-auth November 1, 2026 12:28 — with GitHub Actions Inactive
@eavanvalkenburg

Copy link
Copy Markdown
Member

Thanks for the contribution, lisry-1. The code review is clean, but this changes malformed-history handling from fail-fast to skip-and-warn, so the PR needs the required Related Issue answer before approval. Please link an agreed issue with a closing keyword, or document a maintainer-approved exception. Missing AI Assistance alone is not a blocker.

@lisry-1

lisry-1 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks Eduard van Valkenburg (@eavanvalkenburg). I've opened #9974 to track this bug and linked it in the PR's Related Issue section (Fixes #9974). This is a self-contained defect fix — the Redis provider diverges from the existing Cosmos provider behavior — so I've documented the issue as the agreed requirement. Please let me know if the issue needs a ready-for-implementation label or any further confirmation before you can proceed with approval. Happy to adjust.

@eavanvalkenburg

Copy link
Copy Markdown
Member

lisry-1 The new warning logs str(exc) from deserializing an untrusted Redis history payload. Message.from_dict includes the supplied type value in mismatch errors, so private history data embedded there can be copied into application logs even though the malformed entry is skipped. Please log a safe failure category or exception class instead of exception text, and add a regression asserting that sensitive payload values are absent from the warning.

…ormed history

Message.from_dict embeds the payload's type value in mismatch errors, so
logging str(exc) copies untrusted history data into application logs.
Log the exception class name instead and add a regression asserting the
sensitive value is absent from the warning.
@lisry-1
lisry-1 temporarily deployed to github-app-auth November 4, 2026 07:33 — with GitHub Actions Inactive
@lisry-1

lisry-1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Eduard van Valkenburg (@eavanvalkenburg) thanks for catching that — you're right, str(exc) could copy untrusted history data into application logs, since Message.from_dict embeds the payload's type value in mismatch errors.

I've pushed a fix:

  • The skip warning now logs the exception class (type(exc).__name__) instead of str(exc), so no payload-derived value reaches the log.
  • Added test_malformed_entries_do_not_leak_payload_in_logs, which stores a payload with type: "secret-internal-token" and asserts the warning is emitted while the sensitive value is absent from it.

This branch was previously deployed

1 inactive deployment
github-app-auth — 9a1cd018 Deployed Nov 4, 2026 by lisry-1 via add_label #27054
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants