Skip to content

Bound flow storage and keep detection history - #2199

Open
AlyaGomaa wants to merge 1 commit into
sebas/feat/native_macos_rpi_runtimefrom
sebas/feat/bounded_history_retention
Open

AlyaGomaa wants to merge 1 commit into
sebas/feat/native_macos_rpi_runtimefrom
sebas/feat/bounded_history_retention

Conversation

@AlyaGomaa

@AlyaGomaa AlyaGomaa commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Sub-PR of #2141: part 2 of 12. The parent PR stays open and unchanged; this PR carries one feature of it so it can be reviewed on its own.

What this PR adds

  • Flow retention for live captures (flow_retention in config/slips.yaml): SQLite prunes the oldest raw flows by age and size and protects flows linked to evidence.
  • Detection history (detection_history.py, detection_queries.py): old detections are archived and rotated, hourly summaries are kept, maintenance yields to capture.
  • Flows are written in batches (add_flows); detector flow lookups use composite indexes.
  • Evidence is stored in SQLite before it is published; processed evidence scores are stored; scoring checks an alert index instead of loading past alerts.
  • ICMP evidence keeps type and code (is_icmp_protocol, normalize_icmp_evidence).
  • Log rotation helper for slips.log and evidence logs (log_rotation.py); accepted evidence is drained before workers stop.

Where it sits in the split

Notes for review

  • Holds the shared database and evidence changes that sub-PRs 2, 5, 6, 9 and 11 build on.

Tests

  • Unit suite on this branch (local, pytest tests/unit -n 3): 3572 passed, 2 skipped, 0 failed. Integration tests were not run locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

#2141 sub-PR Part of the split of PR #2141

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants