SentinelAI — AI-Powered Threat Detection & Incident Correlation Platform

High-throughput web access log analysis, deterministic heuristic threat detection, chronological attack sessionization, and SOC-grade incident correlation engine.

Key Capabilities • Architecture • Detector Matrix • Correlation Engine • API Reference • Getting Started

📌 Overview

SentinelAI is an intrusion detection and incident correlation platform engineered for cybersecurity analysts and Security Operations Centers (SOC). It ingests raw web server telemetry (Apache Combined Log Format), streaming each event through a pipeline of deterministic heuristic threat detectors.

Rather than overwhelming security analysts with isolated, noisy log alerts, SentinelAI incorporates an Incident Correlation Engine that groups sequential alerts by attacker IP address within a 600-second sliding time window, collapses 30-second burst duplicates, aggregates targeted URI endpoints, and automatically escalates multi-vector attack campaigns to CRITICAL severity.

All telemetry and correlated sessions are persisted in a relational SQLite data store and exposed via a RESTful FastAPI backend to a high-performance React 19 + TypeScript dashboard equipped with interactive Recharts visualizations, raw payload inspection, and export pipelines.

⚡ Key Capabilities

Streaming Log Ingestion: Memory-efficient generator streaming (ApacheLogParser) capable of tokenizing, regex-validating, and decoding high-volume Apache access logs line-by-line.
Deterministic Heuristic Detection: 5 specialized rule engines inspecting decoded URIs, HTTP methods, status codes, and request metadata and decoded URLs for malicious web signatures with confidence scoring up to 0.99.
State-Aware Brute Force Tracking: Sliding-window session tracking that correlates POST /login.php 302 redirects and HTTP 401 Unauthorized responses against configurable failure thresholds.
Incident Sessionization & Deduplication: Multi-alert attack grouping by Source IP with dynamic sliding time windows (600s), burst deduplication (30s), target endpoint harvesting, and campaign classification.
Composite Severity Escalation: Dynamic risk scoring that escalates multi-vector incidents involving OS Command Injection or Brute Force attempts to CRITICAL.
Threat Explainability & CWE Mapping: Built-in mapping of detection signatures to authoritative Common Weakness Enumerations (CWE-89, CWE-79, CWE-22, CWE-78, CWE-307) with actionable remediation guidance.
Interactive SOC Dashboard: Cyber-defense telemetry interface with real-time incident querying, severity filters, heuristic distribution charts, temporal attack trajectories, and raw log inspection drawers.
Multi-Format Data Export: Direct streaming export of security alerts to structured JSON (/api/v1/export/json) and tabular CSV spreadsheets (/api/v1/export/csv).

🏛 Architecture

The following diagram illustrates the complete data flow from raw access logs to persistent storage and UI presentation:

flowchart TD
    subgraph INGESTION["1. Telemetry Ingestion Layer"]
        A[Raw Apache Access Logs<br/><i>Combined Log Format</i>] --> B[ApacheLogParser<br/><i>Regex Tokenizer & URL Decoder</i>]
        B --> C[SecurityEvent Stream<br/><i>Normalized Data Models</i>]
    end

    subgraph DETECTION["2. Heuristic Detection Layer"]
        C --> D[DetectorManager]
        D --> D1[SQLInjectionDetector<br/><i>Confidence: 0.98 | HIGH</i>]
        D --> D2[XSSDetector<br/><i>Confidence: 0.95 | HIGH</i>]
        D --> D3[DirectoryTraversalDetector<br/><i>Confidence: 0.97 | HIGH</i>]
        D --> D4[CommandInjectionDetector<br/><i>Confidence: 0.99 | CRITICAL</i>]
        D --> D5[BruteForceDetector<br/><i>Confidence: 0.95 | Stateful HIGH</i>]
        D1 & D2 & D3 & D4 & D5 --> E[DetectionResult Stream]
    end

    subgraph PERSISTENCE["3. Persistence & Correlation"]
        E --> F[AlertRepository]
        F --> G[(SQLite Database<br/><i>sentinel.db</i>)]
        G --> H[CorrelationEngine<br/><i>600s Sliding Window & 30s Dedup</i>]
        H --> I[CorrelatedIncident Sessions<br/><i>Multi-Vector Severity Escalation</i>]
    end

    subgraph API_AND_UI["4. API & SOC Interface"]
        I & G --> J[FastAPI REST API<br/><i>/api/v1 Endpoints</i>]
        J --> K[React 19 + Vite Dashboard]
        K --> K1[Incident Cards & Threat Timelines]
        K --> K2[Heuristic & Severity Recharts]
        K --> K3[Raw Payload & CWE Inspector]
        K --> K4[JSON / CSV Export Pipelines]
    end

🛡 Threat Coverage & Detector Matrix

SentinelAI v1.0.0 ships with 5 production-calibrated heuristic detectors registered under backend/detector/:

Detector	Target Attack Vector	CWE Mapping	Base Severity	Confidence	Key Signatures & Heuristic Logic
`SQLInjectionDetector`	SQL Injection (SQLi)	CWE-89	`HIGH`	`0.98`	Matches `UNION SELECT`, boolean tautologies (`OR 1=1`, `AND 1=1`), `DROP TABLE`, `information_schema`, `xp_cmdshell`, `load_file`, `INTO OUTFILE`, `sleep()`, `benchmark()`, and encoded hex/URL SQL tokens (`%27`, `%20or%20`, `%20union%20`).
`XSSDetector`	Cross-Site Scripting (XSS)	CWE-79	`HIGH`	`0.95`	Matches `<script...>`, `</script>`, `javascript:`, inline DOM event handlers (`onerror=`, `onload=`), `alert()`, `document.cookie`, `document.location`, `window.location`, `<img`, `<svg`, `%3Cscript`, and `%3E`.
`DirectoryTraversalDetector`	Path / Directory Traversal	CWE-22	`HIGH`	`0.97`	Identifies relative traversal sequences (`../`, `..\`, `%2e%2e%2f`, `%2e%2e/`, `..%5c`) and direct targets to sensitive system files (`/etc/passwd`, `/etc/shadow`, `boot.ini`, `win.ini`, `system32`).
`CommandInjectionDetector`	OS Command Injection / RCE	CWE-78	`CRITICAL`	`0.99`	Detects shell metacharacters (`;`, `\|`, `&&`, `$()`, backticks) chained with operating system binaries (`whoami`, `cat`, `ls`, `pwd`, `id`, `wget`, `curl`, `powershell`, `cmd.exe`, `/bin/sh`, `/bin/bash`).
`BruteForceDetector`	Credential Brute Force	CWE-307	`HIGH`	`0.95`	Stateful tracker correlating in-flight `POST` submissions with failure redirects (`GET /login.php 200`) or HTTP `401 Unauthorized` responses exceeding 5 failed attempts within a 10.0s window.

🧠 Incident Correlation Engine

Individual alerts often represent symptoms of a broader attack campaign. The CorrelationEngine (backend/services/correlation_engine.py) transforms atomic alerts into high-level attack sessions:

Chronological Sorting & IP Partitioning: Sorts all alerts chronologically and partitions them by unique Source IP.
Sliding Time Window (600 seconds): Groups contiguous security events from the same IP into a single session as long as subsequent events arrive within 10 minutes of the previous alert. Exceeding this delta closes the incident and initiates a new session.
Burst Deduplication (30 seconds): Prevents flood fatigue by collapsing rapid, identical (detector, target_path) alert bursts occurring within 30 seconds into the parent incident metrics.
Target Endpoint Aggregation: Extracts and deduplicates unique URI paths (e.g., /vulnerabilities/sqli/, /login.php) targeted during the session.
Dynamic Severity Escalation:
- Preserves CRITICAL if any constituent alert is CRITICAL.
- Automatically escalates multi-vector attack campaigns containing CommandInjectionDetector or BruteForceDetector to CRITICAL.
- Assigns HIGH to general multi-vector incidents or single-vector high-severity attacks.

🔍 Alert Investigation & Explainability

SentinelAI provides contextual threat explainability inside the frontend (frontend/src/utils/detectorExplainer.ts):

CWE Contextualization: Translates raw detector triggers into standard CWE definitions and risk categories (e.g., Database Exfiltration & Authentication Bypass or Remote Code Execution).
Decoded URL & Query Tokenizer: Decodes URL-encoded parameters (e.g., %27+OR+1%3D1 -> ' OR 1=1) to display the attacker's true intent.
Matched Pattern Isolation: Pinpoints the exact regex token that triggered the alert.
One-Click Log Copy: Allows SOC analysts to copy raw Apache log strings directly to the clipboard for external SIEM or ticket ingestion.
Remediation Recommendations: Surfaces actionable defense instructions for each vulnerability class (e.g., parameterized queries, input sanitization, strict CSP headers).

📊 Dashboard Capabilities

View / Feature	Capabilities
Telemetry Summary Grid	Total detections, critical/high counts, active heuristic rules, and live backend connection telemetry.
Visual Analytics	Interactive Recharts visualizations: Heuristic Distribution Bar Chart, Severity Breakdown Pie Chart, and Incident Velocity Timeline.
Attack Sessions View	Filterable incident cards displaying attack duration, source IP, targeted endpoints, tactic badges, and collapsible event timelines.
Raw Alert Stream	Paginated tabular view of all recorded events with real-time keyword search, detector filtering, and severity badges.
Drag & Drop Ingestion	Upload `.log`, `.txt`, or `.csv` files for immediate parser ingestion and automated database population.
Export Pipelines	One-click downloads for structured JSON (`alerts.json`) and tabular CSV (`alerts.csv`).
Dark / Light Theme	Instant UI theme switching powered by CSS variables and local storage persistence.

💻 Technology Stack

Backend

Language: Python 3.10+ / 3.14
Web Framework: FastAPI (Asynchronous ASGI framework)
Database & ORM: SQLAlchemy 2.0 + SQLite
Server: Uvicorn
Validation: Pydantic v2
Testing: pytest 9.1 + Starlette TestClient + AnyIO

Frontend

Framework: React 19 + TypeScript 6.0
Build Tool: Vite 8.2
Styling: TailwindCSS v4 (@tailwindcss/vite)
State & Data Fetching: TanStack React Query v5 + Axios
Charts & Motion: Recharts 3.10 + Framer Motion 13
Icons: Lucide React

📁 Repository Structure

SentinelAI/
├── backend/                        # Backend FastAPI Application & Engine
│   ├── api/                        # REST API routing, schemas, and middleware
│   │   ├── routes/                 # Route handlers: health, alerts, incidents, stats, upload, export
│   │   ├── app.py                  # FastAPI entry point & CORS configuration
│   │   ├── schemas.py              # Pydantic request/response schemas
│   │   ├── middleware.py           # Request logging & timing middleware
│   │   └── exception_handlers.py   # Global HTTP & custom error handlers
│   ├── core/                       # Core configurations, logger, and data models
│   │   ├── config.py               # Centralized path & parameter configuration
│   │   ├── models.py               # SecurityEvent data models
│   │   ├── detection_result.py     # DetectionResult container
│   │   ├── severity.py             # Severity enumeration (LOW, MEDIUM, HIGH, CRITICAL)
│   │   ├── exceptions.py           # Custom exception definitions
│   │   └── logger.py               # Application-wide structured logger
│   ├── database/                   # Database layer
│   │   ├── database.py             # SQLAlchemy engine & SQLite migration initialization
│   │   ├── models.py               # SQLAlchemy ORM models (Alert, Incident)
│   │   ├── repository.py           # Database CRUD & aggregation queries
│   │   ├── session.py              # Session factory
│   │   └── sentinel.db             # SQLite database file
│   ├── detector/                   # Heuristic detection engines
│   │   ├── base_detector.py        # Abstract BaseDetector interface
│   │   ├── detector_manager.py     # Orchestration of registered detectors
│   │   ├── brute_force_detector.py # Stateful brute force login detector
│   │   ├── command_injection_detector.py # OS command injection detector
│   │   ├── directory_traversal_detector.py # Path traversal detector
│   │   ├── sql_injection_detector.py # SQL injection detector
│   │   └── xss_detector.py         # Cross-site scripting detector
│   ├── exporter/                   # Export handlers (JSON, CSV)
│   ├── parser/                     # Apache access log parsing & regex patterns
│   │   ├── apache_parser.py        # Streaming log parser implementation
│   │   └── regex_patterns.py       # Compiled Apache Combined Log Format regex
│   ├── services/                   # Business logic layer
│   │   ├── correlation_engine.py   # Attack session correlation & severity escalation
│   │   ├── export_service.py       # File generation service for exports
│   │   ├── statistics_service.py   # Aggregate metric calculations
│   │   └── upload_service.py       # File upload & ingestion pipeline
│   └── uploads/                    # Upload cache and DVWA sample attack logs
├── frontend/                       # React 19 + TypeScript + Vite SPA
│   ├── src/
│   │   ├── api/                    # Axios instance & API endpoint definitions
│   │   ├── components/             # Reusable UI cards, tables, charts, layout, badges
│   │   ├── contexts/               # ThemeProvider (Dark / Light mode)
│   │   ├── hooks/                  # React Query hooks (useAlerts, useIncidents, useStats)
│   │   ├── pages/                  # Page views: Dashboard, Alerts, Upload, Statistics, Settings
│   │   ├── services/               # Frontend API service layer
│   │   ├── types/                  # TypeScript interfaces (Alert, Incident, Health, Stats)
│   │   └── utils/                  # Log parser, detector explainer, formatters
│   ├── package.json                # Frontend dependencies & npm scripts
│   └── vite.config.ts              # Vite config with /api reverse proxy
├── tests/                          # Automated Pytest Suite (54 Tests)
│   ├── sample_logs/                # Test log files
│   ├── conftest.py                 # Pytest fixtures
│   └── test_*.py                   # 29 test modules for APIs, detectors, parser, and models
├── docs/                           # Documentation folder
├── pytest.ini                      # Pytest configuration
├── requirements.txt                # Root Python dependencies
└── README.md                       # Project documentation

⚙ Prerequisites

Ensure you have the following installed on your host system:

Python: Version 3.10 or higher (Python 3.11+ recommended)
Node.js: Version 18.0.0 or higher (v20+ recommended)
npm: Version 9.0.0 or higher
Git: For version control

🚀 Quick Start & Installation

1. Clone the Repository

git clone https://github.com/yshivamcodes/SentinelAI.git
cd SentinelAI

2. Backend Environment Setup

Create and activate a Python virtual environment, then install required dependencies:

Windows (PowerShell):

# Create virtual environment
python -m venv .venv

# Activate virtual environment
.\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

Linux / macOS:

# Create virtual environment
python3 -m venv .venv

# Activate virtual environment
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

3. Frontend Environment Setup

cd frontend
npm install
cd ..

🏃 Running SentinelAI

To run the complete platform, start both the backend API and frontend dev server in separate terminal windows:

Terminal 1: Launch Backend API

# Using project virtual environment
.\.venv\Scripts\uvicorn.exe backend.api.app:app --reload --host 127.0.0.1 --port 8000

The FastAPI server will start at http://127.0.0.1:8000 (Interactive Swagger docs available at http://127.0.0.1:8000/docs).

Terminal 2: Launch Frontend Interface

cd frontend
npm run dev

The Vite development server will start at http://localhost:5173 with automatic API proxying to port 8000.

🧪 Testing & Verification

SentinelAI includes an automated test suite covering unit, integration, and API end-to-end functionality.

Run Backend Test Suite (54 Tests)

Always execute pytest using the project's local virtual environment to ensure all dependencies (fastapi, sqlalchemy, httpx2, starlette) resolve correctly:

Windows (PowerShell):

.\.venv\Scripts\pytest.exe -v

Linux / macOS:

./.venv/bin/pytest -v

Verified Test Summary:

============================= test session starts =============================
platform win32 -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: C:\SentinelAI
configfile: pytest.ini
testpaths: tests
plugins: anyio-4.14.2
collected 54 items

tests\test_alerts_api.py ..                                              [  3%]
tests\test_brute_force_detector.py .........                             [ 20%]
tests\test_command_injection_detector.py ..                              [ 24%]
tests\test_config.py .                                                   [ 25%]
tests\test_correlation_engine.py ......                                  [ 37%]
tests\test_cors.py .                                                     [ 38%]
tests\test_csv_exporter.py .                                             [ 40%]
tests\test_detection_pipeline.py .                                       [ 42%]
tests\test_detection_result.py .                                         [ 44%]
tests\test_detector_manager.py .                                         [ 46%]
tests\test_directory_traversal_detector.py ...                           [ 51%]
tests\test_exception_handler.py .                                        [ 53%]
tests\test_exceptions.py .                                               [ 55%]
tests\test_export_api.py ..                                              [ 59%]
tests\test_health_api.py .                                               [ 61%]
tests\test_incidents_api.py ..                                           [ 64%]
tests\test_json_exporter.py .                                            [ 66%]
tests\test_logger.py .                                                   [ 68%]
tests\test_logging_middleware.py .                                       [ 70%]
tests\test_migration.py .                                                [ 72%]
tests\test_models.py .                                                   [ 74%]
tests\test_parser.py ....                                                [ 81%]
tests\test_regex.py ..                                                   [ 85%]
tests\test_repository.py .                                               [ 87%]
tests\test_severity.py .                                                 [ 88%]
tests\test_sql_injection_detector.py ..                                  [ 92%]
tests\test_statistics_api.py .                                           [ 94%]
tests\test_upload_api.py .                                               [ 96%]
tests\test_xss_detector.py ..                                            [100%]

============================= 54 passed in 2.25s ==============================

Frontend TypeScript Build & Lint Check

cd frontend

# Verify TypeScript type-checking and production bundle build
npm run build

# Verify ESLint code quality
npm run lint

📡 REST API Reference

All application endpoints are versioned under the /api/v1 prefix:

Method	Endpoint	Description	Response Model / Output
`GET`	`/`	Root service metadata	`{"application": "SentinelAI", "version": "1.0.0", "status": "running"}`
`GET`	`/api/v1/health/`	System health check	`HealthResponse` (`status`, `service`)
`GET`	`/api/v1/alerts/`	List all recorded security alerts	`list[AlertResponse]`
`GET`	`/api/v1/alerts/{id}`	Retrieve details for a single alert	`AlertResponse` (404 if not found)
`GET`	`/api/v1/incidents/`	List all correlated attack sessions	`list[IncidentResponse]`
`GET`	`/api/v1/incidents/{id}`	Retrieve a single correlated incident	`IncidentResponse` (404 if not found)
`GET`	`/api/v1/statistics/`	Aggregate alert and detector statistics	`StatisticsResponse` (`total_alerts`, `severity`, `detectors`)
`GET`	`/api/v1/statistics/incidents`	Correlated incident statistics	`IncidentStatisticsResponse` (`total_incidents`, `multi_vector`, `actors`)
`POST`	`/api/v1/upload/`	Multipart form log file ingestion	`UploadResponse` (`filename`, `total_logs`, `parsed`, `invalid`, `alerts`)
`GET`	`/api/v1/export/json`	Download all alerts as JSON file	Streaming `FileResponse` (`alerts.json`)
`GET`	`/api/v1/export/csv`	Download all alerts as CSV spreadsheet	Streaming `FileResponse` (`alerts.csv`)

🔬 Empirical Validation (Controlled DVWA Lab)

The heuristic detectors and correlation engine in SentinelAI were calibrated and validated against real attack traffic generated inside a controlled DVWA (Damn Vulnerable Web Application) lab environment:

SQL Injection: Validated against DVWA low/medium/high SQLi endpoints testing UNION SELECT extraction, boolean queries, and error-based payloads (sqli_real.log, sqli_real2.log, sqli_real3.log).
Command Execution: Validated against DVWA ping/command execution modules exercising shell chaining operators (;, |, &&) and binary reconnaissance (command_real.log).
Directory Traversal: Validated against DVWA File Inclusion modules probing ../../ relative path escapes and /etc/passwd disclosure (traversal_real.log).
Brute Force: Validated against automated dictionary attacks directed at DVWA login forms testing rapid POST 302 redirects and HTTP 401 responses (bruteforce_real.log).
Multi-Vector Attack Campaigns: Validated against combined threat sequences spanning credential brute forcing, database exfiltration, and command injection (campaign_real.log).

Sample log files are located in backend/uploads/ and root for test ingestion.

📸 Screenshots

Dashboard Overview & Telemetry	Correlated Attack Sessions & Timeline
Live Threat Stream & Payload Inspector	Threat Analytics & Heuristic Matrix

⚠️ Limitations (v1.0.0)

To maintain absolute transparency regarding what is currently implemented:

Log Format Scope: v1.0.0 parses standard Apache Combined Log Format (.log, .txt, .csv). Additional web formats (Nginx Combined, IIS, JSON structured logs) are planned for v1.1.
Rule-Based Heuristics: Threat detection is deterministic and regex/state-driven. Machine Learning anomaly classifiers and LLM explanation agents are not active in v1.0.0.
Batch & File Upload Ingestion: Telemetry ingestion is performed via multipart file upload or CLI parsing. Live continuous log tailing (watcher.py) is scheduled for the next release.
Single-Node Persistence: SQLite is utilized as the default local relational store. Distributed database backends (PostgreSQL / ClickHouse) are targeted for multi-node deployments.

🗺 Roadmap

📖 Documentation

Interactive API Documentation (Swagger / OpenAPI): Available at http://127.0.0.1:8000/docs when the backend is running.
ReDoc Technical Specification: Available at http://127.0.0.1:8000/redoc.
Deep-Dive Architecture Manual: Refer to docs/V1_COMPLETE_DOCUMENTATION.md for in-depth algorithmic documentation, mathematical models, and deployment configurations (to be generated as part of the extended technical release pack).

📄 License

Distributed under the MIT License. See LICENSE for more information.

_{Engineered by the SentinelAI Team. Designed for high-reliability cybersecurity operations.}

Navigation Menu

Repository files navigation

SentinelAI — AI-Powered Threat Detection & Incident Correlation Platform

📌 Overview

⚡ Key Capabilities

🏛 Architecture

🛡 Threat Coverage & Detector Matrix

🧠 Incident Correlation Engine

🔍 Alert Investigation & Explainability

📊 Dashboard Capabilities

💻 Technology Stack

Backend

Frontend

📁 Repository Structure

⚙ Prerequisites

🚀 Quick Start & Installation

1. Clone the Repository

2. Backend Environment Setup

3. Frontend Environment Setup

🏃 Running SentinelAI

Terminal 1: Launch Backend API

Terminal 2: Launch Frontend Interface

🧪 Testing & Verification

Run Backend Test Suite (54 Tests)

Frontend TypeScript Build & Lint Check

📡 REST API Reference

🔬 Empirical Validation (Controlled DVWA Lab)

📸 Screenshots

⚠️ Limitations (v1.0.0)

🗺 Roadmap

📖 Documentation

📄 License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 21 Commits
backend		backend
deployment		deployment
docs		docs
frontend		frontend
models		models
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
check_database.py		check_database.py
create_database.py		create_database.py
pytest.ini		pytest.ini
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

SentinelAI — AI-Powered Threat Detection & Incident Correlation Platform

📌 Overview

⚡ Key Capabilities

🏛 Architecture

🛡 Threat Coverage & Detector Matrix

🧠 Incident Correlation Engine

🔍 Alert Investigation & Explainability

📊 Dashboard Capabilities

💻 Technology Stack

Backend

Frontend

📁 Repository Structure

⚙ Prerequisites

🚀 Quick Start & Installation

1. Clone the Repository

2. Backend Environment Setup

3. Frontend Environment Setup

🏃 Running SentinelAI

Terminal 1: Launch Backend API

Terminal 2: Launch Frontend Interface

🧪 Testing & Verification

Run Backend Test Suite (54 Tests)

Frontend TypeScript Build & Lint Check

📡 REST API Reference

🔬 Empirical Validation (Controlled DVWA Lab)

📸 Screenshots

⚠️ Limitations (v1.0.0)

🗺 Roadmap

📖 Documentation

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages