Add evidence-first crawl analysis

README.md

@@ -84,6 +84,24 @@ python main.py -u https://example.com --disable-socks5 --visualize tree
 torbot app
 ```
 
+### Evidence-first AI analysis
+
+Save a versioned crawl result and turn it into a cited investigation bundle:
+
+```sh
+torbot --url https://example.com --disable-socks5 \
+  --save result --result-file crawl-result.json
+torbot analyze crawl-result.json --provider ollama --model qwen3 \
+  --output investigation/
+```
+
+Analyst defaults to a local Ollama endpoint. Remote OpenAI-compatible
+providers require the explicit `--allow-remote` flag. Every supported finding
+must cite captured evidence, and a provider outage never prevents the
+deterministic evidence bundle from being written. See
+[TorBot Analyst](docs/ANALYST.md) for the offline quick start, output formats,
+privacy boundary, and schemas.
+
 ### Options
 ```text
 usage: Gather and analyze data from Tor sites.

docs/ANALYST.md

@@ -0,0 +1,82 @@
+# TorBot Analyst
+
+TorBot Analyst turns a versioned crawl result into a local investigation
+bundle. It records stable evidence IDs and content hashes, maps page and
+contact relationships, and requires every supported finding to reference
+captured evidence.
+
+## Five-minute quick start
+
+Create a deterministic report without an AI provider:
+
+```sh
+torbot analyze tests/fixtures/analyst-safe-crawl.json \
+  --provider none \
+  --keyword example \
+  --output investigation/
+```
+
+The output contains:
+
+- `report.md` with supported, conflicted, and unsupported findings
+- `evidence.jsonl` with captured URLs, excerpts, timestamps, and hashes
+- `graph.json` with evidence-linked page and contact relationships
+- `run.json` with reproducibility metadata and provider warnings
+
+## Capture a crawl result
+
+The existing tree and JSON outputs are unchanged. Save the new versioned
+result explicitly:
+
+```sh
+torbot --url https://example.com \
+  --disable-socks5 \
+  --save result \
+  --result-file crawl-result.json
+```
+
+The versioned result contains bounded visible text, not raw HTML, scripts,
+styles, cookies, headers, credentials, or arbitrary exception messages.
+
+## Local AI analysis
+
+Ollama is the default provider and is contacted only on localhost:
+
+```sh
+torbot analyze crawl-result.json \
+  --provider ollama \
+  --model qwen3 \
+  --output investigation/
+```
+
+If Ollama or the model is unavailable, TorBot still writes the deterministic
+evidence bundle and records a warning in `run.json` and `report.md`.
+
+## Remote OpenAI-compatible providers
+
+Remote transmission is fail-closed. Both an explicit provider and
+`--allow-remote` are required. The API key is read from `OPENAI_API_KEY` and is
+never written to an output file. Email addresses and phone numbers are
+redacted before remote transmission. Repeat `--redact-pattern` to add
+case-specific regular expressions.
+
+```sh
+export OPENAI_API_KEY=your-key
+torbot analyze crawl-result.json \
+  --provider openai-compatible \
+  --base-url https://api.openai.com/v1 \
+  --model your-model \
+  --allow-remote \
+  --redact-pattern 'CASE-[0-9]+' \
+  --output investigation/
+```
+
+Captured page text is untrusted data. Analyst exposes no tools to the model,
+does not execute page instructions, discards unknown evidence references, and
+never promotes an uncited model statement to `supported`.
+
+## Schemas
+
+Machine-readable schemas live in `schemas/crawl-result.v1.schema.json` and
+`schemas/investigation-report.v1.schema.json`. Consumers must reject schema
+versions they do not understand.

docs/CHANGELOG.md

@@ -2,6 +2,17 @@
 --------------------
 All notable changes to this project will be documented in this file.
 
+## Unreleased
+
+### Added
+- Added `--save result` for analysis-ready crawl-result v1 files with bounded visible text and content hashes.
+- Added `torbot analyze` for deterministic evidence bundles and optional local or remote OpenAI-compatible analysis.
+- Added stable evidence IDs, claim citation validation, an evidence graph, JSON schemas, and a safe surface-web fixture.
+
+### Security
+- Remote evidence transmission now requires `--allow-remote`, redacts contact data by default, and never stores API keys.
+- Crawled content is treated as untrusted data, model calls receive no tools, and uncited model output cannot become a supported finding.
+
 ## 4.3.0 - 2026-07-28
 
 ### Changed

docs/CRAWL_RESULT.md

@@ -20,6 +20,12 @@ Links and contacts include their source (`anchor`, `mailto`, or `tel`). Raw
 HTML, headers, cookies, credentials, and arbitrary exception strings are not
 part of the contract.
 
+When a caller explicitly requests analysis-ready output, a fetched page may
+also include a `title`, deterministic classification metadata, and `content`.
+Content contains bounded visible plain text and its SHA-256 digest. Scripts,
+styles, templates, and raw markup are excluded. The CLI exposes this form only
+through `--save result`; legacy `--save json` behavior is unchanged.
+
 GoTor's versioned report is the compatibility baseline: both projects use
 `schemaVersion: 1`, `engine`, `target`, depth/settings, timestamps, duration,
 and per-page URL/parent/depth/status/link information. TorBot's fields are

schemas/crawl-result.v1.schema.json

@@ -0,0 +1,33 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/DedSecInside/TorBot/schemas/crawl-result.v1.schema.json",
+  "title": "TorBot crawl-result v1",
+  "type": "object",
+  "required": ["schemaVersion", "target", "pages"],
+  "properties": {
+    "schemaVersion": {"const": 1},
+    "runId": {"type": "string"},
+    "target": {"type": "string", "minLength": 1},
+    "pages": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": ["url", "outcome"],
+        "properties": {
+          "url": {"type": "string", "minLength": 1},
+          "outcome": {"enum": ["fetched", "failed", "skipped"]},
+          "content": {
+            "type": "object",
+            "required": ["mediaType", "text", "sha256"],
+            "properties": {
+              "mediaType": {"const": "text/plain"},
+              "text": {"type": "string"},
+              "sha256": {"type": "string", "pattern": "^[a-f0-9]{64}$"}
+            },
+            "additionalProperties": false
+          }
+        }
+      }
+    }
+  }
+}

schemas/investigation-report.v1.schema.json

@@ -0,0 +1,26 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/DedSecInside/TorBot/schemas/investigation-report.v1.schema.json",
+  "title": "TorBot investigation-report v1",
+  "type": "object",
+  "required": ["schema", "createdAt", "source", "claims", "evidenceIds", "warnings"],
+  "properties": {
+    "schema": {"const": "investigation-report.v1"},
+    "claims": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": ["text", "status", "evidenceIds", "source"],
+        "properties": {
+          "text": {"type": "string"},
+          "status": {"enum": ["supported", "unsupported", "conflicted"]},
+          "evidenceIds": {"type": "array", "items": {"type": "string"}},
+          "source": {"enum": ["deterministic", "model"]}
+        },
+        "additionalProperties": false
+      }
+    },
+    "evidenceIds": {"type": "array", "items": {"type": "string"}},
+    "warnings": {"type": "array", "items": {"type": "string"}}
+  }
+}