Add managed team-deployment platform: run orchestration, security hardening, and a Rust worker

Managed run system (buffdata/runs/, buffdata/server/):
- Submit/claim/execute/finish lifecycle with immutable manifests and resumable/cancellable runs
- Run lineage tracking and HTTPS webhook notifications (HMAC-signed, SSRF-guarded)
- Worker pools: multiple projects sharing one worker token/claim queue, with fair randomized
  claim scanning so no pool member is structurally starved
- Encryption at rest for managed artifacts: AES-GCM envelope encryption, per-project data
  keys wrapped by an operator-supplied master key, never persisted to disk unencrypted

Security hardening:
- Subprocess sandboxing for third-party validator/PII-recognizer plugins
- Bounded-memory minhash deduplication (xxhash fingerprints instead of raw shingle strings)
- Presidio dependency isolation into its own venv (unblocks a cryptography advisory fix in
  the main environment)
- OS-keyring credential storage and file-permission hardening for local CLI use

Team deployment: Docker Compose stack (API, worker, Postgres, nginx/squid egress proxy),
OIDC browser auth, a reproducible synthetic-OIDC smoke test, and CI workflows.

rust-worker/: an opt-in Rust reimplementation of the worker's claim loop and process
supervisor -- identical wire protocol to buffdata/server/worker.py, still delegates the
actual optimization work to the Python buffdata.runs.executor subprocess.

Fix: the managed-run supply-chain identity fingerprint (runtime_identity) no longer depends
on python-dotenv's ambient, invocation-mode-dependent .env discovery, which could make a
submission and its execution disagree about the environment and falsely reject a valid run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: voidstackloop

.dockerignore

@@ -0,0 +1,19 @@
+# Build-context allowlist: datasets, credentials, history and local caches stay out.
+**
+!pyproject.toml
+!README.md
+!buffdata/
+!buffdata/**
+!deploy/
+!deploy/Dockerfile.team
+!deploy/requirements-cpu.lock
+!rust-worker/
+!rust-worker/Cargo.toml
+!rust-worker/Cargo.lock
+!rust-worker/src/
+!rust-worker/src/**
+**/.env
+**/.env.*
+**/__pycache__
+**/*.pyc
+rust-worker/target/

.github/workflows/deployment-smoke.yml

@@ -0,0 +1,44 @@
+name: Team deployment smoke
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - 'deploy/**'
+      - 'buffdata/server/**'
+      - 'buffdata/runs/**'
+      - 'buffdata/security/**'
+      - 'buffdata/governance/oidc.py'
+      - 'pyproject.toml'
+      - '.dockerignore'
+      - '.github/workflows/deployment-smoke.yml'
+permissions:
+  contents: read
+jobs:
+  compose:
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - uses: docker/setup-buildx-action@v3
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: deploy/Dockerfile.team
+          load: true
+          tags: buffdata-team:local
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+      - run: pip install httpx PyJWT cryptography PyYAML
+      - run: python deploy/smoke_test.py
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: compose-smoke-results
+          # Never upload the generated secret files, JWT material, or database backup.
+          path: |
+            /tmp/buffdata-compose-smoke-*/results.json
+            /tmp/buffdata-compose-smoke-*/sbom.json
+          if-no-files-found: warn

.github/workflows/security.yml

@@ -0,0 +1,30 @@
+name: Security and compatibility
+on: [push, pull_request]
+permissions:
+  contents: read
+jobs:
+  boundaries:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: pip install torch --index-url https://download.pytorch.org/whl/cpu
+      - run: pip install '.[server,dev]' pip-audit keyrings.alt
+      - run: pytest -q tests/test_run_management.py tests/test_security_boundaries.py tests/test_team_server.py tests/test_adaptive_pipeline.py tests/test_accuracy_gate.py
+      - run: pip-audit
+      - run: buffdata sbom --output sbom.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dependency-sbom
+          path: sbom.json
+  secrets:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: gitleaks/gitleaks-action@v2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -16,3 +16,28 @@ outputs/
 *.jsonl
 !examples/*.jsonl
 graphify-out/
+deploy/team/
+*.db
+*.db-journal
+*.db-wal
+*.db-shm
+benchmarks/results-managed-security/runs/
+.managed-test-artifacts/
+rust-worker/target/
+
+# Generated benchmark run output, not source -- the benchmark scripts themselves are tracked.
+benchmarks/results-*/
+benchmarks/results-*.log
+benchmarks/_*.json
+
+# Stray artifacts that have shown up in this working tree from local runs -- not part of the
+# project. The venv-shaped one (bin/lib/lib64/include/pyvenv.cfg at repo root) came from a
+# `python -m venv` accidentally targeted at the repo root instead of .venv/.
+/bin/
+/lib/
+/lib64
+/include/
+/pyvenv.cfg
+/.stress_output.checkpoint.json
+/stress_25k_out.report.json
+/train_25k_clean.report.json

README.md

@@ -1,5 +1,29 @@
 # BuffData
 
+## Secure run management — optimization unchanged
+
+Managed CLI/SDK runs now add immutable input snapshots, run history, verified artifact
+manifests, cancellation/resume, and original/generated comparisons. An optional internal-team
+API/dashboard adds OIDC login, project roles, bounded uploads, and project-specific workers.
+
+Compatibility was checked on cached AG News data with the same PyTorch model, six epochs,
+and three seeds for each condition:
+
+| Source / test rows | Original | Legacy optimizer | Managed optimizer |
+|---|---:|---:|---:|
+| 3,000 / 1,000 | 83.90% | 83.90% | 83.90% |
+| 20,000 / 4,000 | 87.80% | 87.80% | 87.80% |
+| 100,000 / 7,000 | 89.38% | 89.38% | 89.38% |
+
+Ordered text/label sequences, accuracy, and macro-F1 matched. These are preservation
+checks, not additional accuracy gains. [Measured results and timing boundaries](benchmarks/results-managed-security/REPORT.md).
+
+Start with `buffdata runs start input.parquet --config pipeline.yaml`.
+See [run management and security policy](docs/run-management.md) and
+[team deployment](docs/team-deployment.md). Docker Compose syntax and image digests are
+verified; live Compose/PostgreSQL/OIDC deployment checks still require a working Docker
+engine and organization identity configuration. Do not treat this as production validation.
+
 ## Verified benchmark results
 
 The latest accuracy gate directly compares matching original AG News records with

benchmarks/_verify_datasets.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""Verify candidate Hugging Face datasets for the binary/multi-class/multi-label CLI
+classification benchmark, without downloading full data -- uses
+load_dataset_builder().info to inspect schema and split sizes.
+"""
+from __future__ import annotations
+
+import json
+
+from datasets import ClassLabel, Sequence, Value, load_dataset_builder
+
+LABEL_COL_CANDIDATES = ("label", "labels", "topic", "answer", "category", "class")
+
+# (hf_id, config_or_None, expected_type)
+CANDIDATES = [
+    # --- binary (SetFit re-hosts + originals known to load) ---
+    ("stanfordnlp/imdb", None, "binary"),
+    ("fancyzhx/yelp_polarity", None, "binary"),
+    ("fancyzhx/amazon_polarity", None, "binary"),
+    ("stanfordnlp/sst2", None, "binary"),
+    ("SetFit/sst2", None, "binary"),
+    ("SetFit/imdb", None, "binary"),
+    ("SetFit/amazon_polarity", None, "binary"),
+    ("SetFit/enron_spam", None, "binary"),
+    ("SetFit/subj", None, "binary"),
+    ("SetFit/CR", None, "binary"),
+    ("SetFit/SentEval-CR", None, "binary"),
+    ("SetFit/mrpc", None, "binary"),
+    ("SetFit/qqp", None, "binary"),
+    ("SetFit/qnli", None, "binary"),
+    ("SetFit/rte", None, "binary"),
+    ("SetFit/wnli", None, "binary"),
+    ("SetFit/hate_speech18", None, "binary"),
+    ("SetFit/hate_speech_offensive", None, "binary"),
+    ("SetFit/ethos_binary", None, "binary"),
+    ("SetFit/toxic_conversations", None, "binary"),
+    ("SetFit/toxic_conversations_50k", None, "binary"),
+    ("SetFit/insincere-questions", None, "binary"),
+    ("SetFit/ade_corpus_v2_classification", None, "binary"),
+    ("SetFit/onestop_english", None, "multiclass"),
+    ("SetFit/wsc_fixed", None, "binary"),
+    ("nyu-mll/glue", "sst2", "binary"),
+    ("nyu-mll/glue", "qqp", "binary"),
+    ("cardiffnlp/tweet_eval", "offensive", "binary"),
+    ("cardiffnlp/tweet_eval", "emotion", "multiclass"),
+    ("cornell-movie-review-data/rotten_tomatoes", None, "binary"),
+    # --- multi-class ---
+    ("fancyzhx/ag_news", None, "multiclass"),
+    ("fancyzhx/dbpedia_14", None, "multiclass"),
+    ("community-datasets/yahoo_answers_topics", None, "multiclass"),
+    ("dair-ai/emotion", None, "multiclass"),
+    ("SetFit/20_newsgroups", None, "multiclass"),
+    ("SetFit/emotion", None, "multiclass"),
+    ("SetFit/ag_news", None, "multiclass"),
+    ("SetFit/bbc-news", None, "multiclass"),
+    ("SetFit/sst5", None, "multiclass"),
+    ("SetFit/yelp_review_full", None, "multiclass"),
+    ("SetFit/TREC-QC", None, "multiclass"),
+    ("SetFit/student-question-categories", None, "multiclass"),
+    ("SetFit/tweet_eval_stance", None, "multiclass"),
+    ("SetFit/amazon_massive_scenario_en-US", None, "multiclass"),
+    ("SetFit/amazon_massive_intent_en-US", None, "multiclass"),
+    ("SetFit/amazon_reviews_multi_en", None, "multiclass"),
+    ("SetFit/ethos", None, "multiclass"),
+    # --- multi-label ---
+    ("google-research-datasets/go_emotions", "simplified", "multilabel"),
+    ("google-research-datasets/go_emotions", "raw", "multilabel"),
+    ("SetFit/go_emotions", None, "multilabel"),
+    ("argilla/go_emotions_multi-label", None, "multilabel"),
+    ("owaiskha9654/PubMed_MultiLabel_Text_Classification_Dataset_MeSH", None, "multilabel"),
+    ("google/civil_comments", None, "multilabel"),
+    ("google/jigsaw_toxicity_pred", None, "multilabel"),
+    ("mteb/toxic_conversations_50k", None, "binary"),
+    ("Arsive/toxicity_classification_jigsaw", None, "multilabel"),
+    ("SetFit/ethos", "multilabel", "multilabel"),
+]
+
+results = []
+for hf_id, config, expected in CANDIDATES:
+    entry = {"hf_id": hf_id, "config": config, "expected": expected}
+    try:
+        builder = load_dataset_builder(hf_id, config) if config else load_dataset_builder(hf_id)
+        info = builder.info
+        features = info.features or {}
+        splits = info.splits
+        train_split = None
+        if splits:
+            for name in ("train", "training"):
+                if name in splits:
+                    train_split = name
+                    break
+        train_rows = splits[train_split].num_examples if train_split else None
+        entry["train_rows"] = train_rows
+        entry["columns"] = list(features.keys())
+
+        label_col = None
+        label_type = None  # "single" or "multi"
+        num_classes = None
+        for name in LABEL_COL_CANDIDATES:
+            if name in features:
+                feat = features[name]
+                label_col = name
+                if isinstance(feat, ClassLabel):
+                    label_type, num_classes = "single", feat.num_classes
+                elif isinstance(feat, Sequence) and isinstance(feat.feature, ClassLabel):
+                    label_type, num_classes = "multi", feat.feature.num_classes
+                elif isinstance(feat, Sequence):
+                    label_type = "multi"
+                elif isinstance(feat, (Value,)) and feat.dtype in ("int64", "int32", "bool"):
+                    label_type = "single"  # plain int/bool label column (e.g. SetFit style)
+                break
+        entry["label_col"] = label_col
+        entry["label_type"] = label_type
+        entry["num_classes"] = num_classes
+        entry["ok"] = bool(train_rows and train_rows >= 10000 and label_col)
+        entry["error"] = None
+    except Exception as exc:
+        entry["ok"] = False
+        entry["error"] = f"{type(exc).__name__}: {exc}"[:150]
+    results.append(entry)
+    status = "OK" if entry.get("ok") else "FAIL"
+    print(f"[{status}] {hf_id} ({config}) expected={expected} -> "
+          f"rows={entry.get('train_rows')} label_col={entry.get('label_col')} "
+          f"label_type={entry.get('label_type')} classes={entry.get('num_classes')} "
+          f"err={entry.get('error')}", flush=True)
+
+with open("benchmarks/_dataset_verification.json", "w") as f:
+    json.dump(results, f, indent=2)
+
+ok = [r for r in results if r["ok"]]
+print(f"\n{len(ok)}/{len(results)} candidates verified OK")
+for kind in ("binary", "multiclass", "multilabel"):
+    matching = [r for r in ok if r["expected"] == kind]
+    print(f"  {kind}: {len(matching)} verified")

benchmarks/_verify_true_classes.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+"""Get the TRUE full-dataset class count for every entry in DATASETS (benchmark_scale_matrix.py),
+using dataset.unique() on the full train split -- not a partial sample, which can miss rare
+classes (exactly what caused the amazon_massive_scenario bug: a 3000-row sample only showed
+14 of the real 17+ classes)."""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent))
+from benchmark_scale_matrix import DATASETS, load_normalized
+
+for name, spec in DATASETS.items():
+    try:
+        train, test = load_normalized(spec)
+        train_labels = set(train.unique("label"))
+        test_labels = set(test.unique("label"))
+        all_labels = train_labels | test_labels
+        true_n = len(all_labels)
+        declared_n = spec["classes"]
+        min_label, max_label = min(all_labels), max(all_labels)
+        contiguous = all_labels == set(range(min_label, max_label + 1))
+        status = "OK" if true_n == declared_n and min_label == 0 and contiguous else "MISMATCH"
+        print(f"[{status}] {name}: declared={declared_n} true={true_n} range=[{min_label},{max_label}] contiguous={contiguous} train_rows={len(train)}", flush=True)
+    except Exception as exc:
+        print(f"[FAIL] {name}: {type(exc).__name__}: {str(exc)[:150]}", flush=True)