Add OS-keyring API key storage, file permission hardening, and pip packaging fixes

Security: buffdata auth set/remove/status stores provider API keys in the
OS-native credential store (Windows Credential Manager / macOS Keychain /
Linux Secret Service) via keyring, now a base dependency. The default
secret resolver checks the environment first and falls back to the keyring
automatically, so a stored key just works with zero other configuration.
Checkpoints, report.json, and the audit database now get owner-only (0600)
file permissions on write.

Packaging: fixed two real dependency bugs a genuinely fresh-venv install
caught -- orjson and xxhash were imported directly but never declared,
so `pip install buffdata` would crash on import (orjson) or on the
default exact-dedup path (xxhash) for every new user. Also dropped a
typer[all] extra that no longer exists upstream, and added PyPI metadata
(classifiers, keywords, project URLs). Verified end-to-end against a
real fresh install from live PyPI (buffdata 0.3.0).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: voidstackloop

README.md

@@ -249,6 +249,8 @@ contract. BuffData never sends a failed request to a different provider automati
 - Local LLM servers (Ollama, LM Studio, vLLM, llama.cpp) or any other OpenAI-compatible
   endpoint, on this machine or another one on your network -- with an optional
   structural guarantee (`--network-policy local`) that calls can never leave it.
+- API keys stored in your OS's native credential store (`buffdata auth set`), not a
+  plaintext `.env` file -- see [docs/security.md](docs/security.md).
 - Local PII redaction before any dataset content reaches a provider.
 - Schema-preserving JSON/JSONL/NDJSON, CSV/TSV, Parquet, Arrow/Feather/IPC,
   YAML, plain-text, and Hugging Face on-disk dataset processing.

buffdata/cli/main.py

@@ -1,4 +1,5 @@
 import asyncio
+import os
 from pathlib import Path
 from typing import List, Optional
 import typer
@@ -1238,3 +1239,102 @@ def sbom_cmd(
     console.print(
         f"[bold green]Wrote SBOM[/bold green] ({len(sbom['components'])} components) to {output}"
     )
+
+
+auth_app = typer.Typer(
+    help="Store or remove provider API keys in the OS-native credential store (Windows "
+    "Credential Manager / macOS Keychain / Linux Secret Service), instead of a plaintext "
+    "file or a shell history entry."
+)
+app.add_typer(auth_app, name="auth")
+
+# Not exhaustive -- any name works, since the OS keyring just stores whatever key you give
+# it -- but this is every secret name a stock BuffData install actually looks for, so
+# `buffdata auth status` has something concrete to check.
+_KNOWN_SECRET_NAMES = [
+    "GEMINI_API_KEY", "GOOGLE_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY",
+    "AZURE_OPENAI_API_KEY", "OPENAI_COMPATIBLE_API_KEY",
+    "OLLAMA_API_KEY", "LMSTUDIO_API_KEY", "VLLM_API_KEY", "LLAMACPP_API_KEY",
+]
+
+
+def _keyring_service_name() -> str:
+    return os.getenv("BUFFDATA_KEYRING_SERVICE", "buffdata")
+
+
+@auth_app.command("set")
+def auth_set_cmd(
+    key_name: str = typer.Argument(
+        ...,
+        help=f"Secret name to store, e.g. one of: {', '.join(_KNOWN_SECRET_NAMES)}",
+    ),
+):
+    """Store a provider API key in the OS keyring -- prompted with hidden input, never
+    echoed to the terminal, never written to any file or shell history. Every buffdata
+    command that needs it picks it up automatically afterward with no other
+    configuration: the default secret backend checks the OS keyring whenever the
+    matching environment variable isn't already set (see buffdata/engine/secrets.py).
+    """
+    try:
+        import keyring
+    except ImportError:
+        console.print("[bold red]Install keyring (pip install keyring) to use `buffdata auth`.[/bold red]")
+        raise typer.Exit(1)
+
+    value = typer.prompt(f"Value for {key_name}", hide_input=True, confirmation_prompt=True)
+    if not value:
+        console.print("[yellow]Empty value -- nothing stored.[/yellow]")
+        raise typer.Exit(1)
+    try:
+        keyring.set_password(_keyring_service_name(), key_name, value)
+    except Exception as exc:
+        console.print(f"[bold red]Could not store {key_name} in the OS keyring: {exc}[/bold red]")
+        raise typer.Exit(1)
+    console.print(
+        f"[bold green]Stored {key_name} in the OS keyring.[/bold green] No .env file or "
+        "BUFFDATA_SECRET_BACKEND change needed -- it's used automatically from here on."
+    )
+
+
+@auth_app.command("remove")
+def auth_remove_cmd(
+    key_name: str = typer.Argument(..., help="Secret name to remove, as previously passed to `buffdata auth set`"),
+):
+    """Remove a key previously stored via `buffdata auth set`."""
+    try:
+        import keyring
+        from keyring.errors import PasswordDeleteError
+    except ImportError:
+        console.print("[bold red]Install keyring (pip install keyring) to use `buffdata auth`.[/bold red]")
+        raise typer.Exit(1)
+
+    try:
+        keyring.delete_password(_keyring_service_name(), key_name)
+        console.print(f"[bold green]Removed {key_name} from the OS keyring.[/bold green]")
+    except PasswordDeleteError:
+        console.print(f"[yellow]{key_name} was not set in the OS keyring.[/yellow]")
+
+
+@auth_app.command("status")
+def auth_status_cmd():
+    """Show which known provider secrets currently resolve, and from where (environment
+    variable vs. OS keyring) -- the values themselves are never displayed, here or
+    anywhere else in buffdata.
+    """
+    table = Table(title="Secret resolution status", border_style="cyan")
+    table.add_column("Name", style="bold")
+    table.add_column("Resolves from")
+    for name in _KNOWN_SECRET_NAMES:
+        if os.getenv(name):
+            source = "[green]environment[/green]"
+        else:
+            found = None
+            try:
+                import keyring
+
+                found = keyring.get_password(_keyring_service_name(), name)
+            except Exception:
+                pass
+            source = "[cyan]OS keyring[/cyan]" if found else "[dim]not set[/dim]"
+        table.add_row(name, source)
+    console.print(table)

buffdata/engine/checkpoint.py

@@ -2,6 +2,7 @@
 from pathlib import Path
 from typing import Dict, List, Optional, Set, Union
 from buffdata.models.schemas import DatasetItem
+from buffdata.security.permissions import restrict_to_owner
 
 
 class CheckpointManager:
@@ -36,6 +37,7 @@ def save_item(self, item: DatasetItem):
         data["_buffdata_id"] = str(item.id)
         with open(self.path, "a", encoding="utf-8") as f:
             f.write(json.dumps(data, ensure_ascii=False) + "\n")
+        restrict_to_owner(self.path)
         self.processed_ids.add(str(item.id))
 
     def load_all(self) -> List[DatasetItem]:

buffdata/engine/pipeline.py

@@ -16,6 +16,7 @@
 from buffdata.engine.validator import DatasetValidator
 from buffdata.models.formats import read_dataset, write_dataset_atomic
 from buffdata.observability import ObservabilityRegistry, disabled_registry
+from buffdata.security.permissions import restrict_to_owner
 from buffdata.models.schemas import (
     ClassificationMode,
     DatasetItem,
@@ -83,6 +84,10 @@ def _atomic_json(path: Path, data: dict[str, Any]) -> None:
         path.parent.mkdir(parents=True, exist_ok=True)
         temporary = path.with_name(f".{path.name}.tmp")
         temporary.write_bytes(orjson.dumps(data, option=orjson.OPT_INDENT_2))
+        # Owner-only permissions before the rename, not after: os.replace preserves the
+        # source inode's mode on POSIX, so this is the only chmod needed, and it means
+        # the file is never briefly world-readable at its final path.
+        restrict_to_owner(temporary)
         os.replace(temporary, path)
 
     async def _save_checkpoint(

buffdata/engine/secrets.py

@@ -3,10 +3,12 @@
 Every LLM client in engine/client.py resolves its API key through whatever
 SecretResolver is configured, instead of calling os.getenv directly. The
 default (EnvSecretResolver) makes this a no-op for every existing setup --
-nothing changes until BUFFDATA_SECRET_BACKEND is set to something else. This
-exists so an enterprise deployment can point BuffData at Vault, AWS Secrets
-Manager, GCP Secret Manager, or Azure Key Vault without any code changes above
-this layer -- only environment configuration.
+nothing changes until BUFFDATA_SECRET_BACKEND is set to something else, except
+that it now also checks the OS keyring (see KeyringSecretResolver) when an
+environment variable isn't set, so a value stored with `buffdata auth set`
+just works. Beyond the default, an enterprise deployment can point BuffData at
+Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault without any
+code changes above this layer -- only environment configuration.
 """
 
 from __future__ import annotations
@@ -21,14 +23,54 @@ class SecretResolver(Protocol):
     def get(self, key: str) -> Optional[str]: ...
 
 
+class KeyringSecretResolver:
+    """Reads secrets from the OS-native credential store -- Windows Credential Manager,
+    macOS Keychain, or Linux Secret Service/KWallet -- via the `keyring` package. The one
+    backend that needs no server, no cloud account, and no infrastructure to already exist
+    to be useful: exactly the gap for a `pip install`ed CLI running on someone's own
+    machine, where Vault/AWS/GCP/Azure secret managers all assume infrastructure that
+    plainly isn't there. Populated with `buffdata auth set <NAME>`, which prompts for the
+    value with hidden input and never writes it to any file.
+    """
+
+    SERVICE_NAME = "buffdata"
+
+    def __init__(self, service_name: Optional[str] = None):
+        self.service_name = service_name or os.getenv("BUFFDATA_KEYRING_SERVICE", self.SERVICE_NAME)
+
+    def get(self, key: str) -> Optional[str]:
+        try:
+            import keyring
+        except ImportError:
+            return None
+        try:
+            return keyring.get_password(self.service_name, key)
+        except Exception:
+            # A missing/misconfigured OS backend (headless Linux with no Secret Service,
+            # a locked keychain, ...) must degrade to "not found" for this one lookup, not
+            # crash every secret resolution on a machine that simply has none configured.
+            return None
+
+
 class EnvSecretResolver:
     """Default resolver: environment variables (including .env, already loaded via
-    engine/client.py's load_dotenv()). Matches BuffData's behavior before any secret-backend
-    integration existed -- this is what every client fell back to already.
+    engine/client.py's load_dotenv()), falling back to the OS keyring
+    (KeyringSecretResolver) for any key not found in the environment. This is the only
+    resolver that falls back to anything -- every other backend below is explicit and
+    deliberately does not -- because the fallback exists specifically to make the
+    unconfigured default case work with zero setup: `buffdata auth set GEMINI_API_KEY`
+    followed immediately by any command that needs it, no BUFFDATA_SECRET_BACKEND change
+    required. An environment variable, when set, always wins over the keyring.
     """
 
+    def __init__(self):
+        self._keyring = KeyringSecretResolver()
+
     def get(self, key: str) -> Optional[str]:
-        return os.getenv(key)
+        value = os.getenv(key)
+        if value:
+            return value
+        return self._keyring.get(key)
 
 
 class VaultSecretResolver:
@@ -178,6 +220,7 @@ def get(self, key: str) -> Optional[str]:
 
 _RESOLVERS = {
     "env": EnvSecretResolver,
+    "keyring": KeyringSecretResolver,
     "vault": VaultSecretResolver,
     "aws_secrets_manager": AWSSecretsManagerResolver,
     "gcp_secret_manager": GCPSecretManagerResolver,

buffdata/governance/audit_store.py

@@ -30,6 +30,8 @@
 
 from pydantic import BaseModel, Field
 
+from buffdata.security.permissions import restrict_to_owner
+
 
 class AuditRecord(BaseModel):
     run_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
@@ -78,6 +80,7 @@ class SQLiteAuditStore:
     def __init__(self, db_path: Union[str, Path] = "buffdata_audit.db"):
         self.db_path = Path(db_path)
         self._init_schema()
+        restrict_to_owner(self.db_path)
 
     def _connect(self) -> sqlite3.Connection:
         conn = sqlite3.connect(self.db_path)

buffdata/security/permissions.py

@@ -0,0 +1,30 @@
+"""Restrictive file permissions for locally-written files that can carry sensitive
+content -- pipeline checkpoints (which can briefly hold pre-PII-redaction text, since
+the "pii" stage runs after "validate"'s own checkpoint is saved -- see
+buffdata/engine/pipeline.py's STAGES), report.json (record counts, provider/model,
+dataset-revealing file paths), and the audit database. Owner-only (0600) on POSIX,
+where "everyone else on this machine can read your files by default" is the real
+default; a best-effort no-op everywhere else, since os.chmod doesn't express meaningful
+ACLs on Windows and NTFS's own per-user-profile isolation already covers the common
+case there.
+"""
+
+from __future__ import annotations
+
+import os
+import stat
+from pathlib import Path
+from typing import Union
+
+
+def restrict_to_owner(path: Union[str, Path]) -> None:
+    """Set owner-read-write-only (0600) permissions on `path`. Silently does nothing if
+    the path doesn't exist, or if the platform/filesystem doesn't support chmod (e.g. a
+    FAT-formatted mount) -- this is defense in depth, not a guarantee the rest of the
+    pipeline should ever depend on for correctness, so a failure here must never fail
+    the write it's protecting.
+    """
+    try:
+        os.chmod(path, stat.S_IRUSR | stat.S_IWUSR)
+    except OSError:
+        pass

docs/README.md

@@ -21,6 +21,7 @@ Written for three readers:
 | [architecture.md](architecture.md) | The 7-stage pipeline, the provider-neutral client contract, data flow, where each guarantee (strict/local network policy, accuracy contracts) actually lives in the code |
 | [cli-reference.md](cli-reference.md) | Every command, grouped by what it's for, with real examples |
 | [configuration.md](configuration.md) | Every `PipelineConfig` field: type, default, what it controls, and the YAML equivalent |
+| [security.md](security.md) | API keys in the OS keyring instead of a plaintext file (`buffdata auth`), and owner-only file permissions on checkpoints/reports/the audit DB |
 | [providers.md](providers.md) | Cloud providers, local LLM servers (Ollama/LM Studio/vLLM/llama.cpp), secret backends (Vault/AWS/GCP/Azure), cloud dataset storage (S3/GCS/ADLS) |
 | [governance.md](governance.md) | Access policy (RBAC), OIDC/SSO bearer-token verification, the durable audit log, Data Contracts, SBOM generation |
 | [observability.md](observability.md) | OpenTelemetry tracing and Prometheus metrics per pipeline run |

docs/security.md

@@ -0,0 +1,103 @@
+# Security
+
+Two independent protections, both on by default with zero configuration: where API
+keys live, and who on the local machine can read the files BuffData writes.
+
+## API keys: the OS keyring, not a plaintext file
+
+```bash
+buffdata auth set GEMINI_API_KEY
+# Value for GEMINI_API_KEY: [hidden input]
+# Repeat for confirmation: [hidden input]
+# Stored GEMINI_API_KEY in the OS keyring.
+```
+
+Stores the key in the platform-native credential store -- Windows Credential Manager,
+macOS Keychain, or Linux Secret Service/KWallet, via the `keyring` package (a base
+dependency, not an extra: this is the default story for every `pip install buffdata`
+user, not an enterprise add-on). The prompt uses hidden input with a confirmation
+step, and the value is never written to any file, never echoed to the terminal, and
+never lands in shell history the way `export GEMINI_API_KEY=sk-...` would.
+
+Every command picks it up automatically afterward -- no `.env` file, no
+`BUFFDATA_SECRET_BACKEND` change. `EnvSecretResolver`, the default backend
+([`buffdata/engine/secrets.py`](../buffdata/engine/secrets.py)), checks the
+environment variable first (so CI/scripted use with real env vars is completely
+unaffected) and falls back to the OS keyring only when that's unset. It's the one
+backend that behaves this way; every other backend (`vault`,
+`aws_secrets_manager`, `gcp_secret_manager`, `azure_key_vault` -- see
+[providers.md](providers.md#secret-backends)) is explicit and doesn't fall back to
+anything, since those are deliberate infrastructure choices, not a smoothing default.
+
+```bash
+buffdata auth status    # which known secrets resolve, and from where -- values never shown
+buffdata auth remove GEMINI_API_KEY
+```
+
+```
+          Secret resolution status
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
+┃ Name                      ┃ Resolves from ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
+│ GEMINI_API_KEY            │ not set       │
+│ GOOGLE_API_KEY            │ not set       │
+│ OPENAI_API_KEY            │ environment   │
+│ ANTHROPIC_API_KEY         │ not set       │
+│ AZURE_OPENAI_API_KEY      │ not set       │
+│ OPENAI_COMPATIBLE_API_KEY │ not set       │
+│ OLLAMA_API_KEY            │ not set       │
+│ LMSTUDIO_API_KEY          │ not set       │
+│ VLLM_API_KEY              │ not set       │
+│ LLAMACPP_API_KEY          │ not set       │
+└───────────────────────────┴───────────────┘
+```
+
+Set `BUFFDATA_KEYRING_SERVICE` to use a different keyring namespace than the default
+`buffdata` (multiple installs/profiles on one machine, for instance).
+
+Verified for real in
+[`tests/test_auth_keyring.py`](../tests/test_auth_keyring.py) -- genuine
+`keyring.set_password`/`get_password`/`delete_password` round trips (via a real,
+file-backed `keyrings.alt` backend in the test environment, since no OS-native
+credential store is available in headless CI; the platform-native ones are what
+production actually uses), the full `buffdata auth set/remove/status` CLI flow, and an
+end-to-end test confirming a key stored via `auth set` is what `create_llm_client`
+actually resolves and uses -- no gap between "stored" and "used."
+
+## Files at rest: owner-only permissions
+
+[`buffdata/security/permissions.py`](../buffdata/security/permissions.py)'s
+`restrict_to_owner()` sets `0600` (owner read/write only) on every locally-written file
+that can carry sensitive content, immediately after writing it:
+
+- **Pipeline checkpoints** (`.{output}.checkpoint.json`) -- can briefly hold
+  pre-PII-redaction text, since the checkpoint after the `validate` stage is written
+  before the `pii` stage runs (see [architecture.md](architecture.md)).
+- **`report.json`** -- record counts, provider/model, dataset-revealing file paths.
+- **The audit database** (`buffdata_audit.db` by default) -- every recorded run across
+  a team, queryable by anyone who can read the file.
+
+A no-op on Windows (`os.chmod` doesn't express meaningful ACLs there, and NTFS's
+per-user-profile isolation already covers the common single-user-machine case) and
+never fails the write it's protecting -- this is defense in depth for a shared POSIX
+machine, not something the rest of the pipeline depends on for correctness.
+
+## What this doesn't cover
+
+- **Encryption at rest** for these same files was deliberately not built: without a
+  passphrase or key-management story for a `pip install`ed CLI to draw from, an
+  encryption key would have to be hardcoded (pointless) or become yet another secret
+  the user has to manage (worse than the file-permission protection it would replace).
+  File permissions are the standard mitigation for this exact scenario --
+  `~/.ssh/id_rsa` and `~/.aws/credentials` use the same approach, not encryption.
+- **The final output dataset** (`optimized.jsonl` and its `.rejected.jsonl`) is *not*
+  permission-restricted -- it's the deliverable you explicitly asked BuffData to
+  produce for downstream use (training, sharing, `buffdata push` to a Hub), and
+  defaulting it to owner-only would just be friction for the common case of a build
+  pipeline or teammate reading it under a different local user.
+- Redacting secrets from third-party SDK exception messages (if `openai`/`anthropic`/
+  `google-genai` ever included a raw key in an error string) is outside BuffData's
+  control -- checked during this work and none of buffdata's own code paths do this
+  (no client's `__repr__`, exception message, or anything written to `report.json`/the
+  audit DB ever includes `self.api_key`), but a third-party library's own error
+  formatting isn't something this codebase can guarantee.