diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..90cd141 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,28 @@ +name: "๐Ÿ› Bug report" +description: Something crashed or behaved incorrectly (not a false positive โ€” use the other form for that). +title: "[bug] " +labels: ["bug"] +body: + - type: textarea + id: what + attributes: + label: What happened? + description: What you ran, what you expected, and what you got. + validations: + required: true + - type: textarea + id: repro + attributes: + label: Steps to reproduce + placeholder: | + 1. autonomyproof scan ./my-agent + 2. ... + validations: + required: true + - type: input + id: env + attributes: + label: Version & OS + placeholder: "autonomyproof 0.15.0, Python 3.12, macOS 14" + validations: + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..f509ba6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: true +contact_links: + - name: ๐Ÿ’ฌ Questions & ideas + url: https://github.com/autonomyproof/autonomyproof-cli/discussions + about: Ask a question, share how you use it, or float an idea before opening an issue. + - name: ๐Ÿ”’ Report a security vulnerability + url: https://github.com/autonomyproof/autonomyproof-cli/blob/main/SECURITY.md + about: Please do NOT open a public issue for vulnerabilities โ€” follow the security policy. diff --git a/.github/ISSUE_TEMPLATE/false-positive.yml b/.github/ISSUE_TEMPLATE/false-positive.yml new file mode 100644 index 0000000..6c5772c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/false-positive.yml @@ -0,0 +1,36 @@ +name: "๐ŸŽฏ Report a false positive" +description: A rule fired on code that is actually safe. These reports directly drive our accuracy work. +title: "[false-positive] AG0XX on " +labels: ["false-positive"] +body: + - type: markdown + attributes: + value: | + False positives are the enemy โ€” we publish our own FP benchmark and take these seriously. + The clearer the minimal repro, the faster it gets fixed (and it often becomes a corpus case). + - type: input + id: rule + attributes: + label: Which rule fired? + placeholder: "AG005" + validations: + required: true + - type: textarea + id: code + attributes: + label: Minimal code that was flagged (but is safe) + render: python + validations: + required: true + - type: textarea + id: why + attributes: + label: Why is it actually safe? + placeholder: "The URL comes from settings.API_URL, a trusted config constant, not model input." + validations: + required: true + - type: input + id: version + attributes: + label: Version + placeholder: "autonomyproof 0.15.0" diff --git a/.github/ISSUE_TEMPLATE/propose-a-rule.yml b/.github/ISSUE_TEMPLATE/propose-a-rule.yml new file mode 100644 index 0000000..ca6f3f1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/propose-a-rule.yml @@ -0,0 +1,60 @@ +name: "๐Ÿ›ก๏ธ Propose a detection rule" +description: Suggest a new authority/guardrail check for the scanner. Great first contribution. +title: "[rule] " +labels: ["rule-proposal", "help wanted"] +body: + - type: markdown + attributes: + value: | + Thanks for helping expand coverage! Every rule is small and self-contained โ€” + one `Rule` subclass + a positive/negative test + a `benchmark/corpus.yaml` case. + See the [Add a rule guide](../blob/main/CONTRIBUTING.md#adding-a-security-rule). + The bar is **zero false positives on the real-repo benchmark** โ€” that's the whole game. + - type: input + id: pattern + attributes: + label: What dangerous pattern should it detect? + description: One sentence โ€” the authority or guardrail gap. + placeholder: "An agent tool that can wipe an entire datastore with no approval." + validations: + required: true + - type: textarea + id: example + attributes: + label: Minimal example that SHOULD fire + description: A few lines of Python. This becomes the positive corpus case. + render: python + placeholder: | + @tool + def reset_db(): + Base.metadata.drop_all(engine) + validations: + required: true + - type: textarea + id: negative + attributes: + label: A similar example that must NOT fire (the zero-FP guard) + description: The benign look-alike that keeps the rule honest. + render: python + placeholder: | + @tool + def trim(df): + return df.drop(columns=["x"]) # pandas column drop โ€” benign + validations: + required: true + - type: input + id: frameworks + attributes: + label: Frameworks / libraries involved + placeholder: "SQLAlchemy, redis-py, pymongo" + - type: textarea + id: mappings + attributes: + label: Standards mappings (if known) + placeholder: "OWASP Agentic: Excessive agency ยท MITRE ATT&CK: T1485 ยท CVE: โ€”" + - type: checkboxes + id: willing + attributes: + label: Would you like to implement it? + options: + - label: I'd like to open the PR myself (we'll help) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..cefb487 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,22 @@ + + +## What this changes + + + +## Checklist + +- [ ] Commits are signed off (`git commit -s`) โ€” required by CI (DCO) +- [ ] `pytest` passes (100% branch coverage is enforced) +- [ ] `ruff check .` and `ruff format --check .` pass +- [ ] `mypy` passes + +### If this adds or changes a detection rule + +- [ ] Added a **positive** test (the rule fires) and a **negative** test (it does not) +- [ ] Added a positive and negative case to `benchmark/corpus.yaml` +- [ ] Ran `python benchmark/run.py` and confirmed **no new false positives** on the real-repo corpus +- [ ] Registered the rule in `rules/registry.py` and added standards mappings + + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7b19d06..058040f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,12 +32,51 @@ If a line genuinely cannot be covered, justify it with `# pragma: no cover` in t ## Adding a security rule -1. Add a `Rule` subclass to the matching thematic module under `src/autonomyproof/rules/` - (`execution.py`, `filesystem.py`, `network.py`, `data.py`, `agent_controls.py`, - `metadata.py`), or create a new module for a new theme. One class per rule ID. -2. Register it in `rules/registry.py`. -3. Add a positive fixture (triggers) and a negative fixture (does not) under `tests/`. -4. Document the rule ID, severity, and framework mappings. +This is the best first contribution โ€” small, self-contained, and objectively gradeable by +the benchmark. Looking for something to build? See [ROADMAP.md](ROADMAP.md) for a wishlist of +`good first issue` rules. + +**The bar:** a rule must add **zero false positives** on the real-repo benchmark. A detection +that can't stay clean on real code doesn't ship โ€” that discipline is the whole point. + +### The six-step loop + +1. **Write the rule.** Add a `Rule` subclass to the matching module under + `src/autonomyproof/rules/` (`execution.py`, `filesystem.py`, `network.py`, `data.py`, + `agent_controls.py`, `metadata.py`, `harness.py`), or a new module for a new theme. One + class per rule ID. Skeleton: + + ```python + class MyNewRule(Rule): + """AG034 โ€” one-line summary.""" + + id = "AG034" + name = "Human-readable name" + default_severity = Severity.HIGH + description = "What the rule detects." + risk = "Why it is dangerous." + remediation = ["Do X", "Prefer Y"] + mappings = Mappings(owaspAgentic=["Excessive agency"], mitre=["T1485"]) + + def check(self, ctx: RuleContext) -> Iterable[Finding]: + for call in ctx.analysis.calls: + if ctx.analysis.resolve_call(call) == "some.dangerous.sink": + yield self.make_finding(ctx, call, evidence="what was found") + ``` + + Use `ctx.tool_functions` to scope a rule to registered agent tools, and the shared helpers + in `astutils.py` / `rules/sources.py` (import resolution, source classification) โ€” that's + how rules stay zero-FP. Read a rule near your theme first; they're short. + +2. **Register it** in `rules/registry.py` (import + add to the ID-ordered list). +3. **Unit tests** โ€” a **positive** fixture (fires) and at least one **negative** (a benign + look-alike that must not fire) in the matching `tests/test_rules_*.py`. Use the `run_rule` + helper. Coverage is 100% and enforced. +4. **Ground-truth cases** โ€” add a positive and a negative to `benchmark/corpus.yaml`. These + are enforced in CI (`tests/test_corpus.py`): precision and recall must stay 1.000. +5. **Prove zero-FP on real code** โ€” run `python benchmark/run.py` and confirm your rule adds + no false positives across the 41 repos. Include the count in your PR. +6. **Run the gates** โ€” `pytest`, `ruff check .`, `ruff format .`, `mypy`. All green. ## Reporting a security vulnerability diff --git a/README.md b/README.md index aa5b9b0..97aba1d 100644 --- a/README.md +++ b/README.md @@ -187,7 +187,18 @@ pytest # runs the suite and enforces 100% branch coverage ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md). Commits must be signed off (DCO). Apache-2.0. +**New detections are the best first contribution.** Each rule is small and self-contained โ€” +one class, a positive/negative test, and a benchmark case โ€” and the real-repo benchmark tells +you objectively whether it holds up: **it must add zero false positives on 41 real repos.** +That's the whole game, and it makes for a satisfying, well-scoped PR. + +- ๐Ÿ”Ž **Pick a rule to build** from the [wishlist in ROADMAP.md](ROADMAP.md) (`good first issue`). +- ๐Ÿ›ก๏ธ **Propose your own** via the [rule proposal form](https://github.com/autonomyproof/autonomyproof-cli/issues/new?template=propose-a-rule.yml). +- ๐ŸŽฏ **Found a false positive?** [Report it](https://github.com/autonomyproof/autonomyproof-cli/issues/new?template=false-positive.yml) โ€” accuracy reports directly drive the backlog. +- ๐Ÿ’ฌ **Questions or ideas?** Open a [Discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions). + +The full six-step "add a rule" walkthrough is in [CONTRIBUTING.md](CONTRIBUTING.md#adding-a-security-rule). +Commits must be signed off (`git commit -s`, DCO). Apache-2.0. ## The rest of the platform diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..c0fa7ee --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,51 @@ +# Roadmap & rule wishlist + +AutonomyProof detects the **authority an AI agent holds in code** โ€” can it move money, run +shell, wipe data, reach any URL, rewrite its own guardrails โ€” and fails the PR that quietly +grants new dangerous authority. The engine ships ~33 rules today; the list below is where +we'd love help. + +## Why contributing a rule is a great first PR + +Every rule is small and self-contained. Adding one is a tight, well-scoped loop: + +1. One `Rule` subclass in the matching module under `src/autonomyproof/rules/`. +2. A **positive** test (it fires) and a **negative** test (it doesn't). +3. A positive + negative case in `benchmark/corpus.yaml`. +4. Run `python benchmark/run.py` โ€” your rule must add **zero false positives** on 41 real repos. + +That last step is the whole game: **a detection that can't stay clean on real code doesn't +ship.** The benchmark tells you objectively whether your rule is good โ€” no bikeshedding. +Full walkthrough in [CONTRIBUTING.md](CONTRIBUTING.md#adding-a-security-rule). + +## Wanted rules (`help wanted` / `good first issue`) + +Each of these is a bite-sized, zero-FP-friendly detection. Pick one, open an issue with the +[rule proposal form](.github/ISSUE_TEMPLATE/propose-a-rule.yml), and go. + +| Idea | Pattern to detect | Why it stays zero-FP | +|---|---|---| +| **Cloud resource destruction** | An agent tool calling `boto3` / GCP / Azure delete/terminate (`delete_bucket`, `terminate_instances`, `delete_*`) | Match specific unambiguous SDK method names, scoped to tool functions (the cloud analog of AG033) | +| **Kubernetes destruction** | `delete_namespaced_*`, `delete_collection_*` from the k8s client inside a tool | Method names are specific; tool-scoped | +| **Money movement without approval** | Stripe/PayPal `Refund.create`, `Transfer.create`, `Payout.create` in a tool with no approval gate | Named SDK calls; reuse the AG007 approval-marker suppression | +| **Repo mutation by code agents** | `git push --force`, GitPython `repo.push()`, force-push via subprocess in a tool | Specific verbs; tool-scoped | +| **Persistence via dotfiles** | Writing to `~/.ssh/authorized_keys`, crontab, `.bashrc`, systemd units | Specific target paths only | +| **Supply-chain RCE** | Download โ†’ deserialize: a URL fetched then passed to `pickle`/`torch.load`/`joblib.load` | Requires both a network source and a deser sink in one flow | +| **Runtime package install** | `pip install` / `npm install` of a model-controlled package via subprocess | shell/subprocess + install verb + non-constant arg | +| **Full-environment passthrough** | `subprocess(..., env=os.environ)` feeding a shell/interpreter tool | Exact kwarg shape | +| **Secret read-and-return** | A tool that reads `.env`/keyring/credential files and returns the value to the model | Credential-path read (AG004) whose value reaches a `return` | +| **World-writable permissions** | `os.chmod(path, 0o777)` / overly-permissive modes | Literal mode check | +| **Model-controlled file write** | `open(path, "w")` / `Path(path).write_*` on a model-controlled path inside a tool | The write analog of AG003; tool-scoped + taint-classified | + +## Bigger pieces (discussion first) + +- **Cross-function taint** โ€” today source-tracking is single-function; whole-program taint + would close the AG005/AG012 config-vs-model gap noted in the benchmark. Start a + [discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions) before diving in. +- **More version-validated CVEs** โ€” extend `cve.py`'s registry with new agent-framework + advisories (version ranges verified). Each is a small, high-signal addition. +- **JavaScript/TypeScript agents** โ€” the rule model generalizes; a TS front-end is a large + but high-impact effort. + +Not sure where to start? Open a [discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions) +and we'll help you scope something.