Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: "🐛 Bug report"
description: Something crashed or behaved incorrectly (not a false positive — use the other form for that).
title: "[bug] "
labels: ["bug"]
body:
- type: textarea
id: what
attributes:
label: What happened?
description: What you ran, what you expected, and what you got.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
placeholder: |
1. autonomyproof scan ./my-agent
2. ...
validations:
required: true
- type: input
id: env
attributes:
label: Version & OS
placeholder: "autonomyproof 0.15.0, Python 3.12, macOS 14"
validations:
required: true
8 changes: 8 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: 💬 Questions & ideas
url: https://github.com/autonomyproof/autonomyproof-cli/discussions
about: Ask a question, share how you use it, or float an idea before opening an issue.
- name: 🔒 Report a security vulnerability
url: https://github.com/autonomyproof/autonomyproof-cli/blob/main/SECURITY.md
about: Please do NOT open a public issue for vulnerabilities — follow the security policy.
36 changes: 36 additions & 0 deletions .github/ISSUE_TEMPLATE/false-positive.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: "🎯 Report a false positive"
description: A rule fired on code that is actually safe. These reports directly drive our accuracy work.
title: "[false-positive] AG0XX on "
labels: ["false-positive"]
body:
- type: markdown
attributes:
value: |
False positives are the enemy — we publish our own FP benchmark and take these seriously.
The clearer the minimal repro, the faster it gets fixed (and it often becomes a corpus case).
- type: input
id: rule
attributes:
label: Which rule fired?
placeholder: "AG005"
validations:
required: true
- type: textarea
id: code
attributes:
label: Minimal code that was flagged (but is safe)
render: python
validations:
required: true
- type: textarea
id: why
attributes:
label: Why is it actually safe?
placeholder: "The URL comes from settings.API_URL, a trusted config constant, not model input."
validations:
required: true
- type: input
id: version
attributes:
label: Version
placeholder: "autonomyproof 0.15.0"
60 changes: 60 additions & 0 deletions .github/ISSUE_TEMPLATE/propose-a-rule.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: "🛡️ Propose a detection rule"
description: Suggest a new authority/guardrail check for the scanner. Great first contribution.
title: "[rule] "
labels: ["rule-proposal", "help wanted"]
body:
- type: markdown
attributes:
value: |
Thanks for helping expand coverage! Every rule is small and self-contained —
one `Rule` subclass + a positive/negative test + a `benchmark/corpus.yaml` case.
See the [Add a rule guide](../blob/main/CONTRIBUTING.md#adding-a-security-rule).
The bar is **zero false positives on the real-repo benchmark** — that's the whole game.
- type: input
id: pattern
attributes:
label: What dangerous pattern should it detect?
description: One sentence — the authority or guardrail gap.
placeholder: "An agent tool that can wipe an entire datastore with no approval."
validations:
required: true
- type: textarea
id: example
attributes:
label: Minimal example that SHOULD fire
description: A few lines of Python. This becomes the positive corpus case.
render: python
placeholder: |
@tool
def reset_db():
Base.metadata.drop_all(engine)
validations:
required: true
- type: textarea
id: negative
attributes:
label: A similar example that must NOT fire (the zero-FP guard)
description: The benign look-alike that keeps the rule honest.
render: python
placeholder: |
@tool
def trim(df):
return df.drop(columns=["x"]) # pandas column drop — benign
validations:
required: true
- type: input
id: frameworks
attributes:
label: Frameworks / libraries involved
placeholder: "SQLAlchemy, redis-py, pymongo"
- type: textarea
id: mappings
attributes:
label: Standards mappings (if known)
placeholder: "OWASP Agentic: Excessive agency · MITRE ATT&CK: T1485 · CVE: —"
- type: checkboxes
id: willing
attributes:
label: Would you like to implement it?
options:
- label: I'd like to open the PR myself (we'll help)
22 changes: 22 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!-- Thanks for contributing! Keep PRs focused — one rule or one fix per PR is ideal. -->

## What this changes

<!-- One or two sentences. Link the issue it closes, e.g. "Closes #123". -->

## Checklist

- [ ] Commits are signed off (`git commit -s`) — required by CI (DCO)
- [ ] `pytest` passes (100% branch coverage is enforced)
- [ ] `ruff check .` and `ruff format --check .` pass
- [ ] `mypy` passes

### If this adds or changes a detection rule

- [ ] Added a **positive** test (the rule fires) and a **negative** test (it does not)
- [ ] Added a positive and negative case to `benchmark/corpus.yaml`
- [ ] Ran `python benchmark/run.py` and confirmed **no new false positives** on the real-repo corpus
- [ ] Registered the rule in `rules/registry.py` and added standards mappings

<!-- The zero-false-positive bar is the whole point of the project. A rule that can't
stay clean on real code won't be merged — and that's a feature, not a rejection. -->
51 changes: 45 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,51 @@ If a line genuinely cannot be covered, justify it with `# pragma: no cover` in t

## Adding a security rule

1. Add a `Rule` subclass to the matching thematic module under `src/autonomyproof/rules/`
(`execution.py`, `filesystem.py`, `network.py`, `data.py`, `agent_controls.py`,
`metadata.py`), or create a new module for a new theme. One class per rule ID.
2. Register it in `rules/registry.py`.
3. Add a positive fixture (triggers) and a negative fixture (does not) under `tests/`.
4. Document the rule ID, severity, and framework mappings.
This is the best first contribution — small, self-contained, and objectively gradeable by
the benchmark. Looking for something to build? See [ROADMAP.md](ROADMAP.md) for a wishlist of
`good first issue` rules.

**The bar:** a rule must add **zero false positives** on the real-repo benchmark. A detection
that can't stay clean on real code doesn't ship — that discipline is the whole point.

### The six-step loop

1. **Write the rule.** Add a `Rule` subclass to the matching module under
`src/autonomyproof/rules/` (`execution.py`, `filesystem.py`, `network.py`, `data.py`,
`agent_controls.py`, `metadata.py`, `harness.py`), or a new module for a new theme. One
class per rule ID. Skeleton:

```python
class MyNewRule(Rule):
"""AG034 — one-line summary."""

id = "AG034"
name = "Human-readable name"
default_severity = Severity.HIGH
description = "What the rule detects."
risk = "Why it is dangerous."
remediation = ["Do X", "Prefer Y"]
mappings = Mappings(owaspAgentic=["Excessive agency"], mitre=["T1485"])

def check(self, ctx: RuleContext) -> Iterable[Finding]:
for call in ctx.analysis.calls:
if ctx.analysis.resolve_call(call) == "some.dangerous.sink":
yield self.make_finding(ctx, call, evidence="what was found")
```

Use `ctx.tool_functions` to scope a rule to registered agent tools, and the shared helpers
in `astutils.py` / `rules/sources.py` (import resolution, source classification) — that's
how rules stay zero-FP. Read a rule near your theme first; they're short.

2. **Register it** in `rules/registry.py` (import + add to the ID-ordered list).
3. **Unit tests** — a **positive** fixture (fires) and at least one **negative** (a benign
look-alike that must not fire) in the matching `tests/test_rules_*.py`. Use the `run_rule`
helper. Coverage is 100% and enforced.
4. **Ground-truth cases** — add a positive and a negative to `benchmark/corpus.yaml`. These
are enforced in CI (`tests/test_corpus.py`): precision and recall must stay 1.000.
5. **Prove zero-FP on real code** — run `python benchmark/run.py` and confirm your rule adds
no false positives across the 41 repos. Include the count in your PR.
6. **Run the gates** — `pytest`, `ruff check .`, `ruff format .`, `mypy`. All green.

## Reporting a security vulnerability

Expand Down
13 changes: 12 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,18 @@ pytest # runs the suite and enforces 100% branch coverage

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). Commits must be signed off (DCO). Apache-2.0.
**New detections are the best first contribution.** Each rule is small and self-contained —
one class, a positive/negative test, and a benchmark case — and the real-repo benchmark tells
you objectively whether it holds up: **it must add zero false positives on 41 real repos.**
That's the whole game, and it makes for a satisfying, well-scoped PR.

- 🔎 **Pick a rule to build** from the [wishlist in ROADMAP.md](ROADMAP.md) (`good first issue`).
- 🛡️ **Propose your own** via the [rule proposal form](https://github.com/autonomyproof/autonomyproof-cli/issues/new?template=propose-a-rule.yml).
- 🎯 **Found a false positive?** [Report it](https://github.com/autonomyproof/autonomyproof-cli/issues/new?template=false-positive.yml) — accuracy reports directly drive the backlog.
- 💬 **Questions or ideas?** Open a [Discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions).

The full six-step "add a rule" walkthrough is in [CONTRIBUTING.md](CONTRIBUTING.md#adding-a-security-rule).
Commits must be signed off (`git commit -s`, DCO). Apache-2.0.

## The rest of the platform

Expand Down
51 changes: 51 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Roadmap & rule wishlist

AutonomyProof detects the **authority an AI agent holds in code** — can it move money, run
shell, wipe data, reach any URL, rewrite its own guardrails — and fails the PR that quietly
grants new dangerous authority. The engine ships ~33 rules today; the list below is where
we'd love help.

## Why contributing a rule is a great first PR

Every rule is small and self-contained. Adding one is a tight, well-scoped loop:

1. One `Rule` subclass in the matching module under `src/autonomyproof/rules/`.
2. A **positive** test (it fires) and a **negative** test (it doesn't).
3. A positive + negative case in `benchmark/corpus.yaml`.
4. Run `python benchmark/run.py` — your rule must add **zero false positives** on 41 real repos.

That last step is the whole game: **a detection that can't stay clean on real code doesn't
ship.** The benchmark tells you objectively whether your rule is good — no bikeshedding.
Full walkthrough in [CONTRIBUTING.md](CONTRIBUTING.md#adding-a-security-rule).

## Wanted rules (`help wanted` / `good first issue`)

Each of these is a bite-sized, zero-FP-friendly detection. Pick one, open an issue with the
[rule proposal form](.github/ISSUE_TEMPLATE/propose-a-rule.yml), and go.

| Idea | Pattern to detect | Why it stays zero-FP |
|---|---|---|
| **Cloud resource destruction** | An agent tool calling `boto3` / GCP / Azure delete/terminate (`delete_bucket`, `terminate_instances`, `delete_*`) | Match specific unambiguous SDK method names, scoped to tool functions (the cloud analog of AG033) |
| **Kubernetes destruction** | `delete_namespaced_*`, `delete_collection_*` from the k8s client inside a tool | Method names are specific; tool-scoped |
| **Money movement without approval** | Stripe/PayPal `Refund.create`, `Transfer.create`, `Payout.create` in a tool with no approval gate | Named SDK calls; reuse the AG007 approval-marker suppression |
| **Repo mutation by code agents** | `git push --force`, GitPython `repo.push()`, force-push via subprocess in a tool | Specific verbs; tool-scoped |
| **Persistence via dotfiles** | Writing to `~/.ssh/authorized_keys`, crontab, `.bashrc`, systemd units | Specific target paths only |
| **Supply-chain RCE** | Download → deserialize: a URL fetched then passed to `pickle`/`torch.load`/`joblib.load` | Requires both a network source and a deser sink in one flow |
| **Runtime package install** | `pip install` / `npm install` of a model-controlled package via subprocess | shell/subprocess + install verb + non-constant arg |
| **Full-environment passthrough** | `subprocess(..., env=os.environ)` feeding a shell/interpreter tool | Exact kwarg shape |
| **Secret read-and-return** | A tool that reads `.env`/keyring/credential files and returns the value to the model | Credential-path read (AG004) whose value reaches a `return` |
| **World-writable permissions** | `os.chmod(path, 0o777)` / overly-permissive modes | Literal mode check |
| **Model-controlled file write** | `open(path, "w")` / `Path(path).write_*` on a model-controlled path inside a tool | The write analog of AG003; tool-scoped + taint-classified |

## Bigger pieces (discussion first)

- **Cross-function taint** — today source-tracking is single-function; whole-program taint
would close the AG005/AG012 config-vs-model gap noted in the benchmark. Start a
[discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions) before diving in.
- **More version-validated CVEs** — extend `cve.py`'s registry with new agent-framework
advisories (version ranges verified). Each is a small, high-signal addition.
- **JavaScript/TypeScript agents** — the rule model generalizes; a TS front-end is a large
but high-impact effort.

Not sure where to start? Open a [discussion](https://github.com/autonomyproof/autonomyproof-cli/discussions)
and we'll help you scope something.
Loading