Make `audit-docs` safe under Codex while keeping Copilot and Claude Code behaviour by AlexJSully · Pull Request #605 · AlexJSully/AlexJSully-Portfolio

AlexJSully · 2026-09-27T23:02:50Z

Makes the audit-docs skill and prompt behave under OpenAI Codex (CLI and IDE extension) without changing how they run under GitHub Copilot and Claude Code, which remain the primary targets.

Why

Run by a GPT model in Codex, audit-docs deleted private Go helper methods, deleted a Markdown file and re-created it under the wrong name with content missing, and wrote overly long comments in a TypeScript file. The same model family under Copilot did not do this. Reading the Codex, Copilot, and OpenAI cookbook sources points to a combination of harness and prompt:

The edit tool is not the difference. Copilot gives GPT models the same apply_patch tool that Codex does.
Codex forces the skill on. Its skill catalogue tells the model it "must use that skill" whenever the task matches a skill's description. The description matches Markdown edits and finished features, so ordinary Codex tasks pulled in the full audit.
Codex has an open bug that matches the file symptom. In apply_patch can delete without visible Guardian review and report Delete+Add as Add-only openai/codex#34515, large rewrites become one patch that deletes and re-adds a file. Codex also applies edits directly, with no Keep/Undo review.
The prompt invited large rewrites. Checklist items applied to every file the run read, splitting a page never said the original keeps its path, and new comments had no length cap. GPT models follow such text literally.

Key changes

Every rule below is worded identically in the prompt half and the skill half, and says nothing host-specific, so Copilot and Claude Code get the same rules.

Edit in place, for every file type. Smallest targeted edits only. Never delete-and-recreate, overwrite, rename, or move a file, or write one through the shell. In source files only comment lines change, and every declaration stays, private helpers included. No formatters or auto-fixers, and no code changes to answer a build or lint failure.
Diff-locality checklist item. Before finishing, the run confirms each file's diff is local: comment lines only in source files, and only the reported passages in documents.
Phase 3 stays in the requested scope. A named directory means every file in it. A pull request means the changed files plus files their changed lines reference directly, one step out, never across a package or workspace. A file opened only to verify a claim is never edited. The fourth scope rung is now "the files the surrounding task named or edited".
Splitting keeps the original file at its path as the overview, and only the depth moves to a new file (report template updated to match).
Documentation comments are one sentence by default and follow the language's own conventions, such as Go's identifier-first style and TSDoc's summary before tags. A new tag adds no type the signature already declares.
Checklist items are limited to files in scope, and absolute wording ("MUST", "No exceptions", "zero tolerance") is softened. No rule is removed.
Codex only: agents/openai.yaml sets policy.allow_implicit_invocation: false, so Codex loads the skill only when it is invoked as $audit-docs. Copilot and Claude Code do not read this file, and claude plugin validate passes with it present.
Supporting docs: skill README.md, .github/prompts/readme.md scope defaults, .claude/rules/prompt-skill-sync.md, and the BUNDLE_DIRS comment in the publishability checker.

The prompt stays under its 36,000-character budget by condensing text it repeated. The prompt-skill-sync subagent confirmed the halves still match and restored one table rule that was missing from the prompt before this PR.

Validation

make -f .claude/Makefile check-skills: pass. make -f .claude/Makefile test-scripts: 104/104 pass.
prettier, eslint, tsc, test:jest (92/92), build, lint:markdown, and their check variants: all pass.
test:cypress:e2e was not run: the Cypress binary is not installed in the cloud container.
Codex behaviour itself could not be run here. To verify it, run $audit-docs in Codex on a fixture containing a Go file with public and private functions, a TypeScript file with undocumented exports, and a Markdown page mixing overview and reference. Expect no deleted or renamed files, local Markdown hunks only, comment-only Go diffs, and new comments of at most two sentences.

Note

Note from @AlexJSully

This PR is purely AI generated. I'm testing out Claude Code cloud sessions with improving the audit-docs skill.

Adds an edit-in-place rule and a diff-locality check covering Markdown and source files, narrows Phase 3 to the requested scope (one step out on a pull request, never across a package or workspace), keeps a split document at its path, caps new documentation comments at one sentence by default, and bounds checklist items to files in scope. Adds a Codex `agents/openai.yaml` that turns off implicit invocation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L7J7gWb6rCBh4Nd4vfKBxR

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L7J7gWb6rCBh4Nd4vfKBxR

vercel · 2026-09-27T23:02:56Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
portfolio	Ready	Preview	Sep 27, 2026 11:16pm UTC

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved moderate issues affect documentation coverage, fragment-link preservation, procedure counting, and synchronization checks.

Review effort: Lite
Findings: 1

Open (1)

Correct procedure count to exclude metadata YAML · New

What changed in this PR

Refines audit-docs scope, editing safeguards, documentation guidance, and Codex invocation metadata.

Changes:

Clarifies scope boundaries and pull-request audit coverage.
Requires minimal in-place edits and improves documentation-comment guidance.
Updates supporting documentation, templates, agents, validation notes, and Codex metadata.

File	Summary	Review findings
`.github/prompts/readme.md`	Documents refined scope behavior.	No final comments.
`.github/prompts/audit-docs.prompt.md`	Updates audit, editing, and documentation rules.	Moderate: allow docstrings and preserve or update fragment links (1 vote each).
`.claude/skills/audit-docs/SKILL.md`	Mirrors refined skill rules.	Moderate: allow docstrings and preserve or update fragment links (1 vote each).
`.claude/skills/audit-docs/README.md`	Updates usage and scope guidance.	No final comments.
`.claude/skills/audit-docs/assets/audit-report.template.md`	Updates split-document reporting.	No final comments.
`.claude/skills/audit-docs/agents/surface-auditor.md`	Clarifies comment expectations.	No final comments.
`.claude/skills/audit-docs/agents/openai.yaml`	Disables implicit Codex invocation.	Moderate: update the procedure count to exclude metadata (2 votes).
`.claude/scripts/check-skill-publishability.mjs`	Documents Codex metadata handling.	No final comments.
`.claude/rules/prompt-skill-sync.md`	Documents Codex integration.	Moderate: include `agents/openai.yaml` in synchronization path globs (1 vote).

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

AlexJSully · 2026-09-27T23:10:50Z


 A skill directory is a plugin root in two ways. In the repository holding it, a `.claude-plugin/plugin.json` inside it makes Claude Code load it as `<name>@skills-dir` on the next session, with no marketplace and no install step, and that manifest is what turns the files in `agents/` into agents a run can delegate to. Everywhere else, the marketplace at [`.claude-plugin/marketplace.json`](../../.claude-plugin/marketplace.json) lists it with a `source` of `./.claude/skills/<name>`, and VS Code and Claude Code install the directory as a plugin whose root `SKILL.md` is its one skill and whose `agents/` they find by default, manifest or not. Without either route the files in `agents/` stay ordinary files, which is what each `SKILL.md` already treats as the default when it tells the run to open one and follow it: `agents/` is a host extension, not part of the Agent Skills specification, which defines `references/`, `assets/`, and `scripts/` and nothing else.

+**Codex reads one file in `agents/`, and it is not an agent.** Codex treats no `agents/*.md` as an agent definition, and reads only `agents/openai.yaml`, for display metadata and invocation policy. [`audit-docs`](../skills/audit-docs/agents/openai.yaml) sets `policy.allow_implicit_invocation: false` there. Codex's skill catalogue tells the model it must use any skill whose description matches the task, and a description as broad as that skill's would otherwise pull a full audit, with its in-code pass, into ordinary feature work. The skill still runs when invoked by name. Claude Code documents an agent as a Markdown file, and `claude plugin validate` passes with the YAML file present.


"Codex reads one file in agents/" Source please? Where did you get this from?

It comes from the openai/codex source on main. OpenAI's docs site was blocked from the session, so I did not use the published docs:

Codex reads agents/openai.yaml: the skill loader looks up exactly that folder and file name and parses its interface, dependencies, and policy keys. See codex-rs/ext/skills/src/loader/metadata.rs and the schema in openai_yaml.md.

Codex does not treat agents/*.md as agents: Codex agent roles come only from .toml files under a config folder's agents/ (such as ~/.codex/agents or .codex/agents) and from [agents.<name>] tables in config.toml. See codex-rs/agent-roles/src/loader.rs and discovery.rs. I found no code in the skill loader that reads any other file under a skill's agents/.

"Reads only" is therefore a statement about the current source, not a documented guarantee. I'll add these links to the paragraph in the next push so the claim carries its own citation.

Generated by Claude Code

AlexJSully · 2026-09-27T23:11:09Z

@@ -0,0 +1,3 @@
+# Read by Codex alone: the skill loads when invoked by name as $audit-docs, never on a description match.
+policy:
+    allow_implicit_invocation: false


What does allow_implicit_invocation do and why is it set to false?

allow_implicit_invocation controls whether Codex puts the skill into the model's context automatically when a task matches its description. Codex's own schema describes it like this: when false, the skill "is not injected into the model context by default, but can still be invoked explicitly via $skill. Defaults to true." (openai_yaml.md in openai/codex, parsed in loader/metadata.rs).

It is set to false for this reason. With implicit invocation on, Codex's skill catalogue prompt tells the model it "must use that skill for that turn" whenever the task "clearly matches a skill's description". The audit-docs description says to use it "when creating or editing Markdown or docs, after implementing a feature, before merging a pull request", which matches most Codex feature work. So an ordinary task could pull in the full three-phase audit, including the in-code comment pass. Turning it off means the audit runs in Codex only when asked for with $audit-docs.

Only Codex reads this file. Copilot and Claude Code ignore it, so their automatic loading is unchanged. If you'd rather Codex keep loading it automatically, deleting the file restores the default.

Generated by Claude Code

Allows the project's own formatter on edited files and in tables, exempts requested code changes and docstrings from the diff check, covers links to headings a split moves, counts only Markdown files as bundled procedures, and cites the Codex sources for `agents/openai.yaml`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

AlexJSully · 2026-09-27T23:20:27Z

Closing this PR because I do NOT like the fact that Claude Code is posting comments & PR using my alias AlexJSully. Example: #605 (comment) .

All Generated by [Claude Code](https://claude.ai/code) will be banned going forward. PRs and thread responses MUST be written by a human.

claude added 2 commits September 27, 2026 20:20

Carry the table re-padding rule into the audit-docs prompt

d255a85

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L7J7gWb6rCBh4Nd4vfKBxR

AlexJSully requested a lite review from Copilot September 27, 2026 23:05

AlexJSully self-assigned this Sep 27, 2026

Copilot started reviewing on behalf of AlexJSully September 27, 2026 23:06 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

Comment thread .claude/skills/audit-docs/agents/openai.yaml

AlexJSully commented Sep 27, 2026

View reviewed changes

AlexJSully changed the title ~~Refine audit-docs scope rules and in-code documentation guidance~~ Make audit-docs safe under Codex while keeping Copilot and Claude Code behaviour Sep 27, 2026

vercel Bot deployed to Preview September 27, 2026 23:16 View deployment

AlexJSully closed this Sep 27, 2026

AlexJSully deleted the claude/brave-bohr-7dvq0u branch October 4, 2026 19:42

Navigation Menu

Uh oh!

Make `audit-docs` safe under Codex while keeping Copilot and Claude Code behaviour - #605

Make `audit-docs` safe under Codex while keeping Copilot and Claude Code behaviour#605
AlexJSully wants to merge 3 commits into
mainfrom
claude/brave-bohr-7dvq0u

AlexJSully commented Sep 27, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Sep 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

AlexJSully Sep 27, 2026

Uh oh!

AlexJSully Sep 27, 2026

Uh oh!

AlexJSully Sep 27, 2026

Uh oh!

AlexJSully Sep 27, 2026

Uh oh!

AlexJSully commented Sep 27, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		A skill directory is a plugin root in two ways. In the repository holding it, a `.claude-plugin/plugin.json` inside it makes Claude Code load it as `<name>@skills-dir` on the next session, with no marketplace and no install step, and that manifest is what turns the files in `agents/` into agents a run can delegate to. Everywhere else, the marketplace at [`.claude-plugin/marketplace.json`](../../.claude-plugin/marketplace.json) lists it with a `source` of `./.claude/skills/<name>`, and VS Code and Claude Code install the directory as a plugin whose root `SKILL.md` is its one skill and whose `agents/` they find by default, manifest or not. Without either route the files in `agents/` stay ordinary files, which is what each `SKILL.md` already treats as the default when it tells the run to open one and follow it: `agents/` is a host extension, not part of the Agent Skills specification, which defines `references/`, `assets/`, and `scripts/` and nothing else.

		Codex reads one file in `agents/`, and it is not an agent. Codex treats no `agents/*.md` as an agent definition, and reads only `agents/openai.yaml`, for display metadata and invocation policy. [`audit-docs`](../skills/audit-docs/agents/openai.yaml) sets `policy.allow_implicit_invocation: false` there. Codex's skill catalogue tells the model it must use any skill whose description matches the task, and a description as broad as that skill's would otherwise pull a full audit, with its in-code pass, into ordinary feature work. The skill still runs when invoked by name. Claude Code documents an agent as a Markdown file, and `claude plugin validate` passes with the YAML file present.

Uh oh!

Conversation

AlexJSully commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Key changes

Validation

Uh oh!

vercel Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

AlexJSully Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

AlexJSully Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

AlexJSully Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

AlexJSully Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

AlexJSully commented Sep 27, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexJSully commented Sep 27, 2026 •

edited

Loading

vercel Bot commented Sep 27, 2026 •

edited

Loading