feat(agent-builder): author, save, and dry-run custom tools by dmarticus · Pull Request #3237 · PostHog/code

dmarticus · 2026-07-07T16:20:55Z

What

Wires the backend's custom-tool authoring loop (posthog#66584) into PostHog Code. Until now the agent builder could only read tool source (tools/<id>/source.ts, read-only) — this adds the client + GUI to edit, save, and dry-run an existing custom tool end to end, behind the existing agent-platform flag.

Scope: this edits tools already declared in the spec — there's no create-a-new-tool or delete-from-UI flow yet, and it edits source.ts only (description / args_schema are round-tripped unchanged).

How

Part A — API client + types (hand-written; agent-platform types are not generated from OpenAPI)

@posthog/shared/agent-platform-types: ToolCompileError/ToolCompileErrorKind, ToolCapabilities, WriteToolRequest/WriteToolResult, DryRunToolRequest/DryRunToolEnvelope/DryRunToolResult.
posthog-client.ts:
- putRevisionTool — 422 returns a typed compile-failed result (rendered inline); other non-2xx (400/409 sealed/…) still throw.
- deleteRevisionTool — 404 (tool_not_found) treated as success.
- dryRunRevisionTool — reads ok from the body (a throwing tool is HTTP 200 with ok:false); 200 and 500 both return the envelope; 429 → throttled (with optional max_concurrent), 503 → unavailable. Never thrown, never retried.
- parseFailedRequest helper recovers status + body from the shared fetcher's Failed request: [status] body throw.

Part B — hooks

useSaveRevisionTool (invalidates the bundle(...) query only when the compile persisted), useDeleteRevisionTool (invalidates bundle), useDryRunRevisionTool (no invalidation). All expose isPending.

Part C — UI (agent builder)

ToolSourcePanel — read-first: the source is a collapsed, read-only block at rest; Edit source (draft-only) and Test tool are explicit toggles, so a tool node stays as compact as every other config node.
ToolSourceEditor (behind the edit toggle) — compile-on-save with inline ToolCompileError diagnostics (kind + message + 1-based line/column), Revert, double-submit guard, and a secret_refs / dynamic_secret_refs readout from the save response.
ToolDryRunPanel (behind the test toggle) — args JSON editor, optional mock_secrets key/value rows, and a result area handling success (result + duration_ms), tool failure (error.code/message), 429 busy, and 503 unavailable.
Everything is gated behind AGENT_PLATFORM_FLAG; non-draft / non-custom tools stay read-only.

Testing

pnpm typecheck (23/23 packages) and pnpm biome check on all touched files pass.
New client tests: 200/422/409 writes, delete 404-as-success, dry-run 200-ok / 200-fail / 500 / 429 (with and without max_concurrent) / 503 + mock_secrets passthrough. New hook test for the invalidate-only-when-persisted branch. The @posthog/api-client and @posthog/ui suites pass.

Note

End-to-end dry-run requires the deployed backend (janitor + main app) to include posthog#66584 with the sandbox backend configured. Until then dry-run returns 503, which is the expected "not available here" state this UI is built against.

🤖 Generated with Claude Code

Wire the backend's custom-tool authoring loop (posthog#66584) into PostHog Code so an internal user can create, save, and dry-run a custom tool end to end, behind the existing agent-platform feature flag. v0: correctness over polish. API client + types (hand-written; agent-platform types are not generated): - Add ToolCompileError(Kind), ToolCapabilities, WriteToolRequest/Result, DryRunToolRequest/Envelope/Result to @posthog/shared. - Add putRevisionTool (422 -> typed compile-failed result, inline; other non-2xx throw), deleteRevisionTool (404 -> success), and dryRunRevisionTool (reads `ok` from the body; 200 and 500 return the envelope; 429 -> throttled; 503 -> unavailable; never thrown, never retried). A parseFailedRequest helper recovers status + body from the shared fetcher's throw. Hooks: - useSaveRevisionTool (invalidates the bundle only when the compile persisted), useDeleteRevisionTool, useDryRunRevisionTool. All expose isPending. UI (agent builder): - ToolSourceEditor: draft-only editable source with a compile-on-save, inline ToolCompileError diagnostics (kind + message + 1-based line/column), double-submit guard, and a capabilities readout from the save response. - ToolDryRunPanel: args JSON editor, optional mock_secrets rows, Test button, and a result area handling success / tool failure / 429 busy / 503 unavailable. - Gate edit + dry-run behind AGENT_PLATFORM_FLAG; non-draft/non-custom stays read-only. Tests: client cases for 200/422/409 writes, delete 404-as-success, and dry-run 200-ok/200-fail/500/429/503 + mock_secrets passthrough; a hook test for the invalidate-only-when-persisted branch. Note: end-to-end dry-run needs the deployed backend (posthog#66584) with the sandbox configured; until then it returns 503, the expected "not available here" state. Generated-By: PostHog Code Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba

trunk-io · 2026-07-07T16:21:21Z

Merging to main in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

github-actions · 2026-07-07T16:21:27Z

React Doctor found no issues in the changed files. 🎉

_{Reviewed by React Doctor for commit 0fc4805.}

greptile-apps · 2026-07-07T16:24:28Z

Comments Outside Diff (1)

packages/ui/src/features/agent-applications/components/ToolSourceEditor.tsx, line 982 (link)

Uncleaned timeout on unmount

setTimeout is started but never cleared, so if the component unmounts within the 2-second window (e.g. the user navigates away right after saving), the captured setJustSaved closure fires on a stale closure. React 18 makes this a silent no-op, but the pattern accumulates stale closures. Store the timer ID in a useRef and clear it in a cleanup callback or on the next onSave call.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "feat(agent-builder): author, save, and d..." | Re-trigger Greptile}

…e timer - Make DryRunToolResult.throttled `max_concurrent` optional and omit it on a 429 without a numeric count, so the panel no longer renders a misleading "0 runs in flight". The dry-run panel shows a count only when present. - Clear the "saved" tick setTimeout in ToolSourceEditor on unmount and before the next save (via a ref) so a stale timer can't setState after unmount. - Add a client test for a 429 with no max_concurrent. Generated-By: PostHog Code Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba

- ToolDryRunPanel: give mock-secret rows a stable id (ref counter) and key/ update/remove by it, instead of the array index. - ToolSourceEditor: make the source textarea uncontrolled (defaultValue + ref) and track only a `dirty` flag, dropping the prop-derived useState and the in-render reconcile. The parent keys the editor by revision+tool so switching either remounts it with a fresh buffer. - Key each compile-error row by its kind/line/column/message instead of the array index. Generated-By: PostHog Code Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba

The tool detail view showed an always-open source textarea and an always- expanded dry-run panel on draft tools, which made the node look like a code editor bolted into the builder. Extract a ToolSourcePanel that defaults to a collapsed, read-only view: - Resting state is a `source · <path>` header plus a "Show source" toggle (read-only CodeBlock when expanded) — no editor, no open dry-run panel. - "Edit source" (draft-only) is an explicit toggle that swaps in the inline editor; "Test tool" is an explicit toggle that reveals the dry-run panel, with a "Hide test" to collapse it. - Keyed by revision+tool so switching either resets the toggles and reseeds the editor buffer. ToolSourceEditor/ToolDryRunPanel are unchanged — just moved behind the toggles. Generated-By: PostHog Code Task-Id: 3dc0ab45-e514-4c17-a185-9aa78871a3ba

daniloc

thanks for your patience. Yeah, I know what you mean. it's fine, it works, and also I can't imagine wanting to do the edit/test loop myself at ALL, that's a robot job

so approving with the same faint enthusiasm as the PR itself

greptile-apps Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread packages/api-client/src/posthog-client.ts

dmarticus added 2 commits July 7, 2026 09:38

dmarticus requested a review from a team July 7, 2026 17:10

daniloc approved these changes Jul 8, 2026

View reviewed changes

Merge branch 'main' into posthog-code/custom-tool-authoring-dry-run

0fc4805

dmarticus merged commit 6b6580e into main Jul 9, 2026
26 checks passed

dmarticus deleted the posthog-code/custom-tool-authoring-dry-run branch July 9, 2026 23:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(agent-builder): author, save, and dry-run custom tools#3237

feat(agent-builder): author, save, and dry-run custom tools#3237
dmarticus merged 5 commits into
mainfrom
posthog-code/custom-tool-authoring-dry-run

dmarticus commented Jul 7, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

Uh oh!

daniloc left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dmarticus commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Testing

Note

Uh oh!

trunk-io Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments Outside Diff (1)

Uh oh!

Uh oh!

daniloc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmarticus commented Jul 7, 2026 •

edited

Loading

github-actions Bot commented Jul 7, 2026 •

edited

Loading

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading