perf(cli): load a manual run's draft once and keep embedded CLI results lean by waleedlatif1 · Pull Request #8625 · simstudioai/sim

waleedlatif1 · 2026-10-05T13:43:50Z

Summary

Cuts fixed overhead from the two CLI commands the in-app agent issues most, workflows run --manual and logs get. The public API and the installed CLI behave exactly as before.

The draft is loaded once per manual run. executeManualWorkflowOperation already loads the saved draft to choose and validate the trigger. The execute service and the execution core then each read it again. They now reuse the snapshot the use case loaded, for both the sync and the stream paths and for run-from-block.
Embedded runs return file references, not inline base64. When the CLI runs embedded (the in-app agent's sim_cli), a synchronous or --follow workflows run sends includeFileBase64: false unless the caller passes --include-file-base64. The agent reads produced files by id. The inline bytes only grew its result and cost the server a storage read and an encode per file. --async runs never send the field, because the route rejects it there.
Embedded logs get leaves out the workflow snapshot. A new embeddedRequestDefault field on the CLI contract makes embedded invocations send includeWorkflowState=false unless --include-workflow-state is passed. The agent diagnoses runs from traceSpans, and workflows get serves the block configuration.
New E2E benchmark: apps/sim/scripts/test-cli-run-latency-e2e.ts (bun run test:cli-run-latency:e2e). It seeds a five-block workflow (Start, two Functions, a file write, a file read) into a disposable database and drives both commands through the embedded CLI over real HTTP. It writes each command's stdout to a file, as the outputFile sink does, and records per-segment timings in a JSON report at CLI_LATENCY_E2E_REPORT_PATH. CI runs a short pass (1 warm-up and 3 measured iterations) in the HTTP E2E job's workflow step, after the --stop-after suite and against the same self-hosted app (hosted billing admits runs through Redis, which that job does not provision). With this checkout's CLI, the run asserts that the results stay lean.

Before / after

Setup:

One machine, a production build (next build and next start) of origin/staging against this branch, and each variant's own CLI source.
Interleaved: 8 rounds × 2 variants, in ABBA order. Each round restarts the server, discards 3 warm-up iterations and measures 5, so n = 40 per variant per condition.
The machine was running an unrelated benchmark at the same time, so all runs were niced. Interleaving gives both variants the same load.
Two database conditions:
- Direct loopback Postgres.
- A TCP shim that adds 1 ms per direction (about 2 ms RTT), to approximate a networked database.
Δ is after minus before. The 95% CI comes from 4,000 bootstrap resamples.

With about 2 ms of database RTT (all values in ms):

Command / segment	before p50	after p50	Δp50 [95% CI]	before p90	after p90
`workflows run`: before execution (request → log `started_at`)	192.5	179.0	−13.5 [−31.5, +4.0]	283.0	287.2
`workflows run`: execution	866.0	757.5	−108.5 [−174.0, −37.5]	1031.1	1001.0
`workflows run`: after execution (`ended_at` → last byte)	62.0	52.0	−10.0 [−27.5, 0.0]	158.2	74.2
`workflows run`: CLI render	2.5	0.9	−1.5 [−2.1, −0.8]	4.7	2.1
`workflows run`: sink write	1.1	0.8	−0.2 [−0.4, +0.1]	1.6	1.4
`workflows run`: total	1155.2	1021.5	−133.7 [−196.6, −62.5]	1362.2	1275.0
`workflows run`: stdout	98,122 B	1,568 B	−96,554 B
`logs get`: HTTP	113.4	105.2	−8.2 [−16.9, +6.4]	134.0	126.3
`logs get`: CLI render	3.7	3.6	−0.1 [−0.8, +0.9]	5.8	6.2
`logs get`: total	134.6	125.4	−9.2 [−17.9, +4.5]	161.3	156.8
`logs get`: stdout	336,228 B	329,311 B	−6,917 B
Per run (`workflows run` + `logs get`)	1285.1	1156.6	−128.5 [−208.9, −54.1]	1519.1	1403.7

Direct loopback database (all values in ms; noisier, because the database costs almost nothing and machine load dominates):

Command / segment	before p50	after p50	Δp50 [95% CI]	before p90	after p90
`workflows run`: before execution	88.5	81.0	−7.5 [−21.0, +11.5]	240.9	216.0
`workflows run`: execution	452.5	448.5	−4.0 [−130.0, +116.0]	1925.2	823.9
`workflows run`: after execution	47.0	39.5	−7.5 [−17.5, +2.5]	141.1	104.0
`workflows run`: total	615.4	569.6	−45.8 [−168.2, +142.4]	2411.4	1035.3
`logs get`: total	78.6	68.2	−10.4 [−25.4, +4.9]	188.0	99.5
Per run	714.1	653.3	−60.8 [−171.4, +118.2]	2699.8	1087.0

What the numbers say:

The clear win is the file-reference default.
- The run result shrinks from 98 KB to 1.6 KB, because this workflow's two file outputs each carried 48 KB of base64.
- With a networked database, the run is about 130 ms faster at p50, and the CI excludes zero.
- Most of the saving is inside "execution": the executor also hydrates file outputs when base64 is on.
The single draft load is real but small here.
- "Before execution" drops 7–14 ms at p50, but the CI includes zero at n = 40.
- The fixture has 5 blocks on a local database. Production drafts are larger and sit behind a network.
logs get returns 6.9 KB less, the size of this workflow's snapshot. Its time change is within noise.
- The production logs get cost after the request comes from the remote workbench write of stdout, which this harness cannot reproduce.
- A smaller stdout reduces it but does not remove it.
- Rendering takes about 3.5 ms for 330 KB, so rendering is not the bottleneck.

Reproduce:

CLI_LATENCY_E2E_BASE_URL=http://127.0.0.1:<port> \
CLI_LATENCY_E2E_DATABASE_URL=postgresql://…/<db with a test segment> \
CLI_LATENCY_E2E_REPORT_PATH=/tmp/report.json CLI_LATENCY_E2E_RUNS=30 \
  bun run --cwd apps/sim test:cli-run-latency:e2e

To compare CLI builds, set CLI_LATENCY_E2E_CLI_MODULE to another checkout's packages/sim-cli/src/embed.ts.

Design

Draft state flows from the use case as a trusted, already-authorized value:
- ExecuteWorkflowServiceParams.draftState → ExecuteWorkflowCoreOptions.draftState, and ExecuteWorkflowOptions.draftState on the stream path.
- The service refuses it without useDraftState.
- The core uses it only on the draft branch. Resumed, deployed and override states load exactly as before.
- HTTP callers cannot supply it.
Embedded defaults live in the CLI because the embedding host is the consumer that doesn't need the payloads. The server's defaults are untouched.
- logs get goes through a generic, contract-declared embeddedRequestDefault, which wins over requestDefault only inside an embedded invocation.
- workflows run applies its default in the synchronous request builder (workflow-run-follow.ts), because the same field is a 400 on --async.
- The flag help and the generated CLI docs state the agent's default, so the agent's reference card describes what it actually gets.

Compatibility

Public API: no default changed. includeFileBase64 still defaults to true and includeWorkflowState still defaults to true.
Installed CLI: unchanged. The embedded defaults apply only inside runEmbeddedCli. A test covers the explicit flags.
In-app agent: this is the one behaviour change.
- Run results no longer carry file.base64, and logs get returns workflowState: null unless asked.
- I checked the worker for consumers. It passes argv through without adding these flags, and nothing parses run output for base64 or workflowState.
- Its skills already read files by id (files read) and logs by traceSpans.
- The agent can still opt back in with --include-file-base64 or --include-workflow-state.
Draft reuse:
- Before, the trigger was chosen from load Workflow Zoom #1 and the graph executed from load Connection line re-render on sub-block input #3. A save between the two could make them disagree.
- Now the trigger and the executed graph are the same snapshot.
- The same holds for --stop-after (feat(workflows): stop a manual v2 run after a block, and workflows run --stop-after #8622): run.stopAfterBlockId is validated against the snapshot the run then executes, and it threads through the service and core alongside draftState.
- The shared snapshot is only read: the use case and the core build merged copies with mergeSubblockStateWithValues, and the service reads the blocks only to resolve --select-output.

Test plan

test-cli-run-latency-e2e.ts, 8 interleaved rounds per variant and condition. Every check passed: the run completes, blockOutputs matches, the file reference is present, and the log has trace spans.
New embedded-default tests in packages/sim-cli/src/embed.test.ts, covering sync, --follow, --async, logs get and explicit opt-in. The sync and logs get tests fail on the pre-change CLI.
execute-manual-workflow, execute-service, execution-core, execute-workflow, v2 execute route and csv-workflow-execution tests
bun run lint, bun run type-check, bun run check:audits, bun run docs-manifest:check, block-registry check against origin/staging
Full bun run test via CI (Lint and Test, PostgreSQL integration), plus the new E2E in the HTTP E2E job: all green. Locally, the suite only timed out under unrelated machine load.

vercel · 2026-10-05T13:43:52Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 6, 2026 11:09pm UTC

waleedlatif1 · 2026-10-05T13:43:55Z

@greptile

waleedlatif1 · 2026-10-05T13:43:57Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-05T13:44:02Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 15 files

_{Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-10-05T13:48:45Z

Confidence Score: 4/5

[Medium risk] Optimizes CLI performance by caching draft workflow state.

The PR is not yet safe to merge because the reorganized HTTP job can start a second server before the first one's workers have exited.

Summary

The PR reuses the authorized draft snapshot for manual execution and makes embedded CLI run and log responses smaller while retaining opt-in flags. It also adds an HTTP E2E benchmark and reorganizes the HTTP suites; the newly separated suite has a server-shutdown race.

_{Reviews (6) · Last reviewed commit: "fix(cli): remove the CLI E2E's execution..."}

waleedlatif1 · 2026-10-05T13:58:41Z

@greptile

waleedlatif1 · 2026-10-05T13:58:43Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-05T13:58:50Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-10-05T14:09:49Z

@greptile

waleedlatif1 · 2026-10-05T14:09:51Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-05T14:10:00Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 15 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-10-05T16:36:26Z

@greptile

waleedlatif1 · 2026-10-05T16:36:27Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-05T16:36:35Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-10-05T16:44:24Z

Comments Outside Diff

These findings could not be posted inline.

Servers can overlap .github/workflows/test-build.yml:341 ▶

This step clears .next/dev and starts another server on port 3018 immediately after the CLI step. The CLI step kills and waits for only the parent next dev process, but its workers can outlive it. If a worker is still running, it can hold the port or write to the cache as this step starts, causing the HTTP suite to fail. The CLI step needs the same session-wide shutdown used by the other suites.

waleedlatif1 · 2026-10-05T16:51:05Z

On the 4/5: the stop-after finding is about assertStopAfterBlock and the engine's stop handling, which came in from staging with #8622. This PR does not change them. The PR diff against staging only threads stopAfterBlockId alongside the preloaded draftState, so the target is validated against the same snapshot the run executes. A target on a branch that a router or condition skips at runtime can't be refused statically. Whether that run should report that it never reached the target belongs in a #8622 follow-up, not in this latency change.

waleedlatif1 · 2026-10-05T16:51:06Z

@greptile

…ts lean - Manual runs reuse the draft state the use case loaded to pick and validate the trigger, instead of reading the draft tables twice more in the execute service and the execution core. The trigger is now chosen from the same snapshot that runs. - Embedded (in-app agent) synchronous runs ask for file references without inline base64 unless --include-file-base64 is passed; --async never sends it. - Embedded logs get leaves the workflow snapshot out unless --include-workflow-state is passed, via a new embeddedRequestDefault contract field. The installed CLI and public API defaults are unchanged. - Add test-cli-run-latency-e2e.ts, which times both commands per segment against a running app and writes a JSON report; CI runs a short pass.

…he CLI E2E - Embedded --follow runs also ask for file references only. - The E2E asserts this checkout's embedded results carry no inline file bytes or workflow snapshot; a compared CLI build is measured, not asserted. - Run the CLI E2E against its own self-hosted app: hosted billing admits runs through Redis, which the HTTP E2E job does not provision.

waleedlatif1 · 2026-10-06T23:06:27Z

@greptile

waleedlatif1 · 2026-10-06T23:06:28Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-06T23:06:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

waleedlatif1 requested a review from a team as a code owner October 5, 2026 13:43

vercel Bot deployed to Preview October 5, 2026 13:46 View deployment

cubic-dev-ai Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread packages/sim-cli/src/commands/protocol/workflow-run-follow.ts

cubic-dev-ai Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread apps/sim/scripts/test-cli-run-latency-e2e.ts

Comment thread packages/sim-cli/src/commands/protocol/workflow-run-follow.ts

greptile-apps Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread packages/sim-cli/src/commands/protocol/workflow-run-follow.ts

Comment thread apps/sim/scripts/test-cli-run-latency-e2e.ts Outdated

vercel Bot deployed to Preview October 5, 2026 14:00 View deployment

cubic-dev-ai Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread apps/sim/scripts/test-cli-run-latency-e2e.ts

vercel Bot temporarily deployed to Preview October 5, 2026 14:09 Inactive

cubic-dev-ai Bot reviewed Oct 5, 2026

View reviewed changes

vercel Bot deployed to Preview October 5, 2026 16:39 View deployment

cubic-dev-ai Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread .github/workflows/test-build.yml Outdated

waleedlatif1 added 2 commits October 6, 2026 16:05

fix(cli): remove the CLI E2E's execution logs and snapshots on cleanup

114cf00

waleedlatif1 force-pushed the fix/cli-run-latency branch from 572982d to 114cf00 Compare October 6, 2026 23:06

waleedlatif1 merged commit 5f1a29e into staging Oct 6, 2026
23 of 24 checks passed

waleedlatif1 deleted the fix/cli-run-latency branch October 6, 2026 23:07

waleedlatif1 mentioned this pull request Oct 6, 2026

v0.9.15: performance improvements, cli run until, memory fixes, xhigh thinking #8654

Closed

vercel Bot deployed to Preview October 6, 2026 23:09 View deployment

waleedlatif1 mentioned this pull request Oct 6, 2026

fix(e2e): compile every timed route first in the HTTP suites and stop sessions completely #8701

Merged

6 tasks done

Conversation

waleedlatif1 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Before / after

Design

Compatibility

Test plan

Uh oh!

vercel Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 5, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments Outside Diff

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 5, 2026

Uh oh!

waleedlatif1 commented Oct 6, 2026

Uh oh!

waleedlatif1 commented Oct 6, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 6, 2026

waleedlatif1 commented Oct 5, 2026 •

edited

Loading

vercel Bot commented Oct 5, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 5, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 5, 2026 •

edited

Loading