fix(core): use effective step timeout for stalled tasks

[jumski]

Summary

Fix stalled-task recovery to use the effective step timeout instead of always using the flow timeout.

This PR stacks directly on #663 (09-01-issue_645_failed_run_terminalization) and preserves its run, step-state, and task eligibility guards.

Root cause

start_tasks() sets PGMQ visibility from the effective timeout:

coalesce(step.opt_timeout, flow.opt_timeout) + 2

requeue_stalled_tasks() used only flows.opt_timeout. A short step timeout could therefore make the PGMQ message visible while its task row remained started until the longer flow timeout and recovery buffer expired.

Behavior

Recovery now requires:

started_at < now()
  - (coalesce(step.opt_timeout, flow.opt_timeout) * interval '1 second')
  - interval '30 seconds'

The comparison stays strict. A null step timeout inherits the non-null flow timeout.

The PGMQ-only two-second margin is not added to recovery. Adding it again would change the existing 30-second recovery grace to 32 seconds. The 15-second cron cadence can add up to roughly 15 seconds after eligibility.

The change preserves:

• run.status = 'started', step_state.status = 'started', and task.status = 'started';
• permanently_stalled_at is null and FOR UPDATE OF task SKIP LOCKED behavior;
• attempts and requeue counters;
• three successful requeues before permanent stall;
• immediate visibility through set_vt_batch(..., 0);
• archive and permanent-stall behavior.

Tests

Added effective_step_timeout.test.sql with deterministic timestamps inside one transaction:

• flow 60 / step 5: exactly 35 seconds stays started; 36 seconds requeues;
• flow 5 / step 60: 36 seconds stays started; 91 seconds requeues;
• flow 5 / null step timeout: 36 seconds requeues through flow fallback.

Before the source fix, the focused test failed 5 of 10 assertions for the expected reason. The short step override returned 0 and stayed started at 36 seconds. The long step override requeued at 36 seconds, so its later 91-second call returned 0.

After the source fix:

• focused test: 1 file, 10 tests, pass;
• all stalled-recovery tests: 6 files, 54 tests, pass;
• full pgTAP: 285 files, 1326 tests, pass.

Migration and release note

Atlas generated 20260901203454_pgflow_temp_effective_step_timeout.sql. It replaces pgflow.requeue_stalled_tasks(), includes the cumulative #645 guards, and performs no backfill.

Added a separate patch changeset for @pgflow/core. The fixed release group expands the patch at release time.

Checks

• pnpm nx verify-migrations core --skip-nx-cache — pass
• pnpm nx gen-types core --skip-nx-cache — pass; no generated type diff
• pnpm nx verify-gen-types core --skip-nx-cache — pass
• pnpm nx test:pgtap core --skip-nx-cache — pass; 285 files, 1326 tests
• pnpm nx test core --skip-nx-cache — pass
• pnpm nx lint core --skip-nx-cache — pass; 0 errors and 2 existing type-test warnings
• pnpm nx build core --skip-nx-cache — pass
• pnpm changeset status — pass; patch fixed group detected
• git diff --check — pass

pnpm nx fix-sql core hit the known Sqruff CLI mismatch: error: unexpected argument '--force' found. The direct repository fallback, sqruff --config=.sqruff fix --parsing-errors pkgs/core/schemas/, processed 37 files and found nothing to fix. Direct Sqruff lint also passed.

Two fresh independent Sol xhigh review rounds returned APPROVED with no required findings.

Out of scope

• Make start_tasks() apply task visibility before handlers run #656 and execution of the unreferenced start_tasks() visibility CTE;
• Use terminal events to cooperatively abort in-flight worker tasks #646 worker-side handler cancellation;
• queue identity or per-step queue routing;
• changes to Failed runs leave unfinished sibling tasks queued or started instead of cancelled #645 cancellation semantics or parent-state guards;
• configurable recovery buffers or cron cadence;
• migration consolidation and release PR Version Packages #659.

The stack still contains temporary migrations, so the main-targeted temporary-migration check can fail until the settled release sequence consolidates them. This PR does not consolidate the parent migration.

Fixes #621

[changeset-bot]

🦋 Changeset detected

Latest commit: cb52bb5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@pgflow/core	Patch
pgflow	Patch
@pgflow/client	Patch
@pgflow/edge-worker	Patch
@pgflow/dsl	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jumski]

• fix(core): use effective step timeout for stalled tasks #664 👈 (View in Graphite)
• fix(core): terminalize unfinished tasks when runs fail #663
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[nx-cloud]

View your CI Pipeline Execution ↗ for commit cb52bb5

Command	Status	Duration	Result
nx run edge-worker:test:integration	✅ Succeeded	5m 49s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

---

☁️ Nx Cloud last updated this comment at 2026-09-02 02:09:49 UTC

[github-actions]

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-664.pgflow.pages.dev

📝 Details:

• Branch: 09-02-issue_621_effective_step_timeout
• Commit: 2cfe6eb34c86575bf18e74a06ff4aa7fbe3dd9e8
• View Logs

_Last updated: _

[jumski]

Merge activity

• Sep 2, 5:21 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Sep 2, 5:22 AM UTC: @jumski merged this pull request with Graphite.

[github-actions]

🚀 Production Deployment: Website

✅ Successfully deployed to production!

🔗 Production URL: https://pgflow.dev

📝 Details:

• Commit: 35d261b95b12b1b47aca4b5103f9e4302be4a747
• View Logs

Deployed at: 2026-09-02T07:22:06+02:00