Version Packages - #659
Open
jumski wants to merge 1 commit into
Open
Conversation
|
View your CI Pipeline Execution ↗ for commit 8683d94
💡 Dealing with memory or CPU issues? See memory and CPU details with the resource usage add-on ↗. ☁️ Nx Cloud last updated this comment at |
jumski
force-pushed
the
changeset-release/main
branch
from
September 1, 2026 12:42
dbf34c1 to
53885a0
Compare
jumski
force-pushed
the
changeset-release/main
branch
from
September 2, 2026 01:12
53885a0 to
b1ba4c2
Compare
jumski
added a commit
that referenced
this pull request
Sep 2, 2026
## Summary Fix stalled-task recovery to use the effective step timeout instead of always using the flow timeout. This PR stacks directly on #663 (`09-01-issue_645_failed_run_terminalization`) and preserves its run, step-state, and task eligibility guards. ## Root cause `start_tasks()` sets PGMQ visibility from the effective timeout: ```sql coalesce(step.opt_timeout, flow.opt_timeout) + 2 ``` `requeue_stalled_tasks()` used only `flows.opt_timeout`. A short step timeout could therefore make the PGMQ message visible while its task row remained `started` until the longer flow timeout and recovery buffer expired. ## Behavior Recovery now requires: ```sql started_at < now() - (coalesce(step.opt_timeout, flow.opt_timeout) * interval '1 second') - interval '30 seconds' ``` The comparison stays strict. A null step timeout inherits the non-null flow timeout. The PGMQ-only two-second margin is not added to recovery. Adding it again would change the existing 30-second recovery grace to 32 seconds. The 15-second cron cadence can add up to roughly 15 seconds after eligibility. The change preserves: - `run.status = 'started'`, `step_state.status = 'started'`, and `task.status = 'started'`; - `permanently_stalled_at is null` and `FOR UPDATE OF task SKIP LOCKED` behavior; - attempts and requeue counters; - three successful requeues before permanent stall; - immediate visibility through `set_vt_batch(..., 0)`; - archive and permanent-stall behavior. ## Tests Added `effective_step_timeout.test.sql` with deterministic timestamps inside one transaction: - flow 60 / step 5: exactly 35 seconds stays started; 36 seconds requeues; - flow 5 / step 60: 36 seconds stays started; 91 seconds requeues; - flow 5 / null step timeout: 36 seconds requeues through flow fallback. Before the source fix, the focused test failed 5 of 10 assertions for the expected reason. The short step override returned 0 and stayed `started` at 36 seconds. The long step override requeued at 36 seconds, so its later 91-second call returned 0. After the source fix: - focused test: 1 file, 10 tests, pass; - all stalled-recovery tests: 6 files, 54 tests, pass; - full pgTAP: 285 files, 1326 tests, pass. ## Migration and release note Atlas generated `20260901203454_pgflow_temp_effective_step_timeout.sql`. It replaces `pgflow.requeue_stalled_tasks()`, includes the cumulative #645 guards, and performs no backfill. Added a separate patch changeset for `@pgflow/core`. The fixed release group expands the patch at release time. ## Checks - `pnpm nx verify-migrations core --skip-nx-cache` — pass - `pnpm nx gen-types core --skip-nx-cache` — pass; no generated type diff - `pnpm nx verify-gen-types core --skip-nx-cache` — pass - `pnpm nx test:pgtap core --skip-nx-cache` — pass; 285 files, 1326 tests - `pnpm nx test core --skip-nx-cache` — pass - `pnpm nx lint core --skip-nx-cache` — pass; 0 errors and 2 existing type-test warnings - `pnpm nx build core --skip-nx-cache` — pass - `pnpm changeset status` — pass; patch fixed group detected - `git diff --check` — pass `pnpm nx fix-sql core` hit the known Sqruff CLI mismatch: `error: unexpected argument '--force' found`. The direct repository fallback, `sqruff --config=.sqruff fix --parsing-errors pkgs/core/schemas/`, processed 37 files and found nothing to fix. Direct Sqruff lint also passed. Two fresh independent Sol xhigh review rounds returned `APPROVED` with no required findings. ## Out of scope - #656 and execution of the unreferenced `start_tasks()` visibility CTE; - #646 worker-side handler cancellation; - queue identity or per-step queue routing; - changes to #645 cancellation semantics or parent-state guards; - configurable recovery buffers or cron cadence; - migration consolidation and release PR #659. The stack still contains temporary migrations, so the main-targeted temporary-migration check can fail until the settled release sequence consolidates them. This PR does not consolidate the parent migration. Fixes #621
jumski
force-pushed
the
changeset-release/main
branch
from
September 2, 2026 05:23
b1ba4c2 to
8683d94
Compare
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.
Releases
pgflow@0.15.1
Patch Changes
@pgflow/client@0.15.1
Patch Changes
@pgflow/core@0.15.1
Patch Changes
7903661: Mark unfinished tasks as cancelled when their run fails, prevent late callbacks and stalled recovery from reviving them, and repair active tasks on historical failed runs.
daadcac: Terminalize queued and started task rows when their parent step is skipped: sibling tasks of a step skipped via
whenExhausted: 'skip'/'skip-cascade'(and cascade-skipped steps) now end asskippedinstead of stayingqueued/startedforever, and a migration repairs existing rows.Tasks are now terminalized before their queue messages are archived, preserving the task-before-queue lock order, and
start_tasksonly returns rows it actually claimed, so workers no longer execute tasks a concurrent skip already markedskipped.Fixes Skip paths archive sibling messages but never terminalize sibling step_tasks rows, leaving status='started' tasks on completed runs #638
35d261b: Requeue stalled tasks using the effective step timeout instead of waiting for the flow timeout.
@pgflow/edge-worker@0.15.1
Patch Changes
@pgflow/dsl@0.15.1