Skip to content

fix(workflow): catch up missed cron fires - #7506

Open
BluRoot01 wants to merge 1 commit into
block:mainfrom
BluRoot01:fix/workflow-scheduler-catchup
Open

fix(workflow): catch up missed cron fires#7506
BluRoot01 wants to merge 1 commit into
block:mainfrom
BluRoot01:fix/workflow-scheduler-catchup

Conversation

@BluRoot01

Copy link
Copy Markdown

Summary

  • persist per-workflow cron cursors so scheduler restarts retain execution progress
  • claim due schedule slots atomically before emitting actions, preventing duplicate execution across concurrent ticks
  • catch up missed cron fires within a bounded window and record skipped backlog safely
  • cover normal due execution, restart catch-up, duplicate prevention, and database persistence

Related issue

  • Buzz production issue: buzz://issue?id=3fdeffee9e1aeb6693e13d0fda17d0be50e301a2847143c81a56466df8c4f45f&owner=0f3a6f2f1e2d60769a231ae87b87f56aa9ddf4f3bdbb0a9d264bdc46cf21f614&d=switchboard-zoho-gateway
  • No matching upstream GitHub issue found during incident triage.

Migration

Adds migrations/0045_workflow_schedule_cursors.sql. The owner approved this migration for deployment on 2026-09-08. It creates the scheduler cursor table and its cleanup trigger; no existing application data is rewritten.

Testing

  • 171 buzz-workflow unit tests passed
  • all 6 PostgreSQL scheduler persistence tests passed
  • Rust formatting passed
  • strict Clippy passed
  • independent Sentry review passed
  • verified tested commit: 0d05ec6e90087b88fea403aa512779908adf8e80

Signed-off-by: Forge <forge@bluroot.co>
@BluRoot01
BluRoot01 requested a review from a team as a code owner September 8, 2026 23:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T23:40:14.910071Z 0d05ec6 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

🔐 Codex Security Review

Status: review required for the current range.

The current range is c045321a7fb3ca8939f28519ce7a555a6f597728...0d05ec6e90087b88fea403aa512779908adf8e80.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 0d05ec6e90087b88fea403aa512779908adf8e80 to authorize a new review.
Any previous review applies only to its recorded range.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d05ec6e90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

-- start at deployment time so rollout never replays historical schedules.
SET LOCAL lock_timeout = '5s';

CREATE TABLE workflow_schedule_cursors (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mirror the cursor into the desired-state schema

This defines the cursor relation only in the incremental migration; a repo-wide search confirms that schema/schema.sql contains neither workflow_schedule_cursors nor its reset trigger. Fresh databases created through the repository's pgschema apply path therefore lack this relation, so every cron tick fails in get_or_seed_workflow_schedule_cursor, and community-deletion catalog validation also expects a table that is absent. Add the table, trigger function, trigger, and write fence to the desired-state schema.

AGENTS.md reference: AGENTS.md:L557-L557

Useful? React with 👍 / 👎.

limit = MAX_SCHEDULE_FIRES_PER_TICK,
"Cron tick: global schedule fire budget exhausted"
);
break;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Make the global fire budget fair across workflows

When more than 100 schedule instants are due per minute, this fixed-order break permanently starves later workflows: list_all_enabled_workflows orders by created_at ASC, the first 100 minutely workflows consume the entire budget, and those same workflows each become due again before the next tick, so workflow 101 is never visited and its backlog only grows. The cap needs fair rotation or pagination across workflows rather than restarting from the oldest row on every tick.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant