Skip to content

Bootstrap resume workflows config options - #6402

Open
guzzijones wants to merge 8 commits into
masterfrom
bootstrap-resume-workflows
Open

Bootstrap resume workflows config options#6402
guzzijones wants to merge 8 commits into
masterfrom
bootstrap-resume-workflows

Conversation

@guzzijones

@guzzijones guzzijones commented Aug 28, 2026

Copy link
Copy Markdown
Contributor
  1. Add a config option to enable bootstrap on workflow engine startup. (disabled by default). This prevents a 'thunderous herd' of actions on startup that may surprise on operator.
  2. Add a config option for look back period for bootstrap. Very old workflows/actions are not worth restarting.
  3. Add script to manually bootstrap a single workflow or workflows. This allows an operator to manually bootstrap a workflow when a workflow engine OOMs holding on to workflows in memory.

Backstory

We restarted both workflow engines in k8s because the bootstrap process never kicked off. In k8s this is always the case because one pod is always available. A 'thunderous herd' of old actions restarted causing massive downstream work and cleanup. These fixes give knobs to control the look back period and enable / disable bootstrap altogether. Also, this solves the OOM one engine down case.

@pull-request-size pull-request-size Bot added the size/M PR that changes 30-99 lines. Good size to review. label Aug 28, 2026
@pull-request-size pull-request-size Bot added size/L PR that changes 100-499 lines. Requires some effort to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Aug 28, 2026
@guzzijones guzzijones added this to the 3.10.0 milestone Aug 28, 2026
@pull-request-size pull-request-size Bot added size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels Aug 28, 2026
@guzzijones guzzijones changed the title Bootstrap resume workflows Bootstrap resume workflows config options Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant