[feature]: wf_spec option to retry only on OOMKilled with escalating memory

[wesbragagt]

Prior Search

• I have already searched this project's issues to determine if a similar request has already been made.

What new functionality would you like to see?

An option on wf_spec that retries a template only after an OOMKilled attempt, and runs each retry with more memory.

Today wf_spec takes a single default_resources value. A workflow that is under-sized for one input either fails, or carries a memory request large enough for its worst case on every run. The first wastes an operator's time; the second wastes node capacity on every normal run.

Argo already supports the two pieces. The request is to express them as module inputs instead of hand-written template overrides:

module "workflow" {
  source = ".../wf_spec"

  default_resources = {
    requests = { cpu = "500m", memory = "1024Mi" }
    limits   = { memory = "1024Mi" }
  }

  oom_retry = {
    max_retries   = 3
    growth_factor = 1.3
  }
}

That would render:

retryStrategy:
  limit: "3"
  retryPolicy: Always
  expression: "lastRetry.message matches 'OOMKilled'"
podSpecPatch: |
  containers:
    - name: main
      resources:
        requests:
          cpu: "500m"
          memory: "{{=sprig.int(retries) == 0 ? '1024Mi' : sprig.int(retries) == 1 ? '1332Mi' : sprig.int(retries) == 2 ? '1731Mi' : '2250Mi'}}"
        limits:
          memory: "{{=sprig.int(retries) == 0 ? '1024Mi' : sprig.int(retries) == 1 ? '1332Mi' : sprig.int(retries) == 2 ? '1731Mi' : '2250Mi'}}"

Memory request and limit stay equal on each attempt, so the scheduler reserves what the attempt needs. CPU keeps a request floor and no limit, which matches the current wf_spec behavior.

One detail is easy to get wrong and is worth encoding in the module. Argo exposes retries as a string inside an expression tag. The obvious form retries == 0 ? ... makes the controller reject the pod with Error applying PodSpecPatch, and the attempt never starts. The cast sprig.int(retries) is required. We lost time to this before we found it.

How would you use this new functionality?

We run data reconciliation workflows on Argo through wf_spec. Memory use scales with the invoice volume of the tenant being processed, and the range is wide. A run that normally needs 1 GiB needs 6 GiB for the largest tenants.

Our current options are both bad. We size every workflow for the worst tenant, which reserves several GiB per pod on runs that need a fraction of it. Or we size for the common case and an engineer re-runs the job by hand after each OOM.

With this option we would set the base at the common case and let the ladder absorb the outliers. A single failure class, OOMKilled, is the one our operators handle by raising memory and re-running, so it is the one worth automating. Other failures should still surface immediately, which the expression gives us.

We have validated the pattern against Argo Workflows v3.6.2 with raw manifests and then through our own Terraform wrapper. A 256Mi attempt and a 333Mi attempt were OOMKilled, the 433Mi attempt completed, and a non-OOM failure in the same workflow was not retried. We would rather delete our wrapper and use a supported wf_spec input.