Skip to content

fix(helm): raise uwsgi memory limit so the first login is not OOM-killed - #16097

Closed
Maffooch wants to merge 1 commit into
bugfixfrom
fix/helm-uwsgi-oom-on-login
Closed

Maffooch wants to merge 1 commit into
bugfixfrom
fix/helm-uwsgi-oom-on-login

Conversation

@Maffooch

@Maffooch Maffooch commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

[sc-15982]

Summary

The test-k8s job has been knocking PRs out of the bugfix merge queue. The cause is the chart's default uwsgi memory limit. At 512Mi, the first password check OOM-kills the container.

  • With the default 4 uwsgi processes, warm workers use about 490Mi of anonymous memory.
  • Django's Argon2PasswordHasher (the first entry in PASSWORD_HASHERS) runs with memory_cost=102400 KiB, so every login, token-auth or basic-auth request allocates about 100Mi more while it hashes.
  • On cgroup v2 the kubelet sets memory.oom.group=1, so the OOM killer takes down the whole uwsgi container, not just one worker, and the kubelet restarts it.

This PR raises django.uwsgi.resources.limits.memory from 512Mi to 1Gi. It also changes the k8s workflow so a restart like this fails the job, and the failure output shows why the container died.

Evidence

From the CI logs:

Local reproduction, using the same uwsgi entrypoint and flags (4 processes, 4 threads) in the debian Django image under a Docker memory limit, with memory.oom.group=1 set the way the kubelet sets it:

Limit After 12 GET /login First token-auth POST
512Mi 511Mi, at the limit connection dropped, container exits 137, OOMKilled=true
768Mi 512Mi 200, peak 614Mi
1Gi 519Mi (anon 485Mi) 200, peak 621Mi; 4 concurrent POSTs: all 200, peak 924Mi

The 512Mi row has the same signature as CI: the last log line is a normal request and nothing explains the exit.

Changes

  • helm/defectdojo/values.yaml: raise the uwsgi memory limit to 1Gi, with a comment on the sizing (about 100Mi per concurrent password check on top of the warm workers, so raise it further when raising processes or threads). Requests are unchanged. README.md and the artifacthub.io/changes annotation are updated to match.
  • .github/workflows/k8s-tests.yml:
    • New "Check django pod did not restart" step. It fails if any django container restarted during the test, so a masked crash like this one can't pass again.
    • The "Failed Logs" step now prints each container's restart count and last termination reason, exit code and time, plus kubectl describe for the django pods, kubectl logs --previous for uwsgi and nginx, and cluster events. Until now the job printed only the current container's logs, so the reason the previous uwsgi died never reached the log.

The probes are not changed. The evidence rules out a probe kill.

Verification

  • helm lint ./helm/defectdojo passes.
  • helm template with the CI job's --set values renders. Compared against bugfix, the only manifest difference (ignoring generated secrets) is the uwsgi limits.memory going from 512Mi to 1Gi.
  • actionlint on the workflow: same 18 existing shellcheck notes as on bugfix, and no new ones.
  • Local Docker reproduction as described above.

Not verified:

  • Nothing here ran on minikube locally. The test-k8s job on this PR is the real check, and the new restart step will fail it if 1Gi is not enough.
  • The local measurements come from an arm64 build of the Django image, while CI runs amd64. The numbers should be close but will not match exactly.
  • I did not confirm memory.oom.group on the CI minikube node. It is the kubelet default on cgroup v2 since Kubernetes 1.28, and it matches the whole-container restarts in CI.
  • README.md was edited by hand, not regenerated with helm-docs. The chart lint workflow's helm-docs check will catch any drift.

The OSS Kubernetes install and upgrade docs are generated from a different chart, so they are not touched here.

🤖 Generated with Claude Code

With the default 4 uwsgi processes, warm workers use about 500Mi of
anonymous memory. Django's Argon2 hasher allocates about 100Mi per
password check, so the first login or token-auth request pushes the
container over its 512Mi limit. On cgroup v2 the kubelet OOM-kills the
whole container, which is what the k8s CI job has been hitting: the
first POST /api/v2/api-token-auth/ gets a 502 in most runs, and the job
fails whenever the restarted container is killed again before the retry
loop gives up.

Raise the default limit to 1Gi. The CI workflow now also fails when a
django container restarts during the test instead of letting the retry
loop hide it, and on failure prints the previous container's logs,
termination reason, pod description and events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch Maffooch added this to the 3.3.300 milestone Sep 25, 2026
@github-actions github-actions Bot added the helm label Sep 25, 2026
@Maffooch Maffooch closed this Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant