Conversation
With the default 4 uwsgi processes, warm workers use about 500Mi of anonymous memory. Django's Argon2 hasher allocates about 100Mi per password check, so the first login or token-auth request pushes the container over its 512Mi limit. On cgroup v2 the kubelet OOM-kills the whole container, which is what the k8s CI job has been hitting: the first POST /api/v2/api-token-auth/ gets a 502 in most runs, and the job fails whenever the restarted container is killed again before the retry loop gives up. Raise the default limit to 1Gi. The CI workflow now also fails when a django container restarts during the test instead of letting the retry loop hide it, and on failure prints the previous container's logs, termination reason, pod description and events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[sc-15982]
Summary
The
test-k8sjob has been knocking PRs out of thebugfixmerge queue. The cause is the chart's default uwsgi memory limit. At 512Mi, the first password check OOM-kills the container.Argon2PasswordHasher(the first entry inPASSWORD_HASHERS) runs withmemory_cost=102400KiB, so every login, token-auth or basic-auth request allocates about 100Mi more while it hashes.memory.oom.group=1, so the OOM killer takes down the whole uwsgi container, not just one worker, and the kubelet restarts it.This PR raises
django.uwsgi.resources.limits.memoryfrom 512Mi to 1Gi. It also changes the k8s workflow so a restart like this fails the job, and the failure output shows why the container died.Evidence
From the CI logs:
POST /api/v2/api-token-auth/. nginx logsupstream prematurely closed connectionon that request, and the new container logs*** Starting uWSGIa few seconds later. The exception is the first death in docs(onprem): fix the Docker Compose install guide from an end-to-end install run #16090, which happened before the API test while only health probes were in flight. That fits the warm workers alone reaching the limit. None of the dying containers log a traceback, segfault or signal. The second instance in fix(jira): order the authorized JIRA project queryset by id #16095 died 8 seconds after it started, 6 seconds after a successful liveness probe, so the liveness probe (6 failures x 10s) did not kill it.Return code 502on the first one or two token-auth attempts, then200after the container restarts. The retry loop has been hiding it. The job only fails when the restarted container is killed again before the three retries run out.Local reproduction, using the same uwsgi entrypoint and flags (4 processes, 4 threads) in the debian Django image under a Docker memory limit, with
memory.oom.group=1set the way the kubelet sets it:OOMKilled=trueThe 512Mi row has the same signature as CI: the last log line is a normal request and nothing explains the exit.
Changes
helm/defectdojo/values.yaml: raise the uwsgi memory limit to 1Gi, with a comment on the sizing (about 100Mi per concurrent password check on top of the warm workers, so raise it further when raising processes or threads). Requests are unchanged.README.mdand theartifacthub.io/changesannotation are updated to match..github/workflows/k8s-tests.yml:kubectl describefor the django pods,kubectl logs --previousfor uwsgi and nginx, and cluster events. Until now the job printed only the current container's logs, so the reason the previous uwsgi died never reached the log.The probes are not changed. The evidence rules out a probe kill.
Verification
helm lint ./helm/defectdojopasses.helm templatewith the CI job's--setvalues renders. Compared againstbugfix, the only manifest difference (ignoring generated secrets) is the uwsgilimits.memorygoing from 512Mi to 1Gi.actionlinton the workflow: same 18 existing shellcheck notes as onbugfix, and no new ones.Not verified:
test-k8sjob on this PR is the real check, and the new restart step will fail it if 1Gi is not enough.memory.oom.groupon the CI minikube node. It is the kubelet default on cgroup v2 since Kubernetes 1.28, and it matches the whole-container restarts in CI.README.mdwas edited by hand, not regenerated with helm-docs. The chart lint workflow's helm-docs check will catch any drift.The OSS Kubernetes install and upgrade docs are generated from a different chart, so they are not touched here.
🤖 Generated with Claude Code