Skip to content

redis: vendor redis.conf and auto-size maxmemory from the memory limit - #922

Draft
michaellzc wants to merge 1 commit into
mainfrom
michaellzc/redis-maxmemory-include-directive
Draft

redis: vendor redis.conf and auto-size maxmemory from the memory limit#922
michaellzc wants to merge 1 commit into
mainfrom
michaellzc/redis-maxmemory-include-directive

Conversation

@michaellzc

Copy link
Copy Markdown
Member

Both redis images bake maxmemory 6gb into /etc/redis/redis.conf and the chart never told redis about the pod memory limit, so lowering resources.limits.memory below 6GiB made the kubelet OOM-kill the pod before LRU eviction ran, restarting it into the same wall. Following the existing pgsql pattern, this vendors the images' config in-repo, renders it into a redis-{cache,store}-conf ConfigMap mounted over the image path with subPath, and appends a maxmemory sized at config.maxmemoryRatio (0.75) of the memory limit — the image entrypoint reads it, so there is no command/args override. A new config block on both services adds existingConfig, maxmemory, maxmemoryRatio, maxmemoryPolicy and additionalConfig, and the quantity parser fails safe: an unset or unrecognised limit emits no maxmemory so the vendored default stands.

Breaking: operators who inject their own redis.conf via redisCache/redisStore.extraVolumeMounts must move to config.existingConfig; a render-time guard fails with a migration message rather than letting it surface as an opaque duplicate-mount rejection. Also note the effective cap drops 6GiB → 5.25GiB at the default 7Gi limit, which for redis-store (noeviction) lowers the write-error ceiling — config.maxmemory: 6gb restores the old behaviour.

Checklist

Test plan

helm lint clean; helm unittest 26 suites / 126 tests pass (two new suites cover the parser matrix, both fail guards, and the mount/volume/checksum wiring). The HEAD~1-vs-HEAD rendered diff is purely additive — two ConfigMaps plus the per-Deployment mount, volume and checksum; every other resource is byte-identical. kubeconform -strict passes 71/71.

Provenance: extracted /etc/redis/redis.conf from both pinned 6.0.0 image digests with crane — identical to the vendored files. Entrypoint confirmed as ["redis-server","/etc/redis/redis.conf"]; neither image bakes a requirepass.

Deployed to a local kind cluster and inspected every path with redis-cli CONFIG GET:

  • Auto-sizing: 768Mi → 603979776, 1Gi → 805306368, 512Mi, 2G, raw bytes, 1.5Gi, maxmemoryRatio: 0.5; limits: null and an unparseable quantity both fall back to the vendored 6gb; explicit maxmemory: 300mb and maxmemoryPolicy: volatile-lru applied verbatim.
  • The fix itself, both at a 512Mi pod limit filled with ~1GB: old behaviour (maxmemory 6gb) was OOMKilled with evicted_keys:0, reproducing the loop; auto-sized (384Mi) plateaued at exactly 384.00M, evicted 154,057 keys, zero restarts, still serving writes. Store noeviction rejected real writes at 189.20M with OOM command not allowed... while staying healthy.
  • requirepass, all three routes: via additionalConfig and via existingConfig, redis enforces auth and the readiness probe still passes (replaying the probe script showed the grep finding the password in the chart-managed file). For a custom image that bakes requirepass, verified side-by-side that the old chart keeps auth while the new chart shadows it and starts unauthenticated — the documented regression, real and silent.
  • existingConfig skips the chart ConfigMap and the checksum annotation; changing maxmemoryRatio rolls the pod via checksum/redis-config (needed because subPath mounts never hot-reload); a real old→new helm upgrade swung maxmemory 6442450944 → 5637144576 with data intact; all four guards fail on helm install --dry-run=server with an unrelated extraVolumeMounts negative control passing.
  • Full-chart helm upgrade --install on kind came up healthy (frontend /healthz → 200) with the app writing to both redis instances.

Two things reviewers should know: with requirepass set the redis-exporter sidecar reports redis_up 0 while the pod still shows 2/2 Ready, so metrics go dark silently (pre-existing, but newly easy to trigger — redisExporter.env.REDIS_PASSWORD fixes it); and under localDevMode: true the container resources are stripped yet maxmemory is still computed from values.

🤖 Generated with Claude Code

`redis-cache` and `redis-store` relied entirely on the image's baked
`/etc/redis/redis.conf`, which hardcodes `maxmemory 6gb`. The chart never
told redis about the pod memory limit, so lowering
`resources.limits.memory` below 6GiB made the kubelet OOM-kill the pod
before LRU eviction ran, and the pod restarted into the same wall.

Follow the pgsql pattern: vendor the images' config in-repo under
`files/redis-{cache,store}/redis.conf`, render it into a
`redis-{cache,store}-conf` ConfigMap and mount it over the image path
with `subPath`. The image entrypoint (`redis-server
/etc/redis/redis.conf`) reads it, so no `command`/`args` override is
needed. The vendored files are copied verbatim from the pinned 6.0.0
image tags.

The chart appends a `maxmemory` sized at
`config.maxmemoryRatio` (0.75) of `resources.limits.memory`. A new
`config` block adds `existingConfig`, `maxmemory`, `maxmemoryRatio`,
`maxmemoryPolicy` and `additionalConfig` for both services. The quantity
parser fails safe: an unset or unrecognised limit emits no `maxmemory`
and the vendored default stands.

`sourcegraph.redis.assertNoConfClash` fails the render with a migration
message when `extraVolumeMounts` already targets `/etc/redis/redis.conf`
or an extra volume reuses the reserved `redis-conf` name, instead of
letting it surface as an opaque duplicate-mount rejection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant