Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@ Three consequences follow, and all three are easy to get wrong:
- **Every** container in the pod needs a limit, init containers included. Kubelet only sets pod-level `memory.max` when all of them have one, so a single limit-free sidecar puts the whole pod back in the candidate set.
- CPU limits are irrelevant here. The ranking expression is given the cgroup's path, its QoS class, and `memory.max`, `memory.current` and `memory.peak` — no CPU information of any kind.

Victim selection is also decoupled from the trigger. The cgroup that caused the pressure and the cgroup that gets `SIGKILL`ed are unrelated by design. In practice the pods carrying limits are overwhelmingly tenant workloads — managed applications are sized through resource presets, and a tenant namespace with `resourceQuotas` configured gets a default limit of its own — while system components carried none until the change this page describes. That inverts the intended order: whatever fires the trigger, `metallb-speaker` or `linstor-satellite`, using a few dozen megabytes and entirely uninvolved, is the one killed for it, repeatedly, until the pressure subsides. On Talos v1.13.x a tenant workload thrashing against its own multi-gigabyte limit is enough to fire it, through the node-wide PSI clause described [below](#when-the-trigger-itself-is-the-problem).
Victim selection is also decoupled from the trigger. The cgroup that caused the pressure and the cgroup that gets `SIGKILL`ed are unrelated by design. In practice the pods carrying limits are overwhelmingly tenant workloads — managed applications are sized through resource presets, and a tenant namespace with non-GPU `resourceQuotas` configured gets a default limit of its own — while system components carried none until the change this page describes. That inverts the intended order: whatever fires the trigger, `metallb-speaker` or `linstor-satellite`, using a few dozen megabytes and entirely uninvolved, is the one killed for it, repeatedly, until the pressure subsides. On Talos v1.13.x a tenant workload thrashing against its own multi-gigabyte limit is enough to fire it, through the node-wide PSI clause described [below](#when-the-trigger-itself-is-the-problem).

## What Cozystack does

**A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded.

**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both.

The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above.
The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when the tenant's `resourceQuotas` contains non-GPU entries. GPU-only budgets do not create this LimitRange; `resourceQuotas` is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above.

## Tuning

Expand Down
Loading