From e0ec7a68d8523c87039356388a115da4274da2f8 Mon Sep 17 00:00:00 2001 From: Komh Date: Fri, 24 Apr 2026 05:20:06 +0000 Subject: [PATCH 1/3] [networking] Change a Node's DNS Servers and /etc/resolv.conf Options via NMState NodeNetworkConfigurationPolicy --- ..._NMState_NodeNetworkConfigurationPolicy.md | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md diff --git a/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md b/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md new file mode 100644 index 000000000..07282eeaa --- /dev/null +++ b/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md @@ -0,0 +1,162 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- +## Issue + +The DNS servers and `/etc/resolv.conf` options on a node need to change — a new internal resolver has been rolled out, the node is being moved onto a different search domain, or options like `rotate` / `attempts` / `timeout` need to be tuned. Editing `/etc/resolv.conf` by hand is not durable: NetworkManager regenerates the file on every reboot (and on every network event), so hand-edits vanish. + +The clean path is to declare the desired DNS shape as a `NodeNetworkConfigurationPolicy` (NNCP) handled by the kubernetes-nmstate operator. NMState reconciles the node's NetworkManager configuration to match the declared state; the node picks up a new `/etc/resolv.conf` and keeps it across reboots. + +## Resolution + +### Prerequisites + +The cluster must have the kubernetes-nmstate operator installed and a `NMState` CR created so its DaemonSet is running on every target node. Verify with: + +```bash +kubectl get nmstate -o custom-columns='NAME:.metadata.name,AGE:.metadata.creationTimestamp' +kubectl -n nmstate get pod -l component=kubernetes-nmstate-handler -o wide +``` + +If either command returns empty, the operator is not yet active; install it through the cluster's operator-management surface before proceeding. + +### Capture the current node DNS state + +Before making any change, record what the node currently serves. On the target node: + +```bash +NODE= +kubectl debug node/$NODE --image=busybox -- \ + chroot /host sh -c 'cat /etc/resolv.conf; echo ---; nmcli dev show | grep -E "DNS|DOMAIN"' +``` + +Typical "before" shape: + +```text +# Generated by NetworkManager +search example.internal +nameserver 192.168.1.249 +``` + +This baseline is what the NNCP will replace on the next reconcile. + +### Declare the desired state via NNCP + +Target a single node first with a nodeSelector so the change is bounded. After the policy reconciles cleanly on that node, widen the selector to the rest of the fleet. + +```yaml +apiVersion: nmstate.io/v1 +kind: NodeNetworkConfigurationPolicy +metadata: + name: dns-worker-2 +spec: + nodeSelector: + kubernetes.io/hostname: worker-2 + desiredState: + dns-resolver: + config: + server: + - 192.168.1.249 + - 192.168.1.1 + search: + - example.internal + - corp.example.com + options: + - "rotate" + - "attempts:3" + - "timeout:2" +``` + +Apply and watch the reconcile: + +```bash +kubectl apply -f dns-worker-2.yaml + +kubectl get nodenetworkconfigurationpolicy dns-worker-2 \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status}{" "}{end}{"\n"}' +# Available=True Degraded=False +``` + +NMState publishes per-node progress on a companion `NodeNetworkConfigurationEnactment` (NNCE); check the one that matches the target node: + +```bash +kubectl get nodenetworkconfigurationenactment \ + -l nmstate.io/policy=dns-worker-2 \ + -o custom-columns='NODE:.status.nodeName,STATUS:.status.conditions[?(@.type=="Available")].status,MSG:.status.conditions[?(@.type=="Available")].message' +``` + +`STATUS=True` means NetworkManager has been reconfigured and the change is live. + +### Verify the effect on the node + +Re-check the node's `/etc/resolv.conf`: + +```bash +kubectl debug node/$NODE --image=busybox -- \ + chroot /host cat /etc/resolv.conf +``` + +Expected "after" shape: + +```text +# Generated by NetworkManager +search example.internal corp.example.com +nameserver 192.168.1.249 +nameserver 192.168.1.1 +options rotate attempts:3 timeout:2 +``` + +Test resolution from inside a workload pod to confirm the cluster-side DNS path is unaffected (pods continue to resolve through CoreDNS; the NNCP only changes the node's own resolver): + +```bash +kubectl run dns-probe -it --rm --restart=Never \ + --image=busybox -- \ + sh -c 'nslookup kubernetes.default.svc; nslookup external-host.example.com' +``` + +The first lookup must succeed (pods use CoreDNS, not the node resolver); the second must succeed through the new node-level resolver if the workload egresses through the host network. + +### Widen the rollout once validated + +After validation on the pilot node, remove the single-host `nodeSelector` (or set a label selector covering the broader fleet). Stagger by node label or by the operator's own batching to avoid reconfiguring every node simultaneously — one node at a time is safest. + +```yaml +spec: + nodeSelector: + node-role.kubernetes.io/worker: "" +``` + +Monitor the `NodeNetworkConfigurationEnactment` list for any node that reports `Available=False`; those will need their NM state inspected before proceeding. + +### Roll back + +Delete the NNCP and NMState reverts the nodes to the NetworkManager configuration they held before the policy applied. The operator stores the pre-policy state internally; no manual restore step is needed. + +```bash +kubectl delete nodenetworkconfigurationpolicy dns-worker-2 +``` + +Confirm each node's `/etc/resolv.conf` returns to the baseline recorded at the start of this procedure. + +## Diagnostic Steps + +If the NNCP goes to `Degraded=True`, read the specific enactment for the failing node to see which NetworkManager operation was rejected: + +```bash +kubectl get nodenetworkconfigurationenactment \ + -l nmstate.io/policy=,nmstate.io/node= \ + -o yaml +``` + +Common failures: + +- `dns-resolver.config.server` contains an IP that does not belong to any reachable subnet from the node. NetworkManager applies the resolver list regardless, but the node cannot resolve anything and subsequent probes fail. +- `search` domains exceed the kernel's `MAXDNSRCH` limit (typically 6 entries). NMState trims the list; document the precedence if that matters for the application. +- `options` list contains a token that the glibc resolver does not recognise. NetworkManager writes the option verbatim; invalid options are silently ignored by glibc. Verify each option against `man 5 resolv.conf`. + +If `/etc/resolv.conf` on the node still shows the old content after `Available=True`, NetworkManager may have been overridden by another service (cloud-init, a cluster addon writing directly to `/etc/resolv.conf`, or a static file). Check the node's `nmcli dev show | grep DNS` versus the file — they should agree. If they disagree, the file is being written after NetworkManager; remove that other writer or let NNCP manage a compatible file instead of `/etc/resolv.conf`. From 7973209f5555c881575acd7a1569351efa9fe2b2 Mon Sep 17 00:00:00 2001 From: Komh Date: Sat, 2 May 2026 12:57:50 +0000 Subject: [PATCH 2/3] [networking] Change a Node's DNS Servers and /etc/resolv.conf Options via NMState NodeNetworkConfigurationPolicy --- ...lvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md b/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md index 07282eeaa..e86f6b0db 100644 --- a/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md +++ b/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md @@ -6,6 +6,8 @@ products: ProductsVersion: - 4.1.0,4.2.x --- + +# Change a Node's DNS Servers and /etc/resolv.conf Options via NMState NodeNetworkConfigurationPolicy ## Issue The DNS servers and `/etc/resolv.conf` options on a node need to change — a new internal resolver has been rolled out, the node is being moved onto a different search domain, or options like `rotate` / `attempts` / `timeout` need to be tuned. Editing `/etc/resolv.conf` by hand is not durable: NetworkManager regenerates the file on every reboot (and on every network event), so hand-edits vanish. From 92e591eaa181e29cdbb332c7d697893735189b13 Mon Sep 17 00:00:00 2001 From: Komh Date: Fri, 17 Jul 2026 04:45:55 +0000 Subject: [PATCH 3/3] [networking] Configuring per-node DNS on ACP clusters with NodeNetworkConfigurationPolicy --- ..._NMState_NodeNetworkConfigurationPolicy.md | 164 ------------------ ...ers_with_NodeNetworkConfigurationPolicy.md | 84 +++++++++ 2 files changed, 84 insertions(+), 164 deletions(-) delete mode 100644 docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md create mode 100644 docs/en/solutions/Configuring_per_node_DNS_on_ACP_clusters_with_NodeNetworkConfigurationPolicy.md diff --git a/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md b/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md deleted file mode 100644 index e86f6b0db..000000000 --- a/docs/en/solutions/Change_a_Nodes_DNS_Servers_and_etcresolvconf_Options_via_NMState_NodeNetworkConfigurationPolicy.md +++ /dev/null @@ -1,164 +0,0 @@ ---- -kind: - - How To -products: - - Alauda Container Platform -ProductsVersion: - - 4.1.0,4.2.x ---- - -# Change a Node's DNS Servers and /etc/resolv.conf Options via NMState NodeNetworkConfigurationPolicy -## Issue - -The DNS servers and `/etc/resolv.conf` options on a node need to change — a new internal resolver has been rolled out, the node is being moved onto a different search domain, or options like `rotate` / `attempts` / `timeout` need to be tuned. Editing `/etc/resolv.conf` by hand is not durable: NetworkManager regenerates the file on every reboot (and on every network event), so hand-edits vanish. - -The clean path is to declare the desired DNS shape as a `NodeNetworkConfigurationPolicy` (NNCP) handled by the kubernetes-nmstate operator. NMState reconciles the node's NetworkManager configuration to match the declared state; the node picks up a new `/etc/resolv.conf` and keeps it across reboots. - -## Resolution - -### Prerequisites - -The cluster must have the kubernetes-nmstate operator installed and a `NMState` CR created so its DaemonSet is running on every target node. Verify with: - -```bash -kubectl get nmstate -o custom-columns='NAME:.metadata.name,AGE:.metadata.creationTimestamp' -kubectl -n nmstate get pod -l component=kubernetes-nmstate-handler -o wide -``` - -If either command returns empty, the operator is not yet active; install it through the cluster's operator-management surface before proceeding. - -### Capture the current node DNS state - -Before making any change, record what the node currently serves. On the target node: - -```bash -NODE= -kubectl debug node/$NODE --image=busybox -- \ - chroot /host sh -c 'cat /etc/resolv.conf; echo ---; nmcli dev show | grep -E "DNS|DOMAIN"' -``` - -Typical "before" shape: - -```text -# Generated by NetworkManager -search example.internal -nameserver 192.168.1.249 -``` - -This baseline is what the NNCP will replace on the next reconcile. - -### Declare the desired state via NNCP - -Target a single node first with a nodeSelector so the change is bounded. After the policy reconciles cleanly on that node, widen the selector to the rest of the fleet. - -```yaml -apiVersion: nmstate.io/v1 -kind: NodeNetworkConfigurationPolicy -metadata: - name: dns-worker-2 -spec: - nodeSelector: - kubernetes.io/hostname: worker-2 - desiredState: - dns-resolver: - config: - server: - - 192.168.1.249 - - 192.168.1.1 - search: - - example.internal - - corp.example.com - options: - - "rotate" - - "attempts:3" - - "timeout:2" -``` - -Apply and watch the reconcile: - -```bash -kubectl apply -f dns-worker-2.yaml - -kubectl get nodenetworkconfigurationpolicy dns-worker-2 \ - -o jsonpath='{range .status.conditions[*]}{.type}={.status}{" "}{end}{"\n"}' -# Available=True Degraded=False -``` - -NMState publishes per-node progress on a companion `NodeNetworkConfigurationEnactment` (NNCE); check the one that matches the target node: - -```bash -kubectl get nodenetworkconfigurationenactment \ - -l nmstate.io/policy=dns-worker-2 \ - -o custom-columns='NODE:.status.nodeName,STATUS:.status.conditions[?(@.type=="Available")].status,MSG:.status.conditions[?(@.type=="Available")].message' -``` - -`STATUS=True` means NetworkManager has been reconfigured and the change is live. - -### Verify the effect on the node - -Re-check the node's `/etc/resolv.conf`: - -```bash -kubectl debug node/$NODE --image=busybox -- \ - chroot /host cat /etc/resolv.conf -``` - -Expected "after" shape: - -```text -# Generated by NetworkManager -search example.internal corp.example.com -nameserver 192.168.1.249 -nameserver 192.168.1.1 -options rotate attempts:3 timeout:2 -``` - -Test resolution from inside a workload pod to confirm the cluster-side DNS path is unaffected (pods continue to resolve through CoreDNS; the NNCP only changes the node's own resolver): - -```bash -kubectl run dns-probe -it --rm --restart=Never \ - --image=busybox -- \ - sh -c 'nslookup kubernetes.default.svc; nslookup external-host.example.com' -``` - -The first lookup must succeed (pods use CoreDNS, not the node resolver); the second must succeed through the new node-level resolver if the workload egresses through the host network. - -### Widen the rollout once validated - -After validation on the pilot node, remove the single-host `nodeSelector` (or set a label selector covering the broader fleet). Stagger by node label or by the operator's own batching to avoid reconfiguring every node simultaneously — one node at a time is safest. - -```yaml -spec: - nodeSelector: - node-role.kubernetes.io/worker: "" -``` - -Monitor the `NodeNetworkConfigurationEnactment` list for any node that reports `Available=False`; those will need their NM state inspected before proceeding. - -### Roll back - -Delete the NNCP and NMState reverts the nodes to the NetworkManager configuration they held before the policy applied. The operator stores the pre-policy state internally; no manual restore step is needed. - -```bash -kubectl delete nodenetworkconfigurationpolicy dns-worker-2 -``` - -Confirm each node's `/etc/resolv.conf` returns to the baseline recorded at the start of this procedure. - -## Diagnostic Steps - -If the NNCP goes to `Degraded=True`, read the specific enactment for the failing node to see which NetworkManager operation was rejected: - -```bash -kubectl get nodenetworkconfigurationenactment \ - -l nmstate.io/policy=,nmstate.io/node= \ - -o yaml -``` - -Common failures: - -- `dns-resolver.config.server` contains an IP that does not belong to any reachable subnet from the node. NetworkManager applies the resolver list regardless, but the node cannot resolve anything and subsequent probes fail. -- `search` domains exceed the kernel's `MAXDNSRCH` limit (typically 6 entries). NMState trims the list; document the precedence if that matters for the application. -- `options` list contains a token that the glibc resolver does not recognise. NetworkManager writes the option verbatim; invalid options are silently ignored by glibc. Verify each option against `man 5 resolv.conf`. - -If `/etc/resolv.conf` on the node still shows the old content after `Available=True`, NetworkManager may have been overridden by another service (cloud-init, a cluster addon writing directly to `/etc/resolv.conf`, or a static file). Check the node's `nmcli dev show | grep DNS` versus the file — they should agree. If they disagree, the file is being written after NetworkManager; remove that other writer or let NNCP manage a compatible file instead of `/etc/resolv.conf`. diff --git a/docs/en/solutions/Configuring_per_node_DNS_on_ACP_clusters_with_NodeNetworkConfigurationPolicy.md b/docs/en/solutions/Configuring_per_node_DNS_on_ACP_clusters_with_NodeNetworkConfigurationPolicy.md new file mode 100644 index 000000000..55c7d35ca --- /dev/null +++ b/docs/en/solutions/Configuring_per_node_DNS_on_ACP_clusters_with_NodeNetworkConfigurationPolicy.md @@ -0,0 +1,84 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- + +# Configuring per-node DNS on ACP clusters with NodeNetworkConfigurationPolicy + +## Issue + +Cluster administrators sometimes need to change DNS settings on an individual node — upstream nameservers, search domains, or resolver options — without logging in to the host and hand-editing files. On Alauda Container Platform this can be done declaratively through the Kubernetes API with the kubernetes-nmstate operator: the operator serves the `NodeNetworkConfigurationPolicy` (NNCP) CRD at `nmstate.io/v1` (served and stored at `v1`), and a policy carrying a `dns-resolver` block rewrites the matched node's `/etc/resolv.conf` to the configured values. + +Prerequisite — NetworkManager-managed nodes only: this mechanism operates through NetworkManager, which owns and generates `/etc/resolv.conf` on the node (the file carries the header `# Generated by NetworkManager`, and NetworkManager renders the DNS values from its connection profiles, e.g. `DNS1=...` in an `ifcfg` profile). The end-to-end flow below was verified on CentOS 7.9.2009 nodes running NetworkManager 1.18.8 with nmstate 2.2.4x. Nodes whose network stack is not managed by NetworkManager (for example netplan- or systemd-networkd-based distributions) do not meet this prerequisite. No nmstate package is required on the host itself — the operator's handler image ships `nmstatectl`. + +## Resolution + +Step 1 — install the kubernetes-nmstate operator: + +The NNCP resource type is not served until the operator is installed; before installation, `kubectl explain nodenetworkconfigurationpolicy` fails with a hard error, and afterwards it resolves to `GROUP nmstate.io, VERSION v1`. The v4.1.4 operator bundle (`kubernetes-nmstate-operator`, channel `alpha`, handler image tag `kubernetes-nmstate-handler:v4.1.4`) is available from the platform package repository and ships the operator manifests together with the four `nmstate.io` CRDs; the operator, handler, and webhook run in the `nmstate` namespace. As shipped, the bundle's CSV may reference an internal build registry; repoint the images to the platform registry per the bundle's `relatedImages` list. After installation the `nmstate.io` API group serves `nodenetworkconfigurationpolicies` (short name `nncp`, `v1`), `nodenetworkconfigurationenactments` (`v1beta1`), `nodenetworkstates` (`v1beta1`), and `nmstates`, and the handler publishes a `NodeNetworkState` object for every node in the cluster. + +```bash +kubectl api-resources --api-group=nmstate.io +kubectl explain nodenetworkconfigurationpolicy +kubectl get nodenetworkstates +``` + +Step 2 — create the DNS policy: + +Set `spec.nodeSelector` to the target node's `kubernetes.io/hostname` label to scope the policy to that single node — on a three-node cluster this produced exactly one enactment object, on the matched node only. The `desiredState.dns-resolver.config` block carries three fields in the nmstate schema shipped with the handler: `server` (list of DNS server IPs), `search` (list of search domains), and `options` (resolver option strings). The option strings exercised end-to-end on this platform were `attempts:3` and `timeout:2`; other option strings are accepted at the schema level. + +```yaml +apiVersion: nmstate.io/v1 +kind: NodeNetworkConfigurationPolicy +metadata: + name: node1-dns +spec: + nodeSelector: + kubernetes.io/hostname: + desiredState: + dns-resolver: + config: + server: + - 192.168.16.19 + - 223.5.5.5 + search: + - internal.example + options: + - attempts:3 + - timeout:2 +``` + +Apply the manifest with the CLI; creation returns the standard confirmation and the policy's `spec.nodeSelector` reads back with the single matched hostname: + +```bash +kubectl create -f node1-dns.yaml +``` + +The policy reaches `Available=True` with reason `SuccessfullyConfigured` within seconds of creation, and one `NodeNetworkConfigurationEnactment` (NNCE) appears for the matched node while unmatched nodes get none. + +On the matched node, `/etc/resolv.conf` is rewritten: the `nameserver` entries become the policy's servers in policy order, the policy's search domains are written into the `search` line, and the option strings land on a single `options` line. The `# Generated by NetworkManager` header remains the first line of the rewritten file; the policy's hostname selector confined the change to the matched node — the control node checked before and after showed an unchanged `/etc/resolv.conf`, and no enactment was created for unmatched nodes — and name resolution from the node keeps working on the new configuration. + +Rollback: applying a follow-up policy that carries the node's original `dns-resolver` values rewrites `/etc/resolv.conf` back to its prior content exactly (the test configuration left no residue), and node name resolution was confirmed working after the restore. + +## Diagnostic Steps + +Check the policy and per-node enactment status; a healthy rollout shows the policy `Available=True` with reason `SuccessfullyConfigured` and one NNCE for the matched node: + +```bash +kubectl get nncp +kubectl get nnce +``` + +Confirm the result directly on the target node by reading `/etc/resolv.conf` — a read-only check performed with a node debug pod: + +```bash +kubectl debug node/ -it --image= -- chroot /host cat /etc/resolv.conf +``` + +The expected file starts with the `# Generated by NetworkManager` header; in the verified run the policy's `search` line came next, followed by the `nameserver` entries and then the `options` line. + +As a final sanity check, resolve an external hostname from the target node (for example with `getent hosts `); this succeeded on the rewritten configuration and again after the rollback.