Skip to content

[18.0-fr6] Fix OVN condition flapping during minor updates#1976

Open
openshift-cherrypick-robot wants to merge 1 commit into
openstack-k8s-operators:18.0-fr6from
openshift-cherrypick-robot:cherry-pick-1966-to-18.0-fr6
Open

[18.0-fr6] Fix OVN condition flapping during minor updates#1976
openshift-cherrypick-robot wants to merge 1 commit into
openstack-k8s-operators:18.0-fr6from
openshift-cherrypick-robot:cherry-pick-1966-to-18.0-fr6

Conversation

@openshift-cherrypick-robot

@openshift-cherrypick-robot openshift-cherrypick-robot commented Jul 8, 2026 •
edited by karelyatin
Loading

Copy link
Copy Markdown

This is an automated cherry-pick of #1966
/assign karelyatin
Related-Issue: https://redhat.atlassian.net/browse/OSPRH-31318

@karelyatin
Fix OVN condition flapping during minor updates
9bc7acd 
During minor updates, reconcileOVNControllers was called on every
reconcile throughout all update phases, causing two issues:

1. OVNReadyCondition flapping: Since InitConditions() resets all
   conditions to Unknown on each reconcile, calling
   reconcileOVNControllers during later phases (RabbitMQ, Galera, etc.)
   could transiently flip OVNReadyCondition to False, blocking the
   OpenStackVersion controller from maintaining MinorUpdateOVNControlplane
   as True.

2. Metrics DaemonSet unnecessary restarts: reconcileOVNControllers
   passed an empty string for the metrics cert name, clearing the
   MetricsTLS fields on the OVNController CR. This changed the metrics
   config hash, causing two spurious DaemonSet rollouts — once when
   MetricsTLS was cleared during the update, and again when the normal
   reconcile path restored it after the update completed.

Fix both by:
- Skipping reconcileOVNControllers after the OVN controlplane phase
  completes, directly marking OVNReadyCondition True instead (needed
  because InitConditions() resets conditions each reconcile).
- Passing the correct metrics cert name via EnsureOVNMetricsCert so
  MetricsTLS stays consistent when reconcileOVNControllers does run.

There was another issue in ovn-operator for pre mature readiness, that is
being fixed as part of openstack-k8s-operators/ovn-operator#587

Related-Issue: https://redhat.atlassian.net/browse/OSPRH-31318
Assisted-By: Claude
Signed-off-by: Yatin Karel <ykarel@redhat.com>
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Jul 8, 2026
Merged
@openshift-ci openshift-ci Bot requested review from abays and slagle July 8, 2026 14:34
@openshift-ci

openshift-ci Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: openshift-cherrypick-robot
Once this PR has been reviewed and has the lgtm label, please assign abays for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

OpenStackControlPlane CRD Size Report

Metric Value
CRD JSON size 349969 bytes (342KB)
Base branch size 349969 bytes
Change +0.00%
Status yellow — growing
Threshold reference
Color Range Meaning
🟢 green < 300KB Comfortable
🟡 yellow 300–400KB Growing
🟠 orange 400–750KB Concerning
🔴 red > 750KB Approaching 1.5MB etcd limit (cut in half to allow space for update)

@karelyatin

karelyatin commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
(failed containers: test): ContainerFailed one or more containers exited

@karelyatin

karelyatin commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
error: gather never finished for pod must-gather-hhnst: context deadline exceeded

@karelyatin

karelyatin commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
gather failure

@karelyatin

karelyatin commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
Must Gather Timeout bumped openshift/release#81723

@karelyatin

karelyatin commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
Process did not finish before 4h0m0s timeout

@karelyatin

karelyatin commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
fails even after timeout bump, this time 1/3 node had issues

@karelyatin

karelyatin commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
kuttl failed, must gather passed

@abays

abays commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@openshift-cherrypick-robot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-18 9bc7acd link true /test openstack-operator-build-deploy-kuttl-4-18

Full PR test history. Your PR dashboard.
Details

When must-gather fails, I see this in the logs (and I see this in other PRs as well):

[must-gather-8p577] OUT 2026-07-13T05:57:16.072618897Z gather logs unavailable: http2: client connection lost
[must-gather-8p577] OUT 2026-07-13T05:57:16.072861544Z waiting for gather to complete
[must-gather-8p577] OUT 2026-07-13T06:12:16.092923397Z gather never finished: context deadline exceeded
[must-gather      ] OUT 2026-07-13T06:12:16.451860695Z namespace/openshift-must-gather-854tr deleted

@karelyatin

karelyatin commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

<< When must-gather fails, I see this in the logs (and I see this in other PRs as well):
yes and looks openshift cluster unstable, as in some runs noticed one(out of three) of the nodes had issues while must gather was running

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
gather fix patch in latest index

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/override ci/prow/openstack-operator-build-deploy-kuttl-4-20
no cluster claim and new job

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@karelyatin: karelyatin unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:.

Details

In response to this:

/override ci/prow/openstack-operator-build-deploy-kuttl-4-20
no cluster claim and new job

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/retest
must gather fixes are merged

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20
Error from server (NotFound): openstacks.operator.openstack.org "openstack" not found

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-18
NodeControllerDegraded: The master nodes not ready: node "oko-23-dc5hz-master-0" not ready since 2026-07-14 13:17:30 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
RevisionControllerDegraded: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:218073607192922440 name:"oko-23-dc5hz-master-2" peerURLs:"https://10.0.3.166:2380" clientURLs:"https://10.0.3.166:2379" Healthy:true Took:1.886157ms Error:} {Member:ID:5474027055766516590 name:"oko-23-dc5hz-master-0" peerURLs:"https://10.0.0.58:2380" clientURLs:"https://10.0.0.58:2379" Healthy:false Took:29.996351976s Error:health check failed: context deadline exceeded} {Member:ID:16012177282994135839 name:"oko-23-dc5hz-master-1" peerURLs:"https://10.0.0.99:2380" clientURLs:"https://10.0.0.99:2379" Healthy:true Took:2.121304ms Error:}]

@karelyatin

karelyatin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20
ClusterOperators:
clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: need at least 3 kube-apiservers, got 2) because APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()
WellKnownReadyControllerDegraded: need at least 3 kube-apiservers, got 2
clusteroperator/dns is progressing: DNS "default" reports Progressing=True: "Have 2 available node-resolver pods, want 3."
clusteroperator/etcd is degraded because EtcdMembersDegraded: 2 of 3 members are available, oko-08-r7hm5-master-2 is unhealthy
NodeControllerDegraded: The master nodes not ready: node "oko-08-r7hm5-master-2" not ready since 2026-07-14 14:20:04 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
RevisionControllerDegraded: CheckSafeToScaleCluster found 2 healthy member(s) out of the 3 required by the HAScalingStrategy

@karelyatin

karelyatin commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20

@karelyatin

karelyatin commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20
ClusterOperators:
clusteroperator/authentication is progressing: OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 3/3 pods have been updated to the latest generation and 2/3 pods are available
clusteroperator/dns is progressing: DNS "default" reports Progressing=True: "Have 2 available node-resolver pods, want 3."
clusteroperator/etcd is degraded because EtcdMembersDegraded: 2 of 3 members are available, oko-11-n8scj-master-2 is unhealthy
NodeControllerDegraded: The master nodes not ready: node "oko-11-n8scj-master-2" not ready since 2026-07-15 03:55:40 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
RevisionControllerDegraded: CheckSafeToScaleCluster found 2 healthy member(s) out of the 3 required by the HAScalingStrategy
clusteroperator/image-registry is progressing: Progressing: The registry is ready
NodeCADaemonProgressing: The daemon set node-ca is deploying node pods

@karelyatin

karelyatin commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20
omc=false merged

@karelyatin

karelyatin commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

/test openstack-operator-build-deploy-kuttl-4-20
resource OpenStackControlPlane:openstack-kuttl-tests/openstack: .status.conditions: slice length mismatch: 29 != 19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abays abays Awaiting requested review from abays
@slagle slagle Awaiting requested review from slagle

At least 1 approving review is required to merge this pull request.

Assignees

@karelyatin karelyatin

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@openshift-cherrypick-robot @karelyatin @abays