OCPBUGS-94183, OCPBUGS-94184: [release-4.22] Add transient error handling across all three controllers, fix progressing=true by theobarberbany · Pull Request #485 · openshift/cluster-cloud-controller-manager-operator

theobarberbany · 2026-07-08T13:50:23Z

Summary

All three reconcilers (CloudConfigReconciler, TrustedCABundleReconciler, CloudOperatorReconciler) now distinguish transient errors (API blips) from permanent errors (corrupt config, unsupported platform). Transient errors are silently requeued and only set Degraded=True after a threshold (2m for sub-controllers, 2m30s for the main controller). Permanent errors degrade immediately without requeue.
Failure handling is consolidated into a shared finalizeReconcile() deferred function in reconcile_dispatch.go, using controller-runtime's TerminalError to classify errors.
Fix Progressing=True being immediately overwritten by Available — sync() now returns (bool, error) so Reconcile skips setStatusAvailable when resources were just updated.

This depends on #484 to avoid the race that required the revert previously.

All three reconcilers (CloudConfigReconciler, TrustedCABundleReconciler, CloudOperatorReconciler) now distinguish between transient errors (API blips) and permanent errors (corrupt config, unsupported platform): - Transient errors are silently requeued and only set Degraded=True after a threshold has elapsed (2m for sub-controllers, 2m30s for the main controller). lastTransientFailureAt is tracked so a stale failure window is restarted rather than causing immediate degradation when errors resume after a long gap. - Permanent errors set Degraded=True immediately and return nil so controller-runtime does not requeue; existing watches re-trigger reconciliation when the underlying data changes. - Infrastructure NotFound returns Available (nil) rather than Degraded, matching the main controller's existing behaviour. Reconcile methods use controller-runtime's TerminalError to mark permanent errors and a shared finalizeReconcile() deferred function (in reconcile_dispatch.go) that routes to the correct handler and clears the failure window on every nil-error return path. Additional improvements: - Wrap errors with %w to preserve context through the call chain. - Consolidate failure-tracking fields into a failureWindow struct with named staleness constants (noStalenessWindow, defaultStalenessWindow). - Fix race in cloud config sync test by using a separate reconciler instance for direct Reconcile() calls. - Fix aggregatedTransientDegradedThreshold mismatch in the main controller test that prevented the degraded condition from ever being set. - Add tests for partial-run degraded scenarios in the clusteroperator and trusted CA bundle controllers. - Use specific Gomega matchers or provide failure descriptions per project conventions. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

openshift-ci · 2026-07-08T13:50:28Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-07-08T13:50:32Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: acc65138-09a4-4a4a-93ab-074656ec4684

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

sync() returned nil on success, so Reconcile always called setStatusAvailable immediately after setStatusProgressing. Change sync to return (bool, error) so Reconcile can skip setStatusAvailable when resources were just updated. Also fix applyResources overwriting the updated flag each iteration — only the last resource's status was returned.

Updates the logic in the clusteroperator_controller to handle progressing more robustly. The previous fix showed a jump from 14 -> 85% in sippy, but was still possible to set progressing=True then immediately unset on next reconcile. This isn't that useful to a cluster admin trying to watch the progress of the rollout / upgrade. This change keeps Progressing=True until we see the deployment/daemonset actually stabilise.

openshift-ci-robot · 2026-07-08T15:17:26Z

@theobarberbany: This pull request references Jira Issue OCPBUGS-94183, which is invalid:

expected Jira Issue OCPBUGS-94183 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-94184, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

All three reconcilers (CloudConfigReconciler, TrustedCABundleReconciler, CloudOperatorReconciler) now distinguish transient errors (API blips) from permanent errors (corrupt config, unsupported platform). Transient errors are silently requeued and only set Degraded=True after a threshold (2m for sub-controllers, 2m30s for the main controller). Permanent errors degrade immediately without requeue.

Failure handling is consolidated into a shared finalizeReconcile() deferred function in reconcile_dispatch.go, using controller-runtime's TerminalError to classify errors.

Fix Progressing=True being immediately overwritten by Available — sync() now returns (bool, error) so Reconcile skips setStatusAvailable when resources were just updated.

This depends on #484 to avoid the race that required the revert previously.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

theobarberbany · 2026-07-08T15:38:01Z

/jira refresh

openshift-ci-robot · 2026-07-08T15:38:07Z

@theobarberbany: This pull request references Jira Issue OCPBUGS-94183, which is invalid:

expected dependent Jira Issue OCPBUGS-42837 to target a version in 5.0.0, but it targets "5.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

This pull request references Jira Issue OCPBUGS-94184, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-64852 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-64852 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
bug has dependents

Requesting review from QA contact:
/cc @miyadav

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

theobarberbany · 2026-07-08T15:38:33Z

/jira refresh

openshift-ci-robot · 2026-07-08T15:38:42Z

@theobarberbany: This pull request references Jira Issue OCPBUGS-94183, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-42837 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-42837 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
bug has dependents

Requesting review from QA contact:
/cc @sunzhaohua2

This pull request references Jira Issue OCPBUGS-94184, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-64852 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-64852 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
bug has dependents

Requesting review from QA contact:
/cc @miyadav

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-07-08T15:38:46Z

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: sunzhaohua2.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

@theobarberbany: This pull request references Jira Issue OCPBUGS-94183, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target version (4.22.0) matches configured target version for branch (4.22.0)

bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

release note type set to "Release Note Not Required"

dependent bug Jira Issue OCPBUGS-42837 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)

dependent Jira Issue OCPBUGS-42837 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0

bug has dependents

Requesting review from QA contact:
/cc @sunzhaohua2

This pull request references Jira Issue OCPBUGS-94184, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target version (4.22.0) matches configured target version for branch (4.22.0)

bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

release note type set to "Release Note Not Required"

dependent bug Jira Issue OCPBUGS-64852 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)

dependent Jira Issue OCPBUGS-64852 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0

bug has dependents

Requesting review from QA contact:
/cc @miyadav

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

theobarberbany · 2026-07-08T15:39:03Z

/pipeline auto

nrb

/approve

openshift-ci · 2026-07-08T20:29:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [nrb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

theobarberbany · 2026-07-08T21:01:09Z

/test e2e-aws-ovn-upgrade

openshift-ci · 2026-07-09T00:40:38Z

@theobarberbany: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

theobarberbany · 2026-07-09T11:40:37Z

/hold

openshift-ci-robot · 2026-07-14T16:37:07Z

@theobarberbany: This pull request references Jira Issue OCPBUGS-94183. The bug has been updated to no longer refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-94184. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

All three reconcilers (CloudConfigReconciler, TrustedCABundleReconciler, CloudOperatorReconciler) now distinguish transient errors (API blips) from permanent errors (corrupt config, unsupported platform). Transient errors are silently requeued and only set Degraded=True after a threshold (2m for sub-controllers, 2m30s for the main controller). Permanent errors degrade immediately without requeue.

Failure handling is consolidated into a shared finalizeReconcile() deferred function in reconcile_dispatch.go, using controller-runtime's TerminalError to classify errors.

Fix Progressing=True being immediately overwritten by Available — sync() now returns (bool, error) so Reconcile skips setStatusAvailable when resources were just updated.

This depends on #484 to avoid the race that required the revert previously.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 8, 2026

theobarberbany added 3 commits July 8, 2026 15:18

Transient Degraded=true test

c50f7d0

theobarberbany changed the title ~~Add transient error handling across all three controllers~~ [release-4.22] Add transient error handling across all three controllers Jul 8, 2026

theobarberbany marked this pull request as ready for review July 8, 2026 15:16

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 8, 2026

openshift-ci Bot requested review from damdo and racheljpg July 8, 2026 15:16

theobarberbany changed the title ~~[release-4.22] Add transient error handling across all three controllers~~ OCPBUGS-94183, OCPBUGS-94184: [release-4.22] Add transient error handling across all three controllers Jul 8, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 8, 2026

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jul 8, 2026

openshift-ci Bot requested a review from miyadav July 8, 2026 15:38

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 8, 2026

theobarberbany changed the title ~~OCPBUGS-94183, OCPBUGS-94184: [release-4.22] Add transient error handling across all three controllers~~ OCPBUGS-94183, OCPBUGS-94184: [release-4.22] Add transient error handling across all three controllers, fix progressing=true Jul 8, 2026

nrb approved these changes Jul 8, 2026

View reviewed changes

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 8, 2026

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2026

theobarberbany closed this Jul 14, 2026

Uh oh!

Conversation

theobarberbany commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 8, 2026

Uh oh!

coderabbitai Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

openshift-ci-robot commented Jul 8, 2026

Uh oh!

theobarberbany commented Jul 8, 2026

Uh oh!

openshift-ci-robot commented Jul 8, 2026

Uh oh!

theobarberbany commented Jul 8, 2026

Uh oh!

openshift-ci-robot commented Jul 8, 2026

Uh oh!

openshift-ci Bot commented Jul 8, 2026

Uh oh!

theobarberbany commented Jul 8, 2026

Uh oh!

nrb left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 8, 2026

Uh oh!

theobarberbany commented Jul 8, 2026

Uh oh!

openshift-ci Bot commented Jul 9, 2026

Uh oh!

theobarberbany commented Jul 9, 2026

Uh oh!

openshift-ci-robot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theobarberbany commented Jul 8, 2026 •

edited

Loading

coderabbitai Bot commented Jul 8, 2026 •

edited

Loading