Operator stops reconciling permanently after its watch connections drop while idle

[sweb]

Affected Stackable version

26.7

Affected Trino version

481

Current and expected behavior

The trino, hive and airflow operators flip from recoverable watch-stream errors to a permanent failed to start watching object: client error (Connect) loop and never reconcile again until the pod is restarted. While wedged the operator looks healthy to Kubernetes: the process is running, its conversion webhook still answers.

Expected

A controller whose watch connection is dropped re-establishes it and continues reconciling, as it does for the first several drops.

Actual

ERROR kube_client::client::builder: failed with error client error (Connect)
WARN stackable_operator::logging::controller: Queued reconcile resulted in an error
  controller.name="trinocluster.trino.stackable.tech"
  error=failed to start watching object: ServiceError: client error (Connect)
  error.sources=[ServiceError: client error (Connect), client error (Connect), deadline has elapsed]

The problems seems to be upstream in kube-rs: https://github.com/kube-rs/kube/blob/main/kube-runtime/src/controller/mod.rs#L1704
Possible explanation for this issue: All watch triggers get merged into a single stream that then gets joint backoff - so if an operator watches e.g. 7 resources, the backoff fires instantly 6 times, which quickly leads to high sleep durations that ultimately go beyond the timeout.

While kube-rs has a long timeout configured (>200 secs) on the observed cluster some kind of proxy for the apiserver seems to terminate idle watches after 60 seconds, which is reliably triggered for the mentioned operators.

Possibly related: kube-rs/kube#1915

As mentioned in the beginning, this potentially affects all our operators

Possible solution

Possible Workarounds:

Smaller, constant backoff

    use kube::runtime::utils::Backoff;

    struct ConstantBackoff(Duration);
    impl Iterator for ConstantBackoff {
        type Item = Duration;
        fn next(&mut self) -> Option<Duration> { Some(self.0) }
    }
    impl Backoff for ConstantBackoff {
        fn reset(&mut self) {}
    }

    Controller::new(api, watcher::Config::default())
        .trigger_backoff(ConstantBackoff(Duration::from_secs(2)))

Reduce timeout on watcher side:

Maybe as an optional parameter or something.

const WATCH_TIMEOUT_SECS: u32 = 40;

fn watch_config() -> watcher::Config {
    watcher::Config::default().timeout(WATCH_TIMEOUT_SECS)
}

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[lfrancke]

Whatever we might fix upstream or in our code: Should something like this not also trigger a condition that marks our operator as unhealthy?

[sweb]

Whatever we might fix upstream or in our code: Should something like this not also trigger a condition that marks our operator as unhealthy?

A regular probe would not detect this - you would have to check the connectivity between controller and api server.