Supersedes #3445, which investigated the wrong path (shared InformerEventSource). This issue identifies the actual root cause in EventFilterWindow.
Summary
When a controller uses SSA (Server-Side Apply) and a foreign actor concurrently changes the same resource's spec, the EventFilterWindow can silently drop the foreign change. The controller never re-reconciles and is stuck until maxReconciliationInterval (default 10 hours).
Mechanism
- Controller reconciles resource at gen=1
- Foreign actor changes spec → gen=2 → new RV on API server
- Controller's SSA
patchResource merges on top → API returns RV N (gen=2, because SSA merged with the already-modified resource)
- The fabric8 informer coalesces both watch events (same resource key) into a single event at RV N
EventFilterWindow records own write RV=N, sees relatedEvents={N} == ownResourceVersions={N} → drops the event (EventFilterWindow.java line 182–186)- Controller finishes gen=1 reconciliation successfully, no pending events → rescheduled for
maxReconciliationInterval
- Resource has gen=2 but
observedGeneration=1 — permanently stale until the 10-hour timer fires
Why SSA is specifically affected
With non-SSA updates (full PUT), a concurrent foreign change causes a 409 Conflict, forcing the controller to retry with the current resource state. With SSA, patches succeed even when the resource was concurrently modified — different field managers don't conflict. The API response RV carries both changes, but EventFilterWindow treats it as a pure own-write echo.
Why this didn't happen before 5.3.x
The EventFilterWindow buffering mechanism was introduced in 5.3.x. It captures events that arrive during the API call window (startEventFilteringModify → doneEventFilterModify). When the informer coalesces the foreign change event with the own-write echo into a single event, the window sees only one event at the own-write RV and drops it.
In 5.2.x, the simpler TemporaryResourceCache / canSkipEvent approach checked events as they arrived. With different timing characteristics (no synchronized onAddOrUpdate, no buffering window), the foreign event was more likely to arrive and be processed before the own-write RV was recorded in the cache.
Observed in production
This was observed in a Kroxylicious operator GHA failure when upgrading from JOSDK 5.2.5 to 5.3.5. A dependent controller sees the stale resource (gen=2, observedGeneration=1), exhausts retries, and also goes to sleep for maxReconciliationInterval.
Supersedes #3445, which investigated the wrong path (shared InformerEventSource). This issue identifies the actual root cause in
EventFilterWindow.Summary
When a controller uses SSA (Server-Side Apply) and a foreign actor concurrently changes the same resource's spec, the
EventFilterWindowcan silently drop the foreign change. The controller never re-reconciles and is stuck untilmaxReconciliationInterval(default 10 hours).Mechanism
patchResourcemerges on top → API returns RV N (gen=2, because SSA merged with the already-modified resource)EventFilterWindowrecords own write RV=N, seesrelatedEvents={N} == ownResourceVersions={N}→ drops the event (EventFilterWindow.javaline 182–186)maxReconciliationIntervalobservedGeneration=1— permanently stale until the 10-hour timer firesWhy SSA is specifically affected
With non-SSA updates (full PUT), a concurrent foreign change causes a 409 Conflict, forcing the controller to retry with the current resource state. With SSA, patches succeed even when the resource was concurrently modified — different field managers don't conflict. The API response RV carries both changes, but
EventFilterWindowtreats it as a pure own-write echo.Why this didn't happen before 5.3.x
The
EventFilterWindowbuffering mechanism was introduced in 5.3.x. It captures events that arrive during the API call window (startEventFilteringModify→doneEventFilterModify). When the informer coalesces the foreign change event with the own-write echo into a single event, the window sees only one event at the own-write RV and drops it.In 5.2.x, the simpler
TemporaryResourceCache/canSkipEventapproach checked events as they arrived. With different timing characteristics (no synchronizedonAddOrUpdate, no buffering window), the foreign event was more likely to arrive and be processed before the own-write RV was recorded in the cache.Observed in production
This was observed in a Kroxylicious operator GHA failure when upgrading from JOSDK 5.2.5 to 5.3.5. A dependent controller sees the stale resource (
gen=2, observedGeneration=1), exhausts retries, and also goes to sleep formaxReconciliationInterval.