test(integration): retry transient 533 in testSynchroniseUserRemovalNotification - #5442
Open
blackheaven wants to merge 1 commit into
Open
Conversation
…otification
Conversation.testSynchroniseUserRemovalNotification[domain=other] failed
intermittently in CI: creating a Proteus conversation with members on two
remote backends returned HTTP 533 with
`unreachable_backends: [<run>-fed2-fed…]` instead of the expected 201. The
sibling fed-v0/v1/v2 variants passed in the same run and the body named only the
static fed2 backend as unreachable (the dynamic backend was reachable),
indicating a transient federation-ping failure rather than a hard outage.
Root cause: conversation creation fans out to every remote backend and pings
each in registerRemoteConversationMemberships
(Wire.ConversationSubsystem.Util). ensureNoUnreachableBackends
(Wire.FederationSubsystem) fails fast on the first unreachable backend with zero
retries, so a single transient ping failure (connection refused, TLS handshake,
DNS) surfaces as 533.
Fix is test-layer only; production stays fail-fast. Promote the existing
retryTransient helper from a local `where` binding in
Test/Migration/Conversation.hs to a shared top-level definition in Testlib.App
(re-exported via Testlib.Prelude), wrap the postConversation call in
testSynchroniseUserRemovalNotification, and delete the now-duplicated local
copy. retryTransient retries on {500,422,521,525,533} with exponential backoff
capped at a cumulative 30s, so genuine failures still surface. The single
call-site edit covers all four StaticDomain variants (other/fed-v0/v1/v2) since
they share it.
The retry is idempotent: on 533, deleteOnUnreachable
(Wire.ConversationSubsystem.Util) deletes the local conversation before
re-throwing, so re-issuing the create cannot corrupt state.
Verified: `cabal build integration` is clean under -Werror (the move surfaced and
fixed a now-redundant `import Control.Concurrent (threadDelay)` in the migration
file); `./dist/integration --list` confirms discovery. Runtime confirmation of
the four variants is CI-only (integration tests require the k8s federation
stack).
zebot
added
the
ok-to-test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
changelog.d