feat: migrate push to SSE with WebSocket rollout compatibility

[niemyjski]

Why

Use SSE for server-to-client updates in the Svelte app while retaining WebSocket support for the legacy Angular app and cached clients during rollout.

What changed

• Serve authenticated SSE at /api/v2/push, preserving the existing snake_case message payloads and WebSocket upgrade path.
• Route messages and revocations to connections owned by each API replica. Redis leases coordinate connection limits and expire after replica loss.
• Bound SSE queues, coalesce repeated invalidations, prioritize critical notifications, and reconnect clients to refresh state after overflow or disconnection.
• Handle OAuth token revocation and membership removal without expanding scoped tokens beyond their organization grants. Refresh client caches when membership changes.
• Keep EnableWebSockets as an alias for EnablePush; add transport metrics, shutdown cleanup, HTTP examples, and the SSE OpenAPI contract.

Rollout prerequisites for production AKS

The chart keeps push disabled. Configure and validate the ingress for long-lived streaming before enabling it. Monitor both transport connection counts and retain the WebSocket compatibility path while legacy clients still use it.

Tests

• Backend test-project build: no warnings or errors; 76 focused transport, access-control, configuration, logging, and throttling tests passed.
• Frontend npm run validate, 757 unit tests, and production build passed. The isolated Chromium SSE-abort regression passed.
• OpenAPI was regenerated from the local app and its four checks passed before the final transport fixes. The final service-backed rerun, HTTP/Redis integration tests, and full browser flows remain blocked by the shared Elasticsearch cluster's inactive primary shards; no runtime or deployment readiness is claimed.
• Regression checks reproduced the original missing OAuth revocation, Redis-only lease registration, premature SSE writes, delayed lease release, and blocked WebSocket teardown before validating their fixes.

Non-obvious trade-offs

Push delivery is best-effort without event replay. Clients refetch current state on reconnect. Redis-backed deployments coordinate connection limits; the in-memory fallback remains process-local.

[Copilot]

Pull request overview

This PR migrates real-time push from WebSockets to SSE while keeping a temporary WebSocket compatibility path for rolling deploys, updates both frontends to consume SSE, and adds rollout/shutdown configuration for long-lived push connections.

Changes:

• Adds SSE connection, middleware, broker fanout, diagnostics, and EnablePush configuration.
• Replaces Svelte/Angular WebSocket clients with fetch/stream-based SSE clients and updates tests.
• Adds Kubernetes drain settings and documents push being disabled until ingress supports SSE.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/Exceptionless.Web/Hubs/SseConnection.cs	Adds bounded SSE write queue, deduplication, keep-alives, and disposal.
src/Exceptionless.Web/Hubs/SseConnectionManager.cs	Manages SSE connections, keep-alives, cleanup, and metrics.
src/Exceptionless.Web/Hubs/SseMiddleware.cs	Adds authenticated /api/v2/push SSE endpoint.
src/Exceptionless.Web/Hubs/WebSocketPushMiddleware.cs	Adds temporary WebSocket compatibility middleware.
src/Exceptionless.Web/Hubs/WebSocketConnectionManager.cs	Reworks WebSocket manager for compatibility fanout and metrics.
src/Exceptionless.Web/Hubs/MessageBusBroker.cs	Fans out push messages to SSE and WebSocket managers.
src/Exceptionless.Web/Hubs/MessageBusBrokerMiddleware.cs	Removes old WebSocket-only middleware.
src/Exceptionless.Web/Startup.cs	Wires SSE and WebSocket push middleware behind EnablePush.
src/Exceptionless.Web/Program.cs	Adds host shutdown timeout for drain alignment.
src/Exceptionless.Web/Bootstrapper.cs	Registers the SSE connection manager.
src/Exceptionless.Web/Utility/Handlers/ThrottlingMiddleware.cs	Renames push throttle exemption to SSE-oriented naming.
src/Exceptionless.Core/Configuration/AppOptions.cs	Replaces EnableWebSockets with EnablePush and legacy fallback.
src/Exceptionless.Core/Bootstrapper.cs	Updates push-disabled startup warning.
src/Exceptionless.Core/Utility/AppDiagnostics.cs	Adds SSE/WebSocket push connection counters.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.svelte.ts	Adds Svelte SSE client with reconnect and stream parsing.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.test.ts	Adds Svelte SSE client unit tests.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.svelte.ts	Removes Svelte WebSocket client.
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.test.ts	Removes WebSocket client tests.
src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte	Switches app layout push integration to SSE.
src/Exceptionless.Web/ClientApp.angular/components/websocket/websocket-service.js	Rewrites Angular legacy push service to SSE.
tests/Exceptionless.Tests/Hubs/SseTests.cs	Adds SSE manager, broker, and deduplication tests.
tests/Exceptionless.Tests/Hubs/SseIntegrationTests.cs	Adds HTTP pipeline tests for SSE endpoint behavior.
tests/Exceptionless.Tests/Hubs/FakeHttpResponse.cs	Adds fake response helper for SSE tests.
tests/Exceptionless.Tests/Hubs/WebSocketCompatibilityTests.cs	Adds WebSocket compatibility and dual-fanout tests.
tests/Exceptionless.Tests/Hubs/WebSocketTests.cs	Removes old WebSocket broker tests.
tests/Exceptionless.Tests/Hubs/WebSocketConnectionManagerTests.cs	Removes old WebSocket manager tests.
tests/Exceptionless.Tests/Hubs/TestWebSocket.cs	Updates test WebSocket encoding and close count layout.
tests/Exceptionless.Tests/appsettings.yml	Enables push for test host configuration.
tests/http/push.http	Adds manual SSE smoke requests for localhost.
k8s/exceptionless/templates/api.yaml	Adds drain lifecycle settings and disables push pending ingress migration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

[niemyjski]

Addressed the outstanding review feedback in the latest pushes: restored the Angular legacy client to the existing WebSocket path with only a deprecation note, tightened SSE/WebSocket lifecycle cleanup, fixed the SSE queue and keepalive backpressure edge case, and stopped the Svelte client from retrying when push is disabled.\n\nLocal validation is green (, , , , and Using launch settings from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/Properties/launchSettings.json...
Running tests from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+94/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(2s)

[+94/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(5s)

[+239/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(8s)

[+338/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(11s)

[+483/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(14s)

[+506/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(17s)

[+569/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(20s)

[+684/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(23s)

[+743/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(26s)

[+835/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(29s)

skipped Exceptionless.Tests.Pipeline.EventPipelineTests.GeneratePerformanceDataAsync (0ms)
Used to create performance data from the queue directory
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+900/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(32s)

[+1042/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(35s)

[+1151/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(38s)

skipped Exceptionless.Tests.Repositories.EventRepositoryTests.GetAsyncPerformanceAsync (0ms)
Performance Testing
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
skipped Exceptionless.Tests.Repositories.EventRepositoryTests.GetAsync (0ms)
elastic/elasticsearch-net#2463
from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64)
[+1236/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(41s)

[+1277/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(44s)

[+1424/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(47s)

[+1567/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(50s)

[+1650/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(53s)

[+1792/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(56s)

[+1864/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(59s)

[+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 02s)

[+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 05s)

[+1874/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 08s)

[+1875/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 11s)

[+1876/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 14s)

/Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64) passed (1m 14s 985ms)

Test run summary: Passed!
total: 1880
failed: 0
succeeded: 1877
skipped: 3
duration: 1m 15s 407ms), the PR checks are green, and the open inline threads have been replied to and resolved.

[ejsmith]

The steady-state SSE fanout design should not require sticky sessions: each long-lived stream remains local to the replica that accepted it, Redis pub/sub delivers notifications to every API replica, and only the replica owning a given local connection writes to it. Reconnects can land on any replica.

However, the shared connection metadata currently has two scale-out failure modes that should be addressed before enabling push in production:

1. Authentication-token revocation can be consumed destructively by a non-owning replica, leaving the owning replica's stream connected.
2. Redis connection reservations have no lease or expiry, so an ungraceful pod/node failure can leave permanent entries that eventually cause users with no live streams to receive 429 responses.

The current tests use a single host/manager and cannot expose these cases. Please add a two-replica test with shared connection state/message bus covering cross-replica delivery and token revocation, plus stale-reservation behavior after simulated replica loss.

[niemyjski]

Follow-up feedback audit complete at 5ee86353a9412d2d19394025fccee7b4dea25b01.

• Live inventory before this summary: 37 review threads, 70 inline comments, 42 submitted reviews, and 2 issue comments. All 37 threads were already resolved and none were open, so there was no open thread to reply to or resolve.
• Classification: 4 live-line findings are already fixed/current (EnablePush with legacy EnableWebSockets, SSE resource disposal, and fetch-based SSE auth); 31 old bot/Copilot findings are outdated or superseded by the lifecycle redesign and deleted code; 2 human scale-out findings are already fixed by process-local ownership and expiring Redis leases. Narrow exception-allowlist prescriptions were not safe as written, but their cleanup and backpressure concerns are covered by the current design and tests.
• The one still-actionable item was @ejsmith requested true two-replica shared-message-bus regression coverage. RCA: production fanout/revocation behavior existed, but the test called handlers directly and therefore did not prove bus subscription behavior across replicas. Commit 5ee86353a adds a shared-bus test proving fanout reaches both local replicas and token revocation closes only the owner replica connection while preserving the non-owner connection.
• Focused proof: SSE broker 10/10, manager 7/7, deduplication 9/9, HTTP integration 7/7, WebSocket compatibility 14/14, in-memory leases 2/2, real Redis lease expiry 1/1, and push option compatibility 3/3. dotnet build Exceptionless.slnx --no-restore passed with 0 warnings and 0 errors.
• Full thermo-nuclear branch-vs-main review found no remaining structural, concurrency, authorization, resource-cleanup, compatibility, duplication, atomicity, or file-size blocker; the largest changed file is 770 lines.
• CI on this exact head is terminal green: API coverage, E2E, client, Docker, website, version, and CLA all passed.

GitHub still reports the historical CHANGES_REQUESTED decision; I did not dismiss it. The requested production fixes and combined replica coverage are now present for reviewer re-approval.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	38%	35%	290	❌
Exceptionless.Core	76%	67%	10203	✔
Exceptionless.Web	85%	70%	8078	✔
Exceptionless.AppHost	38%	41%	147	❌
Summary	79% (25719 / 32669)	68% (12041 / 17734)	18718	✔