Skip to content

fix(logmq): filter unsubscribed operator events before suppression - #1050

Open
samadalishah wants to merge 1 commit into
hookdeck:mainfrom
samadalishah:fix/filter-unsubscribed-operator-events-before-suppression
Open

fix(logmq): filter unsubscribed operator events before suppression#1050
samadalishah wants to merge 1 commit into
hookdeck:mainfrom
samadalishah:fix/filter-unsubscribed-operator-events-before-suppression

Conversation

@samadalishah

Copy link
Copy Markdown

Summary

Filter unsubscribed operator events before entering their delivery suppression window.

This prevents an event that will never be emitted from:

  • reading or writing suppression state in Redis;
  • failing log processing because of an unrelated Redis error;
  • producing a misleading opevent delivery failed log;
  • nacking and redelivering an otherwise successfully persisted log message.

Problem

Operator-event topic filtering currently happens inside Emitter.Emit.

For events with a suppression key, logmq.BatchProcessor.send enters the Redis-backed suppression window before calling Emitter.Emit. This means the topic filter is applied too late.

For example:

OPERATOR_EVENTS_TOPICS=alert.destination.disabled

When an attempt exhausts its retries, the alert evaluator can still plan an alert.attempt.exhausted_retries event. Although that topic is not subscribed, the previous delivery path was:

  1. Build the exhausted-retries event.
  2. Enter its Redis suppression window.
  3. Only then call Emitter.Emit.
  4. Emitter.Emit sees that the topic is disabled and discards it.

As a result, an unsubscribed event could fail before reaching the component responsible for filtering it:

opevent delivery failed
topic=alert.attempt.exhausted_retries
error=context deadline exceeded

The log message was then nacked even though no exhausted-retries event was configured for delivery.

Root cause

Topic eligibility was checked after optional delivery-layer behavior:

suppression window -> emitter -> topic filter

Filtering needs to happen before any behavior specific to delivering the event:

topic filter -> suppression window -> emitter

Fix

BatchProcessor.send now checks Emitter.Enabled before entering the suppression window:

if !bp.alerts.Emitter.Enabled(de.event.Topic) {
    return nil
}

Emitter.Emit retains its existing topic check as a boundary safeguard.

Subscribed events continue through the same suppression and delivery path as before.

Tests

Added a regression test covering the reported configuration:

  • only alert.destination.disabled is subscribed;
  • the evaluated attempt also exhausts its retries;
  • the exhausted-retries suppression dependency is configured to return context.DeadlineExceeded;
  • the log message is acknowledged;
  • alert.destination.disabled is delivered;
  • alert.attempt.exhausted_retries is not delivered;
  • the exhausted-retries suppression window is never called.

The existing delivery-suppression tests continue to cover subscribed exhausted-retry events, including:

  • sequential window suppression;
  • per-destination suppression;
  • tenant isolation;
  • concurrent suppression conflicts;
  • suppression cleanup after delivery failure.

Impact

  • No configuration or API changes.
  • No behavior change for subscribed operator-event topics.
  • Unsubscribed topics no longer perform delivery-related Redis work.
  • Redis failures for unsubscribed topics can no longer nack log messages.
  • Avoids misleading delivery-failure logs for events that were intentionally filtered.

Test command

go test ./internal/logmq

For maintainers

Please review if the change makes sense in a broader scope that I may be missing here as I have found this issue in my use case where I am only interested in alert.destination.disabled events running 3 replicas of outpost-log service specifically.

@samadalishah
samadalishah marked this pull request as ready for review August 25, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant