Repository navigation
Conversation
When HA deduplication is enabled for a tenant the distributor removes the replica label from the stored series, so a remote read query carrying that label (Prometheus adds its external labels to the matchers) never matches anything. Drop matchers on the tenant's HA replica label in the remote read handler for tenants with HA deduplication enabled. Fixes cortexproject#3714 Signed-off-by: Srinivas Chippagiri <cvas22@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
When HA deduplication is enabled for a tenant, the distributor removes the HA replica label (
__replica__by default) from the samples it stores. Prometheus adds its external labels, including that replica label, to the matchers of every remote read request. So a Prometheus HA pair that remote writes to Cortex and also remote reads from it gets no data back. The docs worked around this by saying to keep the replica label out of the external labels used for remote read.As @yeya24 suggested on the issue, I made the querier's remote read handler ignore matchers on the tenant's HA replica label (
-distributor.ha-tracker.replica, or the per-tenant override), but only for tenants with HA deduplication enabled (accept_ha_samples). Other tenants behave as before. With multiple tenants in the org ID, the replica labels of every HA-enabled tenant are dropped. If dropping them would leave a query with no matchers at all, the original matchers are kept, so a request never turns into a match-everything query.RemoteReadHandlernow takes the overrides as an extra argument (pkg/apiandpkg/cortex/modules.gopass them through). I also updated the HA pair handling guide.Which issue(s) this PR fixes:
Fixes #3714
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flagsNotes
TestRemoteReadHandler_HAReplicaLabel. It covers a tenant with HA deduplication enabled, a tenant without it, and a request where the replica matcher is the only matcher. It passes, as dogo vetandgo test ./pkg/api/.TestRemoteReadHandlerpanics inside gogo protobuf'sMarshal(makeslice: cap out of range) under Go 1.27.3. It panics the same way on master without this change, so I'm relying on CI for it.