docs: Guide on how to setup UTF-8 mode in Cortex by RaphSku · Pull Request #7439 · cortexproject/cortex

RaphSku · 2026-04-18T13:11:24Z

What this PR does:
Introduces a Guide on how to setup UTF-8 mode in Cortex

Which issue(s) this PR fixes:
Fixes #7091

Checklist

Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

SungJin1212 · 2026-04-19T23:46:41Z

+name_validation_scheme: utf8
+```
+
+### Impact on Cortex components


Since the Distributor is a sub-item, it would be better to increase to the ##.

Suggested change

### Impact on Cortex components

## Impact on Cortex components

Typo, thank you for pointing it out

SungJin1212 · 2026-04-19T23:48:26Z

@RaphSku
In the "Impact on Cortex components" section, an explanation is needed regarding how Ruler and Alertmanager work when we enable the utf8.

SungJin1212 · 2026-04-19T23:50:00Z

 * [FEATURE] Ingester: Add experimental active series queried metric. #7173
 * [FEATURE] Update prometheus Alertmanager version to v0.31.1 and add new integration to IncidentIO and Mattermost. #7092 #7267
 * [FEATURE] Tenant Federation: Add experimental support for partial responses using the `-tenant-federation.allow-partial-data` flag. When enabled, failures from individual tenants during a federated query are treated as warnings, allowing results from successful tenants to be returned. #7232
+* [ENHANCEMENT] Docs: Guide on UTF-8 Support in Cortex #7091


I don't think it's necessary to leave it in the changelog.

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

RaphSku · 2026-05-03T19:29:59Z

@RaphSku In the "Impact on Cortex components" section, an explanation is needed regarding how Ruler and Alertmanager work when we enable the utf8.

I've added the mentioned components to the guide, thank you for the review. If there is something in particular I should mention about either component, let me know.

SungJin1212 · 2026-05-04T12:53:39Z

@RaphSku
To make it even more practical for users, I think it would be highly beneficial to explicitly cover the scenarios we are validating in utf8_test.go.

Querying: Mention that PromQL queries, as well as API endpoints like LabelNames, Series, and LabelValues, fully support UTF-8 (e.g., querying {"test.utf8.metric"="😄"}).
Alertmanager: Highlight that UTF-8 can be safely used in config matchers, group_by fields, and Silence matchers.
Ruler: Note that rule names, rule labels, and rule group labels support UTF-8 characters.

Adding a few short YAML or PromQL snippets based on our tests would perfectly bridge the gap between the configuration and actual usage.

…tilize UTF-8 support Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

RaphSku · 2026-05-10T17:13:50Z

@SungJin1212 Thank you again for your comment, totally valid to have such a section and a good idea. I have included it in the latest commit.

SungJin1212 · 2026-05-11T10:08:30Z

+series.1{"test.utf8.metric"="😄"}
+```
+
+Note, that the metric name `series.1` is not valid under the `legacy` validation scheme. It is accepted only when UTF-8 validation is enabled. This is useful when metric or label names contain characters, such as `.`, that are supported by UTF-8 validation but not by the legacy validation scheme.


Suggested change

Note, that the metric name `series.1` is not valid under the `legacy` validation scheme. It is accepted only when UTF-8 validation is enabled. This is useful when metric or label names contain characters, such as `.`, that are supported by UTF-8 validation but not by the legacy validation scheme.

Note that the metric name `series.1` is not valid under the `legacy` validation scheme. It is accepted only when UTF-8 validation is enabled. This is useful when metric or label names contain characters, such as `.`, that are supported by UTF-8 validation but not by the legacy validation scheme.

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

RaphSku · 2026-09-08T17:24:44Z

@friedrichg friendly ping :)

sandy2008

Thanks, the earlier review points are all in. I checked the examples and the component notes against the code: two of the snippets don't parse, and the Alertmanager and notifier paragraphs describe a different mechanism from what the code does. Details inline.

sandy2008 · 2026-10-28T11:51:32Z

+PromQL queries support UTF-8 metric names and label names. For example, a UTF-8 metric name can be queried with the following expression:
+
+```promql
+series.1{"test.utf8.metric"="😄"}


This doesn't parse: the vendored PromQL parser stops at .1 (1:7: parse error: unexpected number ".1"). A metric name that isn't a legacy identifier has to go inside the braces as a quoted string, which is also what integration/utf8_test.go queries:

Suggested change

series.1{"test.utf8.metric"="😄"}

{"series.1", "test.utf8.metric"="😄"}

You are totally right, my mistake.

sandy2008 · 2026-10-28T11:51:33Z

+
+```yaml
+matchers:
+  - "🙂.utf8.label"


As written this isn't valid YAML (the list item is a bare string followed by value/isRegex keys); each matcher needs a name field, like the silence the integration test creates:

Suggested change

- "🙂.utf8.label"

- name: "🙂.utf8.label"

Yes, for some reason forgot to include name, thank you for pointing out.

sandy2008 · 2026-10-28T11:51:33Z

+
+Ruler external labels are also validated with the selected scheme. With `legacy`, external label names must follow the legacy validation rules. With `utf8`, external label names that are valid under the UTF-8 validation scheme can be used.
+
+The Ruler also passes the configured validation scheme to the notifier that sends alerts to Alertmanager. As a result, alert labels handled by the notification path are validated according to the selected scheme before alerts are sent.


I don't think this validates alert labels: the vendored notifier.NewManager only uses the scheme as the default for alert relabel configs that don't set their own, and the Cortex ruler doesn't configure any. I'd drop this paragraph.

Dropped, now it should be correct since rule definitions are validated in /pkg/ruler/manager.go and external labels in /pkg/util/validation/limits.go. I hope I didn't miss anything else. Thank you for the information regarding notifier.NewManager.

sandy2008 · 2026-10-28T11:51:33Z

+
+### Alertmanager
+
+The `name_validation_scheme` setting is not applied directly by Cortex [Alertmanager](https://cortexmetrics.io/docs/architecture/#alertmanager) in the same way it is applied by the Distributor or Ruler. Its effect on Alertmanager is indirect. Alerts sent by the Ruler pass through the Ruler notifier, which is created with the configured validation scheme. As a result, Alertmanager may receive alert labels that were validated according to the selected scheme.


This one is applied directly: initAlertManager maps name_validation_scheme to Alertmanager's matcher mode (legacy to classic-mode, utf8 to utf8-strict-mode) and calls compat.InitFromFlags, which decides how matchers in the Alertmanager configuration are parsed and how label names such as group_by entries are validated. So the Alertmanager examples above only work with utf8, and switching to utf8 moves existing Alertmanager configs to the UTF-8 matcher parser with no classic fallback, which seems worth a line in the rollout notes.

You are right, thank you for pointing that out, rewritten that section and improved recommended rollout strategy section.

- Fixing PromQL expression since it doesn't parse. - Matchers missed "name" key and converted to JSON to better represent the API body. - Removed ruler paragraph claiming validating alert labels. - Correcting Alertmanager section since `name_validation_scheme` is directly mapped to Alertmanager's matcher modes. - Improving recommended rollout strategy. Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

RaphSku · 2026-10-28T21:20:18Z

@sandy2008 Thank you for your detailed review, all very good points. I have tried to include them and improve/fix the guide.

sandy2008

Re-checked at c4791e0 against the code. The PromQL example now parses with the vendored parser (the old series.1{...} form fails with unexpected number ".1"), the silence matcher has name/value/isRegex/isEqual and unmarshals into Alertmanager's PostableSilence, the notifier paragraph is gone, and the Ruler and Alertmanager sections match ValidateRuleGroup, the ruler_external_labels check in Limits.Validate and initAlertManager. I also loaded the examples: the route config loads in utf8-strict-mode and is rejected in classic-mode, a classic-only matcher like foo=bar baz fails in strict mode as the rollout note warns, and the rule group parses with utf8 and is rejected with legacy. Thanks for the rewrite.

SungJin1212 · 2026-10-30T06:01:22Z

+## Recommended rollout strategy
+
+1. Inventory metric producers and consumers, Ruler rules, external labels, and Alertmanager configurations. Check whether any tools assume legacy metric and label names.
+2. Verify that existing Alertmanager route matchers parse in UTF-8 strict mode. Update ambiguous or incompatible matchers before switching modes.
+3. Enable `utf8` in a staging environment. Ingest and query representative metrics with non-legacy names, and exercise label name, label value, and series API endpoints.
+4. Load representative recording and alerting rules, including group and rule labels, and verify evaluation and notifications.
+5. Test Alertmanager configuration loading, grouping, routing, and silence creation. Roll out consistently across the Cortex components that need the selected mode.


One thing that might be worth covering: what happens when switching back from utf8 to legacy.

Series already ingested with UTF-8 names stay in storage and remain queryable, but new writes with such names are rejected by the Distributor.

Stored rule groups still load, but rule outputs with UTF-8 names are rejected when written, and rules with UTF-8 record names or rule labels can no longer be uploaded or updated through the Ruler API.

Alertmanager configurations that rely on UTF-8 matchers or group_by label names fail to load in classic-mode, and alerts with UTF-8 label names are rejected.

ruler_external_labels and metric_relabel_configs with UTF-8 label names fail limits validation: in the default limits Cortex refuses to start, and in per-tenant overrides the runtime config file is rejected.

Before switching back from utf8 to legacy, remove or rename UTF-8 metric and label names from Ruler rule groups, ruler_external_labels, metric_relabel_configs, and Alertmanager configurations.

Could you add a short "Rolling back to legacy" section with this after the rollout steps?

Added to the guide. Thank you for the very good addition.

- Addressing Distributor, Ruler, Alertmanager and Limits validation that are affected by the potential rollback. Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

RaphSku · 2026-11-02T20:47:29Z

@SungJin1212 Could you please rerun the failed workflow run? Seems to be flaky since the change is unrelated. Thank you.

Guide on how to setup UTF-8 mode on Cortex

d561d5c

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

pull-request-size Bot added the size/M label Apr 18, 2026

dosubot Bot added the component/documentation label Apr 18, 2026

SungJin1212 reviewed Apr 19, 2026

View reviewed changes

Added ruler and alertmanager as affected components

308fad2

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

Added a Using UTF-8 section to demonstrate a few examples on how to u…

e45f856

…tilize UTF-8 support Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

pull-request-size Bot added size/L and removed size/M labels May 10, 2026

SungJin1212 reviewed May 11, 2026

View reviewed changes

SungJin1212 requested a review from friedrichg May 11, 2026 10:09

Punctuation fix

80309fb

Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

sandy2008 reviewed Oct 28, 2026

View reviewed changes

RaphSku requested a review from a team as a code owner October 28, 2026 21:14

sandy2008 approved these changes Oct 29, 2026

View reviewed changes

SungJin1212 reviewed Oct 30, 2026

View reviewed changes

Add rollback to legacy section to improve completeness of the guide.

6c886d5

- Addressing Distributor, Ruler, Alertmanager and Limits validation that are affected by the potential rollback. Signed-off-by: Raphael Skuza <rapsku.dev@gmail.com>

SungJin1212 approved these changes Nov 2, 2026

View reviewed changes

	### Impact on Cortex components
	## Impact on Cortex components

	Note, that the metric name `series.1` is not valid under the `legacy` validation scheme. It is accepted only when UTF-8 validation is enabled. This is useful when metric or label names contain characters, such as `.`, that are supported by UTF-8 validation but not by the legacy validation scheme.
	Note that the metric name `series.1` is not valid under the `legacy` validation scheme. It is accepted only when UTF-8 validation is enabled. This is useful when metric or label names contain characters, such as `.`, that are supported by UTF-8 validation but not by the legacy validation scheme.

	series.1{"test.utf8.metric"="😄"}
	{"series.1", "test.utf8.metric"="😄"}


		Ruler external labels are also validated with the selected scheme. With `legacy`, external label names must follow the legacy validation rules. With `utf8`, external label names that are valid under the UTF-8 validation scheme can be used.

		The Ruler also passes the configured validation scheme to the notifier that sends alerts to Alertmanager. As a result, alert labels handled by the notification path are validated according to the selected scheme before alerts are sent.


		### Alertmanager

		The `name_validation_scheme` setting is not applied directly by Cortex [Alertmanager](https://cortexmetrics.io/docs/architecture/#alertmanager) in the same way it is applied by the Distributor or Ruler. Its effect on Alertmanager is indirect. Alerts sent by the Ruler pass through the Ruler notifier, which is created with the configured validation scheme. As a result, Alertmanager may receive alert labels that were validated according to the selected scheme.

Conversation

RaphSku commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented Apr 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RaphSku commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SungJin1212 commented May 4, 2026

Uh oh!

RaphSku commented May 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RaphSku commented Sep 8, 2026

Uh oh!

sandy2008 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RaphSku commented Oct 28, 2026

Uh oh!

sandy2008 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RaphSku commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RaphSku commented Apr 18, 2026 •

edited

Loading

RaphSku commented May 3, 2026 •

edited

Loading