Skip to content

Search OTLP logs by their own searchable tags and severity, and keep each LAL rule's layer to itself - #14255

Open
wankai123 wants to merge 3 commits into
masterfrom
feature/otlp-log-searchable-tags
Open

wankai123 wants to merge 3 commits into
masterfrom
feature/otlp-log-searchable-tags

Conversation

@wankai123

Copy link
Copy Markdown
Member

Search OTLP logs by their own searchable tags and severity, and keep each LAL rule's layer to itself

  • If this is non-trivial feature, paste the links/URLs to the design doc.
  • Update the documentation to include this new feature.
  • Tests(including UT, IT, E2E) are added to verify the new feature.
  • If it's UI related, attach the screenshots below.

OTLP logs

  • receiver-otel.otlpLogSearchableTags (SW_OTEL_LOG_SEARCHABLE_TAG_KEYS, default level,severity,ai_route_type): the tag keys indexed as searchable on OTLP logs, in place of core's searchableLogsTags, which the SkyWalking native logs keep. ai_route_type, added by the Envoy AI Gateway LAL rules to its OTLP access logs, moves from the core default to this one. The log query accepts a key of either list, and both lists are trimmed the same way.
  • A record's severity_number becomes the severity tag (its enum name, e.g. SEVERITY_NUMBER_INFO2) and the level tag (its range: TRACE…FATAL), so level=ERROR finds the errors of every SDK whatever their severity_text says.
  • A record, or the logs of a resource, whose attributes repeat a key is kept with the key's last value instead of being lost to a duplicate-key error.
  • OTLP logs' autocomplete tags are written under the new tag type OTLP_LOG. queryLogTagAutocompleteKeys/Values take an optional type (LOG, OTLP_LOG; Add an optional type to the log tag autocomplete queries skywalking-query-protocol#182, bumped here); without it, as the UI and LogQL query, both are read in one storage query. ITagAutoCompleteQueryDAO takes a list of tag type names.

LAL

  • Each rule a log runs gets its own copy of the log's metadata: a layer statement in one rule changed the layer of the rules that ran after it, depending on rule order.
  • A layer statement that evaluates to an empty, null or unregistered layer drops the log with a warning, instead of failing in the traffic sink.
  • A layer: auto rule with a sink but no layer statement is skipped at boot and refused as a runtime rule.
  • OAP warns when no fallback: true rule is loaded, or a runtime change removes the last one.
  • A runtime LAL rule is attributed to lal/<name>.yaml, not lal/lal/<name>.yaml.

Existing tests changed (reviewed before the change)

  • LogAnalyzerRoutingTest.fallbackRulesDoNotSeeALayerAnAbortedAutoRuleSet is removed: the reset it covered is gone, as each rule now has its own metadata; LogFilterListenerPerRuleMetadataTest covers the behavior with real listeners.
  • JDBCTagAutoCompleteQueryDAOTest, JDBCMetadataQueryITBase, MetadataQueryITBase, EsReadIndexSelectionTest, BanyanDBIndexModeQueryIT: pass the tag type as a list of names for the new DAO signature; the JDBC SQL assertion expects tag_type in (?).
  • test/e2e-v2/cases/storage/expected/config-dump.yml: the new setting and the core default without ai_route_type.

Verified with the unit tests, the PostgreSQL, Elasticsearch 9 and BanyanDB ITs, and on the showcase (kind, BanyanDB) with the OpenTelemetry Astronomy Shop: severity/level on every SDK's logs, search and autocomplete by them, type: OTLP_LOG/LOG/none, per-rule layers through runtime LAL rules, and the repeated-key cases.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
  • Update the CHANGES log.

…each LAL rule's layer to itself

OTLP logs:
- Add `receiver-otel.otlpLogSearchableTags` (`SW_OTEL_LOG_SEARCHABLE_TAG_KEYS`, default
  `level,severity,ai_route_type`), the tag keys indexed as searchable on OTLP logs in place of core's
  `searchableLogsTags`, which the native logs keep. `ai_route_type` moves from the core default to it.
  The log query accepts a key of either list. Both lists are split and trimmed the same way.
- Write a record's `severity_number` as the `severity` tag (its enum name) and the `level` tag (its range:
  TRACE, DEBUG, INFO, WARN, ERROR, FATAL).
- Keep a record, and the logs of a resource, whose attributes repeat a key, with its last value.
- Write OTLP logs' autocomplete tags under the new tag type `OTLP_LOG`. `queryLogTagAutocompleteKeys`
  and `queryLogTagAutocompleteValues` take an optional `type` (query-protocol #182); without it, both
  sources are read in one storage query. `ITagAutoCompleteQueryDAO` takes a list of tag type names.

LAL:
- Each rule a log runs gets its own copy of the log's metadata, so a `layer` statement in one rule no
  longer changes the layer of the rules that run after it.
- A `layer` statement that evaluates to an empty, null or unregistered layer drops the log with a
  warning instead of failing in the traffic sink.
- A `layer: auto` rule with a `sink` but no `layer` statement is skipped at boot and refused as a
  runtime rule.
- Warn when no `fallback: true` rule is loaded, or a runtime change removes the last one.
- Attribute a runtime LAL rule to `lal/<name>.yaml`, not `lal/lal/<name>.yaml`.
…archable-tags

# Conflicts:
#	docs/en/changes/changes.md

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The JDBC fixture omits the new accessor stub, breaking searchable-tag tests, while both autocomplete backends apply limits before deduplication and can hide valid suggestions.

3 open issues
What changed in this PR

This PR separates OTLP log search settings from native log settings and makes LAL layer selection independent for each rule in OAP’s log pipeline.

Changes:

  • Adds OTLP-specific searchable tags, severity-derived tags, and duplicate-attribute handling.
  • Supports source-specific or combined log-tag autocomplete across storage backends.
  • Isolates LAL metadata, validates layer decisions, warns about missing fallback rules, and updates documentation and tests.
File Description
config-dump.yml Updates expected searchable-tag configuration.
JDBCTagAutoCompleteQueryDAOTest.java Adapts SQL assertions to tag-type lists.
JDBCTagAutoCompleteQueryDAOTagTypesTest.java Tests combined-source SQL construction.
JDBCMetadataQueryITBase.java Migrates integration calls to the new DAO signature.
JDBCTagAutoCompleteQueryDAO.java Queries multiple autocomplete tag types.
JDBCLogQueryDAO.java Accepts the combined searchable-key set.
MetadataQueryITBase.java Updates Elasticsearch integration calls.
EsReadIndexSelectionTest.java Updates index-selection tests for type lists.
TagAutoCompleteQueryDAO.java Filters Elasticsearch autocomplete by multiple types.
BanyanDBIndexModeQueryIT.java Updates BanyanDB integration calls.
BanyanDBTagAutocompleteQueryDAO.java Supports multi-type autocomplete filters.
application.yml Adds OTLP tag settings and adjusts native defaults.
OpenTelemetryLogHandlerTagsTest.java Tests severity, duplicates, and OTLP metadata.
OpenTelemetryLogHandler.java Builds severity tags and preserves repeated attributes’ last values.
OtelMetricReceiverProvider.java Registers OTLP searchable keys with core.
OtelMetricReceiverConfig.java Parses the OTLP searchable-tag setting.
LogQuery.java Adds optional autocomplete source selection.
LogQLApiHandler.java Combines native and OTLP autocomplete sources.
LogBuilderSearchableTagsTest.java Tests receiver-specific indexing and autocomplete types.
TagAutoCompleteQueryServiceLogTypeTest.java Tests source selection and DAO delegation.
ConfigServiceSplitTagKeysTest.java Tests trimming and empty-key handling.
ITagAutoCompleteQueryDAO.java Changes the contract to accept tag-type lists.
LogMetadata.java Adds copying support and receiver tag metadata.
LogBuilder.java Uses receiver-specific searchable keys and tag types.
TagAutoCompleteQueryService.java Resolves sources and delegates combined queries.
ConfigService.java Normalizes keys and maintains the queryable-key union.
TagType.java Introduces OTLP_LOG.
LalFileApplierSourceTest.java Tests matching runtime and boot source attribution.
LalFileApplier.java Removes the duplicated catalog prefix from attribution.
LogAnalyzerRoutingTest.java Removes the replaced shared-metadata reset test.
LogFilterListenerPerRuleMetadataTest.java Tests layer isolation between rules and passes.
LogFilterListenerAutoLayerStatementTest.java Tests auto-rule layer-statement validation.
LALSinkLayerTest.java Tests sink behavior for valid and invalid layers.
LogAnalyzer.java Removes the now-unnecessary fallback layer reset.
TrafficSinkListener.java Updates the comment describing layer validation guarantees.
LogFilterListener.java Copies metadata, validates rules, and warns about missing fallbacks.
FilterSpec.java Drops logs with invalid rule-selected layers.
ExecutionContext.java Tracks each rule’s layer decision explicitly.
LALScriptModel.java Detects sink blocks and layer assignments.
LALBlockCodegen.java Records generated layer assignments through the context.
CLAUDE.md Documents rule-local metadata and validation behavior.
log-otlp.md Explains severity tags and OTLP search configuration.
configuration-vocabulary.md Documents the new setting and revised defaults.
backend-envoy-ai-gateway-monitoring.md Identifies the OTLP setting controlling access-log tags.
lal.md Documents layer isolation, rejection, and fallback warnings.
changes.md Records behavior changes and migration details.
query-protocol.md Documents optional autocomplete source selection.

Review effort: Balanced

.provider()
.getService(ConfigService.class);
searchableTagKeys = new HashSet<>(Arrays.asList(configService.getSearchableLogsTags().split(Const.COMMA)));
searchableTagKeys = configService.getSearchableLogsTagKeys();

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c5eee8: JDBCLogQueryITBase now stubs getSearchableLogsTagKeys() with the same keys as getSearchableLogsTags(). JDBCLogQueryPostgreSQLIT and JDBCLogQueryMariaDBIT pass.

…taking the limit

- JDBC: select distinct tag_value, so a value's row under each tag type does not take the limit's places.
- Elasticsearch: read the values one query per tag type and merge them within the limit; tag_value has no doc values to aggregate on.
- JDBC log IT: stub the searchable log tag keys on the mocked ConfigService.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the OTLP logging and per-rule metadata changes.

1 open issue
2 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants