Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions docs/en/api/query-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,12 +150,16 @@ extend type Query {
# Test the logs and get the results of the LAL output.
test(requests: LogTestRequest!): LogTestResponse!
# Read the list of searchable keys
queryLogTagAutocompleteKeys(duration: Duration!):[String!]
queryLogTagAutocompleteKeys(duration: Duration!, type: TagType):[String!]
# Search the available value options of the given key.
queryLogTagAutocompleteValues(tagKey: String! , duration: Duration!):[String!]
queryLogTagAutocompleteValues(tagKey: String! , duration: Duration!, type: TagType):[String!]
}
```

The optional `type` of the tag autocomplete queries narrows them to one log source: `OTLP_LOG` for the logs received
through OTLP, whose searchable tags `receiver-otel.otlpLogSearchableTags` sets, or `LOG` for the logs of every other
source, whose searchable tags core's `searchableLogsTags` sets. Without a `type`, the tags of both are returned.

Log implementations vary between different database options. Some search engines like ElasticSearch and OpenSearch can support
full log text fuzzy queries, while others do not due to considerations related to performance impact and end user experience.

Expand Down
8 changes: 7 additions & 1 deletion docs/en/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
* Fix JDBC trace profiling queries failing when a task has no profiled segments: the empty segment-ID list now returns an empty result without generating an invalid SQL `IN ()` clause.
* Add the `query-server` module, so the native query APIs can have their own port, apart from the ports agents report to. Set `SW_QUERY_SERVER_REST_PORT` to a port greater than 0, and GraphQL (`/graphql`), the AI agent conversation view and files (`/ai-agent/conversations/*`), and `/status/config/ttl` move from the core REST port (default `12800`) to it, with their own host, context path and TLS settings (`SW_QUERY_SERVER_REST_*`). Point the UI's `queryUrl`, `swctl --base-url` and other GraphQL clients at the new port. With the default `0`, nothing changes. Together with `receiver-sharing-server`, no agent or query traffic is left on the core ports.
* LAL: save the logs of the OpenTelemetry services that set `service.layer=OTLP_GENERAL` through the new `otlp-general` rule in `lal/default.yaml`; no rule matched that layer, so these logs were dropped. A LAL rule declaring an unregistered or missing layer (names are case-sensitive) is now skipped with a warning at boot and rejected when applied as a runtime rule, instead of loading silently under `UNDEFINED` and never running.
* LAL: a log without a layer that no `layer: auto` rule claims runs the new `fallback: true` auto rules, instead of the rules declared for `GENERAL`, and is dropped when none claims it either, so where such a log is filed is decided by a rule, not hard-coded. The new `default-fallback` rule in `lal/default.yaml` files it under `GENERAL`, as before; edit its extractor to file it under another layer. A fallback rule must set the layer, as any auto rule. A custom `layer: GENERAL` rule written for the logs without a layer must declare `layer: auto` and `fallback: true` and set `layer "GENERAL"` in its extractor; it then runs next to `default-fallback`, as it ran next to `default` before.
* LAL: a log without a layer that no `layer: auto` rule claims runs the new `fallback: true` auto rules, instead of the rules declared for `GENERAL`, and is dropped when none claims it either, so where such a log is filed is decided by a rule, not hard-coded. The new `default-fallback` rule in `lal/default.yaml` files it under `GENERAL`, as before; edit its extractor to file it under another layer. A fallback rule must set the layer, as any auto rule. A custom `layer: GENERAL` rule written for the logs without a layer must declare `layer: auto` and `fallback: true` and set `layer "GENERAL"` in its extractor; it then runs next to `default-fallback`, as it ran next to `default` before. The OAP warns at boot when no `fallback: true` rule is loaded, and when a runtime rule change removes the last one, since the logs without a layer are then dropped with only a debug log each.
* LAL: handle the mini-program error logs with the `miniprogram-errors-wechat` and `miniprogram-errors-alipay` rules in `lal/miniprogram.yaml`, declared for the `WECHAT_MINI_PROGRAM` and `ALIPAY_MINI_PROGRAM` layers, replacing the `layer: auto` rule `miniprogram-errors`. The mini-program SDK v0.5.0 sets `service.layer`, which routes its error logs to these rules; it is now the minimum supported SDK version, since an older SDK's error logs carry no layer and are filed under `GENERAL` without the `error_count` metric.
* Fix bugs of the JDBC storage, `mysql` (MySQL and MariaDB) and `postgresql`. Many showed only when a query's time range crossed midnight, as this storage keeps records and metrics in one table per day:
- The trace list came out of order when its time range crossed midnight, and its pages skipped or repeated traces.
Expand Down Expand Up @@ -83,6 +83,12 @@
* Fix a trace profiling task being refused as overlapping a task of the service that starts later in the minute the new task ends.
* Elasticsearch: write a row into the daily index of its time bucket's day, whatever the bucket's precision. Async-profiler, pprof and continuous-profiling tasks, which carry a second-precision bucket, went into an index per hour that the TTL never deleted, and could not be written at all with `dayStep` above 1; the TTL now deletes the hour-named indices (`*-yyyyMMddHH`) earlier versions left, with their day. The copy of an eBPF network profiling task that keeps it alive carried time bucket 0, so it went into an index dated 0 that the next TTL run deleted; it now carries the minute it is written. BanyanDB: a task list now includes the tasks of the range's last time bucket.
* Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in `remote_out_error_count`, which were only logged before. When a worker fails to process one received remote message, only that message is discarded and counted in `remote_in_error_count`; before, the failure ended the whole stream.
* Keep the severity of an OTLP log, which the OTLP log receiver dropped before: a record's `severity_number` is now written as the `severity` tag, named by its enum constant (`SEVERITY_NUMBER_INFO`, `SEVERITY_NUMBER_ERROR2`, ...), and as the `level` tag, named by its range (`TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`), so `SEVERITY_NUMBER_ERROR2` is `ERROR`. A record whose number is unset gets neither tag, and a record attribute named `severity` or `level` is kept instead.
* Add `receiver-otel.otlpLogSearchableTags` (`SW_OTEL_LOG_SEARCHABLE_TAG_KEYS`), the tag keys indexed as searchable on OTLP logs, in place of core's `searchableLogsTags`, which the SkyWalking native logs keep. Its default, `level,severity,ai_route_type`, keeps `level` and `ai_route_type` and adds `severity`. `http.status_code`, which OTLP logs could be searched by before, is not in it: to search OTLP logs by HTTP status, list it, or `http.response.status_code` as the OpenTelemetry semantic conventions name the attribute. `ai_route_type`, the tag the Envoy AI Gateway LAL rules add to its OTLP access logs, moves from the `searchableLogsTags` default to this one, so `searchableLogsTags` defaults to `level,http.status_code` again. A deployment that customized `searchableLogsTags` (`SW_SEARCHABLE_LOGS_TAG_KEYS`) to search OTLP logs sets those keys in `otlpLogSearchableTags` instead: OTLP logs read `searchableLogsTags` only when `otlpLogSearchableTags` is empty, as it is in an `application.yml` without the setting. The log query accepts a key of either list. Both lists are trimmed the same way: a `searchableLogsTags` key written after a space or an empty entry, such as `level, http.status_code`, used to be kept with the space, so it matched no tag; it now names `http.status_code`.
* LAL: every rule a log runs, auto or declared for a layer, saves the log under the layer the log arrived with or one its own `layer` statement decided. The rules shared the log's metadata, so a `layer` statement in one rule changed the layer of the rules that ran after it on the same log, depending on the order the rules ran in, and an invalid layer from one rule made the next one fail; each rule now runs on its own copy of the metadata. An auto rule with a `sink` but no `layer` statement, which dropped every log it saved, is now skipped with a warning at boot and refused as a runtime rule; one without a `sink` needs no `layer` statement. In any LAL rule, a `layer` statement that evaluates to an empty, null or unregistered layer now drops the log with a warning when it is saved, where the traffic sink threw for it and lost the log with only a stack trace.
* The OTLP log receiver keeps a record whose attributes repeat a key, and the logs of a resource whose attributes repeat one, with the key's last value, instead of losing the record, or every log of the export request, to a duplicate-key error.
* Fix a LAL rule applied through the runtime-rule API being attributed to `lal/lal/<name>.yaml`, the catalog prefix applied twice, in its error messages and its generated class; it is now `lal/<name>.yaml`, as the same file loaded at boot.
* Log tag autocomplete keeps the tags of the logs received through OTLP apart from those of every other source, under the new tag type `OTLP_LOG`. `queryLogTagAutocompleteKeys` and `queryLogTagAutocompleteValues` take an optional `type` (`LOG` or `OTLP_LOG`) to read one source; without it, as the UI and LogQL query, they return both merged, within the same limit. The OTLP logs' tags written before the upgrade stay under `LOG`, so a query with `type: OTLP_LOG` misses them until they expire. Both are read in one storage query: `ITagAutoCompleteQueryDAO` takes a list of tag types, so a storage plugin outside this repository that implements it has to follow.

#### UI

Expand Down
15 changes: 14 additions & 1 deletion docs/en/concepts-and-designs/lal.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,14 +130,27 @@ In `layer: auto` mode, `abort` and `drop` have distinct meanings:
If an auto rule claims the log (doesn't abort) but doesn't set a layer, the log is warned and
dropped at persistence time.

Every rule a log runs, auto or declared for a layer, starts from the layer the log arrived with, and only its own
`layer` statement changes it: a `layer` statement in one rule never changes the layer another rule saves the log
under, whatever order the rules run in.

In any rule, auto or declared for a layer, a `layer` statement whose value is empty, null (for example
`layer parsed.kind as String` on a log without `kind`) or not a registered layer name drops the log with a
warning when it is saved. A rule without a `layer` statement saves the log under the layer it arrived with.

An auto rule with a `sink` must therefore have a `layer` statement, on some branch of its script. One without
it is skipped with a warning at boot, and refused when applied as a runtime rule. An auto rule without a `sink`,
for example one that only extracts metrics, never saves a log and needs no `layer` statement.

All the auto rules run on such a log, in no defined order, and each one that claims it saves a copy of
its own, so an auto rule should claim only the logs it recognizes and `abort {}` on the others.

A rule declaring `fallback: true` (allowed only with `layer: auto`) runs after the other auto rules, only when
none of them claimed the log. It is the catch-all for the logs without a layer: `default-fallback` in
`lal/default.yaml` files them under `GENERAL`. A fallback rule follows the same rules as any auto rule: it
must set the layer, and `abort {}` leaves the log unclaimed, which drops it. Auto and fallback rule names
share one name space.
share one name space. When no `fallback: true` rule is loaded, the OAP warns at boot, and when a runtime rule
change removes the last one, since the logs without a layer that no auto rule claims are then dropped.

```yaml
rules:
Expand Down
3 changes: 2 additions & 1 deletion docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,8 @@ All MCP service-level metrics are also available per instance with `meter_envoy_
### Access Log Sampling

Access logs are tagged with `ai_route_type` (`llm` or `mcp`) for filtering in the log query UI.
The `ai_route_type` tag is searchable by default.
The `ai_route_type` tag is searchable by default, through `receiver-otel.otlpLogSearchableTags`
(`SW_OTEL_LOG_SEARCHABLE_TAG_KEYS`), since the access logs arrive over OTLP.

**LLM route logs:**
- **Error responses** (HTTP status >= 400) — always persisted.
Expand Down
Loading
Loading