Skip to content

feat(observability): logs end-to-end (gateway + UI panel) - #5379

Draft
Ma77Ball wants to merge 139 commits into
apache:mainfrom
Ma77Ball:obs/pr5/logs
Draft

Ma77Ball wants to merge 139 commits into
apache:mainfrom
Ma77Ball:obs/pr5/logs

Conversation

@Ma77Ball

@Ma77Ball Ma77Ball commented Jun 5, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

First complete signal: log search through the gateway plus a UI panel to drive it.
Backend:

  • Logs query builder and response parsers (parseLogs, parseLogSources), applying per-field redaction.
  • LogsResource exposing log search and the source-facets endpoint, registered in TexeraWebApplication.
    Frontend:
  • Logs panel with time-window, service, workflow, computing-unit, execution, level, and free-text filters, server-side paging, and sources-backed autofill.

Any related issues, documentation, or discussions?

Closes: #5371
Part of #4070. Stacked on #5378.

How was this PR tested?

  • Backend specs for the logs query builder and response parsers; sbt scalafmtCheckAll passes.
  • Frontend logs-panel and service specs; prettier-eslint and eslint pass.
  • Compile and the full test suites run in this PR's CI.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 in compliance with ASF

Ma77Ball and others added 5 commits June 5, 2026 04:49
…, SDK bootstrap (default-off)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s panel

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nt scope, health, routing)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ca/eBPF profiling

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tracing primitives

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI docs Changes related to documentations infra common labels Jun 5, 2026
@codecov-commenter

codecov-commenter commented Jun 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.84692% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.77%. Comparing base (24b587f) to head (76eeb09).
⚠️ Report is 928 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...a/org/apache/texera/web/TexeraWebApplication.scala 0.00% 19 Missing ⚠️
.../engine/architecture/coordinator/Coordinator.scala 68.18% 6 Missing and 1 partial ⚠️
...exera/service/resource/AccessControlResource.scala 70.58% 0 Missing and 5 partials ⚠️
...exera/amber/engine/common/client/ClientActor.scala 77.77% 0 Missing and 2 partials ⚠️
...la/org/apache/texera/web/ComputingUnitMaster.scala 0.00% 2 Missing ⚠️
...org/apache/texera/web/StaticAssetCacheFilter.scala 89.47% 0 Missing and 2 partials ⚠️
...mber/src/main/python/core/models/internal_queue.py 94.44% 0 Missing and 1 partial ⚠️
amber/src/main/python/core/models/operator.py 98.38% 1 Missing ⚠️
amber/src/main/python/core/models/tuple.py 92.85% 0 Missing and 1 partial ⚠️
...hitecture/common/PekkoActorRefMappingService.scala 0.00% 0 Missing and 1 partial ⚠️
... and 9 more
Additional details and impacted files
@@              Coverage Diff              @@
##               main    #5379       +/-   ##
=============================================
+ Coverage     56.89%   69.77%   +12.87%     
- Complexity     3058     3542      +484     
=============================================
  Files          1129     1174       +45     
  Lines         43802    46525     +2723     
  Branches       4743     5685      +942     
=============================================
+ Hits          24922    32463     +7541     
+ Misses        17449    12334     -5115     
- Partials       1431     1728      +297     
Flag Coverage Δ *Carryforward flag
access-control-service 80.27% <72.22%> (+10.27%) ⬆️
agent-service 99.32% <100.00%> (+54.73%) ⬆️
amber 66.43% <70.21%> (+7.78%) ⬆️ Carriedforward from fcc8222
computing-unit-managing-service 77.20% <ø> (+77.20%) ⬆️
config-service 87.25% <ø> (+34.94%) ⬆️
file-service 63.84% <ø> (+1.02%) ⬆️ Carriedforward from fcc8222
frontend 60.98% <ø> (+10.85%) ⬆️ Carriedforward from fcc8222
notebook-migration-service 83.73% <ø> (+5.16%) ⬆️
pyamber 98.47% <99.42%> (+7.32%) ⬆️
python ?
workflow-compiling-service 74.25% <ø> (+19.10%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Ma77Ball added 23 commits June 26, 2026 17:04
reduced the comments to include less design details and information not
needed in the codebase.
…WorkflowMetrics rename

Follow the OTel Java demo patterns instead of custom wrappers (zuozhiw
review):

- WorkflowService: start the run-level span with the standard OTel API,
name it WorkflowService.initExecutionService, and drop the
initExecutionServiceSpanned split so no span is passed as an argument.
The real execution failure is now recorded onto the span from
errorHandler, where it is actually caught.

- TexeraTracer: drop the withSpan wrapper, keeping only the tracer
accessor (single instrumentation scope) and currentContext.

- SpanAttrs: drop the awkward with*/set* setter helpers, keep the
standard label keys and sanitizeFreeText; callers set attributes via the
OTel API.

- Rename TexeraMetrics to WorkflowMetrics, document that this facade is
only for the workflow-execution cluster, and add an example of calling
the OTel meter API directly at a call site.

Update SpanAttrsSpec and WorkflowMetricsSpec to match.
@github-actions github-actions Bot added the ci changes related to CI label Sep 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI common dependencies Pull requests that update a dependency file docs Changes related to documentations engine frontend Changes related to the frontend GUI infra platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Observability] Log search: gateway endpoint and dashboard panel

2 participants