feat(logs): add datumctl logs for project-scoped log queries

[0xmc]

What

Adds a top-level datumctl logs command that queries telemetry logs for the active project through the queryapi aggregated APIService.

datumctl logs '{service_name="checkout"} |= "error"' --since 1h

Queries use the LogQL subset queryapi accepts — label matchers and line filters. Project scoping is resolved server-side, so the client never supplies a tenant identifier.

Closes #288, which is tracked as a sub-issue of the milo-os/telemetry#90 epic. The server-side counterpart is milo-os/telemetry#77.

Endpoint wiring

ProjectLogsAPIPrefix() is pinned to o11y.miloapis.com / v1alpha1 / logs, matching queryapi's APIService registration. All three parts are load-bearing — queryapi derives both the paths it serves and the permission it reviews (logs.query) from the same constants, so a mismatch is a 404 from the aggregator rather than a graceful fallback. Appending the prefix to the project control-plane host reproduces the server URL declared in queryapi's own openapi.yaml.

Verification against staging

Confirmed against api.staging.env.datum.net:

• aggregator routing and logs.query authorization both resolve
• a metric query returns queryapi's own parser message: logql: metric queries and aggregations are not supported
• rendering was verified against a fake queryapi returning a deliberately adversarial two-stream response — labels in non-alphabetical JSON order, timestamps interleaved across streams — in both directions, with five consecutive runs producing byte-identical output

A real end-to-end query returns zero rows in staging today, for reasons outside this repo: every row in o11y.logs carries ProjectId = 'internal'. ProjectId is MATERIALIZED ResourceAttributes['milo.project.id'] and the row policy is ProjectId = getSetting('telemetry_project_id'), so no customer project can match until the edge pipeline stamps that attribute with the owning project. Verified directly in ClickHouse, and by generating real traffic through an HTTPProxy in a test project — the Envoy access logs were ingested correctly but tagged internal. Details in #288.

Rendering details

• Label sets print in sorted key order. Ranging the map directly emitted a different permutation per run, so two identical queries produced output that did not compare equal.
• Lines from all streams are flattened and re-sorted by timestamp. queryapi returns one stream per distinct label set, so printing stream by stream walked the clock backwards at every boundary — most of the output for a service spread across several label sets.
• Unparseable timestamps sort together at the end rather than as the epoch, so a malformed value does not masquerade as 1970 data.

Errors and flags

Failures surface the response body, unwrapping either the Loki envelope or a Kubernetes Status, with a per-status hint. Previously a rejected query and a permission denial were indistinguishable.

--since and --start are now mutually exclusive rather than silently preferring one. A query argument is required: queryapi rejects a selector with no label matchers, so the previous {} default could only ever produce a 400.

Not included

• Live tail. queryapi does not implement /loki/api/v1/tail yet.
• Label/series discovery. /labels, /label/{name}/values and /series are served by queryapi but not yet wrapped as subcommands.
• Output formatting. Lines always render as timestamp | labels | line. A kubectl-style default of the bare line with opt-in --timestamps/--labels is worth considering, and would pair with routing output through IOStreams instead of fmt.Printf — which would also make the renderer unit-testable.

🤖 Generated with Claude Code

[scotwells]

@0xmc this should be a datumctl "telemetry" plugin that's exported from the o11y repo

[0xmc]

Leaving this PR open (as Draft) until the work is moved over to https://github.com/milo-os/telemetry.