[anish-sahoo]

This adds opt-in OpenTelemetry log export to Cog. The Python worker exports records from the model's logging calls, and the Rust parent exports Cog's own runtime events. Logs are off by default. Stderr, the prediction logs field, and coglet's structured stderr logs do not change, and exporter failures do not affect predictions.

User experience

Enable logs in cog.yaml:

observability:
  logs:
    enabled: true

The boolean form logs: true is accepted as shorthand.

Collector settings stay at runtime:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME=my-model

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT and OTEL_EXPORTER_OTLP_LOGS_PROTOCOL override the generic settings. Generic HTTP endpoints receive /v1/logs; signal-specific endpoints are used as provided. COG_LOGS_ENABLED=false disables the signal at runtime, and OTEL_LOGS_EXPORTER=none disables only the built-in exporter.

No model changes are required. Existing logging calls are exported once the image opts in and an endpoint is present.

Model logs

import logging

from cog import BaseRunner

log = logging.getLogger(__name__)


class Runner(BaseRunner):
    def run(self, prompt: str) -> str:
        result = self.model.generate(prompt)
        log.info("generated", extra={"prompt_tokens": result.prompt_tokens})
        return result.text

Each exported record keeps:

• severity, with WARNING and CRITICAL mapped to OTel's WARN and FATAL
• the logger name as the instrumentation scope
• the formatted message as the body
• code.file.path, code.function.name, and code.line.number
• exception.type, exception.message, and exception.stacktrace when exc_info is set
• extra fields as attributes
• the prediction's trace and span IDs when tracing is enabled

A message that is a dict or list with no format arguments stays structured, matching the OpenTelemetry SDK handler. The docs recommend extra for fields people want to filter on, because backends differ in whether they index body fields.

Cog does not change log levels. A record is exported only if the model's logging configuration lets it through, so with Python's defaults only WARNING and above are exported. Records from disabled loggers or removed by logger filters are not exported.

How Python records are captured

Export wraps logging.Logger.callHandlers. That method runs once per record, after the logger's level, disabled, and filter checks and before any handler. Cog installs the wrapper when the logger provider is installed and restores the original on shutdown.

Cog does not add a root handler. A root handler would stop logging.lastResort from printing unconfigured warnings to stderr, which would change the prediction logs field, and it would make a later logging.basicConfig() call a no-op. A record factory was also considered, but CPython applies extra after the factory runs, so those attributes would be lost.

The SDK's LoggingHandler is deprecated in 1.44.0, and opentelemetry-instrumentation-logging's instrumentor calls basicConfig() itself, so neither is used. Records are converted with the SDK's own attribute mapping (the LoggingHandler._get_attributes staticmethod, called without constructing the handler) and std_to_otel. Both are private SDK APIs pinned at 1.44.0 and should be checked on the next SDK bump.

Runtime logs

The parent attaches opentelemetry-appender-tracing 0.32.0 to its tracing subscriber and exports Cog's own events, such as prediction start and completion, setup progress, and errors. Event fields like prediction_id become attributes.

The parent re-emits model stdout and stderr as tracing events for its own stderr output. The logs layer filters those out: coglet::setup and coglet::user events, and coglet::prediction events that carry a source field. Cog's own coglet::prediction events do not have that field. Non-coglet targets are also excluded.

Worker runtime events already reach the parent over the control channel and are exported there, so the worker does not export them a second time. Relayed worker events keep only their message and have the event name worker_log.

What is not exported

print() output and raw stdout and stderr are not exported. They are noisy, duplicate logging output when the model writes to stderr, and are already available in the prediction response.

Lifecycle and failure behavior

The parent provider is created before the tracing subscriber and shut down after tracing and metrics, so their shutdown events are still captured. Shutdown alone drains the batch processor in both SDKs, so there is no flush step that could export a record twice.

Missing endpoints and invalid built-in exporter settings log a warning and leave serving unaffected. Invalid custom create_logger_provider factories fail setup, like the other factories.

Records logged from OpenTelemetry loggers, re-entrant calls, and OTel SDK export threads are skipped. Without that, an exporter's HTTP or gRPC client logging a failed export during a collector outage would queue another record for export. The model's own HTTP client logs are still exported.

Custom Python telemetry

observability.config can provide a logger provider:

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource


def create_logger_provider(resource: Resource) -> LoggerProvider:
    provider = LoggerProvider(
        resource=resource.merge(Resource({"model.name": "example"})),
        shutdown_on_exit=False,
    )
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    return provider

The factory is optional. A missing factory uses Cog's default provider, and Cog owns shutdown.

Build and configuration

• observability.logs is added to config parsing, the JSON schema, and validation. observability.config now accepts logs as its only enabled signal.
• COG_LOGS_* is reserved from the cog.yaml environment list.
• Generated images set COG_LOGS_CONFIGURED and COG_LOGS_ENABLED. Custom-Dockerfile images set them explicitly to true or false so a base image cannot pass on its logs settings.
• PythonObservabilityCheck is now a function that checks the coglet capability flag for each enabled signal. Coglet adds _supports_observability_logs.
• No new Python packages are installed. The pinned OTLP exporters already include log exporters.

Documentation and examples

The observability guide has a new Logs section, and the YAML, environment, Python, architecture, and README references cover logs.

examples/hello-concurrency enables logs, defines a custom logger provider with OTLP export and an OTEL_DEBUG_LOGS console exporter, and logs an extra field.

integration-tests/tests/observability_logs.txtar checks that a custom logger provider is installed before model import, that a model warning is exported exactly once, and that it still appears in the prediction's logs.

[ask-bonk]

Review summary

This PR adds opt-in OpenTelemetry log export: the worker exports Python logging records by wrapping logging.Logger.callHandlers (no handler added, so stderr, the prediction logs field, lastResort, and basicConfig() are unchanged), and the parent exports Cog's own tracing events through opentelemetry-appender-tracing, filtering out re-emitted model stdout/stderr. Config, schema, Dockerfile generation, docs, and tests are all updated consistently, and the Go changes build and pass their focused tests.

I found one issue worth addressing (a test that verifies nothing), posted inline.

[ask-bonk]

These assertions are vacuous: the test can pass even if trace/span correlation into exported log records is completely broken.

_logs_script(..., traces=True) sets COG_TRACE_CONFIGURED=true but also OTEL_TRACES_EXPORTER=none, and _LOGS_CONFIG defines only create_logger_provider (no create_tracer_provider). install_providers() therefore installs no tracer provider, so trace.get_tracer("test").start_as_current_span("predict") uses the no-op tracer whose span context is invalid (trace_id == 0, span_id == 0). _export_log_record's context=get_current() then yields the same invalid IDs, so record.trace_id == span.get_span_context().trace_id is 0 == 0.

This is the only test covering the documented "records written during a prediction carry its trace and span IDs" behavior, and it would pass if, say, the context=get_current() argument were removed from _export_log_record. To make it meaningful, give the shared config a real tracer provider (the custom factory is used regardless of OTEL_TRACES_EXPORTER=none):

Suggested change

	with trace.get_tracer("test").start_as_current_span("predict") as span:
	_LOGS_CONFIG = """
	from opentelemetry.sdk._logs import LoggerProvider
	from opentelemetry.sdk._logs.export import InMemoryLogRecordExporter, SimpleLogRecordProcessor
	from opentelemetry.sdk.trace import TracerProvider
	exporter = InMemoryLogRecordExporter()
	def create_logger_provider(resource):
	provider = LoggerProvider(resource=resource, shutdown_on_exit=False)
	provider.add_log_record_processor(SimpleLogRecordProcessor(exporter))
	return provider
	def create_tracer_provider(resource):
	return TracerProvider(shutdown_on_exit=False)
	"""

[ask-bonk]

Review: 1 finding.

One inline finding was posted on python/tests/test_telemetry.py (test_logs_carry_the_active_trace_context): it asserts trace/span ID correlation but no tracer provider is installed (OTEL_TRACES_EXPORTER=none, no create_tracer_provider in the shared config), so both IDs are 0 and the test passes vacuously even if the correlation code were broken. No other actionable issues found; the Rust export/filter logic, Python callHandlers wrapping, config/schema/Dockerfile changes, and docs are consistent, and the Go changes build and pass their focused tests.

github run

[anish-sahoo]

@michaeldwan can I get a review