feat: add OpenTelemetry logs

README.md

@@ -229,7 +229,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for how to set up a development environme
 - [Using Cog with Windows 11](docs/wsl2/wsl2.md)
 - [Browse the example models in this repo](docs/examples.md)
 - [Deploy models with Cog](docs/deploy.md)
-- [Configure OpenTelemetry tracing and metrics](docs/observability.md)
+- [Configure OpenTelemetry tracing, metrics, and logs](docs/observability.md)
 - [`cog.yaml` reference](docs/yaml.md) to learn how to define your model's environment
 - [Run interface reference](docs/python.md) to learn how the `Runner` interface works
 - [Training interface reference](docs/training.md) to learn how to add a fine-tuning API to your model

architecture/04-container-runtime.md

@@ -391,7 +391,9 @@ Opt-in OpenTelemetry tracing uses a provider in the parent process, a provider i
 
 Metrics use separate ownership. The parent creates the runtime MeterProvider after worker setup and exports fixed prediction, rejection, setup, and slot-state measurements. The Python worker creates the model MeterProvider for arbitrary model-owned instruments. It returns a runtime metric configuration in its `Ready` control message so the parent can omit selected fixed instruments before recording terminal setup status.
 
-Models may provide `observability.config`, which Cog validates and stages at a fixed image path. During worker setup, Cog loads that module, creates selected Python tracer and meter providers, validates them, installs them globally, and then calls optional `configure_instrumentation()` before predictor import. Cog owns provider flush and shutdown. Configuration failures stop model setup because the user explicitly selected that module.
+Logs follow the same split. The parent attaches an OpenTelemetry appender to its `tracing` subscriber and exports Cog's own events, including worker events relayed over the control channel; it skips the events that re-emit model stdout and stderr. The worker exports Python `logging` records by wrapping `logging.Logger.callHandlers`, which runs after logger level, disabled, and filter checks but adds no handler, so `lastResort` and `basicConfig()` behave as before.
+
+Models may provide `observability.config`, which Cog validates and stages at a fixed image path. During worker setup, Cog loads that module, creates selected Python tracer, meter, and logger providers, validates them, installs them globally, and then calls optional `configure_instrumentation()` before predictor import. Cog owns provider flush and shutdown. Configuration failures stop model setup because the user explicitly selected that module.
 
 The framework trace covers HTTP handling, validation, the logical prediction lifetime, worker execution, input preparation, predictor invocation, output upload, and setup. Model code creates ordinary child spans through `opentelemetry.trace`.
 
@@ -415,6 +417,7 @@ If the healthcheck fails, the HTTP `/health-check` endpoint returns `UNHEALTHY`
 | `COG_SETUP_TIMEOUT`              | none    | Setup timeout in seconds (0 is ignored)            |
 | `COG_TRACE_ENABLED`              | image   | `true` in tracing-enabled images; absent otherwise |
 | `COG_METRICS_ENABLED`            | image   | `true` in metrics-enabled images; absent otherwise |
+| `COG_LOGS_ENABLED`               | image   | `true` in logs-enabled images; absent otherwise    |
 | `COG_OBSERVABILITY_CONFIG`       | none    | Internal path to staged Python telemetry config    |
 | `COG_THROTTLE_RESPONSE_INTERVAL` | 0.5s    | Webhook response throttling interval               |
 | `LOG_FORMAT`                     | json    | Set to `console` for human-readable log output     |

crates/coglet-python/src/lib.rs

@@ -102,6 +102,7 @@ fn init_tracing(
     _to_stderr: bool,
     setup_log_tx: Option<tokio::sync::mpsc::UnboundedSender<String>>,
     #[cfg(feature = "tracing")] tracer: Option<coglet_core::trace::SdkTracer>,
+    #[cfg(feature = "tracing")] logs: Option<&coglet_core::logs::LoggingRuntime>,
 ) -> Result<Option<tokio::sync::mpsc::UnboundedReceiver<String>>, String> {
     let filter = if std::env::var("RUST_LOG").is_ok() {
         EnvFilter::from_default_env().add_directive("coglet::trace=trace".parse().unwrap())
@@ -132,6 +133,11 @@ fn init_tracing(
     #[cfg(not(feature = "tracing"))]
     let otel_layer = tracing_subscriber::layer::Identity::new();
 
+    #[cfg(feature = "tracing")]
+    let logs_layer = logs.map(|runtime| runtime.layer());
+    #[cfg(not(feature = "tracing"))]
+    let logs_layer = tracing_subscriber::layer::Identity::new();
+
     if let Some(tx) = setup_log_tx {
         let accumulator = coglet_core::SetupLogAccumulator::new(tx);
 
@@ -140,6 +146,7 @@ fn init_tracing(
                 .with(filter)
                 .with(sentry_layer)
                 .with(otel_layer)
+                .with(logs_layer)
                 .with(accumulator)
                 .with(fmt::layer().json().with_writer(std::io::stderr));
             subscriber.try_init().map_err(|error| error.to_string())?;
@@ -148,6 +155,7 @@ fn init_tracing(
                 .with(filter)
                 .with(sentry_layer)
                 .with(otel_layer)
+                .with(logs_layer)
                 .with(accumulator)
                 .with(fmt::layer().with_writer(std::io::stderr));
             subscriber.try_init().map_err(|error| error.to_string())?;
@@ -159,13 +167,15 @@ fn init_tracing(
                 .with(filter)
                 .with(sentry_layer)
                 .with(otel_layer)
+                .with(logs_layer)
                 .with(fmt::layer().json().with_writer(std::io::stderr));
             subscriber.try_init().map_err(|error| error.to_string())?;
         } else {
             let subscriber = tracing_subscriber::registry()
                 .with(filter)
                 .with(sentry_layer)
                 .with(otel_layer)
+                .with(logs_layer)
                 .with(fmt::layer().with_writer(std::io::stderr));
             subscriber.try_init().map_err(|error| error.to_string())?;
         }
@@ -349,13 +359,19 @@ fn serve_impl(
         let _entered = rt.enter();
         coglet_core::trace::TracingRuntime::from_env(coglet_core::trace::ProcessRole::Parent)
     };
+    #[cfg(feature = "tracing")]
+    let logs_runtime = {
+        let _entered = rt.enter();
+        coglet_core::logs::LoggingRuntime::from_env()
+    };
 
     let (setup_log_tx, setup_log_rx) = tokio::sync::mpsc::unbounded_channel();
     #[cfg(feature = "tracing")]
     init_tracing(
         false,
         Some(setup_log_tx),
         trace_runtime.as_ref().map(|runtime| runtime.tracer()),
+        logs_runtime.as_ref(),
     )
     .map_err(PyErr::new::<pyo3::exceptions::PyRuntimeError, _>)?;
     #[cfg(not(feature = "tracing"))]
@@ -421,6 +437,10 @@ fn serve_impl(
         }
         #[cfg(feature = "tracing")]
         coglet_core::runtime_metrics::shutdown();
+        #[cfg(feature = "tracing")]
+        if let Some(runtime) = logs_runtime.as_ref() {
+            runtime.shutdown();
+        }
         return result;
     };
 
@@ -441,6 +461,10 @@ fn serve_impl(
     }
     #[cfg(feature = "tracing")]
     coglet_core::runtime_metrics::shutdown();
+    #[cfg(feature = "tracing")]
+    if let Some(runtime) = logs_runtime.as_ref() {
+        runtime.shutdown();
+    }
     result
 }
 
@@ -700,6 +724,7 @@ fn coglet(py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
     m.add("__version__", env!("COGLET_PEP440_VERSION"))?;
     m.add("__build__", BuildInfo::new())?;
     m.add("_supports_observability_metrics", cfg!(feature = "tracing"))?;
+    m.add("_supports_observability_logs", cfg!(feature = "tracing"))?;
 
     // Frozen server object
     m.add("server", CogletServer {})?;

crates/coglet-python/src/worker_bridge.rs

@@ -64,6 +64,7 @@ fn python_telemetry_enabled() -> bool {
     }
     (env_true("COG_TRACE_CONFIGURED", false) && env_true("COG_TRACE_ENABLED", true))
         || (env_true("COG_METRICS_CONFIGURED", false) && env_true("COG_METRICS_ENABLED", true))
+        || (env_true("COG_LOGS_CONFIGURED", false) && env_true("COG_LOGS_ENABLED", true))
 }
 
 fn runtime_metrics_config_from_python(

crates/coglet/Cargo.toml

@@ -14,6 +14,7 @@ categories.workspace = true
 default = ["tracing", "tracing-grpc"]
 tracing = [
     "dep:opentelemetry",
+    "dep:opentelemetry-appender-tracing",
     "dep:opentelemetry-jaeger-propagator",
     "dep:opentelemetry-otlp",
     "dep:opentelemetry_sdk",
@@ -22,6 +23,7 @@ tracing = [
     "opentelemetry-otlp/reqwest-blocking-client",
     "opentelemetry-otlp/trace",
     "opentelemetry-otlp/metrics",
+    "opentelemetry-otlp/logs",
 ]
 tracing-grpc = [
     "tracing",
@@ -74,10 +76,11 @@ rustls.workspace = true
 # Observability
 tracing.workspace = true
 tracing-subscriber.workspace = true
-opentelemetry = { version = "=0.32.0", default-features = false, features = ["trace", "metrics"], optional = true }
+opentelemetry = { version = "=0.32.0", default-features = false, features = ["trace", "metrics", "logs"], optional = true }
+opentelemetry-appender-tracing = { version = "=0.32.0", default-features = false, optional = true }
 opentelemetry-jaeger-propagator = { version = "=0.32.0", default-features = false, optional = true }
 opentelemetry-otlp = { version = "=0.32.0", default-features = false, optional = true }
-opentelemetry_sdk = { version = "=0.32.0", default-features = false, features = ["trace", "metrics"], optional = true }
+opentelemetry_sdk = { version = "=0.32.0", default-features = false, features = ["trace", "metrics", "logs"], optional = true }
 tracing-opentelemetry = { version = "=0.33.0", default-features = false, optional = true }
 
 [target.'cfg(unix)'.dependencies]
@@ -89,4 +92,4 @@ tempfile = "3"
 wiremock = "0.6"
 tower = { version = "0.5", features = ["util"] }
 http-body-util = "0.1"
-opentelemetry_sdk = { version = "=0.32.0", default-features = false, features = ["testing", "trace", "metrics"] }
+opentelemetry_sdk = { version = "=0.32.0", default-features = false, features = ["testing", "trace", "metrics", "logs"] }

crates/coglet/src/lib.rs

@@ -8,6 +8,8 @@ mod version;
 
 pub mod bridge;
 mod fd_redirect;
+#[cfg(feature = "tracing")]
+pub mod logs;
 pub mod orchestrator;
 pub mod permit;
 #[cfg(feature = "tracing")]