feat(observability): supervisor OTLP telemetry relay

AGENTS.md

@@ -64,7 +64,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
 | `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |
 | `crates/openshell-supervisor-middleware-builtins/` | Built-in middleware | First-party in-process middleware implementations |
 | `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection |
-| `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, and bypass monitoring |
+| `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, bypass monitoring, and the OTLP telemetry relay |
 | `crates/openshell-vfio/` | VFIO support | PCI and GPU passthrough preparation and lifecycle |
 | `python/openshell/` | Python SDK | Python bindings and CLI packaging |
 | `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |

Cargo.toml

@@ -66,6 +66,7 @@ tracing-appender = "0.2"
 opentelemetry = "0.32"
 opentelemetry_sdk = { version = "0.32", features = ["rt-tokio"] }
 opentelemetry-otlp = { version = "0.32", default-features = false, features = ["grpc-tonic", "trace"] }
+opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace", "with-serde"] }
 tracing-opentelemetry = { version = "0.33", default-features = false, features = ["tracing-log"] }
 
 # Metrics

architecture/sandbox.md

@@ -456,6 +456,111 @@ Sandbox logs are emitted locally and can also be pushed back to the gateway.
 Security-relevant sandbox behavior uses OCSF structured events; internal
 diagnostics use ordinary tracing.
 
+## Telemetry Relay
+
+The supervisor can relay OpenTelemetry trace data and OCSF events from agent
+processes to the gateway over the session protocol. This gives OTel-instrumented
+agents (LangChain, CrewAI, etc.) a zero-configuration path to an external
+collector without requiring direct egress from the sandbox.
+
+### Data Flow
+
+```
+Agent process --> OTLP HTTP (127.0.0.1:4318) --> Supervisor receiver
+  --> Enrichment (sandbox resource attributes)
+  --> Bounded buffer (4096 slots, shared traces + OCSF)
+  --> Supervisor session (OtelExportData message on the session stream)
+  --> Gateway --> Dedicated SpanExporter --> External OTLP collector
+```
+
+The supervisor session owns the relay (`openshell-supervisor-process::otlp`).
+There is no separate forwarder task: the session drains the buffer directly
+into its outbound stream, so the buffer is the only queue between the
+receiver and the gateway.
+
+### Receiver Binding
+
+The OTLP HTTP receiver binds to `127.0.0.1:4318` only when the relay is
+active (the gateway has `[openshell.gateway.otlp]` configured and confirms
+the `otel_export` capability). When OTLP is not configured, no port is
+bound and no receiver runs.
+
+The bind is lazy: the session binds the receiver on the first
+`SessionAccepted` that confirms the capability and keeps it bound across
+gateway reconnects. The agent process starts before the session handshake
+completes, so an exporter that flushes in that window sees a refused
+connection rather than a silent drop; OTel SDK exporters retry with
+backoff, so those spans land once the port is up. If a later reconnect
+declines the capability, the receiver stays bound and forwarding pauses
+until a session confirms again. A bind failure is logged and disables the
+relay for the rest of the sandbox lifetime.
+
+The bind address depends on the supervisor topology. In all current
+topologies, the process supervisor runs co-located with the agent workload,
+so `127.0.0.1` is reachable from the agent. For Docker/Podman drivers
+where the supervisor creates a network namespace, the bind happens inside
+the namespace via `bind_tcp_in_netns()`. For Kubernetes and VM drivers,
+the supervisor and agent share the same network namespace, so a direct
+bind suffices. The receiver accepts both `application/x-protobuf` and
+`application/json` content types.
+
+Future topologies where the process supervisor moves out of the workload
+pod would require the bind address and the `OTEL_EXPORTER_OTLP_ENDPOINT`
+env var to reflect the supervisor's reachable address from the agent's
+perspective (e.g., a service IP or pod IP).
+
+### Span Enrichment
+
+Forwarded spans are enriched with sandbox resource attributes:
+`openshell.sandbox.id`, `openshell.workspace.id`, `openshell.sandbox.policy`,
+`openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`.
+The `openshell.telemetry.source` attribute (fixed value `"agent"`) is always
+injected regardless of the enrichment toggle so collectors can filter agent
+spans from gateway infrastructure spans. Enrichment can be disabled for
+pass-through forwarding.
+
+### Activation and Capability Negotiation
+
+The relay is opt-in. It starts only when the gateway has `[openshell.gateway.otlp]`
+configured. The supervisor advertises `"otel_export"` in
+`SupervisorHello.capabilities`. The gateway confirms via `SessionAccepted.capabilities`.
+The supervisor gates `OtelExportData` sending on this confirmation. When the
+relay is active, the supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT` and
+`OTEL_EXPORTER_OTLP_PROTOCOL` in agent child processes via `child_env.rs`.
+
+### Non-Interference
+
+Both hops are non-blocking. The receiver uses `try_send` into the bounded
+buffer: when the buffer is full, the newest item is dropped and a counter
+records each drop. The session uses `try_send` into its outbound stream and
+drops the message when that stream is backed up. A queue depth gauge tracks
+buffer pressure. This ensures telemetry cannot block or degrade sandbox
+control operations.
+
+### Shutdown
+
+After the entrypoint exits and before the exit is reported to the gateway,
+the supervisor asks the session to stop the receiver and flush the buffer.
+The receiver stops accepting, disables keep-alive on open connections (idle
+ones close immediately), waits up to 2 seconds for in-flight requests, and
+aborts stragglers. The session then pushes whatever is still buffered onto
+the session stream. The whole flush is bounded at 3 seconds so an
+unreachable gateway cannot delay the exit report.
+
+### OCSF Event Relay
+
+OCSF events generated inside the sandbox (e.g., network deny events) can also
+be forwarded through the same transport. A per-sandbox token bucket rate limiter
+controls the OCSF event rate, with configurable rate and drop counter.
+
+### Gateway-Side Handling
+
+The gateway receives `OtelExportData` messages and exports trace data through a
+dedicated `OtelRelayExporter` that connects directly to the configured OTLP
+collector. This bypasses the gateway's own `SdkTracerProvider` to preserve the
+supervisor-enriched resource attributes. OCSF events are emitted via
+`tracing::info!` on the `ocsf_relay` target.
+
 ## Policy Proposals
 
 When an L4 CONNECT is denied, the proxy emits a `DenialEvent`. The denial

crates/openshell-core/src/sandbox_env.rs

@@ -235,6 +235,29 @@ pub const SANDBOX_GID: &str = "OPENSHELL_SANDBOX_GID";
 /// OCI only for the former contract.
 pub const OCI_IMAGE_USER: &str = "OPENSHELL_OCI_IMAGE_USER";
 
+/// Standard OpenTelemetry environment variable for the OTLP exporter endpoint.
+///
+/// Set conditionally by the telemetry relay when the gateway has an OTLP
+/// endpoint configured. Points agent SDKs at the supervisor's local OTLP
+/// HTTP receiver.
+pub const OTEL_EXPORTER_OTLP_ENDPOINT: &str = "OTEL_EXPORTER_OTLP_ENDPOINT";
+
+/// Standard OpenTelemetry environment variable for the OTLP exporter protocol.
+///
+/// Set to `http/protobuf` when the telemetry relay is active.
+pub const OTEL_EXPORTER_OTLP_PROTOCOL: &str = "OTEL_EXPORTER_OTLP_PROTOCOL";
+
+/// Default OTLP receiver bind address and port.
+///
+/// All current topologies keep the process supervisor co-located with the
+/// agent, so localhost is correct. Future topologies that move the supervisor
+/// out of the agent's network namespace would derive the address from the
+/// topology (e.g., pod IP via downward API).
+pub const OTLP_RECEIVER_ADDR: &str = "127.0.0.1:4318";
+
+/// Default OTLP receiver endpoint URL for agent env var injection.
+pub const OTLP_RECEIVER_ENDPOINT: &str = "http://127.0.0.1:4318";
+
 // The corporate upstream-proxy configuration deliberately has no reserved
 // environment variables: it travels on the supervisor's argv
 // (`--upstream-proxy` and friends), which a sandbox image cannot forge the

crates/openshell-driver-kubernetes/src/driver.rs

@@ -4326,6 +4326,16 @@ fn build_env_list(
         tls_enabled,
         provider_spiffe_socket_path,
     );
+    upsert_env(
+        &mut env,
+        openshell_core::sandbox_env::OTEL_EXPORTER_OTLP_ENDPOINT,
+        openshell_core::sandbox_env::OTLP_RECEIVER_ENDPOINT,
+    );
+    upsert_env(
+        &mut env,
+        openshell_core::sandbox_env::OTEL_EXPORTER_OTLP_PROTOCOL,
+        "http/protobuf",
+    );
     env
 }
 

crates/openshell-ocsf/src/lib.rs

@@ -64,5 +64,6 @@ pub use builders::{
 
 // --- Tracing layers ---
 pub use tracing_layers::{
-    OCSF_TARGET, OcsfJsonlLayer, OcsfShorthandLayer, clone_current_event, emit_ocsf_event,
+    OCSF_TARGET, OcsfJsonlLayer, OcsfRelayLayer, OcsfRelaySink, OcsfShorthandLayer,
+    clone_current_event, emit_ocsf_event,
 };

crates/openshell-ocsf/src/tracing_layers/mod.rs

@@ -9,8 +9,10 @@
 
 pub(crate) mod event_bridge;
 mod jsonl_layer;
+mod relay_layer;
 mod shorthand_layer;
 
 pub use event_bridge::{OCSF_TARGET, clone_current_event, emit_ocsf_event};
 pub use jsonl_layer::OcsfJsonlLayer;
+pub use relay_layer::{OcsfRelayLayer, OcsfRelaySink};
 pub use shorthand_layer::OcsfShorthandLayer;

crates/openshell-ocsf/src/tracing_layers/relay_layer.rs

@@ -0,0 +1,46 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Tracing layer that captures OCSF events and forwards them as JSON bytes
+//! through a telemetry buffer sender for relay to the gateway.
+
+use std::sync::Arc;
+
+use tracing::Subscriber;
+use tracing_subscriber::Layer;
+use tracing_subscriber::layer::Context;
+
+use super::event_bridge::{OCSF_TARGET, clone_current_event};
+
+/// Callback trait for delivering serialized OCSF events.
+pub trait OcsfRelaySink: Send + Sync + 'static {
+    fn send(&self, json_bytes: Vec<u8>);
+}
+
+/// A tracing layer that captures OCSF events and serializes them to JSON
+/// for relay through the telemetry transport.
+pub struct OcsfRelayLayer {
+    sink: Arc<dyn OcsfRelaySink>,
+}
+
+impl OcsfRelayLayer {
+    pub fn new(sink: Arc<dyn OcsfRelaySink>) -> Self {
+        Self { sink }
+    }
+}
+
+impl<S: Subscriber> Layer<S> for OcsfRelayLayer {
+    fn on_event(&self, event: &tracing::Event<'_>, _ctx: Context<'_, S>) {
+        if event.metadata().target() != OCSF_TARGET {
+            return;
+        }
+
+        let Some(ocsf_event) = clone_current_event() else {
+            return;
+        };
+
+        if let Ok(json) = serde_json::to_vec(&ocsf_event) {
+            self.sink.send(json);
+        }
+    }
+}

crates/openshell-otel/src/lib.rs

@@ -5,7 +5,7 @@
 
 mod driver;
 mod grpc;
-mod propagation;
+pub mod propagation;
 
 pub use driver::{
     BoxGrpcStream, ComputeDriverTracing, DriverTracingConfig, DriverTracingHandle,