From 23f27e495502bfd7fa5b35dc88f9bbda15764510 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 17:44:08 +0200 Subject: [PATCH 01/22] docs(rfc): add RFC 0012 observability architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The centerpiece is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. Relates-to: #1055, #2507, #2508, #909 Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 344 ++++++++++++++++++ 1 file changed, 344 insertions(+) create mode 100644 rfc/0012-observability-architecture/README.md diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md new file mode 100644 index 0000000000..984060f33a --- /dev/null +++ b/rfc/0012-observability-architecture/README.md @@ -0,0 +1,344 @@ +--- +authors: + - "@rhuss" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/1055 + - https://github.com/NVIDIA/OpenShell/issues/2507 + - https://github.com/NVIDIA/OpenShell/issues/2508 + - https://github.com/NVIDIA/OpenShell/issues/909 +--- + +# RFC 0012 - OpenShell observability architecture + +## Summary + +This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The centerpiece is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. The relay complements recently merged infrastructure tracing for the gateway and VM driver, and builds on existing OCSF structured logging and Prometheus metrics foundations. + +The proposal also covers infrastructure trace instrumentation for components that have no OTEL coverage today (in-process compute drivers, CLI, sandbox supervisor), deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTEL correlation, and the metrics maturity path. + +## Motivation + +Operating OpenShell at scale requires answering questions that span multiple layers. A platform operator needs to know why sandbox creation is slow. A security reviewer needs to correlate a network deny event with the request that triggered it. An agent developer needs to see their agent's tool calls and LLM invocations in MLflow without writing OpenShell-specific instrumentation. A platform integrator needs a clean contract for wiring OpenShell into their monitoring stack. + +OpenShell's observability story is still early. That is expected for a project at this stage, and recent contributions have laid real foundations. OTLP trace export recently merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are solid starting points. + +The purpose of this RFC is to define where the story goes from here, so that the next contributions build toward a coherent architecture rather than growing organically. The main gaps today are: + +- Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTEL integration yet. +- No connection between OCSF security events and traces. An operator cannot pivot from a network deny event to the trace that caused it. +- The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is mostly unimplemented beyond the foundation. No ServiceMonitor, dashboards, or alerting rules exist. +- No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or Claude Code running inside sandboxes). The sandbox is network-isolated, so an agent's OTEL SDK cannot reach an external collector. + +If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. + +### Personas + +The personas below are proposed, not settled. OpenShell's persona model is evolving (see [#2615](https://github.com/NVIDIA/OpenShell/pull/2615) for ongoing work on centering issue reports around user stories). This RFC uses five personas as a lens for reasoning about observability needs, but the final definitions should be driven by the broader project discussion. + +| Persona | Role | Observability needs | +|---|---|---| +| **Platform operator** | Deploys and operates the gateway cluster, manages compute drivers, monitors platform health | Distributed traces, SLI/SLO dashboards, alerting rules, Prometheus RED metrics. Tools: Grafana, Jaeger/Tempo, Prometheus/Alertmanager | +| **Security/compliance reviewer** | Reviews audit logs, investigates security events, ensures policy compliance | OCSF JSONL events, trace-to-OCSF correlation. Tools: log aggregation (Loki, Elasticsearch, Splunk) | +| **Agent developer** | Runs agents in sandboxes, debugs agent behavior, optimizes performance | Agent-level traces in MLflow or similar, with infrastructure context (sandbox, policy). Note: agent traces are a separate visibility domain from operator-only infrastructure traces per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) | +| **Workspace administrator** | Manages a workspace (namespace, tenant) in a multi-tenant deployment | Per-workspace metrics: sandbox counts, policy violation rates, resource utilization | +| **Platform integrator** | Integrates OpenShell into a larger platform (managed K8s, cloud AI platform, on-prem) | Documented OTLP endpoint contracts, resource attribute schemas, Helm values, integration guides. Tools: OTel Collector config, Helm values | + +## Non-goals + +This RFC does not propose OTEL log export. Logs continue to go to stdout and OCSF JSONL, since container-level log collection already handles operational logs and adding a parallel OTLP path would not add clear value. The architecture does not preclude it later if a need emerges. + +OpenShell does not deploy an OTel Collector. The platform integrator owns the collector; OpenShell only emits OTLP to a configured endpoint. + +OTLP metrics push is planned as an opt-in complement to Prometheus but is not part of the initial scope. Similarly, TUI and router tracing are future items (the TUI consumes monitoring data, it does not produce it), and Python SDK OTEL hooks are tracked separately in [#1818](https://github.com/NVIDIA/OpenShell/issues/1818). + +The personas proposed in this RFC are a reasoning tool for the observability discussion, not settled project-wide definitions. That broader conversation is happening separately. + +## Proposal + +### Current state + +**Tracing:** + +| Component | OTLP Traces | W3C Propagation | Status | +|---|---|---|---| +| Gateway server | Per-request spans, gRPC conventions | Inbound + outbound to drivers | Recently merged ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) | +| VM driver | Per-RPC spans, lifecycle ops | Receives from gateway | Recently merged ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)) | +| `openshell-otel` crate | Shared provider, layer, error helpers | `TraceContextInterceptor` | Recently merged ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)) | +| In-process drivers (K8s/Docker/Podman) | None | N/A (in-process) | Gap | +| Sandbox supervisor | None | None | Gap | +| CLI | None | None | Gap | +| Agent traces from inside sandbox | None | N/A | Gap (new) | + +Configuration: `[openshell.gateway.otlp]` in `gateway.toml`. Table presence is the on-switch. + +**Logging:** Human-readable logs via `tracing_subscriber::fmt` to stdout. OCSF structured security logs via `openshell-ocsf` as shorthand (always on) and JSONL (opt-in), covering network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. OCSF is sandbox-side only. Centralized sandbox log collection is unsolved ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922), stale). + +**Monitoring:** Prometheus foundation in place (metrics facade, `/metrics` endpoint, Helm port 9090). Basic gRPC/HTTP RED metrics and interceptor metrics implemented. The broader catalog from [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is unimplemented. Zero metrics in the sandbox supervisor. No ServiceMonitor, dashboards, or alerting rules. + +### Supervisor as OTLP relay for sandbox traces + +The sandbox supervisor listens on `localhost:4317` (OTLP/gRPC) and `localhost:4318` (OTLP/HTTP) inside the sandbox. Agent frameworks export traces to this well-known address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. + +``` +Agent process (inside sandbox) + | OTLP/HTTP to localhost:4318 + v +Supervisor (OTLP receiver, inside sandbox) + | Session protocol (existing gRPC) + v +Gateway (OTLP relay) + | OTLP/gRPC to configured endpoint + v +External Collector (platform-managed) + | + v +MLflow / Jaeger / Grafana Tempo +``` + +The supervisor: + +1. Accepts OTLP spans from the agent process (both gRPC and HTTP protocols, matching standard OTel Collector receiver behavior) +2. Buffers in memory (bounded, with explicit drop semantics matching [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s transport design) +3. Optionally enriches spans with sandbox resource attributes (see span enrichment below) +4. Forwards to the gateway over the existing session protocol +5. Flushes buffered spans before shutdown, surviving agent process termination + +The gateway: + +1. Receives spans from supervisors via the session protocol +2. Relays to the configured external OTLP endpoint alongside its own spans +3. Applies sampling if configured + +For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTEL-instrumented framework works without OpenShell-specific code. + +### OTLP receiver reachability per driver + +There is a subtlety that matters for all container-based drivers. The supervisor creates an internal workload network namespace for the agent process, connected to the supervisor's namespace via a veth pair. The proxy already binds to the host-side veth IP (available from `netns.host_ip()`) to intercept agent traffic. This means `localhost` inside the agent's network namespace is not the same `localhost` the supervisor sees. + +The OTLP receiver cannot simply bind to `127.0.0.1:4318` and expect the agent to reach it. It needs to follow the same pattern as the proxy: bind to the veth host IP that is routable from inside the workload netns. The supervisor then sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://:4318` in the agent's environment rather than `http://localhost:4318`. + +How this plays out per driver: + +| Driver | Supervisor-agent network relationship | OTLP receiver binding | +|---|---|---| +| Docker | Same container, but agent enters a separate workload netns via `setns()` | Bind to `netns.host_ip():4318` (same IP the proxy uses). Reachable from agent in the workload netns. | +| Podman | Identical to Docker | Same as Docker | +| K8s (embedded) | Supervisor side-loaded into agent container, same single-container model | Same as Docker | +| K8s (sidecar) | Network sidecar + agent container share the pod network namespace. Sidecar uses nftables for interception, no separate workload netns. | Bind to `localhost:4318` directly. This is the simplest case because the agent process stays in the pod's shared network namespace. | +| VM | Supervisor is PID 1 inside a libkrun microVM, agent is a child process | Same as Docker (within the VM). The VM itself is isolated from the host; traces leave the VM through the supervisor-to-gateway session protocol, not through host networking. | + +The key point: the OTLP receiver reuses the same networking plumbing the proxy already uses. The proxy binds to the veth host IP; the OTLP receiver binds to the same IP on a different port. The infrastructure for making a supervisor-hosted service reachable from the agent process already exists. The OTLP receiver is another listener on that same address. + +The K8s sidecar topology is the exception. Because it uses nftables-based traffic interception in the pod's shared network namespace rather than creating a separate workload netns, `localhost` is truly shared between the supervisor and agent. Binding to `localhost:4318` works directly. + +This pattern has prior art in [Dapr](https://dapr.io). Dapr's [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) intercepts all application communication and automatically generates distributed traces without requiring any SDK or instrumentation in the application code. The sidecar writes traces using [OTLP to a configured collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/), which handles retries, batching, and encryption. Applications just annotate their deployment with `dapr.io/config` and get full distributed tracing transparently. OpenShell's supervisor plays the same role: the agent exports to `localhost:4318`, the supervisor handles buffering, enrichment, and forwarding. The agent never needs to know where traces go or how they get there. + +### Span links for sandbox-to-trace correlation + +Agent traces from inside the sandbox are correlated with gateway infrastructure traces via [span links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links), not parent-child relationships. + +``` +Gateway + sandbox.create span ─────────────────── (short-lived, completes in seconds) + {trace_id: abc, span_id: 123} + ^ ^ + │ span link │ span link + │ │ +Agent trace #1: Agent trace #2: + root: "agent.task" root: "agent.task" + tool_call_1 tool_call_2 + llm_invocation_1 +``` + +Sandbox lifetimes are unpredictable. A CI sandbox runs for 30 seconds, an interactive sandbox might live for hours, a persistent dev environment could run for days. A gateway-owned parent span that lives for the entire sandbox lifecycle is an antipattern: the batch exporter won't export it until the sandbox terminates, trace backends assume traces complete within a time window, and the gateway holds the open span in memory for every concurrent sandbox. + +Span links create a "related to" relationship without parent-child hierarchy. The gateway creates a short-lived `sandbox.create` span (completes in seconds, exported immediately), and the supervisor adds a span link from each agent root span to that gateway span during enrichment. Backends like Jaeger and Grafana Tempo navigate links bidirectionally. + +For batch/CI sandboxes where a single task maps to the entire sandbox lifetime, the supervisor can optionally create a short task-scoped parent span and pass it via `TRACEPARENT`. This keeps the parent-child relationship for the simple case without the long-lived trace antipattern. + +[#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s "Not settled" section discusses injecting `traceparent` into agent outbound HTTP requests (propagation outward). The span link pattern here is correlation inward, connecting agent traces to sandbox context. These are complementary decisions. + +### Configurable span enrichment at the supervisor + +When forwarding agent-emitted spans, the supervisor attaches sandbox context as resource attributes: + +| Attribute | Source | Example | +|---|---|---| +| `openshell.sandbox.id` | Sandbox metadata | `sb-abc123` | +| `openshell.sandbox.policy` | Active policy name | `default-policy` | +| `openshell.sandbox.user` | Authenticated user | `user@example.com` | +| `openshell.sandbox.image` | Container image | `ubuntu:22.04` | +| `openshell.sandbox.driver` | Compute driver type | `kubernetes` | + +Enrichment is configurable and can be disabled for pass-through forwarding. Without enrichment, an operator sees "LangChain called tool X" but cannot tell which sandbox, user, or policy was active. With enrichment, the trace carries full context without the agent needing any OpenShell-specific instrumentation. + +### OCSF-to-OTEL correlation + +When a trace context is active, OCSF events include `trace_id` and `span_id` fields. This lets operators pivot between "I see a network deny in the security log" and "show me the trace that triggered it." + +The OCSF builders already accept arbitrary fields. Add optional `trace_id`/`span_id` fields to the builder pattern, populated from the current `tracing::Span` when available. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 5 ("OCSF correlation") tracks this but has not defined the mechanism. + +### Infrastructure trace instrumentation + +The K8s, Docker, and Podman compute drivers run in-process within the gateway and already inherit its tracing subscriber. What they lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver serves as the reference pattern here, with 10+ instrumented operations using `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right foundation. + +The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. The fix is straightforward: add optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wire `TraceContextInterceptor` onto the gRPC channel, and create a root span per CLI command. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. + +The sandbox supervisor has zero OTEL integration today, which is the biggest single gap. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, OPA evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. + +### Deployment configuration + +Add `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` to Helm `values.yaml`, templated into the gateway's `gateway.toml` ConfigMap. Pass `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_RESOURCE_ATTRIBUTES` as env vars when configured. Add `[openshell.gateway.otlp]` to Docker Compose's `gateway.toml` example (commented out). Update `docs/reference/gateway-config.mdx`. + +OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context has its own collector topology, and deploying one would couple OpenShell to a particular backend. + +### Visibility domains + +Two distinct visibility domains flow through the same OTLP relay pipeline: + +1. **Infrastructure traces** (operator-only): gateway request spans, driver lifecycle, supervisor network/process/middleware spans. Locked as operator-only per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508). +2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow) from infrastructure traces (e.g., Tempo). + +The platform integrator configures this routing at the collector level (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). OpenShell emits both to the same OTLP endpoint; the collector separates them. + +### Metrics direction + +Metrics remain Prometheus-first. Priority order: + +1. Complete P0 metrics from [#909](https://github.com/NVIDIA/OpenShell/issues/909) (supervisor sessions, sandbox phase, relay) +2. Add Helm ServiceMonitor (gated, off by default) +3. P1 metrics (SSH, compute driver, DB, policy) +4. Dashboards and alerting rules +5. OTLP metrics push (opt-in complement) + +## Implementation plan + +### Phase 1: Core instrumentation + +This phase extends the recently merged gateway and VM driver tracing to the remaining infrastructure components. None of this work requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. + +**In-process driver spans.** The K8s, Docker, and Podman drivers run inside the gateway process and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to key operations (provisioning, teardown, status checks, exec) following the VM driver as the reference pattern. Since these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. The integration point is straightforward: instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`. The spans appear as children of the gateway's request span automatically. + +**CLI trace propagation.** The CLI needs to inject `traceparent` headers into its gRPC calls to the gateway. The building blocks already exist: `TraceContextInterceptor` in `openshell-otel` handles W3C trace context injection, and the gateway already extracts incoming `traceparent` in `multiplex.rs`. The work is wiring these together: add a `--otlp-endpoint` flag (or respect `OTEL_EXPORTER_OTLP_ENDPOINT`), build a `SdkTracerProvider` using the shared `openshell-otel` crate, create a root span per CLI command, and attach the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. + +**Helm and deploy configuration.** The Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. The integration point is `deploy/helm/templates/` where the ConfigMap is generated and the StatefulSet picks up env vars. Docker Compose's example `gateway.toml` gets a commented-out OTLP section. The published docs at `docs/reference/gateway-config.mdx` need the OTLP configuration reference. + +### Phase 2: Sandbox OTLP relay + +This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector, something no existing issue covers. + +**Supervisor OTLP receiver.** The supervisor needs an OTLP listener on `localhost:4317` (gRPC) and `localhost:4318` (HTTP) inside the sandbox. The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). The receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` in the agent process environment automatically. This is the integration point with any OTEL-instrumented agent framework: the agent's SDK exports to localhost, and the supervisor receives it. + +**Supervisor-to-gateway span transport.** Buffered spans need to reach the gateway. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 1 tracks this and it depends on [#1731](https://github.com/NVIDIA/OpenShell/issues/1731), which restructures the supervisor-to-gateway session protocol. The integration point is the session event stream between `crates/openshell-sandbox/` and `crates/openshell-server/`. A new message type carries serialized OTLP span batches from the supervisor to the gateway. Drop semantics need explicit design: what happens when the session is down, when the buffer is full, and whether partial traces are emitted or discarded. This is the hardest integration point because it touches the session protocol, which is shared infrastructure. + +**Gateway OTLP relay.** The gateway receives span batches from supervisors and forwards them to the configured external OTLP endpoint. Today the gateway only exports its own spans via the `SdkTracerProvider` built in `otel_tracing.rs`. The relay adds a second export path: spans received from supervisors go through the same OTLP/gRPC exporter but are not generated locally. The integration point is `crates/openshell-server/`, where the session handler receives span batches and the OTLP exporter sends them alongside the gateway's own spans. Sampling configuration (global or per-sandbox) applies here. + +**Span link correlation.** When the gateway creates a sandbox, it emits a short-lived `sandbox.create` span with sandbox metadata (ID, user, policy, driver). The span's `{trace_id, span_id}` is passed to the supervisor as part of the sandbox configuration. When the supervisor's OTLP receiver encounters an agent root span (no parent), it adds a span link pointing back to the gateway's `sandbox.create` span. The integration points are the sandbox creation flow in the gateway (where the span is created and context is passed) and the supervisor's span enrichment pipeline (where the link is added to forwarded spans). + +**Supervisor span enrichment.** Before forwarding agent spans, the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`). This runs in the same pipeline as span link injection. The integration point is between the OTLP receiver buffer and the session transport: spans are read from the buffer, enriched with attributes and links, then serialized for transport. Enrichment is configurable via an extension of the existing `[openshell.gateway.otlp]` config pattern. + +### Phase 3: Supervisor instrumentation depth + +This phase instruments the supervisor's own operations, producing infrastructure spans that serve the platform operator. The instrumentation work is independent of the OTLP relay (Phase 2 provides the transport, Phase 3 fills it with the supervisor's own spans). + +**Network spans.** The primary value. The candidate span sources in `crates/openshell-supervisor-network/` are: `proxy.rs` (L4 connect and tunnel establishment), `opa.rs` (policy evaluation latency), the `l7/` tree (HTTP/REST/GraphQL/WebSocket parsing and enforcement), `token_grant.rs` and `sigv4.rs` (credential injection), and `upstream_proxy.rs` (upstream dispatch). Not every function deserves a span. The decision per operation is span, event on a parent span, or nothing. This is a hot path (every outbound agent connection), so span construction cost needs measuring. The integration point is the shared `openshell-otel` crate for provider setup and the supervisor-to-gateway transport from Phase 2 for span delivery. + +**Middleware spans.** Operator-run middleware is the strongest cross-service tracing case in the product. A middleware call is an external gRPC service call, inline in the egress path, with a 500ms default timeout and fail-open/fail-closed semantics. The `HttpRequestEvaluation` type already carries a `request_id` field, so correlation plumbing partly exists. The integration point is `crates/openshell-supervisor-middleware/`, where the middleware chain executes and results are processed. + +**Process spans.** Entrypoint and process lifecycle, exec invocations, SSH session and channel operations. Smaller surface than the network side. The integration point is `crates/openshell-supervisor-process/`. + +**OCSF-to-OTEL correlation.** Add `trace_id` and `span_id` to OCSF events when a trace context is active. The integration point is the OCSF builder pattern in `crates/openshell-ocsf/`, where optional trace context fields are populated from the current `tracing::Span`. + +**Enablement and sampling.** Per-sandbox and global toggles following the existing `ocsf_json_enabled` settings pattern. Head sampling at the supervisor vs. tail sampling at the gateway have different tradeoffs on the hot egress path. The integration point is the sandbox settings flow from gateway to supervisor. + +### Phase 4: Monitoring maturity + +This phase fills out the metrics catalog and adds the Kubernetes monitoring surface. It is largely independent of the tracing work and can proceed in parallel once the metrics foundation is in place. + +**P0 metrics.** The metrics that tell you if the platform is fundamentally working: supervisor sessions active (gauge), connect/disconnect counters, sandboxes-by-phase gauge, relay opens/claims/pending. These are emitted from `crates/openshell-server/` using the existing `metrics` crate facade. The integration point is the same pattern as the existing gRPC/HTTP RED metrics, just covering new subsystems. + +**Helm ServiceMonitor.** A `ServiceMonitor` CRD template gated behind a Helm value (off by default). When enabled, it wires Prometheus scraping to the gateway's metrics port. The integration point is `deploy/helm/templates/`, adding a new template alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. + +**P1 metrics and beyond.** SSH tunnel saturation, compute driver RPC duration, DB operation latency, policy merge retries. Then dashboards (Grafana JSON models in the Helm chart or as separate artifacts), alerting rules (PrometheusRule CRDs), and eventually OTLP metrics push as an opt-in complement to Prometheus scrape for environments that cannot use pull-based collection. + +## Risks + +### Supervisor complexity and the hot path + +The OTLP relay adds an OTLP receiver, buffering, enrichment, and forwarding to the supervisor, which is already a complex component sitting on every outbound agent connection. Span construction on the egress path has a cost, and that cost needs measuring rather than assuming it is negligible. + +Related to this is the question of what happens when spans cannot be delivered. A dropped span is worse than a dropped log because it corrupts the parent-child structure of a trace, not just loses a record. The relay needs explicit, documented drop behavior when the gateway is unreachable or the buffer fills up. Inheriting the best-effort semantics from the existing log push path is not sufficient. + +### Span attribute cardinality + +Trace backends index span attributes for querying. When an attribute has a bounded set of values (like `rpc.method = "CreateSandbox"` or `openshell.sandbox.driver = "kubernetes"`), indexing is cheap. When the set of values is unbounded, the index grows without limit. + +This matters in OpenShell because the supervisor traces agent network activity. The natural attributes to record are the destination hostname and URL path, since those answer "where did the agent connect?" But the agent controls those values. A misbehaving or compromised agent could connect to thousands of unique hostnames, each becoming a distinct attribute value in the backend's index. At 10,000 unique hostnames from one sandbox, storage costs climb, query performance degrades, and some backends enforce cardinality limits that silently drop data. + +The mitigation is to normalize high-cardinality attributes before they reach the backend. Record known, bounded values like `dst.hostname = "api.openai.com"` as-is, but replace arbitrary agent-chosen hostnames with a hashed or bucketed value, or move them to span events (which are not indexed) instead of span attributes (which are). + +### Relay scale and gateway bottleneck + +The supervisor-to-gateway session protocol was designed for control plane traffic: sandbox lifecycle events, configuration updates, log pushes. Adding trace relay changes the traffic profile substantially. An active agent with OTEL instrumentation can produce 10-100 spans/second. With 100 concurrent sandboxes on one gateway, that adds up to 1,000-10,000 spans/second, roughly 1-10 MB/second sustained through the gateway. + +The network bandwidth alone is manageable. The deeper concern is that the gateway now serves two roles (control plane and data plane relay), and these roles compete for resources. If the external collector is slow or unreachable, the gateway accumulates buffered spans from all sandboxes. Memory pressure from the relay can degrade sandbox lifecycle operations. + +Several mitigations should be part of the implementation: + +- Trace relay should use a separate transport channel (or multiplexed stream) from the control plane session, so trace backpressure does not block sandbox create or delete operations. +- Head sampling at the supervisor reduces volume before it reaches the gateway. This should be configurable per-sandbox or globally (e.g., sample 10% of traces, but always forward traces containing errors). +- Per-sandbox span rate limits prevent one noisy agent from overwhelming the gateway. Excess spans are dropped at the supervisor with a counter metric, not silently. +- The gateway forwards received spans to the external collector asynchronously in batches, not synchronously per-span. +- When the gateway's relay buffer is full, it should signal the supervisor to slow down or drop rather than silently accepting until it runs out of memory. +- For high-volume scenarios, users can bypass the relay entirely by setting `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or a direct collector address. The relay is the zero-config default, not a mandatory path. + +### Other considerations + +The OpenTelemetry Rust SDK is a non-trivial dependency. Feature-gating it at compile time is worth evaluating (alongside [#1943](https://github.com/NVIDIA/OpenShell/issues/1943)) so that builds without OTEL support do not pay the binary size and compile time cost. + +Not all trace backends handle span links equally well. Jaeger and Grafana Tempo support bidirectional link navigation, but MLflow's OTLP ingestion may not surface links in its UI. If that turns out to be the case, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. + +## Alternatives + +### Use `host.openshell.internal` for agent trace collection + +Sandboxes can already reach host-side services via `host.openshell.internal` ([#2478](https://github.com/NVIDIA/OpenShell/issues/2478)). An agent could export OTLP directly to a collector on the gateway host. This works today (when the SSRF engine does not block it) and requires no new supervisor code. + +The supervisor relay is preferred as the default for four reasons: + +- Security: `host.openshell.internal` routes through the egress proxy with no per-sandbox authentication, so any sandbox can send data to the collector port. +- Multi-tenancy: one collector per gateway host does not scale to multi-tenant deployments where workspaces need separate OTLP endpoints. +- Reliability: the SSRF engine blocks `host.openshell.internal` on newer gateway versions ([#2478](https://github.com/NVIDIA/OpenShell/issues/2478)), TCP connections hang on the K8s driver, and it requires per-sandbox policy configuration. +- Enrichment: the relay enables the supervisor to attach sandbox context as resource attributes, which is unavailable on the direct path. + +These approaches are not mutually exclusive. The design follows the same pattern as `inference.local`: the relay is the zero-config default, and users can override `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or any other endpoint. + +### Gateway-owned root span with TRACEPARENT + +An earlier version of this design proposed a long-lived gateway root span for the sandbox lifecycle, with agent spans as children via `TRACEPARENT`. This was rejected because sandbox lifetimes are unpredictable (seconds to days) and long-lived parent spans are an OTEL antipattern: they delay export, strain backends, hold gateway memory, and misrepresent the semantic structure. Span links provide correlation without duration coupling. + +### Direct OTLP export from the sandbox + +[#2508](https://github.com/NVIDIA/OpenShell/issues/2508) considered and rejected direct OTLP export from the sandbox. It would require an egress allowlist hole for the collector endpoint in every sandbox, which is a policy exception with real blast radius. Routing through the gateway keeps the sandbox egress policy closed and gives a single place for sampling and redaction. + +### Do nothing + +Leave the current design unchanged. Platform operators cannot debug latency across the stack. Agent developers cannot get traces into MLflow without building workarounds. The three observability pillars remain disconnected. This is not acceptable for production deployments. + +## Prior art + +**[Dapr](https://dapr.io) sidecar architecture and OTEL integration.** Dapr's [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) intercepts all application communication and generates distributed traces automatically, without requiring any SDK or code instrumentation in the application. The sidecar writes traces using [OTLP to a configured OTel Collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/), which handles retries, batching, and encryption. Applications opt in with a single Kubernetes annotation (`dapr.io/config`). The three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The key lesson: transparent trace collection through a co-located proxy is a proven pattern that scales across diverse application frameworks. + +**OpenTelemetry GenAI semantic conventions.** The emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. + +**Root span flushing in CI pipelines.** Red Hat's CI pipelines for running agents in OpenShell sandboxes independently discovered the root span flushing problem for short-lived sandboxes and worked around it with an orchestrator-owned root span passed via `TRACEPARENT`. The span link approach in this RFC avoids the duration coupling while solving the same correlation problem. + +## Open questions + +- **Config surface precedence.** `OTEL_*` env vars vs TOML gateway config. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and RFC 0003 discuss this but no decision has been made. +- **Per-workspace OTLP endpoints.** Multi-tenant deployments likely need per-workspace OTLP endpoint configuration so different tenants' agent traces route to different collectors or MLflow instances. The configuration model for this is not designed. +- **Span link support in MLflow.** Whether MLflow's OTLP ingestion surfaces span links in its UI needs validation. If it does not, the correlation story for agent developers using MLflow is limited to resource attributes. +- **Compile-time feature gating.** Whether the OTel SDK dependency should be gated behind a Cargo feature flag for builds that do not need tracing. +- **OTEL receiver implementation.** Whether the supervisor should implement the OTLP receiver from scratch using the `opentelemetry-proto` crate, or embed a lightweight collector library. The implementation complexity and binary size tradeoffs need evaluation. +- **`traceparent` injection into agent egress.** [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) deliberately left this unsettled. It is a separate decision from the span link correlation proposed here. From 84cb9df0b57a7a4a9f260fa9e09e7eae7e016b2e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 17:50:17 +0200 Subject: [PATCH 02/22] docs(rfc): convert diagrams to mermaid, improve prose flow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace ASCII art with Mermaid diagrams for GitHub rendering. Expand terse fragments into flowing paragraphs throughout the Current State, Deployment Configuration, and Open Questions sections. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 59 +++++++------------ 1 file changed, 21 insertions(+), 38 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 984060f33a..da0f0c4766 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -58,7 +58,7 @@ The personas proposed in this RFC are a reasoning tool for the observability dis ### Current state -**Tracing:** +The table below summarizes where tracing coverage exists today and where the gaps are. The gateway and VM driver recently gained OTLP trace export through three merged PRs, with a shared `openshell-otel` crate providing the common infrastructure. Everything else has no OTEL integration. Tracing is configured through `[openshell.gateway.otlp]` in `gateway.toml`, where the table's presence acts as the on-switch. | Component | OTLP Traces | W3C Propagation | Status | |---|---|---|---| @@ -70,30 +70,20 @@ The personas proposed in this RFC are a reasoning tool for the observability dis | CLI | None | None | Gap | | Agent traces from inside sandbox | None | N/A | Gap (new) | -Configuration: `[openshell.gateway.otlp]` in `gateway.toml`. Table presence is the on-switch. +On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. It runs only on the sandbox side. Centralized sandbox log collection remains an unsolved problem ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922), currently stale). -**Logging:** Human-readable logs via `tracing_subscriber::fmt` to stdout. OCSF structured security logs via `openshell-ocsf` as shorthand (always on) and JSONL (opt-in), covering network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. OCSF is sandbox-side only. Centralized sandbox log collection is unsolved ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922), stale). - -**Monitoring:** Prometheus foundation in place (metrics facade, `/metrics` endpoint, Helm port 9090). Basic gRPC/HTTP RED metrics and interceptor metrics implemented. The broader catalog from [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is unimplemented. Zero metrics in the sandbox supervisor. No ServiceMonitor, dashboards, or alerting rules. +For monitoring, the Prometheus foundation is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. ### Supervisor as OTLP relay for sandbox traces The sandbox supervisor listens on `localhost:4317` (OTLP/gRPC) and `localhost:4318` (OTLP/HTTP) inside the sandbox. Agent frameworks export traces to this well-known address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. -``` -Agent process (inside sandbox) - | OTLP/HTTP to localhost:4318 - v -Supervisor (OTLP receiver, inside sandbox) - | Session protocol (existing gRPC) - v -Gateway (OTLP relay) - | OTLP/gRPC to configured endpoint - v -External Collector (platform-managed) - | - v -MLflow / Jaeger / Grafana Tempo +```mermaid +graph TD + A["Agent process
(inside sandbox)"] -->|"OTLP/HTTP
localhost:4318"| B["Supervisor
(OTLP receiver)"] + B -->|"Session protocol
(existing gRPC)"| C["Gateway
(OTLP relay)"] + C -->|"OTLP/gRPC
configured endpoint"| D["External Collector
(platform-managed)"] + D --> E["MLflow / Jaeger / Grafana Tempo"] ``` The supervisor: @@ -138,17 +128,10 @@ This pattern has prior art in [Dapr](https://dapr.io). Dapr's [sidecar](https:// Agent traces from inside the sandbox are correlated with gateway infrastructure traces via [span links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links), not parent-child relationships. -``` -Gateway - sandbox.create span ─────────────────── (short-lived, completes in seconds) - {trace_id: abc, span_id: 123} - ^ ^ - │ span link │ span link - │ │ -Agent trace #1: Agent trace #2: - root: "agent.task" root: "agent.task" - tool_call_1 tool_call_2 - llm_invocation_1 +```mermaid +graph BT + A1["Agent trace #1
root: agent.task
├ tool_call_1
└ llm_invocation_1"] -.->|"span link"| GW["Gateway
sandbox.create span
(short-lived, completes in seconds)"] + A2["Agent trace #2
root: agent.task
└ tool_call_2"] -.->|"span link"| GW ``` Sandbox lifetimes are unpredictable. A CI sandbox runs for 30 seconds, an interactive sandbox might live for hours, a persistent dev environment could run for days. A gateway-owned parent span that lives for the entire sandbox lifecycle is an antipattern: the batch exporter won't export it until the sandbox terminates, trace backends assume traces complete within a time window, and the gateway holds the open span in memory for every concurrent sandbox. @@ -189,9 +172,9 @@ The sandbox supervisor has zero OTEL integration today, which is the biggest sin ### Deployment configuration -Add `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` to Helm `values.yaml`, templated into the gateway's `gateway.toml` ConfigMap. Pass `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_RESOURCE_ATTRIBUTES` as env vars when configured. Add `[openshell.gateway.otlp]` to Docker Compose's `gateway.toml` example (commented out). Update `docs/reference/gateway-config.mdx`. +Today, none of the deployment surfaces (Helm, Docker Compose, RPM) include OTLP configuration. The Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. When these values are set, the chart should also pass `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` as environment variables to the gateway container. Docker Compose's example `gateway.toml` should include a commented-out OTLP section so operators can see the option and enable it. The published configuration reference at `docs/reference/gateway-config.mdx` needs to document the OTLP settings. -OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context has its own collector topology, and deploying one would couple OpenShell to a particular backend. +An important design constraint: OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context (managed Kubernetes, standalone cluster, Docker Compose, bare metal) has its own collector topology, and bundling one would couple OpenShell to a particular backend while creating operational overlap with platform-level monitoring infrastructure. ### Visibility domains @@ -336,9 +319,9 @@ Leave the current design unchanged. Platform operators cannot debug latency acro ## Open questions -- **Config surface precedence.** `OTEL_*` env vars vs TOML gateway config. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and RFC 0003 discuss this but no decision has been made. -- **Per-workspace OTLP endpoints.** Multi-tenant deployments likely need per-workspace OTLP endpoint configuration so different tenants' agent traces route to different collectors or MLflow instances. The configuration model for this is not designed. -- **Span link support in MLflow.** Whether MLflow's OTLP ingestion surfaces span links in its UI needs validation. If it does not, the correlation story for agent developers using MLflow is limited to resource attributes. -- **Compile-time feature gating.** Whether the OTel SDK dependency should be gated behind a Cargo feature flag for builds that do not need tracing. -- **OTEL receiver implementation.** Whether the supervisor should implement the OTLP receiver from scratch using the `opentelemetry-proto` crate, or embed a lightweight collector library. The implementation complexity and binary size tradeoffs need evaluation. -- **`traceparent` injection into agent egress.** [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) deliberately left this unsettled. It is a separate decision from the span link correlation proposed here. +- How should `OTEL_*` environment variables and TOML gateway configuration interact when both are set? [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and RFC 0003 discuss this but no decision has been made on precedence. +- Multi-tenant deployments likely need per-workspace OTLP endpoint configuration so that different tenants' agent traces can route to different collectors or MLflow instances. The configuration model for per-workspace endpoints is not designed yet. +- Does MLflow's OTLP ingestion surface span links in its UI? If it does not, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. This needs validation. +- Should the OTel SDK dependency be gated behind a Cargo feature flag so that builds without tracing support do not pay the binary size and compile time cost? +- Should the supervisor implement the OTLP receiver from scratch using the `opentelemetry-proto` crate (smaller binary, more control), or embed a lightweight collector library (more features, larger dependency)? The tradeoffs need evaluation during implementation. +- [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) deliberately left `traceparent` injection into agent egress unsettled. Whether the supervisor stamps W3C trace context onto the agent's outbound HTTP requests is a separate decision from the span link correlation proposed in this RFC. From cda1c832e3e3b9a6358166aa37136c84c058a287 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 18:13:56 +0200 Subject: [PATCH 03/22] docs(rfc): polish RFC 0012 prose, remove AI writing patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace bold-keyword fragments in all Implementation Plan phases with flowing prose and natural transitions. Remove signposting, persuasive authority tropes, and promotional language throughout. Trim redundant Dapr description between Proposal and Prior Art sections. Add cross-references from Implementation Plan back to Proposal to reduce repetition. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 72 +++++++++---------- 1 file changed, 35 insertions(+), 37 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index da0f0c4766..cddc3a0f85 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -13,7 +13,7 @@ links: ## Summary -This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The centerpiece is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. The relay complements recently merged infrastructure tracing for the gateway and VM driver, and builds on existing OCSF structured logging and Prometheus metrics foundations. +This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The core idea is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. The relay complements recently merged infrastructure tracing for the gateway and VM driver, and builds on existing OCSF structured logging and Prometheus metrics. The proposal also covers infrastructure trace instrumentation for components that have no OTEL coverage today (in-process compute drivers, CLI, sandbox supervisor), deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTEL correlation, and the metrics maturity path. @@ -21,13 +21,11 @@ The proposal also covers infrastructure trace instrumentation for components tha Operating OpenShell at scale requires answering questions that span multiple layers. A platform operator needs to know why sandbox creation is slow. A security reviewer needs to correlate a network deny event with the request that triggered it. An agent developer needs to see their agent's tool calls and LLM invocations in MLflow without writing OpenShell-specific instrumentation. A platform integrator needs a clean contract for wiring OpenShell into their monitoring stack. -OpenShell's observability story is still early. That is expected for a project at this stage, and recent contributions have laid real foundations. OTLP trace export recently merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are solid starting points. - -The purpose of this RFC is to define where the story goes from here, so that the next contributions build toward a coherent architecture rather than growing organically. The main gaps today are: +OpenShell's observability story is still early. That is expected for a project at this stage, and recent contributions have made real progress. OTLP trace export recently merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are reasonable starting points, but the gaps are significant enough that the next contributions need a coherent target rather than growing ad hoc: - Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTEL integration yet. - No connection between OCSF security events and traces. An operator cannot pivot from a network deny event to the trace that caused it. -- The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is mostly unimplemented beyond the foundation. No ServiceMonitor, dashboards, or alerting rules exist. +- The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is mostly unimplemented beyond the initial wiring. No ServiceMonitor, dashboards, or alerting rules exist. - No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or Claude Code running inside sandboxes). The sandbox is network-isolated, so an agent's OTEL SDK cannot reach an external collector. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. @@ -58,7 +56,7 @@ The personas proposed in this RFC are a reasoning tool for the observability dis ### Current state -The table below summarizes where tracing coverage exists today and where the gaps are. The gateway and VM driver recently gained OTLP trace export through three merged PRs, with a shared `openshell-otel` crate providing the common infrastructure. Everything else has no OTEL integration. Tracing is configured through `[openshell.gateway.otlp]` in `gateway.toml`, where the table's presence acts as the on-switch. +Tracing coverage today is limited to the gateway and VM driver. Both recently gained OTLP trace export through three merged PRs, with a shared `openshell-otel` crate providing the common infrastructure. Everything else has no OTEL integration. Tracing is configured through `[openshell.gateway.otlp]` in `gateway.toml`, where the table's presence acts as the on-switch. | Component | OTLP Traces | W3C Propagation | Status | |---|---|---|---| @@ -72,7 +70,7 @@ The table below summarizes where tracing coverage exists today and where the gap On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. It runs only on the sandbox side. Centralized sandbox log collection remains an unsolved problem ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922), currently stale). -For monitoring, the Prometheus foundation is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. +For monitoring, the Prometheus wiring is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. ### Supervisor as OTLP relay for sandbox traces @@ -104,7 +102,7 @@ For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automaticall ### OTLP receiver reachability per driver -There is a subtlety that matters for all container-based drivers. The supervisor creates an internal workload network namespace for the agent process, connected to the supervisor's namespace via a veth pair. The proxy already binds to the host-side veth IP (available from `netns.host_ip()`) to intercept agent traffic. This means `localhost` inside the agent's network namespace is not the same `localhost` the supervisor sees. +All container-based drivers share a networking subtlety that affects how the OTLP receiver is reached. The supervisor creates an internal workload network namespace for the agent process, connected to the supervisor's namespace via a veth pair. The proxy already binds to the host-side veth IP (available from `netns.host_ip()`) to intercept agent traffic. This means `localhost` inside the agent's network namespace is not the same `localhost` the supervisor sees. The OTLP receiver cannot simply bind to `127.0.0.1:4318` and expect the agent to reach it. It needs to follow the same pattern as the proxy: bind to the veth host IP that is routable from inside the workload netns. The supervisor then sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://:4318` in the agent's environment rather than `http://localhost:4318`. @@ -118,11 +116,11 @@ How this plays out per driver: | K8s (sidecar) | Network sidecar + agent container share the pod network namespace. Sidecar uses nftables for interception, no separate workload netns. | Bind to `localhost:4318` directly. This is the simplest case because the agent process stays in the pod's shared network namespace. | | VM | Supervisor is PID 1 inside a libkrun microVM, agent is a child process | Same as Docker (within the VM). The VM itself is isolated from the host; traces leave the VM through the supervisor-to-gateway session protocol, not through host networking. | -The key point: the OTLP receiver reuses the same networking plumbing the proxy already uses. The proxy binds to the veth host IP; the OTLP receiver binds to the same IP on a different port. The infrastructure for making a supervisor-hosted service reachable from the agent process already exists. The OTLP receiver is another listener on that same address. +In practice, the OTLP receiver reuses the same networking plumbing the proxy already uses. The proxy binds to the veth host IP; the OTLP receiver binds to the same IP on a different port. The infrastructure for making a supervisor-hosted service reachable from the agent process already exists, and the OTLP receiver is just another listener on that address. The K8s sidecar topology is the exception. Because it uses nftables-based traffic interception in the pod's shared network namespace rather than creating a separate workload netns, `localhost` is truly shared between the supervisor and agent. Binding to `localhost:4318` works directly. -This pattern has prior art in [Dapr](https://dapr.io). Dapr's [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) intercepts all application communication and automatically generates distributed traces without requiring any SDK or instrumentation in the application code. The sidecar writes traces using [OTLP to a configured collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/), which handles retries, batching, and encryption. Applications just annotate their deployment with `dapr.io/config` and get full distributed tracing transparently. OpenShell's supervisor plays the same role: the agent exports to `localhost:4318`, the supervisor handles buffering, enrichment, and forwarding. The agent never needs to know where traces go or how they get there. +This pattern follows [Dapr](https://dapr.io)'s [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) approach, where a co-located proxy intercepts application communication and generates distributed traces through an [OTLP collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/) without requiring application-level instrumentation. The agent exports to a local address; the supervisor handles buffering, enrichment, and forwarding. ### Span links for sandbox-to-trace correlation @@ -164,17 +162,17 @@ The OCSF builders already accept arbitrary fields. Add optional `trace_id`/`span ### Infrastructure trace instrumentation -The K8s, Docker, and Podman compute drivers run in-process within the gateway and already inherit its tracing subscriber. What they lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver serves as the reference pattern here, with 10+ instrumented operations using `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right foundation. +The K8s, Docker, and Podman compute drivers run in-process within the gateway and already inherit its tracing subscriber. What they lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver is the reference here, with 10+ instrumented operations using `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right starting point. -The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. The fix is straightforward: add optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wire `TraceContextInterceptor` onto the gRPC channel, and create a root span per CLI command. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. +The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. Adding optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wiring `TraceContextInterceptor` onto the gRPC channel, and creating a root span per CLI command would close this gap. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. -The sandbox supervisor has zero OTEL integration today, which is the biggest single gap. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, OPA evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. +The sandbox supervisor has zero OTEL integration today, the largest gap in the stack. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, OPA evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. ### Deployment configuration Today, none of the deployment surfaces (Helm, Docker Compose, RPM) include OTLP configuration. The Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. When these values are set, the chart should also pass `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` as environment variables to the gateway container. Docker Compose's example `gateway.toml` should include a commented-out OTLP section so operators can see the option and enable it. The published configuration reference at `docs/reference/gateway-config.mdx` needs to document the OTLP settings. -An important design constraint: OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context (managed Kubernetes, standalone cluster, Docker Compose, bare metal) has its own collector topology, and bundling one would couple OpenShell to a particular backend while creating operational overlap with platform-level monitoring infrastructure. +OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context (managed Kubernetes, standalone cluster, Docker Compose, bare metal) has its own collector topology, and bundling one would couple OpenShell to a particular backend while creating operational overlap with platform-level monitoring infrastructure. ### Visibility domains @@ -199,51 +197,51 @@ Metrics remain Prometheus-first. Priority order: ### Phase 1: Core instrumentation -This phase extends the recently merged gateway and VM driver tracing to the remaining infrastructure components. None of this work requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. +This phase extends the recently merged gateway and VM driver tracing to the remaining infrastructure components. None of it requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. -**In-process driver spans.** The K8s, Docker, and Podman drivers run inside the gateway process and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to key operations (provisioning, teardown, status checks, exec) following the VM driver as the reference pattern. Since these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. The integration point is straightforward: instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`. The spans appear as children of the gateway's request span automatically. +The K8s, Docker, and Podman drivers are the easiest starting point because they run inside the gateway process and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to operations like provisioning, teardown, status checks, and exec, following the VM driver as a reference. Because these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. Instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`, and the resulting spans appear as children of the gateway's request span automatically. -**CLI trace propagation.** The CLI needs to inject `traceparent` headers into its gRPC calls to the gateway. The building blocks already exist: `TraceContextInterceptor` in `openshell-otel` handles W3C trace context injection, and the gateway already extracts incoming `traceparent` in `multiplex.rs`. The work is wiring these together: add a `--otlp-endpoint` flag (or respect `OTEL_EXPORTER_OTLP_ENDPOINT`), build a `SdkTracerProvider` using the shared `openshell-otel` crate, create a root span per CLI command, and attach the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. +The CLI is a different problem. Today it does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new, disconnected traces at the gateway rather than continuing from the user's command. The building blocks already exist: `TraceContextInterceptor` in `openshell-otel` handles W3C trace context injection, and the gateway already extracts incoming `traceparent` in `multiplex.rs`. Wiring these together means adding a `--otlp-endpoint` flag (or respecting `OTEL_EXPORTER_OTLP_ENDPOINT`), building a `SdkTracerProvider` using the shared `openshell-otel` crate, creating a root span per CLI command, and attaching the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. -**Helm and deploy configuration.** The Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. The integration point is `deploy/helm/templates/` where the ConfigMap is generated and the StatefulSet picks up env vars. Docker Compose's example `gateway.toml` gets a commented-out OTLP section. The published docs at `docs/reference/gateway-config.mdx` need the OTLP configuration reference. +On the deployment side, the Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. The integration surface is `deploy/helm/templates/` where the ConfigMap is generated and the StatefulSet picks up environment variables. Docker Compose's example `gateway.toml` should include a commented-out OTLP section, and the published docs at `docs/reference/gateway-config.mdx` need to cover the OTLP configuration. ### Phase 2: Sandbox OTLP relay -This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector, something no existing issue covers. +This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector, something no existing issue covers end to end. The relay design is described in the Proposal section; this section focuses on where the code changes land and what depends on what. -**Supervisor OTLP receiver.** The supervisor needs an OTLP listener on `localhost:4317` (gRPC) and `localhost:4318` (HTTP) inside the sandbox. The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). The receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` in the agent process environment automatically. This is the integration point with any OTEL-instrumented agent framework: the agent's SDK exports to localhost, and the supervisor receives it. +The supervisor needs an OTLP listener on `localhost:4317` (gRPC) and `localhost:4318` (HTTP) inside the sandbox. The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). Either way, the receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` in the agent process environment automatically, so any OTEL-instrumented framework exports to localhost and the supervisor picks it up. -**Supervisor-to-gateway span transport.** Buffered spans need to reach the gateway. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 1 tracks this and it depends on [#1731](https://github.com/NVIDIA/OpenShell/issues/1731), which restructures the supervisor-to-gateway session protocol. The integration point is the session event stream between `crates/openshell-sandbox/` and `crates/openshell-server/`. A new message type carries serialized OTLP span batches from the supervisor to the gateway. Drop semantics need explicit design: what happens when the session is down, when the buffer is full, and whether partial traces are emitted or discarded. This is the hardest integration point because it touches the session protocol, which is shared infrastructure. +Getting those buffered spans to the gateway is the harder part. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 1 tracks this and it depends on [#1731](https://github.com/NVIDIA/OpenShell/issues/1731), which restructures the supervisor-to-gateway session protocol. A new message type would carry serialized OTLP span batches from the supervisor to the gateway over the session event stream between `crates/openshell-sandbox/` and `crates/openshell-server/`. Drop semantics need explicit design: what happens when the session is down, when the buffer fills up, and whether partial traces are emitted or discarded. This is the hardest integration point in the entire proposal because it touches the session protocol, which is shared infrastructure. -**Gateway OTLP relay.** The gateway receives span batches from supervisors and forwards them to the configured external OTLP endpoint. Today the gateway only exports its own spans via the `SdkTracerProvider` built in `otel_tracing.rs`. The relay adds a second export path: spans received from supervisors go through the same OTLP/gRPC exporter but are not generated locally. The integration point is `crates/openshell-server/`, where the session handler receives span batches and the OTLP exporter sends them alongside the gateway's own spans. Sampling configuration (global or per-sandbox) applies here. +On the gateway side, the relay receives span batches from supervisors and forwards them to the configured external OTLP endpoint. Today the gateway only exports its own spans via the `SdkTracerProvider` built in `otel_tracing.rs`. The relay adds a second export path: spans received from supervisors go through the same OTLP/gRPC exporter but are not generated locally. The relevant code is in `crates/openshell-server/`, where the session handler receives span batches and the OTLP exporter sends them alongside the gateway's own spans. Sampling configuration (global or per-sandbox) applies at this stage. -**Span link correlation.** When the gateway creates a sandbox, it emits a short-lived `sandbox.create` span with sandbox metadata (ID, user, policy, driver). The span's `{trace_id, span_id}` is passed to the supervisor as part of the sandbox configuration. When the supervisor's OTLP receiver encounters an agent root span (no parent), it adds a span link pointing back to the gateway's `sandbox.create` span. The integration points are the sandbox creation flow in the gateway (where the span is created and context is passed) and the supervisor's span enrichment pipeline (where the link is added to forwarded spans). +Correlation between agent traces and gateway traces happens through span links, as described in the Proposal. When the gateway creates a sandbox, it emits a short-lived `sandbox.create` span with sandbox metadata (ID, user, policy, driver). The span's `{trace_id, span_id}` is passed to the supervisor as part of the sandbox configuration. When the supervisor's OTLP receiver encounters an agent root span (one with no parent), it adds a span link pointing back to the gateway's `sandbox.create` span. The integration points are the sandbox creation flow in the gateway (where the span is created and context is passed) and the supervisor's span processing pipeline (where the link is added). -**Supervisor span enrichment.** Before forwarding agent spans, the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`). This runs in the same pipeline as span link injection. The integration point is between the OTLP receiver buffer and the session transport: spans are read from the buffer, enriched with attributes and links, then serialized for transport. Enrichment is configurable via an extension of the existing `[openshell.gateway.otlp]` config pattern. +Before forwarding agent spans, the supervisor also attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`). This runs in the same pipeline as span link injection: spans are read from the buffer, enriched with attributes and links, then serialized for transport. Enrichment is configurable via an extension of the existing `[openshell.gateway.otlp]` config pattern. ### Phase 3: Supervisor instrumentation depth -This phase instruments the supervisor's own operations, producing infrastructure spans that serve the platform operator. The instrumentation work is independent of the OTLP relay (Phase 2 provides the transport, Phase 3 fills it with the supervisor's own spans). +This phase instruments the supervisor's own operations, producing infrastructure spans for platform operators. The instrumentation work is independent of the OTLP relay: Phase 2 provides the transport, Phase 3 fills it with the supervisor's own spans. -**Network spans.** The primary value. The candidate span sources in `crates/openshell-supervisor-network/` are: `proxy.rs` (L4 connect and tunnel establishment), `opa.rs` (policy evaluation latency), the `l7/` tree (HTTP/REST/GraphQL/WebSocket parsing and enforcement), `token_grant.rs` and `sigv4.rs` (credential injection), and `upstream_proxy.rs` (upstream dispatch). Not every function deserves a span. The decision per operation is span, event on a parent span, or nothing. This is a hot path (every outbound agent connection), so span construction cost needs measuring. The integration point is the shared `openshell-otel` crate for provider setup and the supervisor-to-gateway transport from Phase 2 for span delivery. +The network path is where most of the value lies. The candidate span sources in `crates/openshell-supervisor-network/` include `proxy.rs` (L4 connect and tunnel establishment), `opa.rs` (policy evaluation latency), the `l7/` tree (HTTP/REST/GraphQL/WebSocket parsing and enforcement), `token_grant.rs` and `sigv4.rs` (credential injection), and `upstream_proxy.rs` (upstream dispatch). Not every function deserves a span; the decision per operation is span, event on a parent span, or nothing. This is a hot path (every outbound agent connection), so span construction cost needs measuring. The supervisor uses the shared `openshell-otel` crate for provider setup and the transport from Phase 2 for span delivery. -**Middleware spans.** Operator-run middleware is the strongest cross-service tracing case in the product. A middleware call is an external gRPC service call, inline in the egress path, with a 500ms default timeout and fail-open/fail-closed semantics. The `HttpRequestEvaluation` type already carries a `request_id` field, so correlation plumbing partly exists. The integration point is `crates/openshell-supervisor-middleware/`, where the middleware chain executes and results are processed. +Operator-run middleware has the strongest cross-service tracing case in the product. A middleware call is an external gRPC service invocation, inline in the egress path, with a 500ms default timeout and fail-open/fail-closed semantics. The `HttpRequestEvaluation` type already carries a `request_id` field, so some correlation plumbing exists. The relevant code is in `crates/openshell-supervisor-middleware/`. -**Process spans.** Entrypoint and process lifecycle, exec invocations, SSH session and channel operations. Smaller surface than the network side. The integration point is `crates/openshell-supervisor-process/`. +Process lifecycle spans (entrypoint start/stop, exec invocations, SSH session and channel operations) are a smaller surface than the network side. The relevant code is in `crates/openshell-supervisor-process/`. -**OCSF-to-OTEL correlation.** Add `trace_id` and `span_id` to OCSF events when a trace context is active. The integration point is the OCSF builder pattern in `crates/openshell-ocsf/`, where optional trace context fields are populated from the current `tracing::Span`. +Connecting OCSF events to traces requires adding `trace_id` and `span_id` fields to OCSF events when a trace context is active. The OCSF builder pattern in `crates/openshell-ocsf/` needs optional trace context fields populated from the current `tracing::Span`. -**Enablement and sampling.** Per-sandbox and global toggles following the existing `ocsf_json_enabled` settings pattern. Head sampling at the supervisor vs. tail sampling at the gateway have different tradeoffs on the hot egress path. The integration point is the sandbox settings flow from gateway to supervisor. +Enablement and sampling follow the existing `ocsf_json_enabled` settings pattern, with per-sandbox and global toggles. Head sampling at the supervisor versus tail sampling at the gateway have different tradeoffs on the hot egress path. The configuration flows from gateway to supervisor through the sandbox settings. ### Phase 4: Monitoring maturity -This phase fills out the metrics catalog and adds the Kubernetes monitoring surface. It is largely independent of the tracing work and can proceed in parallel once the metrics foundation is in place. +This phase fills out the metrics catalog and adds the Kubernetes monitoring surface. It is largely independent of the tracing work and can proceed in parallel. -**P0 metrics.** The metrics that tell you if the platform is fundamentally working: supervisor sessions active (gauge), connect/disconnect counters, sandboxes-by-phase gauge, relay opens/claims/pending. These are emitted from `crates/openshell-server/` using the existing `metrics` crate facade. The integration point is the same pattern as the existing gRPC/HTTP RED metrics, just covering new subsystems. +The first priority is completing the P0 metrics that tell you whether the platform is fundamentally working: supervisor sessions active (gauge), connect/disconnect counters, sandboxes-by-phase gauge, relay opens/claims/pending. These are emitted from `crates/openshell-server/` using the existing `metrics` crate facade, following the same pattern as the existing gRPC/HTTP RED metrics but covering new subsystems. -**Helm ServiceMonitor.** A `ServiceMonitor` CRD template gated behind a Helm value (off by default). When enabled, it wires Prometheus scraping to the gateway's metrics port. The integration point is `deploy/helm/templates/`, adding a new template alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. +Next is a Helm `ServiceMonitor` CRD template, gated behind a Helm value and off by default. When enabled, it wires Prometheus scraping to the gateway's metrics port. The template lives in `deploy/helm/templates/` alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. -**P1 metrics and beyond.** SSH tunnel saturation, compute driver RPC duration, DB operation latency, policy merge retries. Then dashboards (Grafana JSON models in the Helm chart or as separate artifacts), alerting rules (PrometheusRule CRDs), and eventually OTLP metrics push as an opt-in complement to Prometheus scrape for environments that cannot use pull-based collection. +After that comes P1 metrics (SSH tunnel saturation, compute driver RPC duration, DB operation latency, policy merge retries), then dashboards (Grafana JSON models in the Helm chart or as separate artifacts), alerting rules (PrometheusRule CRDs), and eventually OTLP metrics push as an opt-in complement to Prometheus scrape for environments that cannot use pull-based collection. ## Risks @@ -311,11 +309,11 @@ Leave the current design unchanged. Platform operators cannot debug latency acro ## Prior art -**[Dapr](https://dapr.io) sidecar architecture and OTEL integration.** Dapr's [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) intercepts all application communication and generates distributed traces automatically, without requiring any SDK or code instrumentation in the application. The sidecar writes traces using [OTLP to a configured OTel Collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/), which handles retries, batching, and encryption. Applications opt in with a single Kubernetes annotation (`dapr.io/config`). The three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The key lesson: transparent trace collection through a co-located proxy is a proven pattern that scales across diverse application frameworks. +**[Dapr](https://dapr.io) sidecar architecture and OTEL integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. -**OpenTelemetry GenAI semantic conventions.** The emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. +The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define the emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. -**Root span flushing in CI pipelines.** Red Hat's CI pipelines for running agents in OpenShell sandboxes independently discovered the root span flushing problem for short-lived sandboxes and worked around it with an orchestrator-owned root span passed via `TRACEPARENT`. The span link approach in this RFC avoids the duration coupling while solving the same correlation problem. +Red Hat's CI pipelines for running agents in OpenShell sandboxes independently discovered the root span flushing problem for short-lived sandboxes and worked around it with an orchestrator-owned root span passed via `TRACEPARENT`. The span link approach in this RFC avoids the duration coupling while solving the same correlation problem. ## Open questions From d7994880ce4e2657e2ffeee3e4d89ed56fe8c235 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 18:17:56 +0200 Subject: [PATCH 04/22] docs(rfc): restructure summary to frame infra vs agent observability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce the two observability layers (infrastructure-level and agent-level) before presenting the OTLP relay. The summary now establishes what exists on the infrastructure side, then positions the supervisor relay as the solution for the harder agent-level problem where sandbox network isolation prevents direct export. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index cddc3a0f85..2593b3bf51 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -13,9 +13,11 @@ links: ## Summary -This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The core idea is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. The relay complements recently merged infrastructure tracing for the gateway and VM driver, and builds on existing OCSF structured logging and Prometheus metrics. +This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. OpenShell observability splits into two distinct layers: infrastructure-level and agent-level. Infrastructure-level observability covers the platform itself (gateway request traces, compute driver lifecycle, supervisor network decisions, OCSF security events, Prometheus metrics). Agent-level observability covers what happens inside sandboxes (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI). -The proposal also covers infrastructure trace instrumentation for components that have no OTEL coverage today (in-process compute drivers, CLI, sandbox supervisor), deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTEL correlation, and the metrics maturity path. +On the infrastructure side, recent contributions have started building coverage. OTLP trace export merged for the gateway and VM driver, OCSF structured logging covers security events, and Prometheus metrics provide basic request counting. This RFC extends that foundation to the remaining components (in-process compute drivers, CLI, sandbox supervisor) and adds deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTEL correlation, and a metrics maturity path. + +On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OTEL SDK cannot reach an external collector. This RFC proposes a supervisor-local OTLP relay as the solution. The supervisor listens on standard OTLP ports inside the sandbox, collects agent-emitted traces, enriches them with sandbox context, and forwards them through the gateway to an external collector. The agent just exports to `localhost:4318`; the platform handles the rest. ## Motivation From f0386fed2633af56edaaa0291b7de673ad4ed471 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 18:34:38 +0200 Subject: [PATCH 05/22] docs(rfc): address copyedit findings, add multi-tenancy section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix critical localhost vs veth-host-IP contradiction in relay section, diagram, and Phase 2. Standardize OTel/OTLP capitalization throughout. Expand all abbreviations on first use (OCSF, OTLP, OTel, SLI/SLO, RED, OPA, SSRF). Add Proposal and Implementation Plan roadmap paragraphs. Move Visibility domains earlier to establish two-domain framing before the relay design. Add new Multi-tenant observability section covering per-workspace OTLP endpoints, trace isolation, and config model. Add openshell.workspace.id to the enrichment attributes table. Add TOML and Helm YAML configuration examples. Fix Mermaid relay diagram to show gRPC+HTTP ports instead of misleading localhost. Add span link annotation to correlation diagram. Link RFC 0003 and reference #1055 in body. Replace temporal "recently merged" language. Reduce Proposal vs Implementation Plan redundancy. Tighten Do-nothing alternative and cardinality risk section. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 129 +++++++++++------- 1 file changed, 82 insertions(+), 47 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 2593b3bf51..4bf1afe9f2 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -13,24 +13,24 @@ links: ## Summary -This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. OpenShell observability splits into two distinct layers: infrastructure-level and agent-level. Infrastructure-level observability covers the platform itself (gateway request traces, compute driver lifecycle, supervisor network decisions, OCSF security events, Prometheus metrics). Agent-level observability covers what happens inside sandboxes (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI). +This RFC proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. OpenShell observability splits into two distinct layers: infrastructure-level and agent-level. Infrastructure-level observability covers the platform itself (gateway request traces, compute driver lifecycle, supervisor network decisions, Open Cybersecurity Schema Framework (OCSF) security events, Prometheus metrics). Agent-level observability covers what happens inside sandboxes (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI). -On the infrastructure side, recent contributions have started building coverage. OTLP trace export merged for the gateway and VM driver, OCSF structured logging covers security events, and Prometheus metrics provide basic request counting. This RFC extends that foundation to the remaining components (in-process compute drivers, CLI, sandbox supervisor) and adds deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTEL correlation, and a metrics maturity path. +On the infrastructure side, contributions have started building coverage. OpenTelemetry Protocol (OTLP) trace export merged for the gateway and VM driver, OCSF structured logging covers security events, and Prometheus metrics provide basic request counting. This RFC extends that to the remaining components (in-process compute drivers, CLI, sandbox supervisor) and adds deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTel correlation, and a metrics maturity path. -On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OTEL SDK cannot reach an external collector. This RFC proposes a supervisor-local OTLP relay as the solution. The supervisor listens on standard OTLP ports inside the sandbox, collects agent-emitted traces, enriches them with sandbox context, and forwards them through the gateway to an external collector. The agent just exports to `localhost:4318`; the platform handles the rest. +On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector. This RFC proposes a supervisor-local OTLP relay as the solution. The supervisor listens on standard OTLP ports inside the sandbox, collects agent-emitted traces, enriches them with sandbox context, and forwards them through the gateway to an external collector. The agent just exports to a well-known local endpoint; the platform handles the rest. ## Motivation Operating OpenShell at scale requires answering questions that span multiple layers. A platform operator needs to know why sandbox creation is slow. A security reviewer needs to correlate a network deny event with the request that triggered it. An agent developer needs to see their agent's tool calls and LLM invocations in MLflow without writing OpenShell-specific instrumentation. A platform integrator needs a clean contract for wiring OpenShell into their monitoring stack. -OpenShell's observability story is still early. That is expected for a project at this stage, and recent contributions have made real progress. OTLP trace export recently merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are reasonable starting points, but the gaps are significant enough that the next contributions need a coherent target rather than growing ad hoc: +OpenShell's observability story is still early. That is expected for a project at this stage, and contributions have made real progress. OTLP trace export merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are reasonable starting points, but the gaps are significant enough that the next contributions need a coherent target rather than growing ad hoc: -- Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTEL integration yet. +- Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTel integration yet. - No connection between OCSF security events and traces. An operator cannot pivot from a network deny event to the trace that caused it. -- The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, SLI/SLO definitions) is mostly unimplemented beyond the initial wiring. No ServiceMonitor, dashboards, or alerting rules exist. -- No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or Claude Code running inside sandboxes). The sandbox is network-isolated, so an agent's OTEL SDK cannot reach an external collector. +- The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, Service Level Indicator/Service Level Objective (SLI/SLO) definitions) is mostly unimplemented beyond the initial wiring. No ServiceMonitor, dashboards, or alerting rules exist. +- No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI running inside sandboxes). The sandbox is network-isolated, so an agent's OTel SDK cannot reach an external collector. -If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. +[#1055](https://github.com/NVIDIA/OpenShell/issues/1055) tracks the overall enterprise observability effort. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. ### Personas @@ -38,7 +38,7 @@ The personas below are proposed, not settled. OpenShell's persona model is evolv | Persona | Role | Observability needs | |---|---|---| -| **Platform operator** | Deploys and operates the gateway cluster, manages compute drivers, monitors platform health | Distributed traces, SLI/SLO dashboards, alerting rules, Prometheus RED metrics. Tools: Grafana, Jaeger/Tempo, Prometheus/Alertmanager | +| **Platform operator** | Deploys and operates the gateway cluster, manages compute drivers, monitors platform health | Distributed traces, SLI/SLO dashboards, alerting rules, Prometheus Rate/Errors/Duration (RED) metrics. Tools: Grafana, Jaeger/Tempo, Prometheus/Alertmanager | | **Security/compliance reviewer** | Reviews audit logs, investigates security events, ensures policy compliance | OCSF JSONL events, trace-to-OCSF correlation. Tools: log aggregation (Loki, Elasticsearch, Splunk) | | **Agent developer** | Runs agents in sandboxes, debugs agent behavior, optimizes performance | Agent-level traces in MLflow or similar, with infrastructure context (sandbox, policy). Note: agent traces are a separate visibility domain from operator-only infrastructure traces per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) | | **Workspace administrator** | Manages a workspace (namespace, tenant) in a multi-tenant deployment | Per-workspace metrics: sandbox counts, policy violation rates, resource utilization | @@ -46,26 +46,28 @@ The personas below are proposed, not settled. OpenShell's persona model is evolv ## Non-goals -This RFC does not propose OTEL log export. Logs continue to go to stdout and OCSF JSONL, since container-level log collection already handles operational logs and adding a parallel OTLP path would not add clear value. The architecture does not preclude it later if a need emerges. +This RFC does not propose OTel log export. Logs continue to go to stdout and OCSF JSONL, since container-level log collection already handles operational logs and adding a parallel OTLP path would not add clear value. The architecture does not preclude it later if a need emerges. OpenShell does not deploy an OTel Collector. The platform integrator owns the collector; OpenShell only emits OTLP to a configured endpoint. -OTLP metrics push is planned as an opt-in complement to Prometheus but is not part of the initial scope. Similarly, TUI and router tracing are future items (the TUI consumes monitoring data, it does not produce it), and Python SDK OTEL hooks are tracked separately in [#1818](https://github.com/NVIDIA/OpenShell/issues/1818). +OTLP metrics push is planned as an opt-in complement to Prometheus but is not part of the initial scope. Similarly, TUI and router tracing are future items (the TUI consumes monitoring data, it does not produce it), and Python SDK OTel hooks are tracked separately in [#1818](https://github.com/NVIDIA/OpenShell/issues/1818). The personas proposed in this RFC are a reasoning tool for the observability discussion, not settled project-wide definitions. That broader conversation is happening separately. ## Proposal +The proposal is organized around the current state, the novel OTLP relay for agent traces (the architecturally new piece), the supporting mechanisms (span links, enrichment, OCSF correlation), infrastructure instrumentation for the remaining components, deployment configuration, multi-tenant considerations, and the metrics path. The two visibility domains (infrastructure vs. agent traces) cut across all of these. + ### Current state -Tracing coverage today is limited to the gateway and VM driver. Both recently gained OTLP trace export through three merged PRs, with a shared `openshell-otel` crate providing the common infrastructure. Everything else has no OTEL integration. Tracing is configured through `[openshell.gateway.otlp]` in `gateway.toml`, where the table's presence acts as the on-switch. +Tracing coverage today is limited to the gateway and VM driver. Both gained OTLP trace export through three merged PRs, with a shared `openshell-otel` crate providing the common infrastructure. Everything else has no OTel integration. Tracing is configured through `[openshell.gateway.otlp]` in `gateway.toml`, where the table's presence acts as the on-switch. | Component | OTLP Traces | W3C Propagation | Status | |---|---|---|---| -| Gateway server | Per-request spans, gRPC conventions | Inbound + outbound to drivers | Recently merged ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) | -| VM driver | Per-RPC spans, lifecycle ops | Receives from gateway | Recently merged ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)) | -| `openshell-otel` crate | Shared provider, layer, error helpers | `TraceContextInterceptor` | Recently merged ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)) | -| In-process drivers (K8s/Docker/Podman) | None | N/A (in-process) | Gap | +| Gateway server | Per-request spans, gRPC conventions | Inbound + outbound to drivers | Merged ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) | +| VM driver | Per-RPC spans, lifecycle ops | Receives from gateway | Merged ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)) | +| `openshell-otel` crate | Shared provider, layer, error helpers | `TraceContextInterceptor` | Merged ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)) | +| In-process drivers (K8s embedded/sidecar, Docker, Podman) | None | N/A (in-process) | Gap | | Sandbox supervisor | None | None | Gap | | CLI | None | None | Gap | | Agent traces from inside sandbox | None | N/A | Gap (new) | @@ -74,13 +76,22 @@ On the logging side, OpenShell has two systems serving distinct purposes. Human- For monitoring, the Prometheus wiring is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. +### Visibility domains + +Two distinct visibility domains flow through the same OTLP relay pipeline: + +1. **Infrastructure traces** (operator-only): gateway request spans, driver lifecycle, supervisor network/process/middleware spans. Locked as operator-only per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508). +2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow) from infrastructure traces (e.g., Tempo). + +The platform integrator configures this routing at the collector level (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). OpenShell emits both to the same OTLP endpoint; the collector separates them. + ### Supervisor as OTLP relay for sandbox traces -The sandbox supervisor listens on `localhost:4317` (OTLP/gRPC) and `localhost:4318` (OTLP/HTTP) inside the sandbox. Agent frameworks export traces to this well-known address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. +The sandbox supervisor (hereafter "supervisor") listens on standard OTLP ports inside the sandbox, accepting both gRPC (port 4317) and HTTP (port 4318). Agent frameworks export traces to this address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. ```mermaid graph TD - A["Agent process
(inside sandbox)"] -->|"OTLP/HTTP
localhost:4318"| B["Supervisor
(OTLP receiver)"] + A["Agent process
(inside sandbox)"] -->|"OTLP gRPC+HTTP
(ports 4317/4318)"| B["Supervisor
(OTLP receiver)"] B -->|"Session protocol
(existing gRPC)"| C["Gateway
(OTLP relay)"] C -->|"OTLP/gRPC
configured endpoint"| D["External Collector
(platform-managed)"] D --> E["MLflow / Jaeger / Grafana Tempo"] @@ -100,7 +111,7 @@ The gateway: 2. Relays to the configured external OTLP endpoint alongside its own spans 3. Applies sampling if configured -For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTEL-instrumented framework works without OpenShell-specific code. +For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTel-instrumented framework works without OpenShell-specific code. ### OTLP receiver reachability per driver @@ -130,7 +141,7 @@ Agent traces from inside the sandbox are correlated with gateway infrastructure ```mermaid graph BT - A1["Agent trace #1
root: agent.task
├ tool_call_1
└ llm_invocation_1"] -.->|"span link"| GW["Gateway
sandbox.create span
(short-lived, completes in seconds)"] + A1["Agent trace #1
root: agent.task
├ tool_call_1
└ llm_invocation_1"] -.->|"span link
(supervisor adds
during enrichment)"| GW["Gateway
sandbox.create span
(short-lived, completes in seconds)"] A2["Agent trace #2
root: agent.task
└ tool_call_2"] -.->|"span link"| GW ``` @@ -153,10 +164,11 @@ When forwarding agent-emitted spans, the supervisor attaches sandbox context as | `openshell.sandbox.user` | Authenticated user | `user@example.com` | | `openshell.sandbox.image` | Container image | `ubuntu:22.04` | | `openshell.sandbox.driver` | Compute driver type | `kubernetes` | +| `openshell.workspace.id` | Workspace/tenant identifier | `ws-prod-team-a` | Enrichment is configurable and can be disabled for pass-through forwarding. Without enrichment, an operator sees "LangChain called tool X" but cannot tell which sandbox, user, or policy was active. With enrichment, the trace carries full context without the agent needing any OpenShell-specific instrumentation. -### OCSF-to-OTEL correlation +### OCSF-to-OTel correlation When a trace context is active, OCSF events include `trace_id` and `span_id` fields. This lets operators pivot between "I see a network deny in the security log" and "show me the trace that triggered it." @@ -168,26 +180,48 @@ The K8s, Docker, and Podman compute drivers run in-process within the gateway an The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. Adding optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wiring `TraceContextInterceptor` onto the gRPC channel, and creating a root span per CLI command would close this gap. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. -The sandbox supervisor has zero OTEL integration today, the largest gap in the stack. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, OPA evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. +The sandbox supervisor has zero OTel integration today, the largest gap in the stack. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, Open Policy Agent (OPA) evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. ### Deployment configuration -Today, none of the deployment surfaces (Helm, Docker Compose, RPM) include OTLP configuration. The Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. When these values are set, the chart should also pass `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` as environment variables to the gateway container. Docker Compose's example `gateway.toml` should include a commented-out OTLP section so operators can see the option and enable it. The published configuration reference at `docs/reference/gateway-config.mdx` needs to document the OTLP settings. +Today, none of the deployment surfaces (Helm, Docker Compose, RPM) include OTLP configuration. + +The gateway's `gateway.toml` needs an OTLP section: + +```toml +[openshell.gateway.otlp] +endpoint = "http://otel-collector.monitoring:4317" +service_name = "openshell-gateway" +``` + +The Helm chart needs corresponding values that template into the ConfigMap: + +```yaml +otlp: + enabled: false + endpoint: "" + serviceName: "openshell-gateway" +``` + +When these values are set, the chart should also pass `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES` as environment variables to the gateway container. Docker Compose's example `gateway.toml` should include a commented-out OTLP section so operators can see the option and enable it. The published configuration reference at `docs/reference/gateway-config.mdx` needs to document the OTLP settings. OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collector. Every deployment context (managed Kubernetes, standalone cluster, Docker Compose, bare metal) has its own collector topology, and bundling one would couple OpenShell to a particular backend while creating operational overlap with platform-level monitoring infrastructure. -### Visibility domains +### Multi-tenant observability -Two distinct visibility domains flow through the same OTLP relay pipeline: +Multi-tenant deployments need per-workspace observability isolation. Different workspaces may route agent traces to different collectors or MLflow instances, and one tenant's trace data must not leak to another's endpoint. -1. **Infrastructure traces** (operator-only): gateway request spans, driver lifecycle, supervisor network/process/middleware spans. Locked as operator-only per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508). -2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow) from infrastructure traces (e.g., Tempo). +The OTLP relay supports this through per-workspace OTLP endpoint configuration at the gateway. When a supervisor forwards spans, the gateway looks up the workspace's configured endpoint and routes accordingly. If no per-workspace endpoint is configured, spans go to the global default. The `openshell.workspace.id` resource attribute (added during span enrichment) enables collector-side routing for deployments that use a shared collector with attribute-based routing rather than per-workspace endpoints. -The platform integrator configures this routing at the collector level (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). OpenShell emits both to the same OTLP endpoint; the collector separates them. +Per-workspace sampling and rate limits follow the same pattern as per-sandbox limits described in the Risks section, but scoped to the workspace level. A noisy workspace should not exhaust the gateway's relay capacity for other tenants. + +The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before Phase 2 implementation. ### Metrics direction -Metrics remain Prometheus-first. Priority order: +Metrics remain Prometheus-first. The tracing architecture above is the primary focus of this RFC; the metrics path is included here because it is part of the observability scope, but the work is largely independent. + +Priority order: 1. Complete P0 metrics from [#909](https://github.com/NVIDIA/OpenShell/issues/909) (supervisor sessions, sandbox phase, relay) 2. Add Helm ServiceMonitor (gated, off by default) @@ -197,21 +231,23 @@ Metrics remain Prometheus-first. Priority order: ## Implementation plan +The implementation is phased to manage dependencies. Phase 1 extends existing tracing to the remaining infrastructure components and requires no new transport mechanisms. Phase 2 introduces the OTLP relay (the architecturally new piece) and depends on the session protocol restructuring in [#1731](https://github.com/NVIDIA/OpenShell/issues/1731). Phase 3 deepens supervisor instrumentation using the relay transport from Phase 2. Phase 4 covers monitoring maturity and can run in parallel with the other phases. + ### Phase 1: Core instrumentation -This phase extends the recently merged gateway and VM driver tracing to the remaining infrastructure components. None of it requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. +This phase extends gateway and VM driver tracing to the remaining infrastructure components. None of it requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. The K8s, Docker, and Podman drivers are the easiest starting point because they run inside the gateway process and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to operations like provisioning, teardown, status checks, and exec, following the VM driver as a reference. Because these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. Instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`, and the resulting spans appear as children of the gateway's request span automatically. -The CLI is a different problem. Today it does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new, disconnected traces at the gateway rather than continuing from the user's command. The building blocks already exist: `TraceContextInterceptor` in `openshell-otel` handles W3C trace context injection, and the gateway already extracts incoming `traceparent` in `multiplex.rs`. Wiring these together means adding a `--otlp-endpoint` flag (or respecting `OTEL_EXPORTER_OTLP_ENDPOINT`), building a `SdkTracerProvider` using the shared `openshell-otel` crate, creating a root span per CLI command, and attaching the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. +The CLI is a different problem. The gap and building blocks are described in the Infrastructure trace instrumentation section. Wiring them together means adding a `--otlp-endpoint` flag (or respecting `OTEL_EXPORTER_OTLP_ENDPOINT`), building a `SdkTracerProvider` using the shared `openshell-otel` crate, creating a root span per CLI command, and attaching the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. -On the deployment side, the Helm chart needs `otlp.endpoint`, `otlp.serviceName`, and `otlp.enabled` values that template into the gateway's `gateway.toml` ConfigMap as `[openshell.gateway.otlp]`. The integration surface is `deploy/helm/templates/` where the ConfigMap is generated and the StatefulSet picks up environment variables. Docker Compose's example `gateway.toml` should include a commented-out OTLP section, and the published docs at `docs/reference/gateway-config.mdx` need to cover the OTLP configuration. +On the deployment side, the Helm and TOML configuration described in the Deployment configuration section needs to land. The integration surface is `deploy/helm/openshell/templates/` where the ConfigMap is generated and the StatefulSet picks up environment variables. Docker Compose's example `gateway.toml` should include a commented-out OTLP section, and the published docs at `docs/reference/gateway-config.mdx` need to cover the OTLP configuration. ### Phase 2: Sandbox OTLP relay -This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector, something no existing issue covers end to end. The relay design is described in the Proposal section; this section focuses on where the code changes land and what depends on what. +This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector. The relay design is described in the Proposal section; this section focuses on where the code changes land and what depends on what. -The supervisor needs an OTLP listener on `localhost:4317` (gRPC) and `localhost:4318` (HTTP) inside the sandbox. The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). Either way, the receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` in the agent process environment automatically, so any OTEL-instrumented framework exports to localhost and the supervisor picks it up. +The supervisor needs an OTLP listener on ports 4317 (gRPC) and 4318 (HTTP), bound to the address reachable from the agent process (the veth host IP for container-based drivers, localhost for K8s sidecar; see OTLP receiver reachability). The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). Either way, the receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT` in the agent process environment automatically, pointing at the correct address. Getting those buffered spans to the gateway is the harder part. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 1 tracks this and it depends on [#1731](https://github.com/NVIDIA/OpenShell/issues/1731), which restructures the supervisor-to-gateway session protocol. A new message type would carry serialized OTLP span batches from the supervisor to the gateway over the session event stream between `crates/openshell-sandbox/` and `crates/openshell-server/`. Drop semantics need explicit design: what happens when the session is down, when the buffer fills up, and whether partial traces are emitted or discarded. This is the hardest integration point in the entire proposal because it touches the session protocol, which is shared infrastructure. @@ -219,13 +255,13 @@ On the gateway side, the relay receives span batches from supervisors and forwar Correlation between agent traces and gateway traces happens through span links, as described in the Proposal. When the gateway creates a sandbox, it emits a short-lived `sandbox.create` span with sandbox metadata (ID, user, policy, driver). The span's `{trace_id, span_id}` is passed to the supervisor as part of the sandbox configuration. When the supervisor's OTLP receiver encounters an agent root span (one with no parent), it adds a span link pointing back to the gateway's `sandbox.create` span. The integration points are the sandbox creation flow in the gateway (where the span is created and context is passed) and the supervisor's span processing pipeline (where the link is added). -Before forwarding agent spans, the supervisor also attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`). This runs in the same pipeline as span link injection: spans are read from the buffer, enriched with attributes and links, then serialized for transport. Enrichment is configurable via an extension of the existing `[openshell.gateway.otlp]` config pattern. +Before forwarding agent spans, the supervisor also attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.sandbox.policy`, `openshell.sandbox.user`, `openshell.sandbox.image`, `openshell.sandbox.driver`, `openshell.workspace.id`). This runs in the same pipeline as span link injection: spans are read from the buffer, enriched with attributes and links, then serialized for transport. Enrichment is configurable via an extension of the existing `[openshell.gateway.otlp]` config pattern. ### Phase 3: Supervisor instrumentation depth This phase instruments the supervisor's own operations, producing infrastructure spans for platform operators. The instrumentation work is independent of the OTLP relay: Phase 2 provides the transport, Phase 3 fills it with the supervisor's own spans. -The network path is where most of the value lies. The candidate span sources in `crates/openshell-supervisor-network/` include `proxy.rs` (L4 connect and tunnel establishment), `opa.rs` (policy evaluation latency), the `l7/` tree (HTTP/REST/GraphQL/WebSocket parsing and enforcement), `token_grant.rs` and `sigv4.rs` (credential injection), and `upstream_proxy.rs` (upstream dispatch). Not every function deserves a span; the decision per operation is span, event on a parent span, or nothing. This is a hot path (every outbound agent connection), so span construction cost needs measuring. The supervisor uses the shared `openshell-otel` crate for provider setup and the transport from Phase 2 for span delivery. +The network path is where most of the value lies. The candidate span sources in `crates/openshell-supervisor-network/` include `proxy.rs` (L4 connect and tunnel establishment), `opa.rs` (OPA policy evaluation latency), the `l7/` tree (HTTP/REST/GraphQL/WebSocket parsing and enforcement), `token_grant.rs` and `sigv4.rs` (credential injection), and `upstream_proxy.rs` (upstream dispatch). Not every function deserves a span; the decision per operation is span, event on a parent span, or nothing. This is a hot path (every outbound agent connection), so span construction cost needs measuring. The supervisor uses the shared `openshell-otel` crate for provider setup and the transport from Phase 2 for span delivery. Operator-run middleware has the strongest cross-service tracing case in the product. A middleware call is an external gRPC service invocation, inline in the egress path, with a 500ms default timeout and fail-open/fail-closed semantics. The `HttpRequestEvaluation` type already carries a `request_id` field, so some correlation plumbing exists. The relevant code is in `crates/openshell-supervisor-middleware/`. @@ -241,7 +277,7 @@ This phase fills out the metrics catalog and adds the Kubernetes monitoring surf The first priority is completing the P0 metrics that tell you whether the platform is fundamentally working: supervisor sessions active (gauge), connect/disconnect counters, sandboxes-by-phase gauge, relay opens/claims/pending. These are emitted from `crates/openshell-server/` using the existing `metrics` crate facade, following the same pattern as the existing gRPC/HTTP RED metrics but covering new subsystems. -Next is a Helm `ServiceMonitor` CRD template, gated behind a Helm value and off by default. When enabled, it wires Prometheus scraping to the gateway's metrics port. The template lives in `deploy/helm/templates/` alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. +Next is a Helm `ServiceMonitor` CRD template, gated behind a Helm value and off by default. When enabled, it wires Prometheus scraping to the gateway's metrics port. The template lives in `deploy/helm/openshell/templates/` alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. After that comes P1 metrics (SSH tunnel saturation, compute driver RPC duration, DB operation latency, policy merge retries), then dashboards (Grafana JSON models in the Helm chart or as separate artifacts), alerting rules (PrometheusRule CRDs), and eventually OTLP metrics push as an opt-in complement to Prometheus scrape for environments that cannot use pull-based collection. @@ -255,15 +291,15 @@ Related to this is the question of what happens when spans cannot be delivered. ### Span attribute cardinality -Trace backends index span attributes for querying. When an attribute has a bounded set of values (like `rpc.method = "CreateSandbox"` or `openshell.sandbox.driver = "kubernetes"`), indexing is cheap. When the set of values is unbounded, the index grows without limit. +Trace backends index span attributes for querying. Bounded attribute values (like `rpc.method = "CreateSandbox"`) are cheap to index; unbounded values cause the index to grow without limit. -This matters in OpenShell because the supervisor traces agent network activity. The natural attributes to record are the destination hostname and URL path, since those answer "where did the agent connect?" But the agent controls those values. A misbehaving or compromised agent could connect to thousands of unique hostnames, each becoming a distinct attribute value in the backend's index. At 10,000 unique hostnames from one sandbox, storage costs climb, query performance degrades, and some backends enforce cardinality limits that silently drop data. +This matters because the supervisor traces agent network activity, and the agent controls values like destination hostname and URL path. A misbehaving or compromised agent could connect to thousands of unique hostnames, each becoming a distinct attribute value in the backend's index. At 10,000 unique hostnames from one sandbox, storage costs climb, query performance degrades, and some backends enforce cardinality limits that silently drop data. The mitigation is to normalize high-cardinality attributes before they reach the backend. Record known, bounded values like `dst.hostname = "api.openai.com"` as-is, but replace arbitrary agent-chosen hostnames with a hashed or bucketed value, or move them to span events (which are not indexed) instead of span attributes (which are). ### Relay scale and gateway bottleneck -The supervisor-to-gateway session protocol was designed for control plane traffic: sandbox lifecycle events, configuration updates, log pushes. Adding trace relay changes the traffic profile substantially. An active agent with OTEL instrumentation can produce 10-100 spans/second. With 100 concurrent sandboxes on one gateway, that adds up to 1,000-10,000 spans/second, roughly 1-10 MB/second sustained through the gateway. +The supervisor-to-gateway session protocol was designed for control plane traffic: sandbox lifecycle events, configuration updates, log pushes. Adding trace relay changes the traffic profile substantially. An active agent with OTel instrumentation can produce 10-100 spans/second. With 100 concurrent sandboxes on one gateway, that adds up to 1,000-10,000 spans/second, roughly 1-10 MB/second sustained through the gateway. The network bandwidth alone is manageable. The deeper concern is that the gateway now serves two roles (control plane and data plane relay), and these roles compete for resources. If the external collector is slow or unreachable, the gateway accumulates buffered spans from all sandboxes. Memory pressure from the relay can degrade sandbox lifecycle operations. @@ -278,7 +314,7 @@ Several mitigations should be part of the implementation: ### Other considerations -The OpenTelemetry Rust SDK is a non-trivial dependency. Feature-gating it at compile time is worth evaluating (alongside [#1943](https://github.com/NVIDIA/OpenShell/issues/1943)) so that builds without OTEL support do not pay the binary size and compile time cost. +The OpenTelemetry Rust SDK is a non-trivial dependency. Feature-gating it at compile time is worth evaluating (alongside [#1943](https://github.com/NVIDIA/OpenShell/issues/1943)) so that builds without OTel support do not pay the binary size and compile time cost. Not all trace backends handle span links equally well. Jaeger and Grafana Tempo support bidirectional link navigation, but MLflow's OTLP ingestion may not surface links in its UI. If that turns out to be the case, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. @@ -286,7 +322,7 @@ Not all trace backends handle span links equally well. Jaeger and Grafana Tempo ### Use `host.openshell.internal` for agent trace collection -Sandboxes can already reach host-side services via `host.openshell.internal` ([#2478](https://github.com/NVIDIA/OpenShell/issues/2478)). An agent could export OTLP directly to a collector on the gateway host. This works today (when the SSRF engine does not block it) and requires no new supervisor code. +Sandboxes can already reach host-side services via `host.openshell.internal` ([#2478](https://github.com/NVIDIA/OpenShell/issues/2478)). An agent could export OTLP directly to a collector on the gateway host. This works today (when the Server-Side Request Forgery (SSRF) engine does not block it) and requires no new supervisor code. The supervisor relay is preferred as the default for four reasons: @@ -299,7 +335,7 @@ These approaches are not mutually exclusive. The design follows the same pattern ### Gateway-owned root span with TRACEPARENT -An earlier version of this design proposed a long-lived gateway root span for the sandbox lifecycle, with agent spans as children via `TRACEPARENT`. This was rejected because sandbox lifetimes are unpredictable (seconds to days) and long-lived parent spans are an OTEL antipattern: they delay export, strain backends, hold gateway memory, and misrepresent the semantic structure. Span links provide correlation without duration coupling. +An earlier version of this design proposed a long-lived gateway root span for the sandbox lifecycle, with agent spans as children via `TRACEPARENT`. This was rejected because sandbox lifetimes are unpredictable (seconds to days) and long-lived parent spans are an OTel antipattern: they delay export, strain backends, hold gateway memory, and misrepresent the semantic structure. Span links provide correlation without duration coupling. ### Direct OTLP export from the sandbox @@ -307,11 +343,11 @@ An earlier version of this design proposed a long-lived gateway root span for th ### Do nothing -Leave the current design unchanged. Platform operators cannot debug latency across the stack. Agent developers cannot get traces into MLflow without building workarounds. The three observability pillars remain disconnected. This is not acceptable for production deployments. +Leaving the current state unchanged preserves all the gaps described in the Motivation section. This is not acceptable for production deployments. ## Prior art -**[Dapr](https://dapr.io) sidecar architecture and OTEL integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. +**[Dapr](https://dapr.io) sidecar architecture and OTel integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define the emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. @@ -319,8 +355,7 @@ Red Hat's CI pipelines for running agents in OpenShell sandboxes independently d ## Open questions -- How should `OTEL_*` environment variables and TOML gateway configuration interact when both are set? [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and RFC 0003 discuss this but no decision has been made on precedence. -- Multi-tenant deployments likely need per-workspace OTLP endpoint configuration so that different tenants' agent traces can route to different collectors or MLflow instances. The configuration model for per-workspace endpoints is not designed yet. +- How should `OTEL_*` environment variables and TOML gateway configuration interact when both are set? [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and [RFC 0003](../0003-gateway-configuration/README.md) discuss this but no decision has been made on precedence. - Does MLflow's OTLP ingestion surface span links in its UI? If it does not, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. This needs validation. - Should the OTel SDK dependency be gated behind a Cargo feature flag so that builds without tracing support do not pay the binary size and compile time cost? - Should the supervisor implement the OTLP receiver from scratch using the `opentelemetry-proto` crate (smaller binary, more control), or embed a lightweight collector library (more features, larger dependency)? The tradeoffs need evaluation during implementation. From 0bbe74e2b5e1bc444beef9aa72551227a0dea2ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 18:44:18 +0200 Subject: [PATCH 06/22] docs(rfc): extend relay to all three observability pillars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reframe the supervisor relay as a general telemetry channel carrying traces, logs, and metrics out of isolated sandboxes. Add log relay section (OCSF events + agent stdout/stderr via session protocol, solving #1922) and metrics relay section (push-based sandbox metrics, lower priority). Update Mermaid diagram to show all three pillars. Add Summary paragraph explaining why tracing gets architectural depth while logging and metrics extend through the same channel. Add Motivation bullet for centralized log collection gap. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 46 +++++++++++++++---- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 4bf1afe9f2..de6846c986 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -17,7 +17,9 @@ This RFC proposes a unified observability architecture for OpenShell covering tr On the infrastructure side, contributions have started building coverage. OpenTelemetry Protocol (OTLP) trace export merged for the gateway and VM driver, OCSF structured logging covers security events, and Prometheus metrics provide basic request counting. This RFC extends that to the remaining components (in-process compute drivers, CLI, sandbox supervisor) and adds deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTel correlation, and a metrics maturity path. -On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector. This RFC proposes a supervisor-local OTLP relay as the solution. The supervisor listens on standard OTLP ports inside the sandbox, collects agent-emitted traces, enriches them with sandbox context, and forwards them through the gateway to an external collector. The agent just exports to a well-known local endpoint; the platform handles the rest. +On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector, OCSF security events stay local with no centralized collection path ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)), and Prometheus cannot scrape metrics from inside isolated sandboxes. This RFC proposes a supervisor-local telemetry relay as the solution. The supervisor acts as a sidecar that mediates all observability data between the isolated sandbox and the platform. For traces, it listens on standard OTLP ports, collects agent-emitted spans, enriches them with sandbox context, and forwards them through the gateway to an external collector. The same relay channel carries OCSF log batches for centralized collection and can push sandbox-level metrics to the gateway. + +Tracing receives the most architectural attention in this RFC because it requires the most novel design (the relay, span links, enrichment pipeline). Logging and metrics build on existing foundations (OCSF is already designed, Prometheus is already wired) and extend through the same relay channel without needing separate architecture. ## Motivation @@ -28,6 +30,7 @@ OpenShell's observability story is still early. That is expected for a project a - Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTel integration yet. - No connection between OCSF security events and traces. An operator cannot pivot from a network deny event to the trace that caused it. - The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, Service Level Indicator/Service Level Objective (SLI/SLO) definitions) is mostly unimplemented beyond the initial wiring. No ServiceMonitor, dashboards, or alerting rules exist. +- No centralized collection for sandbox logs. OCSF security events are generated inside the sandbox but stay local. Agent stdout/stderr is captured by the process supervisor but has no path to a log aggregator. [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks this and is currently stale. - No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI running inside sandboxes). The sandbox is network-isolated, so an agent's OTel SDK cannot reach an external collector. [#1055](https://github.com/NVIDIA/OpenShell/issues/1055) tracks the overall enterprise observability effort. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. @@ -56,7 +59,7 @@ The personas proposed in this RFC are a reasoning tool for the observability dis ## Proposal -The proposal is organized around the current state, the novel OTLP relay for agent traces (the architecturally new piece), the supporting mechanisms (span links, enrichment, OCSF correlation), infrastructure instrumentation for the remaining components, deployment configuration, multi-tenant considerations, and the metrics path. The two visibility domains (infrastructure vs. agent traces) cut across all of these. +The proposal is organized around the current state, the two visibility domains (infrastructure vs. agent), the supervisor telemetry relay (which carries traces, logs, and metrics out of isolated sandboxes), the supporting mechanisms (span links, enrichment, OCSF correlation), the log and metrics relay extensions, infrastructure instrumentation for the remaining components, deployment configuration, and multi-tenant considerations. ### Current state @@ -72,7 +75,7 @@ Tracing coverage today is limited to the gateway and VM driver. Both gained OTLP | CLI | None | None | Gap | | Agent traces from inside sandbox | None | N/A | Gap (new) | -On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. It runs only on the sandbox side. Centralized sandbox log collection remains an unsolved problem ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922), currently stale). +On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. It runs only on the sandbox side. Centralized sandbox log collection remains an unsolved problem (see Log relay section below). For monitoring, the Prometheus wiring is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. @@ -85,16 +88,21 @@ Two distinct visibility domains flow through the same OTLP relay pipeline: The platform integrator configures this routing at the collector level (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). OpenShell emits both to the same OTLP endpoint; the collector separates them. -### Supervisor as OTLP relay for sandbox traces +### Supervisor as telemetry relay + +The sandbox supervisor (hereafter "supervisor") acts as a telemetry sidecar for the isolated sandbox, mediating traces, logs, and metrics between the sandbox and the platform. The session protocol between supervisor and gateway provides the transport for all three pillars. -The sandbox supervisor (hereafter "supervisor") listens on standard OTLP ports inside the sandbox, accepting both gRPC (port 4317) and HTTP (port 4318). Agent frameworks export traces to this address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. +For traces, the supervisor listens on standard OTLP ports inside the sandbox, accepting both gRPC (port 4317) and HTTP (port 4318). Agent frameworks export traces to this address. The supervisor buffers, optionally enriches, and forwards spans to the gateway over the existing session protocol. The gateway relays them to the configured external OTLP endpoint. ```mermaid graph TD - A["Agent process
(inside sandbox)"] -->|"OTLP gRPC+HTTP
(ports 4317/4318)"| B["Supervisor
(OTLP receiver)"] - B -->|"Session protocol
(existing gRPC)"| C["Gateway
(OTLP relay)"] - C -->|"OTLP/gRPC
configured endpoint"| D["External Collector
(platform-managed)"] - D --> E["MLflow / Jaeger / Grafana Tempo"] + A["Agent process
(inside sandbox)"] -->|"OTLP gRPC+HTTP
(ports 4317/4318)"| B["Supervisor
(telemetry relay)"] + A -->|"stdout/stderr"| B + B -->|"Session protocol
(traces, logs, metrics)"| C["Gateway
(relay)"] + C -->|"OTLP/gRPC"| D["Trace Collector"] + C -->|"Log forwarding"| F["Log Aggregator"] + C -->|"Metrics"| G["/metrics endpoint"] + D --> E["MLflow / Jaeger / Tempo"] ``` The supervisor: @@ -174,6 +182,26 @@ When a trace context is active, OCSF events include `trace_id` and `span_id` fie The OCSF builders already accept arbitrary fields. Add optional `trace_id`/`span_id` fields to the builder pattern, populated from the current `tracing::Span` when available. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) sub-issue 5 ("OCSF correlation") tracks this but has not defined the mechanism. +### Log relay for centralized sandbox collection + +The same session protocol transport that carries trace batches can carry log data from the sandbox to the gateway, solving the centralized log collection problem ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)). + +The supervisor has access to two log sources inside the sandbox: OCSF JSONL events (security-relevant structured events, already generated by the OCSF subsystem) and agent process stdout/stderr (operational output captured by the process supervisor). Today both stay local. The log relay forwards them to the gateway, which exposes them to the platform's log aggregation infrastructure. + +Unlike the trace relay, the log relay does not need an OTLP receiver. OCSF events are generated internally by the supervisor, and agent stdout/stderr is already captured as part of process management. The relay adds a new session protocol message type for log batches, alongside the trace batch message from Phase 2. The gateway forwards received log batches to a configured log endpoint or makes them available for scraping. + +Log enrichment follows the same pattern as span enrichment: the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.workspace.id`, etc.) to log records before forwarding. This gives operators the context they need to filter and correlate logs across sandboxes and workspaces. + +The OCSF-to-OTel correlation described above complements this: when OCSF events carry `trace_id`/`span_id` fields, an operator can follow a log record into the trace that produced it, even after the logs have been forwarded to a separate aggregation backend. + +### Metrics relay for sandbox monitoring + +The sandbox is network-isolated, so Prometheus cannot scrape supervisor metrics directly. The supervisor can push sandbox-level metrics to the gateway via the session protocol, and the gateway can aggregate them into its own `/metrics` endpoint or forward them via OTLP metrics push. + +Candidate sandbox metrics include: agent process CPU and memory usage, OTLP relay buffer depth and drop counts, active network connections, policy evaluation latency, and middleware call duration. These would appear with sandbox resource labels so operators can break down by sandbox, workspace, or driver. + +This is lower priority than the trace and log relays. The gateway-level metrics from [#909](https://github.com/NVIDIA/OpenShell/issues/909) provide platform-wide visibility, and per-sandbox metrics add value mainly for capacity planning and debugging individual sandbox performance issues. The metrics relay can be deferred to a later phase without blocking the overall observability architecture. + ### Infrastructure trace instrumentation The K8s, Docker, and Podman compute drivers run in-process within the gateway and already inherit its tracing subscriber. What they lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver is the reference here, with 10+ instrumented operations using `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right starting point. From 4f9de1e12a1200686947880a8f36aac6d4dda728 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:07:59 +0200 Subject: [PATCH 07/22] docs(rfc): add separation of concerns argument for relay design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The relay decouples agent development from observability operations: agent developers export to a fixed auto-injected endpoint, workspace admins configure where telemetry routes, global admins set defaults and policies. Reference this from the Alternatives section as a fifth reason to prefer the relay over direct collector access. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index de6846c986..c194ae05bd 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -121,6 +121,8 @@ The gateway: For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTel-instrumented framework works without OpenShell-specific code. +The relay also creates a clean separation of concerns between three roles. The agent developer exports to a fixed, auto-injected endpoint and never thinks about collector topology, authentication, or routing. The same agent code works in every sandbox, every deployment, every workspace. The workspace administrator decides where their workspace's telemetry goes (which collector, which MLflow instance, what sampling rates) without touching agent configuration. The global administrator configures the default OTLP endpoint and platform-wide policies like rate limits and enrichment. With direct collector access, the agent developer would need to know the collector address, and that address varies by deployment and workspace. The relay makes observability routing an operational concern rather than a development one. + ### OTLP receiver reachability per driver All container-based drivers share a networking subtlety that affects how the OTLP receiver is reached. The supervisor creates an internal workload network namespace for the agent process, connected to the supervisor's namespace via a veth pair. The proxy already binds to the host-side veth IP (available from `netns.host_ip()`) to intercept agent traffic. This means `localhost` inside the agent's network namespace is not the same `localhost` the supervisor sees. @@ -358,6 +360,7 @@ The supervisor relay is preferred as the default for four reasons: - Multi-tenancy: one collector per gateway host does not scale to multi-tenant deployments where workspaces need separate OTLP endpoints. - Reliability: the SSRF engine blocks `host.openshell.internal` on newer gateway versions ([#2478](https://github.com/NVIDIA/OpenShell/issues/2478)), TCP connections hang on the K8s driver, and it requires per-sandbox policy configuration. - Enrichment: the relay enables the supervisor to attach sandbox context as resource attributes, which is unavailable on the direct path. +- Separation of concerns: the relay decouples agent development from observability operations. With direct access, agents need deployment-specific collector addresses. The relay gives every agent a fixed local endpoint regardless of where traces end up. These approaches are not mutually exclusive. The design follows the same pattern as `inference.local`: the relay is the zero-config default, and users can override `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or any other endpoint. From 206aa913cbc0057d01cd6c72932245bfb05980e9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:18:01 +0200 Subject: [PATCH 08/22] docs(rfc): fix technical accuracy issues from codebase verification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix OCSF scope claim: OCSF runs on both sandbox and gateway sides (TLS events, service routing, policy), not sandbox-only. Correct VM driver instrumentation count from "10+ using ErrorStatusGuard" to "15 instrumented operations (9 via instrument, 6 via .instrument()) and ErrorStatusGuard for error marking". Note that K8s and Podman drivers are dual-mode (in-process default + standalone gRPC binary). Clarify #2615 is a PR, not an issue. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index c194ae05bd..044dd7ead5 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -37,7 +37,7 @@ OpenShell's observability story is still early. That is expected for a project a ### Personas -The personas below are proposed, not settled. OpenShell's persona model is evolving (see [#2615](https://github.com/NVIDIA/OpenShell/pull/2615) for ongoing work on centering issue reports around user stories). This RFC uses five personas as a lens for reasoning about observability needs, but the final definitions should be driven by the broader project discussion. +The personas below are proposed, not settled. OpenShell's persona model is evolving (see PR [#2615](https://github.com/NVIDIA/OpenShell/pull/2615) for ongoing work on centering issue reports around user stories). This RFC uses five personas as a lens for reasoning about observability needs, but the final definitions should be driven by the broader project discussion. | Persona | Role | Observability needs | |---|---|---| @@ -75,7 +75,7 @@ Tracing coverage today is limited to the gateway and VM driver. Both gained OTLP | CLI | None | None | Gap | | Agent traces from inside sandbox | None | N/A | Gap (new) | -On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. It runs only on the sandbox side. Centralized sandbox log collection remains an unsolved problem (see Log relay section below). +On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. OCSF runs on both the sandbox side (network decisions, process lifecycle, security findings) and the gateway side (TLS events, service routing, policy operations). Centralized sandbox log collection remains an unsolved problem (see Log relay section below). For monitoring, the Prometheus wiring is in place: a `metrics` crate facade, a dedicated `/metrics` endpoint, and Helm chart port 9090 exposed. Basic gRPC/HTTP RED metrics and gateway interceptor metrics are implemented and working. The broader catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909), which defines 16 metric families across three priority tiers along with SLI/SLO definitions, is mostly unimplemented. The sandbox supervisor has no metrics at all, and there are no ServiceMonitor CRDs, dashboards, or alerting rules. @@ -206,7 +206,7 @@ This is lower priority than the trace and log relays. The gateway-level metrics ### Infrastructure trace instrumentation -The K8s, Docker, and Podman compute drivers run in-process within the gateway and already inherit its tracing subscriber. What they lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver is the reference here, with 10+ instrumented operations using `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right starting point. +The K8s, Docker, and Podman compute drivers run in-process within the gateway by default and inherit its tracing subscriber. K8s and Podman also have standalone gRPC binary entry points for external deployment, but the in-process path is the current default. What these drivers lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver is the reference here, with 15 instrumented operations (9 via `#[tracing::instrument]`, 6 via `.instrument()`) and `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right starting point. The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. Adding optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wiring `TraceContextInterceptor` onto the gRPC channel, and creating a root span per CLI command would close this gap. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. @@ -267,7 +267,7 @@ The implementation is phased to manage dependencies. Phase 1 extends existing tr This phase extends gateway and VM driver tracing to the remaining infrastructure components. None of it requires new transport mechanisms or the OTLP relay. It builds on what exists and unblocks end-to-end tracing from the CLI through the gateway to any compute driver. -The K8s, Docker, and Podman drivers are the easiest starting point because they run inside the gateway process and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to operations like provisioning, teardown, status checks, and exec, following the VM driver as a reference. Because these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. Instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`, and the resulting spans appear as children of the gateway's request span automatically. +The K8s, Docker, and Podman drivers are the easiest starting point because they run in-process within the gateway by default and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to operations like provisioning, teardown, status checks, and exec, following the VM driver as a reference. Because these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. Instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`, and the resulting spans appear as children of the gateway's request span automatically. The CLI is a different problem. The gap and building blocks are described in the Infrastructure trace instrumentation section. Wiring them together means adding a `--otlp-endpoint` flag (or respecting `OTEL_EXPORTER_OTLP_ENDPOINT`), building a `SdkTracerProvider` using the shared `openshell-otel` crate, creating a root span per CLI command, and attaching the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. From 8b00e8774657846674ba28d5df14268052691132 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:23:55 +0200 Subject: [PATCH 09/22] docs(rfc): descope CLI from full OTel instrumentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CLI runs on laptops, CI runners, and jump hosts where an OTel collector is typically unreachable. The gateway already traces every incoming request. Replace the full CLI OTel provider proposal with passive TRACEPARENT propagation: if the environment has TRACEPARENT set (e.g. CI pipelines), forward it to the gateway. No SdkTracerProvider, no collector configuration, no --otlp-endpoint flag needed. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 044dd7ead5..efe771dc4b 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -72,7 +72,7 @@ Tracing coverage today is limited to the gateway and VM driver. Both gained OTLP | `openshell-otel` crate | Shared provider, layer, error helpers | `TraceContextInterceptor` | Merged ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)) | | In-process drivers (K8s embedded/sidecar, Docker, Podman) | None | N/A (in-process) | Gap | | Sandbox supervisor | None | None | Gap | -| CLI | None | None | Gap | +| CLI | None (not needed, see below) | Passive `TRACEPARENT` forwarding proposed | Lightweight | | Agent traces from inside sandbox | None | N/A | Gap (new) | On the logging side, OpenShell has two systems serving distinct purposes. Human-readable operational logs go to stdout via `tracing_subscriber::fmt`. Security-relevant events are captured by the OCSF structured logging system (`openshell-ocsf`), which emits both a shorthand format (always on) and full JSONL records (opt-in). OCSF covers network decisions, HTTP/L7 enforcement, SSH authentication, process lifecycle, security findings, configuration changes, and application lifecycle. OCSF runs on both the sandbox side (network decisions, process lifecycle, security findings) and the gateway side (TLS events, service routing, policy operations). Centralized sandbox log collection remains an unsolved problem (see Log relay section below). @@ -208,7 +208,9 @@ This is lower priority than the trace and log relays. The gateway-level metrics The K8s, Docker, and Podman compute drivers run in-process within the gateway by default and inherit its tracing subscriber. K8s and Podman also have standalone gRPC binary entry points for external deployment, but the in-process path is the current default. What these drivers lack are explicit `#[tracing::instrument]` annotations that would create spans for driver operations like provisioning, teardown, status checks, and exec. The VM driver is the reference here, with 15 instrumented operations (9 via `#[tracing::instrument]`, 6 via `.instrument()`) and `ErrorStatusGuard` for error marking. [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) notes these drivers are expected to migrate to external gRPC services eventually, at which point they will need their own `SdkTracerProvider` and W3C propagation, but the in-process annotations are still the right starting point. -The CLI currently does not inject `traceparent` into its gRPC calls to the gateway, which means CLI-initiated operations start new traces at the gateway rather than continuing a trace from the user's command. Adding optional OTLP export (via `--otlp-endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`), wiring `TraceContextInterceptor` onto the gRPC channel, and creating a root span per CLI command would close this gap. The gateway already knows how to extract incoming `traceparent`, so the connection happens automatically. +The CLI does not need its own OTel provider or OTLP export. The CLI runs on developer laptops, CI runners, and jump hosts where an OTel collector is typically not reachable, and the gateway already traces every incoming request. The gateway is the instrumentation boundary for platform tracing. + +The one useful thing the CLI can do is passive `traceparent` propagation: if the CLI is invoked from a context that already has `TRACEPARENT` set (e.g., a CI pipeline with its own tracing), it should forward that context to the gateway via `TraceContextInterceptor` on the gRPC channel. This is a lightweight addition (no `SdkTracerProvider`, no collector configuration) that gives CI pipelines end-to-end trace continuity without imposing operational overhead on every CLI installation. The sandbox supervisor has zero OTel integration today, the largest gap in the stack. [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) scopes this into network spans (L4 connect, Open Policy Agent (OPA) evaluation, L7 enforcement, credential injection, middleware calls), middleware spans (operator-run gRPC services with real cross-service tracing value), and process spans (entrypoint lifecycle, exec, SSH sessions). All of these route supervisor -> gateway -> collector per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508)'s settled design decision. @@ -269,7 +271,7 @@ This phase extends gateway and VM driver tracing to the remaining infrastructure The K8s, Docker, and Podman drivers are the easiest starting point because they run in-process within the gateway by default and already inherit its tracing subscriber. The work is adding `#[tracing::instrument]` annotations to operations like provisioning, teardown, status checks, and exec, following the VM driver as a reference. Because these drivers are in-process, they share the gateway's OTLP exporter and need no separate provider setup or W3C propagation. Instrument the `ComputeDriver` trait implementations in `crates/openshell-driver-kubernetes/`, `crates/openshell-driver-docker/`, and `crates/openshell-driver-podman/`, and the resulting spans appear as children of the gateway's request span automatically. -The CLI is a different problem. The gap and building blocks are described in the Infrastructure trace instrumentation section. Wiring them together means adding a `--otlp-endpoint` flag (or respecting `OTEL_EXPORTER_OTLP_ENDPOINT`), building a `SdkTracerProvider` using the shared `openshell-otel` crate, creating a root span per CLI command, and attaching the interceptor to the CLI's gRPC channel. Once connected, a trace initiated by `openshell sandbox create` flows from the CLI through the gateway into whatever driver handles the request. +The CLI needs only passive `traceparent` propagation, not its own OTel provider. If `TRACEPARENT` is set in the environment (e.g., by a CI pipeline), the CLI forwards it to the gateway via `TraceContextInterceptor` on the gRPC channel. This is lightweight (no `SdkTracerProvider`, no collector connectivity required) and gives CI pipelines end-to-end trace continuity. The gateway remains the instrumentation boundary for platform tracing. On the deployment side, the Helm and TOML configuration described in the Deployment configuration section needs to land. The integration surface is `deploy/helm/openshell/templates/` where the ConfigMap is generated and the StatefulSet picks up environment variables. Docker Compose's example `gateway.toml` should include a commented-out OTLP section, and the published docs at `docs/reference/gateway-config.mdx` need to cover the OTLP configuration. From b378447a384924829450f3f2d3d1090e9b07c262 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:36:13 +0200 Subject: [PATCH 10/22] docs(rfc): add per-domain OTLP endpoints as future extension MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Infrastructure and agent traces serve different user groups with different backend needs (Tempo vs MLflow). Initially both go to the same endpoint with collector-side routing. As a future extension, per-domain endpoints (infra_endpoint, agent_endpoint) would let the gateway route directly. This composes with per-workspace endpoints for multi-tenant deployments. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index efe771dc4b..f3a5f838fd 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -86,7 +86,9 @@ Two distinct visibility domains flow through the same OTLP relay pipeline: 1. **Infrastructure traces** (operator-only): gateway request spans, driver lifecycle, supervisor network/process/middleware spans. Locked as operator-only per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508). 2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow) from infrastructure traces (e.g., Tempo). -The platform integrator configures this routing at the collector level (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). OpenShell emits both to the same OTLP endpoint; the collector separates them. +These two domains serve different user groups with different backend needs. Infrastructure traces typically go to the platform operator's monitoring stack (Tempo, Jaeger) for SLI/SLO dashboards and latency debugging. Agent traces go to the agent developer's tools (MLflow) for inspecting tool calls and reasoning steps. These may be entirely different systems, managed by different teams, with different retention and sampling policies. + +Initially, both domains go to the same configured OTLP endpoint, and the platform integrator separates them at the collector level using resource attributes (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). As a future extension, the gateway could support per-domain OTLP endpoints (`infra_endpoint` and `agent_endpoint` alongside the default `endpoint`) so it can route infrastructure and agent traces to different backends directly, without requiring collector-side routing rules. This would also compose with per-workspace endpoints (see Multi-tenant observability below), giving each workspace independent routing for each visibility domain. ### Supervisor as telemetry relay @@ -224,6 +226,9 @@ The gateway's `gateway.toml` needs an OTLP section: [openshell.gateway.otlp] endpoint = "http://otel-collector.monitoring:4317" service_name = "openshell-gateway" +# Future: per-domain endpoints (initially, both go to `endpoint`) +# infra_endpoint = "http://tempo:4317" +# agent_endpoint = "http://mlflow:4317" ``` The Helm chart needs corresponding values that template into the ConfigMap: @@ -247,7 +252,7 @@ The OTLP relay supports this through per-workspace OTLP endpoint configuration a Per-workspace sampling and rate limits follow the same pattern as per-sandbox limits described in the Risks section, but scoped to the workspace level. A noisy workspace should not exhaust the gateway's relay capacity for other tenants. -The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before Phase 2 implementation. +The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before Phase 2 implementation. Per-workspace configuration would compose with the per-domain endpoint extension described in Visibility domains: a workspace could route its infrastructure traces to a shared platform Tempo while sending its agent traces to a workspace-specific MLflow instance. ### Metrics direction From f9ef87201ac351fafe7f06e330bdc9170e39b47c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:42:41 +0200 Subject: [PATCH 11/22] docs(rfc): update relay scale analysis for all three pillars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expand the capacity calculation to cover traces, OCSF logs, agent stdout/stderr, and metrics. OCSF logs may be the dominant stream (10-300 events/sec per sandbox at 1-5 KB each). Add per-pillar volume table. Note that log sampling has different constraints than trace sampling (OCSF events serve audit/compliance, so rate limiting is preferred over probabilistic sampling). Add stdout/stderr byte rate limit as a mitigation. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 28 ++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index f3a5f838fd..4b9d4c0d94 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -336,18 +336,32 @@ The mitigation is to normalize high-cardinality attributes before they reach the ### Relay scale and gateway bottleneck -The supervisor-to-gateway session protocol was designed for control plane traffic: sandbox lifecycle events, configuration updates, log pushes. Adding trace relay changes the traffic profile substantially. An active agent with OTel instrumentation can produce 10-100 spans/second. With 100 concurrent sandboxes on one gateway, that adds up to 1,000-10,000 spans/second, roughly 1-10 MB/second sustained through the gateway. +The supervisor-to-gateway session protocol was designed for control plane traffic: sandbox lifecycle events, configuration updates, log pushes. Using it as a telemetry relay for traces, logs, and metrics changes the traffic profile substantially. -The network bandwidth alone is manageable. The deeper concern is that the gateway now serves two roles (control plane and data plane relay), and these roles compete for resources. If the external collector is slow or unreachable, the gateway accumulates buffered spans from all sandboxes. Memory pressure from the relay can degrade sandbox lifecycle operations. +Estimated per-sandbox volume by pillar: + +| Pillar | Per-sandbox rate | Per-record size | Per-sandbox bandwidth | +|---|---|---|---| +| Traces | 10-100 spans/sec (active agent with OTel) | 200-500 bytes | 2-50 KB/sec | +| OCSF logs | 10-300 events/sec (one per outbound network decision) | 1-5 KB (full JSONL event) | 10 KB - 1.5 MB/sec | +| Agent stdout/stderr | Varies (verbose LLM agent: 1-10 KB/sec) | Raw text | 1-10 KB/sec | +| Metrics | 1 push per 15-60 sec | 1-5 KB | Negligible | + +At 100 concurrent sandboxes, the aggregate could reach 1-15 MB/second sustained. OCSF logs are potentially the largest contributor because every outbound agent connection generates structured security events, and OCSF JSONL records are larger than trace spans. Agent stdout/stderr volume depends entirely on agent verbosity and is unbounded without rate limiting. + +The network bandwidth alone is manageable. The deeper concern is that the gateway now serves two roles (control plane and data plane relay), and these roles compete for resources. If the external collector or log aggregator is slow or unreachable, the gateway accumulates buffered data from all sandboxes. Memory pressure from the relay can degrade sandbox lifecycle operations. + +Log sampling has different constraints than trace sampling. Trace head sampling (forward 10% of traces) is a standard practice. But OCSF security events serve compliance and audit purposes, so dropping them silently is not acceptable. Log relay needs separate controls: rate limiting (cap events/second per sandbox) rather than probabilistic sampling, with dropped event counts tracked as a metric. Several mitigations should be part of the implementation: -- Trace relay should use a separate transport channel (or multiplexed stream) from the control plane session, so trace backpressure does not block sandbox create or delete operations. -- Head sampling at the supervisor reduces volume before it reaches the gateway. This should be configurable per-sandbox or globally (e.g., sample 10% of traces, but always forward traces containing errors). -- Per-sandbox span rate limits prevent one noisy agent from overwhelming the gateway. Excess spans are dropped at the supervisor with a counter metric, not silently. -- The gateway forwards received spans to the external collector asynchronously in batches, not synchronously per-span. +- The telemetry relay should use a separate transport channel (or multiplexed stream) from the control plane session, so telemetry backpressure does not block sandbox create or delete operations. +- Head sampling at the supervisor reduces trace volume before it reaches the gateway. This should be configurable per-sandbox or globally (e.g., sample 10% of traces, but always forward traces containing errors). +- Per-sandbox rate limits prevent one noisy agent from overwhelming the gateway. Excess spans and log events are dropped at the supervisor with counter metrics, not silently. OCSF events should have higher rate limits than traces given their audit value. +- Agent stdout/stderr needs a per-sandbox byte rate limit. Without one, a verbose agent can dominate the relay channel. +- The gateway forwards received telemetry to external endpoints asynchronously in batches, not synchronously per-record. - When the gateway's relay buffer is full, it should signal the supervisor to slow down or drop rather than silently accepting until it runs out of memory. -- For high-volume scenarios, users can bypass the relay entirely by setting `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or a direct collector address. The relay is the zero-config default, not a mandatory path. +- For high-volume scenarios, users can bypass the trace relay entirely by setting `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or a direct collector address. The trace relay is the zero-config default, not a mandatory path. ### Other considerations From cead2ec4e0e6024abee90114644e84143965b4da Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:44:38 +0200 Subject: [PATCH 12/22] docs(rfc): acknowledge existing PushSandboxLogs RPC in log relay section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The supervisor already pushes tracing events to the gateway via LogPushLayer and the PushSandboxLogs client-streaming RPC. The log relay section now builds on this existing mechanism rather than proposing to build from scratch. The two remaining gaps are OCSF JSONL events (which use a separate layer and stay local) and agent stdout/stderr (captured but not pushed). Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 4b9d4c0d94..ef72b4c614 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -188,13 +188,15 @@ The OCSF builders already accept arbitrary fields. Add optional `trace_id`/`span ### Log relay for centralized sandbox collection -The same session protocol transport that carries trace batches can carry log data from the sandbox to the gateway, solving the centralized log collection problem ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)). +A log push mechanism already exists. The `LogPushLayer` in `crates/openshell-supervisor-process/src/log_push.rs` captures tracing events from the supervisor and streams them to the gateway via the `PushSandboxLogs` client-streaming RPC. Each `SandboxLogLine` carries the sandbox ID, timestamp, level, target, message, and structured key-value fields. The push is best-effort (`try_send`; if the channel is full, the event is dropped). The gateway stores these logs and exposes them through `GetSandboxLogs`. -The supervisor has access to two log sources inside the sandbox: OCSF JSONL events (security-relevant structured events, already generated by the OCSF subsystem) and agent process stdout/stderr (operational output captured by the process supervisor). Today both stay local. The log relay forwards them to the gateway, which exposes them to the platform's log aggregation infrastructure. +This existing mechanism covers supervisor-level tracing events but has two gaps that [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks: -Unlike the trace relay, the log relay does not need an OTLP receiver. OCSF events are generated internally by the supervisor, and agent stdout/stderr is already captured as part of process management. The relay adds a new session protocol message type for log batches, alongside the trace batch message from Phase 2. The gateway forwards received log batches to a configured log endpoint or makes them available for scraping. +1. **OCSF JSONL events** are not pushed through `PushSandboxLogs`. OCSF events go through a separate JSONL layer and stay local to the sandbox. These are the security-relevant structured events (network decisions, L7 enforcement, SSH authentication) that operators and compliance reviewers need centralized access to. Extending the log push to include OCSF events means either adding OCSF records to the existing `SandboxLogLine` format (with a source field to distinguish them) or adding a dedicated OCSF push alongside the existing log push. -Log enrichment follows the same pattern as span enrichment: the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.workspace.id`, etc.) to log records before forwarding. This gives operators the context they need to filter and correlate logs across sandboxes and workspaces. +2. **Agent stdout/stderr** is captured by the process supervisor but not pushed to the gateway. For debugging agent behavior, operators need access to agent output without SSH-ing into the sandbox. + +Log enrichment follows the same pattern as span enrichment: the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.workspace.id`, etc.) to log records before forwarding. The existing `SandboxLogLine.fields` map already supports this. The OCSF-to-OTel correlation described above complements this: when OCSF events carry `trace_id`/`span_id` fields, an operator can follow a log record into the trace that produced it, even after the logs have been forwarded to a separate aggregation backend. From 6078418121fdf5cee7b8d5550fee9a4d96c725cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:49:43 +0200 Subject: [PATCH 13/22] docs(rfc): remove agent stdout/stderr from log relay scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Agent sandboxes are interactive sessions with terminal UIs, ANSI escape codes, and multiplexed SSH channels. That output stream is user-facing interaction, not structured log data. Internal agent logging is the agent developer's concern, not the platform's. The log relay scope is now: existing PushSandboxLogs tracing events (already working) plus OCSF event forwarding (the actual gap). Updated capacity table, diagram, and mitigations accordingly. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index ef72b4c614..202eba7674 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -17,7 +17,7 @@ This RFC proposes a unified observability architecture for OpenShell covering tr On the infrastructure side, contributions have started building coverage. OpenTelemetry Protocol (OTLP) trace export merged for the gateway and VM driver, OCSF structured logging covers security events, and Prometheus metrics provide basic request counting. This RFC extends that to the remaining components (in-process compute drivers, CLI, sandbox supervisor) and adds deployment configuration (Helm values, ServiceMonitor), OCSF-to-OTel correlation, and a metrics maturity path. -On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector, OCSF security events stay local with no centralized collection path ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)), and Prometheus cannot scrape metrics from inside isolated sandboxes. This RFC proposes a supervisor-local telemetry relay as the solution. The supervisor acts as a sidecar that mediates all observability data between the isolated sandbox and the platform. For traces, it listens on standard OTLP ports, collects agent-emitted spans, enriches them with sandbox context, and forwards them through the gateway to an external collector. The same relay channel carries OCSF log batches for centralized collection and can push sandbox-level metrics to the gateway. +On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector, OCSF security events stay local with no centralized collection path ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)), and Prometheus cannot scrape metrics from inside isolated sandboxes. This RFC proposes a supervisor-local telemetry relay as the solution. The supervisor acts as a sidecar that mediates all observability data between the isolated sandbox and the platform. For traces, it listens on standard OTLP ports, collects agent-emitted spans, enriches them with sandbox context, and forwards them through the gateway to an external collector. The existing log push mechanism extends to carry OCSF event batches for centralized security log collection, and the relay can push sandbox-level metrics to the gateway. Tracing receives the most architectural attention in this RFC because it requires the most novel design (the relay, span links, enrichment pipeline). Logging and metrics build on existing foundations (OCSF is already designed, Prometheus is already wired) and extend through the same relay channel without needing separate architecture. @@ -30,7 +30,7 @@ OpenShell's observability story is still early. That is expected for a project a - Tracing coverage beyond the gateway and VM driver. The sandbox supervisor, in-process compute drivers (K8s, Docker, Podman), CLI, and Helm chart have no OTel integration yet. - No connection between OCSF security events and traces. An operator cannot pivot from a network deny event to the trace that caused it. - The metrics catalog proposed in [#909](https://github.com/NVIDIA/OpenShell/issues/909) (16 families, 3 priority tiers, Service Level Indicator/Service Level Objective (SLI/SLO) definitions) is mostly unimplemented beyond the initial wiring. No ServiceMonitor, dashboards, or alerting rules exist. -- No centralized collection for sandbox logs. OCSF security events are generated inside the sandbox but stay local. Agent stdout/stderr is captured by the process supervisor but has no path to a log aggregator. [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks this and is currently stale. +- No centralized collection for OCSF security events. OCSF events are generated inside the sandbox but stay local with no path to a log aggregator. [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks this and is currently stale. - No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI running inside sandboxes). The sandbox is network-isolated, so an agent's OTel SDK cannot reach an external collector. [#1055](https://github.com/NVIDIA/OpenShell/issues/1055) tracks the overall enterprise observability effort. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. @@ -99,8 +99,7 @@ For traces, the supervisor listens on standard OTLP ports inside the sandbox, ac ```mermaid graph TD A["Agent process
(inside sandbox)"] -->|"OTLP gRPC+HTTP
(ports 4317/4318)"| B["Supervisor
(telemetry relay)"] - A -->|"stdout/stderr"| B - B -->|"Session protocol
(traces, logs, metrics)"| C["Gateway
(relay)"] + B -->|"Session protocol
(traces, OCSF logs, metrics)"| C["Gateway
(relay)"] C -->|"OTLP/gRPC"| D["Trace Collector"] C -->|"Log forwarding"| F["Log Aggregator"] C -->|"Metrics"| G["/metrics endpoint"] @@ -192,9 +191,9 @@ A log push mechanism already exists. The `LogPushLayer` in `crates/openshell-sup This existing mechanism covers supervisor-level tracing events but has two gaps that [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks: -1. **OCSF JSONL events** are not pushed through `PushSandboxLogs`. OCSF events go through a separate JSONL layer and stay local to the sandbox. These are the security-relevant structured events (network decisions, L7 enforcement, SSH authentication) that operators and compliance reviewers need centralized access to. Extending the log push to include OCSF events means either adding OCSF records to the existing `SandboxLogLine` format (with a source field to distinguish them) or adding a dedicated OCSF push alongside the existing log push. +**OCSF JSONL events** are not pushed through `PushSandboxLogs`. OCSF events go through a separate JSONL layer and stay local to the sandbox. These are the security-relevant structured events (network decisions, L7 enforcement, SSH authentication) that operators and compliance reviewers need centralized access to. Extending the log push to include OCSF events means either adding OCSF records to the existing `SandboxLogLine` format (with a source field to distinguish them) or adding a dedicated OCSF push alongside the existing log push. -2. **Agent stdout/stderr** is captured by the process supervisor but not pushed to the gateway. For debugging agent behavior, operators need access to agent output without SSH-ing into the sandbox. +Agent stdout/stderr is explicitly out of scope for the log relay. Agent sandboxes are interactive sessions with terminal UIs, ANSI escape codes, and multiplexed SSH channels. That output stream is user-facing interaction, not structured log data. If an agent developer wants internal operational logs exported, that is their framework's concern (e.g., configuring Python logging or their agent SDK's log output), not something the platform should capture. Log enrichment follows the same pattern as span enrichment: the supervisor attaches sandbox resource attributes (`openshell.sandbox.id`, `openshell.workspace.id`, etc.) to log records before forwarding. The existing `SandboxLogLine.fields` map already supports this. @@ -346,10 +345,10 @@ Estimated per-sandbox volume by pillar: |---|---|---|---| | Traces | 10-100 spans/sec (active agent with OTel) | 200-500 bytes | 2-50 KB/sec | | OCSF logs | 10-300 events/sec (one per outbound network decision) | 1-5 KB (full JSONL event) | 10 KB - 1.5 MB/sec | -| Agent stdout/stderr | Varies (verbose LLM agent: 1-10 KB/sec) | Raw text | 1-10 KB/sec | +| Supervisor tracing events | Varies (existing LogPushLayer) | 100-500 bytes | 1-10 KB/sec | | Metrics | 1 push per 15-60 sec | 1-5 KB | Negligible | -At 100 concurrent sandboxes, the aggregate could reach 1-15 MB/second sustained. OCSF logs are potentially the largest contributor because every outbound agent connection generates structured security events, and OCSF JSONL records are larger than trace spans. Agent stdout/stderr volume depends entirely on agent verbosity and is unbounded without rate limiting. +At 100 concurrent sandboxes, the aggregate could reach 1-10 MB/second sustained. OCSF logs are potentially the largest contributor because every outbound agent connection generates structured security events, and OCSF JSONL records are larger than trace spans. The network bandwidth alone is manageable. The deeper concern is that the gateway now serves two roles (control plane and data plane relay), and these roles compete for resources. If the external collector or log aggregator is slow or unreachable, the gateway accumulates buffered data from all sandboxes. Memory pressure from the relay can degrade sandbox lifecycle operations. @@ -360,7 +359,6 @@ Several mitigations should be part of the implementation: - The telemetry relay should use a separate transport channel (or multiplexed stream) from the control plane session, so telemetry backpressure does not block sandbox create or delete operations. - Head sampling at the supervisor reduces trace volume before it reaches the gateway. This should be configurable per-sandbox or globally (e.g., sample 10% of traces, but always forward traces containing errors). - Per-sandbox rate limits prevent one noisy agent from overwhelming the gateway. Excess spans and log events are dropped at the supervisor with counter metrics, not silently. OCSF events should have higher rate limits than traces given their audit value. -- Agent stdout/stderr needs a per-sandbox byte rate limit. Without one, a verbose agent can dominate the relay channel. - The gateway forwards received telemetry to external endpoints asynchronously in batches, not synchronously per-record. - When the gateway's relay buffer is full, it should signal the supervisor to slow down or drop rather than silently accepting until it runs out of memory. - For high-volume scenarios, users can bypass the trace relay entirely by setting `OTEL_EXPORTER_OTLP_ENDPOINT` to `host.openshell.internal:` or a direct collector address. The trace relay is the zero-config default, not a mandatory path. From dff90d7bb09c865c9279451ff1d55de328090e4c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:51:20 +0200 Subject: [PATCH 14/22] docs(rfc): correct metrics relay section to reflect current state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The supervisor has no metrics today (no metrics crate, no Prometheus exporter). Only the gateway exposes /metrics. Reframe the metrics relay as a future extension for when sandbox-level metrics are added, not as solving a current scraping limitation. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 202eba7674..36e339ad44 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -201,9 +201,7 @@ The OCSF-to-OTel correlation described above complements this: when OCSF events ### Metrics relay for sandbox monitoring -The sandbox is network-isolated, so Prometheus cannot scrape supervisor metrics directly. The supervisor can push sandbox-level metrics to the gateway via the session protocol, and the gateway can aggregate them into its own `/metrics` endpoint or forward them via OTLP metrics push. - -Candidate sandbox metrics include: agent process CPU and memory usage, OTLP relay buffer depth and drop counts, active network connections, policy evaluation latency, and middleware call duration. These would appear with sandbox resource labels so operators can break down by sandbox, workspace, or driver. +The supervisor has no metrics today (no `metrics` crate dependency, no Prometheus exporter). Only the gateway exposes a `/metrics` endpoint (port 9090). If sandbox-level metrics are added in the future (relay buffer depth, drop counts, active connections, policy evaluation latency, middleware call duration), the supervisor would need to push them to the gateway via the session protocol, since the sandbox is network-isolated and cannot expose its own scrape target. The gateway would aggregate per-sandbox metrics into its own `/metrics` endpoint with sandbox resource labels. This is lower priority than the trace and log relays. The gateway-level metrics from [#909](https://github.com/NVIDIA/OpenShell/issues/909) provide platform-wide visibility, and per-sandbox metrics add value mainly for capacity planning and debugging individual sandbox performance issues. The metrics relay can be deferred to a later phase without blocking the overall observability architecture. From be36515bc38f23b8f5bdd21720ec3d0df1566c8a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 19:56:00 +0200 Subject: [PATCH 15/22] docs(rfc): fix forward reference to Phase 2 in multi-tenancy section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace bare "Phase 2" reference with descriptive text and an anchor link to the Implementation plan section, since it appears before the Implementation plan in the document. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 36e339ad44..f30f907a1b 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -251,7 +251,7 @@ The OTLP relay supports this through per-workspace OTLP endpoint configuration a Per-workspace sampling and rate limits follow the same pattern as per-sandbox limits described in the Risks section, but scoped to the workspace level. A noisy workspace should not exhaust the gateway's relay capacity for other tenants. -The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before Phase 2 implementation. Per-workspace configuration would compose with the per-domain endpoint extension described in Visibility domains: a workspace could route its infrastructure traces to a shared platform Tempo while sending its agent traces to a workspace-specific MLflow instance. +The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before the OTLP relay ships (see [Phase 2: Sandbox OTLP relay](#phase-2-sandbox-otlp-relay) in the Implementation plan). Per-workspace configuration would compose with the per-domain endpoint extension described in Visibility domains: a workspace could route its infrastructure traces to a shared platform Tempo while sending its agent traces to a workspace-specific MLflow instance. ### Metrics direction From b86d781fadf0b00539d3afd3830936798ca6c1f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:08:35 +0200 Subject: [PATCH 16/22] docs(rfc): add anchor link to relay section from Phase 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index f30f907a1b..77a8db7cc3 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -281,7 +281,7 @@ On the deployment side, the Helm and TOML configuration described in the Deploym ### Phase 2: Sandbox OTLP relay -This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector. The relay design is described in the Proposal section; this section focuses on where the code changes land and what depends on what. +This is the architecturally new work. It introduces a trace collection path from inside network-isolated sandboxes to the external collector. The relay design is described in [Supervisor as telemetry relay](#supervisor-as-telemetry-relay); this section focuses on where the code changes land and what depends on what. The supervisor needs an OTLP listener on ports 4317 (gRPC) and 4318 (HTTP), bound to the address reachable from the agent process (the veth host IP for container-based drivers, localhost for K8s sidecar; see OTLP receiver reachability). The implementation choice is between using the `opentelemetry-proto` crate to implement the receiver from scratch (smaller binary, more control) or embedding a lightweight collector library (more features, larger dependency). Either way, the receiver accepts `ExportTraceServiceRequest` messages, validates them, and places them in a bounded in-memory buffer. The supervisor sets `OTEL_EXPORTER_OTLP_ENDPOINT` in the agent process environment automatically, pointing at the correct address. From 5d0c580d9a1285becada27d6ef4745e3a81482e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:10:16 +0200 Subject: [PATCH 17/22] docs(rfc): add span links explanation at first substantive use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Span links are an OTel concept that may not be familiar to all readers. Add a brief explanation at the first use in the correlation section and a clarifying parenthetical at the first casual mention in the summary. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 77a8db7cc3..3e8e4a6f3a 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -19,7 +19,7 @@ On the infrastructure side, contributions have started building coverage. OpenTe On the agent side, the problem is harder: sandboxes are network-isolated, so an agent's OpenTelemetry (OTel) SDK cannot reach an external collector, OCSF security events stay local with no centralized collection path ([#1922](https://github.com/NVIDIA/OpenShell/issues/1922)), and Prometheus cannot scrape metrics from inside isolated sandboxes. This RFC proposes a supervisor-local telemetry relay as the solution. The supervisor acts as a sidecar that mediates all observability data between the isolated sandbox and the platform. For traces, it listens on standard OTLP ports, collects agent-emitted spans, enriches them with sandbox context, and forwards them through the gateway to an external collector. The existing log push mechanism extends to carry OCSF event batches for centralized security log collection, and the relay can push sandbox-level metrics to the gateway. -Tracing receives the most architectural attention in this RFC because it requires the most novel design (the relay, span links, enrichment pipeline). Logging and metrics build on existing foundations (OCSF is already designed, Prometheus is already wired) and extend through the same relay channel without needing separate architecture. +Tracing receives the most architectural attention in this RFC because it requires the most novel design (the relay, span links for cross-trace correlation, enrichment pipeline). Logging and metrics build on existing foundations (OCSF is already designed, Prometheus is already wired) and extend through the same relay channel without needing separate architecture. ## Motivation @@ -148,7 +148,7 @@ This pattern follows [Dapr](https://dapr.io)'s [sidecar](https://docs.dapr.io/co ### Span links for sandbox-to-trace correlation -Agent traces from inside the sandbox are correlated with gateway infrastructure traces via [span links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links), not parent-child relationships. +Agent traces from inside the sandbox are correlated with gateway infrastructure traces via [span links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links), not parent-child relationships. Span links are an OTel concept that lets one span reference another as "related" without implying that one is a child of the other. Unlike parent-child relationships (where the child's duration is contained within the parent's), linked spans are independent and can belong to different traces entirely. ```mermaid graph BT From c804bf540c9025701e896ccb11987bf3821af0aa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:12:48 +0200 Subject: [PATCH 18/22] docs(rfc): attribute ServiceMonitor CRD to Prometheus Operator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ServiceMonitor is a Prometheus Operator CRD, not a Helm concept. The Helm chart includes a template for it. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 3e8e4a6f3a..f43b4d9c4c 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -260,7 +260,7 @@ Metrics remain Prometheus-first. The tracing architecture above is the primary f Priority order: 1. Complete P0 metrics from [#909](https://github.com/NVIDIA/OpenShell/issues/909) (supervisor sessions, sandbox phase, relay) -2. Add Helm ServiceMonitor (gated, off by default) +2. Add Prometheus `ServiceMonitor` CRD to the Helm chart (gated, off by default) 3. P1 metrics (SSH, compute driver, DB, policy) 4. Dashboards and alerting rules 5. OTLP metrics push (opt-in complement) @@ -313,7 +313,7 @@ This phase fills out the metrics catalog and adds the Kubernetes monitoring surf The first priority is completing the P0 metrics that tell you whether the platform is fundamentally working: supervisor sessions active (gauge), connect/disconnect counters, sandboxes-by-phase gauge, relay opens/claims/pending. These are emitted from `crates/openshell-server/` using the existing `metrics` crate facade, following the same pattern as the existing gRPC/HTTP RED metrics but covering new subsystems. -Next is a Helm `ServiceMonitor` CRD template, gated behind a Helm value and off by default. When enabled, it wires Prometheus scraping to the gateway's metrics port. The template lives in `deploy/helm/openshell/templates/` alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. +Next is a Prometheus Operator `ServiceMonitor` CRD template in the Helm chart, gated behind a Helm value and off by default. When enabled, it wires Prometheus scraping to the gateway's metrics port. The template lives in `deploy/helm/openshell/templates/` alongside the existing Service and StatefulSet templates. Per [#2507](https://github.com/NVIDIA/OpenShell/issues/2507), the question of whether `--metrics-port` should auto-enable when the chart wires scraping needs a decision. After that comes P1 metrics (SSH tunnel saturation, compute driver RPC duration, DB operation latency, policy merge retries), then dashboards (Grafana JSON models in the Helm chart or as separate artifacts), alerting rules (PrometheusRule CRDs), and eventually OTLP metrics push as an opt-in complement to Prometheus scrape for environments that cannot use pull-based collection. From 7b2260ebe2ed05dcb3dbc3b4f368882be74f1332 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:28:41 +0200 Subject: [PATCH 19/22] docs(rfc): reframe gateway root span alternative without prior version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Describe the alternative as a design option considered and rejected, not as a prior version of this RFC that readers would not have seen. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index f43b4d9c4c..d5da15f294 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -385,7 +385,7 @@ These approaches are not mutually exclusive. The design follows the same pattern ### Gateway-owned root span with TRACEPARENT -An earlier version of this design proposed a long-lived gateway root span for the sandbox lifecycle, with agent spans as children via `TRACEPARENT`. This was rejected because sandbox lifetimes are unpredictable (seconds to days) and long-lived parent spans are an OTel antipattern: they delay export, strain backends, hold gateway memory, and misrepresent the semantic structure. Span links provide correlation without duration coupling. +Instead of span links, the gateway could own a long-lived root span for the entire sandbox lifecycle and pass it to agents via `TRACEPARENT`, making agent spans children of the gateway span. This does not work well because sandbox lifetimes are unpredictable (seconds to days) and long-lived parent spans are an OTel antipattern: the batch exporter delays export until the span closes, trace backends assume traces complete within a time window, and the gateway holds open spans in memory for every concurrent sandbox. Span links provide correlation without duration coupling. ### Direct OTLP export from the sandbox From 2f3ffbe3a4769aeb50317447dd98d485d6899af4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:31:06 +0200 Subject: [PATCH 20/22] docs(rfc): link Dapr to GitHub repo instead of marketing site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index d5da15f294..8de67315fa 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -144,7 +144,7 @@ In practice, the OTLP receiver reuses the same networking plumbing the proxy alr The K8s sidecar topology is the exception. Because it uses nftables-based traffic interception in the pod's shared network namespace rather than creating a separate workload netns, `localhost` is truly shared between the supervisor and agent. Binding to `localhost:4318` works directly. -This pattern follows [Dapr](https://dapr.io)'s [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) approach, where a co-located proxy intercepts application communication and generates distributed traces through an [OTLP collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/) without requiring application-level instrumentation. The agent exports to a local address; the supervisor handles buffering, enrichment, and forwarding. +This pattern follows [Dapr](https://github.com/dapr/dapr)'s [sidecar](https://docs.dapr.io/concepts/dapr-services/sidecar/) approach, where a co-located proxy intercepts application communication and generates distributed traces through an [OTLP collector](https://docs.dapr.io/operations/observability/tracing/otel-collector/open-telemetry-collector/) without requiring application-level instrumentation. The agent exports to a local address; the supervisor handles buffering, enrichment, and forwarding. ### Span links for sandbox-to-trace correlation @@ -397,7 +397,7 @@ Leaving the current state unchanged preserves all the gaps described in the Moti ## Prior art -**[Dapr](https://dapr.io) sidecar architecture and OTel integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. +**[Dapr](https://github.com/dapr/dapr) sidecar architecture and OTel integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define the emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. From a1f04d72c56813623ec6ba9ad94f571170c7d335 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:35:37 +0200 Subject: [PATCH 21/22] docs(rfc): present MLflow as one example, add Langfuse as alternative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MLflow was used throughout as if it were the only agent trace backend. Reframe it as one example alongside Langfuse and use generic "agent trace backend" where the specific tool does not matter. This makes the RFC backend-agnostic. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index 8de67315fa..ed01950576 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -23,7 +23,7 @@ Tracing receives the most architectural attention in this RFC because it require ## Motivation -Operating OpenShell at scale requires answering questions that span multiple layers. A platform operator needs to know why sandbox creation is slow. A security reviewer needs to correlate a network deny event with the request that triggered it. An agent developer needs to see their agent's tool calls and LLM invocations in MLflow without writing OpenShell-specific instrumentation. A platform integrator needs a clean contract for wiring OpenShell into their monitoring stack. +Operating OpenShell at scale requires answering questions that span multiple layers. A platform operator needs to know why sandbox creation is slow. A security reviewer needs to correlate a network deny event with the request that triggered it. An agent developer needs to see their agent's tool calls and LLM invocations in their observability tool (MLflow, Langfuse, or similar) without writing OpenShell-specific instrumentation. A platform integrator needs a clean contract for wiring OpenShell into their monitoring stack. OpenShell's observability story is still early. That is expected for a project at this stage, and contributions have made real progress. OTLP trace export merged for the gateway ([#2534](https://github.com/NVIDIA/OpenShell/pull/2534)) and VM driver ([#2564](https://github.com/NVIDIA/OpenShell/pull/2564)), with a shared `openshell-otel` crate ([#2567](https://github.com/NVIDIA/OpenShell/pull/2567)). OCSF structured logging covers security events in the sandbox. Prometheus metrics provide basic gateway request counting ([#920](https://github.com/NVIDIA/OpenShell/pull/920)). These are reasonable starting points, but the gaps are significant enough that the next contributions need a coherent target rather than growing ad hoc: @@ -33,7 +33,7 @@ OpenShell's observability story is still early. That is expected for a project a - No centralized collection for OCSF security events. OCSF events are generated inside the sandbox but stay local with no path to a log aggregator. [#1922](https://github.com/NVIDIA/OpenShell/issues/1922) tracks this and is currently stale. - No collection mechanism for agent-level traces (tool calls, LLM invocations, reasoning steps from frameworks like LangChain or CrewAI running inside sandboxes). The sandbox is network-isolated, so an agent's OTel SDK cannot reach an external collector. -[#1055](https://github.com/NVIDIA/OpenShell/issues/1055) tracks the overall enterprise observability effort. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into MLflow without workarounds, and the three observability pillars remain disconnected. +[#1055](https://github.com/NVIDIA/OpenShell/issues/1055) tracks the overall enterprise observability effort. If the current design is left unchanged, platform operators cannot debug latency or failures across the gateway/driver/sandbox stack, agent developers cannot get their traces into tools like MLflow or Langfuse without workarounds, and the three observability pillars remain disconnected. ### Personas @@ -43,7 +43,7 @@ The personas below are proposed, not settled. OpenShell's persona model is evolv |---|---|---| | **Platform operator** | Deploys and operates the gateway cluster, manages compute drivers, monitors platform health | Distributed traces, SLI/SLO dashboards, alerting rules, Prometheus Rate/Errors/Duration (RED) metrics. Tools: Grafana, Jaeger/Tempo, Prometheus/Alertmanager | | **Security/compliance reviewer** | Reviews audit logs, investigates security events, ensures policy compliance | OCSF JSONL events, trace-to-OCSF correlation. Tools: log aggregation (Loki, Elasticsearch, Splunk) | -| **Agent developer** | Runs agents in sandboxes, debugs agent behavior, optimizes performance | Agent-level traces in MLflow or similar, with infrastructure context (sandbox, policy). Note: agent traces are a separate visibility domain from operator-only infrastructure traces per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) | +| **Agent developer** | Runs agents in sandboxes, debugs agent behavior, optimizes performance | Agent-level traces in an LLM observability tool (e.g., MLflow, Langfuse), with infrastructure context (sandbox, policy). Note: agent traces are a separate visibility domain from operator-only infrastructure traces per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) | | **Workspace administrator** | Manages a workspace (namespace, tenant) in a multi-tenant deployment | Per-workspace metrics: sandbox counts, policy violation rates, resource utilization | | **Platform integrator** | Integrates OpenShell into a larger platform (managed K8s, cloud AI platform, on-prem) | Documented OTLP endpoint contracts, resource attribute schemas, Helm values, integration guides. Tools: OTel Collector config, Helm values | @@ -84,11 +84,11 @@ For monitoring, the Prometheus wiring is in place: a `metrics` crate facade, a d Two distinct visibility domains flow through the same OTLP relay pipeline: 1. **Infrastructure traces** (operator-only): gateway request spans, driver lifecycle, supervisor network/process/middleware spans. Locked as operator-only per [#2508](https://github.com/NVIDIA/OpenShell/issues/2508). -2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow) from infrastructure traces (e.g., Tempo). +2. **Agent traces** (agent developer): tool calls, LLM invocations, reasoning steps from agent frameworks inside sandboxes. Routable to a separate backend (e.g., MLflow, Langfuse) from infrastructure traces (e.g., Tempo). -These two domains serve different user groups with different backend needs. Infrastructure traces typically go to the platform operator's monitoring stack (Tempo, Jaeger) for SLI/SLO dashboards and latency debugging. Agent traces go to the agent developer's tools (MLflow) for inspecting tool calls and reasoning steps. These may be entirely different systems, managed by different teams, with different retention and sampling policies. +These two domains serve different user groups with different backend needs. Infrastructure traces typically go to the platform operator's monitoring stack (Tempo, Jaeger) for SLI/SLO dashboards and latency debugging. Agent traces go to the agent developer's LLM observability tools (e.g., MLflow, Langfuse) for inspecting tool calls and reasoning steps. These may be entirely different systems, managed by different teams, with different retention and sampling policies. -Initially, both domains go to the same configured OTLP endpoint, and the platform integrator separates them at the collector level using resource attributes (e.g., route spans with `openshell.sandbox.*` attributes to MLflow, route spans with `openshell-gateway` service name to Tempo). As a future extension, the gateway could support per-domain OTLP endpoints (`infra_endpoint` and `agent_endpoint` alongside the default `endpoint`) so it can route infrastructure and agent traces to different backends directly, without requiring collector-side routing rules. This would also compose with per-workspace endpoints (see Multi-tenant observability below), giving each workspace independent routing for each visibility domain. +Initially, both domains go to the same configured OTLP endpoint, and the platform integrator separates them at the collector level using resource attributes (e.g., route spans with `openshell.sandbox.*` attributes to an agent trace backend like MLflow, route spans with `openshell-gateway` service name to Tempo). As a future extension, the gateway could support per-domain OTLP endpoints (`infra_endpoint` and `agent_endpoint` alongside the default `endpoint`) so it can route infrastructure and agent traces to different backends directly, without requiring collector-side routing rules. This would also compose with per-workspace endpoints (see Multi-tenant observability below), giving each workspace independent routing for each visibility domain. ### Supervisor as telemetry relay @@ -103,7 +103,7 @@ graph TD C -->|"OTLP/gRPC"| D["Trace Collector"] C -->|"Log forwarding"| F["Log Aggregator"] C -->|"Metrics"| G["/metrics endpoint"] - D --> E["MLflow / Jaeger / Tempo"] + D --> E["Agent trace backend / Jaeger / Tempo"] ``` The supervisor: @@ -122,7 +122,7 @@ The gateway: For agents, the experience is: `OTEL_EXPORTER_OTLP_ENDPOINT` is set automatically in the sandbox environment, pointing at the supervisor's OTLP receiver. Any OTel-instrumented framework works without OpenShell-specific code. -The relay also creates a clean separation of concerns between three roles. The agent developer exports to a fixed, auto-injected endpoint and never thinks about collector topology, authentication, or routing. The same agent code works in every sandbox, every deployment, every workspace. The workspace administrator decides where their workspace's telemetry goes (which collector, which MLflow instance, what sampling rates) without touching agent configuration. The global administrator configures the default OTLP endpoint and platform-wide policies like rate limits and enrichment. With direct collector access, the agent developer would need to know the collector address, and that address varies by deployment and workspace. The relay makes observability routing an operational concern rather than a development one. +The relay also creates a clean separation of concerns between three roles. The agent developer exports to a fixed, auto-injected endpoint and never thinks about collector topology, authentication, or routing. The same agent code works in every sandbox, every deployment, every workspace. The workspace administrator decides where their workspace's telemetry goes (which collector, which agent trace backend, what sampling rates) without touching agent configuration. The global administrator configures the default OTLP endpoint and platform-wide policies like rate limits and enrichment. With direct collector access, the agent developer would need to know the collector address, and that address varies by deployment and workspace. The relay makes observability routing an operational concern rather than a development one. ### OTLP receiver reachability per driver @@ -245,13 +245,13 @@ OpenShell is collector-agnostic. The Helm chart does not deploy an OTel Collecto ### Multi-tenant observability -Multi-tenant deployments need per-workspace observability isolation. Different workspaces may route agent traces to different collectors or MLflow instances, and one tenant's trace data must not leak to another's endpoint. +Multi-tenant deployments need per-workspace observability isolation. Different workspaces may route agent traces to different collectors or agent trace backends, and one tenant's trace data must not leak to another's endpoint. The OTLP relay supports this through per-workspace OTLP endpoint configuration at the gateway. When a supervisor forwards spans, the gateway looks up the workspace's configured endpoint and routes accordingly. If no per-workspace endpoint is configured, spans go to the global default. The `openshell.workspace.id` resource attribute (added during span enrichment) enables collector-side routing for deployments that use a shared collector with attribute-based routing rather than per-workspace endpoints. Per-workspace sampling and rate limits follow the same pattern as per-sandbox limits described in the Risks section, but scoped to the workspace level. A noisy workspace should not exhaust the gateway's relay capacity for other tenants. -The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before the OTLP relay ships (see [Phase 2: Sandbox OTLP relay](#phase-2-sandbox-otlp-relay) in the Implementation plan). Per-workspace configuration would compose with the per-domain endpoint extension described in Visibility domains: a workspace could route its infrastructure traces to a shared platform Tempo while sending its agent traces to a workspace-specific MLflow instance. +The configuration model for per-workspace OTLP endpoints is not yet designed. Options include extending `gateway.toml` with per-workspace sections, using workspace-level CRDs, or deriving the endpoint from the workspace settings API. This decision depends on how workspace configuration is structured more broadly, which is outside the scope of this RFC but must be resolved before the OTLP relay ships (see [Phase 2: Sandbox OTLP relay](#phase-2-sandbox-otlp-relay) in the Implementation plan). Per-workspace configuration would compose with the per-domain endpoint extension described in Visibility domains: a workspace could route its infrastructure traces to a shared platform Tempo while sending its agent traces to a workspace-specific agent trace backend. ### Metrics direction @@ -365,7 +365,7 @@ Several mitigations should be part of the implementation: The OpenTelemetry Rust SDK is a non-trivial dependency. Feature-gating it at compile time is worth evaluating (alongside [#1943](https://github.com/NVIDIA/OpenShell/issues/1943)) so that builds without OTel support do not pay the binary size and compile time cost. -Not all trace backends handle span links equally well. Jaeger and Grafana Tempo support bidirectional link navigation, but MLflow's OTLP ingestion may not surface links in its UI. If that turns out to be the case, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. +Not all trace backends handle span links equally well. Jaeger and Grafana Tempo support bidirectional link navigation, but not all agent trace backends surface links in their UI (e.g., MLflow's OTLP ingestion may not). If that turns out to be the case for a given backend, the correlation story would be limited to resource attributes rather than navigable links. ## Alternatives @@ -399,14 +399,14 @@ Leaving the current state unchanged preserves all the gaps described in the Moti **[Dapr](https://github.com/dapr/dapr) sidecar architecture and OTel integration.** Dapr's sidecar model, referenced in the OTLP receiver reachability section above, is the closest prior art for the supervisor relay pattern. Applications opt in with a single Kubernetes annotation (`dapr.io/config`), and the three-tier architecture (application -> sidecar -> collector -> backend) maps directly to OpenShell's design (agent -> supervisor -> gateway -> collector). The lesson is that transparent trace collection through a co-located proxy scales across diverse application frameworks without requiring per-application instrumentation. -The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define the emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for MLflow compatibility. +The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) define the emerging standard for agent trace attributes (`gen_ai.operation.name`, `gen_ai.agent.name`, etc.). OpenShell infrastructure spans use standard RPC/HTTP conventions; agent-emitted spans from inside sandboxes should follow GenAI conventions for compatibility with agent trace backends like MLflow and Langfuse. Red Hat's CI pipelines for running agents in OpenShell sandboxes independently discovered the root span flushing problem for short-lived sandboxes and worked around it with an orchestrator-owned root span passed via `TRACEPARENT`. The span link approach in this RFC avoids the duration coupling while solving the same correlation problem. ## Open questions - How should `OTEL_*` environment variables and TOML gateway configuration interact when both are set? [#2507](https://github.com/NVIDIA/OpenShell/issues/2507) and [RFC 0003](../0003-gateway-configuration/README.md) discuss this but no decision has been made on precedence. -- Does MLflow's OTLP ingestion surface span links in its UI? If it does not, the correlation story for agent developers using MLflow would be limited to resource attributes rather than navigable links. This needs validation. +- Do common agent trace backends (MLflow, Langfuse) surface span links in their UI? If not, the correlation story for agent developers would be limited to resource attributes rather than navigable links. This needs validation per backend. - Should the OTel SDK dependency be gated behind a Cargo feature flag so that builds without tracing support do not pay the binary size and compile time cost? - Should the supervisor implement the OTLP receiver from scratch using the `opentelemetry-proto` crate (smaller binary, more control), or embed a lightweight collector library (more features, larger dependency)? The tradeoffs need evaluation during implementation. - [#2508](https://github.com/NVIDIA/OpenShell/issues/2508) deliberately left `traceparent` injection into agent egress unsettled. Whether the supervisor stamps W3C trace context onto the agent's outbound HTTP requests is a separate decision from the span link correlation proposed in this RFC. From 0d9a4b6d7e0cc1968e9a4fd779b8536192516f5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Wed, 5 Aug 2026 20:40:18 +0200 Subject: [PATCH 22/22] docs(rfc): add span links explanation and OTel docs link in Risks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Explain what span links are and what "surface links in the UI" means concretely (one-click navigation vs manual attribute filtering). Link to OTel docs and the RFC's own span links section. Signed-off-by: Roland Huß --- rfc/0012-observability-architecture/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/0012-observability-architecture/README.md b/rfc/0012-observability-architecture/README.md index ed01950576..a2dd6eaa20 100644 --- a/rfc/0012-observability-architecture/README.md +++ b/rfc/0012-observability-architecture/README.md @@ -365,7 +365,7 @@ Several mitigations should be part of the implementation: The OpenTelemetry Rust SDK is a non-trivial dependency. Feature-gating it at compile time is worth evaluating (alongside [#1943](https://github.com/NVIDIA/OpenShell/issues/1943)) so that builds without OTel support do not pay the binary size and compile time cost. -Not all trace backends handle span links equally well. Jaeger and Grafana Tempo support bidirectional link navigation, but not all agent trace backends surface links in their UI (e.g., MLflow's OTLP ingestion may not). If that turns out to be the case for a given backend, the correlation story would be limited to resource attributes rather than navigable links. +Not all trace backends handle span links equally well. Span links are ["related to" references between spans](https://opentelemetry.io/docs/concepts/signals/traces/#span-links) (see [Span links for sandbox-to-trace correlation](#span-links-for-sandbox-to-trace-correlation)), and a backend that supports them lets users click from an agent trace to the gateway's `sandbox.create` span and back. Jaeger and Grafana Tempo support this bidirectional navigation, but not all agent trace backends surface links in their UI (e.g., MLflow's OTLP ingestion may not). If a given backend does not render span links, the correlation story falls back to resource attributes (filtering by `openshell.sandbox.id`), which still works but requires manual querying rather than one-click navigation. ## Alternatives