Add OpenTelemetry trace correlation across gateway activity #1758

Problem Statement

OpenShell documents OCSF JSON export for SIEM-style events, but a multi-step agent action still needs distributed trace context so operators can see one tree across invocation, tool call, policy decision, broker token acquisition, downstream request, and response. Compliance mapping to EU AI Act Annex IV and NIST AI RMF should be a follow-up documentation issue; this issue stays focused on OTel trace emission and correlation identifiers.

Proposed Design

1. Emit OTel spans for agent invocation, tool call, policy decision, sandbox lifecycle, credential-broker actions, downstream request, and response.
2. Carry stable correlation fields on every span: agent_id, session_id, user_principal where applicable, tool_name, binary_path, binary_sha256 where available, policy_decision, policy_rule_matched, resource/host, and outcome.
3. Preserve OCSF JSON export and align OCSF event correlation IDs with OTel trace/span identifiers where practical.
4. Document a minimal collector setup for Tempo or Jaeger.

Acceptance criteria:

• A multi-step agent action appears as a single trace tree from invocation through downstream response.
• agent_id and session_id correlate consistently across OCSF events and OTel spans.
• Trace emission can be enabled without disabling existing OCSF JSON export.
• A reference deployment exports traces to a stock OTel collector and at least one common backend such as Tempo or Jaeger.

Alternatives Considered

• Keep only OCSF logs: useful for SIEM, but weaker for reconstructing multi-hop execution paths.
• Rely on downstream service traces only: misses gateway policy and sandbox lifecycle decisions.
• Add compliance reporting first: useful, but depends on stable event and trace identifiers.

Agent Investigation

• OCSF JSON export exists and should continue unchanged.
• The Providers v2 Roadmap explicitly lists "Refresh telemetry as OCSF events" as future work — refresh logs today are secret-safe gateway logs only.
• Current public docs do not describe OTel trace export with consistent agent_id/session_id correlation across gateway activity.

References:

• OpenShell OCSF JSON export: https://docs.nvidia.com/openshell/observability/ocsf-json-export
• OpenShell Providers v2 (Roadmap): https://docs.nvidia.com/openshell/sandboxes/providers-v2
• OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
• EU AI Act (for follow-up mapping issue): https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• NIST AI RMF (for follow-up mapping issue): https://www.nist.gov/itl/ai-risk-management-framework

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request