feat(audit): emit structured gateway control-plane mutation events #2911

User Story

As a security or compliance owner, I want every state-changing gateway operation and system-driven mutation recorded as a structured audit event, so that I can determine who changed what, when it changed, and what outcome OpenShell produced.

Problem Statement

OpenShell emits structured OCSF events for sandbox behavior and exports gateway operational spans, but it does not have a comprehensive audit record for gateway control-plane mutations. Operations involving sandboxes, workspaces, providers, credentials, policies, settings, inference routes, and other managed resources can change state without producing a consistent record containing the authenticated actor and terminal outcome. Background work such as reconciliation or credential rotation has no request actor and can be invisible.

Existing issues own adjacent surfaces:

• feat(observability): OCSF event export from gateway to external SIEM destinations #2762 owns external export of OCSF events from the gateway.
• feat(observability): shared telemetry relay infrastructure for supervisor and agent traces #2642 owns telemetry relay buffering, backpressure, and drop accounting.
• feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640 owns OCSF and OpenTelemetry trace correlation fields.
• Centralized audit/event log #1933 owns a client-facing event subscription and replay surface.

This issue owns production and coverage of the gateway audit events that those export and subscription surfaces can consume. It extracts the non-overlapping control-plane audit scope from #2892 and sits under the enterprise observability umbrella in #1055.

Impact / Why This Matters

Operators cannot reliably answer questions such as who deleted a sandbox, whether a credential rotation was user-initiated or automatic, or whether a policy mutation failed or made no change. Their current workaround is to correlate ordinary process logs, request traces, and database state after the fact. Those sources are not a stable audit contract, do not consistently identify the authenticated actor or system job, and may never record the mutation outcome.

This prevents reliable incident reconstruction and forces external integrations to infer authoritative state changes from implementation-specific log text.

Proposed Design

OpenShell should publish a documented, versioned catalog of security-relevant gateway mutations and emit one structured terminal audit event for each operation attempt. The externally observable record should include:

• the operation and resource type, with stable resource identifiers when available;
• the authenticated actor derived from the trusted request context, or a defined system actor for background work;
• an explicit outcome that distinguishes a state change, a successful no-op, a rejection, and an internal failure without changing the API's existing semantics;
• the request or correlation identifier and trace identifiers when available;
• a timestamp and severity suitable for filtering; and
• redacted metadata sufficient to investigate the change without exposing credential-shaped values.

Audit-event production should remain distinct from external export configuration. Disabling or changing an export destination must not silently redefine which gateway operations produce audit records. Optional capture of sensitive-but-non-secret details, such as command arguments or setting values, should have explicit configuration and documented defaults. Credentials, tokens, bearer material, request environment variables, and standard input must never be included.

The event catalog should identify both request-driven and background mutations. New state-changing gateway operations should update the catalog and their audit coverage as part of the same change.

Acceptance Criteria

• A documented event catalog enumerates the current security-relevant state-changing gateway RPCs and background operations covered by this feature.
• Every cataloged operation emits one terminal audit record per attempt, including success, successful no-op, authorization or validation rejection, and internal failure outcomes as applicable.
• Request-driven records identify the authenticated actor from trusted gateway context rather than actor data supplied in the request body.
• Background reconciliation, rotation, startup migration, or similar cataloged work identifies a defined system actor.
• Records include the affected resource type, stable identifier when available, operation, outcome, timestamp, severity, and request or correlation identifier.
• A no-op is represented explicitly without changing the operation's existing API success or failure semantics.
• Credential values, minted tokens, SSH bearer material, request environment variables, and standard input never appear in an audit record; automated tests cover representative secret canaries.
• Configuration for optional sensitive fields is documented separately from external export configuration.
• Contributor documentation requires new security-relevant gateway mutations to update the event catalog and audit coverage.
• The record catalog, field semantics, redaction behavior, and configuration are documented for operators.

Alternatives Considered

Treat operational traces as the audit record. Traces help diagnose latency and causality, but sampling and span attributes do not provide a stable, comprehensive mutation ledger.

Infer mutations from database changes. Store-level observation loses the authenticated request context and cannot reliably distinguish user actions, rejected attempts, and system-driven work.

Add the records only to #1933's client event stream. A subscription API is one consumer and transport. Defining audit-event production independently allows the same authoritative records to feed local logs, #2762's external exporter, and a future client stream without duplicating mutation instrumentation.

Agent Investigation

• Gateway request spans already carry request IDs and can export over OTLP, but the current OTLP integration exports traces rather than audit logs.
• Authenticated Principal data is available to gateway handlers and can supply trusted actor identity.
• crates/openshell-ocsf already provides structured event builders, including API Activity and configuration state-change event types, but gateway mutation handlers do not use them comprehensively.
• Related: Enterprise Observability #1055, Centralized audit/event log #1933, feat(observability): OpenTelemetry export surface for gateway operators #2507, feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640, feat(observability): shared telemetry relay infrastructure for supervisor and agent traces #2642, feat(observability): OCSF event export from gateway to external SIEM destinations #2762, and Enhance OpenShell's OCSF telemetry and export it off-box via OTLP #2892.