Backlog: 14 unscheduled feature/hardening ideas from gaps-implementation-plan.md #242

Summary

Consolidates 14 still-open backlog ideas from docs/gaps/gaps-implementation-plan.md, which is being archived to docs/archived/gaps-implementation-plan.md (no longer actively maintained — new work is tracked as individual GitHub issues going forward, matching how #236/#238/#239/#240 were already handled). None of these are blocking or in-progress; this is a single roundup issue rather than 14 separate ones, since none currently have an owner or a scheduled phase. Full historical context (phase numbers, "from BACKLOG.md"/"from operator comparison" provenance notes) is preserved in the archived doc.

Split any of these out into its own issue once someone picks it up.

Per-sandbox identity binding

Problem: Current TokenReview validates any cloud-agents audience token. Does not bind caller identity to the specific spawned sandbox container.

What to build: Generate scoped ServiceAccounts per sandbox spawn; verify identity matches the specific sandbox when results are returned.

Effort: 3-4 weeks

Risks: K8s SA lifecycle adds 2 extra creates + 2 extra deletes per step, multiplying with parallel workflows. Cleanup on failure: SA + RoleBinding orphaned if Job creation fails — current orphan reconciliation only finds Jobs, not RBAC resources. Token propagation timing: projected SA tokens need the pod running before the token is available. K8s-only, no OpenShell-gateway equivalent yet — needs re-evaluation now that OpenShellSpawner is the sole ephemeral spawner (issue #198). Shares a prerequisite auth-evolution design doc with the dynamic-RBAC item below.

Dynamic RBAC from agent output

Problem: Agent output can declare RBAC requirements, but the framework doesn't create scoped Roles/RoleBindings dynamically.

What to build: After analysis step, read an rbac field from output, create per-proposal Roles/RoleBindings, clean up on completion.

Effort: 4-6 weeks

Risks: Requires controller-like lifecycle management without K8s controller machinery. Runner needs elevated RBAC (create Roles/RoleBindings), expanding the security surface. Cleanup depends on finally blocks (known reliability issues). Sandbox response schema has no rbac field yet. K8s-only — needs re-evaluation post-#198. Depends on per-sandbox identity binding (above) landing first.

Agents-as-tools

Problem: No way for a chatbot conversation to invoke workflows as LLM tools.

What to build: Registry auto-generates LLM tool definitions from workflow definitions, so a chatbot agent can call e.g. start_diagnostic_workflow(cluster, issue) as a tool.

Effort: 4-6 weeks

Risks: Sync/async impedance mismatch — LLM tool calls expect synchronous responses, but Temporal workflows are async (approvals, retries, minutes-long); none of block/poll/restrict-to-fast-auto-approved-only is solved yet. Schema translation from WorkflowInput's 13 fields to tool parameters is non-obvious and workflow-specific. No consumer without the chatbot-trigger item below (circular dependency). Recommendation from the original plan: start with manually-registered tools for one specific workflow, not auto-generation, and solve the async pattern first.

Chatbot trigger

Problem: Only an API trigger exists. No chatbot/conversation trigger.

What to build: Integration with the lightspeed-stack /query conversation flow. Depends on agents-as-tools (above).

Effort: TBD — depends on lightspeed-stack integration scope.

Risks: Double-blocked — depends on agents-as-tools (itself blocked on an unsolved sync/async pattern) AND on lightspeed-stack integration (external team, unknown scope). Don't plan until agents-as-tools is complete and the lightspeed-stack integration surface is documented.

Conversational approval

Problem: Approval gates pause workflows but the only channels are Slack/webhook. No in-conversation approval.

What to build: When a workflow hits an approval gate, the LLM surfaces it to the user in natural language; the user approves/rejects in the conversation flow.

Effort: TBD — depends on chatbot integration (above).

Nested workflows

What to build: Workflow-to-workflow composition (recursive execution).

Hidden complexity: Temporal supports child workflows, but Cloud Agents routes everything through run_sandbox_step. A nested workflow either bypasses the sandbox (needs a new step type) or creates a circular dependency (sandbox calls back to the runner API). Resource exhaustion risk: 3-deep nesting of 3-step workflows is 6+ pods, and MAX_SPAWNED_PODS is a global cap with no per-workflow budget. Approval propagation is undefined — if a nested workflow hits an approval gate, does the parent block?

Workflow versioning and rollback

What to build: Schema migration + state compatibility for definition updates.

Resumable SSE streaming

What to build: Persisted event replay via Last-Event-ID.

Depends on: #241 (progress streaming) — that item enriches what's streamed; this one adds reconnection on top.

Spawner spec caching

What to build: Cache spawner configurations (env vars, volumes, labels) by content hash. When multiple workflow steps use identical sandbox config, reuse the cached spec instead of rebuilding it.

Priority: Low — negligible overhead at expected volumes. Also needs re-evaluation for whether it still applies to OpenShellSpawner's sandbox spec shape (this item predates the OpenShell-only consolidation in issue #198).

Agent artifact storage

What to build: OCI artifacts, derived images, git-sync sidecar for tool/skill distribution.

Note: Partially superseded by the baked-in-image skills approach from issue #202 — re-scope before picking this up.

Workflow visualization

What to build: Graph rendering UI or OpenShift console plugin integration.

SBOM / SLSA provenance

What to build: Image signing attestation and software bill of materials.

Prompt injection guardrails

Problem: Workflow definitions are submitted as arbitrary dicts. Pydantic validates schema but doesn't restrict prompt/instruction content. A malicious prompt could instruct the LLM to exfiltrate data, ignore safety guidelines, or produce harmful output — especially relevant once non-admin users can trigger workflows (post-RBAC).

What to build: Design decision needed — input-side filtering (reject suspicious prompts) vs output-side filtering (scan agent output) vs both. Consider integration with existing guardrail frameworks (pydantic-ai-shields, llm-guard). At minimum: log a warning when prompts contain known injection patterns ("ignore previous instructions", "system prompt override").

Effort: Logging-only detection is 1-2 days; full guardrail integration is 1-2 weeks.

Audit log integrity

Problem: Audit events go to stdout/stderr via structured logging. No signed audit trail, no tamper-evident log chain, no guaranteed delivery. An operator with log access could modify audit records.

What to build: Append audit events to a dedicated audit log file (separate from application logs). Add HMAC signatures or a hash chain for tamper detection. Optionally forward audit events to an external audit service (webhook).

Effort: 1-2 days for a file-based audit log; 1 week for signed/chained logs.