RSPEED-0000: Skills image via Podman driver_config + fragility analysis

docs/spikes/openshell-spawner-spike.md

@@ -305,31 +305,136 @@ Additional changes:
 
 ### Resolved
 
-1. **RESOLVED**: Standalone Podman gateway works on macOS with JWT config. Podman secret file mount is broken in 5.8.x but automated workaround implemented (issue #82).
+1. **RESOLVED**: Standalone Podman gateway works on macOS with JWT config.
 2. **RESOLVED**: Supervisor auth chain works end-to-end with Podman driver (JWT minting, supervisor authentication).
-3. **RESOLVED**: Podman secret file mount workaround implemented in `openshell_spawner.py`.
+3. **RESOLVED**: Podman 5.8.x JWT workaround — **removed**. OpenShell v0.0.79+ (PR NVIDIA/OpenShell#2156) delivers sandbox JWTs natively via Podman secrets when `gateway_jwt` is configured with signing keys. No client-side workaround needed. Requires gateway v0.0.79+.
 4. **RESOLVED**: RHEL 9 verification — gateway + supervisor + sandbox lifecycle works. Seccomp and network namespace isolation confirmed.
 5. **RESOLVED**: UBI 9 sandbox images — requires source-built supervisor (glibc 2.34 compatibility). UBI 10 works with pre-built binaries.
+6. **RESOLVED**: Production gaps closed (PR #105) — skills image, credentials, filesystem policy, MCP secrets, TLS skip, service account skip, provider cleanup. All 7 example workflows pass E2E through OpenShell on RHEL 9 + Kind cluster.
+7. **RESOLVED**: ~~Containerized gateway deployment (DooD pattern)~~ — verified (see "Containerized Gateway" section).
 
 ### Remaining
 
 1. OpenShell team commits to stabilizing the Python SDK (especially `ExposeService` wrapper)
 2. Gateway resource overhead measurement in target deployment environments
 3. L7 network policy testing with live gateway (auto-derivation implemented in `_build_network_policy()`)
 4. Sandbox image must include `iproute2`, `nsenter` (`util-linux-core`), and a `sandbox` user
-5. ~~Containerized gateway deployment (DooD pattern)~~ — verified above (see "Containerized Gateway" section)
-6. Upstream: request musl-static supervisor binary or RHEL 9-compatible builds
+5. Upstream: request musl-static supervisor binary or RHEL 9-compatible builds
+6. Skills image extraction for OpenShell + K8s driver (issue #106, see Fragility Analysis below)
 
 **Immediate value**: Eliminates duplicated spawner code and provides defense-in-depth isolation (seccomp + network namespace). Even without L7 policy, this is a security improvement over container securityContext alone.
 
 **If no-go**: The prototype code remains as a third spawner option. No changes to existing K8s/Podman spawners.
 
+## Fragility Analysis
+
+Assessment of each production feature's robustness, what we've done, and known gaps.
+
+### Skills image — moderate fragility
+
+**What we've done**: `_load_skills()` extracts skills from an OCI image locally (Podman SDK or CLI), creates a tar, streams it into the sandbox via `exec_stream`. Works for OpenShell + Podman driver.
+
+**Fragility**:
+- **OpenShell + K8s driver**: Runner is a K8s pod with no Podman socket. Neither Podman SDK nor CLI is available. `_extract_skills_image()` raises `RuntimeError`. Issue #106 tracks this.
+- **Full image pull**: `crane export` or Podman SDK downloads ALL layers of the skills image, not just the `/skills` directory. A fat base image (e.g. `python:3.12` + 5MB of skills) wastes bandwidth.
+
+**Mitigation**: Build skills images as minimal single-layer images (`FROM scratch` + `COPY skills/ /skills/`). This is a convention, not a code change. A 5MB skills directory = 5MB image. The extraction code works efficiently with minimal images.
+
+**Future options for K8s driver**:
+1. Install `crane` in the sandbox image and pull from inside (no runner-side extraction)
+2. OpenShell `driver_config` volume mounts (needs upstream investigation)
+3. Operator pre-populates skills into sandbox spec
+
+### Credentials (Provider API) — moderate fragility
+
+**What we've done**: `_inject_credentials()` first tries the OpenShell Provider API (`CreateProvider` + `AttachSandboxProvider` via raw gRPC), then falls back to file injection via `_do_write_file()`.
+
+**Fragility**:
+- **Raw gRPC**: Uses `grpc.insecure_channel` and accesses `self._client._endpoint` (private attribute). If the OpenShell SDK changes its internal structure, this breaks silently.
+- **Proto compatibility**: `CreateProviderRequest` and `AttachSandboxProviderRequest` protobuf messages must match the gateway version. No version negotiation.
+
+**Mitigation**: File-based fallback catches Provider API failures and writes credentials to `/var/run/secrets/llm-credentials/`. This degraded path works for all providers. Missing credentials now raise `RuntimeError` (fail-closed) instead of silently skipping.
+
+### MCP secret injection — solid
+
+**What we've done**: `_inject_mcp_secrets()` creates directories via `exec_stream(mkdir -p)` and writes secret values via `_do_write_file()` (base64 + exec).
+
+**Why it's solid**: Uses the same proven base64+exec pattern that `_do_write_file()` and `_do_read_file()` use for transcript collection. No external dependencies. Works on any OpenShell driver.
+
+### Filesystem policy — solid
+
+**What we've done**: `_build_filesystem_policy()` sets `spec.policy.filesystem.read_only = ["/"]` with explicit `read_write` exceptions for `/tmp`, `/home/agent`, `/var/log`, `/app/skills`, `/var/secrets/mcp`, `/var/run/secrets/llm-credentials`.
+
+**Why it's solid**: Sets protobuf fields directly on `SandboxSpec.policy` — same mechanism as `_build_network_policy()`. Clean contract with no external dependencies.
+
+### Network policy auto-derivation — solid
+
+**What we've done**: `_build_network_policy()` derives deny-by-default L7 egress rules from `LIGHTSPEED_PROVIDER` (provider-to-host mapping) and `LIGHTSPEED_MCP_SERVERS` (parsed URLs). Uses scheme-based default ports (80 for http, 443 for https).
+
+**Why it's solid**: Same protobuf contract as filesystem policy. Provider host mapping is a static dict. MCP URLs are parsed with stdlib `urlparse`.
+
+### TLS / Service account — solid (not applicable)
+
+**What we've done**: Log info and skip. The gateway provides transport security (TLS) and manages identity (no K8s SA equivalent). These are architectural decisions, not workarounds.
+
+### Provider cleanup — solid
+
+**What we've done**: `_do_destroy()` detaches providers via `DetachSandboxProvider` gRPC before deleting the sandbox. `_cleanup_sandbox()` does the same on spawn failure. Provider IDs are tracked in `self._provider_ids`.
+
+**Minor fragility**: Uses the same raw gRPC pattern as credential injection. If detach fails, it logs a warning and continues with sandbox deletion (non-blocking).
+
+## Spawner Comparison: OpenShell vs K8s vs Podman
+
+### Isolation & Security
+
+| | Podman | Kubernetes | OpenShell |
+|---|---|---|---|
+| **Isolation** | Linux containers (namespaces/cgroups) | K8s pods with hardened security context | MicroVM (strongest) |
+| **Network egress** | Podman network (manual config) | Cluster NetworkPolicy (manual) | Auto-derived deny-by-default from provider + MCP config |
+| **Filesystem** | Optional read-only flag | Always read-only rootfs | Granular read/write path policy |
+| **TLS** | App-level cert injection | App-level cert injection | Not needed — gateway handles transport |
+
+### Feature Parity
+
+| Feature | Podman | Kubernetes | OpenShell |
+|---|---|---|---|
+| **Skills image** | Volume + transient container | Init container + emptyDir | Podman SDK extract + tar stream (Podman driver ✅, K8s driver ❌ issue #106) |
+| **Credentials** | Skipped (warning) | Secret volume + envFrom | Provider API + file fallback |
+| **MCP secrets** | Rejected (ValueError) | Secret volumes | File injection via exec |
+| **Read-only mode** | Container flag | Always enforced | Policy with write exceptions |
+| **Service account** | Ignored | PodSpec SA + projected token | N/A (gateway manages identity) |
+| **Resource limits** | No | CPU/memory via SpawnConfig | No |
+| **Progress streaming** | No | No | Yes (real-time JSONL tail) |
+
+### Architecture
+
+| | Podman | Kubernetes | OpenShell |
+|---|---|---|---|
+| **Primitive** | Container | Job + Service | Sandbox (microVM) |
+| **Readiness** | Base class HTTP poll | Base class HTTP poll | Custom Host-header poll (gateway-routed) |
+| **Routing** | `localhost:{random_port}` | `{name}.{ns}.svc:8080` | Gateway virtual-host (`Host` header) |
+| **Cleanup** | Stop + remove container | Delete Job + Service | gRPC Delete + provider detach |
+| **Instance tracking** | Podman label query | K8s Job label query | In-memory dicts |
+| **Dependencies** | Podman socket | K8s API server | OpenShell gateway (gRPC) v0.0.79+ |
+
+### Key Findings
+
+1. **OpenShell is the most secure** — microVM isolation + auto-derived L7 network policy + gateway-managed credentials. K8s is second (always-hardened securityContext). Podman is weakest (socket access = host-level control).
+
+2. **OpenShell is the most feature-complete** — only spawner with progress streaming, Provider API, auto network policy, and granular filesystem policy. K8s has resource limits that OpenShell lacks.
+
+3. **Podman has the most gaps** — no credential injection, no MCP secrets (ValueError), no resource limits, no progress streaming. It is a dev/demo spawner.
+
+4. **OpenShell eliminates TLS complexity** — gateway handles transport security, removing cert generation/mounting/cleanup code.
+
+5. **Trade-off**: OpenShell adds a gateway infrastructure dependency. K8s/Podman spawners talk directly to the container runtime.
+
 ## Files
 
 | File | Description |
 |---|---|
-| `src/cloud_agents/spawner/openshell_spawner.py` | Spawner implementation |
-| `tests/unit/spawner/test_openshell_spawner.py` | 27 unit tests |
+| `src/cloud_agents/spawner/openshell_spawner.py` | Spawner implementation (52 tests) |
+| `tests/unit/spawner/test_openshell_spawner.py` | Unit tests |
 | `docs/spikes/openshell-standalone-setup.md` | Standalone gateway setup guide |
 | `pyproject.toml` | `openshell` optional dependency |
 | `docs/gaps/gaps-implementation-plan.md` | T53 entry |

src/cloud_agents/spawner/openshell_spawner.py

@@ -66,6 +66,7 @@ class OpenShellSpawner(AgentSpawner):
     def __init__(
         self,
         openshell_client: Any = None,
+        driver: str = "podman",
         **kwargs: Any,
     ) -> None:
         """Initialize the OpenShell spawner.
@@ -74,6 +75,13 @@ def __init__(
             openshell_client: OpenShell SDK client for sandbox operations.
                 Must implement create(), wait_ready(), exec(), exec_stream(),
                 and delete().
+            driver: OpenShell compute driver type ("podman" or "kubernetes").
+                Controls how skills_image is handled:
+                - "podman": Uses native image mounts via driver_config
+                  (no extraction, no tar streaming — Podman mounts the
+                  OCI image directly at /app/skills).
+                - "kubernetes": Falls back to tar streaming (extract
+                  skills locally, stream into sandbox via exec_stream).
 
         Requires OpenShell gateway v0.0.79+ with gateway_jwt configured.
         The gateway mints sandbox JWTs and delivers them via Podman
@@ -89,6 +97,7 @@ def __init__(
         """
         super().__init__(**kwargs)
         self._client = openshell_client
+        self._driver = driver
         self._podman_cli: str | None = None
         self._sandbox_names: dict[str, str] = {}
         self._sandbox_ids: dict[str, str] = {}
@@ -353,6 +362,27 @@ async def _do_spawn(
         if read_only:
             self._build_filesystem_policy(spec)
 
+        # Skills image: Podman driver mounts the OCI image directly
+        # via driver_config (no extraction needed). K8s driver falls
+        # back to tar streaming after sandbox creation.
+        skills_via_driver_config = False
+        if skills_image and self._driver == "podman":
+            from google.protobuf import struct_pb2
+
+            mount_config = struct_pb2.Struct()
+            mount_config.update({
+                "podman": {
+                    "mounts": [{
+                        "type": "image",
+                        "source": skills_image,
+                        "target": "/app/skills",
+                        "read_only": True,
+                    }],
+                },
+            })
+            spec.template.driver_config.MergeFrom(mount_config)
+            skills_via_driver_config = True
+
         sandbox_ref = await asyncio.to_thread(self._client.create, spec=spec)
         sandbox_name = sandbox_ref.name
         sandbox_id = sandbox_ref.id
@@ -371,7 +401,7 @@ async def _do_spawn(
                     agent_name, sandbox_name, credential_secret_name, env,
                 )
 
-            if skills_image:
+            if skills_image and not skills_via_driver_config:
                 await self._load_skills(
                     agent_name, sandbox_id, skills_image, skills_paths,
                 )
@@ -467,7 +497,13 @@ async def _inject_credentials(
         Uses the OpenShell Provider system for gateway-managed credentials.
         Falls back to file-based injection if Provider RPCs fail.
         """
+        # credential_secret_name may be K8s-normalized (e.g. "openai-api-key")
+        # while the env dict has the original key (e.g. "OPENAI_API_KEY").
+        # Try both forms.
         cred_value = env.get(credential_secret_name)
+        if not cred_value:
+            original_key = credential_secret_name.upper().replace("-", "_")
+            cred_value = env.get(original_key)
         if not cred_value:
             raise RuntimeError(
                 f"Credential '{credential_secret_name}' not found in env "
@@ -631,9 +667,60 @@ async def _extract_skills_image(
     ) -> None:
         """Extract skills content from an OCI image to a local directory.
 
-        Uses the Podman Python SDK (cross-platform, works in any
-        container) or falls back to the podman CLI binary.
+        Fallback chain (first success wins):
+        1. crane — static binary, no daemon needed, works everywhere
+        2. Podman Python SDK — needs Podman socket
+        3. podman CLI — needs binary + socket
+
+        crane is the primary path because it works in K8s pods
+        where no container runtime socket is available. Install it
+        in the runner Containerfile from go-containerregistry releases.
         """
+        import shutil
+        import subprocess
+
+        # 1. Try crane (no daemon needed — works in K8s pods)
+        crane_bin = shutil.which("crane")
+        if crane_bin:
+            try:
+                def _crane_extract() -> None:
+                    result = subprocess.run(
+                        [crane_bin, "export", skills_image, "-"],
+                        capture_output=True,
+                        timeout=120,
+                    )
+                    if result.returncode != 0:
+                        raise RuntimeError(
+                            f"crane export failed: {result.stderr.decode()[:200]}"
+                        )
+                    import tarfile
+                    from io import BytesIO
+
+                    with tarfile.open(fileobj=BytesIO(result.stdout)) as tar:
+                        for member in tar:
+                            if member.issym() or member.islnk():
+                                continue
+                            for skill_path in copy_paths:
+                                prefix = skill_path.lstrip("/") + "/"
+                                if member.name.startswith(prefix):
+                                    member.name = member.name[len(prefix):]
+                                    resolved = os.path.normpath(
+                                        os.path.join(tmp_dir, member.name),
+                                    )
+                                    if not resolved.startswith(tmp_dir):
+                                        continue
+                                    tar.extract(member, tmp_dir)
+
+                await asyncio.to_thread(_crane_extract)
+                logger.info("Extracted skills via crane from '%s'", skills_image)
+                return
+            except Exception:
+                logger.warning(
+                    "crane skills extraction failed, trying Podman SDK",
+                    exc_info=True,
+                )
+
+        # 2. Try Podman Python SDK (needs socket)
         try:
             from podman import PodmanClient
 
@@ -655,6 +742,7 @@ def _sdk_extract() -> None:
                 pclient.close()
 
             await asyncio.to_thread(_sdk_extract)
+            logger.info("Extracted skills via Podman SDK from '%s'", skills_image)
             return
         except ImportError:
             pass
@@ -664,9 +752,8 @@ def _sdk_extract() -> None:
                 exc_info=True,
             )
 
+        # 3. Try podman CLI
         if self._podman_cli:
-            import subprocess
-
             for src_path in copy_paths:
                 await asyncio.to_thread(
                     subprocess.run,
@@ -680,11 +767,12 @@ def _sdk_extract() -> None:
                     capture_output=True,
                     timeout=120,
                 )
+            logger.info("Extracted skills via podman CLI from '%s'", skills_image)
             return
 
         raise RuntimeError(
             f"Cannot extract skills image '{skills_image}': "
-            "neither Podman SDK nor podman CLI available"
+            "none of crane, Podman SDK, or podman CLI available"
         )
 
     async def _inject_mcp_secrets(
@@ -711,8 +799,10 @@ async def _inject_mcp_secrets(
                 )
                 continue
 
+            import posixpath
+
             await self._exec_mkdir(sandbox_id, mount_path)
-            file_path = f"{mount_path}{key}"
+            file_path = posixpath.join(mount_path, key)
             await self._do_write_file(agent_name, file_path, secret_value)
             logger.info(
                 "Injected MCP secret '%s/%s' into sandbox for agent '%s'",
@@ -959,13 +1049,15 @@ async def _do_destroy(self, agent_name: str) -> None:
             except asyncio.CancelledError:
                 pass
 
-        provider_id = self._provider_ids.pop(agent_name, None)
+        provider_id = self._provider_ids.get(agent_name)
         if provider_id:
             try:
                 await self._detach_provider(sandbox_name, provider_id)
+                self._provider_ids.pop(agent_name, None)
             except Exception:
                 logger.warning(
-                    "Failed to detach provider '%s' from sandbox '%s'",
+                    "Failed to detach provider '%s' from sandbox '%s' — "
+                    "retained for retry on next destroy",
                     provider_id, sandbox_name, exc_info=True,
                 )
 
@@ -997,3 +1089,4 @@ async def _do_list_active(
             List of agent names with active sandboxes.
         """
         return list(self._sandbox_names.keys())
+

src/cloud_agents/workflow/temporal_entrypoint.py

@@ -90,11 +90,12 @@ def _create_spawner():
         from openshell import SandboxClient
 
         gateway_url = os.environ.get("OPENSHELL_GATEWAY_URL", "localhost:17670")
+        driver = os.environ.get("OPENSHELL_DRIVER", "podman")
         # Strip http:// scheme — SandboxClient uses gRPC, not HTTP
         endpoint = gateway_url.replace("http://", "").replace("https://", "")
         client = SandboxClient(endpoint=endpoint)
-        logger.info("Using OpenShellSpawner (gateway=%s)", gateway_url)
-        return OpenShellSpawner(openshell_client=client)
+        logger.info("Using OpenShellSpawner (gateway=%s, driver=%s)", gateway_url, driver)
+        return OpenShellSpawner(openshell_client=client, driver=driver)
     logger.info("No spawner configured — sandbox activity will use stub mode")
     return None