[grs]

Summary

Add support for updating a stopped sandbox’s image and configuration while preserving its identity and persistent storage. A new image may require different environment variables or startup commands, so these fields can be updated together before the next start. Labels and annotations are included for completeness and consistency with the sandbox resource.

Related Issue

Closes #4623

Changes

• Add SandboxUpdates.UpdateSandbox with optional image, command, environment, label, and annotation updates.
• Implement storage-preserving updates for the Kubernetes driver.
• Require the sandbox to be stopped and leave it stopped after updating.
• Preserve omitted fields, replace supplied commands and maps, and protect system metadata.
• Use conditional resource versions and durable operation tracking to handle conflicts and interrupted updates.
• Gate updates behind explicit activation after a compatible gateway rollout.
• Add raw SDK access, documentation, and unit and Kubernetes E2E coverage.

Release Notes

Kubernetes sandboxes can now update their image, command, environment, and metadata without recreating the sandbox or losing persistent storage. Updates require operator activation and a stopped sandbox.

Testing

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Relevant crate READMEs and published documentation updated

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[sjenning]

/ok to test 42ae543

[github-actions]

Label test:e2e-kubernetes applied for 42ae543. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway, sandbox, and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

[github-actions]

Label test:e2e applied for 42ae543. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[mrunalp]

Review summary

This adds a new SandboxUpdates.UpdateSandbox RPC. It changes a stopped sandbox's image, command, environment, labels and annotations, and keeps its ID, custom resource and PVC. The sandbox stays stopped. Only the Kubernetes driver implements it, and it is off by default behind enable_sandbox_updates.

The core mechanics look sound:

• Accept and complete are each a single transaction, with resource-version checks.
• The stopped state is re-checked inside the transaction.
• request_id payloads are fingerprinted.
• Private state is keyed by sandbox UUID and removed on delete.
• validate_only writes nothing.
• PVC and volume templates are untouched.
• The supervisor command and OPENSHELL_* env are protected.
• Older drivers and gateways still negotiate correctly.

The concerns below are mostly about recovery and operability.

Concerns

1. A sandbox can become impossible to start, stop or delete.

Before this PR, an operator could always delete a sandbox once its driver work had settled. With this PR, an accepted update blocks Start, Stop, Delete and the GC/TTL sweeps (compute/mod.rs:1900, 2103, 3097, 3253). The only way out is retrying UpdateSandbox with exactly the same fields, and nothing clears the intent in the background.

That retry is refused before the pending path is reached (compute/image_update.rs:58-87) in any of these cases:

• the operator turns the flag back off;
• the Postgres check in (3) fails;
• the driver stops advertising the capability.

A driver error that repeats on every retry has the same effect, for example a precondition failure after an out-of-band change to the custom resource. The only fix left is editing the database by hand.

Suggestion: let completion of an existing intent skip the activation gates, and add an abort or force-delete path.

2. Recovery needs state the client can't see.

The retry must match the stored intent's field selection, including whether each optional field was present or absent (image_update.rs:124-133). That intent is private, and the accept step has already written the target values into the public spec, so a different client, or one that lost the original request, can't rebuild it.

For example, sending image + command with identical values when the original sent only image fails with "another sandbox update is pending". Reusing the original request_id returns REQUEST_OUTCOME_UNCERTAIN. At minimum, the error should include the pending selection.

3. The Postgres activation check blocks updates in common deployments.

persistence/image_activation.rs:41: any client backend on the same database without the openshell-image-update-v2 marker refuses every update, including pending retries. That covers psql sessions, pg_dump, postgres_exporter and managed-DB agents. Two more ways the check misfires:

• Rows with backend_type IS NULL count against it. Those are probably the sessions the gateway role can't fully see, which is common on managed Postgres without pg_read_all_stats.
• The marker is set with set_config after connecting (postgres.rs:33), so each fresh gateway connection briefly looks unmarked.

The marker also unconditionally overwrites any application_name the operator set in the DSN, even with updates disabled, which affects their database monitoring.

Combined with (1), a stray admin session can freeze a pending sandbox. Suggestion: set application_name through PgConnectOptions, and make the check advisory or run it only at activation.

4. Updates are refused (HTTP 403) in the documented split gateway/workspace Helm setup.

The driver now lists every Pod in the namespace (driver-kubernetes/src/image_update.rs:235-245). The openshell-workspace Role grants pods: [get] only (deploy/helm/openshell-workspace/templates/role.yaml:39-45). In shared mode with workspaceResources.enabled=false, every UpdateSandbox call fails as an internal error, the validate_only preflight included. CI uses the default chart, so it doesn't catch this.

Listing by label plus get_opt on recorded pod names would avoid needing list. It would also cut the full-namespace scan, which can hit the 30s API timeout in large shared namespaces.

5. Annotation updates can turn an ephemeral sandbox persistent, and some user keys get locked.

Before this PR, user metadata was only set at create. Now:

• An annotations update that leaves out openshell.nvidia.com/retention=ephemeral drops it, because that key isn't treated as system metadata (core/src/sandbox_update.rs:10, read at compute/mod.rs:5698). The sandbox silently stops being auto-deleted.
• Create only reserves internal.openshell.ai/ (grpc/validation.rs:485), but update protects all of *.openshell.ai/ and *.agents.x-k8s.io/. User keys created under those domains can never be changed or removed.

6. Label-only or annotation-only updates go through the driver and the lock.

These are gateway-only metadata and don't touch pod metadata, yet they still require the configuration capability and a no-op driver RPC, and they set the pending flag (image_update.rs:46-54). So a metadata-only edit returns UNIMPLEMENTED on an image-only driver, and it's exposed to (1). It could be a plain conditional write on the sandbox record.

7. Docs and code describe a "protocol 2" that replaces something never released.

main has no image-update code at all, yet the docs, skills and server README talk about "protocol 2", "gateways that previously supported image-only updates" and "legacy image-only intents". Operators will look for a v1 that doesn't exist, and the compatibility branches are dead code. Collapsing to a single protocol and a single driver RPC (the configuration request already embeds the image request) would simplify both.

8. Bundled lifecycle change.

Stop on a timed-out Error sandbox whose cleanup has finished now writes Stopped directly, without a driver stop. Driver Deleted events no longer prune timed-out Error/Stopped records (compute/mod.rs:1916-1969, 5734-5741). This may be intended for image repair, but it's a separate behavior change and deserves its own description or PR. The heading rewrite in configuration.mdx:1507 is unrelated too.

9. E2E cost.

Adding openshell-sdk and openshell-bootstrap to the e2e-kubernetes feature pulls about 900 lockfile lines (proto codegen, OIDC, tunnel deps) into every Kubernetes e2e binary. The test also re-implements gateway discovery and mTLS loading that tests/testkit already provides, so it would fit better as a #[testkit::test] suite. It pulls images from docker.io, so Docker Hub rate limits apply.

Smaller items

• Two HA replicas retrying at the same time: the one that loses the row lock reports a conflict even though the update succeeded (sandbox_image.rs:228).
• The UNIMPLEMENTED message says "image updates" even when the missing capability is the configuration one (image_update.rs:79).
• UpdateSandboxConfiguration skips the RPC tracer (driver-kubernetes/src/grpc.rs:333).
• Label maps are checked against the annotation entry limit (grpc/sandbox.rs:405).
• metadata.uid in the Kubernetes merge patch isn't a real precondition; resourceVersion does the fencing, so the README's "UID-fenced" claim overstates it.
• The protocol 2 is hard-coded in two places (gateway_members.rs, image_activation.rs:98).

[mrunalp]

Re-review of 6520f083f

Thanks for the quick turnaround. The blocking issues from the first review are resolved. What's left is mostly about the cost of the recovery sweep and the scope of the bundled changes.

Status of earlier concerns

#	Concern	Status
1	A sandbox could get stuck (no start, stop or delete)	Fixed. There are three ways out now: a background recovery sweep, an empty "resume" request that skips the activation gates, and Delete, which now replaces a pending update.
2	Recovery needed private state the client can't see	Fixed. A request with no fields resumes the stored intent, and the error messages say how.
3	Postgres pg_stat_activity check	Fixed. The check is removed, the DSN application_name is preserved, and a test covers both.
4	403 in the split gateway/workspace Helm setup	Fixed. The workspace Role now matches the main chart's permissions, and CI's default Kubernetes e2e lane installs that split setup.
5	Retention annotation dropped; user keys locked	Fixed. Protection is now an explicit list of gateway-owned keys plus internal.openshell.ai/.
6	Label/annotation updates went through the driver	Fixed. They're now a single conditional write on the sandbox record, with no driver call or lock.
7	"Protocol 2" and two driver RPCs	Fixed. One capability and one RPC, protocol 1 as a shared constant, docs cleaned up.
8	Bundled lifecycle change	Still there, and there's more of it (see below).
9	E2E cost	Fixed. The test is now a testkit suite pulling from mirror.gcr.io.
—	Smaller items	Fixed: the HA double-completion error (via the completed receipt), the UNIMPLEMENTED message, RPC tracing, label limits, and the README's UID-fence wording.

Remaining concerns

1. The recovery sweep scans every sandbox, every minute, on every replica, even when the feature is off.

reconcile_loop calls recover_pending_sandbox_updates() on each 60s tick. That function lists every sandbox record and then reads each sandbox plus its private state (compute/image_update.rs:351-400, compute/mod.rs:4859). Nothing skips it when enable_sandbox_updates is false or no driver supports updates.

On a large HA deployment that's one full scan plus 2N point reads per minute per replica, for a feature that's almost never pending. Listing only the private state rows that have a pending intent, or returning early when there is no update driver, would remove nearly all of that cost.

2. More lifecycle changes are bundled in.

Before this PR, sandbox stop on a Completed sandbox, or one whose main process failed, returned immediately with nothing to do. With this commit it moves through Stopping, calls the driver's stop, and keeps the exit code (compute/mod.rs:440-476, 8061). The stop-after-timeout change from the first review is also still there.

Both are plausibly needed so these sandboxes can reach Stopped and be updated. But they change what users see for all sandboxes, so they belong in the PR description and release notes, or in a separate PR.

3. Deleting sandboxes no longer wait for pending driver work, based on phase alone.

Five sites now skip the pending-driver-operation guard whenever the phase is Deleting (compute/mod.rs:3651, 4449, 6182, 6224, 6555). Today only the update path can reach Deleting with that flag set, because Delete still requires settled operations otherwise. A future path that reaches Deleting with a create or start still in flight would silently lose that protection.

Limiting the exemption to the update's own operation ID would keep the protection intact.

4. A driver error that repeats is retried forever.

Recovery retries every minute and logs a warning for each sandbox each time, with no backoff. Delete is now the way out, so this is noise rather than a stuck sandbox. A backoff, or a status condition users can see, would make it easier to diagnose.

5. The main Kubernetes e2e lane changed topology.

The default e2e:kubernetes lane now always runs with the workspace chart installed separately (OPENSHELL_E2E_KUBE_SPLIT_WORKSPACE). That lane no longer exercises the combined chart's own Role. Was that intended, or should one lane keep the default setup?

Nit: the new test hand-builds UUIDs with rand and hex (tests/suites/features/kubernetes-updates/tests/update.rs:32).