Skip to content

[Bug]: workspace up can't restart a Kubernetes workspace whose pod has terminated #1699

Open

Description

@wsmoak

Before submitting

  • I have searched existing issues to confirm this is not a duplicate
  • I am running the latest version of Devsy

What happened?

When a Kubernetes workspace's pod terminates (PID 1 exits, for example an OOM kill or a
crash), the pod goes Completed and the PVC stays bound. devsy workspace status
correctly reports Stopped, but devsy workspace up fails after about 2 minutes instead
of bringing the workspace back. Deleting the pod by hand makes up work again, and the
PVC contents survive.

Likely cause: FindDevContainer (pkg/driver/kubernetes/find.go) treats any pod not in
phase Running as exited, so status is right. But reconcileExistingPod
(pkg/driver/kubernetes/run.go) finds the existing pod, sees the provider options are
unchanged, logs "Pod already exists and nothing changed, skipping update" and returns
without recreating it. It never checks the pod's phase, and only deletes the pod when
options changed. Agent injection then tries to exec into the terminated container.

What did you expect to happen instead?

workspace up recreates the pod (reusing the existing PVC) and the workspace starts.

Steps to reproduce

  1. devsy provider add kubernetes
  2. devsy provider set kubernetes -o KUBERNETES_CONTEXT=docker-desktop -o KUBERNETES_NAMESPACE=devsy-p3test -o DISK_SIZE=1Gi
  3. devsy workspace up ./p3test --id p3test --provider kubernetes --ide none
  4. kubectl -n devsy-p3test exec <pod> -- kill 1 (simulates PID 1 exiting)
  5. kubectl -n devsy-p3test get pods,pvc shows the pod Completed and the PVC Bound.
  6. devsy workspace status p3test reports "state":"Stopped".
  7. Run the workspace up command from step 3 again. It fails.

devcontainer.json

{ "image": "alpine:3.20" }

Error output / logs

platform-native agent delivery failed: deliver agent (post-start): in-container download failed:
  download in container: Internal error occurred: unable to upgrade connection: container not found ("devsy")
exec-stream delivery failed: Internal error occurred: unable to upgrade connection: container not found ("devsy")
legacy agent injection failed: inject agent: [inject] read ping: EOF

# afterwards
NAME                     READY   STATUS      RESTARTS   AGE
devsy-default-p3-30abe   0/1     Completed   0          2m39s

How often does this happen?

Every time

Operating system

macOS

Linux distribution (if applicable)

N/A (the pod image is alpine:3.20)

Architecture

ARM64

Desktop app or CLI?

CLI only

Devsy version

v1.24.0

Devsy provider

Kubernetes

Provider version

v1.0.0

Screenshots

No response

Anything else?

Suggested fix: in reconcileExistingPod, treat a pod in phase Succeeded or Failed
as needing recreation. Delete it as the options-changed path does (waitPodDeleted) and
fall through to creating a new one.

Any way PID 1 can end (OOM kill, crash, a node drain with restartPolicy: Never) leaves the workspace unrecoverable through Devsy until the pod is deleted.

This also blocks one possible fix for issue #1698 (inactivity timeout), which would otherwise end the pod's main process.

Workaround:

kubectl -n <namespace> delete pod <pod>
devsy workspace up ...

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions