Repository navigation
[Bug]: workspace up can't restart a Kubernetes workspace whose pod has terminated #1699
Description
Before submitting
- I have searched existing issues to confirm this is not a duplicate
- I am running the latest version of Devsy
What happened?
When a Kubernetes workspace's pod terminates (PID 1 exits, for example an OOM kill or a
crash), the pod goes Completed and the PVC stays bound. devsy workspace status
correctly reports Stopped, but devsy workspace up fails after about 2 minutes instead
of bringing the workspace back. Deleting the pod by hand makes up work again, and the
PVC contents survive.
Likely cause: FindDevContainer (pkg/driver/kubernetes/find.go) treats any pod not in
phase Running as exited, so status is right. But reconcileExistingPod
(pkg/driver/kubernetes/run.go) finds the existing pod, sees the provider options are
unchanged, logs "Pod already exists and nothing changed, skipping update" and returns
without recreating it. It never checks the pod's phase, and only deletes the pod when
options changed. Agent injection then tries to exec into the terminated container.
What did you expect to happen instead?
workspace up recreates the pod (reusing the existing PVC) and the workspace starts.
Steps to reproduce
devsy provider add kubernetesdevsy provider set kubernetes -o KUBERNETES_CONTEXT=docker-desktop -o KUBERNETES_NAMESPACE=devsy-p3test -o DISK_SIZE=1Gidevsy workspace up ./p3test --id p3test --provider kubernetes --ide nonekubectl -n devsy-p3test exec <pod> -- kill 1(simulates PID 1 exiting)kubectl -n devsy-p3test get pods,pvcshows the podCompletedand the PVCBound.devsy workspace status p3testreports"state":"Stopped".- Run the
workspace upcommand from step 3 again. It fails.
devcontainer.json
{ "image": "alpine:3.20" }Error output / logs
platform-native agent delivery failed: deliver agent (post-start): in-container download failed:
download in container: Internal error occurred: unable to upgrade connection: container not found ("devsy")
exec-stream delivery failed: Internal error occurred: unable to upgrade connection: container not found ("devsy")
legacy agent injection failed: inject agent: [inject] read ping: EOF
# afterwards
NAME READY STATUS RESTARTS AGE
devsy-default-p3-30abe 0/1 Completed 0 2m39sHow often does this happen?
Every time
Operating system
macOS
Linux distribution (if applicable)
N/A (the pod image is alpine:3.20)
Architecture
ARM64
Desktop app or CLI?
CLI only
Devsy version
v1.24.0
Devsy provider
Kubernetes
Provider version
v1.0.0
Screenshots
No response
Anything else?
Suggested fix: in reconcileExistingPod, treat a pod in phase Succeeded or Failed
as needing recreation. Delete it as the options-changed path does (waitPodDeleted) and
fall through to creating a new one.
Any way PID 1 can end (OOM kill, crash, a node drain with restartPolicy: Never) leaves the workspace unrecoverable through Devsy until the pod is deleted.
This also blocks one possible fix for issue #1698 (inactivity timeout), which would otherwise end the pod's main process.
Workaround:
kubectl -n <namespace> delete pod <pod>
devsy workspace up ...
Activity