Repository navigation
Retry transient Forbidden errors on ephemeral runner pod creation - #4719
Open
JoaoManierii wants to merge 8 commits into
Open
JoaoManierii wants to merge 8 commits into
JoaoManierii wants to merge 8 commits into
Conversation
When the API server fails a backend call while admitting a runner pod, it answers 403 with the gRPC status in the message, for example "pods ... is forbidden: rpc error: code = ResourceExhausted desc = stream terminated by RST_STREAM with error code: ENHANCE_YOUR_CALM". The IsForbidden branch only retried "exceeded quota:". Any other Forbidden marked the EphemeralRunner Failed on the first attempt. A Failed runner is counted by scaleTotal() and is not removed until the set is cleaned up, so it keeps a replica slot and the queued job gets no runner. Treat a Forbidden message carrying a ResourceExhausted, Unavailable or DeadlineExceeded gRPC status like the quota case: requeue after 30s while the runner is younger than 10 minutes, then delete it so it is re-created. Other Forbidden errors still mark the runner Failed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JoaoManierii
requested review from
a team,
Steve-Glass,
mumoshu,
nikola-jokic,
rentziass and
toast-gear
as code owners
October 18, 2026 23:49
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This reopens #4711 by @fedor-rubalskiy with his commit unchanged. He closed it to follow the contributing guide and filed #4712 first. Neither has had a maintainer response yet, and we hit the same bug in production, so we would like to get the fix reviewed. @fedor-rubalskiy, if you would rather reopen yours, I'll close this one.
Fixes #4712
Fixes #4718
#4718 is the same failure seen from the scale set side: a transient 403 left a Failed runner that held its slot for hours. This change stops the transient 403 from producing that Failed runner. Points 2 and 3 in #4718 (a Failed runner holding a slot for other reasons, and surfacing it) can be followed up separately if you want them.
What it does
When the API server rejects a runner pod with a Forbidden whose message carries a gRPC status of
ResourceExhausted,UnavailableorDeadlineExceeded, a backend of the API server failed. The pod itself was not rejected. On GKE this shows up as throttling withENHANCE_YOUR_CALM. Those errors now get the handling thatexceeded quota:already has since #4305:Other Forbidden errors, such as a missing PriorityClass, an admission rejection or
PermissionDenied, still mark the runnerFailedwith reasonInvalidPod.Why
Today any Forbidden other than quota marks the runner
Failedon the first attempt.scaleTotal()keeps counting it, so the scale set loses a slot until someone deletes the runner by hand.On controller 0.13.1 on GKE, four runners in four scale sets failed with
ENHANCE_YOUR_CALMwithin one day, while the pod template was fine. On a set with maxRunners 4, the Failed runner held its slot for about 11.7 hours. Jobs waited with no runner for more than 2 hours in total. Logs and details are in #4718.Testing
On this branch, Linux, go 1.27.1:
go test -run TestReconcileForbiddenPodCreation -v ./controllers/actions.github.com/: all 8 cases pass.isTransientForbiddenemptied, the three transient cases fail and the other five pass.go vet ./controllers/actions.github.com/andgofmt -l controllers/actions.github.com/are clean.go test -short ./controllers/actions.github.com/...with envtest (Kubernetes 1.37) and Helm v4.2.2 passes, including the Ginkgo suite.The testing notes in #4711 still apply: the test uses the fake client with an interceptor, because envtest cannot produce this error.
We can also run a controller build with this change in our cluster and report back.
🤖 Generated with Claude Code