[nobbs]

Summary

A stalled broker poll can spend roughly ten minutes in SDK retries. When the listener finally returns an error, session cleanup uses a background context and can block again on authentication. The pod stays alive, delaying listener replacement and the terminal error log.

 listener session
-  poll through five two-minute attempts plus backoff
+  bound the whole poll, including token refresh, to three minutes
   on polling failure
-    report the error after session cleanup returns
+    report the error immediately
-    close the session without an overall deadline
+    give session cleanup an independent ten-second deadline
+    preserve the listener error through intentional metrics shutdown

The three-minute polling budget permits one full failed attempt followed by a normal fifty-second poll. Unexpected metrics-server closure still returns an error. The cause of the initiating broker failure remains unconfirmed.

Evidence

• Before: actual actions/scaleset v0.5.0 requests with virtual time reproduce five stalled attempts taking approximately 10m15s–10m22.5s; token refresh can extend the call further. After: stalled calls terminate at three minutes across retries and refresh. Normal fifty-second polls, transient 503 recovery, recovery after one stalled attempt, and parent cancellation pass in ten repeated runs.
• Before: TestRunReturnsAfterBrokerFailureWhenCleanupAuthenticationStalls drives the real listener entrypoint against local HTTP APIs and fails because cleanup prevents exit after polling stops. After: the same isolated Linux-container regression passes 3/3, exiting after ten seconds of stalled cleanup and retaining the original polling error. Its polling deadline is shortened to four seconds; the separate SDK tests exercise the real two-minute attempt timers.
• Before: intentional metrics shutdown returns http.ErrServerClosed and logs only context canceled. After: cancellation before and after binding returns normally and retains the cancellation cause; unexpected closure remains an error.
• Passed go test -race ./cmd/ghalistener/... -count=1, go vet ./cmd/ghalistener/..., and golangci-lint run ./cmd/ghalistener/... with v2.15.0. The Linux amd64 listener builds. API-server integration coverage requiring KUBEBUILDER_ASSETS and live GitHub/cluster acceptance were not run.

The entrypoint test uses synthetic credentials and local fake GitHub/Actions and Kubernetes APIs. To reproduce it on an Apple Silicon Docker host:

ARC_TEST_DIR=$(mktemp -d)
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go test -c ./cmd/ghalistener -o "$ARC_TEST_DIR/listener.test"
docker build -t arc-listener-recovery-repro -f - "$ARC_TEST_DIR" <<'DOCKERFILE'
FROM scratch
COPY listener.test /listener.test
ENTRYPOINT ["/listener.test"]
DOCKERFILE
docker run --rm --read-only --network none --tmpfs /tmp --tmpfs /var/run/secrets/kubernetes.io/serviceaccount -e ARC_CONTAINER_REPRO=1 arc-listener-recovery-repro -test.run '^TestRunReturnsAfterBrokerFailureWhenCleanupAuthenticationStalls$' -test.v -test.timeout=45s

Use GOARCH=amd64 for an amd64 Docker host. Service-account files are created only on the disposable container's tmpfs.

Merge Danger

Door: two-way

Reverting restores the previous polling and shutdown behavior; there is no persisted API or CRD change.

Blast Radius: listeners

A prolonged broker failure can cause earlier listener replacement. The three-minute poll and ten-second cleanup limits are explicit recovery-policy choices for maintainer review. No live deployment has been performed.

[Copilot]

🔵 Needs a closer look

The polling and cleanup deadlines change production recovery policy and require maintainer confirmation.

No open issues 🎉

What changed in this PR

This PR bounds listener recovery time after stalled broker requests and prevents intentional metrics shutdown from obscuring the listener error.

Changes:

• Limit each broker poll, including retries and token refresh, to three minutes.
• Give session cleanup an independent ten-second deadline and log listener failures before cleanup.
• Preserve shutdown causes and add regression coverage for polling, cleanup, and metrics shutdown.

File	Description
cmd/​ghalistener/​metrics/​shutdown_test.go	Tests intentional shutdown, cause logging, and unexpected closure.
cmd/​ghalistener/​metrics/​metrics.go	Logs cancellation causes and suppresses expected server-closed errors.
cmd/​ghalistener/​main.go	Wires bounded polling, limits cleanup, and logs failures promptly.
cmd/​ghalistener/​main_recovery_test.go	Adds a container-only regression for stalled cleanup authentication.
cmd/​ghalistener/​broker_poll_test.go	Tests polling deadlines, retries, refresh, recovery, and cancellation.
cmd/​ghalistener/​broker_client.go	Adds the three-minute whole-poll deadline wrapper.

_{Review effort: Balanced}