[LIghtJUNction]

SIGTERM during worker maintenance could still claim another effect, and systemd signalled sandbox children before their parent could drain. Stop new work when shutdown begins, allow the current stage 60 seconds, and preserve cancelled jobs' durable leases rather than acknowledging them. HTTP drain is bounded to 60 seconds and all three service units allow 90 seconds including cleanup with parent-first signals.

Storage cleanup now runs even when lifespan/load fails and closes Valkey connections.

Validation: all 7 shutdown tests passed, including real OS SIGTERM during load/maintenance/job, bounded cancellation, a real HTTP socket response draining before storage close while new connections are refused, and PostgreSQL effect leases remaining running on cancellation before expiring to uncertain. Valkey integration regressions passed (10 tests including the initial shutdown cases). Ruff and diff checks passed. Broader shutdown/lease/fencing/retry/PostgreSQL regressions passed: 64 tests.

Based on the deployed money branch to keep the graceful-shutdown diff independent.

[chatgpt-codex-connector]

You have reached your Codex usage limits for security reviews. Please try again later.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[LIghtJUNction]

Update: fixed in e8a49862. I reran the same isolated real-SIGTERM harness against that head's fetched function body: stop_seen=true, claimed_state=null, execute=false, attempts=0, saved_after_stop=0. The final stop check prevents the reported race. Exact-head CI is still pending; the harness is not an end-to-end PostgreSQL or production shutdown test.

Original finding at 4cd0ce18:

P2: the shutdown guard still allows a new tool claim if SIGTERM arrives during the concurrency lookup (head 4cd0ce18).

In EffectWorker._claim, self.stopping() is checked only on transaction entry. The later await tool_concurrency(...) awaits app.contents.read_bytes(...), which yields while reading Git-backed tool content. If shutdown begins there, _claim resumes, saves the pending job as running, and returns execute=True; run_once then starts that new effect.

I reproduced this with the exact fetched _claim and tool_concurrency function bodies in an isolated local harness, using a stub transaction/content reader and real SIGTERM while the content read was paused. After the signal was handled, resuming the read produced stop_seen=true, claimed_state=running, execute=true, attempts=1. This was a focused race reproduction, not an end-to-end PostgreSQL or production shutdown test.

Please recheck the stop flag immediately before the pending-to-running transition and add a regression that signals during the concurrency content read. That should prevent a new claim while preserving the existing drain/lease handling for effects already in flight.

[LIghtJUNction]

Both graceful-shutdown commits were integrated directly into main at 97876ec, without a second merge. Released as msgctl 0.2.1 and deployed as msgd 0.2.1-20261001.2. Both production services are active, with KillMode=mixed and a 90-second systemd stop timeout. Closing this superseded PR; no additional code changes.