You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support graceful runner drain before orchestrator pod upgrades #23
The sandbox allocator must drain an orchestrator and its app runners before an orchestrator-pod upgrade. The agreed maintenance window allows active work up to 12 hours. Today Registry.usable requires Status == "ready" and is used both for new reservations and for sessionTarget / proxyTarget. Changing a runner's status to withdraw capacity therefore also breaks new requests on existing persistent sessions and generated proxies. The trusted sandbox adapter will update status and capacity through Node's heartbeat protocol directly; no Go SDK status/capacity API is required for the first sandbox path.
Separate admission/discovery readiness from service of already admitted sessions. Provide a trusted way for a dynamic runner or local manager to enter and observe draining state without giving an app container runner credentials.
During drain, reject new persistent reservations and single-shot calls, while existing persistent session proxies/callbacks and already in-flight single-shot requests continue within their established limits.
Define a configurable drain deadline that supports a 12-hour grace period. On completion or deadline, release/cancel remaining sessions and report the outcome. Explicit forced termination remains possible under authorized management.
Ensure paid and off-chain paths use the same admission rule; a rejected paid reservation must not charge or consume a ticket.
Document what happens when the orchestrator process crashes or is forcibly restarted: its session/payment control state is currently process-local, so graceful drain must finish before a planned pod replacement.
Done when
Tests demonstrate both session modes, existing proxy continuity during drain, rejection of new work, deadline cancellation, and a planned orchestrator upgrade that waits for drain completion or the 12-hour limit. The node exposes enough state for the local manager to tell whether it is safe to replace the pod. Integration tracking: naap-ops #94 and #92.
Activity