Skip to content

Support graceful runner drain before orchestrator pod upgrades #23

Open

Description

@j0sh

Problem

The sandbox allocator must drain an orchestrator and its app runners before an orchestrator-pod upgrade. The agreed maintenance window allows active work up to 12 hours. Today Registry.usable requires Status == "ready" and is used both for new reservations and for sessionTarget / proxyTarget. Changing a runner's status to withdraw capacity therefore also breaks new requests on existing persistent sessions and generated proxies. The trusted sandbox adapter will update status and capacity through Node's heartbeat protocol directly; no Go SDK status/capacity API is required for the first sandbox path.

Source: orchestrator/registry.go at 7a75438 and naap-ops #99.

Work

  • Separate admission/discovery readiness from service of already admitted sessions. Provide a trusted way for a dynamic runner or local manager to enter and observe draining state without giving an app container runner credentials.
  • During drain, reject new persistent reservations and single-shot calls, while existing persistent session proxies/callbacks and already in-flight single-shot requests continue within their established limits.
  • Define a configurable drain deadline that supports a 12-hour grace period. On completion or deadline, release/cancel remaining sessions and report the outcome. Explicit forced termination remains possible under authorized management.
  • Ensure paid and off-chain paths use the same admission rule; a rejected paid reservation must not charge or consume a ticket.
  • Document what happens when the orchestrator process crashes or is forcibly restarted: its session/payment control state is currently process-local, so graceful drain must finish before a planned pod replacement.

Done when

Tests demonstrate both session modes, existing proxy continuity during drain, rejection of new work, deadline cancellation, and a planned orchestrator upgrade that waits for drain completion or the 12-hour limit. The node exposes enough state for the local manager to tell whether it is safe to replace the pod. Integration tracking: naap-ops #94 and #92.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions