Skip to content

fix(gastown): stop containers when no agent is running - #8613

Merged
eshurakov merged 1 commit into
mainfrom
fix/gastown-stop-idle-containers
Oct 22, 2026
Merged

eshurakov merged 1 commit into
mainfrom
fix/gastown-stop-idle-containers

Conversation

@eshurakov

Copy link
Copy Markdown
Contributor

Summary

Gastown containers stayed awake in towns where nothing was happening. The TownDO alarm kept a container awake while hasActiveWork was true. That rule also counts beads the reconciler never dispatches, for example held issues, or triage requests waiting behind a stuck triage batch. Production logs showed 242 towns with a running container, and only 6 of them did any agent work in 2 hours.

  • scheduling.hasRunningAgents is the rule for "an agent process should be running": an agent that is working, or stalled for less than 30 minutes. hasActiveWork uses it too, so the query has one owner.
  • The rule now controls three alarm paths that keep a container awake:
    • ensureContainerReady: the warm-up and the /health probe on every tick.
    • stopContainerIfIdle: the idle stop.
    • refreshContainerToken: the hourly /refresh-token request. In production, a request to a stopped container starts it again.
  • Other work starts the container when it is dispatched, through prepareContainerBilling → warmUp, as it already does after sleepAfter. The alarm cadence and the reconciler still use hasActiveWork.
  • The idle stop now writes a log line, stopContainerIfIdle: stopped idle container, so you can count stops in Axiom. Before, the stop event went only to Analytics Engine.

Verification

  • New integration test test/integration/container-idle.test.ts:
    • A town with only held work stops its running container. This test fails on main.
    • A town with a working agent keeps its container.
  • Gastown unit suite: 352 passed. typecheck, lint and oxfmt pass.
  • Full integration suite: the same 52 tests fail on main and on this branch. The 2 new tests pass.
  • I checked every other alarm path that sends to the container. Mail goes only to working agents. Nudge expiry changes only the database. Re-escalation to the mayor is bounded by MAX_RE_ESCALATIONS.

Visual Changes

N/A

Reviewer Notes

  • Expected effect. About 236 idle containers stop on their first alarm tick after deploy. In Axiom (cloudflare-logpush), the number of towns with container.ready_observed should drop to about the number of towns with working agents.
  • Mayor and terminal. In a town with no working agent, a waiting mayor or an open terminal stops after the existing 5-minute mayor grace. Towns with no work already followed this rule.
  • One extra cold start. A town with an agent stalled for more than 30 minutes gets one cold start from the status poll. Then the agent comes back not_found, becomes idle, and the container stops again.
  • Not in scope:
    • Stuck towns stay on the 5 s alarm cadence. That costs Durable Object time, not containers.
    • The stuck triage batch (maybeDispatchTriageAgent: triage batch bead active or in cooldown) is a separate bug.

The alarm kept a town's container awake while hasActiveWork was true.
That includes beads the reconciler never dispatches, such as held issues
or triage requests behind a stuck batch, so those containers ran forever.

Only a working or recently stalled agent now keeps the container awake,
gets the health probe, or gets the hourly token refresh. Other work starts
the container when it is dispatched. Alarm cadence still uses
hasActiveWork.
@kilo-code-bot

kilo-code-bot Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Assumptions: [INFERENCE] I could not run the gastown typecheck/lint or integration suites in this read-only review, so correctness is assessed from source. A narrow interleaving remains where a fire-and-forget dispatch (slingBead, dispatchUnblockedBeads) is still idle in agent_metadata during the pre-status awaits in dispatchAgent while the alarm's idle-stop evaluates; the subsequent /agents/start fetch restarts the container, so the worst case is one extra cold start rather than lost work.

Files Reviewed (5 files)
  • services/gastown/src/dos/Town.do.ts
  • services/gastown/src/dos/town/scheduling.ts
  • services/gastown/src/dos/town/container-idle-stop.ts
  • services/gastown/src/dos/town/container-idle-stop.test.ts
  • services/gastown/test/integration/container-idle.test.ts

Reviewed by deepseek-flash-latest · Input: 132.6K · Output: 31.4K · Cached: 2.8M

Review guidance: REVIEW.md from base branch main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants