[igooch]

A suspended actor with a local snapshot is pinned to its node. While that node was full, its resume failed with ResourceExhausted even when other nodes had room, until atelet detached the actor on its own after --snapshot-detach-delay. Resume now detaches the actor itself, with DetachActor from #2852, and restores it on another node.

Part of #589. This covers a full node. A node that is gone, the scale-down case, is still open.

How the move works

Assignment still runs pinned to the node first. When it finds no room there, and a worker on another node has room, resume:

1. Calls DetachActor on the pinned node. atelet uploads the latest snapshot and deletes the local copies.
2. Records the detach, which marks the upload COMPLETED and clears the locality and assigned_node.
3. Assigns a worker on any node and restores from the durable snapshot.

DetachActor is synchronous. If the snapshot isn't durable yet, the resume waits for atelet to upload it.

What the restore keeps

On a worker of the same runtime, the restore keeps the snapshot's fidelity. On a worker of another runtime, a MEMORY snapshot restores at ROOTFS, as for any durable-only actor since #2880. Process memory is lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES snapshots keep their fidelity on any worker of the same sandbox class. A template can't yet refuse the downgrade; #2818 would add that.

When the actor doesn't move

• No worker anywhere has room. The resume returns ResourceExhausted as before, without calling atelet.
• The detach fails. The resume fails, and the actor stays SUSPENDED on its node, not crashed, since it can still resume there.
• The node is gone. DetachActor can't reach atelet, so the resume returns Unavailable and the actor stays pinned.

sequenceDiagram
    participant C as client
    participant A as ate-api-server
    participant L1 as atelet on node1 (full)
    participant O as object storage
    participant L2 as atelet on node2
    C->>A: ResumeActor
    A->>A: assign on node1: no room
    A->>A: node2 has room
    A->>L1: DetachActor
    L1->>O: upload latest snapshot
    L1->>L1: delete local copies
    L1-->>A: state with latest_durable only
    A->>A: mark COMPLETED, clear locality and assigned_node
    A->>A: reload restore source, assign on node2
    A->>L2: Restore(durable snapshot)
    L2->>O: download
    L2-->>A: ok
    A-->>C: RUNNING on node2

Loading

Design choices

• Any runtime counts as room.
  • Memory is an optimization for GA, so a worker that can't restore the snapshot's process memory still triggers the move, and the actor restores there at ROOTFS.
  • Assignment still prefers a worker that keeps it.
• No claim before the detach.
  • Claiming a worker first would leave the claim and the snapshot record disagreeing if ate-api-server crashed before the detach.
  • In this order, every step ends in a state resume already handles.
  • Two actors detached for one free place race for it, and the loser has paid an upload, but it ends durable-only and its next resume schedules it onto any node without another detach.

Also worth knowing

• A concurrent update fails the record write with Aborted, and the next resume detaches again; atelet returns the same state. If the node has room by then, the resume restores there from the upload instead, and the record keeps the upload PENDING, because the detach cancelled atelet's report.
• DetachActor errors map as the tag workflow maps PromoteSnapshot errors, plus atelet's Aborted passed through.
• A resume that detaches records two scheduler samples, no_capacity then assigned. A dashboard that reads no_capacity as a failed resume will count these.
• No API changes.
• TestResumeActor_RelocatesAfterDetach checked ResourceExhausted at a point where node2 already had room, which now detaches. The check moved to before node2 exists.
• The functional fake's DetachActor returns the named actor's last checkpointed snapshot, not the last of any actor.

[Zoe Zhao (zoez7)]

Could this room check move out of detachFromFullNode and into ResumeActor? Something like:

if shouldDetachFromNode(actor, err) {
	// Claim a worker on another node first.
	assigned, worker, err = w.assignWorkerOnOtherNode(...)
	if err != nil {
		return nil, false, err // still ResourceExhausted when no node has room
	}
	// Then detach from the full node.
	if err := w.detachFromFullNode(...); err != nil {
		// release the claim
		return nil, false, err
	}
}

Claiming the place before the detach closes the race noted at line 66, where two actors detached for the same place leave the loser durable-only. It also leaves detachFromFullNode with only the detach to do.

[igooch]

🤖 Before I make this change, a question for you and Michelle Au (@msau42).

DetachActor is synchronous. If the snapshot isn't durable yet, atelet uploads it before returning, and the resume waits for the whole upload. The upload runs on the RPC's context, so if the router's resume budget cancels the request, the upload is canceled too, and the next resume starts it over. With a claim taken first, every such attempt would also hold and release a worker for the length of the upload.

What if resume moves an actor only when its latest snapshot is already COMPLETED? Then DetachActor only prunes, which is fast. Claiming the worker first then costs almost nothing, and there's no upload to cancel. An actor suspended within the last --snapshot-upload-delay (5m by default) would get ResourceExhausted until atelet's background upload lands and is reported, and would move on the next resume after that.

The cost is that a recently suspended actor can't move right away. Is that acceptable for GA? If not, I'll claim first and keep the synchronous upload, and note the latency in the description.

[Michelle Au (msau42)]

1. I don't think we should wait for async upload at least with our current async period of 5 mins. We could force promote and resume faster than that.

2. Claiming before DetachActor vs. keeping the current order (anyWorkerHasRoom -> DetachActor -> ensureWorkerAssigned): I'd recommend keeping the current order (checking room first, detaching while still SUSPENDED, then claiming in ensureWorkerAssigned) rather than claiming a worker before DetachActor:

• If ateapi crashes between assignOnOtherNode and detach, then there will need to be some logic to handle the discrepency between assigned_node and snapshots.
• Keeping the actor SUSPENDED and unclaimed until DetachActor + recordDetach complete preserves clean re-entrancy at every boundary. Even if two actors race for the last free worker and both detach, the loser ends in the normal durable-only SUSPENDED state (Locality = "", AssignedNode = "", Status = COMPLETED), which is still forward progress and lets the next retry schedule it onto any node without calling DetachActor again.

[igooch]

🤖 Thanks, that settles both. I'll keep the synchronous upload and the current order: room check, detach while SUSPENDED, then claim in ensureWorkerAssigned. That's what the branch does now, so no code change.

One follow-up I'd file rather than fix here: the upload runs on the RPC's context, so a resume the router cancels also cancels its upload, and the next attempt starts over. For a snapshot that takes longer than the router's budget to upload, only atelet's background upload would ever finish it. atelet could keep a foreground detach's upload going after the caller leaves.

[Michelle Au (msau42)]

will also let Zoe review

[Zoe Zhao (zoez7)]

🤖 should-fix 🟡 – The other workflows don't retry a version conflict in place. They return Aborted and let the caller re-enter the workflow, which picks up from the stored record. This step can do the same: write the record once and return the error.

[igooch]

🤖 Done. Zoe Zhao (@zoez7) Michelle Au (@msau42), this reverses the bounded retry msau42 asked for, so flagging it for both of you.

The step now writes the record once. A conflict returns Aborted, and the next resume re-enters from the stored record, detaches again, and records the state atelet returns. A test runs that sequence. The re-read before the detach stays, for msau42's stale-actor point.

The case the retry covered: if the actor's node has room by the time the resume re-enters, the resume restores there from the upload instead of detaching again. The record then keeps the upload PENDING, since the detach cancelled atelet's report. The description says so. The window is a concurrent UpdateActor landing during the detach, which takes no lease, so it should be rare.

[igooch]

A possible follow-up that closes the remaining gap is have a foreground DetachActor mark atelet's report due instead of cancelling it, so a record write that loses a conflict is corrected by the report atelet already retries.

[Michelle Au (msau42)]

Returning ABORTED is fine to be consistent with the other workflows. The important part was the re-read.

[igooch]

🤖 Re-requesting review: the merge queue failed because #2879 landed after my last rebase and now rejects workers without a default runtime. I rebased and gave the test workers a runtime. Test fixtures only, no change to the code you approved.