[igooch]

Part of #3101.

When local snapshots fill a node's disk, the kubelet evicts atelet and rejects every replacement while the node has DiskPressure. Uploads and detaches stop, so nothing frees the space. In the runs on #3101, atelet at system-node-critical was never evicted and uploaded on schedule.

The kubelet never evicts a critical pod under node pressure, and admits one while the node is under pressure. GKE admits critical pods outside kube-system only in a namespace with a ResourceQuota scoped to their class, so the manifest adds one. Its pods: 1G limit matches GKE's own gcp-critical-pods quota, so it never blocks atelet on a large cluster. Clusters that don't restrict the critical classes, like kind, admit atelet without it.

There are two trade-offs.

• The kubelet gives node-critical pods oom_score_adj -997, so under memory pressure the kernel kills workers before atelet.
• atelet can preempt system-cluster-critical pods on a full node.

An install upgraded across this change keeps the old ate-node-critical PriorityClass. Delete it with kubectl delete priorityclass ate-node-critical.

I used an AI agent (Claude Code) to run the #3101 tests, make this change and draft this description.

[Benjamin Elder (BenTheElder)]

Agent assisted sources and auditing, review by me :-)

[Benjamin Elder (BenTheElder)]

Did we consider system-cluster-critical? The kubelet treats any priority >= 2000000000 as critical, so it still won't evict atelet or reject its replacement, but the -997 oom_score_adj only applies to the system-node-critical name, and it can't preempt cluster-critical pods.
https://github.com/kubernetes/kubernetes/blob/f28b4c9efbca5c5c0af716d9f2d5702667ee8a45/pkg/kubelet/types/pod_update.go#L188-L195

atelet and objectstore-plugin have no memory limits, and an OOM kill in a worker container takes the whole container and all of its actors with it, so I'd rather a leaking atelet be the one the kernel picks. If we stay at node-critical I think we need memory limits on both ... we might want to do that, but maybe not now.

[igooch]

We did consider it. atelet works much like the kubelet. ateapi reaches every worker through the atelet on its node and won't dial unless there is exactly one, so a node can't host actors without it. system-node-critical is the class for agents like that.

The -997 oom_score_adj is the main difference, and it's what we want when workers fill the node's memory. A worker OOM kill loses its actors' state. atelet restarting would stop activations and checkpoints on the whole node. Both classes already keep atelet safe from kubelet eviction, so the score only matters once the kernel has to pick.

I agree about the leak risk. -997 is what we want when workers fill the node, but a leaking atelet would take workers down with it. Should we add memory limits to atelet and objectstore-plugin in a follow-up, so a leak only restarts atelet? Does that need to be done as part of M3 or can it wait?

[Benjamin Elder (BenTheElder)]

[...] so a node can't host actors without it

I mean, it can't interact with the actors, but they will stay up and not lose state if it goes down.

The -997 oom_score_adj is the main difference, and it's what we want when workers fill the node's memory. A worker OOM kill loses its actors' state. atelet restarting would stop activations and checkpoints on the whole node. Both classes already keep atelet safe from kubelet eviction, so the score only matters once the kernel has to pick.

Er, but my point is that if we do have to pick, we'd rather pick atelet than ate-worker. atelet can be recovered. ate-worker loses actor data.

[...] but a leaking atelet would take workers down with it [...]

how? if atelet crashes and restarts, it should re-discover the worker pod(s), without losing any critical state.

Should we add memory limits to atelet and objectstore-plugin in a follow-up, so a leak only restarts atelet?

Limits will require tuning, but I think we can accomplish the same by using system-cluster-critical instead of node-critical. Despite the names the only difference is a small amount of priority and the oom_score_adj.

If we're at the point of evicting a pod on one of these nodes, probably atelet actually is the most viable to evict.

Another system-node-critical pod would be something like kube-proxy, without which we can't use atelet <> ate-api anyhow.

[igooch]

My main concern is disk. Only atelet frees what substrate writes under /var/lib/ate, which includes local snapshots, actor image layers, actor bundles, and actor dirs. The kubelet doesn't count that hostPath toward any pod, and its image and container GC don't reach it (in #3101 image GC freed 0 bytes). So atelet has to stay on the node, or the node can't recover from disk pressure without manual cleanup. Agreed that both classes keep atelet on the node, so I'm assuming you mean "OOM kill [the container]" not "viable to evict [the pod]"?

On memory, I don't think system-cluster-critical gets us the same as limits. The kubelet's memory-pressure eviction skips critical pods and evicts the next pod in rank under either class. The class only matters when the kernel OOMs before the kubelet acts.

Memory pressure	system-node-critical (-997)	system-cluster-critical (~995 for a 128Mi request)
atelet leaks, kubelet acts first	Workers evicted	Workers evicted
atelet leaks, kernel acts first	Workers OOM-killed	atelet OOM-killed, restarts in place
Workers spike, kernel acts first	A worker OOM-killed	atelet or objectstore-plugin likely killed first, which frees little and interrupts suspends and uploads. Then a worker.

With a memory limit, an atelet leak OOM-kills only atelet before the node runs low, under either class. I slightly prefer system-node-critical with limits on atelet and objectstore-plugin in a follow-up. Although tuning them will take some work, since some of their memory use isn't bounded yet. If you feel strongly about system-cluster-critical, I can make the switch.

[Benjamin Elder (BenTheElder)]

cc Julian Gutierrez Oschmann (@juli4n)

[ajaysundar.k (ajaysundark)]

/cc