Repository navigation
Support sandbox platforms running on Substrate #553
Description
Background
One of the potential consumers today with large scale are evaluation and RL frameworks like Harbor and NeMo Gym. These run agents inside execution sandbox environments, and already integrate with several of these platforms: Harbor ships backends for roughly twenty-five providers, e2b and OpenSandbox among them, each implementing a single interface of start, stop, exec, and file upload and download.
Over the last couple of days I looked at what it would take to run a Harbor or NeMo Gym style platform on Substrate. This issue writes up what those platforms need from a compute backend, and the gaps I hit.
What these platforms are
A sandbox platform (like e2b, OpenShell, OpenSandbox) gives an SDK caller a fresh enforced environment: filesystem restrictions, controlled egress, credentials injection. Over that it exposes an API: run a command, stream output, download and upload files, open a shell.
These platforms let users bring their own images, and inject the sandbox binary at execution time without requiring the user's image to be rebuilt. This aligns with Harbor-style systems, which need to run many image types (100+), one per benchmark environment.
A few examples I looked on:
| Platform | Isolation | Source |
|---|---|---|
| Vercel Sandbox | Firecracker microVM | Closed |
| e2b | Firecracker microVM | Apache 2.0 |
| NVIDIA OpenShell | OCI container | Apache 2.0 |
| OpenSandbox (Alibaba) | OCI container | Apache 2.0 |
I would focus on the two container-based fully open-sourced platforms, OpenShell and OpenSandbox.
There are certainly more than these, and this list is not meant to be exhaustive. Happy to hear from the community about others worth considering.
Both already support Kubernetes as a pluggable compute backend. Adding Substrate alongside it is the goal this issue is framing.
How they work on Kubernetes
Both follow the same shape: the user brings an arbitrary OCI image, and the platform delivers its own binary into that image at pod-creation time.
OpenShell
A single supervisor binary runs as the container's first process and spawns the agent as a child. Everything the platform enforces lives in that supervisor process: policy decisions, a loopback proxy for egress, credential injection, etc. Because the supervisor owns the child, it applies filesystem and syscall restrictions before the agent ever starts.
Supervisor binary delivery: on modern k8s clusters the supervisor's own OCI image is mounted into the pod as a read-only image volume; on older ones an init container copies the binary into a shared volume. Either way the pod's container command is overridden to point at the delivered binary.
The supervisor container runs as root and is granted capabilities like NET_ADMIN and NET_RAW for network setup, and CHOWN to prepare paths for the sandbox user. The supervisor drops privileges for the child.
The supervisor holds an outbound connection to the control plane and receives exec, file, and policy operations over it.
OpenSandbox
A separate daemon, execd, implements the API surface directly: command execution with streaming output, interactive shell sessions over an ssh-like channel, and file operations. It is delivered by an init container that copies it out of a dedicated image into a shared volume, and the container command is overridden to a bootstrap script that starts execd, then launches the user's workload as a child.
Egress is handled by a second container. A sidecar carrying the network policy joins the pod, holds NET_ADMIN, and installs the rules. The workload container has that same capability explicitly dropped. Since all containers in a pod share one network namespace, the workload's traffic necessarily traverses the sidecar's rules, and the workload has no privilege to inspect or remove them.
The pattern
Reduced to essentials, both need two things from their compute backend:
-
Deliver a binary into an arbitrary user image and run it as the container process
-
Give enforcement components privileges the workload does not have
The API for exec, upload and download is not a separate requirement. The injected binary implements it. This is aligned with the comment on #185, which separates exec-into-actor for debugging from the API an execution environment offers, and suggests the latter need not live in Substrate itself.
Gaps identified
1. No way to deliver a binary into an actor
ActorTemplate containers take an image, command, args, env, and volume mounts. Volumes exist, but every source starts empty. Nothing carries content in. The only way to get a binary into an actor today is to have it in the image, which means asking users to rebuild theirs.
2. No per-container capabilities
Every container in an actor receives the same fixed capability set. OpenSandbox's egress model depends on the asymmetry between a privileged sidecar and an unprivileged workload, and OpenShell's supervisor needs elevated privileges to install its controls.
Options to close each gap
Delivering the binary
-
An image volume: a volume source naming an OCI image, mounted read-only into the actor. Something similar to the Kubernetes image volume model.
-
An init container: a phase that runs before the actor's containers, so a user can copy content into a shared volume.
Capabilities
- Per-container
securityContextwith capability add onActorTemplate.
Related work in Substrate
-
#185 exec channel into a running actor
-
#264 an API to push files into an actor
-
#126 transparent egress policy and credential injection
Looking for feedback on the gaps and the options above. Happy to open a separate issue per option we agree on.
Activity
ronlv10 commentedon Jul 27, 2026
I know that Julian Gutierrez Oschmann (@juli4n) worked on how to build "E2B-like" code execution sandbox solution in substrate
and also Tim Hockin (@thockin) and rogeroger (@rogeroger-yu) had discussion around this in #185
happy to hear your thoughts!
BenTheElder commentedon Jul 27, 2026
That seems ~fine. It will add a bit of complexity but we can add another RO mount. The behavior is pretty predictable, and there are probably other use-cases.
We'd also need something like an emptyDir scratch dir between the init container and the working containers
This potentially adds a lot of lifecycle complexity / confusion we should be careful to communicate clearly about (e.g. init probably shouldn't run again on resume but you might assume or want that).
I do think we need this eventually, but I think we should be careful about how we expand this API, communicate defaults, etc.
There's also some awkward things where these have evolved a little in the ecosystem over time ...
BenTheElder commentedon Jul 27, 2026
This smells like two feature requests packed into one issue, support for something e2b like is sort of it's own thing that may or may not require these specific features depending on how we implement it. There are other ways you could solve these problems (e.g. a separate component could patch the binary into an additional image layer as an image pull proxy).
ronlv10 commentedon Jul 28, 2026
Just to make sure I was clear - I think one of them should be sufficient for now. I thought that the init container would be a more powerful tool that users could leverage for more use cases, but it sounds like you think an image volume makes more sense for now?
I agree. I wanted to get feedback on:
BenTheElder commentedon Aug 5, 2026
My take:
I do. But I can't exactly speak for the project on that overall.
Personally I'd rather add read-only image volumes for binary injection than add slow and complex startup ordering (init containers) with confusing semantics. I think that's going to be easier to build performantly and with clear behavior.
zmberg commentedon Aug 6, 2026
Thanks Ron Lev (@ronlv10) for the thorough write-up. This resonates strongly with what we're building.
I'm from the Alibaba Cloud Container Service team. We develop and maintain OpenKruise Agents — a CNCF incubation-stage project under the OpenKruise family. It is a native Kubernetes sandbox orchestration system that manages the full lifecycle of sandbox instances (create, checkpoint-based pause/resume, in-place upgrade, reclaim) on top of OCI containers, with first-class support for Virtual Kubelet and ECI integration.
Interest in Substrate: We are internally exploring Substrate as an additional backend beyond containers. Specifically, we plan to integrate directly with Substrate's interfaces at the sandbox-manager (E2B service layer) level, rather than replacing the underlying runtime. The two gaps you identified resonate strongly with us:
The RL scenario vs. actorTemplate model dilemma. In our RL scenarios, customers have hundreds of thousands of images, each containing different SWE-bench examples. These sandboxes have two main requirements:
Currently, Substrate requires creating an actorTemplate per image, which means users would need to create hundreds of thousands of actorTemplates — not a friendly experience. Moreover, for RL scenarios, what customers truly need is fast startup speed + filesystem fork capability, not full VM-level checkpoint/restore. We haven't yet figured out how to reconcile this massive-image + high-throughput + filesystem-fork pattern with Substrate's current actorTemplate model. We'd love to hear the community's thoughts on this.
We'd love to have more communication with the community to facilitate better integration and iteration. We're also very willing to contribute our experience and code during the actual implementation process.
dberkov commentedon Aug 7, 2026
berg (@zmberg), you raised an interesting topic regarding hundreds of thousands of actorTemplates in your (#553 (comment)). Since the current issue focuses on injecting binaries into the actor, could you please open a separate issue so we can dedicate a discussion to it?
omeryahud commentedon Aug 9, 2026
Hi berg (@zmberg) !
This sentence is interesting to me. Do you have an alternative method in mind? For the RL use case, I can see how this could be a bottleneck.
BenTheElder commentedon Aug 13, 2026
FWIW: I think we can take a stronger stance that yes: This should be supported, based on recent conversations we've had.
So I think the follow-up now is to land the split out feature requests. Feel free to ping me in slack for any related PRs, thanks!
thockin commentedon Aug 19, 2026
Should we consider allowing the Actor to "augment" the template? If those 100K templates are all the same except for a few things (e.g. the OCI image), would that make a better UX?
BenTheElder commentedon Aug 25, 2026
So we landed the features raised here, but I think next up is a POC of integration to see what else is missing?
ronlv10 commentedon Aug 31, 2026
I think this is relevant for RL, but also for any e2b-like environment that supports bootstrapping a sandbox from a user-submitted OCI image. For example, in OpenShell:
Today, to implement this, the platform needs to do the following (if there is no existing actor template for it):
Create an actor template with the OCI image -> wait for it to be ready (wait for the golden snapshot) -> create an actor -> resume the actor.
Tim Hockin (@thockin) - I like the idea of using an existing template and just overriding fields. Another option is to allow CreateActor to create an actor on-demand without an actor template (though this is probably a bigger change to the existing API).
ronlv10 commentedon Aug 31, 2026
We have started to work on a POC for Openshell
iracic82 commentedon Oct 22, 2026
On "next up is a POC of integration to see what else is missing": I tried OpenShell's sandbox runtime inside Substrate actors on kind (OpenShell 0.1.5). It works on the micro-VM class and not on gVisor, and the one Substrate-side gap I hit is already covered by #1904.
gVisor: OpenShell needs Landlock ABI 3+ and seccomp user notification. runsc
release-20261012.0has neither (ENOSYS / EINVAL), so it fails closed. Expected.micro-VM:
openshell-sandbox capability-probe-launchreturns"qualified": true(Landlock ABI 7 on the Kata 6.18 guest), fresh and after suspend/resume. On main it neededip_unprivileged_port_start=0set from inside the guest. With #1904 merged onto main it needs nothing, details on that PR.Two smaller things from the same run:
run-microvm-demo-kind.shstill tells you to port-forwardatenet-routeron port 80, but the router only serves mTLS on 443 now. I used a client pod with apodidentity.podcert.ate.dev/identitypod certificate.drop: [ALL]plusSETPCAP,SETUID,SETGID. Might be worth a line in the docs.