Repository navigation
Proposal: new vm module primitives & loader API for ESM customization #62720
Description
This proposes new vm module primitives that aim to replace the existing vm.SourceTextModule and provide a high-level loader API for ESM (specifically SourceTextModule) loading customization.
Consider this a very early draft for discussion. This is mostly to investigate whether a new design can addresses the existing issues. I am not 100% it's implementable yet, especially the loader customization part, but it's better to discuss what design would help developers before we think about what is easier to implement.
There'll be a session about this new design at the collaboration summit too openjs-foundation/summit#482
Background
The vm module APIs have been behind --experimental-vm-modules for a long time. There was a tracking issue about their stabilization and accumulated several issues over the years.
The meaningful current users are concentrated in test tooling (Jest, Vitest). Recently there has been renewed momentum to look into what it takes to bring the API out of the experimental status. @legendecas and I have been adding some non-breaking changes e.g. linkRequests(), instantiate(), moduleRequests, hasTopLevelAwait(), hasAsyncGraph(), conditionally synchronous evaluate() so that it now provides capabilities required to implement something similar to how ESM is handled in the built-in loader - specifically, the linking process can be driven by those who construct the SourceTextModule instead of being driven from a link method with callbacks, and it can be conditionally synchronous as the spec allows. But it has become awkward to keep piling methods on the existing classes to do things differently without breaking the API.
There are still a few issues with the current design:
- The
importModuleDynamicallyandinitializeImportMetacallbacks are passed as options to the module constructors, requiring careful memory management to avoid leaks when callbacks capture over referrers (#33439, #50113, #59118), or use-after-free when remote code callsimport()indirectly via a closure (#47096). We've addressed for the main context with a very intricate memory management scheme, but for new contexts it's uncertain. The current implementation works but is not very GC-efficient, and this issue may not have existed in the first place if the callbacks are managed differently. evaluate()mixes loader errors (status errors, timeout) with module evaluation errors in a single promise rejection, making it hard to handle them differently (#60242).- Many users have expressed that the current API still requires too much plumbing if they only need partial customization. (#31234, #35848, #43899).
It seems better to consolidate the changes into a new API rather than continuing to pile onto the existing interface, while we can still steer the design during the experimental phase.
A draft for a new API
The new API can live in a 'vm/modules' module and exports SourceTextModule, SyntheticModule, and SourceTextModuleLoader.
Core idea
Provide a SourceTextModuleLoader abstraction that users can subclass and override high-level processes they want to customize (#43899). This loader can be used standalone, or registered in a given context:
dynamicImport(request, context, parent)importMeta(meta, context, parent)getModules(requests, context, parent)(resolving and linking a batch of modules for a set of import requests)
In this model, SourceTextModule and SyntheticModule are primitives of the loader.
When registered for a given context, the loader is responsible for resolving ESM requests, handling dynamic import(), and initializing import.meta. Instead of one callback per module, there is one loader per context.
Note that this means once there's a loader registered, the internals have to wrap several constructs to be a publically accessible shape e.g. wrapping actual context into vm Context. So it can take a bit of refactoring and adds overhead, but should be managable.
This is separate from module.registerHooks() which installs low-level hooks into built-in resolution/loading process for all types of modules - consider the hooks run underneath super.getModules()/super.dynamicImport() etc. as shown below, so they operate in a different layer. SourceTextModuleLoader is a higher level customization, and as the name implies, is specifically for handling SourceTextModules - SyntheticModule does not have import() or import.meta or load other modules, so they are not applicable to the customization.
1. Full customization
import {
SourceTextModule,
SyntheticModule,
SourceTextModuleLoader,
} from 'vm/modules';
// Names starting with `helper` are only for this example,
// they are not part of the API.
class ExampleModuleLoader extends SourceTextModuleLoader {
#sourceStore = new Map();
#moduleCache = new Map();
// Invoked by Node.js to perform dynamic import(), overrides default.
dynamicImport(request, context, parent) {
const mod = this.helperGetSingleModule(request, context, parent);
mod.evaluate();
// topLevelCapability does not need to fulfill here.
// That promise will just be forwarded to the code actually
// awaiting the dynamic import().
return mod.namespace();
}
// Invoked by Node.js to initialize import.meta, overrides the default.
importMeta(meta, context, parent) {
// Objects attached to import.meta should typically be created
// in the target context via vm.runInContext() or similar.
meta.identifier = parent.identifier;
}
// Invoked by Node.js to resolve and compile modules for a set of
// import requests. The returned array must correspond 1:1 to the
// requests array.
// TODO: it's unclear whether getModules or getModule would work better
// for the internal way of resolving modules - supposedly batching helps
// with performance, but it also leaks internal ordering choices.
getModules(requests, context, parent) {
const result = [];
for (const { specifier } of requests) {
// Consult the custom cache first.
const cached = this.#moduleCache.get(specifier);
if (cached) { result.push(cached); continue; }
// Use the custom source store.
const source = this.#sourceStore.get(specifier);
if (!source) {
throw new Error('module not found');
}
const identifier = specifier;
const mod = new SourceTextModule(source, { identifier, context });
// Recursively get dependencies (DFS; users can implement BFS too).
const modules = this.getModules(mod.requests, context, mod);
mod.link(modules);
this.#moduleCache.set(specifier, mod);
result.push(mod);
}
return result;
}
// --- Example helpers (not part of the API) ---
helperGetSingleModule(request, context, parent) {
return this.getModules([request], context, parent)[0];
}
helperAddSource(identifier, source) {
this.#sourceStore.set(identifier, source);
}
helperAddBuiltin(identifier, mapping) {
const keys = Object.keys(mapping);
const mod = new SyntheticModule(keys, function() {
for (const key of keys) {
this.setExport(key, mapping[key]);
}
});
this.#moduleCache.set(identifier, mod);
}
}Preparing the custom loader:
const loader = new ExampleModuleLoader();
// If the loader needs to fetch the source asynchronously,
// they can pre-fetch the whole graph from a known root,
// so later when `getModules()` is run, it just loads from the
// asynchronously prefetched graph synchronously and sets up
// the links. This is similar to what Node.js internally does.
loader.helperAddSource('async-root', `
export { foo } from 'foo';
export let bar;
bar = await import('builtin:bar') + import.meta.identifier;
`);
loader.helperAddSource('sync-root', `
export { foo } from 'foo';
import { default as bar } from 'builtin:bar';
export const baz = bar + import.meta.identifier;
`);
loader.helperAddSource('foo', `
export const foo = globalThis.foo;
`);
loader.helperAddBuiltin('builtin:bar', { default: 'bar' });1.a. Async module graph in a new context
import { createContext } from 'node:vm';
const context = createContext({ foo: 'foo' });
const [mod] = loader.getModules([{ specifier: 'async-root' }], context);
// returns undefined or throws for any loader errors.
mod.evaluate();
mod.hasTopLevelAwait(); // true
mod.hasAsyncGraph(); // true
mod.namespace(); // { foo: .., bar: ... } - not yet populated
// Module completion is tracked separately via topLevelCapability,
// so it's what you await to finish evaluation of the module.
await mod.topLevelCapability;
mod.namespace(); // { foo: 'foo', bar: 'barasync-root' }1.b. Synchronous require(esm) pattern
See #59656 for this use case.
const [mod] = loader.getModules([{ specifier: 'sync-root' }], context);
mod.evaluate(); // returns undefined or throws for any loader errors.
mod.hasTopLevelAwait(); // false
mod.hasAsyncGraph(); // false
mod.topLevelCapability; // Always fulfilled for non-TLA modules.
mod.error; // If the module threw during evaluation, the error is here.
mod.namespace(); // { foo: 'foo', baz: 'barsync-root' }2. Partial customization and registering globally
A loader can delegate to the base class for specifiers it doesn't need to customize, and can be registered for a context so that all ESM running in that context (including vm.Script dynamic import()s) uses it.
import { SourceTextModuleLoader, SyntheticModule } from 'vm/modules';
import { createContext, runInContext } from 'node:vm';
class CustomLoader extends SourceTextModuleLoader {
getModules(requests, context, parent) {
return requests.map((req) => {
if (req.attributes?.type === 'foo') {
return new SyntheticModule(['foo'], function() {
this.setExport('foo', 'foo');
}, { context });
}
return super.getModules([req], context, parent)[0];
});
}
// Other methods are left to the default.
}
const loader = new CustomLoader();
// Register the loader for a vm context.
// Only one loader can be registered per context.
loader.register(context);
console.log(
await runInContext(
`import('foo', { with: { type: 'foo' } })`,
context
)
); // { foo: 'foo' }
// Unregister when done.
loader.deregister(context);
// Register for the main context.
loader.register();
console.log(await import('foo', { with: { type: 'foo' } }));Design details
Error model: loader errors vs. module errors
See #60242 for background.
In the current API, evaluate() wraps all errors (status errors, timeout, and actual module exceptions) into a single promise rejection. This makes it difficult to distinguish errors from the loader (implemented by e.g. a framework) from module-level errors (thrown from e.g. user-provided code being tested by a framework), and makes re-implementation of require(esm) awkward.
In the new API, evaluate() separates the two:
- Synchronous throws for loader/infrastructure errors: wrong module status, timeout (
ERR_SCRIPT_EXECUTION_TIMEOUT), signal interruption (ERR_SCRIPT_EXECUTION_INTERRUPTED). - Use
module.topLevelCapability(maps toCyclicModuleRecord.[[TopLevelCapability]]in the spec) to access evaluation resolution or rejections - Use
module.errorto accessCyclicModuleRecord.[[EvaluationError]]once it's evaluated.
Source phase imports
Module requests carry phase information (request.phase). For source-phase static imports (import source x from 'y'), the module returned from getModules() must have a source object set via mod.setSourceObject(obj). For dynamic source-phase imports (import.source('y')), dynamicImport() receives request.phase === 'source' and should return the source object (e.g., a WebAssembly.Module) instead of a namespace.
dynamicImport(request, context, parent) {
const mod = this.helperGetSingleModule(request, context, parent);
if (request.phase === 'source') {
return mod.sourceObject;
}
mod.evaluate();
return mod.namespace();
}TODO: figure out what to do for deferred imports, but it'll be a phase as well.
One loader per context
When a loader is registered for a context, it handles dynamic import() for all code evaluated in that context, including vm.Script. With the new design, omitting importModuleDynamically from a vm.Script should mean "delegate to the context's registered loader if any, otherwise throw." This makes loader.register(context) a single point of configuration for all ESM loading in a given context.
The new dynamicImport(request, context, parent) callback receives the context as a parameter, so it can construct modules in the right context. This addresses the issue where importModuleDynamically for vm.Script didn't have context information (#35714).
loader.register(context) throws if a loader is already registered for that context. For users who want hooks-style middleware composition, module.registerHooks(hooks, context) is still a a separate, orthogonal mechanism that allow nesting, and it runs in a lower layer underneath the loader's getModules()/dynamicImport().
Different level of customizations
See #31234, #35848, #43899 #61127.
The base SourceTextModuleLoader needs a meaningful default getModules() implementation for partial customization to work. The plan is to expose Node's built-in resolution and loading as composable functions (someting like module.resolve(specifier, parentURL, context) and module.load(url, context)) that the base getModules() use (see #55756). This way:
- Partial customizers call
super.getModules()for requests they don't handle. - Advanced users can call
module.resolve/module.loaddirectly to implement theirgetModules()override, these in turn runs the hooks registered bymodule.registerHooks()and/or the built-in resolution/loading logic, so the hooks and the loader are composable in a flexible way.
Activity
joyeecheung commentedon Apr 13, 2026
cc @nodejs/loaders
GeoffreyBooth commentedon Apr 14, 2026
I like it! You show an example of full customization; what's the inverse, of someone who just wants to run a virtual module with as little boilerplate as possible?
What are some examples of how Vitest or Jest would use the new API?
joyeecheung commentedon Apr 14, 2026
That's the partial customization example if I understand you correctly - you just extend the class but inherit most methods, and use
supermethods for cases that don't need cutomizations.I think only their maintainers can answer this question probably but from what I can tell they'd need to go with a full customization loader + prefetching the graph if they need to build it asynchronously (they can still use
SourceTextModuleLoadermethods or the proposedmodule.resolve/module.loador something similar to reduce the amount of work for non-customized requests) and load from a prefetched map ingetModules()(which is similar to what we do internally for the async path & what the spec intentionally allows), For testing frameworks specifically, they do know the root so prefetching should work. I added some comments to theloader.helperAddSourcecalls in the OP to explain.JakobJingleheimer commentedon Apr 15, 2026
This looks really good/promising!
A high-level concern:
Why not literally throw? AJV behaves like
mod.errorand it's really frustrating to use.Bikeshedding:
topLevelCapability- I think the name will be gobbledygook to almost all users. How aboutdone?mod.done(sync) &await mod.done(async) →trueWould
SourceTextModuleLoadernot have its own#sourceStore&#moduleCache? If it does, I think the subclass having one too could create a Where's Wally situation (JS doesn't haveprotected😢, so the subclass couldn't accessSourceTextModuleLoader#sourceStoreetc).joyeecheung commentedon Apr 15, 2026
In the new design,
evaluate()only throws loader errors, because they come from whoever implements the loader (in our default implementation, this can be timeout, assertion failure, someone sends SIGINT to interrupt the evaluation etc.).module.errorcomes from the modules (e.g. if it throws in the code) that go through the loader, and is likely not from those who implement the loader. If we also throw that inevaluate()it would be difficult to distinguish the two e.g. in the case of testing framework, the former can be user Ctrl+C to cancel the test running, the latter is the test actually failing, and they likely want to handle the two differently. See #60242 for more details. In the V8 API/spec this doesn't throw for evaluate either, it technically returns a promise that rejects,mod.erroris just a shortcut to unwrap the rejection if it's synchronously rejected.This comes from the specification https://tc39.es/ecma262/multipage/ecmascript-language-scripts-and-modules.html
I think we may need to decide if we want the API to map more closely to the spec so that it's easier to look up what the spec says when you want to ensure the loader conforms to the spec, or if we want to read less gobbledygook. I am inclined to follow the spec though, because we already have been doing that with
SourceTextModuleandSyntheticModuleand it's easier to look things up that way.No, that's just for the demo. The default base class just does what Node.js internally does (which has its own similar stores, but the whole thing is more complex), the #sourceStore & #moduleCache there are for the example overriding methods. To access the way Node.js handles things they could either use
supermethods, or themodule.resolve/module.loadproposed, but both would be much more sophisticated than just querying things from a map.JakobJingleheimer commentedon Apr 15, 2026
ohhhhh, got it. nevermind, good idea.
I think a mix is okay / desirable:
SourceTextModuleandSyntheticModuleare easy (with not too much knowledge) to guess their purpose. Whereas I'm in TC39 Harmony, and I couldn't have guessed whattopLevelCapabilityis (with a name like that, I would've assume it does something else far more nuanced or sophisticated, but it looks like it's just a terribly over-complicated name). If something is directly analogous to spec, maybe just cite that in the API doc (with a link)?Ahh, okay.
guybedford commentedon Apr 15, 2026
Overall this seems like a solid design to me. For the source phase integration, I agree with the rough proposed framing for this.
One thing I would also add is that
ModuleSourcewould be beneficial to support in all places where a string source is currently provided. The framing here is kind of like thatModuleSourceis like the "trusted types" version of a raw string. Sonew SourceTextModule(source)could even be beneficial. Or hooks that support direct source strings allowingModuleSource. It can also be possible to returnModuleSourcefromgetModulesfollowing the same semantics asimport(module)for the module source (i.e. every source has a canonical instance associated with it in a given loader context).Then, since
WebAssembly.Moduleis also a module source, it would be beneficial to allownew SourceTextModule(wasmModule)to allow proper linking of Wasm too. I think the answer given to @nicolo-ribaudo's question yesterday on this topic of assuming explicit instantiation isn't sufficient without this.The reason we can't just rely on explicit instantiation for Wasm is because there are features of the cycle module record integration that cannot be simulated this way. In particular:
exportsshape of the instantiated Wasm module. This is becauseWebAssembly.Globalis unwrapped on the instantiation boundary in ESM exports to provide direct values instead of theWebAssembly.Globalwrapper object allowing JS value types to be exported instead of just the limited set of Wasm types.WebAssembly.namespaceInstance(ns)which performs a lookup from a Wasm module namespace object, and returns itsWebAssembly.Instancecannot be supported without tying Wasm instantiations through the loader semantics into a global weakmap, which is the responsibility of the loader. (see https://webassembly.github.io/esm-integration/js-api/index.html#dom-webassembly-namespaceinstance-namespace).joyeecheung commentedon Apr 22, 2026
Thanks for the feedback! Supporting module source should be more straightforward or I think can fit into the current draft incrementally. But good points on wasm module. I didn't fully understand @nicolo-ribaudo 's question at the session though we later discussed about it and I agree we need to sketch out a bit more concretely about how this would work with wasm imports especially if there's a cycle. I will brew a bit on that and see what I can come up but ideas would also be welcomed - the draft is early and it's the kind of feedback I was looking for to make sure that we can tweak it to be more optimal for the cross-cutting concerns while we still can.
SimenB commentedon Apr 23, 2026
Thanks for putting this together Joyee! 🙏 Sorry for the slow reply, only just saw your email this morning.
First off, really happy to see this getting attention at all, and thanks for pushing to unblock this. Jest has been living on
--experimental-vm-modulesfor years, and the flag gate is one of our most frequent ESM pain points, both for us to explain and for users to stumble into. Gettingvmmodules on a path to stable is a huge deal for Jest and for our users. 😀Direction feels right to me, and the error-model split in particular is going to save us a lot of grief. Writing from Jest's perspective, assuming this is the shape we'd rewrite
jest-runtimeonto.Before the list:
jest-resolveowns every resolution path in Jest (import,requireincludingrequire.resolve, andimport.meta.resolve). Two drivers:__mocks__folder,automock,moduleNameMapper). Runtime mocking comes from test code (jest.mock,jest.doMock,jest.unstable_mockModule). Both intercept at the resolve layer, and both have to reach any specifier user code can resolve. So all resolve surfaces have to agree, and all of them have to be hookable.tsconfig.json'spaths,moduleNameMapper, Haste, pluggable custom resolvers. None of that is expressible via Node's resolution alone.Consequence is that we can't delegate to
super.getModules()today, even though it's really appealing (caching, linking, source phase, future Node features, all for free). Two API shapes would let us (and probably anyone with a richer resolver or runtime module interception) stay on the base-loader happy path:loader.resolve(specifier, parentURL, context)method thatsuper.getModules()calls via method dispatch, so subclasses can override just the resolve step and keep the base loading/linking/caching, orCalling this one out up front because it shapes everything below. The numbered list assumes we're running our own
getModulestop to bottom.1. Cache invalidation
jest.resetModules()andjest.isolateModules(fn)are public API, so every ESM record has to be disposable. Sincejest-resolveowns resolution and mocking has to reach every specifier (includingnode:*), we'd avoidsuper.getModules()and maintain our own cache stack (we need a stack anyway forisolateModules). That narrows the question to: which helpers we do call carry internal state?What we'd need:
module.load(url, context)in particular - we'd want to use it as a pure source-fetch + compile helper. If it caches per-context, we can't, and we'd have to reimplement that layer ourselves.loader.invalidate(specifier)/loader.invalidateAll(context)or similar). (Aside, not a Jest concern: bundlers and dev servers would possibly want the same surface at some point for HMR-style use cases.)dynamicImportpromises when caches mutate under them.2. Mocking as a first-class use case
jest.unstable_mockModule('./x.js', factory)replaces./x.jsfor the current test's graph with aSyntheticModulebuilt from the factory. Mapping that onto the proposal is a branch at the top of ourgetModules. Workable, as long as the base loader doesn't short-circuit us.Two things that'd help:
getModulescan returnSyntheticModulefor arbitrary specifiers without the base resolver being consulted, and without the base cache poisoning the identity.SyntheticModuleon the next request (i.e. identity per-specifier isn't a Node-side invariant across time).We'd love to unify
jest.mock(CJS, hoisted) andjest.unstable_mockModule(ESM, async) at some point. That's blocked byimportevaluating before user code regardless of the loader, not by anything in this proposal. Just want to make sure the proposal doesn't make convergence harder.3. Supporting
require(esm)This is a long-standing goal on our side that we just haven't gotten to yet. With
hasTopLevelAwait()/hasAsyncGraph()plus conditionally-syncevaluate(), we should be able to support it, as long as the sync-eval path is reachable from a framework-owned loader (not just Node's internal one).When the graph has TLA and can't be sync-evaluated, a typed error (e.g.
ERR_REQUIRE_ASYNC_MODULEwith the module identifier) would let us surface a good message instead of a generic failure.4. Loading Node builtins into VM contexts
When we resolve a specifier to a builtin ourselves (
jest-resolveroutesfstonode:fsetc.), we still need a module record the VM context can link to. Today werequire(name)from outside the VM and wrap the result in aSyntheticModule. Works, but it's a hack. We had to guess at identifier shape (there's literally a comment in jest-runtime asking whether the identifier should benode://${name}), and there's nothing context-aware about it.Would love a cleaner API here. Something like
module.loadBuiltin(name, context)returning a linkable module record, or a way to callsuper.getModules()just for builtin specifiers without re-entering the base resolver.5. Loader lifecycle
Two small asks:
loader.register(context)returning aDisposable/AsyncDisposableso teardown order is obvious.dynamicImportacrossderegister.Jest's environment owns the VM context, and we need to deregister cleanly before the context is torn down. Today there's no clear contract for what "cleanly" means.
6. Isolation pattern docs
"One loader per context" collides with
jest.isolateModules(fn), where we want a scoped, disposable module cache inside the callback. We'll handle it in user-land (cache stack inside our loader), but the same pattern is going to come up for anyone else offering module-graph isolation. A documented convention for nested disposable caches would save a lot of reinvention.The wins 😀
Don't want to bury this. What you've drafted kills several recurring pain points for us on contact:
importModuleDynamically/initializeImportMetaclosures go away, which means the GC leak class you called out evaporates once callbacks live on the loader. Also wins back per-module closure allocation (we build thousands of these per test run, so this is not just correctness but memory).dynamicImportfor ESM + CJS (compileFunction) +vm.Scriptdeletes a bunch of duplicate plumbing. As a bonus this finally addressesvm.Script'simportModuleDynamicallyshould get a reference to thecontextused to run it #35714 (I think at least 😀)module.errorvstopLevelCapabilitysplit maps exactly onto "test failed" vs "framework blew up", which is the distinction circus needs. Today we catch everything coming out ofmodule.evaluate()and have to guess which bucket an error belongs to.getModulesreturning linked modules, ormodule.link(specifiers, modules)) lets us drop_esmModuleLinkingMap(our WeakMap of in-flight linking promises, needed today to dedupe concurrentlink(linker)calls across parents) and_fileTransformsMutex. Both exist purely as workarounds for shape mismatches against the current API.new SourceTextModule(wasmModule)replaces ~50 lines of hand-linked wasm.loader.importMetais where we'd setmeta.resolve(routed throughjest-resolve) andmeta.jest. Clean fit, one call per module instead of a closure per module.import source, deferred) come for free as long asrequest.phasepropagates.evaluate()gated onhasTopLevelAwait()/hasAsyncGraph()is exactly the shape we want: prefer sync, fall back to async only when the graph forces it. Puts the onus on us to use sync transform paths where we can (babel/swc/esbuild/ts all have them).Modulesubclass (currently overridingcreateRequireinside VM contexts and stubbingsyncBuiltinESMExportsas a no-op) can probably shrink significantly once the loader handles context-aware module construction (depending on how CJS interop would be done).Happy to iterate on any of the above 🙏 If a sketch would help, I have a rough skeleton of
getModules+ a Jest mock/cache layer I can clean up and post.Sidebar: not ESM related, but
vmUnrelated to this proposal, but since we're on the topic of
vm: re-raising the per-context Node builtins ask (#31852). This has been languishing since 2020 and was auto-closed by the stale bot 😢 Per your comment there about ShadowRealms integration making this closer to feasible, it feels worth reconsidering.For Jest this is long-standing pain: user code running in a VM context gets
fs,Buffer,setTimeout, errors, etc. from the outer realm, soinstanceofchecks fail, global mutations bleed across test files, and we can't cleanly isolate a test's globals. Cross-context interop isn't a requirement for us (per the original discussion).Also saw your comment on #31658 pointing to
vm.constants.DONT_CONTEXTIFY, thanks for that! Very happy there's a better workaround path now, we'll likely adopt it in the meantime. That said, it addresses the symptom (interceptor overhead from ourGlobalProxyinjection dance) rather than the problem (Jest having to inject globals at all because the context doesn't have its own). If #31852 lands, we'd stop injecting entirely andDONT_CONTEXTIFYbecomes moot on our side (I think - haven't dug into it).Also, small helper ask: CJS named-exports
We use
cjs-module-lexerto derive named exports forimport {foo} from './bar.cjs'. Node uses it internally forrequire(esm)too. Would be great to have it built in:module.cjsNamedExports(source | path) → string[], or aSyntheticModule.fromCjsExports(exports)helper. Saves every framework re-vendoring the lexer.SimenB commentedon May 1, 2026
Quick update: we're shipping
require(esm)in Jest in jestjs/jest#16074 (will be in the next 30.x release).A few concrete things from the implementation that I think might be worth folding into the proposal, on top of the earlier asks:
1. Which module forced async?
When
hasAsyncGraph()is true on the root, we want to name the offending file in the error: "require()can't loadfoo.mjssynchronously: a dependency uses top-level await (bar.mjs)". Today we walk the graph callinghasTopLevelAwait()on eachSourceTextModuleto find the culprit. It works, but feels unelegant.A predicate on the module would let us build error messages without iterating the graph:
rootModule.asyncReason()returning{kind: 'tla' | ..., identifier: string} | null. Node already has the info internally I think (per--experimental-print-required-tla); exposing it would let our error messages match Node's quality without duplicating the bookkeeping.2. SyntheticModule has no sync evaluation path
The redesign'sgetModules({sync: true})returning ready-to-use records will make this moot, but worth flagging as a concrete thing it solves: todaySyntheticModulecan only be linked and evaluated through the asynclink()/evaluate()pair, so it can't be the root of a syncrequire(). For us that blocksjest.unstable_mockModulefactories from replacing a module that'srequire()d directly.3.
linkRequests+instantiatesplitWe always call these as a pair, so the redesign's single
mod.link(deps)collapsing them is a nice simplification.The direction laid out here looks great to me, and it should, from what I can tell, solve the issues we have with the current implementation. Thanks again for working on it!
joyeecheung commentedon May 4, 2026
@SimenB Thanks for the thorough feedback! It took me a while to reason through them, to respond to the points where I have some idea on how to address them for now:
On the ability to customize resolve more thoroughly, I think the current design in combination with
module.registerHooks(which shall be picked up bySourceTextModuleLoader#getModules) could be adequate enough? For example:Cache invalidation
I think we can make
module.resolve/module.loadtake an option regarding whether it should consult/bypass the cache, without the cache it's stateless. For invalidating the built-in cache, that's being implemented in #61767 which I think would work with this proposal too, just that if we were to have per-context cache, that API needs to take another argument for the context.I think this is a challenge that can't be guaranteed by the API alone - internally in Node.js it's still an unsolved issue and we've had races before we synci-fy most of our loading paths. For the built-in loader are currently leaning towards making all loading fully synchronous (#55782 ) and so in this case, cache access is generally deterministic. From discussions with someone else who had a similar use case in their internal runtime, syncrhonicity seems to be the most viable way to make the cache access deterministic when it needs to support both
require()andimport().require(esm)To help surfacing "which module contains TLA", currently V8 only has an API that allows us to detect it after the module graph is evaluated, which is obviously not the best since this is statically detectable, and it's why
--print-required-tlais currently still experimental. It's still a TODO for us to replace that with trackinghasTopLevelAwaitinformation collected in the graph after compilation, but yes it's implementable and I think when we have it we can expose it to the vm Module API too to reduce the churn for users.Loading Node builtins into VM contexts
This was also raised by a vitest maintainer in review of this proposal - I think there are two things about builtins:
import 'node:foo"going through the built-in loader, so we can just wrap them and expose them to the new APII think for
super.getModules(), the best we can offer for now is to document that "when it resolves to a builtin, the context is currently ignored and we always return the module record from the main context", which is also effectively what the conceptual "default implementation" would do today. When we do have internal support for context-specific builtins, we can add an option togetModules()to return a copy and eventually with a breaking change, flipcontextAwareBuiltin: trueas the default behavior.Loader lifecycle
I think the asks are implementable; for the in-flight
dynamicImportI think we can have some kind of API to track them and users can query them before deregistration/thederegistermethod can return them. Probably don't need to be in the initial implementation but can fit in to a follow up?Isolation pattern docs
I think maybe one thing that would help is to add an API that returns the currently active loader? Technically one could still nest them by calling the previous loader's methods inside their own methods. So instead of
super.getModules(), just callsavedLoader.getModules().module.cjsNamedExports()Idea SGTM, maybe it needs a separate feature request issue?
Mocking as a first-class use case
getModules()should consult theparentparameter for base URLs, so it's configurable by the user. For resolution/loading cache consultation, we can make it configurable via an optionThe following is one thing I think can not be fully addressable by the proposal, as it touches the ESM invariants in the spec and isn't something that Node.js alone can provide/guarantee.
I think this depends on the exact input/output pairs and whether they conform to https://tc39.es/ecma262/multipage/ecmascript-language-scripts-and-modules.html#sec-HostLoadImportedModule - if the parent module is technically different (e.g. it has a cache-busting URL), or it's in a different context, then it can be considered legal. Otherwise the invariant comes from the spec and V8 is free to misbehave on them (it currently may not do that, but it's not guaranteed as the invariant is in the language spec)
It is true that "being able to easily swap out things in the ESM namespace" is not going to be made easy by this proposal, as that's made difficult by the language spec. From what we've discussed with V8 previously it would also be difficult to convince the upstream to open up a hook like this as this can easily messes up the ESM state machine management in JS engines that is already quite brittle. Both https://github.com/nodejs/import-in-the-middle and Node.js's experimental module mocking tries to implement it at the source level which are both quite convoluted and have some limitations, a
SyntheticModuleprimitive can make it less convoluted but still it's mostly dancing around the spec limitations of ESM namespaces being immutable.SimenB commentedon May 5, 2026
Thanks @joyeecheung! The concrete sketches really help. A few follow-ups on the
registerHooksshape.module.registerHooks+hookDataI like it. 😀
registerHooks+hookDatafits partial customization well from what I can tell; for Jest's case (own the whole pipeline, want Node's compile/fetch/link/cache underneath) a directsuper.load(url, context)we could call after resolving viajest-resolveourselves might be cleaner? Both shapes worth having, I think.Questions:
Does
import.meta.resolvederive from the loader's resolve? Saves us wiring it up per module ininitializeImportMeta.How do
require(esm)andawait import()distinguish? Today we thread'sync-preferred' | 'sync-required'through the graph walk soawait import()can fall back on async edges (async mock factory, async-only transformer) whilerequire(esm)throwsERR_REQUIRE_ASYNC_MODULE. Prefetch covers the async entry; forrequire(esm)we can't prefetch. Separate sync entry, request flag, something else? (TLA is fine via conditional-syncevaluate().)One shape thought: constructor options instead of subclassing -
new SourceTextModuleLoader({ getModules, dynamicImport, importMeta }). Subclassing forces a newclass extends ...per Jest Runtime (methods bind at class-definition time, can't close over per-Runtime state otherwise). Same shapevm.Script'simportModuleDynamicallyalready uses.Happy to prototype against
jest-runtime- picks up where the sketch I offered in the original comment would have gone.Cache eviction
The
Module.clearCacheshape from #61767 covers what we need 👍. One alternative to the per-context arg you mentioned: since loaders are per-context anyway, an instance methodloader.clearCache(specifier?)might be cleaner?Other points
super.getModules()would already be a nice win for us in the meantime - lets us drop our hand-rolled shimhasTopLevelAwait: 👍require(esm)- our internaltryLoadGraphSyncalready operates that way. I guess the async path would still exist, but would be raredynamicImportas follow-up is perfectly fine 👍. Open from our side still:Disposable/AsyncDisposableonregister, and what an in-flightdynamicImportdoes when its context is torn down - a typed rejection would be enough; we've had real bugs from imports outliving the envmodule.cjsNamedExports: agreed - module: expose CJS named-exports detection (cjs-module-lexerequivalent) #63123One thing this whole conversation got me thinking about: CJS in
vmalongside ESM. Today we docompileFunction+ manual module-function wrapping (then execute it and wrap inSyntheticModuleon the ESM side), which works but is a fair bit of plumbing for what's conceptually "load a CJS module into this context." Something like aSyntheticModule.fromCjsExports(exports, options)convenience that combined the lex with the synthetic construction would be a much nicer shape than what we hand-roll today. Probably its own conversation, not asking for anything concrete here - just flagging that the new ESM loader landing might be a good moment to look at the CJS side too.joyeecheung commentedon May 5, 2026
I think a phase helper wrapping "compile/fetch/link/cache" makes sense though to avoid ambiguity we should probably avoid naming it
load, since that identifier is taken by the hooks to mean "the process of fetching source code and determining module format, given a resolved URL". I am not sure what's the best way to name it - internally we call this processloadAndTranslate()(translate includes compilation + linking);Another idea can be to just reuse
getModules()but you simply pass the resolved URL to it, in general the two are equivalent (e.g.import('<full URL>')usually operates the same asimport('<relative specifier>')without hooks). This is conceptually what the built-in implementation ofrequire(esm)does.If
the loader's resolvemeans what goes on ingetModulesthen no,getModulesare conceptualized as callers of the resolve step. In the new loader'simportMetaoverride, if you invokesuper.importMeta(meta)it will install the built-in loader'simport.metastuff on it, which includesimport.meta.resolvebut that'll be the built-in loader's version (which goes through resolve hooks registered bymodule.registerHooks). To install a resolve fully controlled by the user, they can simply assign a function that's fully controlled by the user (this can be done as an override after callingsuper.importMeta(meta)too)I think in this case, we may need to add an argument to
getModulesto carry the metadata? Perhaps it should begetModules(request, data, context)instead, wheredata.parentis the parent module initiating the request, and we put other kind of information (e.g. it's coming fromrequireorimport()or staticimportindata).This is an interesting idea - the subclass design was mostly for connivence of inheriting default implementations of each methods, but I think an option bag with defaults can do the same. So far I am leaning towards this design 👍
I think this may introduce some confusions: the loader is in charge of loading ESM (lower part of
require(esm)- after we turn require into a "synchronous import" after knowing the target is ESM - orimport(esm)), but is orthogonal torequire(cjs)loading (it will be otherwise very difficult to fit it into the same API, because CJS loading is by design very different from ESM loading), thereforeloader.clearCacheof the built-in ESM loader cannot touch the require cache (this is also apparent in themodule.clearCacheimplementation).On that note I think there may be some subtleties in how the loader methods are invoked for the main context's
require(esm)- we can not know that we are trying to load a source text module until after the resolution (and if not.mjs, including source loading) is done. Only when we know that the target is a source text module will we invoke theSourceTextModuleloader (likely just callinggetModuleswith the full URL and the already fetched source code + format in the data, this is conceptally how our built-inrequire(esm)works currently too), To hook into the part prior toSourceTextModuleLoader#getModulesinvocation, the user would have to usemodule.registerHooksto e.g. virtualize the resolution and source loading coming fromrequire, though this would need to be done ifrequire(cjs)needs to be handled anyway asrequire(cjs)fully bypass theSourceTextModuleLoadermechanism.I am wondering if introducing an abort-controller-based mechanism makes sense? At least there can be a per-context signal (can be created during context creation) loader implementers can grab from
dynamicImport's context parameter? I think V8 should be able to GC it properly if the signal is implemented correctly (e.g. removing the abort listener oncedynamicImportis completed), though won't be able to know for sure without diving into an implementation.I think what you are asking is pretty much what we have here. When we finish deprecation of
module.register(possibly some time in 27/28, which removes on variant of the quirkyloadCJSimplementation) I can see it become feasible. But before we stablize the internal implementation, in the near term some hand rolling may be inevitable.SimenB commentedon May 5, 2026
Thanks @joyeecheung, lots to like here - I'm optimistic about the direction of the design. 😀 Happy to take a look again later if there are changes or implementation begins 👍
A few final follow-ups from my side:
That makes sense. Presumably handing it a fully-resolved URL also skips the
module.registerHooksresolve chain since we've already done that step?That works for what we need. 👍 Anything else you're considering carrying in
data(just curious)?Yeah,
AbortControllerworks. I'd lean toward one signal per context rather than per-call - on env teardown we want one place to abort everything in flight, and per-call granularity (e.g. timing out a single import) feels like a separate concern. Does that line up with what you were picturing? I guess there could be oneAbortSignal.any()aggregate thing as well?Cool - I think that simplifies things on our side at least without creating odd closures.
Right, that makes sense.
The
SyntheticModule.fromCjsExports-equivalent existing internally is good to know about - happy to wait on themodule.registerdeprecation timeline.SimenB commentedon May 8, 2026
Hello again! I just filed #63186 - flagging here since it relates to
vm.SourceTextModuleand may or may not be addressed by this redesign 🙂(in semi-related news - I released Jest 30.4 yesterday with
require(esm)👍)joyeecheung commentedon May 12, 2026
I think whether the hooks are skipped can be an option to
getModules(internally the built-in loader has a similar option which is useful for re-entrance ofimport cjsetc.)I think so far the "where this request is coming from" is the most obvious thing (we might want to replicate this in the
module.registerHooksAPI too). But also maybe users can put custom data inside as well which then can be passed into the lower level hooks?Yes, I think per-context signal makes sense. Not sure about aggregates, I feel that per-context signal might be the safest in terms of risk of leaks.
As commented in #63186 (comment) - I think it's a generic V8 GC laziness problem, not a leak, it might even be unrelated to modules.
joyeecheung commentedon Jul 21, 2026
Some updates:
loader.register(context), we should also offer something likeloader.requestGraph(specifier, data, context)that can be invoked to start the fetching and linking process for a subgraph - all modules fetched by a loader from the root will be "minted" by the loader automatically i.e. their dynamicimport()dispatches would also be dispatched through the loader'sgetModules()if overridden, and so on.resolve/load, which can in turn callmodule.resolve/module.loadand choose whether they want to include global hooks as part of the chain.import source/import.source()instantiate()explicitly (i.e.getModulesshould not encompass instantiation, but only provisionally saves the links). ForWebAssembly.namespaceInstancewe just need to make sure the API design leaves room for us to dispatch the query to user-land loader when V8 starts to consult the embedder for the instance.request.phaseingetModules()and accept module source as an element in the array returned by the user (ifphaseis'source'), the same goes todynamicImport()as well. Whether the returned module source is tied to aSourceTextModuleis irrelevant to us - it is true that to mimic how WASM integration works in Node.js internally, you'd build a SourceTextModule-based facade, but that's an implementation detail. If there are any other module types supported in the future, you don't necessarily need aSourceTextModuleto back the source. We could consider exposing a helper that wraps "how Node.js builds a module source from a WASM module with a live binding" in the future, but that doesn't really need to be part of the MVP.On a higher level of the API shape, something like this (I renamed
getModulestorequestModules, and it's now calledModuleLoaderbecause instances of this may be used to handle dynamic imports fromvm.Script/vm.compileFunction, and doesn't necessarily returns aSourceTextModuleor has anySourceTextModulein the graph in that case).For each of the hooks above, we'll try to provide a freestanding helper in the
node:modulenamespace that captures "what Node.js's builtin loader would do in this case", and these helpers can take an option to specify if they should consult the custom loader, any other lower-level hooks, or should they just do the default behavior (as if there's no customization). To reduce the surface subject to Hyrum's law, it's better avoid exposing a default loader instance (lest people start to expect e.g. monkey patching prototype methods of it would do anything) - freestanding helper functions have a much smaller compatibility surface.For sync customization driven completely by the loader:
For async customization driven externally, with the loader only "connecting the dots" afterwards:
Will create a more consolidated proposal incorporating other points we discussed above as a PR to nodejs/loaders in the coming days...
ScriptedAlchemy commentedon Sep 8, 2026
@joyeecheung love this! Is there any chance VM could support some option like codeCache: false?
v8 code cache can hold onto old versions of a module up to its max old space limits.
module-federation/core#4566 (comment) I believe the source of this users leak however is in timers pinning the old graph, but in doing so i did notice that using VM/eval etc will hog the max space available given enough 'hot reloads'
joyeecheung commentedon Sep 11, 2026
@ScriptedAlchemy
I think you may be thinking about #63186 which I have a WIP fix working and from offline discussion the V8 folks are open to the approach. Still need to polish the tests. My current plan is to land that CL, and also add an API for "opting out of the compilation cache", to reduce memory consumption for frameworks that are compiling code that are known to be constantly changing e.g. modules being watched and HMR-ed. This also appears to be acceptable, though I will need to demonstrate that the changes won't affect Chromium negatively. When the V8 changes are accepted, I can try them with a prototype of this proposal to verify a leak-free approach for this type of use cases.
Note: there are two types for compilation cache in V8 - the in-memory one, that is handled by V8 and the fix generally have to be done in V8 like in the CL, not from Node.js; or, the serializable one e.g. what you get from
produceCachedData: true, which is handled by the user and leak there generally arise from user land.ScriptedAlchemy commentedon Sep 11, 2026
@joyeecheung you never cease to amaze me