Repository navigation
Proposal: --default-type #335
Description
This is a proposal to provide a way to change Node’s default module system from CommonJS to ESM (#318).
There are two places where the module system can be specified, with a lack of specificity defaulting to CommonJS:
-
package.json"type"field, where the lack of the field is interpreted as"commonjs". -
--input-type, where the lack of the flag is interpreted ascommonjs.
We propose a new flag --default-type, to control the default values of package.json "type" and --input-type:
-
--default-type=commonjsis the same as the current behavior, where the lack of the flag or field is interpreted as CommonJS. -
--default-type=modulewould cause the lack of the flag or field to be interpreted as ESM.
The expectation is that users who prefer ESM to be their default would set this flag in their NODE_OPTIONS, e.g. NODE_OPTIONS=--default-type=module.
With the flag set to module, some potential use cases are:
-
Commands like
node --eval 'import { version } from "process"; console.log(version)'could be run without needing--input-type=module. -
Extensionless files to be run like shell scripts could use ESM syntax, without needing to be symlinks to
.mjsfiles or use other workarounds.
Under --default-type=module, most typical projects with dozens of CommonJS dependencies would break. Most, if not all, of those dependencies would lack "type": "commonjs" in their package.json files, at least for now while public awareness of the new field grows. One potential solution is a script that users could run to add the field for any dependencies that lack it:
#!/usr/bin/env node
const { argv, exit } = require('process');
const { basename, resolve } = require('path');
const { readdir, open } = require('fs').promises;
if (argv.length !== 3 || argv[2] === '-h' || argv[2] === '--help') {
const scriptCommand = basename(argv[1]);
console.log(`This script searches a folder and its subfolders for package.json files,
adding "type": "commonjs" to any such files that lack a "type" field.
Usage: ${scriptCommand} folder
Example: ${scriptCommand} ./node_modules`);
exit(1);
} else {
walk(resolve(argv[2])).catch(console.error);
}
async function walk(folder) {
const entries = await readdir(folder, {withFileTypes: true});
await Promise.all(entries.map(async (entry) => {
if (entry.isDirectory()) {
return walk(resolve(folder, entry.name));
} else if (entry.name === 'package.json') {
const resolvedFilePath = resolve(folder, entry.name);
let handle;
try {
handle = await open(resolvedFilePath, 'r+');
const manifest = JSON.parse(await handle.readFile('utf-8'));
if (manifest.type === undefined) {
manifest.type = 'commonjs';
await handle.writeFile(JSON.stringify(manifest, null, 2));
}
} catch (exception) {
console.error(`Error updating ${resolvedFilePath}:`, exception);
} finally {
if (handle) {
await handle.close();
}
}
}
}));
}We expect that package managers might soon begin to add "type": "commonjs" automatically for dependencies that lack a "type" field. If they don’t, and no other userland solution finds broad acceptance, another solution is for Node to add a third option to --default-type, namely --default-type=module-except-dependencies. This would behave the same as --default-type=module except that it wouldn’t apply to folders under a node_modules folder. We would prefer not to need to implement this, however, as a “pure” solution where package managers or userland utilities bridge the gap would let Node provide a more ideal API. This option can always be added later if it turns out to be necessary.
Activity
ljharb commentedon May 31, 2019
This would be highly dangerous and break a lot of code by parsing with the wrong goal. The default is CommonJS, and back compat means that can’t change if, and until after, the community largely settles on a different default.
It should not be possible to silently parse third-party code with any parse goals that the author didn’t explicitly choose - and the lack of an explicit signal definitively means CommonJS now, and for the foreseeable future.
guybedford commentedon May 31, 2019
We do need to start thinking about what the upgrade paths will be in shifting the default to modules, and I'd hope we can have a clear guide and proposal on what users can expect in the long term here.
I very much support having a proposal like this, and possibly even an implementation in order to allow those who are interested to start exploring these workflows. Ideally we should start coming up with recommendations for this process as well.
For example, I chatted briefly with @arcanis (Yarn maintainer), about if it might be suitable to be injecting a "type": "commonjs" on install into node_modules already now, as that is the kind of work that would allow us to flip the default as in this proposal in due course, without breaking code.
It is somewhat of a chicken and egg situation, but please lets get this discussion going.
hybrist commentedon May 31, 2019
Maybe a better intermediate flag would be "--require-package-type" which throws when an explicit package type hasn't been provided?
devsnek commentedon May 31, 2019
i don't want to ship a system where it needs some weird script that recursively patches already working code to run.
Even if the default system is esm, an extensionless file could be json or css or html etc, so this still wouldn't really be feasible.
weswigham commentedon May 31, 2019
?
bmeck commentedon Jun 1, 2019
arcanis commentedon Jun 11, 2019
Sorry I didn't get the bandwidth to jump in before! As we discussed I have a lot of concerns about doing something like this, as it would imo be a relatively flaky solution: it wouldn't work well with PnP (because then we'd have to modify the files from the cache), it wouldn't work for node_modules that already got created (due to cache systems), it would cause a change of behavior from one day to the next, ...
I would tend to agree with this statement. Imo we should always consider "no type field" to be CommonJS, and the package manager shouldn't take responsibility in deciding otherwise.
If this kind of logic was to be implemented, then the package registry would imo be the right place, as they're free to decide whatever rule they want to implement as to what is valid and what isn't. It would be possible for the npm folks to enforce that all packages published as of now must have a valid
typefield (this is assuming that the npm cli is able to properly print how to fix the problem).guybedford commentedon Jun 11, 2019
Can you expand on this a bit? I'm not sure I follow this logic.
This is exactly why we need to start doing this now to prep for the 5 year plan. Otherwise it could be more like a 10 year transition.
I agree, but we have to be practical here. If npm would implement this (and every other registry), then great. But otherwise how would we go about this?
Or shall we just let the 10 year transition play out on its own here?
ljharb commentedon Jun 11, 2019
A smooth long transition is much better than a shorter rougher one, imo. If everyone moving to ESM requires a push, then consider that it may not be the best end state (i believe everyone will and it is; but only with a smooth unforced migration path)
bmeck commentedon Jun 11, 2019
Any sort of bytecode caches or snapshotting would be affected, consider:
If we compile it as CJS and cache the compiled form (using things like vm.Script#createCacheData) we get a specific buffer back that we can use to speed up bootstrapping.
If we swap the translator used to be ESM:
I personally think we should expose capabilities to handle this as other global flags like
--preserve-symlinkscan have similar effects in this area now withpackage.jsonboundary scoped settings causing the mutations to be non-trivial.arcanis commentedon Jun 11, 2019
Conceptually, all packages belong to the same cache that is shared by all the projects. They aren't copied to the
node_modulesdirectory anymore. As such, we can't just replace the content from thepackage.jsonwithin the projectnode_modules- the only place where we can do this is inside the cache itself.It also goes both way: under the current circumstances this patch approach would have to be implemented by everyone for it to be possible. Otherwise Node would have to keep the default to commonjs anyway.
If the issue here is "how to make
--default-typeaffect the main package but not others", that seems like a reasonable compromise? Especially since it would be quite difficult to ask users to add--default-typeto all their JS scripts, but adding a configuration settings in thepackage.jsonis reasonable enough. It would have more design space as well, such as opening the way to support Node configuration within thepackage.jsonfile (which will be required for loaders anyway).The main concern would be with workspaces (since you'd then have to replicate the field everywhere), but since we're introducing constraints it would just be a
fixaway 🤷♀️bmeck commentedon Jun 11, 2019
ljharb commentedon Jun 11, 2019
That’s what type module already does in package.json, no?
GeoffreyBooth commentedon Jun 12, 2019
If we can get back to the original feature request, it was for a user to flip Node’s defaults to be ESM first rather than CommonJS first. The question is what that means in practice. As I wrote at the top, if we don’t want to confront the complication caused by extending this to dependencies, we can just provide an option that limits the scope to
--input-typeand the scope surrounding the initial entry point.GeoffreyBooth commentedon Jun 19, 2019
Just tested
--input-typeinNODE_OPTIONS:And it works. So really the proposed
--default-typewould only be needed to flip the default of thepackage.json"type"field. Per the 2019-06-19 meeting we’re deciding to instead encourage all package authors to include the field, for both ESM and CommonJS, rather than make it easier to leave the field out. This would mean that the field is with us forever, but that’s good in case we ever need another parse goal/module system type in the future.The only other case I can think of is non-
.mjsshell scripts or files outside of packages, likenode loose-file-with-esm-syntax.jsor/usr/local/bin/extensionless-javascript-file-with-hashbang. Users could put apackage.jsonwith{ "type": "module" }in their user or system root to flip their defaults that way. I guess that’s good enough for now, unless/until we get a concrete use case.