Feature: ESM in executable files

Coming from this comment: nodejs/modules#151 (comment)

It is now possible to create executable, extension-less, files:

#!/usr/bin/env node
console.log(__filename);

But it's not possible to enable ESM as parsing goal.

#!/usr/bin/env node -m
console.log(import.meta.url);

Even if there are OS incapable to parse the whole shebang up to the -m flag (or whatever flag will land in nodejs), there is no way to even define an executable that would like to parse the source file without any extension.

Example

Given the following esm file, reachable through /usr/local/bin or similar OS folder:

#!/usr/bin/env bash
node --module $1

And given the following executable:

#!/usr/bin/env esm
console.log(import.meta.url);

It should be possible to have extensions-less files parsable as ESM.

Update

There is a solution to the single file problem that would still require --module hook to bootstrap the file as ESM parse goal.

There are a variety of solutions we should probably evaluate and see if they are or are not feasible that I can think of.

Right now we can point executables without extensions to other files using symlinks. I'm not curious about what file extension maps to what format, but I am curious about if that disambiguation and/or indirection method is infeasible. I'd be curious about places that cannot support that behavior. Most installations put the .cmd/Symlink as an indirection layer to a folder and the main file for "bin" installations through package.json so this probably falls on if the file extension can be used, whatever it maps to.

In addition the executable could stay CJS and require/import() to another file to change the mode. This also is a method of indirection, but is not relying on file extension and instead relying on w/e disambiguation method is done at runtime, even if it uses the file extension it isn't mandating such.

I also am concerned about hashbang being unusable not just on linux shells but on windows. I agree it is a design space that we can look into but I am not sure how feasible it is. I do not think loaders can solve this problem using a hashbang either because they currently are not runtime configurable, nor am I convinced they should become runtime configurable due to this single issue. If we were to use a loader configuration it would need to be done outside of the file using CLI/ENV/package.json/etc. That makes me think we might want to look at indirection so that we can get that out of band data somehow.

I do have scaling concerns for having multiple executables because things like BinaryAST / WASM / WebPackage are other goals I'd like to support in the future and they would explode the number of executables that node ships with. I think however this format is configured and sent to node it should be planned around adding at least those 2 formats as well.

I do not think wrapping processes are a way to solve this problem either. Notably, Windows does not have an exec capability like unix/shells that can replace the same process ID, and using child_process would create wrappers that might be problematic for various process ID tracking service manager.

Forgot to add, that even though Loaders are not configurable, they could be done in a per package manner for things that are not installed using "bin" from package.json approaches.

Not sure I follow, but I've used Linux in pretty much every dev/production env I know and written dunno how many binary files based on the hashbang #!/usr/bin/env node.

I also am concerned about hashbang being unusable not just on linux shells but on windows.

NodeJS ignores hashbang too since ever because it's been a valid use case for long time so it shouldn't probably be under discussion the ability to create nodejs executable based on ESM, but I hope I've misunderstood your reply.

@WebReflection

@WebReflection that is executing the node executable as the shell that invoked the file parsed the hashbang (well env is invoked really). My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable. Even then, the declaration of the executable through the hashbang is picked up by the shell and not node itself.

My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable.

then I misunderstood, and indeed since it doesn't work in the only env I've used it (Linux) I guess we would need a better way.

I don't have ideas for now but I'll keep thinking about it.

If we want to avoid passing parameters in the shebang, we could create an excutable wrapper that passes them (nodem, node-m).

But from my point of view that would be confusing for people used to run node.

@xtuc

This is one of the benefits of the package.json "mode" proposal in that it can allow this scenario to be handled clearly (when executing the "bin", the package.json is loaded and used as source-of-truth of the format).

@xtuc's suggestion is an interesting alternative here too, although agreed being weary of adding to confusion.

Don’t forget the case where the extensionless file is outside the package folder, e.g. /usr/local/bin. For example, when CoffeeScript is installed globally, e.g. npm install --global coffeescript, it puts a symlink /usr/local/bin/coffee that points to /usr/local/lib/node_modules/coffeescript/bin/coffee (on Mac/Linux systems, anyway). Maybe in the case of followed symlinks Node can look for a package.json in the folder where the symlink resolves.

node by default uses the extension of the resolved file, not the symlink e.g. file.json -> file.js is seen as cjs not json

@xtuc

FWIW @xtuc suggestion is exact equivalent of my ./esm binary with content:

#!/usr/bin/env bash
node --module $1

This is a pragmatic solution to the shebang gotcha but it feels future hostile needing that in order to have an executable based on modern code so, unfortunately, we might need a better idea than just an indirection 😢

@WebReflection

@WebReflection whats wrong with the symlink indirection that most things do today?

@bmeck

@bmeck if I understand correctly that would require a package.json somewhere else, right? In AUR (or other packaging formats different from npm) that might not be easy/straight forward to implement.

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

@WebReflection

@WebReflection it would rely on some file that it points to being unambiguous, yea. Not necessarily using package.json, could be a file extension, etc.

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

I think this is where all of these are breaking down. Perhaps, the way that things are done today doesn't scale well, even at 2 possibilities we are seeing problems unless we define a different mechanism than today. Wait for BinaryAST and WASM entrypoints and we have 4 possibilities.

@bmeck

@bmeck I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?

Anyway, I forgot I have some hackery to start GJS so that this would be my solution:

#!/usr/bin/env bash
Function=Function//; node -m "$0" "$@"; exit

console.log('ESM');

Explanation

the shebang is the most widely compatible one
in bash, Function has no meaning but you can assign it to anything, including slashes //
the ; after slashes ends the variable assignment and bootstrap node -m "$0"
the rest of the arguments is passed along via "$@" and the bash program exits (after node exits too so nothing else will be executed / parsed / interpreted from bash)
the node will execute the file as module, it will ignore the shebang and it will also ignore the assignment of the Function to itself, or better, there won't be any side effect, and the comment will nullify bash.

All it's missing now, is this mechanism to bypass the file extension and enable the --module like parser so that import.meta.url or any other ESM related syntax would be valid.

Being able to signal a short cli script to use esm over cjs is important. For small, general purpose, bash-like, copy-paste-able, executable scripts like, for example:

#!/usr/bin/env node
import http from 'http'
const host = "0.0.0.0"
const port = 8000

const requestListener = function(req, res){
  res.writeHead(200)
  res.end("Hello World")
}

const server = http.createServer(requestListener)
server.listen(port, host, () => {
  console.log(`server is listening on port ${port}`)
})

I'd like to name this script 'hello-world' instead of 'hello-world.mjs` for it to run without error.

@mshiltonj

@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.

@ljharb

I'm a long term user of Node.js (since v0.4.0) and totally happy with require('module') syntax. It does the job fine, probably even better than ESM modules from a practical standpoint.

I do however DISAGREE with your comment @ljharb. By not supporting ESM module syntax in extensionless shebang executables you are not only forcing programmers to use the CommonJS require() syntax inside the bin file, BUT ON EVERY SINGLE DEPENDENCY from that file up the module tree.

If the problem was with a single file it would really be (as you point out) "a few users of a niche use case", but by making every dependency use CommonJS you are dooming extensionless bin files as a whole which are part of a lot of major Node.js libraries.

Just to pick a few examples: mocha, tape, semver, uuid, he, ncp.

Quoting NPM:

A lot of packages have one or more executable files that they'd like to install into the PATH. npm makes this pretty easy (in fact, it uses this feature to install the "npm" executable). To use this, supply a bin field in your package.json which is a map of command name to local file name. When this package is installed globally, that file will be linked where global bins go so it is available to run by name.

https://docs.npmjs.com/cli/v8/configuring-npm/package-json#bin

No? You can, in a CJS bin, async import any ESM module.

Extensionless bin files I mean. If you add the extension .mjs you can start using import (or even the .js extension with "type": "module" in the package.json)

Sure. But in CJS, you can always use import().

@bmeck

await import('module') syntax inside an extensionless bin file still produces the same issue.

Without "type": "module" it kicks off an error in the dependency chain.

import axios from 'axios';
^^^^^^

SyntaxError: Cannot use import statement outside a module

Using "type": "module" there is still a problem with the resolver.

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /path/to/binfile

Using import or import() or require() inside the dependencies is something I cant always control. So back to square one... 🤔 unless I understood you wrong.. and if that's the case please feel free to explain further.

On the other hand, it seems like @bmeck is still dedicating brain time to this (#42301).

@sdesalas

@sdesalas type module only determines whether your own .js files are ESM or not (it has no effect on things in node_modules). You can still use .mjs for ESM (and should).

@sdesalas

@sdesalas CommonJS doesn't have top-level await, so instead of this ext.mjs content:

#!/usr/bin/env node

const {default: axios} = await import('axios');

console.log(axios);

You should write something like this in a noext file:

#!/usr/bin/env node

(async () => {
  const {default: axios} = await import('axios');

  console.log(axios);
})();

if you have these excusable files in the same folder where node_modules/axios is installed, you won't have any issue.

@WebReflection

@WebReflection I used an async IIFE for the await part.

The problem was not with axios on the extensionless shebanged file, but on the dependency chain from its required modules. I'll write up a working example when i have a mo.

@sdesalas

@sdesalas the dependency chain shouldn't have any impact - altho CJS can't require ESM, it can dynamically import it, so any module format can be accessed from any module format, just not always synchronously.

Don't use ESM in executables

@mshiltonj

Hey all, I am new to this thread, but I am in the process of upgrading from Node 12 to Node 14 and thought it would be good to convert everything to ESM. However I am running into a bunch of problems, it seems like ESM is not fully baked.

One of our major problems is getting our .bin executables to run with ESM. I have tried using .mjs I have tried type: "module" but it always seems to have some problem. Is this just not possible currently? Is the current guideline to only use CommonJS in executables? If it is possible is there a fully baked example to refer to? For example, in the package.json do I need to specify the .mjs extension in the key? The examples in the doc do not specify.

@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.

Build Script

Navigation Menu

Repository navigation

Feature: ESM in executable files #49444

Example

Update

Explanation

49 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature: ESM in executable files #49444

Description

Example

Update

Activity

bmeck commented on Jul 12, 2018

bmeck commented on Jul 12, 2018

WebReflection commented on Jul 12, 2018

bmeck commented on Jul 12, 2018

WebReflection commented on Jul 12, 2018

xtuc commented on Jul 15, 2018

guybedford commented on Jul 15, 2018

GeoffreyBooth commented on Jul 16, 2018

devsnek commented on Jul 16, 2018

WebReflection commented on Jul 17, 2018

bmeck commented on Jul 17, 2018

WebReflection commented on Jul 17, 2018

bmeck commented on Jul 17, 2018

bmeck commented on Jul 17, 2018

WebReflection commented on Jul 17, 2018

Explanation

49 remaining items

mshiltonj commented on Aug 4, 2021

ljharb commented on Aug 5, 2021

sdesalas commented on Mar 15, 2022

ljharb commented on Mar 15, 2022

sdesalas commented on Mar 15, 2022

ljharb commented on Mar 15, 2022

sdesalas commented on Mar 15, 2022

ljharb commented on Mar 16, 2022

WebReflection commented on Mar 16, 2022

sdesalas commented on Mar 16, 2022

ljharb commented on Mar 16, 2022

rmclaughlin-nelnet commented on Jun 13, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions