Repository navigation
[RFC] Import query params #3477
Description
Problem Statement
Currently in Parcel v2, dependencies are resolved to files and the resulting file paths are matched against the configured globs in the .parcelrc file to see which pipeline the file should be transformed by. This works for most cases, but sometimes you might want to force a dependency down a specific pipeline. For instance, let's say you are building documentation for a React component. By default, importing the React component will bundle the component as a JavaScript module, but what if you additionaly want the component's source as plain text? Maybe you also want to run some transformer over the component that extracts what prop types it expects and bundle that as well.
A very naive solution that would work for any and all tooling would be to just create files for the source text and prop types in a prebuild step and then you can import three individual files. The downside to this approach is that you would then have two build steps, which means you have two build commands that you have to string together and two separate watchers if you want to build in watch mode. Imagine in this scenario you updated MyComponent.js. Both the prebuild and the bundle watchers would pick up the change and start rebuilding. Then when the prebuild step finishes, the derived files are updated, which causes the bundler to rebuild yet again. It makes for a much better developer experience if this can all be handled in one process.
With webpack, this is usually solved by using inline loaders.
import MyComponent from './MyComponent';
import MyComponentSrc from '!!raw-loader!./MyComponent';
import MyComponentPropTypes from `!!extract-react-types-loader!./MyComponent`;One downside to this approach is that the detail that you are using webpack leaks into your source code. Ideally this could be done in a more bundler agnostic way.
Proposed Solution: Import Query Params
The ECMAScript modules standard (ESM for short) is now supported across all modern browsers and will soon be supported by Node.js as well. One neat thing about ESM is that it essentially treats all module specifiers as URLs, which means adding query parameters is perfectly valid syntax and not something bespoke to a specific bundler. This seems like a great way to specify how you would like a dependency loaded:
import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js?as=raw-text';
import MyComponentPropTypes from './MyComponent.js?as=react-types';So what would it take for Parcel to support this? The most straightforward approach requires no changes to Parcel's configuration so it's got that going for it. All you would have to do to force a file to be transformed by a different pipeline is add a query param to the module specifier and configure a pipeline in your .parcelrc with a glob that matches it.
{
"transforms": {
"**/*.js\?as=react-types": ["parcel-transformer-extract-react-types"]
}
}The only change we would need to make in Parcel would be to slice off the query params from the module specifiers to resolve the file and then tack them back on to resolve the transformation pipeline. Perhaps it would be nice to have a special glob matcher that makes it easier to write globs of this type. Query params could also be passed on to transformers as well.
import profilePic from './profile.png?width=300&height=300';Drawbacks
- Since this solution is based on ESM it makes it pretty JavaScript centric. Some other languages and module formats support module specifiers as URLs, but what about the ones that don't? Parcel will either end up encouraging developers to write non standard code that only works because it's run with Parcel or this behavior may not end up being supported in all types of files.
- Text matching can get hairy. It seems possible that this could cause Parcel config files to balloon especially considering there could be multiple query parameters that might influence what pipeline a file should be transformed by.
- This is more of a drawback for the transformation configuration model in general, but one thing it doesn't accommodate is influencing what pipeline to use based on the file that defined the dependency. For instance you might want something different when importing
as=urlfrom a JavaScript file vs a CSS file. This could be solved by using a more specific parameter likeas=js-url, but the problem still remains for imports without query parameters. If we decided to change the configuration to allow for this type of thing then it could potentially lead to a cleaner/smaller/simpler config.
Alternatives
One alternative would be to make a change to Parcel's configuration. I do like how simple Parcel's configuration is at the moment, but I could imagine this not working perfectly for all cases. I'm not aware of any of these cases at the moment, but that's what the RFC is for. If no one brings up a compelling case that proves that a change to configuration would either be necessary or would be a much cleaner solution, then this seems like a reasonable place to start.
Another alternative could be to use a special extension like extensions leading with a certain number of underscores.
import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js.__raw';
import MyComponentPropTypes from './MyComponent.js.__react-types';Parcel could split the module specifier like it would have to with query parameters and use the parts before the special extension to resolve the file and tack the special extension back on to resolve the pipeline. You could also have a prebuild step that actually creates these files which means the imports would still work with other tooling. One potential issue is that the special extension might conflict with legitimate extensions.
Open Questions
- ESM treats module specifiers as URLs but CommonJS does not. Should Parcel only support this feature on module specifiers that are known to be URLs or should it be less strict?
- Should this be supported in
node_modulesor only source code? - What happens when the configured transformer changes the type of the asset? Do we use the query parameters to match the next pipeline or do they only apply to the first pipeline? (Seems like you would probably have to use them to match if you wanted any control over whether to jump to a new pipeline or not)
- Any examples where this type of configuration doesn't cut it?
Activity
mischnic commentedon Aug 30, 2019
I think we shouldn't encourage publishing packages with this (currently) non standard code, at least in the beginning.
devongovett commentedon Aug 30, 2019
I think we should only support query params for specifiers that are URLs, so for JS, only ESM. CSS and HTML are all URLs, but langs like SASS and LESS may not be. Some research required. CJS shouldn't be supported -
?is a valid character in a filename.Agreed with @mischnic.
I think it should only apply to the first pipeline, and should be stripped off afterward.
devongovett commentedon Aug 30, 2019
I think in order to make the config easiest to use, we shouldn't require escaping the query params. We can build a custom matcher, that strips off the query param first and matches the glob on the left side, and then checks the query params separately if it matches.
devongovett commentedon Aug 30, 2019
It would also be pretty nice if config like this worked:
{ "transforms": { "*.png": ["parcel-transformer-png"], "**?as=url": ["parcel-transformer-js-url"], "**?as=base64": ["parcel-transformer-js-base64"] } }Currently, that wouldn't work - for a PNG file, only the
.pngpipeline would run, and not the url or base64 pipeline. It could be made to work if you configured the order differently:{ "transforms": { "**?as=url": ["...", "parcel-transformer-js-url"], "**?as=base64": ["...", "parcel-transformer-js-base64"], "*.png": ["parcel-transformer-png"] } }The
...would cause the next matching pipeline (*.png) to run first, and then continue with url or base64. But that's kinda confusing vs the first.Any ideas on how we could make this work?
rtsao commentedon Aug 30, 2019
One alternative is to expose a Parcel API for conveniently allowing transformations to write files to disk. This way it's still a single process with a single command and single file watcher, but there's zero "magic" in the JS module bundling phase as only JS files are importing other JS files (as opposed to arbitrary file types). This seems the most bundler agnostic solution, as any other bundler could pick up these plain JS files.
One huge advantage to on-disk code generation is generated code can be statically typed. Type checkers are not going to be able to understand query parameters. However, the transformer could output a file on disk that any type checker will be able to understand.
The main potential drawback I can see is passing configuration at the import site might not be possible (as the files would have to exist already) -- so the config would have to be centralized. This might not necessarily be bad, though. The other drawback is potentially extra work compiling files before they are used. However, this might be avoided using conventions around filenames where transformed files are only generated on-demand.
jakearchibald commentedon Sep 2, 2019
My opinions here may not fit with what Parcel is trying to achieve, so feel free to ignore.
I don't think build systems should hide. If a file is to be transformed in an uncommon way, it should be obvious from the import that something special is happening.
The above doesn't really look special. It's just a regular URL.
If the above is passed to a system that doesn't recognise the 'as' query parameter, it'll interpret the module as JavaScript, which feels like a bad fallback. If the transform doesn't have an 'acceptable' fallback (if a file fails to minify, the result still works), the import should fail as soon as possible.
Other ideas:
The
as-textscheme is invalid in module identifiers, so it's obvious something special is happening. Without the transform, this will fail at the parse stage.This treats
whatever.jsas a directory, when it isn't. Transforms would define what a file could 'contain'. Without the transform, this will fail at the fetch stage. That might be too late, and the special nature of the import may not be obvious enough.I'm not sure if files within
node_modulesshould support this syntax. However, I think files outside ofnode_modulesshould be able to use it on files withinnode_modules.Eg:
devongovett commentedon Sep 2, 2019
That's interesting feedback @jakearchibald! One reason we were thinking about query parameters was to avoid making
as"special". It's just an option that is passed to transformers, like other query params (e.g. image width/height). It could be calledtypeorbananafor all Parcel core cares.If we went with something more specific like a URL protocol, it would also affect resolution in addition to transformation and would need to become a bit more integral to Parcel core. But I can see your point about fallback in case a build system isn't configured to use
asvs failing early. We can think about that further.As for the specific suggestions, protocols seem like an interesting idea, but feel a bit like a "source" rather than a transform. e.g. normal protocols like
http:,file:, and evenstd:modules are all sources - you cannot request the same file from different protocols generally. The directory idea is hard as well because it requires cooperation from the resolver plugin as well as the transformer.jamiebuilds commentedon Sep 4, 2019
There's a good point made that "falling back" to JavaScript when
?as=textis not supported, is actually worse than not resolving to anything. If you're going to fail, fail early.A couple constraints:
url(...)so push people that direction.protocols://because we allow importing URLs in many forms!raw-loader!is a bad solution because it specifies the transformer directly.I was in favor of query parameters before, but @jakearchibald made a good point that unknown query parameters rarely change how a route is resolved by a server.
Right now I'm leaning more towards the additional extension:
This has the benefit of reusing existing configuration:
{ "*.{txt,text}": ["@parcel/transformer-raw"] }The problem comes in trying to resolve it. How do you decide
./file.js.txtshould resolve to./file.js?The answer could be
stat-ingbasename(name)recursively until you resolve to something, but I haven't seen anything that behaves like that before.jakearchibald commentedon Sep 5, 2019
@devongovett
The only exception I can think of is
view-source:http://example.com/which a few browsers do.@jamiebuilds
Yeah, that could work. Would it be chainable? Say I had two transforms:
…would I be able to do:
jamiebuilds commentedon Sep 22, 2019
For that specific case I don’t think we’d encourage the use of optimizers via file extension vs Parcel’s config format:
{ "transforms": {...}, "optimizers": { "*.png": ["@parcel/optimizer-pngmin"] } }However for the generic case of chaining I’m a little bit torn.
There is some precedence for this in build systems as well, Ruby on Rails’ asset pipeline uses extensions to turn on different file transforms (although that’s at the file system level, not at import-time).
We’d have to consider how the Parcel pipeline interacts with this:
If the config were to look something like this:
{ "transforms": { "*.png": ["@parcel/transform-png"], "*.img-data": ["@parcel/transform-img-data"], }, "optimizers": { "*.png": ["@parcel/optimizer-pngmin"] } }There’s a lot of questions about which plugins run and when they run:
*.pngmatch.png.img-data?.js.raw-sourcewould you want Babel to run first? (Probably not)I’m not sure we can come up with a pipeline algorithm that will work for all cases. In which case I’d prefer forcing users to do a little more work to be explicit than to have them be frustrated with a bad abstraction.
jamiebuilds commentedon Sep 22, 2019
Vague late night thoughts:
view-source:http://...but I think it implies it's a standard of some sortloader!./filesyntax would be nice here, but I explicitly want to use a different syntax than them to avoid feature confusion.Some ideas on syntaxes other than
loader!that don't conflict with the URL standard or what I've seen in any programming language import specifiers, and which will fail to load if you try loading them outside Parcel:Example
{ "transforms": { "*.js": ["@parcel/transform-babel"], "*.png": ["@parcel/transform-png"] }, "optimizers": { "*.js": ["@parcel/transform-uglify"], "*.png": ["@parcel/optimizer-pngmin"] }, "viewers": { "image-data": ["@parcel/viewer-image-data"], "raw-source": ["@parcel/viewer-raw-source"] } }Morgul commentedon Sep 22, 2019
Just a random thought.
Personally, I dislike having to change my import statements because of a bundler. Even the query parameter is annoying, because it feels like I'm mixing the concern of loading modules with the concern of bundling code. They're related, but should be more explicitly separate, imho.
Combined with the concern that no all languages support urls for their module imports, the solution seems pretty clear:
Here's the same examples from the OP:
Simple, easily understood, not bundler specific, and supported in all languages with comments. (Syntax obviously up for debate, that was with about 10 seconds of consideration.)
devongovett commentedon Sep 22, 2019
I still think query params are by far the best option here, even with the concerns raised.
giraffequery param for all Parcel core cares - it's up to transformer plugins to support options, and the user to configure their pipelines in.parcelrcif necessary.The fallback issue when switching tools is valid, but honestly not that bad I think. You'll already have other problems in anything complex. There are many cases where behavior of imports relies on specific configuration.
import './foo.png'in a tool that isn't configured to load PNGs, and it'll either try to process it as JS directly, or maybe give you back a URL or base64 string. Depends on how the tool is configured. You need to configure your whole toolchain to support processing images in order to fix it. Query params are the same thing.I think we should try to keep this as simple as possible and not try to make Parcel even more complicated to support this one feature. Query params can be widely used for many purposes, including changing output type or passing other options (e.g. image resizing). Supporting them is simple, and opens up the doors to many features without much added complexity.
jamiebuilds commentedon Sep 23, 2019
I strongly disagree with that.
We already should be supporting query parameters for other purposes, if someone specifies a file
https://example.com/image.png?width=100&height=100then we should be respecting that and fetching the file as written.By assigning a query parameter, we're changing the meaning of query parameters. What if someone actually means to specify a
?as=query param? We would have to strip it out because we will inherently have a different meaning than they give it in their API.Not to mention that using an existing standard would be harder to unwind than something non-standard. Failing early is far better than failing softly or silently. Making it easier for users to debug ahead of time.
It costs us nothing to have the syntax be something other than query parameters. Having it be query parameters does come with a cost.
It wouldn't try to do any of those things, unless you're talking about Parcel in which case it also should not do any of those things, it should fail the same way it's meant to fail:
Module loading is incredibly complicated today because of a whole bunch of early decisions that were not all that well thought out. Having been one of the people (on Babel) that helped make decisions that directly led to the ecosystem having a ton of "problem code" today, I really wish we had done things differently.
There are a lot of things Parcel can make short term decisions with, but one of them should not be module loading or user code, because it sticks around for a very long time.
There is a bigger problem here than parameters to transformers though. Something like
?as=raw-sourcerequires a completely different pipeline than whatever it's original extension was. And that pipeline looks completely different from what?as=image-datashould be.There are two problems:
A. Passing parameters to transformers.
B. Modifying the pipeline that a particular import goes through.
I agree that both need to be solved, but:
If you solve A before you solve B, you will have to support two things. But if you solve B and A at the same time, you can have a unified solution.
Not to mention that query parameters decentralize your pipeline from out of your config file and plugins and into your code.
devongovett commentedon Sep 23, 2019
I don't think it's changing the meaning at all.
aswouldn't be a special parameter in Parcel core, just what a plugin has implemented. Nothing is stopping transformers from implementing query parameters that return different types of assets depending on the value. For example, the raw transformer could implement support foras=base64andas=textin the same plugin based on the query param. This would work even without modifying the pipeline config at all.So, if we add another syntax for this, we'd effectively be supporting two ways to do the same thing.
jamiebuilds commentedon Sep 24, 2019
But that is changing the meaning. Query parameters, and module specifiers in general only affect the way a module is resolved. There is currently no standard for transformations and if one existed it would not be based on query params because it would be a backwards-incompatible change.
The only question you need to ask is “If I removed Parcel from the equation would this fail immediately or potentially quietly/silently.”
That’s not how the pipeline config works though. If I import
file.js?as=raw-sourceit’s gonna match*.jsunless the config file also specifies*?as=raw-sourcein which case it does require a modification to the pipeline config and it does not give the raw transformer control over the query param.The only alternative I can come up with is specifying raw transformer to match
**and fall through to other transformers if it doesn’t have the query param, but now you’re breaking the design contract of.parcelrcand you’re loading in a transformer on every file.People are always going to find ways to use Parcel in ways we did not intend or want them to. There are dozens of Babel plugins that I wish didn’t exist.
But that’s an argument for being more restrictive and provide more guidelines. Letting people hack away in whatever way they please is only going to make it harder to make changes to Parcel in the future.
jakearchibald commentedon Sep 24, 2019
Something came up at TPAC that feels relevant, but sorry if this is just noise:
There's a proposal to make this work in browsers, where
whatever.jsonis parsed as JSON and the resulting JS object is imported. But, this creates a situation whereexample.comcan change the content type ofwhatever.jsontotext/javascript, and now they're executing JS.It wasn't resolved at TPAC, but to me it feels like the importer should have primary control over how the resource is processed, not the server. Something in the import statement needs to say "treat this as JSON", and it can't be a syntax that, if JSON imports are unsupported, falls back to "treat this as JS".
@devongovett
It comes down to whether build tools should behave like JS or CSS. In JS, if you call an unrecognised function, it throws. In CSS, if you set an unrecognised property, it's a no-op.
devongovett commentedon Sep 24, 2019
@jamiebuilds
img.png?width=100&height=100doesn't affect resolution, but transformation. In this case it just doesn't change the type of asset being returned.Servers can implement an
asparameter today and return different things if they want. Not sure why parcel is any different. Just pretend transformer plugins are servers. If the server changes what the query params it supports mean, then it'll break.jamiebuilds commentedon Sep 24, 2019
If you fetch that URL it does resolve to an asset, the server might be behind the scenes “transforming” that, but from the client you are only aware of resolution — and that resolution is important for the client for caching.
Except Parcel and the server can conflict.
If my server wants
?as=foo|barand parcel only allows?as=baz|batthen we now need to do some parcel magic to get it all working together.Exactly, and it will probably break quietly or silently which is why well designed APIs will intentionally make it loudly fail when someone passes a deprecated/removed query param.
We don’t have the ability to do that, if someone uses a module relying on Parcel-specific query params outside of Parcel it’s going to fail quietly or silently. If they use something like
raw-source~file.jsit will fail immediately every time.devongovett commentedon Sep 24, 2019
Yup, and they'll likely have no recourse but to change their code, thus making it incompatible between tools. At least with query params there is some chance of other tools implementing them. I really don't want to be in the business of inventing syntax with Parcel.
devongovett commentedon Sep 24, 2019
FWIW, there was a discussion in create-react-app a while back about this: react/create-react-app#3722. @gaearon made a proposal to use named imports for this. Not sure where it ended up, but appears that query params were also proposed as an option in the thread. Would be useful to get some buy in from other tools for whatever we move forward with.
jamiebuilds commentedon Sep 24, 2019
Other tools could also implement a
viewer~specifiersyntax. That is not limiting. But we need to design something that would get more consensus, which query params will not.There’s still always the possibility that a standards body will want to get involved here, there’s been talks about having “loaders” for years now.
By designing something more robust we can have more influence on the design from standards bodies. But if we design something with obvious flaws, then it will be discarded.
kwelch commentedon Sep 24, 2019
I am sort of intrigued by the idea of protocol again with the combination of viewers. Using a protocol like
view-*:, could sort of blend the line between standard and new.view-source:./foo.jsview-data:./butterfly.pngIt would be nice to also be able to select if optimizers are ran or not.
devongovett commentedon Oct 10, 2019
FYI I implemented "named pipelines" in #3613 using a protocol-like syntax. It's not implemented in import specifiers yet, only in
.parcelrc. For now, it's used to run a different pipeline based on the target (e.g.main,module,types).devongovett commentedon Nov 20, 2019
Named pipelines are implemented in import specifiers now. Query params for options are not yet implemented.
svr93 commentedon Jan 8, 2020
I think it will be good to use a single syntax for the whole JS ecosystem so I suggest combining different solutions into one - tc39/proposal-import-attributes#22