This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
A couple months ago, I asked about creating a module-manifest file while targeting the ESM format. Apparently module-manifests are a :browser
format feature, which is nice, and I thought that made sense since maybe ESM modules don’t need a manifest.
Recently though, I’ve noticed that if I use shadow-cljs for configuring module-splitting, it works for ESM modules, but there’s no manifest file for determining which split-modules depend on other split-modules. And it’s important to know the dependencies so I can load certain split-modules before others.
For example, if I created a “commons” split-module, that contains everything shared amongst two other modules, then I would like a manifest to tell me that each of the other two split-modules depend on the commons split-module. Otherwise I’ll need to track dependencies by referencing the shadow-cljs.edn
or something.
Can this be resolved by generating a module-manifest? Or should I try configuring module-splitting with a separate tool if I’m using ESM modules?
esm files can load their dependencies themselves, so you just load the file you need and don't worry about the others
even loading them in the wrong order is no problem, as they'll all sort themselves out
For example, in production, it’s common to bundle stuff into one chunk, but sometimes you want to split it up into more chunks.
Ahh I think I’m seeing the results your describing. I thought that Shadow-CLJS wouldn’t include the import statements to the dependent chunks inside a split-module. So I was confused as to how it would know to load the dependencies
the :browser
builds can't do that, so you need manifests and support from the html side
Will shadow-cljs automatically create a module-split if it sees a dynamic import? Or are splits only configured through the shadow-cljs.edn file?
okay great! I think that’ll have the least amount of surprises for me haha. Thanks again for the help 🙏
Okay so I’m noticing something odd with the format for source-maps created by shadow-cljs. It seems we use this format as described here: https://github.com/tc39/source-map-spec/blob/main/source-map-rev3.md#index-map-supporting-post-processing
In that format we store the map
field inside of the sections
field. But it seems tools like Vite don’t check for source-map data that way and will instead look at the root of the source-map for the map
field.
From what I can tell, it seems the source-map v3 supports both styles, but maybe the un-nested one is more popular (?). Is there a reason we use the nested version instead?
the sections version is used because the final artifact consists of many files concatenated together
what are you doing with vite? I mean processing the files with two build tools is generally not the best idea 😛
I'm using NCC and have similar challenges. Deploying to AWS lambda. Would be great if exceptions could report accurate source stack traces
My use case is to use the dev server from Vite with output files from shadow-cljs. So Vite loads everything but shadow handles cljs stuff and Vite handles JS + CSS stuff. This setup does seem to work most of the time, but source maps are assumed to be structured differently (for some reason) in Vite. And I agree that processing files in multiple steps is odd, but it is suppose to work since shadow supports the use of js bundlers for handling js dependencies. That’s basically the use-case but atm the js bundler (Vite) expects different source maps.
can you setup a repo that demonstrates the setup and the problem? maybe I can figure something out
Happy to do so. Should be able to tomorrow. @U04H1SFLY5Q will you do one for vite?
Sure thing! I have something that I’ll send soon, just need to clean it up and make a readme for reproducing the issue.
Just wanted to add more context here (still planning on sharing some code):
• I double checked the source code of Vite and it seems they very much assume a certain structure to the sourcemap file.
◦ Here’s a link to part of their source that parses the sourcemap: https://github.com/vitejs/vite/blob/23ef8a1a7abdb4a7e0400d7dd6ad3f7d444c548f/packages/vite/src/node/server/middlewares/transform.ts#L85-L94
◦ Here’s another link to a function that throws an exception because the sourcemap is shaped differently than how Vite expects it to be: https://github.com/vitejs/vite/blob/6868480d0036f08388e82611992d58ee52cf97b7/packages/vite/src/node/server/sourcemap.ts#L103-L107
▪︎ Specifically the expression map.sources.length
because sources are not defined at the root level of the sourcemap data.
◦ Here’s the rough shape of the data that they support for a sourcemap: https://github.com/rollup/rollup/blob/15d321bf7da5d48ed9a8ed9f87d7f88736ce837d/src/rollup/types.d.ts#L62-L71
▪︎ This is based on Rollup’s definition of a sourcemap, which doesn’t seem to be aware of sections
• I also looked around for alternatives to Vite/Rollup, and found that esbuild also doesn’t support sourcemaps with sections: https://github.com/evanw/esbuild/blob/02dae18a70b3d26dcb5b9599f02a9a36d88f809d/internal/js_parser/sourcemap_parser.go#L38-L41
• I’m looking into some more alternatives to try out, maybe webpack or something supports this properly, but it’s little troublesome that so many JS tools don’t support this part of the sourcemap spec already 🤷
note that you can use :target :npm-module
instead. there each file has a single source map
I appreciate the suggestion, but I thought that was deprecated 😿 ? Or at least we had discussed that using ESM is probably the way forward right?
well ... the recommendation changes based on what you are actually doing. I'm mostly guessing what you are doing
there are instances where :npm-module
will integrate easier with JS tools (e.g. source maps)
well I guess i’m observing that shadow is not at fault with these issues with esm support, but JS tooling just follows part of the spec (for some reason).
I also talk too many people with too many questions .. so I can't quite remember what we talked about last time 😛
I did find some good news though, it seems webpack’s sourcemap handling does support sections somehow: https://github.com/webpack-contrib/source-map-loader/blob/5e265904edf5c04d8dc209755926441a2732fc1e/src/index.js#L89-L92
if you want to write such a function for CLJ I'm happy to make use of it in shadow-cljs, but writing such a function seemed very low priority for myself
Well I can try testing it out on my side and see if this flattenSourceMap
works. Ideally, Vite and ESbuild and whoever would implement this ahah, but who knows if I’ll get that merged everywhere. Otherwise, I’ll share some CLJ code for the same function and it can be reviewed if it’s worth it or not.
Maybe it can be an optional process when you’re combining into a larger build with NCC or Vite. And it wouldn’t be used if shadow is the final build step.
Okay after some time-traveling, I’ve found the original webpack-related issue that asked for “Indexed Sourcemaps” (or sourcemaps with sections) support: https://github.com/webpack-contrib/source-map-loader/issues/89 • This PR used the source-map library by mozilla for handling this stuff, so the original support was included there and extended to webpack stuff: https://github.com/mozilla/source-map/issues/16 ◦ Although the entire spec is still not supported yet: https://github.com/mozilla/source-map/issues/437 • What’s interesting though is that the webpack related changes were requested by a ClojureScript user! This makes me think we’re on the right track here, but who knows aha 😸
Yeah it’s probably used elsewhere, since it’s valid piece of the spec. I think it’s sorta interesting to see how it’s been (or not been) implemented though in these bundlers. Maybe this history gives more credibility to opening issues in Vite or Esbuild, since webpack “fixes” the issue, maybe they will too
well generating source maps and consuming them are two entirely different subjects. so I can totally see webpack even generating source maps with sections
the closure compiler isn't exactly great at consuming input source maps either, just doesn't seem like a high priority often
Yeah I imagine sections sourcemaps are created in other places still, I wonder if Sentry supports soucemaps with sections (probably do) since they’re likely a huge consumer of sourcemaps
here’s a repo for the NCC use-case https://github.com/stevebuik/shadow-cljs-ncc-source-maps
in a related issue, I just hit this in my client app https://github.com/google/closure-compiler/issues/2731 I’m pretty sure it’s because we upgraded our version of https://remirror.io/ so I might try your vite bundling config to workaround this when you upload it
the error I get is “Class in a non-extractable location with ES2022 features’ is not yet implemented”. trying to use the :js-provider :external workaround but it doesn’t seem to change anything. will wait to see the vite sample
@U0510KXTU in your example config ncc
doesn't seem to make any use of the input source maps at all. even with a flat one, so nothing for me to do.
But even before NCC is involved, the shadow release JavaScript also lacks source info in exceptions
So then I/we need to learn how NCC can use shadow source maps. You can see the flag enabled when invoked
yes, generation. that is an entirely different subject. it can generate just fine, for the sources it generates
but it doesnt "consume" the input source maps, i.e. the source maps generated by shadow-cljs do map back to CLJS
ncc uses webpack internally, so the sections
issue doesn't apply here since webpack definitely supports that
Error: don't know how
at new Je (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/cljs/core.cljs:11623:11)
at Function.Le.N (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/cljs/core.cljs:11655:5)
at Le (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/computer.cljs:4:1)
at [eval]:1:33
at Script.runInThisContext (node:vm:122:12)
at Object.runInThisContext (node:vm:298:38)
at node:internal/process/execution:83:21
at [eval]-wrapper:6:24
at runScript (node:internal/process/execution:82:62)
at evalScript (node:internal/process/execution:104:10)
for
node -r source-map-support/register -e "require('./public/computer.js').call(null,'how?')"
So webpack does support input source maps, but you’ll need to configure the plugin for webpack. So if NCC doesn’t do that by default then input source maps may not be supported.
This plugin is for input source maps : https://webpack.js.org/loaders/source-map-loader/
I could just zip the whole node_modules dir but that will probably slow down the aws lambda cold start so not ideal
shadow-cljs can do some bundling for the node targets, so if you only use basic npm packages that might be enough
since I’m also working on the client side and trying Vite there, I might try Vite for the server side as well. then I can benefit from Seans work
tbh source maps in errors is nice to have for me currently. the client side problem is a blocker so I’m going to focus on that first
when I do I might start a new thread to ask for your help on that but thanks for this feedback already. it’s pointing me in directions I can understand
@U04H1SFLY5Q are you client or server side for your problem?
looked into aws lambda nodejs and it doesn’t appear possible to provide node runtime flags so enable -r for my runtime. annoying but still happy to learn about this
I’m experimenting with both client and server side stuff, but my initial source-maps issue came from some client stuff. here’s a small repro of the source-maps issue with Vite + Shadow: https://github.com/seanstrom/cljs-dev-template/tree/minimal-sourcemaps-debug
If you’re interested in bundling for server stuff, I have some more experiments in this branch: https://github.com/seanstrom/cljs-dev-template/tree/client-server-template
Take a look at the package.json scripts for some of the ways we can bundle with esbuild
and then use pkg
to make a binary.
I just noticed in Intellij you can click play on the cli commands in the readme. such a nice way to work through a repro
I’m interested in both side. thanks for the examples. I’ll learn from them tomorrow. dinner time now so signing off. thanks to you both
now that I’ve had a chance to think about this, since NCC doesn’t support using source-maps as input, I’m going try not using NCC. instead I’ll try zipping node modules and see if the size is reasonable/fast enough for aws lambda cold starts. if so, then I can use the “require” technique @U05224H0W points out. that would be a great result because the source line numbers are then visible enough to be useful. hopefully zipping node_modules isn’t crazy big. npm install --production may help
if the packages are simple enough you can try https://github.com/thheller/shadow-cljs/issues/290#issuecomment-459818765
Small update:
I’ve formerly requested for Vite to also help out here with some sections
support in the source-maps: https://github.com/vitejs/vite/issues/14573
I’ve also started patching Vite in my own fork, but I’ll be fighting JS/TypeScript interop for a little bit.
I also asked esbuild for sections
support, and I’m waiting to see what they say: https://github.com/evanw/esbuild/issues/3439