Fork me on GitHub
#shadow-cljs
<
2023-10-08
>
seanstrom07:10:44

A couple months ago, I asked about creating a module-manifest file while targeting the ESM format. Apparently module-manifests are a :browser format feature, which is nice, and I thought that made sense since maybe ESM modules don’t need a manifest. Recently though, I’ve noticed that if I use shadow-cljs for configuring module-splitting, it works for ESM modules, but there’s no manifest file for determining which split-modules depend on other split-modules. And it’s important to know the dependencies so I can load certain split-modules before others. For example, if I created a “commons” split-module, that contains everything shared amongst two other modules, then I would like a manifest to tell me that each of the other two split-modules depend on the commons split-module. Otherwise I’ll need to track dependencies by referencing the shadow-cljs.edn or something. Can this be resolved by generating a module-manifest? Or should I try configuring module-splitting with a separate tool if I’m using ESM modules?

thheller14:10:58

esm files can load their dependencies themselves, so you just load the file you need and don't worry about the others

thheller14:10:05

so no manifest is needed

thheller14:10:51

even loading them in the wrong order is no problem, as they'll all sort themselves out

seanstrom14:10:36

What if you intend to module-splitting that groups modules into 2 or 3 chunks?

thheller14:10:14

any configuration will work

seanstrom14:10:14

For example, in production, it’s common to bundle stuff into one chunk, but sometimes you want to split it up into more chunks.

thheller14:10:11

I'm still not sure what you are asking

seanstrom14:10:40

Ahh I think I’m seeing the results your describing. I thought that Shadow-CLJS wouldn’t include the import statements to the dependent chunks inside a split-module. So I was confused as to how it would know to load the dependencies

thheller14:10:17

yeah thats the good thing about esm, it knows how to load other esm files

thheller14:10:31

the :browser builds can't do that, so you need manifests and support from the html side

seanstrom14:10:06

pretty neat! definitely feels great to have it know what to load by itself

seanstrom14:10:46

Will shadow-cljs automatically create a module-split if it sees a dynamic import? Or are splits only configured through the shadow-cljs.edn file?

thheller14:10:49

only the configured ones, no automatic splits

👍 1
seanstrom14:10:44

okay great! I think that’ll have the least amount of surprises for me haha. Thanks again for the help 🙏

seanstrom15:10:32

Okay so I’m noticing something odd with the format for source-maps created by shadow-cljs. It seems we use this format as described here: https://github.com/tc39/source-map-spec/blob/main/source-map-rev3.md#index-map-supporting-post-processing In that format we store the map field inside of the sections field. But it seems tools like Vite don’t check for source-map data that way and will instead look at the root of the source-map for the map field. From what I can tell, it seems the source-map v3 supports both styles, but maybe the un-nested one is more popular (?). Is there a reason we use the nested version instead?

thheller15:10:13

the sections version is used because the final artifact consists of many files concatenated together

thheller15:10:49

i.e. each section is basically one file

seanstrom15:10:52

Okay that makes sense 👍

seanstrom15:10:12

I’ll try asking the Vite people to update the code to support both somehow

thheller15:10:44

what are you doing with vite? I mean processing the files with two build tools is generally not the best idea 😛

steveb8n19:10:30

I'm using NCC and have similar challenges. Deploying to AWS lambda. Would be great if exceptions could report accurate source stack traces

steveb8n19:10:19

Would switch to vite if that works for lambda

seanstrom05:10:41

My use case is to use the dev server from Vite with output files from shadow-cljs. So Vite loads everything but shadow handles cljs stuff and Vite handles JS + CSS stuff. This setup does seem to work most of the time, but source maps are assumed to be structured differently (for some reason) in Vite. And I agree that processing files in multiple steps is odd, but it is suppose to work since shadow supports the use of js bundlers for handling js dependencies. That’s basically the use-case but atm the js bundler (Vite) expects different source maps.

thheller06:10:06

can you setup a repo that demonstrates the setup and the problem? maybe I can figure something out

steveb8n07:10:06

Happy to do so. Should be able to tomorrow. @U04H1SFLY5Q will you do one for vite?

seanstrom07:10:04

Sure thing! I have something that I’ll send soon, just need to clean it up and make a readme for reproducing the issue.

seanstrom09:10:31

Just wanted to add more context here (still planning on sharing some code): • I double checked the source code of Vite and it seems they very much assume a certain structure to the sourcemap file. ◦ Here’s a link to part of their source that parses the sourcemap: https://github.com/vitejs/vite/blob/23ef8a1a7abdb4a7e0400d7dd6ad3f7d444c548f/packages/vite/src/node/server/middlewares/transform.ts#L85-L94 ◦ Here’s another link to a function that throws an exception because the sourcemap is shaped differently than how Vite expects it to be: https://github.com/vitejs/vite/blob/6868480d0036f08388e82611992d58ee52cf97b7/packages/vite/src/node/server/sourcemap.ts#L103-L107 ▪︎ Specifically the expression map.sources.length because sources are not defined at the root level of the sourcemap data. ◦ Here’s the rough shape of the data that they support for a sourcemap: https://github.com/rollup/rollup/blob/15d321bf7da5d48ed9a8ed9f87d7f88736ce837d/src/rollup/types.d.ts#L62-L71 ▪︎ This is based on Rollup’s definition of a sourcemap, which doesn’t seem to be aware of sections • I also looked around for alternatives to Vite/Rollup, and found that esbuild also doesn’t support sourcemaps with sections: https://github.com/evanw/esbuild/blob/02dae18a70b3d26dcb5b9599f02a9a36d88f809d/internal/js_parser/sourcemap_parser.go#L38-L41 • I’m looking into some more alternatives to try out, maybe webpack or something supports this properly, but it’s little troublesome that so many JS tools don’t support this part of the sourcemap spec already 🤷

thheller09:10:39

note that you can use :target :npm-module instead. there each file has a single source map

seanstrom09:10:31

I appreciate the suggestion, but I thought that was deprecated 😿 ? Or at least we had discussed that using ESM is probably the way forward right?

thheller09:10:17

well ... the recommendation changes based on what you are actually doing. I'm mostly guessing what you are doing

thheller09:10:34

there are instances where :npm-module will integrate easier with JS tools (e.g. source maps)

thheller09:10:41

there are instances where :esm will be the better choice

thheller09:10:49

it varies based on requirements

seanstrom09:10:25

well I guess i’m observing that shadow is not at fault with these issues with esm support, but JS tooling just follows part of the spec (for some reason).

thheller09:10:38

I also talk too many people with too many questions .. so I can't quite remember what we talked about last time 😛

😸 1
seanstrom09:10:57

I did find some good news though, it seems webpack’s sourcemap handling does support sections somehow: https://github.com/webpack-contrib/source-map-loader/blob/5e265904edf5c04d8dc209755926441a2732fc1e/src/index.js#L89-L92

thheller09:10:17

maybe there is a tool to "merge" source map sections back into a flat map?

thheller09:10:48

like that flattenSourceMap? 😛

thheller09:10:29

if you want to write such a function for CLJ I'm happy to make use of it in shadow-cljs, but writing such a function seemed very low priority for myself

thheller09:10:51

plus its probably not free, so emitting sections is more efficient

seanstrom09:10:14

Well I can try testing it out on my side and see if this flattenSourceMap works. Ideally, Vite and ESbuild and whoever would implement this ahah, but who knows if I’ll get that merged everywhere. Otherwise, I’ll share some CLJ code for the same function and it can be reviewed if it’s worth it or not.

seanstrom09:10:13

Maybe it can be an optional process when you’re combining into a larger build with NCC or Vite. And it wouldn’t be used if shadow is the final build step.

seanstrom09:10:51

Okay after some time-traveling, I’ve found the original webpack-related issue that asked for “Indexed Sourcemaps” (or sourcemaps with sections) support: https://github.com/webpack-contrib/source-map-loader/issues/89 • This PR used the source-map library by mozilla for handling this stuff, so the original support was included there and extended to webpack stuff: https://github.com/mozilla/source-map/issues/16 ◦ Although the entire spec is still not supported yet: https://github.com/mozilla/source-map/issues/437 • What’s interesting though is that the webpack related changes were requested by a ClojureScript user! This makes me think we’re on the right track here, but who knows aha 😸

thheller10:10:51

I have definitely seen sections elsewhere, not just CLJS

seanstrom10:10:38

Yeah it’s probably used elsewhere, since it’s valid piece of the spec. I think it’s sorta interesting to see how it’s been (or not been) implemented though in these bundlers. Maybe this history gives more credibility to opening issues in Vite or Esbuild, since webpack “fixes” the issue, maybe they will too

thheller11:10:51

well generating source maps and consuming them are two entirely different subjects. so I can totally see webpack even generating source maps with sections

thheller11:10:43

the closure compiler isn't exactly great at consuming input source maps either, just doesn't seem like a high priority often

thheller11:10:09

adding to that many npm modules don't even ship source maps in the first place

seanstrom11:10:29

Yeah I imagine sections sourcemaps are created in other places still, I wonder if Sentry supports soucemaps with sections (probably do) since they’re likely a huge consumer of sourcemaps

steveb8n23:10:18

let me know if anything isn’t working

steveb8n00:10:46

in a related issue, I just hit this in my client app https://github.com/google/closure-compiler/issues/2731 I’m pretty sure it’s because we upgraded our version of https://remirror.io/ so I might try your vite bundling config to workaround this when you upload it

steveb8n00:10:40

the error I get is “Class in a non-extractable location with ES2022 features’ is not yet implemented”. trying to use the :js-provider :external workaround but it doesn’t seem to change anything. will wait to see the vite sample

thheller06:10:24

@U0510KXTU in your example config ncc doesn't seem to make any use of the input source maps at all. even with a flat one, so nothing for me to do.

steveb8n07:10:08

Yeah, illustrating my lack of understanding of NCC and how it works

steveb8n07:10:09

But even before NCC is involved, the shadow release JavaScript also lacks source info in exceptions

thheller07:10:19

no, that works fine. in node you however need to "enable" source map support first

steveb8n07:10:40

Really, I'll check again when I'm home

thheller07:10:52

you can do so by adding node -r source-map-support/register -e .... to the command

steveb8n07:10:01

Oh ok. Cool. Learning already

steveb8n07:10:06

So then I/we need to learn how NCC can use shadow source maps. You can see the flag enabled when invoked

thheller07:10:32

it doesn't seem like there is a flag for that

steveb8n07:10:39

How can you tell that it's not using the shadow output

steveb8n07:10:05

Hmm. Pretty sure that NCC help shows a source map option

steveb8n07:10:26

I'll be home in 30 mins. Will check then

thheller07:10:38

yes, generation. that is an entirely different subject. it can generate just fine, for the sources it generates

thheller07:10:04

but it doesnt "consume" the input source maps, i.e. the source maps generated by shadow-cljs do map back to CLJS

steveb8n07:10:12

Ah ok. Doesn't support source maps coming in

thheller07:10:27

ncc uses webpack internally, so the sections issue doesn't apply here since webpack definitely supports that

thheller07:10:22

Error: don't know how
    at new Je (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/cljs/core.cljs:11623:11)
    at Function.Le.N (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/cljs/core.cljs:11655:5)
    at Le (/mnt/c/Users/thheller/code/tmp/shadow-cljs-ncc-source-maps/public/computer.cljs:4:1)
    at [eval]:1:33
    at Script.runInThisContext (node:vm:122:12)
    at Object.runInThisContext (node:vm:298:38)
    at node:internal/process/execution:83:21
    at [eval]-wrapper:6:24
    at runScript (node:internal/process/execution:82:62)
    at evalScript (node:internal/process/execution:104:10)
for
node -r source-map-support/register -e "require('./public/computer.js').call(null,'how?')"

👏 1
thheller07:10:38

not perfect, but works.

seanstrom07:10:40

So webpack does support input source maps, but you’ll need to configure the plugin for webpack. So if NCC doesn’t do that by default then input source maps may not be supported.

thheller07:10:37

yeah, ncc doesn't use that and doesn't appear to have a flag to add it

steveb8n07:10:01

ok, so it can work if we avoid ncc. that’s something to ponder and good progress

steveb8n07:10:37

I could just zip the whole node_modules dir but that will probably slow down the aws lambda cold start so not ideal

thheller07:10:13

shadow-cljs can do some bundling for the node targets, so if you only use basic npm packages that might be enough

steveb8n07:10:22

since I’m also working on the client side and trying Vite there, I might try Vite for the server side as well. then I can benefit from Seans work

steveb8n07:10:22

I’m using relatively complex npm deps so not sure but will explore that too

steveb8n07:10:04

tbh source maps in errors is nice to have for me currently. the client side problem is a blocker so I’m going to focus on that first

steveb8n07:10:06

when I do I might start a new thread to ask for your help on that but thanks for this feedback already. it’s pointing me in directions I can understand

steveb8n07:10:29

@U04H1SFLY5Q are you client or server side for your problem?

steveb8n07:10:54

looked into aws lambda nodejs and it doesn’t appear possible to provide node runtime flags so enable -r for my runtime. annoying but still happy to learn about this

thheller07:10:21

you can just require it in your own source code

thheller07:10:28

-r is just a quick trick

thheller07:10:42

(:require ["source-map-support/register"]) in your main file works too

🙏 1
seanstrom08:10:43

I’m experimenting with both client and server side stuff, but my initial source-maps issue came from some client stuff. here’s a small repro of the source-maps issue with Vite + Shadow: https://github.com/seanstrom/cljs-dev-template/tree/minimal-sourcemaps-debug

seanstrom08:10:21

If you’re interested in bundling for server stuff, I have some more experiments in this branch: https://github.com/seanstrom/cljs-dev-template/tree/client-server-template Take a look at the package.json scripts for some of the ways we can bundle with esbuild and then use pkg to make a binary.

steveb8n08:10:14

I just noticed in Intellij you can click play on the cli commands in the readme. such a nice way to work through a repro

steveb8n08:10:33

I’m interested in both side. thanks for the examples. I’ll learn from them tomorrow. dinner time now so signing off. thanks to you both

❤️ 1
steveb8n23:10:33

now that I’ve had a chance to think about this, since NCC doesn’t support using source-maps as input, I’m going try not using NCC. instead I’ll try zipping node modules and see if the size is reasonable/fast enough for aws lambda cold starts. if so, then I can use the “require” technique @U05224H0W points out. that would be a great result because the source line numbers are then visible enough to be useful. hopefully zipping node_modules isn’t crazy big. npm install --production may help

thheller06:10:40

might work after all 😛

seanstrom10:10:05

Small update: I’ve formerly requested for Vite to also help out here with some sections support in the source-maps: https://github.com/vitejs/vite/issues/14573 I’ve also started patching Vite in my own fork, but I’ll be fighting JS/TypeScript interop for a little bit. I also asked esbuild for sections support, and I’m waiting to see what they say: https://github.com/evanw/esbuild/issues/3439

👏 1
steveb8n10:10:55

Super valuable for me. We use typescript with vite and integrate that into a reframe app. Very similar