Can anyone tell me where where clojurescript imports its reader? (I mean, it seems like its from clojure directly, but the question is where it does the importing.)

@leif do you mean this? https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/analyzer.cljc#L27-L28

tools.reader as a library has a clojure and cljs variant (for self-hosted)

@borkdude maybe better to talk about the details here - what went wrong w/ the require?

@dnolen first let's establish the right context for discussing the problem. My goal is to create a node script that imports an ES module from vanilla CLJS, to compare if it ostensibly looks the same as what nbb scripts are going to look like. I am able to require term-size and some other ES module using shadow's `:esm` build type: https://gist.github.com/borkdude/7e548f06fbefeb210f3fcf14eef019e0 I have no idea how to accomplish the same thing with vanilla CLJS. Then there is another thing: First I used createRequire to implement :require in nbb. But this disallowed using ES Modules from NPM and some libraries (like term-size) only offer their new versions as ESM. That is the only reason I switched to ESM-based :require again implemented using shadow.esm which is essentially js/import. Simply because this lets you deal with all libs available in nodeJS and not only the commonJS ones.

Hope the issue is clear, if not, let's try to get it clearer.

@borkdude and what happens if you don't use this this createRequire trick?

require itself is forbidden in ES modules in Node.js and importing ES modules from CommonJS doesn't work either

but let's just focus on how would one do this in normal CLJS in the ns form: how does one go about making a script using vanilla CLJS like this one: https://gist.github.com/borkdude/7e548f06fbefeb210f3fcf14eef019e0. I can make the nbb scripts look the same, there is enough flexibility

@borkdude right then it probably doesn't work in ClojureScript in the same way it doesn't work in JS since we generate CommonJS

it works on the Web and React Native (via :bundle) - because not working is terrible

My main concern is compatibility: I want nbb scripts to look like CLJS scripts and not make any breaking changes later on. But it seems so far nobody every really tried to use ES modules from nodeJS via CLJS? This surprises me.

@borkdude but I think you already solved the problem wrt. compatibility? that what it ends up looking like on the Web & for RN

I get the whole "we don't care about ES modules" but if there is no other choice, they are forced upon you now.

I think probably you are overestimating the number of people actually doing Node.js specific stuff

if you are doing Node.js specific you already have to know about the work arounds

use bridging libraries / hacks, don't bump versions etc.

truly popular libraries probably haven't to switched to only ES because of these issues

you have 12 years of Node.js code sitting around

@borkdude all that said, this createRequire thing seems like a simple enough workaround - does it work regardless of what you try to require?

I'm not using createRequire anymore since you can't load ES modules with it, only CommonJS modules from an ES module

oh so you're saying dynamic import (i.e. as a function) solves this problem)?

well I am using it indirectly. I am using (createRequire "script.cljs") then I use its resolve method to find library in the relative node_modules directory and feed that into dynamic import.

what I don't understand is how this works with a nested dep graph?

i.e. you have things in your graph that are not ESM?

this sounds so broken

This approach works for both ESM and CommonJS.

I think Node.js lets you require CommonJS modules using import as well, which is probably why it works.

ok

ESM -> CommonJS -> CommonJS etc. is allowed

yeah

but ESM is not a thing either

like how does Node.js know it's ESM? Did they decide on .mjs?

There are two things: you can define "type": "module" in your package.json to define your package as ESM and if you're going to use the import ... syntax, then you use .mjs, at least, this is how far my understanding currently goes

but how does this work for index.js then?

This is index.mjs

do you have to declare that or does it have to be index.mjs?

yeah, I think so

I have tried this yesterday and that worked, with "main": "index.mjs"

the levels of breakage are mind-boggling

note that this is more or less my first encounter with this ecosystem in any serious way, so I could totally be missing something

but how does import as fn work? it's async and returns a promise

In my naivety I thought: let's try to replicate the experience of lumo scripts with what I've currently got with SCI and see what happens.

yes, dynamic import returns a promise. but nbb deals with this by evaluating everything async from top to bottom

oh but you can await at the top level

alright

yeah kind of, but I'm in CLJS there, so I'm just chaining promises myself for now

but this is ok, this solves a different problem I had with SCI for a while: it will enable loading code asynchronously from other places lazily, so I'm not unhappy that I was forced down this path

so I'd be ok w/ an experimental patch for :esm output for :nodejs target only for now

1. require -> await import

2. a warning if :output-to is not .mjs extension

this is a relatively trivial patch, so if you're interested let me know

maybe the only place w/ potential gotcha might be REPL stuff - but probably not

yeah, makes sense. I'm also curious about @thheller's input here since he has made the :esm target in shadow already.

oh there is a 3

3. CLJS files must be emitted .mjs in this mode

we feed compiler opts everywhere - so this shouldn't be too hard

let me compile nbb and see what shadow currently emits in terms of .js files.

:esm in shadow-cljs does a bunch of stuff and support more usecases - not really interested in using it as a reference

Just .js files so I'm probably wrong about that .mjs stuff. This is why I would appreciate thheller's input, he's already figured this stuff out.

Maybe the rule is, if "type" : "module" then ES is assumed or something like this.

$ head out/nbb_main.js

import "./nbb_core.js";
import "./cljs-runtime/shadow.module.nbb_main.prepend.js";
SHADOW_ENV.setLoaded("shadow.module.nbb_main.prepend.js");
import "./cljs-runtime/shadow.esm.esm_import$module.js";
SHADOW_ENV.setLoaded("shadow.esm.esm_import$module.js");
import "./cljs-runtime/nbb.main.js";
SHADOW_ENV.setLoaded("nbb.main.js");
import "./cljs-runtime/shadow.module.nbb_main.append.js";
SHADOW_ENV.setLoaded("shadow.module.nbb_main.append.js");

you don't need await import ..., only when you are using the import(...) function I think (aka dynamic import)

> Node.js will treat .cjs files as CommonJS modules and .mjs files as ECMAScript modules. It will treat .js files as whatever the default module system for the project is (which is CommonJS unless package.json says "type": "module",).

(https://stackoverflow.com/a/57492606/6264)

interacting w/ package.json is bad idea

ok, so we can make it explicit using .mjs then

note that this kind of build can also benefit the deno stuff (although that sounds a bit speculative)

dynamic async import is simpler because we don't have to mangle the code even more

or do other things

it looks just like require(...)

I think you can also emit import * as something from "./nbb_core.js"; and then do something with something, which is probably similar to what CLJS is doing now with require

what I said about is how it needs to be done if it's going to be simple

ok, if dynamic import is closer, then let's go with that

with Node.js modules we assign the result to a namespace var

async import(...) let's us change nothing

indeed

there are probably dragons still, but I will write up a ticket with these notes

🙏

FWIW the only thing shadow-cljs really does is add glue code to make import work. so it compiles CLJS as regular code with (:require ["thing" :as x]) setting an alias. eg. module$thing, so any use of x/foo gets turned into module$thing.foo. after cljs compilation is done and "packaging" starts shadow-cljs basically just adds import * as module$thing from "thing" via a prepended string

all the extra glue it does is just to work around quirks in the closure compiler not being configurable enough when I did this

it still isn't. one hurdle with async import() will be that closure will want to optimize it (or others in case of :target :bundle)

but since its "dynamic" that'll end up less ideal

@thheller so you want to support bundling for Node.js? Is there are any reason for that? or you were trying to make this work for both Web and Node.js?

@borkdude https://clojure.atlassian.net/browse/CLJS-3325

I wanted this for platforms that don't support commonjs, see https://clojureverse.org/t/generating-es-modules-browser-deno/6116

right, don't care about deno - ok

didn't build this for node at all

yes this was my understanding a wide scope which I'm not interested in

but given that ESM is a standard it'll work everywhere ESM works

so the point really is supporting the standard, assuming everything moving to it over time

@borkdude the scope of the above is all I'm interested in for now

@thheller but is there any requirement for this to work for the Web given all the bundlers solve this for you?

many bundlers starting going ESM first (eg. webpack v5). so the whole interop between commonjs <-> ESM will get much stricter than it used to be

and as such more and more people moving over to ESM. until we get to the point where its mostly ESM and commonjs is the problem case

@thheller for shadow-cljs if you emit CLJS as ESM how does that impact the REPL?

works fine. in development its kinda fake ESM. just puts everything in the global scope and operates on globalThis

really not doing any adjustments to the code generated by cljs.compiler at all

I see your :esm thing in many ways works just like :target :bundle

just wrapping it in a bit more boilerplate

lifting the requires to some other place

yes, pretty much

I think @thheller would be an ideal candidate for this issue, much more ideal than me ;)

@borkdude I doubt he has time for this 🙂

ok shadow-cljs just solves a general problem

:target :bundle could do what shadow does so it's orthogonal IMO

the thing :esm does "more" than :bundle is the usual npm processing and actually bundling the packages

It's not like I have seas of time either, but to re-iterate, the main issue I'm after is: nbb should behave the same as (future) CLJS with regards to this stuff. If I can help ensuring that, then I would give it some time.

but with :js-provider :import it is basically :target :bundle, just import not require

CLJS-3325 IMO is just about mitigating the current Node.js situation

@borkdude if all you care about is compatibility - then there's nothing more to do - somebody else can work on this as well

the impact is too small for me or anyone else at the moment - I don't hear people clamoring for this

at least I was able to verify how it works with the :target :esm in shadow-cljs already: the nbb script looks identical there

and at least people have 1 option to migrate from interpreted scripts to compiled scripts with shadow, CLJS could follow later (and hopefully the .cljs code stays the same)

I just care a lot of about not making breaking changes, that's why I brought this all up. I'd be ok if CLJS waited a bit on this issue as there are options now.

Over time the issue may become more important as ESM is more forced upon you.

yeah, seems slow going but stuff is definitely moving over

I could add some docs to nbb how people can migrate to shadow if they want more perf out of their scripts.

you could provide a shadow-cljs run nbb.shadow my-script.cljs out/my-script.js (or I could)

just as a convenience to not need a build config 😉

yeah, not all nbb projects have just one single script though, there's already one with 3 and a util namespace: https://github.com/chr15m/c64core

so let's go with some docs for now and wait how it will unfold

some docs around the :esm target may be nice for starters

yeah docs are upcoming 😉

ok thanks all, appreciate the conversation

(https://github.com/borkdude/nbb#migrating-to-shadow-cljs)

@thheller Ah, okay, so it just uses the parser of the version of clojure its written in then?

err, reader*

no. tools.reader is a separate library https://github.com/clojure/tools.reader

@thheller AH, okay, thanks. (I thought it was using Clojure's /src/jvm/clojure/lang/LispReader.java for some reason (except for self-hosted obviously).

Anyway, that seems to work, thanks.

It used to but we switched to tools.reader when we went self hosted