Clojurians Log v2

I am using shadow's :esm target if I'm using shadow-cljs. For #cherry which is a little alternative CLJS compiler I just generate import directly from :require.

but you always emit import rather than require

e.g. in Node.js you cannot use require 1. from modules 2. to load other libraries (unless you use some Node.js specific API, module.createRequire(import.meta.url) For this reason, always import

but if you use import how does this work from a REPL?

In #nbb what I do is use dynamic import, always. But this is a CLJS interpreter, not targeted at emitting code, but just executing on the fly. So (require '["whatever-lib"]) executes as a dynamic import of whatever lib and then defines whatever is necessary in the environment regarding referred functions, aliases etc. Each top level REPL expression returns a promise (not automatically wrapped like JS promises) and it's "awaited" result becomes the REPL evaluation result. dynamic imports don't play well with optimizations done by JS tooling though #cherry doesn't have much of a REPL yet and this is indeed one of the unsolved annoying ES6 problems.

right but that is quite ugly for many reasons wrt. REPL

(do (require …) …) for example

not criticizing specifically how you did it

just pointing probably not acceptable for ClojureScript

$ npx nbb
Welcome to nbb v1.2.173!
user=> (do (require '["fs" :as fs]) (js/console.log fs))
[Module: null prototype] {
  Dir: [class Dir],
  Dirent: [class Dirent],
  F_OK: 0,

top-level do is unwrapped

but aren’t there cases you can come up that won’t work because of the promise?

or are you saying because the REPL deals w/ promises it works?

for squint (similar project as cherry, but no immutable data structures, etc) I've implemented something that would be closer to the CLJS compiler problem since it emits code.

$ npx squint-cljs repl
user=> (require '["fs" :as fs])
user=> fs/readFileSync
[Function: readFileSync]

It's also ugly, but ES6 complicates stuff a lot since it's async + immutable

but in both cases, the REPL knows that the REPL evaluation result is a promise

but it does not unwrap promises directly:

user=> (js/Promise.resolve 1)
Promise { 1 }

ok I’m curious how shadow-cljs solves the import + REPL problem

last time I asked the target :esm didn't have a REPL, but curious about hearing more from @thheller

require in a do has never worked in a REPL where IO is required ie. browser?

hrm, actually you are right - require support is a REPL affordance

(assuming something must actually be loaded)

you write require , and then something else

so it would have to be dynamic import

does shadow-cljs do that?

no, the import is only emitted for :target :esm. which currently doesn't support requiring "new" stuff not part of the build yet

otherwise for regular builds import is added as a prepend, so the closure compiler never sees it

@thheller What do you mean with "regular build", aside from target esm you don't need import, right?

by regular build I mean "not the REPL"

is there a target esm REPL then?

sure, when loaded in the browser. just not for node or deno

https://clojurians.slack.com/archives/C6N245JGG/p1662922223854529

ah right, that was the caveat

@thheller hrm so shadow-cljs just doesn’t support requiring libs in Node.js that for some wild reason declare "module" only

not at the REPL. a regular build can do that just fine

I get the build case of course

the way I have done it in shadow is emitting import * as some$alias$var from "x". which is added as a prepend and loaded normally via node

(require '["x" :as x]) then maps to that some$alias$var, but as of now does NOT do a dynamic import for create that var if it not already exists

it could certainly do that but a REPL and ESM suck so much

yep :( I'm considering just doing nested global objects for development in cherry and squint. I mapped "CLJS" namespaces to ES6 modules directly, but this doesn't play well with a REPL for sure

What is the current status of Google Closure and ES6 output? Still nothing?

yeah, thats why during development :esm maps everything in globalThis to emulate "everything is global" as much as possible

has been supported for ages, with a million unsupported cases 😛

I meant ES6 module output

https://github.com/google/closure-compiler/blob/49472a63ab90f9c06303dc9ee389ce88a46fa0c3/src/com/google/javascript/jscomp/CompilerOptions.java#L2984-L2992

impossible to make anything work with that though

but shadow isn't using this right?

no, doesn't allow me to do half the stuff :esm needs to do

I wish there would be a more dev-friendly ES6 module format with at least mutable bindings, async or not

ok, I thought the library that I was looking at only supports module but actually that’s not true - it does have a CJS file in the build dir

I guess even JS people realize that ESM only in Node.js leads to madness?

progress to adopt ESM as universal has been extremely slow yes

right I remember chatting about this two years ago - like any day now - but based on how bad this stuff is I think it will never happen

@dnolen #nbb defaults to loading ESM but falls back to CJS via module.createRequire if nothing else is available

I mean the Node.js REPL appears to have the same problem right?

yeah I remember your rants and you were of course right ;)

It is impossible to import in a JS REPL?

yes, but you can do dynamic import

but that is so broken

await import("fs")

total language design fail

well, yeah, the certainly didn't adopt "top level expression is the compilation unit" idea...

ok anyways, I’ve collected a good amount of information

I think my feeling is still “do nothing” 🙂

Some tools like next.js (I can't keep up with what they have now again, vite, or so?) do hot-reload ES6 modules, but how they do it is a complete mystery to me

I share the "do nothing" sentiment. Always feels like things should settle around this stuff but they somehow don't

bun, which is a Node.js competitor allows more stuff like sync loading ES6 stuff and mixing import and require, I believe.

☠️ - this is how I feel after reading this thread (fixed). I never realized how repl-unfriendly this es6 has been. Enlightening at least.

@borkdude afaik vite works by keeping track of the module graph and reloading the relevant part of the tree by pretending it’s a new module via some query param. Modules can opt into hot-module-reloading (HMR) see https://vitejs.dev/guide/api-hmr.html

right, still it's pretty gross

I mean, it's great they solved the problem, but it shouldn't have to be that complex

no support for exports at all as of now. same nightmare of inconsistent support and undefined uses.

https://webpack.js.org/guides/package-exports/

directories, wildcards, ordering matters, ambiguities all over the place. just horrible.

there is a :js-options {:entry-keys ["browser" "main" "module"]} config option that lets you re-order so "module" is picked first for example. just not in the context of package.json "exports"

but I'd reuse this for exports for same. same logic really.

ok, it also seems way too complex to me, I think I will only solve a couple of cases so that there are some clear examples

community can submit patches to expand the functionality using the refactored code / tests as launching pad

all I was really going to do was calculate explicit subpath stuff

so that react-select/creatable works

and I guess just always resolve to require for now?

if you add the same simplification that shadow-cljs does for strings you don't really need to do anything

ie. any (:require ["any-string" :as x]) is just passed through, without attempting to resolve for node at all

you only needed to know node_modules packages for symbols, so that (:require [any-sym :as x]) can be checked if it exists

It’s not possible to change anything here anymore really

you are not breaking anything, just relaxing requirements

symbols you'd still need to check. or is your intent to allow (:require [react-select/creatable :as x])?

for shadow-cljs in any build where it doesn't try to bundle npm dependencies (so node, react-native, etc) it just accepts any string and lets webpack/node/whatever figure it out

so it doesn't need to look at exports for those cases

I think it is breaking because you could have used a string for a regular CLJS require

for symbols it does the node_modules presence check, just to complain if they don't exist

well you could disallow that. shadow-cljs doesn't allow that, and I have never seen it used in the wild.

if you mean (:require ["clojure.string" :as x])

I always thought that was a bug and not intentional

not a bug

so altering it is a problem

I know these aren't the official CLJS rules, but some tools (shadow, nbb, clj-kondo) rely on string = JS lib, symbol = CLJS lib. Keeps things much simpler

we just don’t rely on it and it been that way for years

specifically clj-kondo for linting would be wrong here

I know, but there's no other way for clj-kondo (static analysis) to know whether you're using a JS lib or not (and hence you can use the alias as a JS object or not)

strings are supported because symbols have character restrictions - that’s it, nothing about JS or not

but clojure namespaces can't have those characters, so it makes sense and everything so much clearer

pretty sure clojuredart treats it that way as well

at this point I’m just pointing out that it’s just very unclear who would be affected so it’s not really worth altering anymore

this must go back to 2017/18 at least

@borkdude btw, you could probably call into the ClojureScript functions to figure this out

yeah, it could do that, but clj-kondo is pretty unassuming, it works on random .clj(s) files without looking at the rest of your project and incrementally learns more if you throw more code at it

I haven't gotten any complaints about it, I think most people like the shadow convention. But I understand that making breaking changes is a no-go

the other thing to consider is some point we could parse the entrypoints as we do w/ GCL so we validate usages / :refers etc.

I’m not saying that clj-kondo should do anything - but figuring this stuff does have value on the ClojureScript side

yeah at some point it might. we're scratching the surface with Java analysis (source + byte code) now to improve Java interop linting/completions

That is nice if it is working for react-select. I’ve seen a few libs in recent times that have this same structure. So that certainly covers the cases I’ve been exposed to so far (I read the threads and know that package.json “exports” can get a lot more messy & we aren’t dealing w/that now).

I like the refactored code org. It makes it clearer where this stuff happens and I just think its easier to read through than the previous.

definitely … also you know … TESTS 🙂

I just realized there is a release version of CLJS that has this change. I am going to try it out. Been hitting the same react-select style indexing in many node modules I’ve encountered lately.

I’m still messing with it, but so far it’s looking good. @codemirror modules were another major example of this and we have a bunch of them that now work with CLJS v1.11.121. Quite nice.

I get (js-await ...) but where would you use js-async? Some experimental projects and ClojureDart have used:

(defn ^:async foo [])

that might be sufficient, honestly I haven’t been following along too closely, so if there’s prior art that is good to know

but what about the anon fn case?

^:async (fn [])

yes

it is not sufficient

there are very many cases in the compiler that generate IIFEs. that will break await expectations

let being the most prominent I guess, but pretty much all expressions get wrapped in (function() { return ... })();

so async function() { var x = (function() { return await something() + 1 })() } breaks

oh nvm. didn't see other thread

Just emitting the keyword won't be enough since expressions that result into compiled fn expressions (to make JS behave as an expression oriented language) also need to be marked async. So (I think) it's necessary to use the CLJS analyzer to remember if you are in an async function and then propagate that to the emitted self-calling fn helpers. I've been doing this work in cherry. E.g.:

$ ./node_cli.js --show --no-run -e '(defn ^:async foo [] (let [x (do (js/await 1) 2 3)] 1))'
import * as cherry_core from 'cherry-cljs/cljs.core.js';
var foo = (async function () {
let x1 = (await (async function () {
 (await 1);
2;
return 3;
})());
return 1;
})
;

export { foo }

Tests here: https://github.com/squint-cljs/cherry/blob/ac89d93f136ee8fab91f62949de5b5822ba08b3c/test/cherry/compiler_test.cljs#L276-L310

I should have introduced a special form js-await for which I have an issue

So far the CLJS project (and also shadow) resisted adding async/await support, I'd be happy to contribute or help

hrm right that is the main problem

I wonder if GCC optimizes async fns?

trivial cases like that

you mean strip out the await if it's not necessary?

oh but I see, this is like list of side effects

right, anything can happen

So far shadow covered this use case by implementing js-await as a macro (shadow.utils.js-await..., don't remember the exact namespace) which just compiled into a .then expression. It was the user-space solution which gets you 90% there (no top level await), but because JS isn't Lisp they have to solve it at the language committee level

async/await in CLJS will be super tedious unless let is changed to not compile down to IIFEs, I think

it's not that complex, at least not how I did it in cherry, just remember some state in the context and pass it down

e.g. one may end up doing sth like

(defn ^:async f [x]
  (let [y x]
    (yield ...)
    (await ...)))

(function(){   // <-- inserted by compiler
  yield ...;   // <-- error (must be inside function*)
  await ...;   // <-- error (must be inside async function)
}();

I assume you saw the example here:

https://clojurians.slack.com/archives/C07UQ678E/p1684782676083269?thread_ts=1684782243.166459&cid=C07UQ678E

also how does recur work in an async context? async iteration is relatively new to ES (though before es2020 people in JS/TS world missed Promise.all and would incorrectly use await in for-each loops)

yeah that’s right… I remember searching around the net seeing what would be the pros/cons to async/await in CLJS, and so far it seems like a huge refactoring of the compiler would be needed…

with that said, I agree with sufficient effort and refactoring we could likely have async/await in CLJS, but it seems like a huge amount of effort for something that is honestly kinda “solved” by core.async, promesa, and JS interop with promises

imho js-await should look like this https://clojureverse.org/t/promise-handling-in-cljs-using-js-await/8998

simple to do now, and conceptually easy to translate to real await later

> it seems like a huge refactoring of the compiler would be needed I don't think it will be a huge change, but only a PoC will make this evident I guess

oh I tried. it is a substantial change.

in cherry I only have a few lines of code that are about async/await and I haven't come across an example that didn't work sufficiently. this is why my hunch was that the change to the CLJS compiler wouldn't have to be that big, but I could be wrong

@thheller hrm, why couldn't this be done w/ a compiler pass?

if it is a lot of work - then probably not, but if can be done as a pass then probably we can do this

re: top level await maybe that is off the table?

if you're not emitting ES6 modules, then top level await doesn't even work

can't remember the exact details of why this was hard. but just search for function in cljs.compiler. there are a lot of them emitted, with no analyzer involved. so a pass won't do anything for those.

hrm, that is an interesting point right

we might not be so disciplined when we emit extra fns so maybe that's a problem