To avoid reading through the wall of text below, I'll start a thread.

I'm not completely sure what you mean with runner.js, I will have to look into it. I'll try to break down the problems I encounter one by one and keep it focussed. The "hack" of require is done so the script user, who will call this hacked require, can load local npm modules.

(I'm looking at : https://github.com/thheller/shadow-cljs/blob/master/packages/shadow-cljs/cli/runner.js)

in your package.json you'll specify a bin. that should be the runner. as seen here https://github.com/thheller/shadow-cljs/blob/master/packages/shadow-cljs/package.json#L21

that "runner" is then responsible for locating the correct .js file and "running" it

like my variant

ok, cool

the hack then becomes irrelevant as far as I can tell?

so when you run npx shadow-cljs it'll call the runner.js and that will either locate the local node_modules/shadow-cljs install and run that or if not found use a global install

well, SCI has access to the global object, so it can do interop on "everything" including require. So when someone writes js/require, we look up require from the global object and call it. This is why we need to put an override for require on the global object.

But the hacked require can then contain similar logic as your runner

I must be missing something because I don't know why you need to provide a custom require

I was surprised by this too, but this problem is explained here: https://swizec.com/blog/making-a-node-cli-both-global-and-local/

the problem is that node require doesn't look in the local node_modules when calling a globally installed CLI script

I think we are talking past each other here

the runner.js in shadow-cljs takes care of running either the LOCAL install OR the global install if that is missing

yes, that is what the hack for require also does

bah no it does not

that overrides that GLOBALLY and does it for every single require

which is just not needed

it is needed for the js/require calls within the scripts that are interpreted

so people can install libs locally in their interpreted script dir

if I don't apply this hack, people will have to write (js/require "./node_modules/my_lib") or so

but this is just a detail, not the major problem I'm having with shadow-cljs

you are talking about node_modules which already has special rules so the user should NEVER type out node_modules in anything and instead they should be tying (js/require "my_lib")

we're on a tangent here

which then resolves properly and doesn't require any overrides anywhere

unless I'm completely missing the point on what you are trying to do here

yes, exactly! but this doesn't work for tbd, see the blog article :)

that blog post is not about the problem you are trying to solve

like I explained with the runner.js logic. it maintains which version of YOUR tool is loaded. either the local or the global. YOUR tool, so tbd. why would you want that for the code the USER is loading?

they can already load local versions by default and you specically do NOT want it to fall back to global versions

let me put it differently, to make you understand what is not working. and then you can decide if you would solve this different.

because we are indeed talking past each other

I will make a branch without the hack

I think what you are looking for is https://nodejs.org/api/module.html#module_module_createrequire_filename

I must have been confused before since it does work. You are right, once again :)

there are uses for a custom require but never ever mess with the "global" js/require. since that is not actually global but belong to the module you are currently in

and you definitely do not want that to the the reference of all your future requires

you are right

so if you want to expose a require for sci use the module.createRequire based on the file it is currently eval'ing

also try node --input-type=module in the hashbang stuff

the hashbang worked just with node, it just didn't have any hashbang, was the problem. But this "missing hashbang" problem isn't present in my non-ESM branch

well yeah this is only for ESM. in case you ever want to "future proof" the thing

never know if node people are actually going to remove support for commonjs at some point

that's kind of a conflict right: should I continue to work with the ESM stuff to make it "future proof" (with all kinds of async stuff around import) or continue with the :node-library approach

You could also ask yourself: is node still around in 3 years or will people go all in on deno?

I'm not that familiar with the node ecosystem that much, it's an experiment for me at this point

I doubt deno will replace node ever but that doesn't mean ESM won't become more adopted generally

I don't think the async import stuff is a dealbreaker, I can make similar restrictions to dynamic requires as CLJS does, just support one top level ns form or require form

commonjs is certainly more convenient in some of these dynamic aspects but the focus has definitely shifted towards ESM

judging from all docs using ESM examples etc

@thheller I've successfully got your shadow.esm/dynamic-import exposed to SCI so you can write:

(-> (js/import "csv-parse/lib/sync.js")
    (.then (fn [csv]
             (let [csv-parse (.-default csv)]
               (-> (csv-parse "foo,bar,baz\n1,2,3" #js {:columns true})
                   (js->clj :keywordize-keys true)
                   prn)))))

Should I just leave this alone in the ns or require form and just let the user figure it out?

why not just (js/import "csv-parse")? specifying the extension when referencing a file is node stuff. I think thats a new requirement to esm, should have been that way always tbh

does node support it like this, as in (esm/dynamic-import "cvs-parse") ?

node that this lib/sync is a module within that package

I'd assume so? don't know the lib

part of their api

ah ok

@thheller Now got this working as a "tbd" script:

(ns foo
  (:require ["fs" :as fs]
            ["csv-parse/lib/sync.js" :default csv-parse]
            [reagent.core :as r]
            #_[reagent.dom.server :as rds]))

(println (str (.readFileSync fs "test.cljs")))

(prn :hello-the-end)

(prn (csv-parse "foo,bar"))

#_(prn (rds/render-to-string [:div [:p "hello"]]))

not loading reagent saves a bit of time, like 20ms or so. it's not a lot, but it shows that, architecturally, it's now possible to add more and more built-in libs, without impacting startup if you don't load all of those

Now I remember what the "hack" was for.

If you have a local file relative to the script, e.g. foo.js and you try to load it with:

(ns script (:require ["./foo"]))

again DO NOT USE THAT HACK!

use the module.createRequire if you must have a custom require (pretty sure there is a import equivalent)

aha

note that this doesn't hack import globally though

but I'll look into it :)

@thheller I'm trying to do the createRequire thing. For:

import { createRequire } from 'module';

The required JS dependency "module" is not available, it was required by "nodashka/core.cljs".

Dependency Trace:
	nodashka/core.cljs

When I try:

(def createRequire (.-createRequire js/module))
(def require* (createRequire js/module.meta.url))

ReferenceError: module is not defined in ES module scope

I have removed the hack as you suggested, but when I install nodashka (current working name) globally using npm install -g nodashka and then have a local node_modules with e.g. the ink dependency, it can't find it. That is the problem I'm trying to solve.

When you npm install -g ink then it does work. Perhaps this should just be how it works...?

I guess this is perhaps just the way it should be. Mixing global deps from nodashka itself with local deps would perhaps be weird.

if you are still using the esm target you need to add "module" to the :keep-as-import set in your build config

otherwise it'll try to bundle it but since its a built-in node package it can't do that

ah

now I remember from the fs thing, thank you

I think I'm able to get "require" back through this createRequire thing even in ESM. I thought ESM would prevent me from doing so and all requires are now async. But I could revert that.

On the other hand, maybe for the future, one could require from some http address and then it would have be async anyway

I'm not sure how this all looks for pure esm. the createRequire looks commonjs-ish

I'll give it a try. I'm still confused about having nbb (yes, changed the name) as a global tool from npm and using it with node_modules in a local script dir. Do you think this is possible, or should a global tool always use the global npm deps?

https://nodejs.org/api/esm.html#esm_no_require_exports_or_module_exports

guess its ok to use require that way?

yeah

btw you should maybe try to not use :default. use the official ["thing$default" :as x] instead of ["thing" :default x]

oh I didn't know about that

it does ring a bell, it was a recent addition right

yes, https://clojurescript.org/news/2021-04-06-release#_library_property_namespaces

quite useful addition. very handy if you have a lot of nested properties

How do I write import.meta.url in CLJS/shadow?

I tried with (.. js/import -meta -url), and even js* but both fail

fail how?

fail in release you mean?

yes

in non-release it did work

yeah the closure-compiler wants to process it statically

so using it dynamically requires some trickery, as seen in shadow.esm/dynamic-import

so js/shadow_esm_import.meta.url should work

hmm. TypeError: Cannot read property 'url' of undefined

check if js/shadow_esm_import is actually set

can't remember how I did this exactly

if all else fails (js/eval "import.meta.url")

It looks like a function over here: https://github.com/thheller/shadow-cljs/blob/ba0a02aec050c6bc8db1932916009400f99d3cce/src/main/shadow/build/targets/esm.clj#L327

I think it's actually set, else meta would have been null already

trying eval now

meta is undefined thats why you get url of undefined

yes, sorry, else I would have gotten an error about meta I mean: meta of undefined

with js/eval:

import.meta.url
       ^^^^

SyntaxError: Cannot use 'import.meta' outside a module

ah right it wraps import since closure otherwise complains

well yes, as always import is only available in modules

perhaps you can put meta on that function as well? :-D

that wouldn't do anything to fix your problem. what are you trying to do exactly?

I was trying to follow your advice of using createRequire

(def require* (createRequire (js/eval "import.meta.url") #_(.. js/shadow_esm_import.meta.url -meta -url)))

ehm thats wrong

you want to create a require that uses the source of the current file as a reference

so not the file you are in but the file you are eval'ing

oh that you mean, so cwd() basically

no

the file you are evaling

tbd some-script.cljs

so some-script.cljs

alright

but full path resolved using node path/resolve

ok, this sounds promising... trying

now I finally get what you were getting at

dont' forget to add "path" to :keep-as-import when you import it 😉

:-D

I have much to learn in this node ecosystem, I'm glad you're patient ;)

its not that complicated but helps to understand it. saves you from trying weird hacks 😉

absolutely

require in node always has a "context". ie. the module you are currently in so it knows how to follow relative paths

and follow the node resolve rules of finding node_modules

great, now my requires look like this again:

(ns script
  (:require
   ["csv-parse/lib/sync" :as csv-parse]
   ["fs" :as fs]
   ["shelljs" :as sh]))

(ns script
  (:require
   ["csv-parse/lib/sync.js" :default csv-parse]
   ["fs" :as fs]
   ["shelljs" :default sh]))

I'll move everything back to require, much more node-ish

well maybe add ["csv-parse/lib/sync.js$default" :as csv-parse]

has the added benefit of csv-parse/foo working since its an alias

yeah, I will do that after I get this sorted

probably shouldn't adopt :default at this point since it was rejected from CLJS proper

yeah, it was my ignorance :)

now people can also just use js/require again like they are used to

I'll probably move this $ back to SCI proper so it can work for all CLJS applications using it

Hmm, it seems I'm still stuck in the async import stuff. I did manage to create require which resolves according to the file but you cannot require an ES Module with require, so I'll have to keep doing that using dynamic import.

(this is for the lazy loading of optional modules at runtime)

and I don't know if there is a createImport instead of createRequire

I found this issue about it: https://github.com/nodejs/node/issues/30645

It seems for the import stuff to work, I need access to:

import.meta.resolve also accepts a second argument which is the parent module from which to resolve from:

await import.meta.resolve('./dep', import.meta.url);

(source: https://nodejs.org/api/esm.html#esm_import_meta_resolve_specifier_parent)

I tried to apply a similar trick to yours, but it didn't work.

"globalThis.shadow_esm_resolve = function(x,y) { return import.meta.resolve(x,y); }"

TypeError: (intermediate value).resolve is not a function

I admit fully that I didn't know what I was doing, but it was worth a shot ;)

But I think, either the above import.meta.resolve should be used, or, I should try something else than these ESM modules, or I should write my own resolver...

Perhaps continueing with :node-library and then making the code splitting work

So, I've also created a branch node-library here: https://github.com/borkdude/nbb/tree/node-library The compilation works nicely. It's just that I want code splitting so I can load "splits" dynamically, as needed.

So perhaps we can look into making this works. It seems a lot "easier" from the nodeJS side than this ESM stuff

@thheller Good news. I was able to use the createRequire stuff while also loading my own ESM modules async. So now the programs look "node-ish" , while still leveraring your ESM target with module support.

Once you support modules in the node-library target I'll likely move to that, but for now, we're good :-D

Also the local npm library require stuff works as expected now.

I will keep an eye on shadow for that feature, but I'm out of the woods now for my own needs

@thheller Btw, the way of resolving from the script file root got me thinking. I don't think the clojure CLI does this, does it.

e.g. foo/deps.edn + foo/script.clj + clojure foo/script.clj will not use foo/deps.edn but only the local deps.edn

I think it could be useful for normal babashka if it did...

It does. Thank you 👍

As you said, doesn't seems to be the issue