As a more generic question, is there a way to consume ESM-only npm packages in :node-script and :browser builds?

:browser will most likely just work if you just require it normally. dynamic import will not work unless you use :target :esm instead

same for :node-script really. if you plan on using lots of ESM it is best to use ESM yourself

https://clojureverse.org/t/generating-es-modules-browser-deno/6116

the :esm target doesn't seem to resolve any of the node native modules though...

The required namespace "child_process" is not available, it was required by "blah-blah".

for node targets you should set :js-options {:js-provider :import} in your build config

otherwise it'll try to bundle everything which can't work for the node packages

hm... child_process is now seemingly imported fine, but the very next import of electron fails: The required namespace "electron" is not available, it was required by "blah-blah". I made sure electron is in the dependencies in package.json, cleaned everything up, re-installed with npm i, the result is the same. any hints?

looks like you are trying to use (:require [electron ...])? A symbol? should be (:require ["electron" ...]) isntead?

Nope, it's

(:require
   ["child_process" :as cp]
   ["electron"] ...

I was happy in the cjs world until I stumbled upon an esm-only package I struggle with consuming now 🙂

hmm what is your build config?

and this is all installed in the correct folder? happens quite frequently that people look in the wrong folder

{:target :esm
   :output-dir "resources/js/compiled/main"
   :modules {:main {:entries [app.main]}}
   :devtools {:repl-pprint true}
   :compiler-options {:output-feature-set :es-next}
   :js-options {:js-package-dirs ["node_modules"]
                :js-provider :import}
   :main app.main/main
   :dev {:closure-defines
         {app.config/DEV true}}}

:main app.main/main has no effect in :esm but otherwise looks fine

so you can confirm there is node_modules/electron/package.json?

oh, hold on pls

yep, the npm module is installed correctly.

I figured out the reason though: there was another file with similar name that tried to require electron as a symbol...

that should still be ok though if the packages exist

but always safer to use a string

yep, checked all the code, that was a single occurance of that.

even though it compiles now, it fails to run under electron with:

Error: ENOENT: no such file or directory, open '...app/.shadow-cljs/builds/main/dev/out/cljs-runtime/goog.debug.error.js'

any other things I've missing in the build config potentially?

that doesn't look an esm build?

hm... let me clean and rebuild....

maybe still using the old output file?

you're absolutely right. I should take a break probably... 😅

Well, in the end it hits the wall in that electron tries to load the entrypoint with a require itself, which being an esm module, fails...

App threw an error during load
Error [ERR_REQUIRE_ESM]: require() of ES Module ...app/resources/js/compiled/main/main.js from ...app/node_modules/electron/dist/Electron.app/Contents/Resources/default_app.asar/main.js not supported.
Instead change the require of ...app/resources/js/compiled/main/main.js in ...app/node_modules/electron/dist/Electron.app/Contents/Resources/default_app.asar/main.js to a dynamic import() which is available in all CommonJS modules.

hmm I guess then esm isn't supported

you can make dynamic import work but the trouble is the closure compiler may break it when optimizing

it basically requires the entrypoint to be a cjs module. I can potentially just have that as plain .cjs file and then dynamically import the result of the build from there.... at least i hope it'll work then.

at least the last time I checked it did

js/import getting renamed to import$ is just the compiler being annoying. you can use (js* "import(\"whatever\)") to not have it do that

but that may only work in development

hm... let me play with this a bit further... i'll try all of the above perhaps, and will also check the esm build works in the browser...

thanks for your help, really appreciated 🙂

Well, continuing on the same journey of enabling esm in an electron app built with shadow-cljs... The build config is now this:

{:main
  {:target :esm
   :output-dir "resources/js/compiled/main"
   :modules {:main {:entries [relm.main]}}
   :devtools {:console-support false
              :hud false
              :repl-pprint true}
   :compiler-options {:output-feature-set :es-next}
   :js-options {:entry-keys ["module" "browser" "main"]
                :js-package-dirs ["node_modules"]
                :js-provider :import}
   :dev {:closure-defines
         {relm.config/DEV true}}}}

It all compiles alright:

[:main] Build completed. (144 files, 2 compiled, 0 warnings, 0.18s)

There's also a esm-enabler entry point .cjs script to make it possible to load everything in electron:

global.__relm__dirname = __dirname;
global.Electron = require("electron");
global.WebSocket = require("isomorphic-ws");
import("./compiled/main/main.js");

If I try to load it as-is, the following happens:

(node:28006) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `Electron --trace-warnings ...` to show where the warning was created)
(node:28006) UnhandledPromiseRejectionWarning: ...app/resources/js/compiled/main/main.js:3
import "./cljs-runtime/shadow.module.main.prepend.js";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1038:15)
    at Module._compile (node:internal/modules/cjs/loader:1072:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:829:12)
    at Function.c._load (node:electron/js2c/asar_bundle:5:13343)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:190:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
(node:28006) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 1)
^C...app/node_modules/electron/dist/Electron.app/Contents/MacOS/Electron exited with signal SIGINT

To mitigate this I've put a bare package.json with { "type": "module" } in resources/js, so electron treats everything under that folder as being esm by default.

It all works alright until I try to require a node js package that happens to require another package...

(node:27663) UnhandledPromiseRejectionWarning: ReferenceError: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '...app/resources/js/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
    at file://...app/resources/js/compiled/main/cljs-runtime/cljs.nodejs.js:3:23
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
    at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
(Use `Electron --trace-warnings ...` to show where the warning was created)
(node:27663) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 2)
^C...app/node_modules/electron/dist/Electron.app/Contents/MacOS/Electron exited with signal SIGINT

@U05224H0W should cljs-runtime/cljs.nodejs.js maybe be using esm-style imports under the hood when compiled with :target :esm?

It all bails out when I try to use the macchiato.server ns from the macchiato/core package. Looking at its source, I guess the problem is caused by lines like this one: https://github.com/macchiato-framework/macchiato-core/blob/85c5e3b0b55095565543a3dc876849f71df354d4/src/macchiato/server.cljs#L16

Where node stands for:

(ns macchiato.server
  (:require
    [cljs.nodejs :as node]
...

I'm kind of lost. Does the above mean the macchiato library is only meant to be used as a part of a normal node script, and will not work in an esm environment ever?

I've also just realised that cljs.nodejs is not a shadow-cljs wrapper, but a part of the cljs runtime: https://github.com/clojure/clojurescript/blob/master/src/main/cljs/cljs/nodejs.cljs

So, there's no way I can prevent macchiato from using the node's require I guess...

Will try to see if I can replace it by the regular express from npm

hmm yeah should definitely not be using the cljs.nodejs ns at all

guess you are stuck with commonjs then

did you try the import hack instead of going full esm?

Yeah, I did. There was something not working with the approach as well. Though I don't remember details by now. 🙂 Anyway, I ditched macchiato and switched to express. Everything else seems to work fine under esm.

One thing I noticed though, when building with :target :esm the resulting code insists on using the global WebSocket object. That works in a browser, but is not available by default in node. That's why I have the following bit in the loader file:

global.WebSocket = require("isomorphic-ws");

:target :esm defaults to assuming a browser runtime. thus it injects some code for the REPL which uses the websocket so hot-reload and stuff work

using it for two different runtimes is currently not officially supported but I guess you can hack it like you did

the docs talk about lifecycle but it seems to be cljs only

My clj-backend has lifecycle methods with component. the server is started with it https://figwheel.org/docs/ring-handler.html

Now I need a way to do it with shadow-cljs so that my frontend is served on the same port as my backend runs on.

ok shadow-cljs doesn't support this since you won't be using it in production and I'm a firm believer your backend should match the production setup as closely as possible

you can just run a regular ring setup with that handler though

nothing special to do

but I'm guessing you already did that part? since you have something on 3000 running?

you mean I can run the shadow-cljs lifecycle-hooks with a clj-function?

no, you misunderstand the setup

right, I am starting shadow-cljs, then starting my lifecycle-components from the clj-repl serving on 3001, and then starting the cljs-repl that serves the frontend on 3000

shadow-cljs does nothing for your backend. that should run separately on its own using nothing from shadow-cljs (since you don't want to have that either in production)

ok, that means I need to run my cljs-app so that it can point to another endpoint as it was served from?

what do you mean by "cljs-repl that serves the frontend on 3000"?

{:deps {:aliases [:dev :cljs]}
 :builds {:app {:target :browser
                :output-dir "dev-resources/public/js"
                :asset-path "/public/js"
                :modules {:main {:init-fn com.aareon.dp.monitor.core/run}}}}
 :dev-http {3000 ["dev-resources/public" "dev-resources/public/js"]}}

no. As I said the .js files shadow-cljs produces can be served by any server. if you have a backend you do not need to use :dev-http at all. just remove it

ah ok

that's probably it...

this is also not related to the REPL in any way

alright, I get it

I will try that

thanks

just make sure your backend server serves the dev-resources/public as files via either the ring-resource middleware (probably just public then, assuming dev-resources is part of the classpath) or the ring-file middleware

ok, very helpful. thanks a lot