Fork me on GitHub
#shadow-cljs
<
2022-04-22
>
neilyio00:04:49

I’m trying to follow the shadow-cljs instructions for https://shadow-cljs.github.io/docs/UsersGuide.html#_requiring_js. I have :require form that looks like this:

(ns cljs.user
  (:require ["lodash"]
            ["/common/relay/relay-env"]))
…where my config has :source-paths ["src"] and my file system has src/common/relay/relay-env.js. When I evaluate the above form, shadow-cljs seems to find relay-env.js just fine, but trips up on the first line of relay-env.js:
import ajaxPath from "common/api/ajaxPath";
…where src/common/api/ajaxPath.js is also on my file system. I get the error:
The required JS dependency "common/api/ajaxPath" is not available, it was required by "common/relay/relay-env.js
…but it works if I manually change the import statement (note the leading /)
import ajaxPath from "/common/api/ajaxPath";
I’m getting the impression that when shadow-cljs resolves the imports inside of .js files, it follows the same convention as resolving .js files in ClojureScript :require forms. That’s to say, it requires the leading / if it’s going to look on the local file system, otherwise it checks node_modules. Is there a way I can get around this without manually changing all my .js import statements? I’m trying to work with a large codebase of existing .js files that all use this relative import style.

eploko02:04:13

I have a :target :node-script build in which I try to dynamically load an ESM module with (js/import "module-name-here"), which gives me ReferenceError: import$ is not defined. Is there a work around to this? Is there a way to consume an ESM module dynamically?

eploko03:04:44

As a more generic question, is there a way to consume ESM-only npm packages in :node-script and :browser builds?

thheller03:04:43

:browser will most likely just work if you just require it normally. dynamic import will not work unless you use :target :esm instead

thheller04:04:14

same for :node-script really. if you plan on using lots of ESM it is best to use ESM yourself

eploko04:04:58

the :esm target doesn't seem to resolve any of the node native modules though...

The required namespace "child_process" is not available, it was required by "blah-blah".

thheller04:04:28

for node targets you should set :js-options {:js-provider :import} in your build config

thheller04:04:47

otherwise it'll try to bundle everything which can't work for the node packages

eploko05:04:19

hm... child_process is now seemingly imported fine, but the very next import of electron fails: The required namespace "electron" is not available, it was required by "blah-blah". I made sure electron is in the dependencies in package.json, cleaned everything up, re-installed with npm i, the result is the same. any hints?

thheller05:04:22

looks like you are trying to use (:require [electron ...])? A symbol? should be (:require ["electron" ...]) isntead?

eploko05:04:18

Nope, it's

(:require
   ["child_process" :as cp]
   ["electron"] ...

eploko05:04:20

I was happy in the cjs world until I stumbled upon an esm-only package I struggle with consuming now 🙂

thheller05:04:50

hmm what is your build config?

thheller05:04:14

and this is all installed in the correct folder? happens quite frequently that people look in the wrong folder

eploko05:04:51

{:target :esm
   :output-dir "resources/js/compiled/main"
   :modules {:main {:entries [app.main]}}
   :devtools {:repl-pprint true}
   :compiler-options {:output-feature-set :es-next}
   :js-options {:js-package-dirs ["node_modules"]
                :js-provider :import}
   :main app.main/main
   :dev {:closure-defines
         {app.config/DEV true}}}

thheller05:04:31

:main app.main/main has no effect in :esm but otherwise looks fine

thheller05:04:31

so you can confirm there is node_modules/electron/package.json?

eploko05:04:36

oh, hold on pls

eploko05:04:59

yep, the npm module is installed correctly.

eploko05:04:37

I figured out the reason though: there was another file with similar name that tried to require electron as a symbol...

thheller05:04:55

that should still be ok though if the packages exist

thheller05:04:01

but always safer to use a string

eploko05:04:47

yep, checked all the code, that was a single occurance of that.

eploko05:04:20

even though it compiles now, it fails to run under electron with:

Error: ENOENT: no such file or directory, open '...app/.shadow-cljs/builds/main/dev/out/cljs-runtime/goog.debug.error.js'

eploko05:04:51

any other things I've missing in the build config potentially?

thheller05:04:04

that doesn't look an esm build?

eploko05:04:28

hm... let me clean and rebuild....

thheller05:04:34

maybe still using the old output file?

eploko05:04:24

you're absolutely right. I should take a break probably... 😅

eploko05:04:43

Well, in the end it hits the wall in that electron tries to load the entrypoint with a require itself, which being an esm module, fails...

App threw an error during load
Error [ERR_REQUIRE_ESM]: require() of ES Module ...app/resources/js/compiled/main/main.js from ...app/node_modules/electron/dist/Electron.app/Contents/Resources/default_app.asar/main.js not supported.
Instead change the require of ...app/resources/js/compiled/main/main.js in ...app/node_modules/electron/dist/Electron.app/Contents/Resources/default_app.asar/main.js to a dynamic import() which is available in all CommonJS modules.

thheller05:04:05

hmm I guess then esm isn't supported

thheller05:04:02

you can make dynamic import work but the trouble is the closure compiler may break it when optimizing

eploko05:04:18

it basically requires the entrypoint to be a cjs module. I can potentially just have that as plain .cjs file and then dynamically import the result of the build from there.... at least i hope it'll work then.

thheller05:04:18

at least the last time I checked it did

thheller05:04:04

js/import getting renamed to import$ is just the compiler being annoying. you can use (js* "import(\"whatever\)") to not have it do that

thheller05:04:15

but that may only work in development

eploko05:04:09

hm... let me play with this a bit further... i'll try all of the above perhaps, and will also check the esm build works in the browser...

eploko05:04:30

thanks for your help, really appreciated 🙂

eploko02:04:25

Well, continuing on the same journey of enabling esm in an electron app built with shadow-cljs... The build config is now this:

{:main
  {:target :esm
   :output-dir "resources/js/compiled/main"
   :modules {:main {:entries [relm.main]}}
   :devtools {:console-support false
              :hud false
              :repl-pprint true}
   :compiler-options {:output-feature-set :es-next}
   :js-options {:entry-keys ["module" "browser" "main"]
                :js-package-dirs ["node_modules"]
                :js-provider :import}
   :dev {:closure-defines
         {relm.config/DEV true}}}}

eploko02:04:48

It all compiles alright:

[:main] Build completed. (144 files, 2 compiled, 0 warnings, 0.18s)

eploko02:04:07

There's also a esm-enabler entry point .cjs script to make it possible to load everything in electron:

global.__relm__dirname = __dirname;
global.Electron = require("electron");
global.WebSocket = require("isomorphic-ws");
import("./compiled/main/main.js");

eploko02:04:48

If I try to load it as-is, the following happens:

(node:28006) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `Electron --trace-warnings ...` to show where the warning was created)
(node:28006) UnhandledPromiseRejectionWarning: ...app/resources/js/compiled/main/main.js:3
import "./cljs-runtime/shadow.module.main.prepend.js";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1038:15)
    at Module._compile (node:internal/modules/cjs/loader:1072:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:829:12)
    at Function.c._load (node:electron/js2c/asar_bundle:5:13343)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:190:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
(node:28006) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 1)
^C...app/node_modules/electron/dist/Electron.app/Contents/MacOS/Electron exited with signal SIGINT

eploko02:04:25

To mitigate this I've put a bare package.json with { "type": "module" } in resources/js, so electron treats everything under that folder as being esm by default.

eploko02:04:03

It all works alright until I try to require a node js package that happens to require another package...

(node:27663) UnhandledPromiseRejectionWarning: ReferenceError: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '...app/resources/js/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
    at file://...app/resources/js/compiled/main/cljs-runtime/cljs.nodejs.js:3:23
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
    at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
(Use `Electron --trace-warnings ...` to show where the warning was created)
(node:27663) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see ). (rejection id: 2)
^C...app/node_modules/electron/dist/Electron.app/Contents/MacOS/Electron exited with signal SIGINT

eploko02:04:12

@U05224H0W should cljs-runtime/cljs.nodejs.js maybe be using esm-style imports under the hood when compiled with :target :esm?

eploko02:04:43

It all bails out when I try to use the macchiato.server ns from the macchiato/core package. Looking at its source, I guess the problem is caused by lines like this one: https://github.com/macchiato-framework/macchiato-core/blob/85c5e3b0b55095565543a3dc876849f71df354d4/src/macchiato/server.cljs#L16

eploko02:04:22

Where node stands for:

(ns macchiato.server
  (:require
    [cljs.nodejs :as node]
...

eploko02:04:58

I'm kind of lost. Does the above mean the macchiato library is only meant to be used as a part of a normal node script, and will not work in an esm environment ever?

eploko02:04:28

I've also just realised that cljs.nodejs is not a shadow-cljs wrapper, but a part of the cljs runtime: https://github.com/clojure/clojurescript/blob/master/src/main/cljs/cljs/nodejs.cljs

eploko02:04:29

So, there's no way I can prevent macchiato from using the node's require I guess...

eploko02:04:09

Will try to see if I can replace it by the regular express from npm

thheller04:04:53

hmm yeah should definitely not be using the cljs.nodejs ns at all

thheller04:04:15

guess you are stuck with commonjs then

thheller04:04:08

did you try the import hack instead of going full esm?

eploko23:04:31

Yeah, I did. There was something not working with the approach as well. Though I don't remember details by now. 🙂 Anyway, I ditched macchiato and switched to express. Everything else seems to work fine under esm.

eploko23:04:23

One thing I noticed though, when building with :target :esm the resulting code insists on using the global WebSocket object. That works in a browser, but is not available by default in node. That's why I have the following bit in the loader file:

global.WebSocket = require("isomorphic-ws");

thheller03:04:30

:target :esm defaults to assuming a browser runtime. thus it injects some code for the REPL which uses the websocket so hot-reload and stuff work

thheller03:04:04

using it for two different runtimes is currently not officially supported but I guess you can hack it like you did

thheller03:04:34

@neil.hansen.31 those rules are pretty much the same as in webpack. as in webpack would also look for common/api/ajaxPath in node_modules. unless you explicitely configure it not to. there is no matching option for this currently though.

Patrick Brown11:04:40

Has anyone run into this error while trying to use workspaces with fulcro3? My other builds work, just not the workspaces one. The shadow build settings are near identical, so I’m lost. I don’t think this is shadow related, probably a Fulcro3 versus 2 deps thing, but here seemed like the place to find an answer. The required namespace "goog.debug.Logger.Level" is not available, it was required by "fulcro/logging.cljc"

thheller12:04:34

@pat561 yes, most likely a dependency version problem. lots of stuff changed in the closure library and you likely are just using some incompatible versions

timo13:04:53

I am currently migrating away a project from lein-figwheel to shadow-cljs and I am having a problem of understanding. I was used to have my app started when I started my cljs-repl. It was both running on the same port so I had no problem. Now I am starting shadow-cljs first as clojure-repl starting my app on port 3001 usually and then starting my frontend repl which then runs on 3000. Now my frontend-app tries to reach the backend-app on 3000 but backend is running on 3001. Do you run your frontend and backend on different ports? Do you then configure that somewhere?

timo14:04:12

the docs talk about lifecycle but it seems to be cljs only

thheller14:04:09

@timok would help if you share the configs. I'm not sure I understand that you are describing. as far as shadow-cljs is concerned everything can serve the file .js files it produces. so you should just have your backend serve them as regular files

timo14:04:36

My clj-backend has lifecycle methods with component. the server is started with it https://figwheel.org/docs/ring-handler.html

timo14:04:12

Now I need a way to do it with shadow-cljs so that my frontend is served on the same port as my backend runs on.

thheller14:04:27

ok shadow-cljs doesn't support this since you won't be using it in production and I'm a firm believer your backend should match the production setup as closely as possible

thheller14:04:43

you can just run a regular ring setup with that handler though

thheller14:04:49

nothing special to do

thheller14:04:03

but I'm guessing you already did that part? since you have something on 3000 running?

timo14:04:28

you mean I can run the shadow-cljs lifecycle-hooks with a clj-function?

thheller14:04:51

no, you misunderstand the setup

timo14:04:31

right, I am starting shadow-cljs, then starting my lifecycle-components from the clj-repl serving on 3001, and then starting the cljs-repl that serves the frontend on 3000

thheller14:04:38

shadow-cljs does nothing for your backend. that should run separately on its own using nothing from shadow-cljs (since you don't want to have that either in production)

timo14:04:31

ok, that means I need to run my cljs-app so that it can point to another endpoint as it was served from?

thheller14:04:34

what do you mean by "cljs-repl that serves the frontend on 3000"?

timo14:04:02

{:deps {:aliases [:dev :cljs]}
 :builds {:app {:target :browser
                :output-dir "dev-resources/public/js"
                :asset-path "/public/js"
                :modules {:main {:init-fn com.aareon.dp.monitor.core/run}}}}
 :dev-http {3000 ["dev-resources/public" "dev-resources/public/js"]}}

thheller14:04:20

no. As I said the .js files shadow-cljs produces can be served by any server. if you have a backend you do not need to use :dev-http at all. just remove it

timo14:04:38

that's probably it...

thheller14:04:40

this is also not related to the REPL in any way

timo14:04:02

alright, I get it

timo14:04:10

I will try that

thheller14:04:03

just make sure your backend server serves the dev-resources/public as files via either the ring-resource middleware (probably just public then, assuming dev-resources is part of the classpath) or the ring-file middleware

timo14:04:18

ok, very helpful. thanks a lot