Fork me on GitHub
#shadow-cljs
<
2021-09-30
>
gdubs04:09:36

I'm stuck. I'm trying to bootstrap a little project and am having trouble in situations where my .cljs file requires a .js file which requires an npm module. See below for a ~10-line example.

gdubs04:09:57

The error says: Module not provided: shadow.js.shim.module$express, but I can confirm that that file is buried in my .shadow-cljs/... directory as I would expect.

thheller05:09:28

hmm which shadow-cljs version is this? something similar was fixed a rather long time ago?

gdubs05:09:51

just upgraded to the latest. Is the problem my use of require(...) as opposed to import?

thheller05:09:44

no, both should be ok

thheller05:09:39

can you open an issue about this? can't take a look right now

Phillip Mates09:09:53

hi folks, I'm interested in doing a bit performance-related explorations of cljs code in various JS engines (V8, JSC, etc). The project I'd like to profile uses shadow-cljs. thus I'm wondering: Is there a way in shadow-cljs to compile to a single js file that doesn't make use of require, nor node.js-specific features like filesystem access? Such that I can run the code in the d8 or jsc javascript consoles?

Phillip Mates09:09:51

I tried some of the different :target configurations with :optimizations set to simple but couldn't get anything rolling. I was able to find some benchmark code in the clojurescript repo that runs cljs stuff in d8 and jsc, so I know at some level this sort of thing is tenable

thheller15:09:59

a :node-script release build should be ok in d8/jsc. either simple or advanced. watch or compile won't work

Phillip Mates17:09:59

I'm giving it a shot now and getting > public/script.js:3420: ReferenceError: global is not defined and when I replace goog.global=global with goog.global=globalThis (not sure if that is correct though) then I get > public/script.js:4203: ReferenceError: require is not defined > ... > shadow.js.shim.module$react=require("react"); > ... I don't think d8 has require available 😕

thheller17:10:07

it does not but it'll also probably not have many other things if you plan on using react and stuff

thheller17:10:40

you can set :js-options {:js-provider :shadow} in your build config. that'll make shadow-cljs bundle all dependencies so the result doesn't contain require anymore

thheller17:10:45

maybe thats enough

Phillip Mates20:10:18

I gave it a try with a build entry like:

:alone
  {:target :browser
   :debug false
   :output-to "public/script.js"
   :main re-db.test/-main
   :js-options {:js-provider :shadow}
   :compiler-options {:optimizations :simple :pseudo-names true}
   :output-dir "public/bench/js"
   :modules {:bench {:entries [re-db.test]}}}
public/bench/js/bench.js:4428: TypeError: Cannot read properties of undefined (reading 'abs')
shadow.js.files[$i$jscomp$309_name$$],$moduleFn$$=shadow$provide[$i$jscomp$309_name$$];if(void 0===$module$$){if(void 0===$moduleFn$$)throw"Module not provided: "+$i$jscomp$309_name$$;$module$$={exports:{}};shadow.js.files[$i$jscomp$309_name$$]=$module$$}if($moduleFn$$){delete shadow$provide[$i$jscomp$309_name$$];try{$moduleFn$$.call($module$$,goog.global,shadow.js.jsRequire,$module$$,$module$$.exports)}catch($e$$){throw console.warn("shadow-cljs - failed to load",$i$jscomp$309_name$$),$e$$;}if($opts$$){var $globals$$=
                                                                                                                                                                                                                                                                                                                                                                                                                                   ^
TypeError: Cannot read properties of undefined (reading 'abs')
    at Fa (public/bench/js/bench.js:208:349)
    at Object.<anonymous> (public/bench/js/bench.js:225:174)
    at Object.shadow$provide.<computed> (public/bench/js/bench.js:225:258)
    at Object.shadow.js.jsRequire (public/bench/js/bench.js:4428:332)
    at Object.shadow.js.require (public/bench/js/bench.js:4429:417)
    at public/bench/js/bench.js:10843:137
    at public/bench/js/bench.js:11772:4

simongray09:09:36

When using a custom handler fn, is there any way to intercept and modify the request map being sent to the handler in the shadow-cljs.edn configuration?

thheller15:09:20

that question doesn't compute. you said custom handler fn, you can do whatever you want in that? I assume that is what you configured in shadow-cljs.edn?

thheller15:09:35

but as always I recommend using your own server if you need anything special from your server

thheller15:09:59

you absolutely do not need to use the built-in shadow-cljs servers, they are purely for convenience. they are not required for hot-reload or repl to work

simongray07:10:34

what I meant was, is there a way I can modify the contents of the request being sent to the handler, but never mind.

simongray07:10:05

It’s because I needed to add some authorization-override data to it . I just use a separate handler for shadow-cljs now.

Carlo12:09:14

question: I started splitting the dependencies in a deps.edn file because I needed a dependency to be a local checkout (I followed https://shadow-cljs.github.io/docs/UsersGuide.html#deps-edn). But now, when I cider-jack-in-cljs, should I choose clojure-cli or shadow-cljs as the connection method? (before, I was choosing the latter). Both give me problems: • clojure-cli will give me:

Execution error (ExceptionInfo) at shadow.cljs.devtools.server.runtime/get-instance! (runtime.clj:11).
shadow-cljs has not been started yet!
In embedded mode you need to call (shadow.cljs.devtools.server/start!) to start it.
If you have a shadow-cljs server or watch running then you are not connected to that process.
shadow-cljs will give me:
[:app] Build failure:
The required namespace "minimal.core" is not available.

Carlo12:09:29

solved this particular problem, using shadow-cljs in the end (I was using a wrong path in paths)

Joshua Smock17:09:25

Hi 👋 new to the Slack group 🙂 We (Pitch) have been looking into using https://jestjs.io as a test runner for some of our code. We’ve gotten an initial prototype working by using the :browser-test target combined with enabling :devtools :loader-mode :script, as well as loading the individually compiled files using an approach similar to the :node-test target, using vm. It more or less works, but is a little hacky. Thinking about some improvements to this, one thing that looks interesting is Jest’s https://jestjs.io/docs/code-transformation capability, as it could theoretically allow us to “run” CLJS files directly rather than running Jest against the compiled target output. To do this we would need to be able to do two things: 1. Compile an arbitrary file, with goog.require statements that also handles node modules in a nice way, and 2. Generate the dependencies for that arbitrary file. I’m curious how feasible this would be, or if there would need to be work done to enable this. Any thoughts?

thheller18:10:14

:npm-module is probably your best bet. I don't know how jest works internally though so no clue what it needs. the code transformation definitely won't work since that can't invoke JVM code

thheller18:10:31

(or could but in a rather slow way, making it effectively unusable)

Joshua Smock18:10:39

My thinking for the transformation was to call out to some JVM process via a thread/process call, and then do some light transformation so that Jest can use it (unless :npm-module already handles the require calls correctly?)

Joshua Smock18:10:21

Jest caches the transformed code so speed is not too important, but if we needed it we could spin up some daemon when Jest starts so that the JVM is warm

thheller18:10:47

I don't know how jest operates or what it needs

thheller18:10:01

:npm-module is just the most generic way to run CLJS code in JS based tools

thheller18:10:13

it is meant to just work and usually does

thheller18:10:39

but again ... no clue how Jest operates 😛

Joshua Smock18:10:36

it does take JS code 😄 It’s kind of hard to describe how Jest works succinctly, but it basically can take some JS code and use globals like describe and it to run tests. If it sees a require statement it does some additional stuff too. Would using :npm-module work well with :loader-mode :script ? Jest works best with individual files because it parallelizes the files it runs as well, so a single bundled file isn’t ideal

thheller18:10:25

:loader-mode :script is an entirely different thing and doesn't apply

thheller18:10:55

just try :npm-module. if jest really doesn't do anything special it'll just work

Joshua Smock18:10:27

I’ll give it a try and let you know how it works! I’m not fully confident because Jest is a bit complex but maybe it does

thheller18:10:38

if :browser-test worked then it'll most definitely work

Joshua Smock18:10:42

ah, but I rely on ns-regexp, will this also work with :npm-module?

Joshua Smock18:10:15

I can make a registry but prefer not to

thheller18:10:34

yes, :ns-regexp works

👍 1
Joshua Smock18:10:09

I’m having an issue with the build that seems related to ns-regexp. When it’s present, I get a stack trace:

[:jest] Configuring build.
[:jest] Compiling ...
[:jest] Build failure:
NullPointerException: Cannot invoke "java.util.concurrent.Future.get()" because "fut" is null
    clojure.core/deref-future (core.clj:2304)
    clojure.core/deref (core.clj:2324)
    clojure.core/deref (core.clj:2310)
    shadow.build.classpath/get-classpath-entries (classpath.clj:808)
    shadow.build.classpath/get-classpath-entries (classpath.clj:807)
    shadow.build.classpath/find-cljs-namespaces-in-files (classpath.clj:1250)
    shadow.build.classpath/find-cljs-namespaces-in-files (classpath.clj:1247)
    shadow.build.test-util/find-test-namespaces (test_util.clj:16)
    shadow.build.test-util/find-test-namespaces (test_util.clj:7)
    shadow.build.targets.npm-module/resolve* (npm_module.clj:78)
    shadow.build.targets.npm-module/resolve* (npm_module.clj:67)
    clojure.core/apply (core.clj:669)
    clojure.core/update-in/up--6870 (core.clj:6174)
    clojure.core/update-in/up--6870 (core.clj:6173)
    clojure.core/update-in (core.clj:6175)
    clojure.core/update-in (core.clj:6161)
    shadow.build.targets.npm-module/resolve (npm_module.clj:102)
    shadow.build.targets.npm-module/resolve (npm_module.clj:100)
    shadow.build.targets.npm-module/process (npm_module.clj:112)
    shadow.build.targets.npm-module/process (npm_module.clj:105)
    clojure.lang.Var.invoke (Var.java:384)
    shadow.build/process-stage/fn--16319 (build.clj:160)
    shadow.build/process-stage (build.clj:157)
    shadow.build/process-stage (build.clj:149)
    shadow.build/resolve (build.clj:405)
    shadow.build/resolve (build.clj:401)
    shadow.build/compile (build.clj:439)
    shadow.build/compile (build.clj:432)
    shadow.cljs.devtools.server.worker.impl/build-compile (impl.clj:363)
    shadow.cljs.devtools.server.worker.impl/build-compile (impl.clj:344)
    shadow.cljs.devtools.server.worker.impl/eval17936/fn--17938 (impl.clj:439)
    clojure.lang.MultiFn.invoke (MultiFn.java:234)
    shadow.cljs.devtools.server.util/server-thread/fn--17698/fn--17699/fn--17707 (util.clj:284)
    shadow.cljs.devtools.server.util/server-thread/fn--17698/fn--17699 (util.clj:283)
    shadow.cljs.devtools.server.util/server-thread/fn--17698 (util.clj:256)
    java.lang.Thread.run (Thread.java:831)
if I remove :ns-regexp it goes further and produces an expected error rather than the stack trace

thheller18:10:37

upgrade shadow-cljs. that should be fixed

👀 1
Joshua Smock19:10:27

It works! and actually out of the box, without much extra work. I’m assuming it’s possible to do this programmatically (building a target on the fly rather than one from shadow-cljs.edn), so the only question remaining is if it’s possible to get a manifest.json file for the :npm-module build. I added :build-options :manifest-name "manifest.json" to my config but it didn’t seem to work

thheller05:10:18

what do you use manifest for? thats a feature specific to browser based builds and doesn't really make sense for npm-module?

Joshua Smock11:10:58

The manifest might not be the best tool for this but I would like to get an analysis of the dependencies (filenames, specifically) of a specific file. If I have that then I can give it to Jest and it can figure out which tests to run based on changed files