I am noticing weird things too when reloading and I thought it was on my code base. I will try to repro after the update

is there a way to use node_modules that are npm linked? It seems shadow-cljs doesn't pick them up.

@thheller getting this error

The required namespace "konserve.filestore" is not available, it was required by "infinitor/collector/database.cljs".
"konserve/filestore.clj" was found on the classpath. Should this be a .cljs file?

@emil0r looking at the repo isn't always the correct answer. looking at the .jar is. and that doesn't include the filestore.cljs. looks like that was created after the last release.

indeed

@bbss how so? it should be working as long as node_modules/the-thing/... exists. the-thing is allowed to be a link. maybe need to upgrade if you get a complaint about not being allowed to use sources outside the project. that was fixed.

ah, my local global version might be out of date. Cheers!

I'm seeing linear growth in compilation times:

[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 1.62s)
------ WARNING #1 --------------------------------------------------------------
point_cloud_layer_fragment_glsl assigned a value more than once.
--------------------------------------------------------------------------------
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 2.72s)
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 3.80s)
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 6.44s)
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 12.04s)
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 23.20s)
[:main] Build completed. (1197 files, 3 compiled, 0 warnings, 57.57s)
;;after shadow-cljs restart
[:main] Build completed. (1197 files, 1040 compiled, 0 warnings, 112.28s)

however after restart + rm -rf .shadow-cljs the recompilation is fast again

which version are you on? there was a bug like that a while ago

2.6.2

Updating before asking my next question 😅

hmm no that definitely had the fix for that problem

if you see this again please run with --verbose

otherwise impossible to tell where the time is going

okay, will do

if you have side-effecting macros I would check those first

Not that I'm aware of.

the cache issue where compiles got slower over time was fixed in 2.4.20 so must be something else

they also didn't get slower but that much so quickly 😉

> ... hangs a long time here....

where? the point of the log is to first log when something starts and then again when something completes

so there should be a <- msg with a high ms number?

that's where it hangs. exactly there

hmm thats odd .. there is no pending stuff unless you removed some lines?

any -> without a matching <-?

> <- Closure JS Cache read: 4 JS files (2427 ms)

this is suspiciously slow

please check how large .shadow-cljs/builds/main/dev/shadow-js/index.json.transit is?

and .shadow-cljs/builds/main/dev/closure-js/index.json.transit

-rw-r--r-- 1 baruchberger staff 155600 Oct 24 22:35 .shadow-cljs/builds/main/dev/shadow-js/index.json.transit -rw-r--r-- 1 baruchberger staff 298854289 Oct 24 22:55 .shadow-cljs/builds/main/dev/closure-js/index.json.transit

what the heck 😛

300mb?

😅

does transit somehow cache requests from my app?

No clue why it would be that big

no that is compiler cache data

do you import any .js files directly?

Yes

are they this large?

Hmm, I don't think so.

The files themselves are small, maybe their deps?

check the output directory .shadow-cljs/builds/main/dev/closure-js

whats the largest file?

that one, by far

yeah thats clear since it basically contains all the others

-rw-r--r--  1 baruchberger  staff  298854289 Oct 24 22:55 index.json.transit
-rw-r--r--  1 baruchberger  staff       3958 Oct 24 19:43 module$gsv$api$partition.js
-rw-r--r--  1 baruchberger  staff      20392 Oct 24 22:53 module$gsv$ui$custom_point_cloud.js
-rw-r--r--  1 baruchberger  staff       5023 Oct 24 19:43 module$gsv$ui$point_cloud_layer_fragment_glsl.js
-rw-r--r--  1 baruchberger  staff       7160 Oct 24 19:43 module$gsv$ui$point_cloud_layer_vertex_glsl.js

so it looks like closure is doing crazy stuff to one of your inputs

it should never get this large

the only thing I can imagine somehow, this morning I tried to use shadow-cljs to build a npm package, on a different project, it npm packs the directory (100mb) in the process.

but it's a different project entirely.

it makes no sense for the file to be that big .. its just an index ...

can you send it to me?

zipped please 😛

Sorry I just deleted it. When it happens again I will.

Judging by how it went previously that will take around 20 minutes..

gah make a copy next time please

Sorry..

impossible to tell whats going on if you delete the evidence 😛

😅

btw the :module-move messages are gone in the latest version

I've pulled the latest into my shadow-cljs repo, but what was the command to install it locally again?

why do you have a local repo?

lein install

I have a local repl because earlier you pushed a fix for something and I wanted to have it quick.

its all released now 😉

if you run lein install you must run the build-all.sh first

otherwise some files will be missing

Okay, will do after I reproduce.

just observe the cache index on each compile. it should be almost constant in size depending on the amount of JS files used

even 1mb is probably too big already

-rw-r--r--    1 baruchberger  staff   10699 Oct 24 23:21 start.index.json.transit
->
-rw-r--r--  1 baruchberger  staff  26659 Oct 24 23:27 index.json.transit

no thats enough already

unless you added many js files

Do you want me to send them?

yes that would help lots

No added none, just editted a bit.

Where to send?

edits should definitely not affect the size

<mailto:[email protected]|[email protected]>

sent

ok found the problem I think

so quick 🙂

its pretty obvious if you look at the data

the index is basically just a map that tracks whatever compiler settings were active and so on

it tracks which polyfills closure injected

thats supposed to be a set

:injected-libs ("es6/util/inherits"
                 "es6/util/setprototypeof"
                 "util/objectcreate"
                 "es6/util/arrayfromiterable"
                 "es6/util/arrayfromiterator"
                 "es6/util/makeiterator"
                 "es6/symbol"
                 "es6/object/assign"
                 "util/polyfill"
                 "util/global"
                 "util/defineproperty"

but it ends up as a list. so it basically doubles in size on each compile since it re-adds everything

that's why it was so linear in time increase 🙂

now only to figure out where it becomes a list ...

Alright, no rush from my side. I can live with a rm + restart once in a while.

no need for restart. you can just delete the index file

even better :thumbsup:

wow doh ...

injected-libs
        (-> (.get injected-libraries-field cc)
            (keys)
            (into #{})

-> instead of ->> ...

1 char bug ㅋㅋㅋ

TIL: (clojure.set/union #{} '(1 2 3)) => (1 2 3)

meh

Hmm, that is surprising indeed.

set/union has so many weird behaviours ... should just use into ...

dev> (clojure.set/union #{1 2 3} '(1 2 3))
#{1 3 2}
dev> (clojure.set/union #{1 2 3} '(1 2 3 4))
(2 3 1 1 2 3 4)

source explains it all:

dev> (source clojure.set/union)
(defn union
  "Return a set that is the union of the input sets"
  {:added "1.0"}
  ([] #{})
  ([s1] s1)
  ([s1 s2]
     (if (< (count s1) (count s2))
       (reduce conj s2 s1)
       (reduce conj s1 s2)))
  ([s1 s2 & sets]
     (let [bubbled-sets (bubble-max-key count (conj sets s2 s1))]
       (reduce into (first bubbled-sets) (rest bubbled-sets)))))

Hmm, I see the implementation is very elegant. But a few warnings on non-sets would be nice.

@bbss fixed in 2.6.17

Thank you!

I'm having trouble mapping react to preact-compat when targeting :npm-module to build the leaves (view components) of an existing JS preact app in CLJS. Would you all expect this to work for my shadow-cljs.edn?

{:source-paths ["src"]
 :dependencies [[cljs-css-modules "0.2.1"]
                [reagent "0.8.1"]]
 :builds {:npm {:target :npm-module
                :output-dir "build/cljs"
                :compiler-options {:infer-externs :auto}
                :js-options {:resolve {"react" {:target :npm
                                                :require "preact-compat"}
                                       "react-dom" {:target :npm
                                                    :require "preact-compat"}}}}}}

[:npm] Build failure:
The required namespace "react" is not available, it was required by "reagent/core.cljs".

@jeromedane :resolve is not supported for :npm-module or rather it is very limited

it only works reliably for :browser really. for :npm-module you really need to configure it in whatever you are targetting

We might want to reflect that in the documentation for :resolve?

eg. webpack or node

it is?

https://shadow-cljs.github.io/docs/UsersGuide.html#js-resolve-limitations

Ah, I see. Thanks. Suggested add > It is therefore not recommended to be used with targets that use require directly (eg. :node-script[, :npm-module]) for those of us that are a little thicker than most. Thank you!! ❤️

@gklijs asked a question about loading .wasm files in cljs

created an issue here with some context: https://github.com/thheller/shadow-cljs/issues/397

in the latest shadow, whever I do a require in the REPL I get:

another.ns> (require '[policy-fn.env :as env])
Circular dependency detected: policy-fn.env -> policy-fn.env
 (require '[policy-fn.env :as env])

only custom namespaces, no core namespace like clojure.string

actually, let me differentiate more

it is a namespace that has been added to the path but at runtime, while the JVM was running

I restarted the JVM and now I receive a bunch of [:did-not-find ...]

compilation is successful

for this file the watcher does not pick up the changes

@thheller about the code you were fixing this morning why not set instead of (into #{})

Works for -> and ->>

was there a change made to the way that module splitting works?

when I do a release build, I see this message now:

Module Entry "cambia.web.experience.dashboard.components.survey-prompt.client" was moved out of module ":survey-prompt".
It was moved to ":main" and used by #{:onboarding :survey-prompt}.

it looks like there are cross-references between the two modules

but this worked in a previous version

@lilactown yes I added that as a hard error because otherwise it goes by unnoticed

you specified that namespace as as :entries for one of your modules

but another module directly required it so it was never in that module

so effectively making the :survey-prompt module useless

Gotcha

It looks like adding : survey-prompt to the dependencies of :onboarding fixed it

if that makes sense for your module structure then that is the correct fix yes

that error message probably will get more details at some point

but the previous behavior with no errors was worse 🙂

Heh. Well luckily for us it didn't change the behavior at all while it was wrong

well it changed your modules. the code was moved to :main previously as well

so you thought you were splitting but actually everyone was always loading it

making :main larger that it would otherwise be

probably not by much if its just a few components but still

Yeah. It hadn't affected us yet because ATM we always load those modules

Using the shadow-cljs - browser quickstart should it be possible to just do an npm install to use an npm module. I tried, https://github.com/gklijs/shadow-gol-3d (last commit, rest is copy) but it keeps saying

The required JS dependency "game-of-life-3d" is not available, it was required by "gol/main.cljs".

Searched in:/Users/gerardklijs/Workspace/shadow-gol-3d/node_modules

You probably need to run:
  npm install game-of-life-3d

@gklijs there is no support for .wasm yet so that module won't load

ok, that too bad

but you found a bad incorrect error message for packages that don't have a main but only module

that could be fixed but then it would fail when trying to load the .wasm file

so not much better

no.. It's not really well supported yet. Even with webpack I could not get a production build. I could take the compiled javascript and serve it with webpack probably, but then I lose easy reloading.

I presume the .js file is Rust output?

thats really where the problem is

but thanks for the example. I can use that to test some stuff

yes, almost the default wasm-pack output, only added some utils methods at the end

I never used the Rust stuff before so no idea how that part works

Kind of surprised how hard it's with clojurescript, did get it to work with Elm.

what did you do in elm?

From rust it's pretty easy to create some wasm module, also there's a lot of js/wasm interop work done.

So far in Elm I just had the basics working, but I could load and start the wasm part, and get info back through a port.

geez the generate file is just weird

@thheller I have a Q...what can be the cause [:did-not-find policy-fn.api.query-events] error when connecting to nrepl? it does not happen in socket REPL but it does in nrepl, I switch a couple of versions of shadow, now on 2.6.10

then a REPL command timed out. follows

in-ns before require or ns

oh but has that changed? I never needed that

basically in-ns before the policy-fn.api.query-events has been compiled

still need to check how CLJS behaves in that case. I just don't know what to do

no that has not changed. has always been that way

oh ok, so there must be something on the cider side

no clue what cider does

@gklijs is it possible to tune the wasm that gets generated? right now the .js files imports the .wasm and the .wasm imports the .js file right back

thats kind of annoying to deal with since you can't process the .js in any meaningful way 😛

the browser must also be able to load it

@thheller if you can help me on the shadow side I can debug what broken

cider does this:

(cider-nrepl-request:eval (format "(in-ns '%s)" ns)
                                (cider-repl-switch-ns-handler connection))

@richiardiandrea in-ns is fine but what is happening BEFORE that? where is the require or so?

which REPL are you connected to? your app? plain node-repl?

it is a select-nrepl

thats CLJ

@thheller I don't think so, it's probably also why you need to load asynchronously

There's some sort of chicken and egg problem there

right now this is part of the code

(import "./game_of_life_3d" "__wbg_time_ff03c978915ebe44" (func $./game_of_life_3d.__wbg_time_ff03c978915ebe44 (type $t2)))
  (import "./game_of_life_3d" "__wbg_timeEnd_5ba22134470392e6" (func $./game_of_life_3d.__wbg_timeEnd_5ba22134470392e6 (type $t2)))
  (import "./game_of_life_3d" "__wbg_random_86efc8986c8a8805" (func $./game_of_life_3d.__wbg_random_86efc8986c8a8805 (type $t3)))
  (import "./game_of_life_3d" "__wbindgen_throw" (func $./game_of_life_3d.__wbindgen_throw (type $t2)))

so no matter what it will try to import the JS file relative to the path of the wasm

thats just always going to break

well it'll work if you leave the code alone and just use it via <script type="module" ...>

but typically you can pass in an import import

so you can pass in the functions instead of the wasm file trying to load them itself

Yes, or import it in index.ts, and load that with import("../ts/index.ts")

@thheller this is what is sent

(-->
  id         "20"
  op         "eval"
  session    "b5cba90b-dd4f-446a-81ea-2a02be297cb2"
  time-stamp "2018-10-24 14:32:34.813926979"
  code       "(in-ns 'laputa.policy-fn.api.query-events)"
)
(<--
  id         "7"
  session    "b5cba90b-dd4f-446a-81ea-2a02be297cb2"
  time-stamp "2018-10-24 14:32:34.864485854"
  out        "[:did-not-find laputa.policy-fn.api.query-events]
"
)
(<--
  id         "20"
  session    "b5cba90b-dd4f-446a-81ea-2a02be297cb2"
  time-stamp "2018-10-24 14:32:44.826639415"
  err        "REPL command timed out.
"
)

you didn't answer my questions

what REPL is this? where is the require for that ns?

right sorry 😄

it just isn't loaded. thats all there is to it

(in-ns 'i.do.not.exist)

this is the form sent: (do (require '[shadow.cljs.devtools.api :as shadow]) (shadow/nrepl-select %s))

after the clojure REPL boots up

yes .. that is also fine. but you are still completely ignoring my question

what REPL is this? what build :target? how is it started?

node-repl? :browser?

where is the require?

let me answer all the Qs

so

I start with yarn shadow-cljs watch az

classpath resolved with deps.edn

I launch and connect with the above

:target :azure-app?

yeah

after that I don't do any require

I connect with cider and try to switch namespace

run yarn shadow-cljs watch az --verbose

trying

does it compile your ns?

did I get REPL working for the azure stuff? can't remember really

ah node-lib. so should be fine

@thheller it does not seem to compile that namespace for some reason

then nothing requires it?

whats your fn-map?

yes I think that might be right

so weird

ok but thanks for getting my head straight

I think it would be much safer and less headache if cider just sent (require 'laputa.policy-fn.api.query-events) (in-ns 'laputa.policy-fn.api.query-events)

yeah I agree with you and I will open an issue

in-ns without require first also sucks in clojure

it does not make sense to do anything different

yeah

(in-ns 'foo.bar)
=> #object[clojure.lang.Namespace 0x25e67a35 "foo.bar"]
(+ 1 2)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: + in this context, compiling:(C:\Users\thheller\AppData\Local\Temp\form-init11452515739617339391.clj:1:1) 

also safer with require first if you switch into non-existent namespaces

yep let me check with the cider folks but I totally agree

@thheller I am opening an issue in cider but I have a small suggestion for shadow as well. What if the error messages returned [:not-required .....]? It would be clearer what you have to do next and what the problem was

its basically no "not implemented error" because I don't know what to do

need to check what the default CLJS repl does first

ok cool I will open an issue then

thanks a lot again

ok opened and this is what happens in vanilla:

$ cljs-node-repl 
[Rebel readline] Type :repl/help for online help info
cljs.user=> (in-ns 'foo.bar)

foo.bar=> (+ 1 2)
3
foo.bar=> 

ok so it just creates an empty ns?

yeah...

what happens if you in-ns into a ns that actually exists but was not required yet?

uhm I would have to try on a project, will do it later ok? now I need to get stuff done for the day 😄

I can try myself later

or rather tommorrow 😉

oh, also one day I think you suggested me to use shadow/repl instead shadow/select-nrepl, maybe I should PR cider

yeah 😄

doesn't really matter for cider given that it always uses nrepl

ok coolio

as a user its nicer to type shadow/repl IMHO but for a tool it doesn't matter

totally, good point

OK I have a weird thing I'm trying to do

(defmacro defstart! [args & body]
  `(do
     (defn ~'-start! ~args
       ~@body)
     (with-meta
       (defonce
         ~'start!
         (hecate.loader/reloadable (var ~'-start!)))
       {:export true :dev/after-load true})))

I'm trying to set the meta-data {:export true :dev/after-load true} from a macro. not having any success yet

ideas?

ah.

(defmacro defstart! [args & body]
  `(do
     (defn ~'-start! ~args
       ~@body)
     (defonce
       ~(with-meta 'start! {:export true :dev/after-load true})
       (hecate.loader/reloadable (var ~'-start!)))))