Fork me on GitHub
#shadow-cljs
<
2019-08-20
>
Quest02:08:19

I have an inconsistently occurring bug to report -- In Emacs, if I start M-x cider-jack-in-cljs it usually (90% of time) fails with below output. One in 10 runs inexplicably succeeds.

[:app] Configuring build.
[:app] Compiling ...
[:app] Build completed. (608 files, 1 compiled, 0 warnings, 3.68s)
To quit, type: :cljs/quit
user> [:selected :app]
shutting down ...
cljs.user> 
*** Closed on Mon Aug 19 18:38:22 2019 ***
If I delete the .shadow-cljs folder before running cider-jack-in-cljs, it always succeeds.
[:app] Build completed. (608 files, 607 compiled, 0 warnings, 16.99s)
To quit, type: :cljs/quit
user> [:selected :app]
cljs.user> 
Starting via lein dev works fine 100% of the time. Deleting the folder .shadow-cljs/builds/app/dev/ana/northstar seems to raise the success rate to around ~50%

Quest02:08:42

output from the nrepl-server buffer:

shadow-cljs - server pid exists but server appears to be dead, proceeding without server.
shadow-cljs - config: /Users/quest/repos/ns/browser/shadow-cljs.edn  cli version: 2.8.46  node: v12.8.0
shadow-cljs - running: lein update-in :dependencies conj [nrepl "0.6.0"] -- update-in :dependencies conj [cider/piggieback "0.4.0"] -- update-in :dependencies conj [refactor-nrepl "2.4.0"] -- update-in :dependencies conj [cider/cider-nrepl "0.21.1"] -- run -m shadow.cljs.devtools.cli --npm -d nrepl:0.6.0 -d cider/piggieback:0.4.0 -d refactor-nrepl:2.4.0 -d cider/cider-nrepl:0.21.1 server
shutting down..

Quest02:08:17

* Whether I launch the browser or not (when prompted) seems to have no effect. * Deleting .shadow-cljs/builds/app/dev/ana/cljs seems to raise success rate to 100%. I notice that it's a 5MB+ folder, which probably causes the compilation step to take longer. * However deleting .shadow-cljs/builds/app/dev/ana/re_frame still results in a noticeable amount of failure -- but it's only a 500K library, so maybe similar to deleting my ana/northstar/browser (project's) folder

Quest02:08:15

My guess is some sort of race condition that's negated by sufficiently increased compilation time. Unfortunately the guess is all I have, but I'll assist with troubleshooting however possible

Quest02:08:03

Running on OSX Mojave 10.14.6, GNU Emacs 26.2

thheller06:08:43

@quest try limiting the amount of RAM shadow-cljs uses by setting :jvm-opts ["-Xmx2G"] (or 1G) in project.clj

Quest16:08:58

Can confirm -- setting :jvm-opts ["-Xmx1G"] in project.clj :profiles -> :dev completely fixes it. Doesn't seem to matter if it gets 1G or 2G. (FYI to anyone later reading this, tried setting it in shadow-cljs.edn with no effect, so make sure you set it in project.clj)

thheller16:08:30

yes, when using lein you must set it in project.clj

thheller16:08:48

still would like to know why it runs out of memory when running in cider but not otherwise though 😛

Quest16:08:53

@thheller Thanks for the help. I was quite baffled by this for the last week -- I recall that you recommended bumping memory a few days ago, but I thought the cause was something else 😅

thheller16:08:27

1G is usually enough unless you have really big builds then :advanced compilation might require more

Quest16:08:28

yeah, that I'm not sure of. only guess is the extra CIDER dependencies increasing mem-usage past some critical threshhold

kasuko17:08:28

So I was running into something similar, when a I would run shadow-cljs watch app without the server running and I would get the attached error. This ran fine if I started the server ahead of time. Sure enough limiting the RAM helped. I have no clue why running this with more RAM causes this to break. If I run the command while providing 4GB to Java it will cause the same issue. (I'm sitting at 6GB in use out of 16GB right now, so it's not running out of memory) EDIT: Just to be clear, this issue is FIXED by adding :jvm-opts ["-Xmx2G"] to my shadow-cljs.edn file. I was just wondering if there was any reason why that fix works?

thheller18:08:35

on CircleCI reaching a certain amount of memory caused the OOM Killer from the OS to kill the process

thheller18:08:56

so maybe something similar exists on OSX? I'm not sure

thheller18:08:19

might be something in emacs itself that causes the kill

thheller18:08:07

something from the OS must be doing that. the JVM doesn't kill itself and neither does shadow-cljs

thheller18:08:26

I'd be very curious to find out what is happening but dunno where to look and this has never happened to me either

Quest19:08:35

Did some more testing. If I set :jvm-opts ["-Xmx4G"] in project.clj, it gives the shutting down.. start message at least 30%+ of the time. Values of 1G or 2G seem 100% stable. Setting :jvm-opts in shadow-cljs.edn seems to have no effect regardless of 1G, 2G, or 4G

Quest19:08:18

No idea why this is -- but here's a standing offer to help troubleshoot if you get any ideas

Quest19:08:42

(My system is at 18 / 32gb, so don't think it's related to OOM)

thheller19:08:06

dunno I also have 32gb with about 16gb free

thheller19:08:17

the only thing I don't have is emacs and cider

thheller19:08:29

no clue what does

Quest20:08:44

did some more testing, can repro the shutdown >50% time with /usr/local/bin/npx shadow-cljs watch app

Quest20:08:14

Tried to look through the OSX kernel logs but didn't notice anything suspicious. I'm out of ideas for now :man-shrugging:

Quest20:08:47

(did try setting :jvm-opts in shadow-cljs.edn to 1/2/4G values, didn't seem to make a difference)

thheller20:08:32

is that with cider being connected or completely standalone?

Quest20:08:57

completely standalone via /usr/local/bin/npx shadow-cljs watch app

thheller20:08:37

no clue. the only thing triggering a shutdown should be either the JVM process getting killed, the node process getting killed or explicitely calling (shadow.cljs.devtools.server/stop!)

Quest20:08:28

Possible that it's self killing due to requesting too many threads too quickly & getting execution denied? seems tied to OSX somehow. Running on a 2019 macbook pro.. wonder if an artificial Thread/sleep at some point in startup would "fix" this

Quest20:08:51

(remembering that deleting the 5MB cljs folder from .shadow-cljs cache made the issue go away 100% of the time for that run)

thheller20:08:19

[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 242]

thheller20:08:31

the thread pool is getting shutdown, no overload in sight

thheller20:08:13

please try running shadow-cljs clj-repl and just let it sit for a bit

thheller20:08:22

see if shuts down on its own too

thheller20:08:40

of not try running (shadow/watch :app {:verbose true}) after a bit

Quest20:08:44

initialized to shadow.user=> just fine, letting it sit a minute

Quest21:08:22

a subsequent run got through the [:app] Build completed (608 files, 1 compiled, 0 warnings, 4.34s) and gave me back the [1:0]~shadow.user=>, so seems that we did trigger a shutdown somehow

thheller22:08:43

curious that it even gets to the build completed message since the shutdown is triggered somewhere in the middle

thheller22:08:19

but I guess the work is already queued so it just keeps going and waits with the actual shutdown

thheller22:08:14

otherwise looks fine. nothing suspicious except the shutdown getting triggered in the middle

thheller22:08:39

which java -version do you use?

Quest22:08:22

> java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

Quest23:08:34

I can try a different one if you'd like, just let me know which

thheller06:08:23

no idea. you could try jdk11 or 12 maybe but I'd be very surprised if that changes anything

thheller06:08:26

you could maybe try running (shadow/compile :app) to see if that also gets terminated

thheller06:08:43

and then maybe (shadow/release :app)

thheller06:08:57

the compilation process is almost identical so I presume they are all going to be shut down

thheller06:08:14

but release won't include all the re-frame-10x code so it might survive due to using less resources

thheller06:08:16

dunno why deleting cache files changes your success rate since compilation recreates the exact same state again anways

thheller06:08:18

no clue what emacs does when running processes. maybe it is somehow monitoring the processes or causes it to have other resource constraints

thheller06:08:30

but other people have reported issues with cider as well and limiting the memory seems to help

jpmonettas12:08:29

hi everybody! is there a place in shadow-cljs.edn to add es6 js files I want to add locally (not coming from npm). My use case is that I'm using a library that needs me to provide it objects that extends from others and I think I prefer to leave those as es6 js code instead of replicating that in clojurescript with goog.object/extend etc

thheller12:08:31

just put it next to your .cljs files

jpmonettas12:08:37

thanks a lot!

kasuko17:08:28

So I was running into something similar, when a I would run shadow-cljs watch app without the server running and I would get the attached error. This ran fine if I started the server ahead of time. Sure enough limiting the RAM helped. I have no clue why running this with more RAM causes this to break. If I run the command while providing 4GB to Java it will cause the same issue. (I'm sitting at 6GB in use out of 16GB right now, so it's not running out of memory) EDIT: Just to be clear, this issue is FIXED by adding :jvm-opts ["-Xmx2G"] to my shadow-cljs.edn file. I was just wondering if there was any reason why that fix works?

tianshu18:08:35

can react and react-dom be benefit from optimizations of closure compiler?

tianshu18:08:47

the react-dom bundle size is pretty large

thheller18:08:30

in theory they can but it is not recommended and requires tweaking by hand

thheller18:08:47

could also break with any new release and most OTHER libraries don't work

thheller18:08:02

also doesn't make it that much smaller

thheller18:08:06

so not worth the effort

tianshu18:08:30

I want to reduce the bundle, in my case now it's about 370kb but use on not so good network

thheller18:08:45

is that gzip'd?

tianshu18:08:51

Yes, gzip'd

thheller18:08:21

hmm yeah thats a bit large. did you create a build report? https://shadow-cljs.github.io/docs/UsersGuide.html#_build_report

tianshu18:08:59

There're few dependencies I have plan to remove, like expound

thheller18:08:00

clojurescript seems too large? maybe you included cljs.pprint?

tianshu18:08:19

I don't know why, is it normal?

thheller18:08:49

click the little expand icon in front of clojurescript

tianshu18:08:03

It's about 5731 lines of cljs

tianshu18:08:23

cljs/core.cljs 232.04 KB cljs/pprint.cljs 98.44 KB cljs/analyzer.cljc 95.13 KB cljs/spec/alpha.cljs 54.73 KB cljs/spec/gen/alpha.cljs 23.27 KB cljs/spec/test/alpha.cljs 12.15 KB cljs/stacktrace.cljc 5.56 KB cljs/reader.cljs 3.46 KB

tianshu18:08:36

Ohh, pprint

thheller18:08:45

hmm do you use self hosted stuff?

thheller18:08:07

hmm then cljs.analyzer shouldn't be in there either

tianshu18:08:08

wait, what you mean by self hosted

tianshu18:08:34

Is that if I use it in macro, that cljs will be included as well?

thheller18:08:52

depends on how you included it?

tianshu18:08:27

Just like pprint, I didn't use it, just forgot to delete the require in ns form

thheller18:08:27

if you use the web UI (default at http://localhost:9630)

thheller18:08:39

there is a little UI thingy to tell you why a namespace was included in a build

tianshu18:08:22

wait a moment, I'm going to start this

thheller18:08:09

pprint unfortunately isn't compatible with DCE in the way it is currently written https://clojure.atlassian.net/browse/CLJS-2885

tianshu18:08:08

web UI looks great

thheller18:08:31

if you select your build and hit compile

thheller18:08:46

it'll have a "Build Namespaces" select box at the bottom

thheller18:08:02

select cljs.analyzer and it'll tell you why it is included (which namespace included it)

thheller18:08:25

but I guess you already found it 😉

tianshu18:08:33

reitit.coercion.spec -> spec-tools.core -> spec-tools.impl -> cljs.analyzer.api -> cljs.analyzer

thheller18:08:01

interesting. someone should open a bug report there 😉

tianshu18:08:14

the spec-tools is using this...

thheller18:08:25

yeah but it very likely doesn't actually need it

thheller18:08:45

cljs.analyzer.api is very likely only used from the CLJ macro side

thheller18:08:53

so including it in the CLJS side is pointless

tianshu18:08:25

cljs.spec is also not so small

tianshu18:08:01

I found one thing, report is not support if there's multiple modules?

tianshu18:08:57

but In my project it doesn't, I have two moudles, one is :base :base {:entries []}

tianshu18:08:06

:strip-type-prefixes #{"cljs.spec"} where can I put this

tianshu18:08:18

I also just want to remove all spec

thheller18:08:22

doesn't work for spec anymore

thheller18:08:01

an empty :base module doesn't make sense when you only have 2 modules

tianshu18:08:42

In my dev build, I have another module called devcards, main and devcards both depends on base

thheller18:08:03

dev build? you are likely using shadow-cljs wrong if you have a build named :dev

thheller18:08:14

thats not how it works at all 😛

tianshu18:08:21

no, actually it is app.

thheller18:08:40

devcards should likely be its own build

thheller18:08:48

definitely wrong to include that as a :module

tianshu18:08:01

should I make it a build?

thheller18:08:05

a good rule to remember for :modules is that you should only use them if and only if ONE user may end up using ALL modules

thheller18:08:28

ie :devcards likely won't be used by your users and only by you during development

thheller18:08:34

therefore it should be in a different build

tianshu18:08:36

But If I make it another build, I know I can start two build in shadow, but Can I connect to two browser environment in one cljs repl?

thheller18:08:55

shadow-cljs watch app devcards starts two builds

thheller18:08:10

no to the REPL. they are separate

thheller18:08:32

but they would be anyways?

thheller18:08:48

you likely aren't loading your app and devcards in the same browser tab?

tianshu18:08:51

You know, cider (basically except cursive) rely on repl to provide auto-completion and many other features.

thheller18:08:07

Cursive doesn't rely on that no

tianshu18:08:15

Yes, I know

tianshu18:08:34

but it's easy to solve, I can just start two emacs instances

thheller18:08:54

you can also just load your main entries in the devcards build

tianshu18:08:55

I'll make it another build.

thheller18:08:06

so they have the code loaded

thheller18:08:37

but yeah cider relying on nrepl so much may be a problem with too many builds

tianshu18:08:07

yes, it is

tianshu18:08:13

I just remove pprint

tianshu18:08:27

how can I easily remove all specs? I have a few cljc file

thheller18:08:58

unfortunately can't

thheller18:08:45

or move all specs in a separate files and include them via :preloads

tianshu18:08:47

How can I add conditional dependencies depend on :dev and :release with tools-deps

tianshu18:08:11

:preloads?

tianshu18:08:31

currently I have devtools, but it appears in the report.

thheller18:08:45

in the build report?

tianshu19:08:37

No, I mean

tianshu19:08:42

In the WEB UI

thheller19:08:43

if you included that properly via preloads then it shouldn't be in there

thheller19:08:54

the web ui shows you what you built

tianshu19:08:58

but pprint is still in report

thheller19:08:05

so if you pressed the compile button it shows you a :dev build

tianshu19:08:12

and in web UI I found it is required by devtools

thheller19:08:17

press the release button instead to see the release deps/namespaces

tianshu19:08:39

I'm trying to regenerate a report

tianshu19:08:30

org.clojure/clojurescript is still 400k

thheller19:08:11

well whats in it?

thheller19:08:44

with spec that seems about right

tianshu19:08:30

Yes, only one thing, cljs.analyzer. I have to switch a router to remove that.

tianshu19:08:45

OH, I can patch that maybe

thheller19:08:01

can probably switch that to :clj instead

thheller19:08:07

I hate .cljc so much. so misunderstood and often used incorrectly

tianshu19:08:45

sometimes i like it because can share code, especially configuration.

thheller19:08:25

yeah but it is so easy to get wrong, especially with macros

thheller19:08:42

I guess they are doing this to support the self-hosted stuff but even then it seems wrong

thheller19:08:00

but yeah conditional code include is tough and needs a better solution

tianshu19:08:34

don't know the reason, maybe caused by eval loader

tianshu19:08:44

sometimes, the chrome hangs

thheller19:08:18

depends on how much code you load I guess. from your build-report it doesn't seem too excessive. but dunno how much dev stuff you add (eg. re-frame-10x adds a whole lot)

tianshu19:08:29

often happened when I switch build

tianshu19:08:40

not happened in common usage

thheller19:08:58

dunno. never happens for me so you'll have to debug a bit yourself

tianshu19:08:31

I copy that file change that macro to always return nil. I think it won't affect anything in my project

tianshu19:08:55

cljs is 296 kb now

tianshu19:08:56

I was trying to use module split in my project, and it works pretty good. however it won't load module in a browser, (that is our mainly target browser), no tools or any information to debug. So have to give it up.

tianshu19:08:39

The result is good to me now, it's gizp'd 300kb. I'm going to remove a few more things. thanks for the great help today!!

tianshu19:08:45

shadow is awesome!

4
Quest19:08:57

shadow-cljs is a great tool & the level of support thheller provides here is exceptional. If you agree, please consider subbing https://www.patreon.com/thheller/overview to keep it rolling 🙂

❤️ 16
metehan19:08:57

;; shadow-cljs configuration
{:source-paths
 ["src/dev"
  "src/spill"
  "src/test"]

 :dependencies
 [[org.clojure/core.async "0.4.500"]
  [reagent "0.8.1"]
  [re-frame "0.10.6"]
  [bouncer "1.0.1"]
  [com.andrewmcveigh/cljs-time "0.5.2"]
  [clj-commons/secretary "1.2.4"]
  [venantius/accountant "0.2.4"]
  [binaryage/oops "0.7.0"]
  [binaryage/devtools "0.9.10"]
  [day8.re-frame/re-frame-10x "0.3.7-react16"]
  [day8.re-frame/tracing "0.5.1"]]

 :builds {:app {:output-dir "web/root/js/shadow"
                :asset-path "js/shadow"
                :target :browser
                :closure-defines {"re_frame.trace.trace_enabled_QMARK_" true}
                :preloads [devtools.preload day8.re-frame-10x.preload]
                :modules {:app {:init-fn spill.app.core/init}}
                :main "spill.app.core"
                :devtools {:http-root "web/root"
                           :http-port 8080}
                :compiler-options {:optimizations :none}}}}

metehan19:08:26

with this configuration i can't run re-frame-10x

metehan19:08:31

what i am doing wrong

stefan20:08:24

that config works for me

stefan20:08:12

I think moving :preloads into :devtools, and adding re-frame.trace/trace-enabled? true to your :closure-defines map may do it

thheller20:08:11

:main "spill.app.core" that does nothing and can be removed

thheller20:08:24

:compiler-options {:optimizations :none} also does nothing and can be removed

thheller20:08:47

besides that I think it may require an extra closure-define?

thheller20:08:23

oh and :preloads belongs in :devtools

thheller20:08:47

or into :modules {:app {:init-fn ... :preloads [...]}}

metehan20:08:56

@thheller @stefan.age thank you very much it's working now 🙂

isak22:08:06

Is it possible to configure something like an :asset-path to go along with [:devtools :watch-dir] ? Watch-dir does not work for me, and I think it is because the URLs are not generated like :watch-dir expects (just /some/path under :watch-dir) - in my case I have a prefix.

thheller12:08:54

not sure I understand you question? can you provide an actual example of paths?

isak13:08:56

I have this in my HTML: /f/css/front_dev.css , and in [:devtools :watch-dir] I have this: resources/public/css

thheller13:08:46

so the actual files lives in resources/public/css/f/css/front_dev.css?

thheller13:08:12

watch is assuming it is watching files that are directly served by a webserver

thheller13:08:30

if it is not then it won't work

isak13:08:31

No, the actual paths lack the /f/ there

thheller13:08:00

so what is /f/? why does that exist?

isak13:08:50

It just makes the hosting easier. For example for the main build, I have this:

:output-dir "resources/public/js/compiled"
	 :asset-path "/f/js/compiled"
and therefore that part works

thheller13:08:52

the system would expect you to use :watch-dir "resources/public" and then put the actual file into resources/public/css/front_dev.css and use /css/front_dev.css as the path in HTML

thheller13:08:16

still doesn't explain what the /f is

isak13:08:26

yea, but that is because there is no equivalent of the asset path for the watch dir, though

isak13:08:50

its just a "virtual directory" on the web server that hosts everything under "resources/public"

thheller13:08:38

why is it a virtual directory? why is it not an actual directory?

isak13:08:51

It is kind of a routing simplification, since this is clojurescript being added to a legacy application

thheller13:08:48

sorry I don't follow how that plays into anything? why not just create resources/public/f/css/front_dev.css? and have you webserver serve resources/public like every web server out there?

isak13:08:18

It already has a different public directory

thheller13:08:39

and you can't put files there?

isak13:08:54

I can, but then I'd have other problems with source control

thheller13:08:54

I can't think of an easy way to do this right now. I don't want to overcomplicate the config

isak13:08:28

Ok, I'll look into the hard way

thheller13:08:54

I could maybe add a secondary attribute to the <link tag itself that is used to compare

thheller13:08:42

right now it only checks the href but it could also check something like <link rel="stylesheet" href="/f/css/front_dev.css" data-css-path="/css/front_dev.css" />

thheller13:08:16

or some other html attribute that lets you specify which actual file this is supposed to be

isak13:08:28

yea that would work. though isn't this conceptually the same as asset-path, just for the css?

isak13:08:00

{:watch-dir "resources/public" :watch-asset-path "/f"}

thheller13:08:57

hehe. just looked at the code and guess what ...

thheller13:08:07

{:watch-dir "resources/public" :watch-path "/f"}

thheller13:08:22

already exists 😛

isak13:08:35

omg 🙂

thheller13:08:05

I can't even remember that I ever added that

thheller13:08:21

wasn't even that long ago 😛

thheller13:08:37

but that should do what you need right?

isak13:08:53

nice, and it is implemented? yea, sounds perfect

thheller13:08:21

yeah apparantly has been there for about 6 months 😛

isak13:08:23

wow that worked, thanks!!

thheller13:08:23

added it to the docs as well now. maybe this time I won't forget this exists 😉