Fork me on GitHub
#shadow-cljs
<
2018-10-19
>
grounded_sage04:10:07

I’m thinking hard about the caching provided by shadow and how to avoid wrecking it with my ideal stack. Just curious whether FactUI breaks caching due to being based on Clara. https://github.com/arachne-framework/factui

thheller08:10:06

@grounded_sage yeah clara-rules is not very cache friendly. IIRC the problematic part of fact-ui is the f/rulebase call

thheller08:10:28

so the ns that calls that has to be tagged with ^:dev/always

thheller08:10:25

@richiardiandrea the threadpool was Terminated? did you ctrl+c or something? normally a watch never terminates the threadpool? not even when the watch is stopped since its shared on the entire system (only stop! does that)

urbanslug12:10:06

Hey, does shadow-cljs have issues with cond->

urbanslug12:10:14

or re-frame maybe?

wilkerlucio15:10:45

@thheller just gotta say, the repl fix that uses the current code NS is awesome! much better experience, and with your tips to remove warnings and the latest shadow my compilation times got down from 15s (for file with lots of dependants) to 3s!! awesome!!

👍 1
thheller15:10:00

for future reference: if anyone is seeing regular watch recompile times of 10sec+ please let me know. there is probably something to be improved just waiting to be found! do not assume that 10s+ is normal

thheller16:10:17

heck even 5sec is probably too much 😛

urbanslug16:10:19

hmm there's a new REPL?

thheller16:10:04

no, just fixed a bug

urbanslug16:10:33

So umm how to get this new repl? Update? I use emacs and connect to nrepl

urbanslug16:10:55

:s/update/update shadow version

thheller16:10:24

the change was for Cursive only I think. not sure emacs actually sends the :ns when evaling from a file

thheller16:10:57

the behaviour in Cursive is that it uses the namespace of a file when you send a form from a file to the REPL

thheller16:10:18

eg. if you send ::foo from that will be :

thheller16:10:28

even if the REPL is currently in cljs.user

thheller16:10:12

my emacs setup is completely busted so I have no idea if emacs actually sends :ns or not

richiardiandrea19:10:04

Had this problem agian in CI:

[2018-10-19 19:41:40.023 - INFO] :shadow.build.classpath/jar-cache-read-ex - {:file #object[java.io.File 0x1f0f5174 ".shadow-cljs/jar-manifest/1539977882000-core.rrb-vector-0.0.12.jar.manifest"]}
RuntimeException java.io.EOFException
	com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read (ReaderFactory.java:114)
	cognitect.transit/read (transit.clj:319)
	cognitect.transit/read (transit.clj:315)
	shadow.build.cache/read-cache (cache.clj:49)
	shadow.build.cache/read-cache (cache.clj:33)
	shadow.build.classpath/find-jar-resources/fn--11757 (classpath.clj:565)
…	cognitect.transit/read (transit.clj:319)

thheller20:10:10

sounds like maybe a job got killed while writing cache output?

thheller20:10:20

its no biggy as the cache will just be re-built

thheller20:10:36

its just a warning not a critical error

👍 1
lilactown20:10:05

I'm using 2.6.10 and getting very intermittent errors when doing a release build in CI:

aborted par-compile, [:shadow.build.classpath/resource "clojure/string.cljs"] still waiting for #{cljs.core}
{:aborted [:shadow.build.classpath/resource "clojure/string.cljs"], :pending #{cljs.core}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "clojure/string.cljs"] still waiting for #{cljs.core}
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)
	shadow.build.compiler/par-compile-one (compiler.clj:760)
	shadow.build.compiler/par-compile-one (compiler.clj:729)
	shadow.build.compiler/par-compile-cljs-sources/fn--12969/iter--12970--12974/fn--12975/fn--12976/fn--12977 (compiler.clj:822)
	clojure.core/apply (core.clj:657)
	clojure.core/with-bindings* (core.clj:1965)
	clojure.core/with-bindings* (core.clj:1965)
	clojure.core/apply (core.clj:661)
	clojure.core/bound-fn*/fn--5471 (core.clj:1995)
	java.util.concurrent.FutureTask.run (FutureTask.java:266)
	java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
	java.lang.Thread.run (Thread.java:748)

lilactown20:10:55

might this be fixed by some of the changes to parallel compilation that I see since 2.6.10? It's hard for me to test since it only happens intermittently

TeMPOraL20:10:28

a question about minification/advanced optimizations

TeMPOraL20:10:22

I've tried manually specifying :optimizations :advanced, :infer-externs :auto and :shadow-keywords true, and still, a release build looks like this:

TeMPOraL21:10:43

(not sure if the image uploaded correctly, as slackbot is saying something about no storage left)

TeMPOraL21:10:16

basically: #1, a curious unminified polyfill always shows up, and #2 plenty of full names visible in code

TeMPOraL21:10:06

if I include reagent and re-frame, I get ~300k of release bundle; I tried adding a button and textfield component from material-ui, and the bundle size shoots up to 600k +

TeMPOraL21:10:19

(it gets up to 1MB if I include all of material-ui-core)

TeMPOraL21:10:31

is this normal behaviour, or am I missing some extra minification setting?

lilactown21:10:11

material-ui is incredibly heavy, yeah

TeMPOraL21:10:59

I thought so, but then I stumbled upon some issues about material-ui bundle size, and people apparently expect a release bundle to be ~100-300k max

lilactown21:10:05

are you using a minified version of react & material-ui?

lilactown21:10:52

they might be talking gzipped

thheller21:10:55

@temporal.pl run this to get a details breakdown of all the stuff in your build https://shadow-cljs.github.io/docs/UsersGuide.html#_build_report

thheller21:10:04

and yes material-ui is huge if you import all of it

thheller21:10:19

you typically want to limit it to only import what you are actually using

TeMPOraL21:10:28

this is the report I get; do sizes there look normal?

thheller21:10:30

thats not the report 😉

thheller21:10:40

the html file is more detailed

TeMPOraL21:10:46

I'm new to shadow-cljs, and cljs in general, so I have no point of reference

TeMPOraL21:10:50

uh, ok 😄

lilactown21:10:57

those sizes look pretty normal

thheller21:10:09

@temporal.pl use things like (:require ["@material-ui/core/Button" ...]) to only include parts of the package

TeMPOraL21:10:44

yes, this case uses only

["@material-ui/core/Button" :default muiButton]
["@material-ui/core/TextField" :default muiTextField]
for test

TeMPOraL21:10:58

if I include it all, it's + 400k to total size

TeMPOraL21:10:17

still; it's a fresh project, and I'm not married to any UI library yet, so if I could ask you for recommendation?

TeMPOraL21:10:33

friends over the JS land discouraged me from using react-bootstrap

thheller21:10:23

dunno really .. they all get pretty large pretty quickly

lilactown21:10:16

I just bumped to 2.6.14 and I'm getting this error in CI

TeMPOraL21:10:23

ok, thanks for help and good night!

thheller21:10:34

@lilactown thats a bad closure compiler version

lilactown21:10:43

aackclkasdlfds

lilactown21:10:53

I do this everytime I update shadow-cljs. I'm sorry

thheller21:10:10

thats what you get for not using shadow-cljs to manage your deps 😉

1
thheller21:10:40

not a fan that closure keeps changes their public interfaces. they don't seem to care much about backwards compatibility 😞

thheller21:10:03

well to be fair that interface wasn't public and I had to cheat to get access to it 😛

lilactown21:10:45

heh, when I try and run the release command locally i get an OOM error

lilactown21:10:29

one release build works fine, but concurrent ones: npx shadow-cljs release server client dev-client

lilactown21:10:37

hm, fixed by killing my watch's

lilactown21:10:19

ah I bet I was using the old version shadow-cljs I had running as a server

lilactown22:10:59

OK, the previous error occurred again after upgrading

thheller22:10:15

hmm don't scare me like that ... I actually changed how parallel compile works. don't tell me I actually broke it?

thheller22:10:40

are you absolutely 100% sure that the correct code is loaded

thheller22:10:51

no bad AOT cache or anything?

lilactown22:10:53

I've been seeing this error intermittently since 2.6.10

lilactown22:10:03

it could be cache related

lilactown22:10:29

in the past, re-running the build has resolved it, but it was popping up relatively often

thheller22:10:08

thats definitely not normal and should never actually happen. its just a safeguard so the process doesn't wait forever

lilactown22:10:15

I'm re-running it right now so we'll see if it happens twice 😛

thheller22:10:25

run with :verbose true in the config please

thheller22:10:34

otherwise its gonna be impossible to reconstruct what happens

thheller22:10:45

(or with --verbose on the CLI)

lilactown22:10:09

I'll add that once this job finishes

thheller22:10:39

hmm is the machine maybe just overloaded?

thheller22:10:03

if the other 2 builds proceed normally the third may just not be getting a chance to run before its timeout

thheller22:10:32

still need to add that --parallel=false option

thheller22:10:03

you should try creating a clojure fn that just does the builds sequentially

lilactown22:10:20

I might do that

lilactown22:10:33

ok, this job succeeded

thheller22:10:55

the way this works is that all builds more or less start at the same time and enqueue all the CLJS compiles onto a threadpool

thheller22:10:15

which then compiles them in order they were submitted

thheller22:10:36

so its entirely possible that the queue just gets too long

thheller22:10:04

kinda unlikely but possible

thheller22:10:17

you were running on a toaster before 😉

lilactown22:10:20

I wonder if it might have to do with other jobs running on the same machine

lilactown22:10:27

lol, well we upgraded the toaster

lilactown22:10:03

we now have a group of toasters in AWS ECS for our pipeline

lilactown22:10:25

I'm wondering if a particular EC2 gets used for multiple concurrent builds if it's what causes the error

lilactown22:10:15

it might be poisoning the cache, or just overloading it

thheller22:10:30

definitely try running with verbose AND running them after each other instead of parallel

thheller22:10:40

cache is always isolated per build

thheller22:10:47

so they can't interfere with each other

lilactown22:10:22

alright, I wrote a simple script to do the build:

(ns release
  (:require [shadow.cljs.devtools.server :as server]
            [shadow.cljs.devtools.api :as shadow]))

(defn -main [& args]
  (server/start!)
  (shadow/release :server {:verbose true})
  (shadow/release :client {:verbose true})
  (shadow/release :dev-client {:verbose true})
  (server/stop!)
  (System/exit 0))
I'll report back if I see those errors again

thheller22:10:25

don't really need a server instance for release

thheller22:10:57

shadow-cljs run release/all also works a bit better probably

thheller22:10:17

(defn all [] ...) of course .. not -main

thheller22:10:39

or wrap the calls in (shadow/with-runtime ...) yourself

thheller22:10:19

(defn -main [& args]
  (shadow/with-runtime
    (shadow/release :server {:verbose true})
    (shadow/release :client {:verbose true})
    (shadow/release :dev-client {:verbose true})))

lilactown22:10:05

what does with-runtime do?

thheller22:10:52

share some components between the builds

thheller22:10:14

otherwise they all go off and re-index the classpath 3 times and stuff

thheller22:10:40

if you use shadow-cljs run thats done automatically

thheller22:10:01

(defn all [& args]
  (shadow/release :server {:verbose true})
  (shadow/release :client {:verbose true})
  (shadow/release :dev-client {:verbose true}))