This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-13
Channels
- # aleph (8)
- # announcements (3)
- # beginners (36)
- # calva (1)
- # cider (1)
- # circleci (3)
- # clj-kondo (9)
- # cljdoc (58)
- # clojars (7)
- # clojure (70)
- # clojure-europe (13)
- # clojure-japan (1)
- # clojure-nl (8)
- # clojure-russia (2)
- # clojure-uk (7)
- # clojuredesign-podcast (4)
- # clojurescript (67)
- # clojureverse-ops (14)
- # cursive (6)
- # data-science (1)
- # datahike (3)
- # datomic (6)
- # docker (1)
- # events (2)
- # figwheel-main (9)
- # fulcro (4)
- # graalvm (33)
- # graphql (8)
- # helix (1)
- # jobs (1)
- # leiningen (3)
- # lsp (39)
- # malli (9)
- # minecraft (31)
- # nextjournal (7)
- # off-topic (6)
- # portal (22)
- # re-frame (5)
- # reitit (5)
- # remote-jobs (1)
- # shadow-cljs (55)
- # sql (22)
- # tools-build (3)
- # tools-deps (4)
- # xtdb (6)
Q: I’m getting OOM errors in CI release builds. anyone seen this before?
Execution error (OutOfMemoryError) at java.util.Arrays/copyOf (Arrays.java:3332).
@steveb8n fairly common in CI envs with limited memory yes. You can set :jvm-opts ["-Xmx512M"]
or so to limit the memory or upgrade your JDK
memory detection used to be wrong on older JVMs but I think thats fixed now and properly detects VMs and their memory config
thanks. I’m up to 3gb so I suspect something else. haven’t tried JDK version so that’s next
you should never need 3 gigs for shadow-cljs unless you have like a billion lines of code 😛
so https://github.com/thheller/reagent-react-native/blob/master/react-native/app.json#L2 needs to match the argument here https://github.com/thheller/reagent-react-native/blob/master/src/main/test/app.cljs#L34
@steveb8n finding a local repro case could help finding the cause
Is it possible to get a nodejs repl when using target:
:npm-module
I need to use :npm-module
because the tool I am trying to integrate with does not like the umd style exports which you get from :node-library
@mitchelkuijpers just node-repl
is not enough? :npm-module
works too but requires manual work. as in set :runtime :node
in your build config and then from the JS that is consuming the compiled CLJS code somewhere you need to require("path-to-cljs/shadow.cljs.devtools.client.node")
REPL should work after that but just node-repl
is generally simpler and usually enough
Question for anyone who has been using shadow-cljs with electron. How did you go about compiling the preload.js? I have mine as a separate build from the main and the renderer, and shadow compiles it fine, but electron complains it can't load it properly. This is what I have in my shadow-cljs.edn
builds:
:preload {:target :node-script
:output-to "resources/preload.js"
:main electron-app.preload.core/main}
This is the error showing up in the devtools
(function (exports, require, module, __filename, __dirname, process, global, Buffer) { return function (exports, require, module, __filename, __dirname) { #!/usr/bin/env node
^
SyntaxError: Invalid or unexpected token
at new Script (node:vm:100:7)
at createScript (node:vm:257:10)
at Object.runInThisContext (node:vm:305:10)
at wrapSafe (node:internal/modules/cjs/loader:1021:15)
at Module._compile (node:internal/modules/cjs/loader:1066:27)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1140:10)
at Module.load (node:internal/modules/cjs/loader:982:32)
at Module._load (node:internal/modules/cjs/loader:823:12)
at Function.c._load (node:electron/js2c/asar_bundle:5:13331)
at Function.o._load (node:electron/js2c/renderer_init:29:379)
We just use js for preload
I don't think you're meant to include a ton of code there
Yeah I was able to workaround the compile issue by just using JS as well. Ideally I'd like to not have to write any JS 🙂
From what I've been reading about context isolation, preload seems to be where you can securely expose functionality to the renderer via contextBridge
so there can potentially be a non trivial amount of code there
you can set :hashbang false
in your build config to get rid of the #!/usr/bin/env node
line
it is my impression that you are supposed the bulk of your "node" interop code in the preload
Oh that works, thanks so much!
I have a large project using shadow-cljs. As part of our CI process, we create a production build pretty frequently. I have been looking at ways to speed up the build. Is it a bad idea to cache the .shadow-cljs
directory on the build server? I think it could shave a couple minutes off our build time, but am worried about it making our builds unstable.
It does save a lot of time and time to time it happens that deployed app doesn't render due to some issues ($jscomp something) - clearing the cache and rebuild helps. I am not working on the project - just helped to set it up, so I am not sure how often that happens. I just know it happened a few times in last year. In case you have some functional test, you should be able to catch it though (e.g. you could open homepage and check that at least something is rendered) before deploying to some env.
Thanks for sharing your experience. I'll probably talk this over with my team. We can run some experiments to see if the time saved is worth the hassle of having to clear the cache and re-build on occasion.
That's quite the endorsement, thanks for the response!
Would it be a good idea to invalidate the cache on certain events, e.g. if dependencies are changed?
While developing locally I update dependencies, both npm and clojure dependencies, without having to blow away my shadow-cljs cache and don't have issues. But the only thing it would hurt is your build times.
caches invalidate themselves but most CI cache systems I have seen are too dumb to recognize changes and thus always restore old state. at which point it becomes useless so you do have to somewhat integrate it with your dependencies
Yeah, CircleCI won’t change a cache once it was created, so we create a file hash over package.json & project.clj. If this changes, the cache gets thrown out completely
Thanks for bringing the topic up, just tried it and it brought the uberjar build step in our pipeline down from 2:50min to 2:15min 👍
I was able to shave about 1 minute off our (7 minute) build time by including .shadow-cljs in the build's cache. I think I'll get even better time savings by increasing the compute resources. Currently using a 3GB aws codebuild machine, which leaves a lot to be desired. Every second matters. Thanks again for the advice!
do you build more than one thing? you can save a lot of time by reducing the number of JVM startups
I think we startup 3 jvms in this particular build • download npm dependencies • compile sass to css • compile and run tests, using a :node-test build target (jvm 1 + executing the test node script) • compile a shadow-cljs release build, a :browser build target (jvm 2) • build an uberjar with our clojure server code and the compiled js output (jvm 3) I don't think there is a way around these steps for us, which is okay. The benefits shadow-cljs has brought are well worth the tradeoffs in ci build time. Right now it's probably 2 minutes slower than when we used figwheel, and we weren't able to use advanced compilation with that.
you might gain a lot by compiling via a
function run via https://shadow-cljs.github.io/docs/UsersGuide.html#clj-run
besides that you probably want to shadow-cljs release build test
and then node wherever/your-test.js
to run the tests
startup time is rather significant on most CI systems so avoiding it can result in big improvements
I've never poked around the clj-run
part of the shadow docs. I'll experiment with that and combining the builds. Do some benchmarks. I suspect eliminating one jvm startup will have a big impact.
Unfortunately our tests fail when compiling them with release, and that's not something I'm too keen on tackling right now. However, using clj-run with this
(defn release []
(shadow/release :app)
(shadow/compile :test))
Seems like an easy win so far. Shaved off another minute.you probably want to use the !
variants to release!
and compile!
. otherwise errors are just printed without "failing" (ie. process exit code)
Done. I was wondering what those !
variants were for. Thanks again for the pro-tips
I have a project that has a dependency “@accordproject/markdown-transform”: “0.14.1" that crashes the closure compiler - not pdfmake itself but something related to pdfmake — I can get it to work with nbb — @borkdude mentioned that nbb wasn’t running it through the closure compiler and that there might be a way to webpack or some setting with shadow-cljs to run the dependency without going through closure. Anyone know how to do this?