Fork me on GitHub
#shadow-cljs
<
2021-10-13
>
steveb8n07:10:23

Q: I’m getting OOM errors in CI release builds. anyone seen this before?

Execution error (OutOfMemoryError) at java.util.Arrays/copyOf (Arrays.java:3332).

thheller07:10:05

@steveb8n fairly common in CI envs with limited memory yes. You can set :jvm-opts ["-Xmx512M"] or so to limit the memory or upgrade your JDK

thheller07:10:04

memory detection used to be wrong on older JVMs but I think thats fixed now and properly detects VMs and their memory config

steveb8n07:10:08

thanks. I’m up to 3gb so I suspect something else. haven’t tried JDK version so that’s next

thheller07:10:40

you should never need 3 gigs for shadow-cljs unless you have like a billion lines of code 😛

thheller07:10:03

whats the rest of the stacktrace though? curious where it fails

steveb8n07:10:46

I agree. unzipped release js file is 5Mb so it’s pretty large

steveb8n07:10:55

just ran build-report

thheller07:10:21

I've seen 12MB compile fine with 1gig ram so that doesn't mean much

steveb8n07:10:24

lemme try and get that stacktrace for you

steveb8n07:10:44

can’t repro locally so it’ll take a few mins

steveb8n08:10:43

bitbucket is difficult for catching files in /tmp. can’t get stack trace sorry

thheller16:10:45

react native expects that you register a root component with the proper name

pesterhazy11:10:10

@steveb8n finding a local repro case could help finding the cause

mitchelkuijpers14:10:11

Is it possible to get a nodejs repl when using target:

:npm-module

mitchelkuijpers14:10:46

I need to use :npm-module because the tool I am trying to integrate with does not like the umd style exports which you get from :node-library

thheller16:10:41

@mitchelkuijpers just node-repl is not enough? :npm-module works too but requires manual work. as in set :runtime :node in your build config and then from the JS that is consuming the compiled CLJS code somewhere you need to require("path-to-cljs/shadow.cljs.devtools.client.node")

thheller16:10:07

REPL should work after that but just node-repl is generally simpler and usually enough

jthibaudeau18:10:40

Question for anyone who has been using shadow-cljs with electron. How did you go about compiling the preload.js? I have mine as a separate build from the main and the renderer, and shadow compiles it fine, but electron complains it can't load it properly. This is what I have in my shadow-cljs.edn builds:

:preload {:target :node-script
          :output-to "resources/preload.js"
          :main electron-app.preload.core/main}
This is the error showing up in the devtools
(function (exports, require, module, __filename, __dirname, process, global, Buffer) { return function (exports, require, module, __filename, __dirname) { #!/usr/bin/env node
                                                                                                                                                           ^

SyntaxError: Invalid or unexpected token
    at new Script (node:vm:100:7)
    at createScript (node:vm:257:10)
    at Object.runInThisContext (node:vm:305:10)
    at wrapSafe (node:internal/modules/cjs/loader:1021:15)
    at Module._compile (node:internal/modules/cjs/loader:1066:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1140:10)
    at Module.load (node:internal/modules/cjs/loader:982:32)
    at Module._load (node:internal/modules/cjs/loader:823:12)
    at Function.c._load (node:electron/js2c/asar_bundle:5:13331)
    at Function.o._load (node:electron/js2c/renderer_init:29:379)

pesterhazy19:10:16

We just use js for preload

pesterhazy19:10:45

I don't think you're meant to include a ton of code there

jthibaudeau19:10:58

Yeah I was able to workaround the compile issue by just using JS as well. Ideally I'd like to not have to write any JS 🙂 From what I've been reading about context isolation, preload seems to be where you can securely expose functionality to the renderer via contextBridge so there can potentially be a non trivial amount of code there

thheller20:10:32

you can set :hashbang false in your build config to get rid of the #!/usr/bin/env node line

thheller20:10:06

it is my impression that you are supposed the bulk of your "node" interop code in the preload

thheller20:10:14

so should be fine to run a bunch of code there

jthibaudeau20:10:50

Oh that works, thanks so much!

lispers-anonymous19:10:31

I have a large project using shadow-cljs. As part of our CI process, we create a production build pretty frequently. I have been looking at ways to speed up the build. Is it a bad idea to cache the .shadow-cljs directory on the build server? I think it could shave a couple minutes off our build time, but am worried about it making our builds unstable.

nenadalm19:10:34

It does save a lot of time and time to time it happens that deployed app doesn't render due to some issues ($jscomp something) - clearing the cache and rebuild helps. I am not working on the project - just helped to set it up, so I am not sure how often that happens. I just know it happened a few times in last year. In case you have some functional test, you should be able to catch it though (e.g. you could open homepage and check that at least something is rendered) before deploying to some env.

lispers-anonymous19:10:19

Thanks for sharing your experience. I'll probably talk this over with my team. We can run some experiments to see if the time saved is worth the hassle of having to clear the cache and re-build on occasion.

thheller20:10:44

I recommend caching the .shadow-cljs dir in CI envs yes

lispers-anonymous20:10:35

That's quite the endorsement, thanks for the response!

javahippie20:10:00

Would it be a good idea to invalidate the cache on certain events, e.g. if dependencies are changed?

lispers-anonymous21:10:27

While developing locally I update dependencies, both npm and clojure dependencies, without having to blow away my shadow-cljs cache and don't have issues. But the only thing it would hurt is your build times.

👍 1
thheller06:10:52

caches invalidate themselves but most CI cache systems I have seen are too dumb to recognize changes and thus always restore old state. at which point it becomes useless so you do have to somewhat integrate it with your dependencies

javahippie06:10:52

Yeah, CircleCI won’t change a cache once it was created, so we create a file hash over package.json & project.clj. If this changes, the cache gets thrown out completely

thheller06:10:08

yeah exactly. unfortunately that blows away too much usually but still useful

javahippie06:10:12

Thanks for bringing the topic up, just tried it and it brought the uberjar build step in our pipeline down from 2:50min to 2:15min 👍

👍 1
lispers-anonymous19:10:09

I was able to shave about 1 minute off our (7 minute) build time by including .shadow-cljs in the build's cache. I think I'll get even better time savings by increasing the compute resources. Currently using a 3GB aws codebuild machine, which leaves a lot to be desired. Every second matters. Thanks again for the advice!

👍 1
thheller19:10:58

do you build more than one thing? you can save a lot of time by reducing the number of JVM startups

lispers-anonymous19:10:37

I think we startup 3 jvms in this particular build • download npm dependencies • compile sass to css • compile and run tests, using a :node-test build target (jvm 1 + executing the test node script) • compile a shadow-cljs release build, a :browser build target (jvm 2) • build an uberjar with our clojure server code and the compiled js output (jvm 3) I don't think there is a way around these steps for us, which is okay. The benefits shadow-cljs has brought are well worth the tradeoffs in ci build time. Right now it's probably 2 minutes slower than when we used figwheel, and we weren't able to use advanced compilation with that.

thheller20:10:18

you might gain a lot by compiling via a function run via https://shadow-cljs.github.io/docs/UsersGuide.html#clj-run

thheller20:10:43

(shadow/release :build) there is identical to shadow release build from the CLI

thheller20:10:27

besides that you probably want to shadow-cljs release build test and then node wherever/your-test.js to run the tests

thheller20:10:43

(ie. compile using both builds using one JVM)

thheller20:10:37

startup time is rather significant on most CI systems so avoiding it can result in big improvements

lispers-anonymous20:10:54

I've never poked around the clj-run part of the shadow docs. I'll experiment with that and combining the builds. Do some benchmarks. I suspect eliminating one jvm startup will have a big impact.

thheller20:10:02

if you just need to build two builds shadow-cljs release build-a build-b is fine

lispers-anonymous21:10:27

Unfortunately our tests fail when compiling them with release, and that's not something I'm too keen on tackling right now. However, using clj-run with this

(defn release []
  (shadow/release :app)
  (shadow/compile :test))
Seems like an easy win so far. Shaved off another minute.

👍 2
thheller21:10:54

you probably want to use the ! variants to release! and compile!. otherwise errors are just printed without "failing" (ie. process exit code)

lispers-anonymous21:10:51

Done. I was wondering what those ! variants were for. Thanks again for the pro-tips

tbrooke21:10:36

I have a project that has a dependency “@accordproject/markdown-transform”: “0.14.1" that crashes the closure compiler - not pdfmake itself but something related to pdfmake — I can get it to work with nbb — @borkdude mentioned that nbb wasn’t running it through the closure compiler and that there might be a way to webpack or some setting with shadow-cljs to run the dependency without going through closure. Anyone know how to do this?