Fork me on GitHub
#shadow-cljs
<
2018-10-25
>
vigilancetech05:10:18

what's the advantage or disadvantage of using shadow's built in nrepl vs cider-nrepl (when using cider)?

thheller08:10:42

@vigilancetech that is a question for the cider folks. its all about their middleware and what features it provides. I think it provides auto complete and other things but I'm not sure what.

👍 1
koz12:10:45

I might be missing something, however when trying to load modules on demand using shadow.loader in dev, it seems to 404 since it’s making the request for the additional compiled JS files to the port that the actual application is running in (5000), and not the port that the shadow dev server is serving the JS files from (3449). I have {:devtools {:loader-mode :eval :http-port 3449 :http-root "resources/public"}} set in the associated build, is there anything else I might be missing?

thheller14:10:14

@koz its just a static file request on the same server that also served the initial .js file

thheller14:10:34

so if the initial is from :5000 it must also serve the modules

thheller14:10:50

if it 404s check that your :asset-path is correct

koz14:10:06

ok will do that, thanks!

koz14:10:57

Also, is :loader-mode :eval still a valid flag? I think at one time it was used to opt-in to a new loading mode, but can’t find any mention of it in the docs.

thheller14:10:41

yes thats still a thing

thheller14:10:55

oh right if you use that it may have problems with shadow.loader depending on the version you are on

koz14:10:10

i see, is there a version that it should be OK?

thheller14:10:17

forgot which version I fixed that in but it was some time after I added loader-mode :eval

thheller14:10:30

the latest definitely has that fixed

koz14:10:37

I’m on 2.6.4 right now, but can try updating

thheller14:10:31

I think it was fixed in 2.6.10 so that might be it

koz14:10:27

cool thanks

koz15:10:48

Just updated to 2.6.18 and still running into the same issue. The app has a script tag to load the main JS bundle from the shadow dev server (http://localhost:3449/js/compiled/dev/main.js), which is working fine. Are you saying that any modules loaded via shadow.loader (e.g. /js/compiled/dev/cljs-runtime/lazily_loaded_module.js) won’t load from that same localhost:3449, but that our application server (instead of the shadow-cljs devtools server) needs to be configured to statically serve the compiled JS bundles?

thheller15:10:38

so it has <script src="">?

thheller15:10:52

then you need to set :asset-path to

koz15:10:04

oh ok, didn’t know that took a host/port

thheller15:10:55

basically in case of the loader it will combine the asset-path + filename

thheller15:10:26

typically thats just a path so it would request relative to the location of the current document

koz15:10:44

that worked, however I’m getting CORS errors from accessing a resource on a different port via XHR. I tried adding

{:devtools {:push-state/headers {"content-type" "text/html; charset=utf-8"
                                    "access-control-allow-origin" "*"}
to my shadow-cljs.edn, but don’t see the second header.

thheller15:10:43

hmm would it not be possible to make :5000 just serve the files?

thheller15:10:55

its just static files

thheller15:10:05

CORS can currently not be configured for static files

thheller15:10:23

the push-state handler is only for actual push-state requests (ie. files that don't exist)

koz15:10:30

i see. yeah it would take a bit of extra configuration on our end, but we can make something work for now. thanks!

thheller15:10:52

you could also use :3449 and let it proxy to :5000 for API stuff

thheller15:10:31

:devtools {:proxy-url ""}

koz15:10:31

cool let me give that a try.

thheller15:10:26

localhost:3449 should then serve whatever :5000 serves

👍 1
wilkerlucio16:10:44

hello, sometimes I'm getting this in my CI release build for shadow:

wilkerlucio16:10:46

aborted par-compile, [:shadow.build.classpath/resource "garden/compiler.cljc"] still waiting for #{garden.units}
{:aborted [:shadow.build.classpath/resource "garden/compiler.cljc"], :pending #{garden.units}}
ExceptionInfo: aborted par-compile, [:shadow.build.classpath/resource "garden/compiler.cljc"] still waiting for #{garden.units}
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)

wilkerlucio16:10:45

the ns after still wanting for usually changes, so I'm wondering if that's some shadow timeout? if it is, can I increase it somehow? as the codebase grows I'm feeling its starting to happen more often, a second trigger of the build usually works

thheller17:10:55

@wilkerlucio hmm do you have a verbose log? the timeout is 60000 already and its very unusual if it hits that? you can increase is via :build-options {:par-timeout <num-in-ms>}

lilactown17:10:26

@wilkerlucio I was running into a similar issue on CI as well.

lilactown17:10:54

I was running a number of release builds concurrently, and switching to running them synchronously seems to have resolved the issue

thheller17:10:49

ah right I never asked if that changed anything

thheller17:10:30

this is so weird logically. the tasks are submitted in dependency order so it is ensured that garden.units is compiled before garden.compiler.

thheller17:10:05

so the only case where garden.compiler is even waiting for compile is when a thread is becoming available while the others are still compiling

thheller17:10:20

but that means that garden.units takes 60sec to compile

thheller17:10:06

please turn on :verbose true (globally in config) or run with --verbose

thheller17:10:46

I don't get how this would happen even with many builds running in parallel

thheller17:10:21

garden.units should never take this long to complete and it must have started no matter how many other jobs there are

thheller17:10:01

doesn't matter which namespaces. it the same logic everywhere

thheller17:10:30

a verbose log should at least end with the time it took for garden.units to complete

wilkerlucio18:10:52

@thheller a lot of times I get this with cljs.core as well, and I'm running 2 parallel builds, the total compilation time gets to about 6 minutes when it works

thheller18:10:32

what the heck. is it running on a toaster?

😂 1
wilkerlucio19:10:08

just a lot of files, 70K CLJS lines without counting deps, 😛

wilkerlucio19:10:47

@thheller I'm having an issue that I had before with release builds regarding externs on .js files, but I can't remember how I fixed that, is that a way to tell the compilation to dont mangle code from .js files?

thheller20:10:09

it doesn't matter how many files there are. the queuing ensures that they are compiled in proper order with minimzed wait times

thheller20:10:35

@wilkerlucio shadow-cljs check the-build should tell you

thheller20:10:00

i'll definitely add the sequential option so that release builds don't run in parallel by default

thheller20:10:09

seems to cause more issues than it solves

wilkerlucio20:10:36

is there a way to run shadow using a diferent profile then the one defined on shadow-cljs.edn?

thheller20:10:44

you mean deps.edn alias?

thheller20:10:53

shadow-cljs -A :foo:bar?

pat20:10:17

Ok so I've figured a nice way to generate bindings for Wasm modules. I'm a shadow-cljs noob. Where do I start to work out a packaging solution

pat20:10:54

Theres an array buffer that must be loaded async, and functions defined afterward. Would be nice to treat it as a module

thheller20:10:21

define module? as in code split :modules?

pat20:10:07

I trust your judgment on this, I'd like shadow-cljs to inform how I proceed

thheller20:10:31

well it depends all on how the .wasm is structured

thheller20:10:48

I only checked the Rust output and thats not going to work in either case

pat20:10:55

why is that

thheller20:10:00

since the .js so so coupled to the .wasm

pat20:10:11

no im completely outside that

pat20:10:24

generating our own bindings, not rust-wasm wasm-bindgen

pat21:10:14

yes talking linking up to exported functions ourselves. I think I've got it but next step after is the important bit

thheller21:10:34

can you give me an example?

pat21:10:40

can define functions for you given a module description

pat21:10:57

right now you have to use a string to lookup the exported function and use that to call it

pat21:10:24

wasm-bindgen uses an ast to rig this up, we can do the same

pat21:10:29

you can define functions statically if you want its just not a pleasant dev experience

pat21:10:19

Am I explaining this ok? Im probably alittle close to this

thheller21:10:09

where is the wasm?

thheller21:10:16

how to I generate it?

pat21:10:29

Im not sure what you mean

pat21:10:02

I've already built a cargo layer. Just need to load arraybuffer and instantiate it

thheller21:10:20

I don't know what cargo is

pat21:10:36

ok yeah thats rust build tool. got all that figured out

pat21:10:05

you get your array buffer, separately a description of the module, and generate cljs bindings from that

thheller21:10:23

yeah thats not an option for the browser

pat21:10:36

i dont understand

pat21:10:03

its just a build tool you can send buffer to browser no problem

pat21:10:44

that would I assume be some hook step in shadow-cljs

thheller21:10:49

but its written in cljs. so its of no use for shadow-cljs

thheller21:10:02

shadow-cljs is written in Clojure and hooks are Clojure

pat21:10:31

ok you can shell out and do it

pat21:10:53

i can rewrite it in rust if need be

thheller21:10:38

so what I ideally want is a .wasm file and a .json or .edn file that has some data about the file

thheller21:10:52

(which would save parsing the .wasm)

pat21:10:00

yeah im there

pat21:10:11

build details are less interesting, getting async needs of namespaces etc is where I need help

thheller21:10:03

async is not going away probably

pat21:10:50

yes, how to get it all to play nicely with closure etc

thheller21:10:17

you don't? you just js/fetch the .wasm file an instantiate it

pat21:10:43

sure people can already do that

thheller21:10:46

or instantiateStreaming

pat21:10:31

that too but is poorly supported. Im looking for easy integration with cljs project minus dealing with this stuff.

pat21:10:41

if not possible thats ok

thheller21:10:46

which stuff? its like 3 lines of code? the complicated part as I see it is passing function INTO the wasm so it can call js functions

thheller21:10:55

and getting functions out so that normal code can call it

thheller21:10:11

and that kinda needs compiler support

pat21:10:45

Yes it would be nice to distrubute wasm modules as libs is all. I agree the existing burden is really all that bad

pat21:10:16

ok I thought maybe

pat21:10:41

isn't that bad *

thheller21:10:50

yes I want that but as I said I need a .wasm file + some metadata about it

thheller21:10:02

otherwise there is no way to "automate" it into a build

pat21:10:10

yes I've got that!

pat21:10:16

whats next

thheller21:10:21

give it to me?

thheller21:10:38

I don't care about anything else

thheller21:10:53

since you can't expect a user of a lib to install the entire Rust tool pipeline

thheller21:10:05

the final result is what I'm interested in

pat21:10:06

ofcourse

thheller21:10:10

not how it got there

pat21:10:57

I was asking you to write something, I'm offering a contribution. but if you just want the raw tools thats cool

pat21:10:20

geez i cant type

thheller21:10:58

yeah I totally want to explore how I can integrate .wasm into shadow-cljs to make things nicer

pat21:10:05

I will clean things up and try to put them in clj for you

thheller21:10:08

for that I need a .wasm file I don't have and can't produce

thheller21:10:16

I do not want to install Rust or anything

thheller21:10:00

compiling rust is just never going to become part of shadow-cljs ever

thheller21:10:17

sure it can shell out but thats not the integration point

thheller21:10:26

the integration point is a .wasm file + metadata

thheller21:10:50

so if you can give me that I'd the grateful

pat21:10:02

i didnt explain myself very well I apologize

pat21:10:50

I am saying, we can have a binary and a description, and turn it into something that feels like a namespace

pat21:10:17

the process of packaging that up should inform how the description is used with the binary

pat21:10:44

people dont have to care about rust if they dont want to

thheller21:10:58

we are running in circles

thheller21:10:19

if you can give me a .wasm file + metadata I can work something out

thheller21:10:23

if not I can't do anything

thheller21:10:41

the only other .wasm file I have is never going to work

thheller21:10:36

I'll eventually look into the rust wasm tools but that will probably be next year

gklijs04:10:45

The one from me does work, and since you only need to use it by calling the functions in the JavaScript that comes with it, there is no need to do anything to the wasm at compile time, besides copying it.

gklijs05:10:30

Apparently there was a pr for adding wasm to externs for closure, but it wasn't accepted. https://github.com/google/closure-compiler/pull/2534

thheller07:10:13

yes the one you have works .. unless you modify or rewrite the JS in any way then it doesn;t

thheller07:10:28

so I meant that it'll never work with shadow-cljs without processing the .wasm

gklijs08:10:18

Missed the contib map

wilkerlucio22:10:02

@thheller sorry, wasn't clear, I mean lein profiles, because I have a specific type of run that I want to have different deps there

thheller22:10:51

@wilkerlucio no thats not supported but you can just use lein with-profiles +whatever run -m shadow.cljs.devtools.cli release whatever

thheller22:10:04

its the same as calling shadow-cljs release whatever

thheller22:10:47

just lets you do whatever lein thing you want

thheller22:10:48

@lilactown @wilkerlucio on your failing builds. how many CPU cores do the machine have? 1-2?

thheller22:10:07

and how long do builds take that don't fail?

wilkerlucio22:10:24

5 ~ 6 min here, I don't how many cpus are on this one, I can check, but those are shared instances doing CI for a lot of projects

thheller22:10:05

I think it may be the closure optimization stalling things

thheller22:10:18

I know that it was taking 45sec for one of your builds the other day

thheller22:10:36

so its easy to imagine that it gets past the 60sec mark when under load

thheller22:10:04

which means it can fully utilize a core and stall other work trying to use it

thheller22:10:52

not actually sure how java schedules work like this

thheller22:10:30

I know in erlang processes only get a certain number of reductions but I think thats different on the JVM

lilactown22:10:23

when they were done concurrently, the builds were taking ~400s

lilactown22:10:44

the builds run on a m4.2xlarge EC2 inside a docker container

lilactown22:10:56

it says it has 8 vCPUs

thheller22:10:39

but the issue has not happened again since switching to sequential builds?

thheller22:10:08

probably the only area where parallel build is useful to begin with is running multiple closure optimizations in parallel

thheller22:10:14

since they are single threaded and take a while

thheller22:10:03

but if they completely prevent a parallel build from making any progress that kinda defeats the purpose

lilactown22:10:25

yeah, since switching to sequential builds I haven't seen an issue