Fork me on GitHub
#shadow-cljs
<
2018-02-27
>
kenny00:02:28

I see shadow-cljs uses thheller/cljs-dev Docker image for running CI tests. Is it recommended to use this image for our CI tests as well?

kenny00:02:34

Also, is it possible to specify :repositories in my shadow-cljs.edn?

kenny00:02:18

I don't think shadow-cljs provides a way to configure a private Maven repository.

kenny00:02:47

I guess I can move deps, source paths, and repositories into a project.clj.

kenny01:02:27

Is there a reason the wait time is hardcoded for 60s? https://github.com/thheller/shadow-cljs/blob/d59524d3ef9dfd41a2b6e5ec58fa69a1ed5002e2/src/main/shadow/build/compiler.clj#L683 Seems like this should be configurable. My CLJS is failing to compile from hitting that limit (it's due to using some generators at macro time to generate some code).

thheller08:02:27

@kenny I can make that timeout configurable. the "slowest" compile for a single ns I saw so far was ~15sec. what are you doing that it takes 60s+?

thheller13:02:47

@kenny I changed the timeout condition in 2.1.28 which might just fix your problem. if not you can set :build-options {:par-timeout some-high-number-in-ms}

thheller13:02:59

private maven repos might work. I'm not sure. Using the same lib as lein but not sure if I'm maybe not setting something I would need to set. lein or deps.edn also work though.

lilactown16:02:24

I really appreciate the lein integration - if only because it helps Cursive users 🙂 but also the ecosystem of plugins is quite nice

kenny19:02:37

@thheller Thanks! I set it to 2 mins and I'm still having the issue. It appears to only happen on the CI though, and it's not clear why it is happening. The namespaces that it is printing out should not take a long time to compile. For example, DataScript is failing on the CI:

:shadow.build.classpath/resource "datascript/pull_api.cljc"] #error {
 :cause "aborted par-compile, [:shadow.build.classpath/resource \"datascript/pull_api.cljc\"] still waiting for #{datascript.pull-parser}"
 :data {:aborted [:shadow.build.classpath/resource "datascript/pull_api.cljc"], :pending #{datascript.pull-parser}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "aborted par-compile, [:shadow.build.classpath/resource \"datascript/pull_api.cljc\"] still waiting for #{datascript.pull-parser}"
   :data {:aborted [:shadow.build.classpath/resource "datascript/pull_api.cljc"], :pending #{datascript.pull-parser}}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
Is there a way to get more information printed out? I'd bet something else is going on here.

kenny19:02:34

I added the -v flag and I get more information. I don't understand why some namespaces are taking a long time to compile. For example

Cache write: datascript/impl/entity.cljc
<- Compile CLJS: datascript/parser.cljc (78929 ms)

kenny19:02:22

Could this be happening as a result of another namespace taking a long time to compile? Or are these all compiled independently and DataScript is just taking a long time for some reason?

kenny19:02:09

Turns out the DataScript thing was a red herring. I removed the code created by generators and it worked (getting a No binary for ChromeHeadless browser on your platform. error not but that's something else I'm guessing). Confusing debugging process though. Not sure why it was telling me DataScript was taking a long time to compile.

thheller19:02:27

@kenny I have never seen such high numbers. what kind of setup is this on?

thheller19:02:58

No binary for ChromeHeadless browser on your platform. is no error from shadow-cljs. probably karma?

kenny19:02:40

Right. Trying to figure that one out now. The vanilla example in the docs does not work.

thheller19:02:48

karma-chrome-launcher doesn't install chrome so you need to install chrome manually

kenny19:02:19

That's where I'm at now. Do you use puppeteer?

thheller19:02:15

I haven't used puppeteer no

kenny19:02:54

What do you use on the CI? I only suggested it because it was suggested here: https://github.com/karma-runner/karma-chrome-launcher

kenny19:02:10

Simply adding the steps they have there does not work. It appears other binaries are missing to start the headless chrome.

thheller19:02:29

should just be chrome although headless is fairly new

thheller19:02:35

maybe you just have an older verison?

kenny20:02:03

You install chrome on the CI?

thheller20:02:17

I haven't used this stuff on CI

thheller20:02:31

but others have (I think)

kenny20:02:54

Know any projects off the top of your head that do?

thheller20:02:06

open source no

thheller20:02:37

but yes you need to install chrome on the CI

thheller20:02:17

or use an image that has all of this setup. I know there was one but I forgot which

thheller20:02:23

let me see if I can dig it up

kenny20:02:34

It's somewhat surprising how much work goes into getting this working.

thheller20:02:43

there are probably a couple of images that have all this setup and ready to go.

thheller20:02:47

I can't find it though

thheller20:02:02

doesn't look like it has java

kenny20:02:18

Righttt. Forgot that's needed.

thheller20:02:58

lein-2.8.1-node-browsers should work

kenny20:02:10

Ah cool. I'll try that now.

thheller20:02:25

as for the datascript issue I can't find anything that would cause this slowdown. it might just be the CI acting up. I hear they have issues with rogue crypto miners running on their VMs

thheller20:02:40

might just be using all the CPU and causing yours to get slow

kenny20:02:07

It was a red herring. Turns out it was a completely different ns causing the issue but DataScript was the one printed out for some reason.

kenny20:02:16

That image worked!

thheller20:02:49

if namespaces take too long to compile other namespaces "behind" it may get delayed a lot if there aren't enough threads

thheller20:02:19

but that means the problem ns would spend 60sec+ compiling a single form.

kenny20:02:46

It would've been helpful if the problem ns was the one printed out instead of a tangential one.

kenny20:02:14

But yes that was the case, I guess. I have code that runs at macro time which generates some static data for the runtime environment (using test.check generators). It does not take a long time to run on my machine but it does on the CI.

thheller20:02:53

I'll see if I can find some better diagnostics for this

kenny20:02:16

Perhaps simply printing out the one that is taking the longest?

thheller20:02:55

maybe I'll think about it

thheller20:02:07

a "hint" would be the verbose messages printed

thheller20:02:18

-> indicates that an ns started compiling

thheller20:02:25

<- means it completed

thheller20:02:32

so look for -> I guess

thheller20:02:44

but that could be more obvious

thheller20:02:54

especially when throwing

kenny20:02:58

Sure but there are so many messages printed. Would be very time consuming to go through.

kenny20:02:28

Above that I have about 100 lines of

Feb 27, 2018 3:03:01 AM shadow.build.compiler invoke
WARNING: [:shadow.build.classpath/resource "cljs/test.cljs"] waiting for #{cljs.pprint}
with the namespaces in the log messages varying.

kenny20:02:29

There's a lot of noise to sort through. Maybe printing a summary of all the namespaces still compiling and how long they are taking?

lilactown21:02:45

I just use nrepl-select / cemerick.piggieback ¯\(ツ)/¯

hlolli21:02:09

@lilactown you use piggieback with shadow-cljs? I deleted my original question, because I still wasn't sure about how to formulate it. As I'm trying to connect to node repl, I'm trying to find out if that's an edge case, but I'm realizing it's actually the same procedure.

lilactown21:02:49

yeah, it functions exactly the same

thheller21:02:30

its fake piggieback though

hlolli21:02:29

what I started to do, without success, was to make a huge emacs function for starting shadow-cljs node repl, start cider connect, paste the code in, and fork the shadow-cljs process to the cider-repl buffer. To get the print callbacks in 1 buffer. But I didn't get success, as it's no way to know when the subprocess has finished starting shadow-cljs node-repl for cider to start the connection.

thheller22:02:10

I'm missing some context. what is going on?

hlolli22:02:13

I feel somehow this all could be simpler 🙂 so only thinking out loud, nothing is broken

hlolli22:02:54

well, best would be if cider could start shadow-cljs, just open a ticket and ask if the maintainers have the time in the future to implement shadow-cljs connection like they do with lein and boot....

thheller22:02:43

what is the problem?

lilactown22:02:28

I think @hlolli wants an Emacs fn that starts up a build and connects to it with CIDER

lilactown22:02:26

similar to how you can run cider-jack-in and it uses lein to start a headless REPL and connect to it

lilactown22:02:30

is that correct?

hlolli22:02:03

bingo, so no problem really, it's a minor distraction that I need a terminal window next to my emacs, and the connection it a 4 step. cider-connect -> select root folder -> go to repl buffer -> go to end of buffer -> type in (shadow.cljs.devtools.api/nrepl-select :node-repl) -> press enter twice

hlolli22:02:59

and on top of that, all print/console.log is in the terminal buffer, but I get the return value tough to my emacs.

thheller22:02:30

I have no idea what you are talking about sorry.

lilactown22:02:39

for piping console.log to the REPL, I’m not sure that’s really related

lilactown22:02:02

I don’t think even CIDER with lein (and e.g. figwheel) does that

thheller22:02:54

if you can describe what you want to happen I can maybe help

hlolli22:02:03

yes that's right, browser repl connection doesn't do that, don't think it makes sense there, but this is only an extra feature, I know it's possible tough to implement in emacs with subprocesses

thheller22:02:08

but those emacs steps you mention mean nothing to me

thheller22:02:18

so I can't tell what you actually want

thheller22:02:12

I can make prn output appear in the REPL window

hlolli22:02:27

hehe, I feel like @lilactown understands me. And could clairify if I'm explaining this inaccurately? But this is nothing that requires change from shadow-cljs itself, as far as I see.

hlolli22:02:55

ok, you mean you could send the prn return values trough the nrepl connection?

hlolli22:02:11

"return" sorry side-effects I mean

thheller22:02:44

prn yes, console.log no

thheller22:02:00

well actually in the case of node-repl I could make ALL output appear

thheller22:02:16

since I control the node process

thheller22:02:24

just doesn't work for non :node-repl

hlolli22:02:03

sounds good to me, I guess if you pipe it, it wouldn't add load to the node process?

thheller22:02:09

I think the issue you have is that you can't start a node-repl via nrepl

thheller22:02:15

you have to run it externally

thheller22:02:29

which is absolutely not want I want .. its just the way it is for now

thheller22:02:53

if the process is started via nrepl you'd already get all the output it produces

hlolli22:02:47

if you look at cider and clojurescript in leiningen, you see that when you close emacs with running cider, it will prompt the user for closeing 3 processes, cider process/cljs process and nrepl process. So I can imagine cider startup function, where shadow-cljs would be started as a process within emacs.

hlolli22:02:38

to proof, I did hack something here which almost works https://gist.github.com/hlolli/d8a28c88ee90c5a57d15299f21528030

thheller22:02:40

ok .. let me first take out all emacs stuff you are doing an deconstruct the commands you are running

thheller22:02:08

first of all never hardcode the port plz

thheller22:02:25

I emit a target/shadow-cljs/nrepl.port file which contains the port number

thheller22:02:34

similar to .nrepl-port from lein

thheller22:02:34

might need to parse shadow-cljs.edn though since :cache-root can be configured

hlolli22:02:00

ah nice, ok! Glad I posted this work in progress then, the explicit port is there because it's a work in progress, guess I was debugging/ruleing out. Did this month ago..

thheller22:02:41

you then start shadow-cljs node-repl

thheller22:02:57

you could wait for the mentioned file to appear instead of look at the output

hlolli22:02:14

ah!! ah!! ok

hlolli22:02:19

🙂

hlolli22:02:51

and also, I'm pipeing the stdout to the message buffer. I'd want to change that to the cider-repl buffer. Essentially printing all output to 1 buffer.

thheller22:02:52

defun shadow-node-repl () is defined twice?

hlolli22:02:07

ok, I will try to make something better out of this, I had commented line 60 to bottom, because I was giving up starting shadow-cljs as a subprocess, due to exacly the problem that, I didn't know when shadow-cljs repl was ready, but if the behaviour is like you say, the file nrepl.port exists when the port is open and available, I have then a solution. I recommend not reading this until I've made changes 🙂

thheller22:02:42

I think those are the main issues right now related to nrepl

thheller22:02:07

starting node-repl from within nrepl and seeing its stdout/stderr

thheller22:02:39

this currently doesn't work since I'm unsure how I want to handle external processes

thheller22:02:40

(shadow.cljs.devtools.api/node-repl) does since simply by reading until eof on stdin

thheller22:02:56

which is what shadow-cljs node-repl uses and just kills the node process aftrer

thheller22:02:12

nrepl however doesn't have a "disconnect" notification

thheller22:02:20

since the server keeps running

thheller22:02:41

so the node process would keep running

thheller22:02:08

I'll probably add some kind of method to manage processes

thheller22:02:15

going to need it anyways for some plugins

thheller22:02:34

eg. starting a css compile thingo in the background

hlolli22:02:54

Wish I knew the nrepl protocol mechanics better, but if you're keeping the nrepl connection alive by simply look for eof. Don't know what to say, seems magic to me that nrepl can know which stdout to take as return value and send across the wire and which text to ignore (ex. the print stuff).

thheller22:02:23

uhm thats what I meant .. nrepl works differently. thats why the wait-for-eof method doesn't work

thheller22:02:18

nrepl in a nutshell just sends messages in the form of {:op "do-a-thing" :some "thing"} and gets messages {:out "this is printed"} or {:value "foo"}

thheller22:02:27

bit more complex than that but thats the basics

thheller22:02:48

messages back and forth encoded via bencode+edn

hlolli22:02:30

ok so I take it that it needs to by synchronous, the :out would the the first output from the operation.

thheller22:02:49

no :out is always stdout

thheller22:02:16

can get many

thheller22:02:21

related to other messages or not

thheller22:02:08

{:op "eval" :code "(+ 1 2)" :id 1} would get {:value "3" :id 1}

hlolli22:02:22

ok can be, haven't bumped into that so far. But in lumo this could become very messy, getting return values into the eldoc buffer, and eldoc completions into the repl buffer