Fork me on GitHub
#boot
<
2017-07-11
>
djebbz10:07:24

Hi, We're having a problem with boot-cljs. Running the same task several times doesn't produce the same build. Specifically, sometimes it's build with :advanced optimizations as specified, sometimes not. Similarly, it doesn't always take the :libs option into account, resulting in missing JS code in the final build. To be clear, we tried to the :optimizations options everywhere : in a .cljs.edn file, in the :compiler-options of the task option, and at the top-level of the task option itself. The only almost reliable way we found was to duplicate the options everywhere applicable (both :libs and :optimizations). Yes, this is silly, but this is the way it works the most often. The problem exists with the latest 2.0.0 version of boot-cljs and the previous one. Code (stripped out data, with some comments for you) :

clojure
;; build.boot
(def trackers-libs [ "trackers/google_analytics.js"]) ;; real list is longer
(def ref-src-trackers "trackers/src")

;; this one is used at several places, hence a raw call to `(cljs)` configured elsewhere with `(task-options!)`
(deftask task-assets
  "Main task to build assets"
  [c css-output-style VAL kw "Css output style"
   s css-sourcemap bool "Css sourcemap"]
  (comp
    (sift :invert true :include #{  #"^(.+?)main.scss$"
                                    #"^(.+?)mainfloat.scss$"
                                    #"^(.+?)duplos.scss$"
                                     })
   (sass :source-map css-sourcemap :output-style css-output-style)
   (cljs)
   (sift :include #{  #"^(.+?).js$"
                      #"^(.+?).map$"
                      #"^(.+?)yoda.css$"
                      #"^img[/\\](.+?)$"
                      #"^fonts[/\\](.+?)$"})
   (sift :move { #"^scss[/\\]yoda.css$" "css/main.css"
                 #"^scss[/\\]yoda.css.map$" "css/yoda.css.map"})))

(deftask build-assets
  "build yoda assets"
  []
  (set-env!
    :source-paths #{"src/cljs" "src/cljc" ref-src-trackers}
    :resource-paths  #{"resources" "cantina/duplos/src"})
  (task-options! cljs { :ids #{"js/main"}
                        :optimizations :advanced ;; DUPLICATE!
                        :source-map false ;; DUPLICATE!
                        :compiler-options { :optimizations :advanced ;; DUPLICATE!
                                            :source-map false ;; DUPLICATE!
                                            :libs trackers-libs}}) ;; var trackers-libs reused everywhere we need to build cljs
  (comp
    (task-assets :css-output-style :compressed :css-sourcemap true)
    (sift :include #{ #"^js/main.out(.+?)$" } :invert true)
    (target :dir #{"yoda-assets"})
    (tar-assets)))


;; main.cljs.edn
{:require  [yoda.core]
 :init-fns [yoda.core/init]
 :compiler-options {:optimizations :advanced ;; DUPLICATE!
                    :source-map false ;; DUPLICATE!
                    :libs ["trackers/google_analytics.js"] ;; DUPLICATE!
                    :parallel-build true}}
Note that we ended up in this crazy duplication state because things started not working in almost 100% cases. The bug happens on Linux, locally or in Jenkins (also runs under Linux). Help, please 😢

djebbz10:07:03

Maybe I should open an issue, I don't know. Even duplicating options everywhere doesn't help getting a consistent output.

richiardiandrea11:07:05

@djebbz options to tasks are global when you use task-options! (that is why there is an exclamation mark), so unfortunately once you set them, you can change them only by overriding (duplicating in your case)

djebbz11:07:01

We'll try using only task-options then. I'll report back (after lunch 🍜). Thanks @richiardiandrea

richiardiandrea11:07:21

You would probably need to drop it actually, depending on how many things you need to customize - or use task-options! for options that are not dynamic and direct options otherwise

juhoteperi11:07:12

Don't use task-options with cljs at all

juhoteperi11:07:34

Especially from deftask

juhoteperi11:07:57

task-options is like defn, you don't want to create side-effects from tasks like that

djebbz12:07:27

@juhoteperi So you mean one should use task-options! only for options that stay the same across all tasks ?

djebbz12:07:06

I get it. Indeed studying all examples repo show this patterrn

juhoteperi12:07:33

preferably set-env! should also be used only at top-level, not inside tasks, but unfortunately there are some cases where it can be required

djebbz12:07:26

Yes, I definitely see in our boot file cases for task-level set-env! (different classpaths for clj and cljs)

djebbz13:07:48

Does -v[vvv] works at the pipeline level or at the task level ? Said another way, do I need to use only once at the beginning of the command-line or before each task invoked ?

djebbz13:07:00

indeed, boot -h says so. Sorry for the disturbance

martinklepsch14:07:17

MERGE TIME 😄

martinklepsch14:07:38

I’ll push a changelog entry once it’s merged

alandipert14:07:05

things are looking good for the summer release!

alandipert14:07:19

really appreciate the work people have done vetting and tagging

alandipert14:07:23

(and untagging)

martinklepsch14:07:23

@alandipert I think @richiardiandrea deserves most of the praise 😛

alandipert14:07:58

retain pillsbury theme

martinklepsch14:07:08

Haha yeah, key point. 😄

alandipert14:07:48

it sucks that changes.md is always a conflict

alandipert14:07:03

i wonder about compositing it at build time from a few different files to make things easier on git

alandipert14:07:28

or rather, a script to generate it from a few flat files or something

martinklepsch14:07:45

@alandipert ah yeah that’s annoying indeed. (reminds me of that migration system I’ve been working with where migrations where numbered w/ incrementing integers)

martinklepsch14:07:35

PRs could add files with their changelog entries to .changes, script could consolidate them into changes.md

alandipert14:07:36

@martinklepsch btw i’m done with the merging, feel free to add to changes.md now

martinklepsch14:07:17

https://github.com/boot-clj/boot/pull/612 — this one might be worth consideration as well?

richiardiandrea14:07:55

I'd release and merge this one for the next -alpha

alandipert14:07:12

new 2.7.2-SNAPSHOT deployed, here’s what’s in it: https://github.com/boot-clj/boot/milestone/23?closed=1

richiardiandrea15:07:47

@alandipert there are a couple of docs PR, shall we merge them

alandipert15:07:56

@richiardiandrea sure, pls also add issues to next-release milestone

richiardiandrea15:07:18

I am rebasing 623

juhoteperi15:07:05

@martinklepsch Looks good, I just need to check what with-let does...

juhoteperi15:07:15

Okay, it binds fs/skip-subtree to _, evaluated body, and returns _

juhoteperi15:07:24

I don't see why the macro is used here, but works

djebbz15:07:44

@richiardiandrea @juhoteperi reporting : I rewrote our tasks that used boot-cljs so that they use only the :compiler-option : no task-options!, no x.cljs.edn file, no top-level cljs options (`:optimizations` and :source-map). It just works. Sad to say that what boot-cljs adds isn't very reliable (yet?). It's almost like we're using the raw clojurescript compiler. Thank you guys for your assistance !

juhoteperi15:07:06

Well, I haven't had problems with Boot-cljs in long time, which is why it hasn't seen much development. And it is supposed to be only very light tooling over ClojureScript compiler.

richiardiandrea15:07:26

@djebbz I think that a light wrapper on top of the compiler is the only thing you actually need, in case things go wrong, you don't want to debug layers on top of layers of indirections...`boot-cljs` does its job very well imho 😄

juhoteperi15:07:45

Hmm, though .cljs.edn definitely works for me.

djebbz15:07:36

As a light wrapper indeed it works very well

martinklepsch15:07:10

also very happy with .cljs.edn stuff usually 🙂

djebbz15:07:31

I think it's using cljs.edn files and task-options and inline options at the same time that caused us problems

juhoteperi15:07:12

There might be problems especially with merging some compiler-options.

juhoteperi15:07:54

well, in fact that says that task-options are merged over .cljs.edn

djebbz15:07:16

My understanding is that it's either cljs.edn or not at all, but not mixing both and hoping that options will merge : looks like they don't

djebbz15:07:49

I read the code that merge the options, indeed it should behave as documented

richiardiandrea15:07:00

these issues are difficult to debug, it would be great to have some reproducible case and see which options don't work for you

djebbz15:07:05

No idea why our builds were producing different results

juhoteperi15:07:14

Hmm, though that doesn't describe how task-options! work, that is provided by Boot and not seen by Boot-cljs

juhoteperi15:07:36

"task options" are both the parameters to cljs task and options to task-options! :cljs

juhoteperi15:07:42

Not sure how Boot merges those

juhoteperi15:07:12

I think I usually don't use task-options! with Boot-cljs.

djebbz15:07:32

I followed this advice and it worked for me

juhoteperi15:07:54

Hmm, I quite sure that Boot core can't merge :compiler-options maps if you set one using task-options! and another on cljs params

djebbz15:07:00

I don't think I'll be able to give a reproducible use case, we need to move on

juhoteperi15:07:06

Because Boot can't know how Boot-cljs options should be merged

djebbz15:07:48

Boot can't, but boot-cljs can

juhoteperi15:07:07

But Boot-cljs doesn't see task-options!, Boot merges those before calling Boot-cljs code

djebbz15:07:10

There's a function compiler-options that does just that

djebbz15:07:43

Ok I got it

djebbz15:07:42

Thanks again for your assistance, invaluable for us

martinklepsch15:07:18

What you can do is process your .cljs.edn file in a boot task - this way you can get all the dynamic-ness you want while still keeping the nice aspects of having a .cljs.edn

juhoteperi15:07:48

I'll add issue to Boot-cljs about this, there might be some way to workaround this, or give warning.

juhoteperi15:07:27

Looks like task-options! is implemented so that it saves the task options to task function metadata, so Boot-cljs could at least check if something is set there.

juhoteperi15:07:59

And if the original cljs call parameters can be found some where, Boot-cljs can merge them correctly

djebbz15:07:15

Boot-cljs uses *opts*, I thought it was that

juhoteperi15:07:29

I think that already has task-options! merged into

grzm16:07:12

My ~/.boot/cache is huge. It's over 8GB, and that was after I stopped to check the size when`rm -rf` was taking over a couple of seconds. Is this expected? Is there a standard way of clearing it out?

grzm16:07:54

One reason to clear it out is just to reclaim space: the other is that I had a file showing up (in clj-refactor IIRC) that was no longer in my source tree.

juhoteperi16:07:52

I'm not sure but I think boot cleans the old tmp-dirs of a project when starting a new Boot process

juhoteperi16:07:29

But the caches are per project, so if you have large project a, the old tmp-dirs for that will stay around until you start Boot process on that project, and then it will create new dirs

grzm16:07:17

so, just running boot in a project should clear out the cache for that project?

martinklepsch16:07:45

@richiardiandrea I thought I had a solid repro for https://github.com/boot-clj/boot/pull/474 but can’t find it right now, will try to provide one this week

juhoteperi16:07:21

Starting boot process on my saapas project, removed one tmp-dir and creates another

juhoteperi16:07:33

But there is a second directory that it didn't touch

juhoteperi16:07:49

Not sure why

richiardiandrea16:07:03

@martinklepsch that would be awesome, however I spent some time on it without finding a solution and that is why I guess micha put "help wanted" 😄

martinklepsch16:07:28

yeah, could be I’m confusing it with something else as well

juhoteperi16:07:58

When Boot initializes TmpRegistry for a project, the old directories with .delete-me file are removed, when Boot successfully exists, that file is created: https://github.com/boot-clj/boot/blob/ac835cea8c18e926bb843e01579b86e3f30458d2/boot/core/src/boot/tmpregistry.clj#L79-L80

juhoteperi16:07:29

If Boot crashes, no .delete-me file is created, so in that case the directory is not removed

grzm16:07:45

I can understand that why you might want to keep a crashed tmp directory from a debugging perspective. Would it make sense to have some kind of ensure/finally that creates that file otherwise? I've definitely been crashing boot quite a bit as I've been learning.

juhoteperi16:07:21

Not sure if finally or such would be enough, in some cases, even that can't be called

juhoteperi16:07:45

e.g. OS kills the process due to out of memory etc.

juhoteperi16:07:04

Instead we could write PID to the file and check if it is still running

juhoteperi16:07:35

Similar to what daemon processes usually do in unix

juhoteperi16:07:19

But I'm not sure how easy that would be to implemented so it works for Windows also, and if Java process can even check for running programs easily

grzm16:07:03

Do you think it would make sense to have a boot task that would clear out the cache manually? Or is rm the appropriate interface?

zilti17:07:22

What do I have to do to get a proper uberjar? I did specify the task options, aot the main ns, and set it as main ns for the jar, then did boot aot pom uber jar target. But executing java -jar target/myproject.jar only gives me a Error: Could not find or load main class.

mpenet17:07:23

(jar :main 'foo.the-ns-where-your-main-is)

mpenet17:07:41

ahh from the cli, I guess there's an equivalent

zilti17:07:46

@mpenet that's exactly what I have

zilti17:07:07

(task-options! jar {:main 'foo.bar})

mpenet17:07:46

odd, the task chain looks the same as mine. You're launching the right jar? 🙂

zilti17:07:42

I guess. I mean, it's not like there's a lot of choice in the target dir

mpenet17:07:56

I am boot newbie so I wouldn't know from the cli, I usually just run my own tasks

zilti18:07:47

@mpenet I forgot to add (:gen-class) to the main namespace facepalm

grzm21:07:50

I'm working on a library that I'd like to have work with two different versions of one of its dependencies. So for one of the tasks I'd like to pull in this different version. Is this just a matter of munging the boot environment with set-env! ? I was eyeing the "Task environment conflicts" snippet and thinking that might be a good starting point. https://github.com/boot-clj/boot/wiki/Snippets#task-environment-conflicts

grzm21:07:00

The alternative of commenting/uncommenting portions of build.boot doesn't really appeal 🙂

thosmos22:07:45

I have a project with separate client CLJS and server CLJ app repos. When I'm working on both the server and client, I'm running each in a separate Boot process. The output of the CLJS app dumps its compiled CLJS files into the server app's file tree, which then refreshes its file index. Currently Boot-Reload reloads the browser as soon as the CLJS files are done compiling, but that is before the the server Boot process has had a chance to reload its file index. I'm wondering if there's a trick to getting Boot-Reload to run AFTER the server Boot process refreshes its file index. Another option would be to have the server load from an external resources folder that is updated directly by the CLJS process.

grzm23:07:38

@thosmos I don't have a solution at hand, but to better understand the behavior, you need to force reload the browser again to see the server changes?