Fork me on GitHub
#boot
<
2016-02-23
>
mobileink03:02:01

I'm getting a strange crash that occurs every other time I run my build task:

mobileink03:02:30

clojure.lang.ExceptionInfo: java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: /Users/gar/.boot/cache/tmp/Users/gar/boot/boot-bowdlerize/jetty/1s66/-rcsl8f/hello/styles.clj

mobileink03:02:19

the boot-bowdlerize repo now has basic doc and a jetty testapp that uses boot-http, compojure, hiccup, and some bower resources.

mobileink03:02:18

the jetty testapp is where the crash occurs.

onetom08:02:47

@danielsz: we were looking into system.components.mongo with @mbut and we were wondering why the start method is not idempotent as recommended in https://github.com/stuartsierra/component#idempotence we just had an issue with the open connections, because accidentally our test suite was creating new mongo components on every run and our ~6 workstations collectively exhausted the connection limit every few days. (plus we had some orphaned test suites stuck in the background which were maintaining thousands of open connections by reconnecting to the mongo server even after we restarted the server simple_smile

danielsz08:02:00

Great. I'll be happy to merge. Thanks!

onetom08:02:02

so should we make start and stop idempotent or was there any reason why u didn't make them so?

danielsz08:02:43

onetom: Yes, by all means. No reason other than youth and foolishness. simple_smile

juhoteperi08:02:02

Also, under the same idempotence header there is note about catching all errors on stop

onetom08:02:05

we also have an adi (http://docs.caudate.me/adi/) component on the way, btw which we just tried to fork with https://github.com/vvvvalvalval/datomock thx to @val_waeselynck very exciting; thanks for making this component collection! simple_smile

onetom08:02:51

@juhoteperi: thanks for the reminder, i forgot about that!

juhoteperi08:02:38

I don't remember if I have ever hit problems caused by stop throwing, but should be good idea still

onetom09:02:29

not sure what should we do in the catch block though... just do a print? none of the other components handle exceptions either 😕

nha09:02:31

I use https://github.com/MichaelDrogalis/dire in my system :

(require '[dire.core :refer [with-handler!])
(with-handler! #'com.stuartsierra.component/stop
  "catch all component stop exceptions so that the whole system can always shutdown"
  Exception
  (fn [e &args]
    (println "Stop exception ! " e)))

danielsz09:02:02

You could also throw the exception.

danielsz09:02:50

That would allow the consumer to act on it.

nha09:02:27

I let it throw for the start. But for the stop I prefer to make sure every component got a chance to stop. Not sure yet what it the best approach.

nha09:02:53

Unrelated question : what kind of args can I pass to boot defcli (there is only bool there https://github.com/boot-clj/boot/wiki/Scripts#command-line-args ) ? I would like to pass Files/String paths.

nha09:02:35

(Also, I can't get the example CLI to get a false value).

(defclifn -main
  [b test-bool bool "A test boolean"]
  (println test-bool))
./my-script.clj --test-bool false ;;=> true

onetom09:02:43

@nha: i think i hit the same issue and discussed it with micha, but there is no way to represent false flags currently (as in a half year ago). isn't the GNU convention applies a --no- prefix for this purpose?

nha09:02:46

@onetom: I see, that could work. For now I am just trying to understand the defcli macro : in the snippet below, what are the third and fourth elements ?

(defcli -main
  [t              ;; short name
   test           ;; long name
   WHATISTHIS     ;; ?
   str            ;; parse fn ? (where can I get a list ?)
   "Some doc   "  ;; docstring
   ]
  )

nha13:02:26

@micha perfect, thanks simple_smile

nha13:02:18

(I already used tools.cli though so this will be for next time - added a link in the wiki also)

mobileink13:02:19

Seeking feedback/collaboration on https://github.com/mobileink/boot-bowdlerize, which basically generates runtime config namespaces from buildtime config namespaces, with support for bower. This tasklib resulted from my pondering the advice from @micha and @alandipert regarding management and partitioning of config data in https://github.com/migae/boot-gae . I haven't reorganized the latter, but needed something like boot-bowdlerize for another project. I have one good use case for it (not yet illustrated, the lib is still under dev); I'm curious if anybody else would find this useful. Or is it overkill?

jaen15:02:26

Quick question - how do you create a file in the temporary directory created by tmp-dir!? Usual Clojure file manipulation suspects or are there some helpers I'm missing (at least I can't see them in boot.core)?

dm315:02:17

it's a normal directory, only managed by boot

dm315:02:28

so you can use normal Java tools

dm315:02:42

meaning , etc

jaen15:02:23

I see; I figured maybe there may be something akin to tmp-dir! for tmp files.

jaen15:02:35

But yep, make-parents work as it should; thanks!

micha15:02:54

boot operates at the directory level for temp files

micha15:02:21

you can do whatever you want inside the direcotry tmp-dir! returns

micha15:02:40

boot will just delete everything when it cleans up

jaen15:02:18

Right, that makes sense; I was just kinda confused by parent directories not being created, but that's not boot's fault.

mobileink16:02:56

fyi boot-bowdlerize now has a sample bowdlerized Polymer testapp.

tolitius16:02:20

I'd like to give a boot talk (mostly intro, explaining core concepts and "why") at the Philadelphia Clojure Meetup (a.k.a. Clojadelphia), it's coming in a couple of weeks, but I am a bit swamped with work and won't have time to make solid slides. I'll mostly do coding examples, but does anyone have any ready Boot materials that is ok for me to show for the intro? I'll of course mention the author and give all the proper credit.

tolitius16:02:22

@juhoteperi: thanks! I'll pick out some slides from there, especially the "why" part, since those problems with lein are fundamental and still are there

dm317:02:09

what's the proper way to reload the build.boot file when in the repl?

micha17:02:34

(load-file "build.boot")

dm317:02:46

ooh, forgot about load-file simple_smile

pandeiro17:02:55

say I store in the initial boot set-env! some info that I pass to tasks; is there a way to make those tasks "see" changes to that info in the env while in the watch pipeline?

micha17:02:36

i don't recommend doing that

micha17:02:55

because it breaks the pipeline model in some ways

micha17:02:19

tasks should be able to participate in the configuration process

micha17:02:31

which they can't do if the configuration is in a global var

micha17:02:39

because global vars are not immutable

micha17:02:58

if you put your configuration info in the fileset though, then it's open rather than closed

micha17:02:11

and tasks can participate in the configuration process

micha17:02:17

because the fileset is immutable

micha17:02:41

so tasks only see what the tasks preceding them in the pipeline did

micha17:02:53

that's better than having all the tasks banging on shared mutable state

pandeiro17:02:32

OK, so I move that config from set-env! in build.boot to a config.edn file or something?

micha17:02:44

so for tl;dr; i would recommend making a file with a unique extension, yes

micha17:02:08

that way other tasks can modify that to help automate things

pandeiro17:02:14

I thought it was OK that say a complex dev task knows how to extract info from the env and pass it to different tasks as appropriate though

micha17:02:34

you can do it, but you are then just storing stuff in a data bag

pandeiro17:02:41

I don't really need tasks to modify that, but take it as an input

micha17:02:46

so you lose the ability for tasks to modify that

micha17:02:03

well that means you need to specify everything explicitly

micha17:02:18

if all tasks worked that way boot would be just a huge configuration map

micha17:02:28

without the power to compose tasks

pandeiro17:02:51

sure, I see the point in that

micha17:02:01

like the boot-cljs, cljs-repl, and cljs-reload tasks would need to be combined into one monolithic task

pandeiro17:02:11

yeah, that is the classic example

micha17:02:14

and for hoplon i'd need to combine all four

pandeiro17:02:23

but there are plenty of tasks we have that don't actually modify the fileset

pandeiro17:02:32

testing tasks for instance

micha17:02:37

they also get their configuration from command line args

micha17:02:47

which doesn't change in the pipeline

micha17:02:53

that's the key thing

micha17:02:10

they don't have a data bag of config map

pandeiro17:02:30

ok so say I have a testing task that takes which namespaces to test as an arg...

pandeiro17:02:45

And I'd like to be able to change that set, while developing, without restarting the pipeline

micha17:02:14

that's a perfect place to use a spec.test.edn

micha17:02:19

or something like that

micha17:02:28

a specification

micha17:02:39

that informs your thing how to configure itself

micha17:02:56

because if you're already doing fancy stuff at runtime you are going to want to make tasks to automate that

micha17:02:13

even if you don't need to now, you will soon enough

micha17:02:42

another option is to add metadata to namespaces

micha17:02:48

things like that

micha17:02:00

store the information in the environment itself, not in a map

micha17:02:14

you can alter-meta on namespaces

pandeiro17:02:39

ok, i think the spec file is an approach we could use

micha17:02:54

if you just use a regular var you can add a watch to it

pandeiro17:02:27

would there be any philosophical argument against generating that spec file from a map in build.boot itself? simple_smile

pandeiro17:02:47

i've been meaning to do that with the cljs.edn files for a while now

micha17:02:57

yeah that's the whole point!

micha17:02:07

i never create those files manually

micha17:02:13

i ahve tools that do it automatically

micha17:02:27

like framework level things

micha17:02:44

for example hoplon automatically generates both the html file and the cljs.edn file

micha17:02:05

you just have a cljs program, and hoplon can generate all the stuff you neeed to make it a webapp

pandeiro17:02:25

and can you modify the hoplon task while it's in the watch pipeline, or do you need to restart?

pandeiro17:02:35

like if some arg to that task should change

micha17:02:36

the task is reading the file

micha17:02:45

so if the file changes it will see a new thing

pandeiro17:02:54

reading a file then generating other files?

micha17:02:06

not following

pandeiro17:02:16

you said it generates the cljs.edn files

micha17:02:27

yes, based on other files in the fileset

micha17:02:34

it looks for *.hl files

micha17:02:46

those are the cljs programs

pandeiro17:02:50

right, ok, so that's not even passed in via params

micha17:02:57

each one gets a html file generated for it, and a .cljs.edn file

pandeiro17:02:58

it just knows where to look in the fileset

micha17:02:04

yeah it scans

micha17:02:21

the task could be looking in other places

micha17:02:36

the point is that it's generating the cljs.edn files

pandeiro17:02:02

so should it be possible to switch boot-cljs from :none to :advanced without restarting the pipeline?

micha17:02:25

in theory yes

pandeiro17:02:28

if I had a wrapper task that reads some file to determine what :optimizations to use

micha17:02:43

that still depends on how the cljs compiler handles that

micha17:02:58

it might mean trashing the compiler state

pandeiro17:02:12

ok yeah I've never actually tried that craziness, just using it as an example of a task that currently requires an arg for configuration

micha17:02:54

the main idea is that if the state of the build is contained inthe fileset then you can do all things

micha17:02:15

once you start storing state in mutable places in the clojure runtime you lose that ability

pandeiro17:02:39

i think i see the error of my ways

micha17:02:01

the crucial thing the fileset provides is the automatic rewinding to the initial state each time the pipeline runs

micha17:02:20

that means that tasks can progressively enhance the configuration

micha17:02:35

without running into the problems of unmanaged shared state

pandeiro17:02:28

thanks for setting me straight once again @micha, think I know how I want to proceed now

mobileink19:02:25

anybody else out there using boot to work with webjars? i didn't even know there was such a thing until @juhoteperi clued me in this morning, so i've whipped up a few tasks, e.g. webjars (in my world) are build-time only, so boot show -d won't show them, so i've got a show task in boot-bowdlerize to show build-time webjar deps. boot is so cool. anyway, i'm interested in knowing how people deal with webjars in clojure apps.

mobileink19:02:23

a related question: any particular reason boot.aether is not documented and exposed as an api? i'm using boot.aether/dep-tree, there must be other useful stuff in there.

richiardiandrea20:02:44

Hey guys, what is wrong with:

(pod/with-eval-in pod
        (boot.util/info "%s" (str (:source-paths boot.pod/env)))

micha20:02:32

looks good to me...?

micha20:02:00

you may need to require the namespaces?

micha20:02:08

in the pod

richiardiandrea20:02:13

I get

Caused by java.lang.IllegalArgumentException
   Arguments must be either all strings or all fns

richiardiandrea20:02:24

you mean boot.pod?

micha20:02:48

that exception is coming from the boot function i think

micha20:02:59

i think the pod expression is okay, the problem is elsewhere where you call boot

richiardiandrea20:02:52

I had forgotten with-pass-thru 😞

richiardiandrea20:02:41

and I never remember what I need to do to print info in a pod..

richiardiandrea20:02:07

(pod/with-eval-in @pod
          (require 'boot.pod)
          (require 'boot.util)
          (reset! boot.util/*verbosity* 3)
          (boot.util/info "Hey %s" boot.pod/env))

richiardiandrea20:02:58

ah @micha wait, the info is printed, but not in the repl

richiardiandrea20:02:05

it works on the command line

micha20:02:42

weird, could be some nrepl weirdness with threads

richiardiandrea20:02:28

@micha so if I have my files on :source-paths in a pod, shouldn't I be able to launch a method on them? In my case the -main?

micha20:02:00

pods look at :directories, but yes

micha20:02:10

require the namespace, etc

micha20:02:23

depending on how you require you may need resolve too

richiardiandrea20:02:46

I have not changed :directories but outside the pod I did not have the same source path

micha20:02:07

pods don't look at :source-paths or :resource-paths etc

micha20:02:15

they just care about the classpath, which is :directories

micha20:02:30

that should include both ;source and :resource-paths

richiardiandrea20:02:12

but if I create a pod by modifying the old env, and adding my source-paths are :directories changing accordingly?

richiardiandrea20:02:23

from what I see the answer is not...at least here

richiardiandrea20:02:52

(let [options (options :generator)
          pod-env (merge-with (fn [_ new] new) (get-env) (:env options))] ;; <- brand new source and dependencies
      (let [pod (future (pod/make-pod pod-env))]
        (pod/with-eval-in @pod
          (boot.util/info "Directories %s\n" (:directories boot.pod/env))
          (require 'cljs-api.generator)
          (cljs-api.generator/-main))
        (pod/destroy-pod @pod)))

micha20:02:09

yeah that won't work

micha20:02:23

pods don't care about :source-paths

micha20:02:29

so that will be ignored

micha20:02:13

:source-paths is just a thing in the core pod

micha20:02:32

and the classpath is really defined by :directories there too

richiardiandrea20:02:58

ah, I expected make-pod to take care of that too

micha20:02:14

pods i think should be lower level things

richiardiandrea20:02:25

so if I point my :directories to my new source path it should work..

micha20:02:29

i'm not sure it's a good idea to mock core pod behavior for them

micha20:02:38

makes things less direct

richiardiandrea20:02:58

the goal is to obtain classpath isolation so maybe I am not Doing It Right 😄

micha20:02:01

:directories points to a set of paths to directories on the filesystem

micha20:02:24

if you add a path to another direcotry then that one will be on the classpath when you make the pod

micha20:02:55

if you're making a pod the classpath will be isolated, for sure

richiardiandrea20:02:13

great...so what am I missing, add-classpath with my new source path?

micha20:02:12

if it's in the :directories set that you pass to make-pod then it will be on the classpath

richiardiandrea20:02:01

(boot.pod/add-classpath "src/clj") works too

richiardiandrea20:02:41

java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #' found for class: nil
     clojure.core/-cache-protocol-fn                  core_deftype.clj:  554
                                            io.clj:   69
                                          io.clj:  102
                                 ...                                        
                  clojure.core/apply                          core.clj:  632
                  clojure.core/slurp                          core.clj: 6653
                                 ...                                        
cljs-api.generator/load-cljs-api-edn                     generator.clj:  176
            cljs-api.generator/-main                     generator.clj:  207

richiardiandrea20:02:51

I can launch it now at least 😄

micha20:02:12

that's what you see when the path doesn't point to a directory

micha20:02:37

actually you're calling slurp on nil

richiardiandrea20:02:50

yes, it is a problem in my code now I think

richiardiandrea20:02:33

actually this guys reads a file that I need to add to the class path as well

richiardiandrea20:02:31

Cljs api dumped in src/cljs/cljs_repl_web/cljs_api.cljs
Generated docs for 120 symbols
Victory!

richiardiandrea21:02:07

thank you a lot 😉

mobileink21:02:56

the dreaded "No reader function for tag object" is giving me fits. again. this time, I do let [tgt (boot/tmp-dir!)] at the beginning of my task. then as soon as i enter boot/with-pre-wrap` i do (boot/empty-dir! tgt). but then, inside my pod, any reference to ~tgt` gives me the dreaded exception. i'm trying to use pod/unpack-jar with no luck. a string dest didn't work so I'm trying to make a File object on a tmp-dir.

micha21:02:51

@mobileink: only forms that can be printed with pr-str and read via read-string can pass betwen pods

micha21:02:01

that's why you see that exception

micha21:02:21

you want to pass in the string path to the java.io.File object

micha21:02:25

not the object itself

micha21:02:47

i.e. pass ~(.getPath tgt) instead of ~tgt

micha21:02:54

the test it must pass is (= tgt (read-string (pr-str tgt)))

mobileink21:02:55

thanks, let [destdir (str ~(.getPath tgt) "/" ~base)] in the pod did the trick. That definitely deserves a branch in the boot-fails project.

mobileink21:02:43

out of curiosity, what's the reason for this restriction?

micha21:02:24

it's the way clojure works, interfaces and classes are created dynamically at runtime

micha21:02:41

so two clojure runtimes can't understand each other's clojure things

mobileink21:02:57

right, forgot pods are runtimes

micha21:02:39

i have an experimental method that removes this restriction mostly

micha21:02:05

uses reflection heavily though so is slower than pr-str

mobileink22:02:40

fyi i just copied this to a new item on the Troubleshooting page in the wiki. Not the first time I've encountered it and I'm pretty sure it won't be the last.

micha22:02:02

👍 thanks!

richiardiandrea22:02:06

I think I have also added it to the docstring PR

mobileink22:02:40

FWIW I'm beginning to see that filesets are not the only killer feature of boot. pods are nutty cool. I wonder, can competing java-based build tools do this kinda stuff? I'm looking at you, gradle. stuff like support isolated class paths .