Fork me on GitHub
#boot
<
2016-04-21
>
alandipert01:04:47

@mynomoto: yeah, we do deploys on the web things like that

alandipert01:04:48

@mynomoto: on new things though we scale set instance protection on new instances, scale ASG to 2x, and then scale down to 1x which kills the old nodes. so a really easy rolling deploy

alandipert01:04:26

@mynomoto: if you are interested maybe micha and i can do a screenshare of our eb replacement tools, we definitely want to share the stuff when it's a little more polished

alandipert01:04:33

@seancorfield: hmm maybe inside a pod there sholud be something like a shutdown hook/destructor one can register for? to clean up resources in the pod before it gets reaped

alandipert01:04:02

makes sense to me that pods should have this, as tasks do

alandipert01:04:14

considering pods are the primary resource used in tasks

alandipert01:04:39

so in your case your shutdown handler inside the pod could clear the timers

seancorfield01:04:57

But how would it know what to do in the general case? @alandipert The symptom is an exception about a class not found in a library I don’t even depend on.

seancorfield01:04:39

boot show -d highlighted that Carmine pulls in commons-pool and it took some Googling to make the connection with the missing class.

alandipert01:04:43

the library with the timer doesn't expose a way to clear it?

seancorfield01:04:26

It’s a deep system internal to how connection pooling is handled for Redis inside Carmine, as far as I can tell.

seancorfield01:04:07

It doesn’t seem right that I would need to know about that transitive dependency and figure out a way to clean up after that library just because I’m running inside a pod…?

seancorfield02:04:29

I’ve run into this a few times with pods — with Expectations you have to ensure you disable run on shutdown if you require the library in anywhere inside Boot; with Hazelcast I ended up writing a shutdown hook specifically to expose it to Boot to clean up the pod during testing… but that felt hacky.

alandipert02:04:35

it is unusual, especially considering code runs like this in servlets all the time... which is the same scenario

seancorfield02:04:49

Except that Servlets don’t unload classes?

alandipert02:04:03

do they not? i was under the impression that's how hot deploys work

mynomoto02:04:27

@alandipert: that would be cool (replacements for eb). There are so many options. Ecs + docker also looks pretty good

seancorfield02:04:44

Well, that a) doesn’t happen very often and b) no one really cares about a handful of cleanup exceptions when you do a deploy...

seancorfield02:04:02

…but reliably getting exceptions to the console while running tests is… annoying as heck simple_smile

seancorfield02:04:00

If the recommendation is simply to explicitly shutdown the pool eviction timers in the pod code, sure, I’ll go ahead and figure out how to do it… But I was hoping there was something cleaner and more generic that I was just missing 😞

alandipert02:04:30

i would say the underlying problem is downstream, it's just wrong to use resources without providing a means to deallocate/cancel

alandipert02:04:46

so anything we can do in boot would be a workaround, which is cool, and some come to mind

alandipert02:04:50

maybe the thing registers a jvm shutdown hook?

alandipert02:04:26

oh wait, so you do have a way to cancel the timers yourself?

alandipert02:04:06

i'm with you it doesn't sound generic, but it does seem clean

alandipert02:04:03

also as i haven't seen this before i'm probably missing stuff, so if you have time to post an example on gh that would be super useful. would be interested to poke around

alandipert02:04:39

@mynomoto: yeah! it's a jungle. we're settling on our shell scripts and eb tool with a few conventions, and deploying standalone uberjars that run their own web servers. and a custom AMI

seancorfield02:04:18

Looking at the Carmine source, it’s because we’re using the default connection mode so it creates the connection pool behind the scenes and just assumes it can live until the end of the program...

alandipert02:04:08

maybe it can be configured to not use a connection pool? then you could do that, in testing

seancorfield02:04:18

Or we’d have to switch our code around to explicitly create the pooled connection ourselves and pass it into Carmine — and then expose a hook to explicitly close it.

seancorfield02:04:29

(which is essentially what we ended up doing with Hazelcast)

seancorfield02:04:04

And, yeah, we could use a non-pooled connection during testing… not sure how much that would slow things down… I could certainly try it.

alandipert02:04:05

incidentally, are you aware of any JVM implementation of the redis server? i am curious about slaving a JVM directly to a redis so that the cache is in-process

seancorfield02:04:42

I’m new to Redis so, no, sorry simple_smile

seancorfield02:04:30

We’re ultimately moving from Hazelcast to ElastiCache so we’re evaluating Redis as part of that. We’re also starting to look at DynamoDB (so we’ll be playing with Faraday which is another taoensso library!).

seancorfield02:04:55

So. Many. Freakin’. Options.

dm308:04:14

is there a from-webjars task for Boot (like boot-cljsjs/from-cljsjs) ?

dm308:04:12

lol, it's in the same boot-cljsjs ns

juhoteperi08:04:25

dm3: you could also use sift :add-jar. And depending on what you want to do either might not be needed.

dm308:04:16

hmm:

(from-webjars :name "font-awesome/css/font-awesome.min.css")

Adding META-INF/resources/webjars/font-awesome/4.6.1/css/font-awesome.min.css to fileset
java.lang.NullPointerException: 
               io.clj:  404
                           io.clj:  416
        cljsjs.boot-cljsjs/copy-file  boot_cljsjs.clj:   20
cljsjs.boot-cljsjs/eval2736/fn/fn/fn  boot_cljsjs.clj:   75

dm308:04:54

ah, must specify target

dm308:04:16

@juhoteperi: what's the best way to mark task option as mandatory in boot?

dm308:04:22

the idiomatic way

eveline08:04:09

Hi everyone, I'm looking for some documentation on the boot show -p option, but can't seem to find it...

eveline08:04:06

I'm mostly wondering what the ! means here:

[!] riddley
    ✔ 0.1.12
      manifold
    ✘ 0.1.10
      clj-http
      clj-http-fake
      clj-webdriver

micha11:04:08

@eveline: the [!] there means that the version of that dependency was decided by Aether and not specified by you as a direct dependency

micha11:04:48

@eveline: like in that case you have four dependncies that depend on riddley, three of them depend on version 0.1.10, and one depends on 0.1.12

micha11:04:17

aether has a sort of arbitrary way of resolving transitive dependency conflicts like this one

micha11:04:37

it picks the version with the shortest "path"

eveline11:04:28

Oke, good to know

dominicm12:04:34

I've never used figwheel/sidecar before. I'm quickly seeing why boot is so superior.

dominicm12:04:07

I can't even get sidecar to build some cljs

mynomoto13:04:39

@dominicm: yesterday a coworker started a project using figwheel and the cljs-repl was taking 1s to return (inc 1). I converted that to boot and then it returned instantaneously. That's have been my experience on both sides consistently.

dominicm13:04:21

@mynomoto: I'll let you know when I've set it up.

dominicm13:04:59

Big project, we have custom leiningen plugins and such, so I can't justify switching to boot at this point really. I'd love to at some point.

mynomoto13:04:47

dominicm: only client side or server side also?

martinklepsch13:04:59

@dominicm: what are the custom lein plugins (just out of curiosity)

dominicm13:04:43

@martinklepsch: We have some kind of docker/beanstalk leiningen plugin. I could probably port to boot, but I don't understand it very well. I think it's just taking a jar, and passing it into a library.

dominicm13:04:53

But the declaration of that in leiningen is very difficult.

dominicm13:04:15

The hardest part of boot, is understanding the leiningen stuff, so you can port it to boot.

martinklepsch13:04:33

I'm sure all docker/eb related things are covered by Alan & Micha already 😄

mynomoto13:04:52

Yeah, not sure about the docker part but the eb should be fine. But converting large projects is hard...

micha13:04:56

@dominicm have you used the aws eb tool?

micha13:04:25

that's what we used back w when we were on beanstalk

dominicm13:04:33

@micha: I have. I really hated it. I'll admit to being a luddite in regards to AWS.

micha13:04:06

we hated the lein plugin way more

micha13:04:40

i made boot-beanstalk that wrapped the plugin

micha13:04:02

but it has to hardcode things

micha13:04:25

the eb tool is at least minimally invasive

micha13:04:30

for me it was easy better to configure beanstalk directly than to make a Java program to do it via the clojure plugin

dominicm13:04:21

That might be why this project uses it's own

micha13:04:03

the plugin had just as much of the magical map configuration but wasn't aws maintained so it had to make assump tons

gtrak13:04:27

so, I'm trying to get a clj boot workflow that matched what I had with leiningen in addition to the cool component refresh stuff (a separate task leveraging 'system'), how do I turn off the fileset stuff when building a classpath and use source directories directly? boot.user=> (http://clojure.java.io/resource "/my_app/core.clj") #object[java.net.URL 0x3ddc44a6 "file:/home/gary/.boot/cache/tmp/home/gary/dev/my_app/169/-grrwi1/my_app/core.clj"]

gtrak13:04:55

I'd like to be able to use tools.namespace manually at the repl

micha13:04:43

@gtrak: did you see the wiki page about repl reloading with tools.namespace?

gtrak13:04:51

yes, I found that

micha13:04:59

does it not do what you want?

gtrak13:04:28

I will double-check

micha13:04:40

like what is the goal you want to achieve?

gtrak13:04:52

it looks like (get-env :directories) returns the cache dirs and not the source dirs. Perhaps I'm missing a step that copies modified source into the cache.

micha13:04:06

why do you care about the source dirs?

gtrak13:04:30

I'm modifying the source with a standard REPL workflow

micha13:04:00

boot doesn't know about source dirs really

micha13:04:12

that's a core design feature

micha13:04:22

the project files belong to the user

micha13:04:31

boot doesn't have direct access to them

gtrak13:04:36

it seems like some variant of the watch task is needed, really I'm just getting started with the mental model.

micha13:04:43

so you will need to build something outside of boot

gtrak13:04:08

actually, already using watch, though maybe incorrectly

micha13:04:14

that will also potentially introduce race conditions

micha13:04:37

the files in your project that you add to the fileset via :source-paths etc are copied into temp dirs

micha13:04:49

this way boot can modify them, delete them, move them around etc

micha13:04:59

in a reversible way

micha13:04:18

boot needs to be able to do that for the fileset abstraction to work

micha13:04:40

but you can do like ( "src/foo.clj") of course

micha13:04:43

and you can write to it

micha13:04:48

but that file is not on the classpath

micha13:04:00

a copy of it is, and that copy will be in a temp dir

dominicm13:04:36

@micha: Interesting. It's possible that we hard-code that stuff, I really couldn't tell you how that plugin works

micha13:04:00

refactoring seems like a job for a standalone program

micha13:04:06

not part of the build workflow

micha13:04:15

like in the IDE or whatever

dominicm13:04:16

Oh, it literally just wraps leiningen-beanstalk

micha13:04:20

@dominicm: it's been a year or so since i looked at lein-beanstalk, but the version i used in boot-beanstalk needed to hardcode things internally that don't need to be hardcoded if you use the eb tool

micha13:04:39

and there are a lot of things that were not implemented in the plugin that are implemented in the eb tool

micha13:04:52

it seemed like a waste of time to try to reimplement the eb tool in clojure

dominicm13:04:58

Yeah, makes sense based on what I'm looking at.

micha13:04:01

since for us anyway deployment is a totally separate operation

micha13:04:14

and beanstalk really sucks at deployment

dominicm13:04:19

I just hate AWS in general. I can't understand it.

micha13:04:33

yeah beanstalk in particular is a pile of crap imho

dominicm13:04:43

I want a server I can hack on, and I don't want to learn something new to be able to use it.

micha13:04:49

we are in the process of de-beaning

dominicm13:04:59

I have to click so much crap for AWS. None of it makes sense.

micha13:04:05

yeah and the worst thing is that they change things in the beanstalk stack all the time

dominicm13:04:06

If that is leaving aws, we just did that on another project.

dominicm13:04:17

Oh really? Ouch.

micha13:04:23

we're still 100% AWS, just off beanstalk

dominicm13:04:30

We have a dated stack. Ah, the bean part.

micha13:04:44

beanstalk is built on a teetering tower of ruby and python scripts

micha13:04:57

and all kinds of needlessly complicated infrastructure

micha13:04:07

so we regularly have rolling deploys fail

micha13:04:12

in bad ways

micha13:04:27

and it's not possible to manually intervene if a deploy is going south

micha13:04:39

so we moved to blue/green deployments

micha13:04:51

but that also has issues because there isn't a way to manage that cleanly

micha14:04:13

like it's very easy to have some small difference between the two environments

micha14:04:22

like security group stuff mostly

dominicm14:04:34

I don't evenly remotely understand security groups.

mynomoto14:04:34

@dominicm: they are a sort firewall rules for instances

dominicm14:04:20

@mynomoto: But they're a complex hierarchy system really right? Easy to have small mistakes cause issues

mynomoto14:04:46

Yeah, unless you want something like allow all and bypass it...

mynomoto14:04:58

@dominicm: I actually think it is easier to use security groups as target rules on other security groups. That makes harder to make mistakes imo.

lwhorton15:04:51

can someone point me to an example of a working cljs-test boot task? I just cannot get mine to work for some reason...

mynomoto16:04:07

@lwhorton: https://github.com/martinklepsch/tenzing is a template that should setup it for you using +test option

lwhorton16:04:45

I have been looking at that, thanks for the ref though @mynomoto. What’s stranger still is that I can get the tests to run as expected without hiccup, but only if I use :advanced optimizations

lwhorton16:04:02

:none just give me

Running cljs tests...ClojureScript could not load :main, did you forget to specify :asset-path?

ERROR: doo was not loaded from the compiled script.

Make sure you start your tests using doo-tests or doo-all-tests
and that you include that file in your build

mynomoto17:04:27

@lwhorton: you may open an issue on https://github.com/crisptrutski/boot-cljs-test . You probably could use :simple optimizations or :whitespace that should be faster than :advanced. The test runner is probably not loading the js referenced by main.js.

lwhorton17:04:56

i’ll take a closer look at that repo and see what I can find, thanks

twashing20:04:59

Being new to boot, I have a rudimentary boot project that I want to i) Dockerize, then ii) start a server in the background, so I can have iii) the option to stop it later

twashing20:04:40

$ boot run
             clojure.lang.ExceptionInfo: java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@4d32b021
    data: {:file
           "/var/folders/dv/qph8cnxn5c5fqyxbc60bj16m0000gn/T/boot.user8563596840446670088.clj",
           :line 21}
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@4d32b021
     java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@4d32b021
                                      ...                  
org.httpkit.server/run-server/stop-server  server.clj:   26
                                      ...                  
                      boot.core/run-tasks    core.clj:  794
                        boot.core/boot/fn    core.clj:  804
      clojure.core/binding-conveyor-fn/fn    core.clj: 1938
                                      ...

twashing20:04:02

boot run works if the run task just has (println “Hello World”), so there’s something I’m missing when requiring in the beatthemarket.shell namespace.

martinklepsch20:04:31

@twashing: if you do boot repl and require that ns, does it work?

twashing20:04:24

$ boot cider repl
nREPL server started on port 61756 on host 127.0.0.1 - 
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.8.0
...
boot.user=> (require '[beatthemarket.shell])
nil

twashing20:04:18

and here’s my boot profile and properties files...

twashing20:04:27

$ cat ~/.boot/profile.boot 
(deftask cider "CIDER profile"
  []
  (require 'boot.repl)
  (swap! @(resolve 'boot.repl/*default-dependencies*)
         concat '[[org.clojure/tools.nrepl "0.2.12"]
                  [cider/cider-nrepl "0.12.0"]
                  [refactor-nrepl "2.2.0"]])
  (swap! @(resolve 'boot.repl/*default-middleware*)
         concat '[cider.nrepl/cider-middleware
                  refactor-nrepl.middleware/wrap-refactor])
  identity)

twashing20:04:34

$ cat ~/.boot/boot.properties 
#
#Sun Apr 03 19:27:50 PDT 2016
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_VERSION=2.5.5
BOOT_CLOJURE_VERSION=1.8.0
BOOT_EMIT_TARGET=no

richiardiandrea20:04:00

@twashing: which version of cider are you using?

richiardiandrea20:04:31

if 0.12 you should not need any cider task as middlewares are auto-injected

twashing20:04:28

I believe it’s the latest cider (Seattle)...

twashing20:04:36

I’ll try switching it off

richiardiandrea20:04:45

just to be sure 😉

twashing20:04:01

oh absolutely

twashing20:04:36

so I went ahead and backed up profile.boot, but am still getting the same error

twashing20:04:47

$ ll ~/.boot/
total 16
drwxr-xr-x   5 timothyw  staff   170 Apr 21 13:32 .
drwxr-xr-x+ 63 timothyw  staff  2142 Apr 21 13:02 ..
-rw-r--r--   1 timothyw  staff   155 Apr 19 16:30 boot.properties
drwxr-xr-x   6 timothyw  staff   204 Mar 12 14:35 cache
-rw-r--r--   1 timothyw  staff   430 Apr 19 16:41 profile.boot.1

$ boot run
             clojure.lang.ExceptionInfo: java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@2997042b
    data: {:file
           "/var/folders/dv/qph8cnxn5c5fqyxbc60bj16m0000gn/T/boot.user3887614655177056510.clj",
           :line 15}
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@2997042b
     java.lang.IllegalArgumentException: No value supplied for key: boot.core$run_tasks$fn__901@2997042b
                                      ...                  
org.httpkit.server/run-server/stop-server  server.clj:   26
                                      ...                  
                      boot.core/run-tasks    core.clj:  794
                        boot.core/boot/fn    core.clj:  804
      clojure.core/binding-conveyor-fn/fn    core.clj: 1938

twashing20:04:30

and again, that task just looks like this

twashing20:04:37

(require '[beatthemarket.shell])
(deftask run []
  (beatthemarket.shell/start-server))

twashing20:04:39

I’m gonna grab a coffee. back online in 10.

richiardiandrea20:04:18

it looks like the culprit is -> org.httpkit.server/run-server/stop-server server.clj: 26

richiardiandrea20:04:32

but just guessing 😄

lwhorton20:04:16

has anyone seen a speclj-boot task floating around?

lwhorton20:04:28

im not experienced enough with boot to understand/construct my own without it taking all day, and I really just want to move on

twashing21:04:27

@richiardiandrea hmm, is there a way in boot, to see the elided stacktraces?

twashing21:04:43

or more generally, is there a straight forward way to use http-kit with boot?

twashing21:04:50

… because that code works from the boot repl

micha21:04:42

you can do boot -vvv ... @twashing

twashing21:04:32

Seems my deftask needed to return a fn, that returns a fn, like here: https://github.com/boot-clj/boot/wiki/Tasks#the-null-task

twashing21:04:43

now I just need to daemonize this, so it serves files… currently it just exits after running.

twashing21:04:31

(require '[beatthemarket.shell])
(deftask run []
  (fn [next-task]
    (fn [fileset]
      (beatthemarket.shell/start-server))))

richiardiandrea21:04:19

@twashing: try to append wait at the end of your task chain

twashing21:04:41

@richiardiandrea boot run -- wait doesn’t work

richiardiandrea21:04:00

Why --? try to remove them

twashing21:04:07

oh, thought that’s how you chained tasks… but it still doesn’t wait

richiardiandrea21:04:41

Uhm...gotta go now but it definitely should, I usually use it with serve

twashing21:04:35

@richiardiandrea ok, thanks for the help

richiardiandrea21:04:19

NP sorry I could not help more ;)

micha21:04:27

@twashing: the -- is ignored in between tasks because of how the cli parsing works

micha21:04:04

@twashing: your handler should return a fileset

micha21:04:07

like this:

micha21:04:35

(deftask run []
  (with-pass-thru [_]
    (shell/start-server)))

micha21:04:47

with-pass-thru is a macro

micha21:04:51

equivalent to this:

micha21:04:29

(deftask run []
  (fn [next-task]
    (fn [fileset]
      (shell/start-server)
      (next-task fileset))))

micha21:04:43

you never called next-task

micha21:04:56

which is in your command line case, the wait task

micha22:04:08

so the wait task never would run

twashing22:04:29

hmm… lemme try

twashing22:04:13

ok, the with-pass-thru seems to have done the trick

micha22:04:47

yeah the issue you saw was that your task never called next-task

micha22:04:00

that's how the pipeline continues

micha22:04:23

you always need to call next-task with a fileset as its argument

micha22:04:36

the with-pass-thru macro hooks that up for you

twashing22:04:51

cheers mate

twashing22:04:55

now, what’s the best way docker running this ...Dockerfile is:

FROM adzerk/boot-clj

twashing22:04:11

$ docker build -t beatthemarket .
...
Successfully built 4d764e6ade15

twashing22:04:43

Is it $ docker run beatthemarket boot run wait

micha22:04:54

hm i'm kind of rusty with the docker stuff

micha22:04:33

you may need to specify CMD or ENTRYPOINT

twashing22:04:32

… looks like docker run beatthemarket run wait gets me closer there (although I don’t understand how docker knows how to run boot waith said arguments)

micha22:04:17

i think that's set in the boot-clj container's Dockerfile

micha22:04:24

that one has a CMD clause

twashing22:04:32

but it doesn’t seem to know about my run task

clojure.lang.ExceptionInfo: java.lang.IllegalArgumentException: No such task (run)
    data: {:file "/tmp/boot.user1450022986807314043.clj", :line 7}
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No such task (run)
     java.lang.IllegalArgumentException: No such task (run)
          boot.core/construct-tasks  core.clj:  754
                                ...                
                 clojure.core/apply  core.clj:  630
                  boot.core/boot/fn  core.clj:  805
clojure.core/binding-conveyor-fn/fn  core.clj: 1916

micha22:04:05

seems like maybe your bui.dboot file isn't making it into the container

micha22:04:38

you'll probably need to make some ADD lines in your Dockerfile

twashing22:04:50

hmm, what’s the default app directory. ie, which directory should I copy build.boot to?

twashing22:04:04

digging now...

twashing23:04:31

FYI, looks like WORKDIR is the relevant command: https://docs.docker.com/engine/reference/builder/#workdir