Fork me on GitHub
#boot
<
2015-09-27
>
lukemorton09:09:45

I've got a very basic macro that compiles markdown into a quiescent component. What I'd like to do is retrigger a boot compile every time I change the markdown file. I've added my "markdown" folder to sources in build.boot and saving a .md file does trigger (watch) but doesn't change any cljs so the macro doesnt get run.

juhoteperi10:09:48

@lukemorton: Nothing Boot can do about that, Cljs compiler decides itself when to recompile cljs

juhoteperi10:09:43

Instead you could have task which watches .md files and compiles them into cljs files

martinklepsch11:09:28

@juhoteperi: @lukemorton or you could write a task that "touches" the relevant cljs file

micha14:09:36

@lukemorton: @juhoteperi: it could also create a clj file with multimethod dispatch that the macro will use to emit cljs

micha14:09:03

this way you can have a known clj macro namespace for use in your cljs app

micha14:09:23

i.e. macro calls multimethod

micha14:09:39

multimethod dispatch implementation provided by your md task

micha14:09:07

this is what i'm planning to do for my own zerkdown library

pandeiro14:09:28

How can I "unroll" a map into a task invocation with keyword args?

pandeiro14:09:59

e.g. {:a true 😛 false} -> (some-task :a true :b false)

pandeiro14:09:14

heh, without the smiley

pandeiro14:09:33

I'm assuming (apply some-task {:a true :b false}) doesn't work for some reason... i'll try that

pandeiro15:09:29

yeah i can't get apply to work this way:

clojure.lang.ExceptionInfo: Assert failed: cli options must be unique
                            (apply distinct? opts)

martinklepsch15:09:20

pandeiro: how about just putting (task-options! some-task {:a true :b false}) before the task?

martinklepsch15:09:53

I don’t see why apply would not work but I guess this would at least get the job done

pandeiro15:09:28

@martinklepsch: ok, hadn't thought of that... but would that task-options! need to be manually reset after the task?

pandeiro15:09:46

like, what is the scoping for task-options! ?

martinklepsch15:09:13

it’s side effectful so if you want to go the the previous state you have to do that yourself

pandeiro15:09:40

ok, is there someway i can store the previous state for executing the reset! afterwards?

pandeiro15:09:41

@martinklepsch: thanks for the idea but if (apply ...) should work maybe I need to investigate that better; much less complex solution

martinklepsch15:09:04

@pandeiro: why do you need this kind of behaviour? (just curious)

pandeiro15:09:33

i want to be able to pass in options to a task that composes several subtasks

pandeiro15:09:03

e.g., I don't want to have to write two compile-assets-dev and compile-assets-prod tasks

pandeiro15:09:31

I want to be able to do (compile-assets :cljs {:optimizations :advanced})

pandeiro15:09:49

is there a better way to do that?

micha15:09:47

how is that different from (cljs :optimizations :advanced)?

pandeiro15:09:10

compile-assets is composing several other tasks as well

pandeiro15:09:27

writing some env stuff, minifying html, compiling less, etc

pandeiro15:09:55

it also merge-env's specific src dirs

micha15:09:50

i have tasks like this

micha15:09:53

i can paste one

pandeiro15:09:03

awesome @micha, thanks

micha15:09:10

it's a little hairy but i think it's the right level of abstraction for this job

micha15:09:35

like we need to be able to just go in there and change things without thinking about abstractions

micha15:09:46

those tasks are very linear and impreative

micha15:09:51

*imperative

micha15:09:02

we add and remove options all the time

colin.yates16:09:51

hi all - is there an example for boot + immutant + cljs + hoplon? If not, no worries simple_smile

colin.yates16:09:48

actually, just an example with immutant and cljs would be great - I can’t see how to replace the standard jetty based (serve) task with immutant...

pandeiro16:09:19

@colin.yates: if you figure it out and it is something that could be added to boot-http (a la :httpkit true), i'm happy to include it there

pandeiro16:09:36

i don't know enough about immutant to know if that is even possible as such

colin.yates16:09:07

me neither - I will have a look but it won’t be for a while unfortunately (time pressure, lein works etc. you know - the usual excuses :-)).

micha16:09:27

i think the latest immutant versions are a lot less integrated with leiningen

pandeiro16:09:29

@micha: so the difference between your example and what i'm thinking is that in yours, we need to know in advance which options (e.g., :optimizations the caller of the task may want to set/override

micha16:09:42

@pandeiro: yes, that's true

micha16:09:54

it was a good level of abstraction for us though

pandeiro16:09:55

but there may be other options that i want to be able to pass through

micha16:09:07

because it's a place we want to just type and go

micha16:09:12

without designing anything

pandeiro16:09:16

yeah i thought about doing it similarly but i would prefer something more generic if possible

micha16:09:17

that's the scripting part

pandeiro16:09:24

because i have things like closure-defines, etc

micha16:09:44

yeah you can see we don't use hardly any task options

pandeiro16:09:02

just to clarify -- (apply some-task some-task-opts-map) is not supposed to work, right?

micha16:09:07

i'd use the environment instead of closure defines

micha16:09:15

that's what we do there

micha16:09:30

yeah they're like transducers

micha16:09:36

stateful transducers

micha16:09:44

they have the same sorts of caveats

pandeiro16:09:44

yeah the thing with environment is that the only way i know how to do it involves writing to a special file that gets picked up in the fileset

micha16:09:00

we use environment variables

pandeiro16:09:06

what if i want to be able to change that while another task pipeline is running?

pandeiro16:09:23

like say for instance i have my dev task up and running a server and compilation...

micha16:09:26

we do it, that's what the calls to alter-var-root are doing

pandeiro16:09:31

and i want to run tests from the REPL

pandeiro16:09:44

but i want those tests to hit a different db than the server is using

micha16:09:51

oh, i dunno

micha16:09:00

you would add a new env var for that then

pandeiro16:09:13

if i set db via some env mechanism that writes to a file, won't the server also pick that up?

micha16:09:16

the test db would be defined in a different environment variable

pandeiro16:09:33

two different env vars, that is possible

pandeiro16:09:43

kindy ugly but ... simple_smile

micha16:09:51

i don't think it's ugly

micha16:09:56

i think it's the opposite

micha16:09:11

ahha excellent

micha16:09:35

i think the 12-factor app guidelines are pretty good

pandeiro16:09:44

yeah i agree

micha16:09:44

the best thing i've seen anyway

micha16:09:56

this dovetails with that very naturally

micha16:09:09

like for dev i have a file on my machine that's encrypted

micha16:09:14

it looks like

pandeiro16:09:20

problem is i would have 5 different vars for a db connection

pandeiro16:09:31

so multiplying that by 2... 😕

micha16:09:45

export ADZERK_API_KEY=XXXX
export ADZERK_UI_DB_HOST=YYY
export ...

micha16:09:59

i encrypt this with pgp

pandeiro16:09:18

why encrypt?

micha16:09:22

then when i build the app and whatnot i just do eval $(gpg -d envfile.asc)

micha16:09:30

because it contains passwords and things

micha16:09:42

i don't want those to exist in unencrypted form anywhere

pandeiro16:09:54

you have to type a passphrase?

micha16:09:06

only once of course

micha16:09:21

i do the same thing on the server

micha16:09:39

if i deploy to beanstalk i can set the env vars on the environment in one place

micha16:09:43

during provisioning

micha16:09:58

and if i change them beanstalk will restart my app servers automatically

pandeiro16:09:10

that is neat

micha16:09:13

this also works with pretty much any other way you might deploy

micha16:09:20

tomcat servlet container etc

micha16:09:39

we keep the environment configuration in the environment

micha16:09:43

not in the application

pandeiro17:09:09

how do you run service tests that need to use a real db connection while in dev?

micha17:09:08

you mean like for CI?

pandeiro17:09:51

sure, or locally in your dev machine?

pandeiro17:09:03

ideally, both

micha17:09:51

i'd have to see the setup

micha17:09:09

we use the same db connection for the dev build as for tests

pandeiro17:09:48

hmm... my tests mutate the db obviously so it would warp existing state in dev

pandeiro17:09:29

so in leiningen i used the special :test profile + lein-env to ensure that the tests ran against apptest db instead of regular app db

pandeiro17:09:04

meaning i couldn't run tests safely from the repl... but i could do lein test myapp.service.tests and it would use the right test db

micha17:09:08

tests i imaine you could just hardcode in

micha17:09:20

it's not like you're going to deploy tests to many places

micha17:09:26

unles you have a complex CI situation

pandeiro17:09:35

no, not complex

pandeiro17:09:08

my problem is that i (def db-conn ...) in a namespace that boot loads

pandeiro17:09:21

and it looks for env vars to determine the connection params

pandeiro17:09:59

so it would have to be something special like (or (env :test-db-host) (env :db-host)) ...

pandeiro17:09:18

i guess that solution works though...

pandeiro17:09:28

and could even allow for running tests from the repl

pandeiro17:09:40

if the tests task wrote those env vars in itself

pandeiro17:09:12

i noticed that my namespaces are loaded separately for every task in the pipeline -- that's the expected behavior, right?

pandeiro17:09:59

and if the task executes in a pod, that loaded ns is separate from other instances of the same loaded ns

micha17:09:06

that's right yes

pandeiro17:09:25

whereas if the task doesn't use a pod, the last loading of the ns 'wins' ?

micha17:09:40

the first wins

pandeiro17:09:50

the first wins?

micha17:09:06

require finds the .clj source file in the class path and compiles it

pandeiro17:09:14

ok so i would want to run the tests in a pod ...

micha17:09:14

then adds the namespace to a global map

micha17:09:23

if you call require again it gets it from the map

pandeiro17:09:29

ah ok sure, so require is sort of idempotent that way

pandeiro17:09:36

unless it has :reload ?

pandeiro17:09:17

cool... cool... so if i run tests in a pod, when the db.clj gets required, it sees the test db env vars and makes a separate connection based on those -- is my logic sound?

micha17:09:57

i think so, but it seems tricky

micha17:09:22

why not just have the tests look in their own env var for their configuration?

micha17:09:33

since this is a bona fide orthogonal concern

micha17:09:36

separate it out

pandeiro17:09:47

tests know nothing about the db

pandeiro17:09:52

they just hit the API

micha17:09:10

i'm confused now simple_smile

pandeiro17:09:04

(let [result (-> (session app) (get :some-endpoint))]
  (is (= (:status result) 200)))

pandeiro17:09:43

that's all the tests are, they don't know which db the API is hitting

pandeiro17:09:46

maybe one of the reasons i want it to work this way is that i'm using yesql and using (defqueries "queries.sql" {:connection db-conn}) so I don't need to pass the db conn into every query function

pandeiro17:09:31

it ends up meaning the API can just do (db/get-foo-by-id {:id foo-id}) without having to specify the connection

micha17:09:36

so you have 2 api servers?

pandeiro17:09:41

no just one

micha17:09:42

one for the dev build and one for tests?

juhoteperi17:09:07

Ugh I wonder why yesql moved backwards and started using global state

pandeiro17:09:09

in effect i would be running the tests in a pod so they'd hit a different version of the API right?

pandeiro17:09:23

@juhoteperi: is this a terrible anti-pattern i'm embracing?

pandeiro17:09:29

i am starting to worry about that

juhoteperi17:09:41

All sane libraries move from global state to passing connection as parameter

juhoteperi17:09:51

So you can manage the state e.g. with Component

pandeiro17:09:11

yeah like i said above, this is like the exact problem component exists for, i think

pandeiro17:09:24

I just dislike all the records and boilerplate...

juhoteperi17:09:28

Though I don't see this {:connection ...} example on yesql readme anymore?

micha17:09:43

i haven't found component to be helpful for how i work

pandeiro17:09:50

make sure you're in the 0.5.x yesql branch

micha17:09:07

i'd have 2 api servers on different ports i think

micha17:09:10

in your situation

juhoteperi17:09:19

Oh right, they still haven't released that

pandeiro17:09:37

micha: interesting solution... i'm actually using peridot so it's not really happening over HTTP

micha17:09:45

i use boot tasks the way people use component i think

pandeiro17:09:47

it's just calling the handler functions

pandeiro17:09:19

@micha Yeah I think there's some overlap for sure... doing mostly frontend stuff, i don't see component's usefulness for most of my work

micha17:09:31

i separate out all the orthogonal concerns of configuring the application into env vars

juhoteperi17:09:46

Regardless of whether one uses Component, I think it's best not force global state in libraries and let the apps decide how they manage it

micha17:09:47

i can compute other env vars in tasks

micha17:09:01

yeah that's true

micha17:09:14

but i've found that applications want the opposite

micha17:09:35

an application is basically a collection of global definitions

pandeiro17:09:38

or I suppose my tests could just use (with-redefs ...) simple_smile

micha17:09:47

i mean global state

micha17:09:16

i would probably start a separate instance of whatever it is for the tests

micha17:09:46

i don't ever have case analysis in my code to detect test environment

pandeiro17:09:58

@micha: yeah i think that is essentially what i'd be doing by loading the app (but not starting the server, to avoid port conflict) in a separate pod, right?

pandeiro17:09:04

i think pod + with-redefs may give me all i need, really... it's fairly uglegant, even simple_smile

pandeiro17:09:30

definitely not robust and super configurable, but i'm ok with that

lukemorton17:09:14

Can I see the underlying directory name of things returned from boot.core/input-dirs ?

micha17:09:07

@lukemorton: sure, those contain that info

lukemorton17:09:36

ah, I printed out the dir and couldn't see the directory name I was looking for

micha17:09:28

the fileset only deals with files and directories owned and managed by boot itself

micha17:09:48

it will never know anything about your source files or things in the target dir etc.

micha18:09:24

this is so that you can add any file to the fileset you like, or remove files from the fileset etc

micha18:09:31

without deleting your project source files simple_smile

lukemorton18:09:41

yeah I just wanted to know if files inside the source directory "markdown" had changed

lukemorton18:09:47

I've ended up matching by filename instead

micha18:09:53

yep that's the best way

lukemorton18:09:55

(b/deftask markdown
  "Touch markdown.clj if .md files change"
  []
  (let [prev (atom nil)]
    (fn middleware [next-handler]
      (fn handler [fileset]
        (let [changed (if (nil? @prev) fileset (b/fileset-diff @prev fileset))
              changed-paths (map #(:path %) (b/input-files changed))
              md-paths (filter #(.endsWith % ".md") changed-paths)]
          (println "Markdown changed:" md-paths)
          (reset! prev fileset)
          (next-handler fileset))))))

micha18:09:58

by extension really

micha18:09:05

you should make your own file extension

micha18:09:15

like maybe .cljs.md or something

micha18:09:20

you can then dispatch on that

micha18:09:45

this is a pretty foolproof way to tell boot what your intentions are

micha18:09:58

and eliminates 90% of the configuration boilerplate

micha18:09:17

if you have files that are used for a new purpose, make it official, give them their own extension

micha18:09:23

then all the tooling can configure itself

lukemorton18:09:38

Just to clarify, is the task I've written the right way to go about detecting if any markdown files have changed?

micha18:09:40

you don't need the (if (nil @prev) ...) test

micha18:09:53

(fileset-diff nil fileset) => fileset

micha18:09:26

sure that looks good

lukemorton18:09:15

seems like this idiom could be wrapped up in a helper function, not too hard to implement but maybe fairly common use case?

micha18:09:37

that's equivalent to what you have

micha18:09:15

oops there is a bug

lukemorton18:09:16

ah yes I've read about with-pre-wrap

lukemorton18:09:22

thanks for this simple_smile

micha18:09:23

one sec i update

micha18:09:02

ok there we go

micha18:09:21

each of the things in md-tmpfiles will be a TmpFile record

micha18:09:34

you can use b/tmp-path and b/tmp-file on them

micha18:09:49

tmp-path gives you the relative path in the fileset

micha18:09:09

tmp-file gives you a java.io.File object you can use to read the file etc

micha18:09:42

basically what you had was totally fine, but you asked about helpers

micha18:09:19

there are many helpers in the boot.core namespace

lukemorton18:09:59

ah I see, yeah thanks for the insight

lukemorton18:09:42

My solution to detecting markdown file changed and touching my markdown macro file to trigger live reload: https://github.com/drpheltright/big-time.cljs/blob/master/src/big_time/tasks.clj

lukemorton18:09:58

I know you said something about multi methods earlier @micha but I don't understand them enough to run with that solution. If you have any resources that could help me see what you mean that would be good simple_smile

micha18:09:24

@lukemorton: great name, i love it

micha18:09:28

looks great

micha19:09:45

@lukemorton: are you interested in pull requests?

lukemorton19:09:19

yeah sure! whether I merge them or not... my discretion 😛

micha19:09:29

naturally!

bsima22:09:26

if boot pods are just normal jvm, then can't i manage elasticsearch inside a boot pod? that would make things so much easier

micha22:09:48

@bsima: i'm not that familiar with elasticsearch but i would imagine you could

micha22:09:13

i've used pods to manage all kinds of hairy java dependencies

micha22:09:21

like jruby, jython, things like that

bsima22:09:15

yea i'll give it a try tomorrow

bsima22:09:29

boot-elasticsearch task is in the future, perhaps

micha22:09:38

very nice

micha22:09:37

do you happen to know of a good resource that explains all the tradeoffs involved with the various web server configurations? i.e. interceptors, thread pools, http-kit, jetty etc

micha23:09:23

i'm working on a web service that is IO intensive and will wait for IO a lot

micha23:09:00

want to be pretty confident i can provision it corerctly so it won't fall over under load

micha23:09:23

aleph also looks pretty sweet

micha23:09:17

the thing i really need to understand is how the different architectures behave under sustained load

micha23:09:38

like where things start to back up, how to monitor it, how to get it to autoscale, etc

bsima23:09:15

sounds super interesting, and as much as id like to talk about that, i need some sleep. its 2am here simple_smile

bsima23:09:33

i'll let you know if i find anything tho

micha23:09:36

haha sleep, is that still a thing?