Fork me on GitHub
#boot
<
2016-08-21
>
onetom13:08:25

@darwin i just saw your dirac intellij integration document https://github.com/binaryage/dirac/blob/master/docs/integration.md you are saying: > Then I used Cursive's tools to switch REPL namespace to my current file, loaded the file, and executed the selected form in the editor. I wondering what's your workflow. Do you use boot-reload? I was trying to use Cmd-Shift-M to sync all changed files and their dependencies but it was very slow.

onetom14:08:15

and also just reloading the recompiled cljs into the browser is not enough, since most of the time i modified view code so i want a re-render without a page reload

darwin14:08:39

@onetom I don’t use boot, I’m stuck with lein (because of figwheel)

darwin14:08:44

I’m not sure if your question is really dirac-related, hot-reloading is not necessarily relevant to state of your (n)REPL environment

Chris Bidler15:08:57

so here’s a newbie question: I’m trying to muddle through taking some existing re-frame and cljs tutorials and re-do them in Boot instead of Leiningen, and I’ve gotten to the point where I have a dev that does all the things for the cljs side, serving up target on port 3000, listening for changes, and so forth

Chris Bidler15:08:35

but I want to also add a “server” for a CQRS-y app model and I can’t seem to figure out how to comp together things so that dev does both

Chris Bidler15:08:28

I have to (def p (future (boot (dev)))) and then later (cqrs/start-server) - if I add the server startup fn to dev then nothing is listening on port 3000

micha15:08:21

@chris_johnson usually youd have your server added to your ring stack

Chris Bidler15:08:53

I’m sorry, I don’t understand what you mean - it is a ring server, but are you saying there’s some built-in Boot machinery I should be leveraging to start it up?

micha15:08:34

sorry, how do you serve your assets on port 3000?

micha15:08:52

you'd have your backend serving in that same stack

Chris Bidler15:08:13

oh, I see what you mean

micha15:08:53

yeah you don't want to get too fancy with the boot part probably

micha15:08:03

it's simpler to just make your ring stack do what you want

Chris Bidler15:08:10

so in dev I have a call to pandeiro.boot-http/serve :dir target

micha15:08:17

and then you can use the serve task to hook up to it

micha15:08:36

yeah i'd probably not use the :dir option, and delegate to your ring handler

Chris Bidler15:08:37

I’d have to modify that somehow to also serve up my clj on port 8080 or whatever

micha15:08:57

also i think it's better to serve from the classpath

micha15:08:16

the target directory is in an undefined state while boot is running

micha15:08:47

ring has the wrap-resource middleware, to serve static assets from the classpath

micha15:08:54

that's what you'd have to do in production anyway

micha15:08:07

basically just set up your ring stack the way you would for production

micha15:08:14

it'll work in dev

micha15:08:52

the serve task has a repl option that might be useful

micha15:08:04

get you a repl in the right context

Chris Bidler15:08:20

I was coming at this sort of backwards because I have a dev built specifically to run a re-frame SPA in development mode, including running cljs-repl-env so I can get my browser REPL inside CIDER

onetom15:08:23

@chris_johnson do you want to serve your frontend and the backend on the same port?

Chris Bidler15:08:45

and now I want to “bolt on” a server component just to demonstrate CQRS-style architecture

onetom15:08:10

so you would need CORS setup too, then, right?

Chris Bidler15:08:14

and in fact it might not be the worst thing in the world to just have to start the server “manually”

micha15:08:15

hmm, the way i work is my dev setup is pretty much the same as my prod setup

Chris Bidler15:08:20

yeah, I have ring-cors on the server

micha15:08:26

it's just running locally and recompiling

Chris Bidler15:08:22

@micha: I definitely respect that, there is no “prod setup” here yet, I’m just noodling around with an eye toward maybe publishing a blog post about going from published examples of Reagent/Leiningen to Boot/Re-Frame

Chris Bidler15:08:37

approximately zero percent of the production cliff has been climbed for this project 😄

micha15:08:57

haha, well prod for me is just an uberjar

micha15:08:21

there isn't anything fancy going on

Chris Bidler15:08:06

so the real answer here, I think, is for me to read up more on boot-http and see if I can’t build something to comp in that does “serve this ring handler on port 8080, serve these assets on port 3000"

Chris Bidler15:08:17

instead of the received :dir target

micha15:08:43

yes you can start two servers

Chris Bidler15:08:46

though to be fair, I do also have something in my dev fn that puts things in target,

micha15:08:50

use the serve task twice in the pipeline

micha15:08:14

why not put them in the fileset instaed of the target dir?

onetom15:08:23

micha, shouldn't we recommend boot-jetty instead of pandeiro-http? or even boot-static?

micha15:08:30

writing to the target dir while boot is running is going to have unpredictable results

micha15:08:43

whoa i was not aware of boot-static

onetom15:08:04

really?! we are running it for about 2month now. no issues so far

micha15:08:18

canyou paste a link pls?

micha15:08:34

wow that looks ideal

onetom15:08:48

it's very little code, though it pulls in a lot of vertx libs, yet it's supposed to be a lot leaner than jetty

micha15:08:00

i'm sure it pulls those things into a pod

micha15:08:09

so it won't pollute the project

onetom15:08:10

sure, it uses a pod

micha15:08:24

yeah this looks great

Chris Bidler15:08:30

now I’m trying to remember where I got my current Boot workflow from

Chris Bidler15:08:53

I want to say it was a tutorial of some kind but it does many things that I am reading here are not ideal

micha15:08:42

this is going to improve my dev setup

micha15:08:59

cause i always push static assets to s3 and do the cors thing

micha15:08:34

like the index.html is s3 cloudfront, backend cqrs thing on aws

onetom15:08:38

chris_johnson there are a lot of obsolete stuff out there. things moved quite fast and there are just not enough resources to keep docs coherent

onetom15:08:21

that was on irc 🙂

micha15:08:32

haha i know, good times

onetom15:08:15

@chris_johnson one important thing which is also noted in the readme: you have to use jumblergs cors lib (https://github.com/jumblerg/ring.middleware.cors) because i had issues with https://github.com/r0man/ring-cors (at least when i was trying to use castra)

Chris Bidler15:08:17

another thing I’m doing wrong, but at least I came by that one honestly

Chris Bidler15:08:24

not reading the README is my own fault hehe

Chris Bidler16:08:09

Well alright, this dev task with boot-static gets me my SPA served up on port 3000 and my http-kit ring stack on port 8080

(deftask dev
  "Launch all the things"
  []
  (comp
   (watch)
   (reload)
   (cljs-repl-env) ; before cljs task
   (cljs)
   (serve :port 3000)
   (mount.core/start)))

Chris Bidler16:08:40

boot-static does log an NPE to the REPL a little bit after starting up, but then seems to work okay?

Chris Bidler16:08:26

and I get a browser REPL with (start-repl) as expected/desired

Chris Bidler16:08:13

thank you very much @micha and @onetom - I learned a lot from this discussion

Chris Bidler16:08:12

frustratingly I feel like the blog post I want to write should probably go back to the woodshed and be less about “moving from lein/reagent to boot/re-frame” and more focused on building a CQRS-y re-frame app in boot

mitranim16:08:29

@onetom Good timing with boot-static, I've been elbow deep in boot and boot-http code trying to debug why it doesn't stop the server. No such problem with boot-static

mitranim16:08:39

(originally got the boot-http recommendation from https://github.com/adzerk-oss/boot-cljs/wiki/Serving-files)

mitranim16:08:27

solved my headache, thanks

mitranim16:08:36

boot-static also serves files from the temporary dir, great

mitranim16:08:01

boot-jetty works just as well, any meaningful difference with boot-static?

jumblerg16:08:52

@mitranim: the interface to both boot-static and boot-jetty should be about the same, either will work, but boot-static may be a bit snappier

jumblerg16:08:17

boot-static is based on vertx, which in turn wraps netty

jumblerg16:08:45

boot-jetty wraps the traditional bloatware

jumblerg16:08:37

boot-static is a step on a more ambitious refactoring path that will integrate into a build pipeline with the not-ready-for-use boot-vertx task.

mitranim16:08:16

the readme seems fairly vague

jumblerg16:08:10

one of the differentiating ideas behind the boot-vertx task is that each time the task is invoked (typically by the boot watch task) it will load in a fresh servlet with your application code from a pod pool

mitranim16:08:53

we'll see when it's done I guess 🙂

jumblerg16:08:56

i’m also using a branch of boot-jetty locally that does this

jumblerg16:08:26

the only major feature that is missing from boot-vertx right now is the implementation of the input-stream expected by the ring request map

mitranim16:08:19

aha so it's for server code?

mitranim16:08:28

sorry I was in my cljs thought bubble

jumblerg16:08:30

i haven’t done this because i’m not sure what the right approach is yet; vertx uses its own reactive stream standard

mitranim16:08:32

sounds great

jumblerg16:08:46

yeah, boot-static works great now for cljs

jumblerg16:08:17

later on, boot-static will require the boot-vertx task once it has been completed, but for now, it is everything you need

jumblerg16:08:58

(at least for the client, which i develop as a project separate from my service)

Chris Bidler17:08:08

@jumblerg: as I was just pointed to do in discussion here a bit ago, I’ve adopted your ring.middleware.cors in my toy app but I seem to be maybe doing it wrong? This is what my handler compositon looks like:

(def handler
  (-> routes
      (wrap-transit-body {:keywords? true})
      (wrap-cors identity)
      wrap-edn-params
      wrap-params))

Chris Bidler17:08:29

so I have static assets being served by boot-static on port 3000, and a ring stack on port 8080

Chris Bidler17:08:41

when I dispatch a POST from the SPA on port 3000, I see logging on the server as if the POST went through, but in the SPA console log I get XMLHttpRequest cannot load . Origin is not allowed by Access-Control-Allow-Origin.

Chris Bidler17:08:02

I had r0man/ring-cors in place of this middleware earlier and it was working, I also earlier had #".*localhost:3000.*” in place of identity in this current call

Chris Bidler17:08:50

I guess I’m hoping that you’ll look at my handler and say “oh, nope, you have to do X instead of what you have there” hehe

Chris Bidler17:08:15

looking at the source it certainly appears that just supplying identity should cause the right headers to be sent

jumblerg17:08:55

@chris_johnson: i think you may want wrap-cors on the outside

jumblerg17:08:48

i.e. last in the -> list

jumblerg17:08:44

i also know that there’s a small issue with the way certain errors are handled in that library that may cause other errors on the server to be masked by access-control errors; i’ve been meaning to fix this for a while.

jumblerg17:08:42

here’s an example from an application i’m running locally (and in production) right now:

(defn handle [req]
  {:status  200
   :headers {"content-type" "text/plain"}
   :body    "My Service"})

(def serve
  (-> handle
    (wrap-castra {:state-fn #(q/model *db*)} 'my.service.command 'my.service.query)
    (wrap-authentication)
    (wrap-castra-session secret)
    (wrap-tenant #(first (db/q db/tenant-query %)))
    (wrap-s3 aws-access-key aws-secret-key bucket-name)
    (wrap-smtp aws-access-key aws-secret-key smtp-host smtp-from-email smtp-reply-email)
    (wrap-status "/status")
    (wrap-datomic datomic-uri)
    (wrap-rewrite #"^/service" "")
    (wrap-cors #".*localhost.*" #".*$")))

onetom17:08:49

@chris_johnson here is our ring stack:

(defn make-ring-handler []
  (-> (constantly {:body "Not a castra request"})
    (wrap-castra 'app.users 'app.docs 'app.etc)
    (wrap-castra-session (str "**********"))
    (wrap-cors
      #""
      #"http://.*\.local:8100"
      #""
      #"")))

pesterhazy17:08:28

out of curiosity, do you guys not have a db-conn parameter that you need to pass through?

pesterhazy17:08:44

or is everyone using mount now 🙂

onetom17:08:54

🙂 true, i would need that probably

micha17:08:02

i don't use any sort of component thing

onetom17:08:06

im just thinking about how to switch to mount though 😉

jumblerg17:08:06

i do, that’s what my wrap-datomic wrapper does

jumblerg17:08:12

and i don’t use mount either

pesterhazy17:08:30

@jumblerg, does wrap-datomic attach the db-conn to the request?

micha17:08:35

in my workflow boot tasks serve the same role as components

jumblerg17:08:40

yes, exactly

micha17:08:19

i use the boot.core/cleanup macro to deallocate/shutdown state established by the task

micha17:08:50

so you can start and stop things just by running the pipeline

pesterhazy17:08:52

@micha, interesting. how does a handler get access to, say, the db conn in your setup?

micha17:08:59

environment variables

onetom17:08:09

@micha: not the uri, the connection

micha17:08:35

oh it establishes the connections when the server starts

micha17:08:47

the cleanup macro stops the server, freeing them

pesterhazy17:08:13

so the connection is stored in a var/atom? (not saying that's a bad thing)

micha17:08:43

i dunno, i never do anything super fancy there, i don't need much infrastructure around it

jumblerg17:08:54

here’s the full entrypoint to my service:

;;; utils ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defn assert-prop [var]
  (or (System/getProperty var)
      (throw (Exception. (str "Required system property " var " not set")))))

;;; config options ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(def secret           (assert-prop "16_BYTE_SECRET"))
(def aws-access-key   (assert-prop "AWS_ACCESS_KEY"))
(def aws-secret-key   (assert-prop "AWS_SECRET_KEY"))
(def bucket-name      (assert-prop "BUCKET_NAME"))
(def datomic-uri      (assert-prop "DATOMIC_URI"))
(def smtp-host        (assert-prop "SMTP_HOST"))
(def smtp-from-email  (assert-prop "SMTP_FROM_EMAIL"))
(def smtp-reply-email (assert-prop "SMTP_REPLY_EMAIL"))

;;; clojure servlet api ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defn handle [req]
  {:status  200
   :headers {"content-type" "text/plain"}
   :body    "My Service"})

(def serve
  (-> handle
    (wrap-castra {:state-fn #(q/model *db*)} 'my.service.command 'my.service.query)
    (wrap-authentication)
    (wrap-castra-session secret)
    (wrap-tenant #(first (db/q db/sponsor-query %)))
    (wrap-s3 aws-access-key aws-secret-key bucket-name)
    (wrap-smtp aws-access-key aws-secret-key smtp-host smtp-from-email smtp-reply-email)
    (wrap-status "/status")
    (wrap-datomic datomic-uri)
    (wrap-rewrite #"^/service" "")
    (wrap-cors #".*localhost.*" #".*$")))

onetom17:08:22

@micha: @tolitius has these example apps for his mount library: https://github.com/tolitius/stater would be hyper-educational if you could show an equivalent solution to his neo or smsio apps using your boot task approach

onetom17:08:59

@micha but do you have a test suite which works against temporary dbs and mock services?

pesterhazy17:08:03

@micha, cool. I love hearing when people do things the simple way.

micha17:08:27

i try to avoid mocking dbs whenever possible

micha17:08:47

i avoid mocking anything really

onetom17:08:09

well, i meant (gen-sym "datomic:mem://test-") dbs

onetom17:08:24

u should mock emailing though, no? (just to give a frequent example)

micha17:08:42

when i need stuff from the database i use a test account in the production db if possible, because the database is normally not owned entirely by one system

onetom17:08:30

so when u run tests on your CI it runs against test accounts on the production db?

micha17:08:49

or a snapshot of it as a gzipped json file for instance

micha17:08:06

like just the test account

micha17:08:43

but for me the mocking is where the tests start to become a liability

pesterhazy17:08:32

this is how I mock: (when (not= :testing env) (send-email!))

micha17:08:58

to send an email you'd put something on a queue, so you can just have sa test queue that goes nowhere

onetom17:08:17

@pesterhazy so you dont test email sending? and the content of the sent email? or just test it at a unit test level or how?

pesterhazy17:08:36

ok you got me, I'm terribly sloppy with testing

pesterhazy17:08:05

I wish it wasn't so, but not enough to change my ways

onetom17:08:28

@micha so when you are running your tests you should a. initialize your global email queue to something like /dev/null b. pass the queue as a parameter to your function which happens to send email too

micha17:08:31

my dream is to test only versioned apis

micha17:08:13

i would like my tests to test only well-defined, public api functions

micha17:08:38

that way when i refactor implementation details i don't need to analyze all my tests to be confident that they're still testing the right behavior

micha17:08:56

that's the part that i can't handle, really

micha17:08:23

implementation detail tests maybe can be assertions

micha17:08:00

like the last service i wrote most of the tests are assertions now

onetom17:08:13

sure and none of my questions were suggesting anything like that

micha17:08:16

i'd have my global email queue be sqs or kinesis

micha17:08:34

you can have a kinesis stream for testing, they're not expensive

onetom17:08:55

they are still slow to create and your tests depend on an external service and network connection availability

micha18:08:15

that's true, although i'd just create one for myself

micha18:08:25

i wouldn't create one every time i run a test

micha18:08:37

but yeah i accept that i need internet to work 🙂

onetom18:08:00

then you would need to take care of clearing the queue every time for example before re-running the tests

onetom18:08:31

i had this exact same situation in the past with sqs queues and it's not obvious, neither fast to clear an sqs queue

onetom18:08:05

completely deleting and re-creating them is also not possible because they throttle how quickly can you create a queue with the same name...

micha18:08:18

well it is fast to do it, but i'd imagine the test runner that is looking at things in sqs would delete messages it has handled

onetom18:08:33

unless your test fails

onetom18:08:57

and the message is not deleted but they it will only re-appear in the queue after the whatever-timeout has expired

micha18:08:24

you could just delete it when you receive it

micha18:08:31

it's an option when you receive the message

onetom18:08:26

so ideally every external service (which has some state what you would like to inspect from your tests) should have a test variant, which makes it easy to 1. initialize a new instance of the external system preferably within the same process where your app and tests run 2. inspect its state 3. clean it up fast too

micha18:08:25

usually the external resources are things that require configuration so i don't make a unique instance for each test run

micha18:08:31

like setting permissions, etc

onetom18:08:37

unfortuantely most services we use often lack this kind of variant...

onetom18:08:10

luckily datomic is an exception with its in-memory mode

micha18:08:41

if you're in a situation where your canonical data is only touched by your code then it might make sense

micha18:08:59

but in my experience there are other systems that contribute to that data

onetom18:08:01

but for example the filesystem is something which should have a mock version, but i haven't found any in the java/clojure world which would be good

micha18:08:28

once you have interaction with other systems the mocks are imho a liability

onetom18:08:30

and obviously every aws service...

micha18:08:49

because you don't know if your mock really corresponds to the current shape of the data anymore

micha18:08:08

not just shape but internal consistency also

onetom18:08:22

well, that's why these mock services should be provided by the original service provider... not just client SDKs...

onetom18:08:37

here is a relatively good example of a filesystem mock library (for nodejs): https://github.com/tschaub/mock-fs

micha18:08:48

typed clojure + spec project seems like it could be helpful here

micha18:08:12

to unobtrusively generate tests for clojure functions

micha18:08:43

that might be resilient to refactoring

onetom18:08:30

im getting clearer and clearer on how to structure an clojure app in a test-friendly way, so probably in a few months i will have the chance to release some good example app. maybe for this xmas 🙂

onetom18:08:51

@chris_johnson btw, i've noticed you are also using mount. i might have some questions later regarding that. in the mean time, i should point out that (mount/start) is probably shouldn't be composed with those other tasks, since it probably doesn't return a boot task

Chris Bidler18:08:18

re: SQS testing, ElasticMQ has a mode that SQS clients can treat as “an SQS queue"

Chris Bidler18:08:33

and which can be, say, stood up inside your unit test @Before

Chris Bidler18:08:41

(to use a crude Java-ism)

Chris Bidler18:08:51

oh that’s a good point, I was just trying to drive the number of things to type at repl startup from 2->1 but that really should be a deftask like start-httpkit! or something that just does a side-effect server start and returns its argument, huh

Chris Bidler18:08:59

again, not the ideal of having dev look just like production, but I’m literally just building a thing to do it so I get a better handle on modern Clojure services

micha18:08:58

i have dev and prod pretty much the same just out of laziness really

Chris Bidler18:08:14

laziness is a virtue

micha18:08:16

that's why i use real sqs queues for dev etc

micha18:08:32

my build is usually pretty simple

onetom18:08:50

@chris_johnson (with-pass-thru [_] (mount/start)) is what you want if im correct

micha18:08:15

i do usually spend some time making a nice repl environment though

micha18:08:29

with a repl namespace that uses alter-var-root a lot

micha18:08:45

my idiom for my own things is all the repl functions start with !

micha18:08:04

so in the repl i get tab completion of them

onetom18:08:11

@chris_johnson or you can even just say (do (mount/start) identity), though that's not that explanatory

micha18:08:19

that's my repl setup for a queue worker service

micha18:08:23

haha what's scary?

micha18:08:37

handler there is my worker function that accepts the parsed sqs message body

onetom18:08:56

that every 2nd word is alter-var-root 🙂

micha18:08:06

that's so i don't need to type parens in the repl

micha18:08:37

i can do gymir.repl=> !next instead of gymir.repl=> (!next) etc

onetom18:08:42

> in the repl what do you mean by that? don't u use neovim as a repl?

micha18:08:52

i use it for some things

micha19:08:08

but when i'm testing a sequence of things the repl itself is handy

micha19:08:15

with readline etc

micha19:08:40

i do the vim repl by starting a standalone boot repl anyway

micha19:08:56

so i can switch from the editor to the repl client in tmux pretty easily

micha19:08:22

i tried the neovim terminal thing, but it's kind of a pain to use because of the way it deals with keybindings

onetom19:08:38

well, me too, but only in server mode and connect to it from intellij, where i have proper multiline editing and paredit and syntax highlight and help and go-to-definition and peek into source, etc...

micha19:08:02

yeah the repo functions there are using alter-var-root to avoidn needing to type lots of things

micha19:08:31

also my vim setup sometimes eats stack traces

onetom19:08:33

anyway, thanks for widening our perspectives

micha19:08:39

or i just don't know how to see them properly

onetom19:08:41

but now i shall really sleep

micha20:08:04

ah can you make an issue please?

micha20:08:07

i can update it

micha20:08:52

Given dest and a list of srcs, updates the contents of dest such
that it is the product of srcs, overlayed on top of each other. 
Files that are not in any of the srcs will be removed from dest.

This function returns an instance of boot.filesystem/FilesystemTree
reflecting the final state of dest, suitable to pass as the
:state option to the next invocation (provided of course that
files are not added to, removed from, or modified in dest in the
meantime).

The dest and srcs must be of a type that satisfies the
boot.filesystem/IToPath protocol.

The :state option can be used to provide a FileSystemTree instance
reflecting the state of dest. This eliminates the need to walk dest
to form the patch, so can be much more efficient. Of course, if the
actual contents of dest are not the same as state the result is not
well defined.

The :ignore option specifies a function of one argument, the file 
path (as a string), which returns a true value if the file is to be 
ignored.

The :link option can be specified with a true value to enable use
of hard links instead of copying files from srcs to dest.

Chris Bidler20:08:00

so thanks once again for all your help @micha, @onetom, et. al

micha20:08:18

everything working the way you want it now?

Chris Bidler20:08:39

everything except re-frame barfing and dying when I hand it a new app-state

Chris Bidler20:08:55

but that’s not boot/workflow, that’s me not understanding something about re-frame

Chris Bidler20:08:05

everything else is super lovely, yes 😄

Chris Bidler20:08:52

I will mention it here once I ever get my blog post refactored and posted

Chris Bidler20:08:31

I’m sure there’s a glut of “how to build a simple thing with modern tools” posts out there but I’ve decided that I’m going to post something twice a month about what I’m learning, and if it’s just one voice in a chorus, so be it.

micha20:08:18

yess feed the googles

micha20:08:07

examples of how people do stuff is pretty important