Fork me on GitHub
#boot
<
2016-10-21
>
anmonteiro07:10:59

Right, I confused things. I wanted to find artifacts of ClojureScript compilation, so had to resort to by-re

martinklepsch10:10:23

@anmonteiro I think in theory cljs compilation artifacts should be available on the classpath as well https://github.com/adzerk-oss/boot-cljs/blob/master/src/adzerk/boot_cljs.clj#L238

anmonteiro10:10:58

hrm weird, didn’t work for me but could probably be unrelated

anmonteiro10:10:31

I’ve got it working now anyway, that’s what matters 🙂

anmonteiro12:10:46

can anyone help me with a Windows issue?

anmonteiro12:10:12

trying to run a command via boot.util/dosh but it doesn’t work

anmonteiro12:10:49

clojure.lang.ExceptionInfo: Cannot run program "yarn": CreateProcess error=2, The system cannot find the file specified
    data: {:file
           "C:\\Users\\IEUser\\AppData\\Local\\Temp\\boot.user4802722906876953601.clj",
           :line 43}
       java.io.IOException: Cannot run program "yarn": CreateProcess error=2, The system cannot find the file specified
       java.io.IOException: CreateProcess error=2, The system cannot find the file specified
                                ...
     boot.from.me.raynes.conch/proc                         conch.clj:   40
                                ...
    clojure.core/apply/invokeStatic                          core.clj:  657
                 clojure.core/apply                          core.clj:  652
                       boot.util/sh                          util.clj:  389
                                ...
    clojure.core/apply/invokeStatic                          core.clj:  657
                 clojure.core/apply                          core.clj:  652
                     boot.util/dosh                          util.clj:  400
                                ...
         boot.user/eval845/fn/fn/fn  boot.user4802722906876953601.clj:   19
                boot.core/run-tasks                          core.clj:  938
                  boot.core/boot/fn                          core.clj:  948
clojure.core/binding-conveyor-fn/fn                          core.clj: 2020
                                ...

anmonteiro12:10:26

yarn is definitely there, also error with cp for example

alandipert13:10:54

what does the call to dosh look like?

anmonteiro13:10:26

@alandipert (dosh "yarn" "install")

anmonteiro13:10:30

works on linux & mac

alandipert13:10:44

maybe try fully-qualifying the path to yarn?

anmonteiro13:10:48

this also fails:

boot.user=> (require ’[clojure.java.shell :as sh])
nil
boot.user=> (sh/sh "cp" "build.boot" "x.boot")

IOException CreateProcess error=2, The system cannot find the file specified  java.lang.ProcessImpl.create (ProcessImp
java:-2)

anmonteiro13:10:29

OK progress: (u/sh "cmd" "/c" "copy" "build.boot" "x.boot")

mitchelkuijpers14:10:37

Just out of curiosity how do people here use boot-cljs, boot-reload and boot-cljs-repl with multiple cljs builds? It seems I need to restart boot to switch to a certain build and then the repl. I see a lot of tickets in boot reload with would-be-fixed-with-figwheel-client and I checked out boot-figwheel but that seems to just write to a target folder which seems like a big hack

mitchelkuijpers14:10:55

Oh I see @richiardiandrea is doing some very good work on this part

dominicm14:10:21

@mitchelkuijpers I think you're supposed to create a .cljs.edn per build?

mitchelkuijpers15:10:15

@dominicm Yeah I know how boot-cljs works we have 3 builds, but the problem is in dev recompiling only one cljs file and switching the cljs-repl

dominicm15:10:01

Ah, switching between repls for each build is the problem?

mitchelkuijpers15:10:54

Which figwheel solves very elegantly

mitchelkuijpers15:10:03

And reload always recompiles all 3 builds

dominicm15:10:15

@mitchelkuijpers how does it solve it, out of interest?

mitchelkuijpers15:10:12

in the figwheel cljs-repl you can do (switch-to-build :main)

borkdude16:10:34

@mitchelkuijpers I’m actually just finding out how to use devcards next to a ‘normal’ cljs build

mitchelkuijpers16:10:48

If you need help let me know @borkdude

borkdude16:10:17

well, the road I’m going down now is to make a separate build for devcards, because I’m not sure how to merge it within the normal build

borkdude16:10:59

is there a shorter way to write this:

(fn [next-task]
     (fn [fileset]
       (set-env!
        :source-paths #(conj % "src-cljs-devcards"))
       (next-task fileset)))

richiardiandrea16:10:27

@mitchelkuijpers I am trying yes, I'll keep the chan posted ;)

anmonteiro17:10:48

@borkdude there is with-pass-thru

anmonteiro17:10:00

(with-pass-thru _
  (set-env! ...))

borkdude17:10:12

@anmonteiro I thought I read about that one, but where is that documented? Couldn’t find it

anmonteiro17:10:19

I don’t know 🙂

anmonteiro17:10:33

just laying down what I’ve been absorbing over time

anmonteiro17:10:10

FWIW, this normally works when setting env for me:

(deftask foo []
  (set-env! …)
  identity)

anmonteiro17:10:58

though now that I think about it it’s probably evaluated when composing tasks and not when executing them

anmonteiro17:10:05

but for my purposes it has been working OK

micha17:10:27

@anmonteiro that shouldn't work, because tasks need to return middleware

micha17:10:59

you wouldn't be able to use that in a pipeline with other tasks

anmonteiro17:10:05

@micha oh right I forgot I return identity from the task 🙂

micha17:10:09

because nothing would be calling the next handler

anmonteiro17:10:16

fixing the snippet

borkdude17:10:23

I eventually just made a defn that return the piece of code I wrote above and use that as a task.. but I reverted back to one build anyway

borkdude17:10:47

I want devcards to work just with the one dev cljs build we have

micha17:10:50

(doc with-pass-thru)

micha17:10:09

that should provide good documentation

borkdude17:10:52

@micha thanks. I googled for 'clojure boot with pass through’ (I heard about it) and read this entire page, but couldn’t find it there: https://github.com/boot-clj/boot/wiki/Tasks

micha17:10:09

github is down for me

micha17:10:14

so i can't check it

anmonteiro17:10:17

there you go!

borkdude17:10:26

thanks 🙂

borkdude17:10:08

the type of null-task differs

anmonteiro17:10:18

looks OK to me

borkdude17:10:56

one is identity, other is a function that returns identity

borkdude17:10:05

or something

borkdude17:10:26

no wait. ah well, it’s weekend…

anmonteiro17:10:42

yeah, both return a function that returns its argument

borkdude17:10:38

(fn [next-task]
    (fn [fileset]
      (next-task fileset)))
==
(fn [x] x)
?

micha17:10:15

@borkdude it's confusing because a task is a middleware factory

anmonteiro17:10:24

@borkdude would it help if written like this?

(fn [next-task]
  #(next-task %))

micha17:10:32

task is a function that when applied returns middleware

micha17:10:43

it isn't the middleware itself

micha17:10:58

so (fn [& _] identity)

micha17:10:03

that's the identity task

borkdude17:10:50

thanks guys, I’m off for now

borkdude22:10:38

finally got devcards integrated with our dev build 🙂

hlship22:10:50

I’m starting to look at this. My particular application is going to be mostly about building Docker images, so I need to merge together some files from the file system and maybe some artifacts from Maven.

hlship22:10:15

I understand the middleware concept, but I’m having trouble getting started with the whole filesystem thing.

micha22:10:45

do you have a small example of what you'd like to do?

hlship22:10:53

Like, I guess I should a new tmpdir and copy files into it, commit!, then run Docker in a subshell.

hlship22:10:08

Not a small example, not public. Still exploratory.

hlship22:10:55

Ok, what would this look like. Combine a folder that contains Dockerfile and a couple of other files. Add in a Maven artifact w/ transitive dependencies. Shell out to Docker to run a docker command on that directory.

hlship22:10:22

Do this in a way that makes sense for perhaps 20 or more Docker images.

micha22:10:27

that looks like the standard boot pattern to me

micha22:10:40

if i am not misunderstanding

hlship22:10:40

Later complications: may treat many of the input files as template, want to edit them while copying them into final position.

micha22:10:05

sure that's what boot is designed for

hlship22:10:09

I’m at hour one of using boot, give or take. Just getting my feet wet.

micha22:10:32

ok so the general pattern is a sequence of transformations of the "fileset"

micha22:10:49

it's a little similar to transients maybe in clojure

micha22:10:09

where there is mutation going on in a place that is hidden from the rest of the program

micha22:10:22

and the thing that is passed around updates atomically and immutably

micha22:10:34

the fileset value is what is passed from task to task

hlship22:10:36

which is very cool

micha22:10:44

thanks 🙂

micha22:10:03

each task uses temp directories for its mutation and whatever craziness it does

hlship22:10:14

and there will be craziness

micha22:10:20

it can populate the temp dirs from tthe fileset value it receives from the task that called it

micha22:10:43

and after the craziness it can add those temp dirs to the fileset

micha22:10:52

and commit! then pass the fileset on to the next task

micha22:10:15

since the fileset is an immutable value you can do aggressive caching

micha22:10:27

to make the copying you need to do more efficient

micha22:10:13

so suppose you have some process that expects files in the filesystem in certain places

hlship22:10:28

(bated breath)

micha22:10:28

and it expects to be able to write to some directory or directories

micha22:10:52

this can easily cause issues if you have many of these kinds of processes that all want to own the same filesystem

micha22:10:06

they would need to know all about each other so they don't step on each other's toes

micha22:10:15

in boot you would make a task for each process

micha22:10:27

in the task you would first allocate a temp directory or two

micha22:10:30

like this

micha22:10:50

(deftask foo
  []
  (let [tmp (boot.core/tmp-dir!)]
    ...

micha22:10:10

that temp dir's lifetime is one boot pipeline

micha22:10:36

that is the place where you will have the crazy process dump the artifacts it emits

micha22:10:00

you can also make temp dirs for the inputs to the process

micha22:10:16

so then to set things up and do the work:

micha22:10:52

(deftask foo []
  (let [tmp (tmp-dir!)]
    (with-pre-wrap [fileset]
      (doit :output-dir (.getPath tmp))
      (-> fileset (add-resource tmp) commit!))))

micha22:10:10

where doit does the actual work

micha22:10:36

with-pre-wrap is a macro that emits the middleware

micha22:10:48

just sugar

hlship22:10:06

So maybe I want to build a pipeline where I do a couple of steps: move the simple files to the output dir, move the editted files to the output dir, move the Maven artifacts to the output dir, then run Docker on that output dir … what does that look like?

micha22:10:13

how do the maven artifacts relate to docker?

hlship22:10:01

Building a Docker image that runs a Clojure application; need the dependencies.

micha22:10:19

i usually have boot run in the container

micha22:10:32

like in my dockerfile i call boot ...

micha22:10:39

that will populate the .m2 in the container

hlship22:10:44

So our primary build will install them locally or to a remote repo, and now building the tooling that grabs them and builds the image around them.

borkdude22:10:19

I’ve had this situation where it was desirable to have the dependencies in the image, because on the cloud CI it would take very long to download them every time

hlship22:10:23

I’d like to avoid that option and do the build first. We tend to keep versions stable and note our Git tag or hash, want reproducable builds.

micha22:10:53

i mean just run boot in the container at build time

micha22:10:59

not when the container runs

hlship22:10:00

Yes, when did this kind of thing on AWS it was better to build it into the Docker image as much as possible; though less urgent when we were running a private Docker hub inside AWS as well.

borkdude22:10:16

and that CI was running based on Docker images

borkdude22:10:41

(Codeship Docker platform)

micha22:10:35

that line is there only to download dependencies into the container

micha22:10:53

not sure if that is what you mean?

hlship22:10:53

Even if it isn’t Maven artifacts, we are putting together files from a few different places and passing them though a bit of editting in flight.

micha22:10:40

ok so one other example that might help is how boot-hoplon works

micha22:10:18

it consumes files with the .hl extension and emits corresponding .cljs and .html files

hlship22:10:14

I kind of imagine (deftask create-query-service-image [] (comp (docker-template :dir “query-service”) (with-artifacts :artifacts [[io.aviso/rook “0.2.1”]]) (build-docker-image “com.walmartlabs/query-service:1.4.7”)))

hlship22:10:22

with a couple of more stages

hlship22:10:08

Is this a meta-data kind of thing? Feels like docker-template would create a temp directory and some of the other tasks would need to know what it is to move additional files there, before build-docker-image does its magic.

micha22:10:32

creating the docker image seems identical to creating an uberjar

micha22:10:41

i mean conceptually

hlship22:10:42

And trust me, I want to avoid writing this as a Makefile!

micha22:10:17

so i would say let the build-docker-image just consume the entire fileset

micha22:10:52

then you have tasks that populate the fileset

micha22:10:58

or modify objects in the fileset

micha23:10:09

and at the end you package the result in the docker image

micha23:10:18

with your build-docker-image task

hlship23:10:46

think I’m getting there conceptually, I just don’t get what starts inside the fileset exactly

micha23:10:01

you can use fileset metadata, but i would try it first the simple way

hlship23:10:15

so my query-service folder would not be part of the fileset until docker-template executed, and it would add it to the fileset.

micha23:10:41

the fileset is like a snapshot of the filesystem

micha23:10:01

it's an immutable value that represents the state of the filesystem at an instant in time

micha23:10:45

you can make the actual filesystem state correspond to a fileset value by doing (commit! fileset)

micha23:10:09

inside the fileset is a map of path -> TmpFile record

micha23:10:34

it's similar to how git works

micha23:10:58

in git the actual information is stored in an immutable, content-addressed blob storage area

micha23:10:18

and the working set of files is constructed from the information in the tree and the blobs

micha23:10:28

this is also how the fileset works

micha23:10:44

the actual files reside in a content-addressed blob store on disk

micha23:10:01

when you call commit! boot makes a bunch of hard links to those blob files

hlship23:10:11

I’ll start playing with it and see where I get stuck.

micha23:10:30

in general you don't want to have named directories

micha23:10:35

that you pass around

hlship23:10:44

right, I see the appeal of that

micha23:10:48

the fileset allows actual directories to be anonymous

micha23:10:19

each step in the pipeline would add things to the fileset

micha23:10:26

or modify. delete, whatever

hlship23:10:55

So I could start by loading the contents of the query-service directory into the fileset

micha23:10:00

there is the sift task that has lots of options for transformations on the fileset

hlship23:10:19

Later tasks could add additional files or edit the existing ones

hlship23:10:33

Towards the end I commit!

micha23:10:40

the edit them, btw, by creating a new file and then commit!

micha23:10:13

i mean the updated version of the file is added to the fileset with the same path

micha23:10:26

it then replaces the old version of the file in the fileset

hlship23:10:27

after commit! I can get a reference to a file system directory which will have the files (as hard links) but I can run docker there.

micha23:10:46

you can, but it's not really convenient

micha23:10:02

usually you want to copy things out, do your business, then add them back

hlship23:10:22

In this case, docker will just be reading those files and not updating the file system

micha23:10:44

there are multiple directories and stuff

micha23:10:51

implementation details of the fileset and boot

hlship23:10:49

In a worst case, I can run docker and pass it a tar stream of the content to create as an image

micha23:10:35

i would start with just copying whatever you need to a temp dir and then operate on it with your docker stuff

micha23:10:54

it's not a big deal to copy, boot has ways to make that efficient

hlship23:10:25

you mean, copying it out of the fileset?

hlship23:10:52

let me see what I’ve got here. Get a mini prototype going. Thanks for the head start!

micha23:10:01

good luck 🙂