Fork me on GitHub
#boot
<
2016-12-02
>
grounded_sage04:12:05

I'm a little stuck with my project which is essentially generating a a static website / web app hybrid which can be hosted entirely on a CDN. When I generate pages should I put them into a generated folder straight away or should I be putting them into a folder similar to what this function is doing (garden :styles-var 'vbn.styles/screen :output-to "css/garden.css") or is there some more trickier stuff I should be doing with filesets?

flyboarder04:12:32

@grounded_sage generally you would put things in a (tmp-dir!) then add that to the fileset

grounded_sage04:12:32

@flyboarder ok so I would put it into a (tmp-dir!) then serve it from there during dev and drop them into a target folder when I run build.

martinklepsch11:12:15

@grounded_sage most people don't serve from one specific temp dir but instead from the classpath (or a subset like public/) which can consist of multiple directories

vikeri13:12:31

Any classic gotchas when macros are not recompiling on change when using boot-cljs?

erwin13:12:06

I am trying to find a way in boot to write out files with 644 permission instead of 600, is this solvable in boot? My current strategy would be changing all the files in output dir of target with a shell command, is there a better way?

martinklepsch13:12:53

@erwin https://github.com/boot-clj/boot/issues/189 this might be of interest β€” I think a shell command at the end of the pipeline after putting it somewhere using target is probably the path of least resistence

erwin13:12:37

Ah thanks, I will create a task target-chmodded if only for the name

martinklepsch13:12:10

@erwin haha πŸ˜„ you can use target and put another task behind it that just does the chmodding stuff in the target directory

erwin13:12:33

mm, maybe better yes πŸ™‚

nicolas13:12:11

Hello, I am building an uberjar just fine except it is missing my (io/file (io/resources config.edn)) file. It works in dev, and after building (aot pom uber jar) my config file ends up in target/config.edn right next to the target/my-uberjar.jar. What am I missing?

martinklepsch13:12:45

@erwin or you just make it a bash script πŸ™‚

erwin13:12:16

yes, but then I have a bash script that calls boot and I have this problem (thanks to Docker) more than once and it gets anoying to solve everything with another bash script ...

martinklepsch13:12:09

Not sure I understand... I was more thinking along the lines of boot build && ./bin/fix-permissions

erwin13:12:33

😁 oh yes that is easier ...

erwin14:12:05

but still, my colleagues will end up saying: "see, leiningen is better", cannot let that happen

nicolas14:12:28

(trying to rm -rf target/ to see if I had stale artefacts)

erwin14:12:00

I use (comp (aot) (uber) (pom) (jar :main 'clojure.main) (sift :include [#"project.jar"]) (target))) which works ...

nicolas14:12:53

So the order is slightly different, and I don't use sift. Will try that thanks. What is it for?

martinklepsch14:12:16

@erwin leiningen probably wouldn't be of help with that chmod stuff either (but not sure)

erwin14:12:45

@martinklepsch only thing it helps with is that default is 644, so no issue

martinklepsch14:12:46

@nicolas the contents of the target directory are cleared before the fileset is written if I'm not mistaken

erwin14:12:24

sift is for not writing everything to the target directory

nicolas14:12:27

that would make sense. I know I should not rely on previous files anyway πŸ˜‰

nicolas14:12:21

oh I see. (for sift). Not sure if that would make a difference then, besides not "polluting" my target dir.

nicolas14:12:33

(will try anyway)

erwin14:12:42

indeed, that will not help I think

nicolas14:12:28

(tried cleaning target, still fails: Exception in thread "main" java.lang.IllegalArgumentException: Not a file: jar:file:/Users/.../target/clojure-backend.jar!/config.edn) trying your suggestion now erwin

nicolas14:12:29

I do set :resource-paths #{ "resources/"}, is that the right thingβ„’ ?

nicolas14:12:07

(just trying to make guesses while aot runs πŸ˜‰ )

erwin14:12:15

I have without "/"

erwin14:12:57

I am trying to clean up my own mess, can tryout a small project after that if you still run into problems

nicolas14:12:32

sift worked as intended - so still the same problem. Trying "resources" now

nicolas14:12:58

That's very kind of you

nicolas14:12:37

(it is not exactly a small project now - that's why I'd like to understand what happens)

nicolas14:12:48

Is it because of the file type? .edn?

erwin14:12:20

does the output of boot uber --help help?

nicolas14:12:27

...ok same result without / in "resources"

nicolas14:12:36

Not really. I saw that it includes everything by default

nicolas14:12:48

at least that's how I understood it

nicolas14:12:46

(that's annoying - everything else is there: it prints my startup message, tries to read the file and fails)

nicolas14:12:05

...and it is in my .jar !

nicolas14:12:18

So maybe I need to remove io/file

micha14:12:35

@nicolas what does jar tf target/project.jar |grep config.edn show?

nicolas14:12:05

config.edn it is there πŸ™‚

nicolas14:12:25

(I can see it in emacs too - there is a mode for jars)

nicolas14:12:09

(trying to read it with (io/resources config.edn) now instead of (io/file (io/resources config.edn)))

nicolas14:12:58

nope.. πŸ˜• trying io/reader now (one of these has to work ^^)

micha14:12:01

what happens if you do (in the repl) boot.user=> (io/resource "config.edn")?

micha14:12:29

you should get a java.net.URL object

nicolas14:12:32

let me try πŸ™‚ (repl starting)

erwin14:12:47

@nicolas my very small project works (also with .edn file)

nicolas14:12:02

it should work though, this is how it works in dev to read the config

nicolas14:12:12

@erwin nice !

micha14:12:21

where does it not work?

micha14:12:32

when you do java -jar target/project.jar ?

nicolas14:12:56

Yep I get aURL: #object[java.net.URL 0x5549c970 "file:/Users/nha/.boot/cache/tmp/.../j97/fwjp2k/config.edn"]

nicolas14:12:27

when I do java -jar target/clojure-backend.jar

nicolas14:12:00

It goes through my -main, prints "i will read the file" and then dies with Exception in thread "main" java.lang.IllegalArgumentException: Not a file: jar:file:/Users/.../target/clojure-backend.jar!/config.edn

micha14:12:16

can you paste that piec of code please?

nicolas14:12:44

Sure πŸ™‚ which one though? the file reading part?

micha14:12:55

looks like you're probably doing (io/file (io/resource ...

micha14:12:01

somewhere

micha14:12:10

ah sweet!

micha14:12:16

problem solved then πŸ™‚

nicolas14:12:33

Ooh ? how ? πŸ˜„

micha14:12:34

io/resource returns a java.net.URL object (or nil)

micha14:12:56

io/file can not coerce a URL into a file

micha14:12:11

if you want to read the edn you can do this:

micha14:12:37

(def config (read-string (slurp (io/resource "config.edn"))))

micha14:12:46

no io/file involved

micha14:12:54

because a jar entry is not a file

nicolas14:12:04

Oh ok. I am surprised that iw worked in the repl

nicolas14:12:10

Ok makes sense for the file πŸ˜„

micha14:12:18

it works in the repl because it is a file

nicolas14:12:31

Yeah get it πŸ˜„ Thanks thanks thanks!

micha14:12:33

the config.edn in the repl is not a jar entry

micha14:12:52

it's an actual file on the claspath

micha14:12:24

but the way i pasted above will always work

micha14:12:26

in dev and prod

nicolas14:12:36

Great πŸ˜„ thanks a ton

micha14:12:42

basically you never want to assume that something on the classpath is a file

jfntn15:12:44

Hi folks, we have a big mono repo for our backend, frontend and backend services. We’re starting to run into dependency hell, and I’m evaluating migrating to boot as an alternative to using something like https://github.com/jcrossley3/lein-modules I was wondering if there are any examples of something like this, especially setting up pods?

micha15:12:32

@jfntn hi! can you describe the high level goals you have?

jfntn15:12:45

@micha sure, basically we need to define sub-projects that will share a portion of the repo but have isolated dependencies

micha15:12:50

@jfntn so you have separate jars for each module?

jfntn15:12:01

I’d guess

micha15:12:02

and they depend on each other?

jfntn15:12:50

they’d depend on a small number of core files, the conflicts are at the edges in the different targets

jfntn15:12:00

More precisely we just ran into an issue where we want to run a load-testing process that uses aleph, but we also use datomic and they conflict over netty with 2-way breakage depending on which version you use

micha15:12:11

yeah pods solve that problem completely

micha15:12:42

especially when the conflict is a build or testing component (ie. not part of the production runtime)

jfntn15:12:54

Yup that sounds like what we want

micha15:12:14

that's a separate concern from the multi-module configuration

micha15:12:45

in boot all tasks are expected to encapsulate their own dependencies in a pod

micha15:12:52

so they don't pollute the project

micha15:12:58

like the cljs task for instance

micha15:12:07

the cljs compiler depends on a ton of hairy dependencies

micha15:12:26

like google closure, which depends on like 3 or 4 apache ones that conflict with other things

micha15:12:31

in a large project

micha15:12:50

but the boot-cljs dependency that you add to your build has no transitive dependencies

jfntn15:12:59

Would the multi-module setup come down to properly splitting your source-tree and internal deps so that you can add sets of non-conficting sources the class-path in separate tasks that spin their own pod?

micha15:12:05

the transitive dependencies are added to its own internal pod

micha15:12:40

"multi module" means a lot of different things, so it's hard to say one optimal way to do it

micha15:12:57

generally i will try to factor the modules into separate versioned artifacts

micha15:12:22

is that feasible in your case?

jfntn15:12:22

Actually at this point we’d like to avoid that

micha15:12:36

so at the end of the day you want to have a single artifact?

jfntn15:12:00

Makes more sense to hold everything together at the repo level until our deployment situation gets more complicated

jfntn15:12:17

No we need multiple artifacts

micha15:12:48

i find it easiest to organize my project around the artifacts

jfntn15:12:56

Unless we’re able to build a big one that would take args and spin the appropriate module

micha15:12:14

so i will first decide that: what are the artifacts i want to produce

erwin15:12:58

would it be an option to let boot target task write files with a default permission, such that umask and the likes work?

micha15:12:34

@erwin the problem there is with the way hard links would work

micha15:12:53

and also permissions are somewhat platform specific

micha15:12:16

@erwin the fileset uses hard links instead of copying files for efficiency

erwin15:12:22

but 600 is so restrictive, would 644 also be an option?

micha15:12:31

a hard link can't have different permissions from the referent

erwin15:12:50

maybe with a binding or redef or something?

jfntn15:12:04

@micha one thing I don’t get is how does the repl work with multiple pods? Say I want to run a backend and a service worker, both isolated and both with their own repl. Do I start boot multiple times and kick the repl task inside their pods?

micha15:12:53

@jfntn well for example, boot always has at least 2 pods running at all times

micha15:12:18

if you do boot show --list-pods you can see them

micha15:12:43

you can start a repl server in any pod

jfntn15:12:58

ah ok that sounds good

micha15:12:00

and connect to it with a repl client via the socket it's listening on

micha15:12:11

or emacs cider, vim fireplace

micha15:12:36

@erwin can you describe your use case please?

erwin15:12:08

the usecase is Jenkins awkwardness with docker. Jenkins (with docker in pipeline) does funky stuff with the users and it is just easier to make sure everything is at least world readable.

micha15:12:08

when you say "in the pipeline" do you mean in the boot task pipeline?

micha15:12:17

or just in your general workflow?

erwin15:12:04

no, in Jenkins pipeline, scriptable Jenkins basically

micha15:12:43

i think maybe boot could allow a configurable umask-like setting, but have you tried just chmod -R prior to feeding to jenkins?

erwin15:12:04

Jenkins is a bit akward and I kinda need a workaround. Boot is not really the problem, but would be nice to have a bit of flexibility.

martinklepsch15:12:36

@erwin did the boot build && ./bin/fix-permissions approach not work out?

erwin15:12:13

yes that works

micha15:12:09

the issue with 644 permissions is the boot cache

erwin15:12:13

maybe I should just let it be, but I don't like needing different scripts just for Jenkins

micha15:12:28

you don't want boot secretly exposing your maybe confidential things

erwin15:12:43

and fixing Jenkins is ... difficult πŸ˜‰

micha15:12:23

personally i like solution that @martinklepsch proposes

micha15:12:55

manipulating files in the shell is nice and simple and straightforward

micha15:12:22

i kind of prefer that to an obscure configuration parameter in boot

erwin15:12:05

ok πŸ™‚

micha15:12:00

@erwin the change would be pretty simple to make in boot, if you want to make a PR we can consider it

micha15:12:11

i'm not opposed to having a setting for that

micha15:12:34

i don't think 644 should be the default though, because of the cache security issue

micha15:12:58

@erwin would an option to the target task work for you?

erwin15:12:09

I completely agree with not having a default of 644

micha15:12:10

to have the target task set permissions?

erwin15:12:18

yes that would also work, I tried one just jet and Make did not like it (not sure why), as Make did like the && chmod -R a+r target

erwin15:12:03

I will try this over the weekend and if it does work I can make a pull request, but I have the feeling I just have obscure requirements ...

micha15:12:12

adding the option to the target task seems nice to me

micha15:12:28

you can look at it and easily know what it is supposed to do

erwin15:12:35

ok, I will make a pull request

micha15:12:23

the target task has an optimization to use hard links in the target dir when it can do so safely -- if you add the :mode option or whatever you'd want to not make hard links in that case

micha15:12:26

disable that optimization

micha15:12:54

because if you chmod a hard link you also chmod the thing it points to

micha15:12:01

which we probably don't want

erwin15:12:15

because there can be some task after target ?

micha15:12:52

well the thing the hard link would point to would be a file in the boot cache

erwin15:12:17

yes, but that is also true if you chmod after boot is finished?

micha15:12:31

yes that's true

micha15:12:41

i hadn't thought of that

micha15:12:05

maybe that target optimization is ill advised

erwin15:12:28

most of the time target is the last step I would think

micha15:12:56

but the cache files are not deleted when boot exits

micha15:12:03

because you may need to debug something

erwin15:12:09

and you will only change the permissions of a few files

micha15:12:12

the cache is deleted the next time you run boot

micha15:12:33

so if you chmod files in the target dir and those files are links you will be chmodding files in the cache

micha15:12:50

so there will be world readable files in the cache that you don't necessarily expect or know about

micha15:12:02

and that might hang around in the cache

micha15:12:14

thereby exposing confidential info without your knowledge

erwin15:12:21

yes, maybe a midway solution that gives you the option to forcibly switch back to hardlinks if you really want it?

erwin15:12:38

for performance reasons maybe

micha15:12:47

yeah currently there is an option to not use links

micha15:12:54

but i thnk maybe we never want to use links

micha15:12:15

so yeah, i think you're right

micha15:12:28

disregard what i said about the optimization stuff πŸ™‚

dvorme17:12:54

@micha Is the boot cache a random dir under $TEMP on a per-run basis? E.g.: is there ever a chance that concurrent Boot builds’ caches might collide?

dvorme17:12:01

(We currently orchestrate multiple dependent lein builds via make -j8 and are migrating to Boot.)

alandipert17:12:07

it's $BOOT_HOME/.cache, which befaults to ~/.boot/cache... i don't think there's collision potential tho

hlship18:12:02

Any plans for another boot release soon?

onetom18:12:12

πŸ™‚ i was just about to ask that too

micha19:12:46

this weekend, for sure

juhoteperi20:12:08

@micha This change could be a start for error handling improvements: https://github.com/boot-clj/boot/pull/532

juhoteperi20:12:10

Other PRs also look good to me (with exception of memory leak one, which I guess is still open)