I'm a little stuck with my project which is essentially generating a a static website / web app hybrid which can be hosted entirely on a CDN. When I generate pages should I put them into a generated folder straight away or should I be putting them into a folder similar to what this function is doing (garden :styles-var 'vbn.styles/screen :output-to "css/garden.css") or is there some more trickier stuff I should be doing with filesets?

@grounded_sage generally you would put things in a (tmp-dir!) then add that to the fileset

@flyboarder ok so I would put it into a (tmp-dir!) then serve it from there during dev and drop them into a target folder when I run build.

yeah

@grounded_sage most people don't serve from one specific temp dir but instead from the classpath (or a subset like public/) which can consist of multiple directories

Any classic gotchas when macros are not recompiling on change when using boot-cljs?

I am trying to find a way in boot to write out files with 644 permission instead of 600, is this solvable in boot? My current strategy would be changing all the files in output dir of target with a shell command, is there a better way?

@erwin https://github.com/boot-clj/boot/issues/189 this might be of interest — I think a shell command at the end of the pipeline after putting it somewhere using target is probably the path of least resistence

Ah thanks, I will create a task target-chmodded if only for the name

@erwin haha 😄 you can use target and put another task behind it that just does the chmodding stuff in the target directory

mm, maybe better yes 🙂

Hello, I am building an uberjar just fine except it is missing my (io/file (io/resources config.edn)) file. It works in dev, and after building (aot pom uber jar) my config file ends up in target/config.edn right next to the target/my-uberjar.jar. What am I missing?

@erwin or you just make it a bash script 🙂

yes, but then I have a bash script that calls boot and I have this problem (thanks to Docker) more than once and it gets anoying to solve everything with another bash script ...

Not sure I understand... I was more thinking along the lines of boot build && ./bin/fix-permissions

😁 oh yes that is easier ...

but still, my colleagues will end up saying: "see, leiningen is better", cannot let that happen

😉

(trying to rm -rf target/ to see if I had stale artefacts)

I use (comp (aot) (uber) (pom) (jar :main 'clojure.main) (sift :include [#"project.jar"]) (target))) which works ...

So the order is slightly different, and I don't use sift. Will try that thanks. What is it for?

@erwin leiningen probably wouldn't be of help with that chmod stuff either (but not sure)

@martinklepsch only thing it helps with is that default is 644, so no issue

@nicolas the contents of the target directory are cleared before the fileset is written if I'm not mistaken

sift is for not writing everything to the target directory

that would make sense. I know I should not rely on previous files anyway 😉

oh I see. (for sift). Not sure if that would make a difference then, besides not "polluting" my target dir.

(will try anyway)

indeed, that will not help I think

(tried cleaning target, still fails: Exception in thread "main" java.lang.IllegalArgumentException: Not a file: jar:file:/Users/.../target/clojure-backend.jar!/config.edn) trying your suggestion now erwin

I do set :resource-paths #{ "resources/"}, is that the right thing™ ?

(just trying to make guesses while aot runs 😉 )

I have without "/"

I am trying to clean up my own mess, can tryout a small project after that if you still run into problems

sift worked as intended - so still the same problem. Trying "resources" now

That's very kind of you

(it is not exactly a small project now - that's why I'd like to understand what happens)

Is it because of the file type? .edn?

does the output of boot uber --help help?

...ok same result without / in "resources"

Not really. I saw that it includes everything by default

at least that's how I understood it

(that's annoying - everything else is there: it prints my startup message, tries to read the file and fails)

...and it is in my .jar !

So maybe I need to remove io/file

@nicolas what does jar tf target/project.jar |grep config.edn show?

config.edn it is there 🙂

(I can see it in emacs too - there is a mode for jars)

(trying to read it with (io/resources config.edn) now instead of (io/file (io/resources config.edn)))

nope.. 😕 trying io/reader now (one of these has to work ^^)

what happens if you do (in the repl) boot.user=> (io/resource "config.edn")?

you should get a java.net.URL object

or nil

let me try 🙂 (repl starting)

@nicolas my very small project works (also with .edn file)

it should work though, this is how it works in dev to read the config

@erwin nice !

where does it not work?

when you do java -jar target/project.jar ?

Yep I get aURL: #object[java.net.URL 0x5549c970 "file:/Users/nha/.boot/cache/tmp/.../j97/fwjp2k/config.edn"]

when I do java -jar target/clojure-backend.jar

It goes through my -main, prints "i will read the file" and then dies with Exception in thread "main" java.lang.IllegalArgumentException: Not a file: jar:file:/Users/.../target/clojure-backend.jar!/config.edn

can you paste that piec of code please?

Sure 🙂 which one though? the file reading part?

looks like you're probably doing (io/file (io/resource ...

somewhere

yes I am

ah sweet!

problem solved then 🙂

Ooh ? how ? 😄

io/resource returns a java.net.URL object (or nil)

io/file can not coerce a URL into a file

if you want to read the edn you can do this:

(def config (read-string (slurp (io/resource "config.edn"))))

no io/file involved

because a jar entry is not a file

Oh ok. I am surprised that iw worked in the repl

Ok makes sense for the file 😄

it works in the repl because it is a file

Yeah get it 😄 Thanks thanks thanks!

the config.edn in the repl is not a jar entry

it's an actual file on the claspath

but the way i pasted above will always work

in dev and prod

Great 😄 thanks a ton

basically you never want to assume that something on the classpath is a file

Hi folks, we have a big mono repo for our backend, frontend and backend services. We’re starting to run into dependency hell, and I’m evaluating migrating to boot as an alternative to using something like https://github.com/jcrossley3/lein-modules I was wondering if there are any examples of something like this, especially setting up pods?

@jfntn hi! can you describe the high level goals you have?

@micha sure, basically we need to define sub-projects that will share a portion of the repo but have isolated dependencies

@jfntn so you have separate jars for each module?

I’d guess

and they depend on each other?

they’d depend on a small number of core files, the conflicts are at the edges in the different targets

More precisely we just ran into an issue where we want to run a load-testing process that uses aleph, but we also use datomic and they conflict over netty with 2-way breakage depending on which version you use

yeah pods solve that problem completely

especially when the conflict is a build or testing component (ie. not part of the production runtime)

Yup that sounds like what we want

that's a separate concern from the multi-module configuration

in boot all tasks are expected to encapsulate their own dependencies in a pod

so they don't pollute the project

like the cljs task for instance

the cljs compiler depends on a ton of hairy dependencies

like google closure, which depends on like 3 or 4 apache ones that conflict with other things

in a large project

but the boot-cljs dependency that you add to your build has no transitive dependencies

Would the multi-module setup come down to properly splitting your source-tree and internal deps so that you can add sets of non-conficting sources the class-path in separate tasks that spin their own pod?

the transitive dependencies are added to its own internal pod

"multi module" means a lot of different things, so it's hard to say one optimal way to do it

generally i will try to factor the modules into separate versioned artifacts

is that feasible in your case?

Actually at this point we’d like to avoid that

so at the end of the day you want to have a single artifact?

Makes more sense to hold everything together at the repo level until our deployment situation gets more complicated

No we need multiple artifacts

i find it easiest to organize my project around the artifacts

Unless we’re able to build a big one that would take args and spin the appropriate module

so i will first decide that: what are the artifacts i want to produce

would it be an option to let boot target task write files with a default permission, such that umask and the likes work?

@erwin the problem there is with the way hard links would work

and also permissions are somewhat platform specific

@erwin the fileset uses hard links instead of copying files for efficiency

but 600 is so restrictive, would 644 also be an option?

a hard link can't have different permissions from the referent

maybe with a binding or redef or something?

@micha one thing I don’t get is how does the repl work with multiple pods? Say I want to run a backend and a service worker, both isolated and both with their own repl. Do I start boot multiple times and kick the repl task inside their pods?

@jfntn well for example, boot always has at least 2 pods running at all times

if you do boot show --list-pods you can see them

you can start a repl server in any pod

ah ok that sounds good

and connect to it with a repl client via the socket it's listening on

or emacs cider, vim fireplace

etc

@erwin can you describe your use case please?

the usecase is Jenkins awkwardness with docker. Jenkins (with docker in pipeline) does funky stuff with the users and it is just easier to make sure everything is at least world readable.

when you say "in the pipeline" do you mean in the boot task pipeline?

or just in your general workflow?

no, in Jenkins pipeline, scriptable Jenkins basically

i think maybe boot could allow a configurable umask-like setting, but have you tried just chmod -R prior to feeding to jenkins?

Jenkins is a bit akward and I kinda need a workaround. Boot is not really the problem, but would be nice to have a bit of flexibility.

@erwin did the boot build && ./bin/fix-permissions approach not work out?

yes that works

the issue with 644 permissions is the boot cache

maybe I should just let it be, but I don't like needing different scripts just for Jenkins

you don't want boot secretly exposing your maybe confidential things

and fixing Jenkins is ... difficult 😉

personally i like solution that @martinklepsch proposes

manipulating files in the shell is nice and simple and straightforward

i kind of prefer that to an obscure configuration parameter in boot

ok 🙂

@erwin the change would be pretty simple to make in boot, if you want to make a PR we can consider it

i'm not opposed to having a setting for that

i don't think 644 should be the default though, because of the cache security issue

@erwin would an option to the target task work for you?

I completely agree with not having a default of 644

to have the target task set permissions?

yes that would also work, I tried one just jet and Make did not like it (not sure why), as Make did like the && chmod -R a+r target

I will try this over the weekend and if it does work I can make a pull request, but I have the feeling I just have obscure requirements ...

adding the option to the target task seems nice to me

you can look at it and easily know what it is supposed to do

ok, I will make a pull request

the target task has an optimization to use hard links in the target dir when it can do so safely -- if you add the :mode option or whatever you'd want to not make hard links in that case

disable that optimization

because if you chmod a hard link you also chmod the thing it points to

which we probably don't want

because there can be some task after target ?

well the thing the hard link would point to would be a file in the boot cache

yes, but that is also true if you chmod after boot is finished?

yes that's true

i hadn't thought of that

maybe that target optimization is ill advised

most of the time target is the last step I would think

but the cache files are not deleted when boot exits

because you may need to debug something

and you will only change the permissions of a few files

the cache is deleted the next time you run boot

so if you chmod files in the target dir and those files are links you will be chmodding files in the cache

so there will be world readable files in the cache that you don't necessarily expect or know about

and that might hang around in the cache

thereby exposing confidential info without your knowledge

yes, maybe a midway solution that gives you the option to forcibly switch back to hardlinks if you really want it?

for performance reasons maybe

yeah currently there is an option to not use links

but i thnk maybe we never want to use links

so yeah, i think you're right

disregard what i said about the optimization stuff 🙂

ok

@micha Is the boot cache a random dir under $TEMP on a per-run basis? E.g.: is there ever a chance that concurrent Boot builds’ caches might collide?

(We currently orchestrate multiple dependent lein builds via make -j8 and are migrating to Boot.)

it's $BOOT_HOME/.cache, which befaults to ~/.boot/cache... i don't think there's collision potential tho

Any plans for another boot release soon?

🙂 i was just about to ask that too

this weekend, for sure

:)

@micha This change could be a start for error handling improvements: https://github.com/boot-clj/boot/pull/532

Other PRs also look good to me (with exception of memory leak one, which I guess is still open)