Fork me on GitHub
#boot
<
2015-11-03
>
pandeiro00:11:32

No didn't go far debugging it at all. I'm pretty sure my code is passing it in, though. Hunch is that the key needs to be nested in something else (`:compiler-options` maybe?).

alqvist11:11:48

Is there any option to omit clj and cljs when building uberjars?

martinklepsch11:11:13

@alqvist: only files in :asset-paths and :resource-paths end up in target or other packages like jars.

martinklepsch11:11:25

@alqvist: are you using bootlaces by any chance?

martinklepsch11:11:54

You can learn more about the roles files can have in the fileset here: https://github.com/boot-clj/boot/wiki/Filesets#fileset-components

alqvist11:11:30

@martinklepsch: Files from

:source-paths
ends up in the jar

alqvist11:11:54

@martinklepsch: I am not specifying :asset-paths

alqvist11:11:12

@martinklepsch: will try bootlaces

martinklepsch11:11:58

@alqvist: Bootlaces will not fix that issue but rather would have been a cause under some circumstances.

martinklepsch11:11:09

@alqvist: can you gist your build.boot?

martinklepsch11:11:14

looks fine to me

martinklepsch11:11:56

@alqvist: just to clarify are your own clj files included or the clj files of your dependencies and you want to remove those?

alqvist13:11:50

@martinklepsch: It seems like only the source from my dependencies are included.

martinklepsch13:11:49

@alqvist: that’s what an uberjar is supposed to do simple_smile Do you want to include compiled java classes instead? Maybe try using (aot :all true) after the (uber) task. I’m not sure if this will work though simple_smile

alqvist13:11:43

@martinklepsch: (aot :all true) produces a compile error for me

alqvist13:11:09

@martinklepsch: But thank you for all the help, I think this is enough for me

martinklepsch13:11:50

@alqvist: why did you want to remove those files originally?

alqvist13:11:22

@martinklepsch: for a false sense of security simple_smile

alqvist13:11:09

@martinklepsch: I suppose an obfuscator should be used

martinklepsch13:11:48

simple_smile I suppose you could try aot :all where it was before, I think that will compile all namespaces you depend on. Not sure if (uber) is still required then actually. Would be interesting to try

alqvist14:11:02

@martinklepsch: `(aot :all) still fails on compile nr 66 or something. One of the asyncs

alqvist14:11:33

But I can do aots for the libraries manually if needed

alqvist14:11:40

again thanks for your help

pandeiro17:11:31

how can i do a fresh install of boot 2.4.2? is there no more boot.sh now?

taylor.sando17:11:41

I think you have to set your BOOT_VERSION file in your ~/.boot/boot.properties file. Then run the script, https://github.com/boot-clj/boot-bin/releases/download/2.4.2/boot.sh

pandeiro17:11:14

Ah I didn't see that file in the Github releases page

pandeiro17:11:01

Ah, it's boot-bin now?

martinklepsch18:11:53

@pandeiro: the binary has been moved there because effectively with 2.4.0 there has been a new binary that should rarely require updates

pandeiro18:11:36

ah ok, cool... so in the future will there just be one boot install script?

pandeiro18:11:53

...and in-app update of everything?

micha18:11:17

@pandeiro: if you download boot.sh from there today it will continue to work forever

micha18:11:45

we may release a bugfix to boot.sh from time to time, but new versions will always work with it

micha18:11:02

so if we fix a bug it will be a bug that exists with earlier versions too

micha18:11:15

i mean earlier BOOT_VERSION too

pandeiro18:11:37

that's a very nice improvement from a usability perspective, nice job 👍

micha18:11:55

also boot.sh works with any BOOT_VERSION back to 2.0.0

micha18:11:17

which isn't the case for say boot.sh version 2.3.0

micha18:11:59

also it's like 5k in size, so you can commit it in a git repo if you want

micha18:11:13

so someone can clone the repo and they have boot already there

micha18:11:19

like gradlew or whatever

martinklepsch18:11:10

2.5.0 2.5.0 2.5.0? simple_smile

juhoteperi19:11:57

GPG PR is up to date again, would be cool to have it merged

chrisn21:11:15

So I am studying pods and how they work. Why is the first thing App/newPod does is seal the app classloader? What advantage does a sealed classloader have?

chrisn21:11:21

actually app/newShim

martinklepsch22:11:36

@chrisn: I think the point of sealing is that no other things can be added to the classloader after the pod is created

micha22:11:08

@chrisn: yes, what @martinklepsch said: boot uses dynapath to mark classloaders as immutable which is really only an opt-in thing, but it is there to prevent programs from leaking clojure into classloaders that will pollute all the other pods

micha22:11:46

if a program doesn't modify the classloader via dynapath then the seal-classloader thing won't have any effect

micha22:11:01

but programs shouldn't be modifying these classloaders anyway

micha22:11:11

if they are then who knows what will result

chrisn22:11:33

The thing is that a new URLClassLoader is created that is that pod's specific classloader. How would a program pollute another pod?

micha22:11:07

a program that walks the classloader chain and uses reflection to add things to a classloader higher up in the chain

micha22:11:17

then those classes pollute all the pods

micha22:11:31

and isolation is broken

chrisn22:11:51

OK, I see your point.

micha22:11:05

doing that is defeating the purpose of classloader heirarchies

chrisn22:11:08

So all dependencies need to be added to a pod on initialization?

micha22:11:11

so it's a party foul

chrisn22:11:15

lol, right simple_smile.

micha22:11:28

no, you can still use any of the functions in boot.pod to dynamically add dependencies

micha22:11:38

but it's usually not necessary

micha22:11:51

because the pod is constructed at a time when you know what the dependencies should be

micha22:11:59

the exception to this is the core pod

micha22:11:03

the one where the build.boot runs

chrisn22:11:08

Is this the worker pod?

micha22:11:19

of course there is no way to know which dependencies your build.boot will need

micha22:11:39

so its dependencies (other than clojure) will be added dynamically

micha22:11:17

all the other pods, the worker pod, the aether pod, and any pods that are created by tasks or by the user will be created with all the dependencies they will need specified at construct time

micha22:11:48

the worker pod is a pod containing all the various dependencies that boot needs for its built-in tasks

micha22:11:10

if the tasks weren't built in the tasks would create their own pods

micha22:11:41

but since boot ships with like 15 tasks that are all maintained together it simplifies things to let them all share the worker pod

chrisn22:11:08

And the aether pod is there to check dependencies and download them if necessary.

chrisn22:11:28

I guess also to get the classpath needed for a specific version of a dependency.

micha22:11:56

the aether pod is for when you first install boot

micha22:11:03

when you have nothing

micha22:11:21

it's packaged as an uberjar that is contained in the resources of boot.jar

micha22:11:35

because boot.jar needs to be able to resolve the clojure dependencies

micha22:11:01

so the very first time you run boot it needs to bootstrap itself

micha22:11:07

because its own code is mostly clojure

micha22:11:29

so boot.jar will extract the aether uberjar from its own resources and write it out to a cache dir

micha22:11:47

then it loads that and uses it to fetch the rest of boot from clojars

micha22:11:58

using the aether maven machiery

micha22:11:04

*machinery

chrisn22:11:34

This entire class isolation things is pretty darn interesting. Leiningen does a similar process which we have seen several pretty irritating issues with and we are using onyx which has peers and having dynamic pods for those would be very useful.

micha22:11:01

leiningen uses multiple jvms for isolation

micha22:11:16

which is foolproof, but not ideal for other reasons

micha22:11:34

and it limits the granularity of the isolation

chrisn22:11:44

Right. And it uses a somewhat bugprone system of having each plugin mangle the project and eval-in-project

micha22:11:55

with boot it's common to have 5 or 6 separate pods

chrisn22:11:08

Clearly, you mentioned at least as many right off the bat.

micha22:11:19

tasks don't load their own dependencies into a classloader that has any other task's dependencies

micha22:11:34

in boot, that is

micha22:11:53

so there isn't a problem of transitive deps of plugins conflicting with each other

chrisn22:11:04

How does the pod pool work? It seems that reusing a pod would be tough unless you unloaded its classes.

micha22:11:14

yeah they're not reused

micha22:11:26

it's a way to front load a queue of warmed up pods

micha22:11:44

so like suppose you want to run your tests in a fresh pod each time you build your project

micha22:11:02

and you're developing with a watcher based incremental compiling thing

micha22:11:06

like the watch task

micha22:11:21

since it takes like 2 sec for clojure.core to compile and get ready to run

micha22:11:32

you'd be waiting for the tests to start all the time

micha22:11:49

the pod-pool thing lets you prebuild these pods you'll be running your tests in

micha22:11:07

so there are always N pods waiting int he queue with clojure already loaded and ready to go

micha22:11:35

when you do (pool :refresh) you get a fresh new pod, and the previous head of the queue is disposed of

micha22:11:08

pods aren't reused, so maybe "pool" is a misleading name

chrisn22:11:19

Hmm. Then going back boot is more like a completely bootstrapping clojure system.

micha22:11:33

boot isn't even a build tool

micha22:11:37

in my opinion

chrisn22:11:38

at least the pod/aether aspect.

micha22:11:42

it's just a way to bootstrap clojure

micha22:11:58

to the point where you can write a program in clojure that will build your thing

micha22:11:22

and it comes with various library functions that are helpful for making your build program

chrisn22:11:38

You could imagine a sort of boot daemon that would just always keep some pods running.

micha22:11:41

hence the name: "boot"

micha22:11:45

yes totally

chrisn22:11:48

And you would instantly get pods based on the jvm opts you need.

micha22:11:54

we have a golang client in the repo

micha22:11:06

from a prototype made long ago

micha22:11:19

but the main problem there is that you can't change the CWD of the JVM

micha22:11:37

so relative paths will be rooted at some arbitrary directory

chrisn22:11:47

haha. Law of unintended consequences.

micha22:11:57

i think we could still do it

micha22:11:07

the new boot.sh simplified the bootstrapping process

micha22:11:35

boot.App has the runBoot() function

micha22:11:41

which can be used to run boot in boot

micha22:11:51

for example, you can do this in the repl

chrisn22:11:03

Really mitigating the startup time would go a long way towards using clojure as a scripting language.

chrisn22:11:22

but the CWD thing would also make stuff a PITA.

micha22:11:51

(App/runBoot
  (App/newCore)
  (future pod/worker-pod)
  (into-array String ["show" "-h"]))

micha22:11:23

that will run boot show -h

micha22:11:13

i'd like to see the startup time for cljs in nashorn

micha22:11:19

perhaps that's the way to go

micha22:11:28

but that's cljs, not clojure

chrisn22:11:28

And what you did was create a new core, create a new worker pod, and run the show task?

micha22:11:53

that reuses the existing worker, but otherwise yes

micha22:11:12

that does everything that running it form the command line would do

micha22:11:24

so like if you have the repl going

micha22:11:28

and you edit your build.boot

micha22:11:38

if you did that expression above you'd see the effects

micha22:11:52

because runBoot starts boot, which reads build.boot etc

chrisn22:11:58

So, how does a central datastructure like a fileset get communicated between pods?

micha22:11:04

it doesn't

chrisn22:11:15

oh, that solves that problem simple_smile,

micha22:11:21

take the cljs task for example

micha22:11:43

it's pretty important to isolate that in a pod because there are some really hairy deps there

micha22:11:00

or actually take the boot-hoplon task for example

micha22:11:12

that's probably abetter example because cljs is crazy with deps

micha22:11:30

hoplon uses stuff like tagsoup and jython

micha22:11:54

not great transitive dependencies because they pull in all kinds of ancient apache http client nonsense

micha22:11:06

which causes all kinds of problems in a large complex project

micha22:11:18

so boot-hoplon does its business in a pod

micha22:11:47

so one of the things it does is transform .hl files into .html and .cljs files

micha22:11:54

using all this tagsoup and whatnot

micha22:11:14

the functions in the pod that will be called to do the work take file paths as arguments

micha22:11:39

and they write the results of their work to some directory whose path is provided in the arguments

micha22:11:53

so the stuff in the pod doesn't need to know about boot or anything

micha22:11:24

the pod setup ensures that stuff is on the classpath in the pod the way it needs to be

micha22:11:57

and when the work in the pod is finished the task will add the contents of the directory where the stuff in the pod wrote its results to to the fileset

micha22:11:15

pods are used basically like transients in clojure

chrisn22:11:31

OK, but is the task itself running in its own pod? And it has a child hoplon pod, correct?

micha22:11:45

no, the task is running in the main thread

micha22:11:09

the task creates pods if it needs to, and has references to them

micha22:11:18

and can call functions in them

micha22:11:30

it's important for all the tasks to run in the same thread, because the fileset is an abstraction that's coupled to the JVM classpath

chrisn22:11:44

Ah, now that makes much more sense.

micha22:11:47

which is shared by all threads with no synchronization mechanism

micha22:11:05

the relationship of pods to the fileset is sort of like transients to persistent collections

micha22:11:32

pods are a place with their own isolated state

micha22:11:43

in which all kinds of crazy mutation takes place

micha22:11:02

(consider the crazy mutation of the classpath done by google closure compiler or whatever)

micha22:11:22

but those changes are isolated and separated from the main thread

micha22:11:56

when all the crazy mutation is finished the task extracts the results and atomically adds them to the fileset and the main classpath

chrisn22:11:48

OK, thanks! I have a ton more questions but that is enough for now

micha22:11:58

haha np! 🍻