Fork me on GitHub
#boot
<
2016-12-30
>
mynomoto00:12:19

Could someone release boot-cljs-devtools?

richiardiandrea00:12:03

@mynomoto I am afk (not true, I am away from my other laptop's keyboard), I will as soon as I get home πŸ™‚

richiardiandrea00:12:43

I think I will release SNAPSHOT first, so that folks can try the new split between tasks, how does it sound?

mynomoto00:12:28

Sounds great to me.

mattly01:12:04

anyone know how I can pass the {:formatter :cljs.test/pprint} option to the runner in cljs-test ?

mattly01:12:00

nvm, looks like doo won't let me

richiardiandrea04:12:09

@mynomoto deployed boot-cljs-devtools 0.1.3-SNAPSHOT

martinklepsch05:12:55

@richiardiandrea just tested the devtools only branch and it breaks when dirac is not on the classpath. Is this line needed? https://github.com/boot-clj/boot-cljs-devtools/blob/master/src/powerlaces/boot_cljs_devtools.clj#L8

martinklepsch05:12:04

(It seems the alias isn't used anywhere)

richiardiandrea05:12:20

Uhm, It used to be need @martinklepsch let me fix it

richiardiandrea05:12:41

@martinklepsch do you know if dirac requires cljs-devtools on the classpath?

richiardiandrea05:12:59

(googling in the meantime)

martinklepsch05:12:57

@richiardiandrea that's a lot of deps haha. Can you let me know when/if you push a new snapshot? Then I'll give it another go. (Or if you're out right now I can also remove that line and push another snapshot)

richiardiandrea05:12:48

I am making the assertion of deps more solid as well

richiardiandrea05:12:25

if you have some mini-minute you might want to try with/without dirac and/or cljs-devtools (three cases really) πŸ˜„

richiardiandrea05:12:06

@martinklepsch new 0.1.3-SNAPSHOT deployed

richiardiandrea05:12:50

(make sure you clean your .m2/repository/powerlaces/boot-cljs-devtools if you notice inconsistencies)

richiardiandrea05:12:36

Time for 😴 :sleeping_accommodation: here πŸ˜‰

martinklepsch06:12:03

@richiardiandrea tried the following: - only devtools dep: worked - only dirac dep: warning about missing devtools dep - both: worked

martinklepsch06:12:37

(Didn't try actually using the dirac task though, but I guess it's not affected by the dependency assertion stuff)

richiardiandrea15:12:44

@martinklepsch thanks a lot I will try with the dirac task only

richiardiandrea15:12:09

I put the assertion in both for coherence πŸ˜„

darwin16:12:34

just a clarification: you can trust clojars[1] about transitive deps brought by the dirac library. In another words, all :scope β€œtest” deps in dirac’s project.clj do not have effect when bundling dirac library[2] [1] https://clojars.org/binaryage/dirac [2] https://github.com/binaryage/dirac/blob/master/project.clj#L107-L113

mynomoto16:12:10

Is there a way for boot-reload have different callbacks for different builds?

mynomoto16:12:53

@richiardiandrea thanks for the release, I'm removing my manual stuff for cljs-devtools πŸ’―

richiardiandrea16:12:14

@mynomoto i noticed the problem while working on figwheel and my branch solves that problem too ;)

pesterhazy16:12:48

@mynomoto you mean on-jsload ?

pesterhazy16:12:13

I use something like this:

(ns my.reload)

(defn on-js-reload []
  (if (.startsWith js/window.location.pathname "/whatever")
    (js/whatever.core.on_js_reload)
    (js/something_else.core_js_reload)))

mynomoto16:12:41

@richiardiandrea I'm really eager for that. Is it a good idea build your branch?

richiardiandrea17:12:31

Well, there are three branches to buiud, depends how much tolerant you are πŸ˜€πŸ˜€

mynomoto17:12:32

@pesterhazy thanks! If the new stuff is not ready that will help a lot.

richiardiandrea17:12:24

The last patch with multi on-jsload is still WIP

richiardiandrea17:12:38

I will have time today for that so it is a matter of some patience and many things will not work for sure πŸ˜€πŸ˜€

mynomoto17:12:29

@richiardiandrea well I think I can wait a bit more. Being productive beats cool things πŸ˜‰ @pesterhazy solution will do for now πŸ‘

richiardiandrea17:12:56

Yep I think too πŸ˜‰

mynomoto18:12:21

@richiardiandrea I'm getting

WARNING: :preloads should only be specified with :none optimizations
WARNING: :preloads should only be specified with :none optimizations

mynomoto18:12:18

It don't show if I explicitly pass :optimizations :none

richiardiandrea18:12:37

check boot-cljs options

mynomoto18:12:54

But none is the default right?

richiardiandrea18:12:00

yeah true πŸ˜‰

richiardiandrea18:12:36

the :preloads feature is supposed to be a :none thing by design (according to the D.Nolen law πŸ˜„)

mynomoto18:12:39

No problem with that... Only with the warning when using :none πŸ˜‰

mynomoto18:12:51

I wonder what causes that.

mynomoto18:12:05

I will check that out later.

richiardiandrea18:12:14

so even if you have :none it shows the warning right?

richiardiandrea18:12:36

could you open an issue by any chance? I will investigate

mynomoto18:12:39

No, if I have :none the warning don't show.

mynomoto18:12:59

But if I don't set :optimizations which default to :none it shows.

richiardiandrea18:12:29

ok cool, so this can be a problem with boot-cljs as well, dunno

richiardiandrea18:12:55

I would start by opening an issue on boot-cljs-devtools πŸ™‚

mynomoto18:12:12

I will dig a bit before that, thanks!

richiardiandrea18:12:50

awesome thank you!

bcbradley18:12:47

i'm trying to understand how boot is put together

bcbradley18:12:06

boot seems to split middleware configuration from invocation

bcbradley18:12:41

so is it true that the actual composition of the middleware functions occurs implicitly for you, via boot?

bcbradley18:12:25

for instance (boot (task :arg "foo" :param "bar") (task2 :pie "yum"))

bcbradley18:12:11

task and task2 are functions that take some configuration arguments that will be closed over by the middleware function that they each return

bcbradley18:12:35

however, I didn't specify the order in which these middleware functions should be composed

bcbradley18:12:39

is it implicitly composed left to right?

richiardiandrea18:12:04

@bcbradley yes it is like that, order left to right as in trasducers

micha18:12:19

you may compose them yourself too

micha18:12:30

(boot (comp (foo) (bar)))

micha18:12:32

that is ok too

bcbradley18:12:45

invoking any middleware returns yet another function, this one takes the fileset as an argument (a map that is constructed by boot), and it is in this function that actual work is usually (not always) done, right?

bcbradley18:12:03

and because composed middleware is also middleware, after the implicit composition of all middleware from left to right, boot will invoke the completely composed middleware in order to yield a single function that accepts the filset map that boot is responsible for generating, correct?

micha18:12:37

yes exactly

micha18:12:03

that's why you can easily compose tasks at the command line the same way as you would do with lisp expressions

micha18:12:14

the tree is flattened

bcbradley18:12:17

that seems pretty powerful

bcbradley18:12:44

I feel like the explanation on github could be a bit better though, I had to go digging for this info

bcbradley18:12:55

eventually i had to read the source

micha18:12:12

does that help clarify how it would work?

micha18:12:33

i guess that's too low level maybe

bcbradley18:12:45

i went through that but all the lingo makes it too terse

bcbradley18:12:58

i think it would be better explained if it began with an unfulfilled need

bcbradley18:12:07

what does boot's middleware model really try to solve?

bcbradley18:12:30

start with something simple, like composing plain old functions

bcbradley18:12:38

then improve it slowly

bcbradley18:12:53

by adding abstraction (deferred functions, aka lambdas)

bcbradley19:12:06

maybe someone has already taken this approach I'll look

bcbradley19:12:05

by starting with something simpler and then aggregating i mean like this: http://www.braveclojure.com/appendix-b/

bcbradley19:12:43

however, brave and true still sort of glazes over "why" this is, focusing too much on "how" it is

bcbradley19:12:12

what is the motivation? Why should we want to compose tasks, why should we want to separate configuration from invocation?

bcbradley19:12:22

thats the sort of thing I would really like in the official documentation

micha19:12:56

one important thing with separation of invocation and configuration

micha19:12:17

it's important that the configuration of tasks doesn't change once the pipeline has started

bcbradley19:12:25

i mean boot doesn't just seperate configuration and invocation, it seperates configuration, invocation and composition

bcbradley19:12:28

its 3 levels deep

micha19:12:43

because a task should not be able to see the effects of any operation that follows it in the pipeline

micha19:12:12

so once the tasks are invoked and the middleware formed, those options should not be changed

bcbradley19:12:50

i guess the way I'd probably write it would be to start with a "bad" way of doing things: making one build per project and explaining / showing why its bad. Then I'd "solve" that by breaking things up into composable pieces, so adding 1 level of abstraction.

bcbradley19:12:19

Then I'd "show" why that isn't totally great either, and motivate the need for adding another level of abstraction with seperation of composition and invocation

bcbradley19:12:24

etc for configuration

bcbradley19:12:26

see what I mean?

bcbradley19:12:30

by doing that a person's understanding of how boot operates is made more concrete

bcbradley19:12:40

they are less likely to forget any of it because they know why they need all of it

micha19:12:49

i think there is some parallel with transducers

micha19:12:03

but that probably doesn't help make things more clear

bcbradley19:12:10

transducers don't seperate configuration, but its similar

micha19:12:25

i mean like (partition 2)

micha19:12:31

returns a middleware

micha19:12:35

with config set

micha19:12:40

and it's stateful like tasks

micha19:12:56

so you don't want to reuse it, just like you wouldn't reuse a task middleware

bcbradley19:12:05

the only reason I was even able to make sense of the source code is because i'm already familiar with transducers

bcbradley19:12:39

Idk I just feel like everyone who has worked on boot understands implicitly why its "the right way" because they've been shot in the foot by lein

bcbradley19:12:03

but they aren't explaining why, which makes everyone else scratch their heads about "why 3 levels of abstraction"?

micha19:12:42

there might actually be one layer we don't need in there

micha19:12:55

we went back and forth about it quite a bit

micha19:12:13

so went with the more flexible design

bcbradley19:12:14

honestly i'm fine with the 3 levels now that I understand why they are needed / what problem they solve

bcbradley19:12:22

i just don't like that i had to go digging and thinking for hours to figure it out

bcbradley19:12:04

the documentation right now is fantastic for explaining WHAT boot is doing under the hood

bcbradley19:12:14

its terrible at explaining WHY it does it that way and not some other way

bcbradley19:12:29

or why it has to be done in the first place

micha19:12:38

interesting yeah

bcbradley19:12:44

its like reading a solution to a math problem without being given the problem itself

bcbradley19:12:48

it doesn't make any sense without context

micha19:12:35

there are a number of concerns that are all relevant there

micha19:12:46

like we wanted to be able to do things from the command line or in the repl

micha19:12:59

things like that

micha19:12:24

and since the jvm and clojure are heavy weight processes we need to leverage incremental builds as much as possible

micha19:12:37

which means some kind of framework

micha19:12:57

and we wanted to be able to write small unixlike tasks that can be composed in pipelines like you'd do in the shell

micha19:12:03

and write them in clojure

bcbradley19:12:15

thats the kind of stuff that should be in the docs

bcbradley19:12:26

honestly after figuring it out i don't see why i would ever want to use lein

bcbradley19:12:37

but i can still remember, just earlier this morning, that i didn't know hardly anything about boot

bcbradley19:12:46

and i felt like the motivation was so hidden

bcbradley19:12:52

that i couldn't justify using it

micha19:12:19

yeah if i was starting out with clojure i would use lein for sure

micha19:12:27

cause i can get stuff from stack overflow

micha19:12:32

to get myself computing

bcbradley19:12:17

another thing i had trouble finding was the order in which tasks ought to be run

bcbradley19:12:34

err middleware, not tasks

bcbradley19:12:43

sometimtes specific middelware has to be composed before others

bcbradley19:12:01

it'd be nice to have a reference for that order

micha19:12:14

yeah that can be tricky, but i thnk it's less tricky than trying to imagine a DAG

bcbradley19:12:39

better yet, since boot is architected to separate composition from invocation and configuration,

bcbradley19:12:45

boot could do that for you, in theory

bcbradley19:12:59

it could just take a set of middleware and then totally order it

micha19:12:09

the problem is that the tasks don't know what their products will be used for

micha19:12:22

just like unix programs don't know where in the pipeline they need to be

micha19:12:32

because they can't enumerate all the uses you will have for them

bcbradley19:12:42

you agree that currently boot composes left to right?

micha19:12:50

that's definitely true

bcbradley19:12:59

so let there be an additional boot function

bcbradley19:12:06

whose job is to take a set of tasks

bcbradley19:12:20

and returns a single middleware, composed

micha19:12:22

but how do it know

bcbradley19:12:24

after ordering the arguments

bcbradley19:12:45

that allows the end user to selectively choose what things to automatically order

bcbradley19:12:56

and its visible, because the name of this new boot function is glaring at them

micha19:12:57

how would it ever know how to order things?

bcbradley19:12:22

that could be a key/value in the fileset

bcbradley19:12:39

that different tasks can "register" themselves to on configuration

bcbradley19:12:45

since configuration happens before composition

micha19:12:58

but what would they register

bcbradley19:12:06

i'm not sure

bcbradley19:12:10

a map or something?

micha19:12:21

like how can you statically compute the ordering

micha19:12:30

what information would be the input to that function

micha19:12:14

the idea of the pipeline is based on the assertion that that is a fool's errand and impossible to do correctly

micha19:12:30

leiningen and maven and so on attempt this

micha19:12:43

they establish well-known phases of the build

micha19:12:52

like :compile, :uberjar, etc

micha19:12:02

but this is insufficient

bcbradley19:12:12

lets say you had 3 middlewares a, b, and c

bcbradley19:12:23

and they each have various configuration arguments

bcbradley19:12:50

generally you'd yield a handler through them with (boot (a ...) (b ...) (c ...))

bcbradley19:12:25

what we are really talking about has nothing to do with auto ordering and has a lot to do with information hiding / abstraction

micha19:12:28

or (comp (a..) (b...) (c...)) even

bcbradley19:12:44

what we would really want is something like (boot (foo ...) (c ...))

bcbradley19:12:58

where (foo ...) is tantamount to (comp (a ...) (b ...))

bcbradley19:12:32

now its simple to make a single foo that is bound to a single a and b

bcbradley19:12:35

but that isn't very interesting

bcbradley19:12:50

what we are talking about really, is having a single foo that can be bound to any a, b, c, d, e, whatever

micha19:12:59

do you mean like

(deftask foo []
  (comp (a...) (b...)))

micha19:12:27

that would work ^^^

bcbradley19:12:28

that is the way of binding foo to a single a and b yeah

micha19:12:10

one thing that i think is very important is that tasks are not coupled to each other

bcbradley19:12:15

here is NOT what i'm saying:

micha19:12:17

no task knows about another task

bcbradley19:12:27

`(deftask foo []

micha19:12:31

unless you as the user decide to compose them

bcbradley19:12:53

(deftask foo [a b c] (comp a b c))

bcbradley19:12:00

thats obviously not very useful

bcbradley19:12:17

neither is

(deftask foo [& more] (apply comp more))

bcbradley19:12:19

what i'm saying is something like

(deftask foo [sorter & more] (apply comp (sorter more)))

bcbradley19:12:41

but how do you sort arbitrary middleware?

bcbradley19:12:50

thats the question you are throwing at me

bcbradley19:12:19

i honestly don't know

bcbradley19:12:30

some of the middleware could be sorted without user intervention

bcbradley19:12:36

not all of it can be

micha19:12:46

i assert that the following facts are true:

micha19:12:27

1. the user needs to be able to understand how the machine orders operations, so they can use the tools

micha19:12:06

2. the user needs to understand how the different operations relate to each other, so they can understand the ordering

micha19:12:43

3. given 1 and 2, we should cut out the middleman and make it easy for the user to just directly specify the order

micha19:12:01

because they already need to know all the information that makes the ordering process trivial

bcbradley19:12:32

here is what I would like to have:

mynomoto19:12:38

So about the order when there are dependencies among middleware I think what yogthos did for Macchiato interesting using metadata: http://yogthos.net/posts/2016-12-20-MacchiatoModules.html

bcbradley19:12:43

NOT a system that tries to do total ordering for you

bcbradley19:12:48

and NOT a system that does no ordering for you

bcbradley19:12:07

but more of a system that takes some "tautologies" as arguments (think core.logic)

bcbradley19:12:21

and will do as much ordering as it can given the claims you've made

micha19:12:29

i will study that

micha19:12:36

but it seems like the worst of all worlds

micha19:12:01

now you not only need to know how to do it yourself, but you need to understand a complicated thing that is completely incidental that was bolted on

alandipert19:12:06

also i believe Make is such a system

alandipert19:12:24

in that it unifies the objects you want with the objects you already have via a process you specify

micha19:12:34

it's adding incidental complexity i think

micha19:12:50

like :required [:wrap-params]

bcbradley19:12:08

not necessarily

bcbradley19:12:22

you can actually cause it to devolve to boot's model

bcbradley19:12:28

by specifying no claims whatsoever

bcbradley19:12:31

then it wouldn't perform any ordering

bcbradley19:12:36

the order would be the order you gave it

bcbradley19:12:59

if you wanted lein's model, you'd provide some maven repository style total ordering "sorter" processor

micha19:12:00

the boot philosophy, if there is such a thing, is that the tasks shouldbe small enough to be easily understood by the user

micha19:12:17

if that's done then the user can trivially make their pipeline

micha19:12:30

that means tasks doing only one thing, etc

micha19:12:56

like boot pom uber jar push

micha19:12:04

that seems very straightforward to me

bcbradley19:12:10

well i'm partially for and against that design

mynomoto19:12:14

Well I had lots of middleware bugs and if some tool can help avoid that by declaring dependencies I would use it. But in bootland that's a more rare occurrence.

micha19:12:16

and would only add incidental complexity to try to automate it

bcbradley19:12:25

the thing that pulls me for it is the proven success of the unix command line

bcbradley19:12:33

its been around almost 50 years and its proven itself

micha19:12:51

well consider this:

bcbradley19:12:12

the thing pushing me away from it is practicality

mynomoto19:12:14

I don't mean automation, only sanity checks if the declared order is know to be wrong.

bcbradley19:12:27

i don't think everyone is really going to read and understand each task

micha19:12:32

boot pom sift -m foo:bar uber sift -i baz jar push

bcbradley19:12:39

or that the community as a whole has any way of making sure each task is "simple and small"

bcbradley19:12:47

even in unix, the programs aren't really "simple and small"

bcbradley19:12:53

some of them are freakishly large and complex

micha19:12:00

that example above is very simple, unless you have something that's going to reorder them

micha19:12:13

then it is very complicated to do

micha19:12:19

with lots of magical metadata

micha19:12:30

like in lein the whole ^:replace business

micha19:12:41

and "robert hooke"

micha19:12:47

injections

micha19:12:04

you now need like 10 different things

micha19:12:13

and lifecycle hooks

micha19:12:17

and aliases

bcbradley19:12:00

is it true that there can be two orderings with the same semantics?

micha19:12:09

i think if it's difficult to know how to order the task then the tasks are not well designed or documented

micha19:12:20

i guess it depends what you mean by semantics

micha19:12:32

like there are different boot invocations that could produce the same result

micha19:12:46

just like unix pipes

bcbradley19:12:01

alright, and there are clearly two ordering with have different semantics

micha19:12:09

cat foo |grep hi |grep bye vs cat foo |grep bye |grep hi

bcbradley19:12:57

so depending on which semantics the user wants, he would want a different order

bcbradley19:12:07

even if the pieces all consist of middleware with the same names

micha19:12:16

yeah the example with sift above is a good one i think

bcbradley19:12:22

so its impossible to automatically order it unless you have some way to encode the user's intention

bcbradley19:12:30

what you are saying is that you can't know the user's intention in symbols

bcbradley19:12:54

well thats an interesting claim, and i'm not sure i agree with it

bcbradley19:12:57

i haven't thought about it deeply

micha19:12:02

the user wants boot to let them compute in peace

micha19:12:25

without imposing some arbitrary framework on them

micha19:12:58

the only framework in boot is the minimum necessary for composition

bcbradley19:12:01

if there were a way to symbolize user intentions, would it be less complex than a turing complete language or would it have to be turing complete?

bcbradley19:12:04

that is the real question

bcbradley19:12:16

if it has to be turing complete, might as well use the language of choice

micha19:12:23

the user would still need to understand what they are trying to do

bcbradley19:12:24

and in that case, boot does it correctly

micha19:12:33

and how the tools work

micha19:12:07

like i don't think that boot should be a substitute for understanding what you are doing

micha19:12:20

like the whole boot pom uber jar push thing

bcbradley19:12:24

let me just throw this out there

micha19:12:38

some people don't know what a pom.xml is, but that's not helping them

bcbradley19:12:53

cells in a multicellular organism's zygote (a small bundle of cells that eventualy develop into a fetus)

bcbradley19:12:15

are able to work collaboratively to grow into an organism with a well defined shape and features

bcbradley19:12:30

even though there is no "overlord" or governing process

bcbradley19:12:43

its truly decentralized

bcbradley19:12:02

in other words, tasks could probably order themselves

bcbradley19:12:22

but they would each have to carry around information about how to be independent

micha19:12:04

is there a "user" in your analogy?

micha19:12:17

seems like god maybe

micha19:12:24

but he understands all the cells very well

bcbradley19:12:24

i suppose the user just awaits the birth of the organism

bcbradley19:12:44

maybe the father for instance

bcbradley19:12:48

or the mother

micha19:12:49

we can make things that program themselves as long as we don't care what they produce

micha19:12:04

if we need to get a specific job done though

micha19:12:20

we want to just do it

micha19:12:33

and move on with life

bcbradley19:12:17

yeah i guess if we try to go too far down the rabbit hole programming will start to look less like specifying an exact process and more like selecting a mate for approximate features / "healthy" children / etc

bcbradley19:12:22

it would be less exact

micha19:12:29

like when i use boot it's basically just some extra work i need to do toget my primary task accomplished

micha19:12:44

i really want to make some clojure program or something

micha19:12:05

so the stupider boot itself is the better for me

bcbradley19:12:36

well then i'd just note this: if we are going to pin ourselves to boot's level of abstraction, and not attempt to abstract further

bcbradley19:12:44

we need really really really damn good documentation on middlewares

bcbradley19:12:50

because thats the endpoint

bcbradley19:12:52

it doesn't go higher

bcbradley19:12:25

i need to know enough about middlewares to know exactly what they do, why they do it, how they do it, and how that might interact or interfere with other middleware

micha19:12:39

middleware is only one way to build stuff

bcbradley19:12:43

composability seems to imply linearity, but that isn't true

bcbradley19:12:48

even in mathematics that isn't true

micha19:12:54

i would like to makea more low level interface too

bcbradley19:12:06

you can study up on mathematical chaos to learn about nonlinear differential functions

micha19:12:16

like one thing rich pointed out was that tasks do two unrelated things now

micha19:12:28

they do flow and work

micha19:12:53

like the middleware continuation thing plus the actual work of compiling or building something

micha19:12:06

so i could imagine separating those at a lower level

micha19:12:26

where there is a function that accepts a fileset, does some work, and returns a fileset

micha19:12:35

and a function that is the "connective"

micha19:12:41

that's the middleware part

bcbradley19:12:49

i think i see what he's saying

micha19:12:52

or if you don't want to use middleware you could do something else to connect them

micha19:12:56

like a DAG

micha19:12:59

or whatever

bcbradley20:12:08

i think he means that there might be tasks for which you do the same "kind" of operations in the same order

micha20:12:37

well like the aot compiler task

bcbradley20:12:38

it'd be nice if you could specify the "kindness" and order of the middleware first

micha20:12:39

for example

bcbradley20:12:44

that would be flow

micha20:12:00

there could be a function that gets a fileset as its argument

micha20:12:13

it could compile namespaces and add them to the fileset it returns

micha20:12:30

no middleware or calling next task or anything

bcbradley20:12:48

i think the really problematic issue is that state is lurking in places

bcbradley20:12:55

and you can't really know it unless you dig for it

bcbradley20:12:07

a nonstateful middleware is trivial, and the order doesn't matter

bcbradley20:12:20

with the exception that it depends on the return value of a previous middleware

micha20:12:24

it still matters

bcbradley20:12:26

(aka a modification in the map)

bcbradley20:12:31

but that is well documented

micha20:12:33

like if you have a task that creates cljs namespaces

micha20:12:42

that has to go before the task that compiles them to js

bcbradley20:12:53

if there is a side effect that is produced by a middleware, and another middleware came before it but depends on that side effect

micha20:12:53

just like cat needs to come before grep

bcbradley20:12:55

then you run into issues

bcbradley20:12:13

so if it were up to me, I'd marshal all side effects away

bcbradley20:12:36

only the fileset should be modifyable

bcbradley20:12:46

really its not even being modified

micha20:12:47

that's how it should be

bcbradley20:12:49

its just an immutable map

micha20:12:59

i mean you can have local state

micha20:12:03

as long as it's not exposed

bcbradley20:12:05

if a user wants to produce side effects based on the values in those maps, then he can do it himself

bcbradley20:12:46

i would even be fine with having "middleware" that does that

bcbradley20:12:00

but you can't just mix it in with normal middleware and call it normal middleware

bcbradley20:12:10

and you shouldn't have middleware that does that and does something else, like modify the map

micha20:12:16

like what kind of side effects are you talking about?

bcbradley20:12:36

in the strictest sense, printing to the console

bcbradley20:12:45

in the minimal sense, changing the filesystem

micha20:12:27

tasks need to change the filesystem and the classpath

bcbradley20:12:40

i don't believe they should

micha20:12:44

otherwise they become useless

micha20:12:55

try to compile a java class that isn't on the classpath

bcbradley20:12:04

i think there should be tasks that just read the fileset and return a new one (for the next handler)

bcbradley20:12:16

er middleware (not tasks)

bcbradley20:12:24

and there should be middleware that has side effects

bcbradley20:12:28

the former is pure

micha20:12:34

you would end up reinventing the classpath

bcbradley20:12:36

the latter should end in a !

micha20:12:38

eventually

micha20:12:58

because like what if a process depends on dependencies in jars?

micha20:12:12

you'd need to somehow incorporate those into the immutable thing

bcbradley20:12:13

then its impure, and the middleware should end in a !

bcbradley20:12:46

i fully expect the task to be impure

micha20:12:52

the ! is inconvenient for the command line

bcbradley20:12:54

the middleware can be a mix of impure things and pure things

bcbradley20:12:08

then use a different convention

bcbradley20:12:30

the point is you should be able to know whether a middleware is pure or not

bcbradley20:12:38

either in its documetnation at the very least, or in its name

bcbradley20:12:52

you shouldn't have to guess, based on its functionality

micha20:12:18

i don't know of any pure tasks, except maybe the diagnostic ones

bcbradley20:12:22

not pure tasks

bcbradley20:12:24

pure middleware

micha20:12:35

i mean middleware yeah

micha20:12:04

most of the handlers exist to mutate the filesystem

bcbradley20:12:55

you know it occurs to me there is a way to introduce purity even for middleware that does what you just said, and without duplicating the classpath

bcbradley20:12:28

clojure immutable data structures, strictly speaking, mutate main memory

bcbradley20:12:43

but they do it in a very organized fashion so that it looks immutable on the outside

bcbradley20:12:04

the list, vector and map are all DAGs

bcbradley20:12:26

the same thing can be implemented on disk

bcbradley20:12:52

in other words, make the filesystem immutable

micha20:12:17

we did consider a fuse filesystem initially

micha20:12:23

but it's too slow

bcbradley20:12:39

well thats a shame

bcbradley20:12:58

it would have been a beautiful way to manage the complexity of state

micha20:12:50

i wonder if there are new alternatives

micha20:12:59

something at the system level perhaps

bcbradley20:12:02

i don't know but its worth pursuing

bcbradley20:12:17

if you can implement it

bcbradley20:12:59

the benefits that you gain are that you can't have a task "overwrite" the effects of a previous task

micha20:12:25

but you need that

micha20:12:32

or clojure won't work properly

micha20:12:43

or any of the jvm stuff

bcbradley20:12:05

not necessarily

bcbradley20:12:23

lets take a map as an example

bcbradley20:12:39

if you update the map, the previous map still exists

bcbradley20:12:43

they share structure

bcbradley20:12:53

with the jvm, it would be pointing to the new map

bcbradley20:12:06

it would believe that there was some kind of statefulness going on, because it can't see the old map

bcbradley20:12:27

however, anyone who dependend on the old map, and who doesn't want to be run over by "statefulness" still can

bcbradley20:12:31

because the old map still exists

bcbradley20:12:43

now replace the word "map" with "filesystem"

micha20:12:54

currently boot provides this to taks

micha20:12:59

with the fileset

micha20:12:21

like the handler fn is passed the current fileset

micha20:12:40

meaning an immutable fileset object that corresponds exactly to the state of the filesystem at this moment in time

bcbradley20:12:59

does the fileset contain references to files, or does it actually contain the information that was in the filesystem (a copy)

micha20:12:03

that state can be restored via (commit! old-fileset)

micha20:12:14

it contains the information

bcbradley20:12:19

ok then that works for me

micha20:12:50

the way it works is whenever you add a file to a fileset, like (add-resource fileset some-dir)

micha20:12:02

that returns a new immutable fileset object

micha20:12:22

but it also stores the files in some-dir in its "blob" storage area

micha20:12:37

this is a content-addressed database of files

micha20:12:01

the filename in there is generated from a md5 hash of the contents plus the lastmod time

bcbradley20:12:10

similar to git

micha20:12:26

the immutable fileset object contains a mapping of path to blob id

bcbradley20:12:48

all this sounds a little slow

micha20:12:57

like {"foo/bar.clj" "9c59ec5631dd7bd4d3053bd58f098078.123456"}

bcbradley20:12:08

but i like the priority given to immutability and good architectural princples

micha20:12:21

it can be less slow because it can cache aggressively

micha20:12:30

you can diff two immutable filesets

micha20:12:55

the files that are in the classpath are also just hard links to files in blob storage

bcbradley20:12:56

you are talking about incremental compilation now aren't you?

micha20:12:08

well boot does this internally too

micha20:12:14

when you add that directory to the fileset

micha20:12:20

it's internally diffing and patching

micha20:12:29

to do the minimum number of filesystem operations

micha20:12:54

boot can also cache immutable filesystem trees

micha20:12:58

like the uber task for example

bcbradley20:12:02

how slow would this be on a distributed filesystem?

micha20:12:14

well it needs to use hard links

micha20:12:22

and locking

micha20:12:33

so it would probably not work well over nfs

micha20:12:03

it works well for in memory filesystems i hear

bcbradley20:12:38

i would expect so, since the cpu cache is never invalidated as the information is immutable

bcbradley20:12:24

it can still miss / run out and be replaced

bcbradley20:12:27

but invalidation is a big deal

bcbradley20:12:04

when the cpu cache is invalidated it means it has to use the bus to reload into the cache from main memory the "new stuff"

bcbradley20:12:22

if you just write once and read a bunch rather than write a bunch and read a bunch you are going to get some stupid high speeds

micha20:12:24

also java nio is pretty fast

micha20:12:45

i didn't see a big difference when i tried it, but i didn't really try too hard

bcbradley20:12:02

that might be because most of the usual tasks still create something new

micha20:12:07

i have a ssd locally

bcbradley20:12:09

so you still end up writing a lot

micha20:12:14

maybe that is also relevant

bcbradley20:12:16

you just don't end up overwriting

bcbradley20:12:36

overwriting is only going to be worse than writing if you could avoid overwriting entirely

bcbradley20:12:48

for instance, if you can reuse the structure of the blob

bcbradley20:12:01

so, incremental compilation rather than recompilation for instance

micha20:12:08

maybe in-memory vs ssd is a small difference, as i think ssds have pretty fancy caching and onboard memory for it

bcbradley20:12:15

if you did that in memory you are going to have some crazy speeds

micha20:12:02

i think in general you are far more often reusing than overwriting in boot

bcbradley20:12:26

well i feel a lot happier about knowing how boot is put together now

micha20:12:28

especially since tasks are deterministically regenerating the same things over and over

bcbradley20:12:33

seems like a lot of good ideas were used

micha20:12:08

it's amazing how many things you need to be able to compose tasks

bcbradley20:12:08

i'm also glad you didn't go down the rabbit hole of trying to do too much for the user

bcbradley20:12:29

even though it means the burden is on the user to understand the semantics of the tasks he uses

micha20:12:51

the good part about that is that you can not use tasks that are too complex or weird

bcbradley20:12:02

i guess in the end he can't avoid it, he can only fool himself into thinking he's avoiding it, when really the complexity is transduced (as in energy) into a different place or form

micha20:12:22

most of the time you can hack together your own task to do a thing

micha20:12:31

without needing to learn how to use something complicated

micha20:12:40

which is the ultimate goal

micha20:12:00

likelet the user make their own thing if they need to get stuff done

bcbradley20:12:01

i think there is something in information theory that says there is a certain amount of complexity in any given system that cannot be destroyed, but its form can be changed

bcbradley20:12:25

trying to do things for the user (as with lein / maven) just splits up the complexity into a dozen pieces and makes it harder to understand

bcbradley20:12:34

if you go with boot's approach its all in one spot and doesn't infect anything else

micha20:12:00

yeah it's kind of the general principle behind all of our things

micha20:12:18

figure out what the primitives are, make those, and let the user assemble them

bcbradley20:12:30

seems an awful lot like the principle of thermodynamics that says that thermal energy in a closed system cannot decrease

micha20:12:18

maybeit can be called "linearly scaling irritation" factor

micha20:12:30

like you can only be annoyed enough to make your own task

micha20:12:53

which is designed to be a low annoyance process

bcbradley20:12:31

only thing i'd like to say is that if the boot community is really committed to helping eachother make simple, small tasks, then the official documentation needs to include a treatise on the motivation for separating composition from invocation from configuration

bcbradley20:12:46

they need to communicate the motivations as much as possible

bcbradley20:12:53

show the pitfalls of not sticking to the "good path"

bcbradley20:12:08

the more concrete you make it, the more likely the community is to produce "good" tasks

micha20:12:24

yeah it also involves some hard design work

micha20:12:42

in many cases you need to design something to achieve the goal

micha20:12:50

like cljs for example

micha20:12:07

it's good to have the live reload and the cljs repl and so on be different separate tasks

micha20:12:22

not all monolithically glommed onto the compiler

micha20:12:27

with a mess of options

micha20:12:56

but to make that work we needed to devise the .cljs.edn file convention

micha20:12:27

and for maven stuff, the pom.xml is a good descriptor, but there are things it lacks

micha20:12:59

designing the decriptors, the spec files for various processes

micha20:12:14

maybe clojure.spec can solve this in some awesome way

bcbradley20:12:08

how about instead of having the cljs task do this and that

bcbradley20:12:16

it literally invoke boot

bcbradley20:12:31

so you'd have a boot within a boot

bcbradley20:12:38

that way cljs can do whatever it wants in its own world

bcbradley20:12:44

and it closes over the clojure world

micha20:12:55

that's where pods come in i think

micha20:12:02

although boot-in-boot is thing too

micha20:12:44

currently the cljs compiler lives in its own pod

micha20:12:55

it's in an isolated clojrue runtime

bcbradley20:12:11

what i mean is boot is a way of compiling clojure code, you have build.boot written in clojure and it runs on the jvm

micha20:12:21

pods are important for making small tasks because with lots of small tasks you would get terrible dependency hell if they all lived in the same clojure runtime

micha20:12:42

lots of small tasks -> many dependency conflicts

bcbradley20:12:33

what if you had another version of boot for parsing a build.boot that is written in clojurescript

bcbradley20:12:42

and it runs in a javascript runtime like node.js

bcbradley20:12:28

that way you could theoretically have the node.js boot invoke the jvm boot (or the more regular approach which is vice versa)

micha20:12:51

what is the advantage of nodejs in that scenario?

bcbradley20:12:28

because it means clojurescript isn't sort of "bolted" on to the current incarnation of boot (which itself depends on jvm)

bcbradley20:12:37

in other words, it is its own thing

bcbradley20:12:08

consider the scenario where you want to compile some clojurescript into javascript

bcbradley20:12:21

if you have boot the way it is currently you need to depend on clojure (and have a jvm)

bcbradley20:12:33

if you do it the way I'm suggesting, you don't need to have any jvm or dependon clojure

bcbradley20:12:35

its more direct

bcbradley20:12:00

for the same reason, if you wanted to run boot within boot, the clarity of what is going on increases

bcbradley20:12:12

instead of having things like .cljs.edn

bcbradley20:12:16

you just have another build.boot

bcbradley20:12:28

with a different boot program, maybe boot-cljs

neupsh20:12:04

Hello again my knowledgeable friends πŸ™‚ I have been trying to resolve this issue without any success.. I am hoping someone could help me figure it out: (basically, it looks like classloader related issue) I have a namespace with gen-class which extends a class from library (jnr-fuse, which uses jnr-ffi library). all the required libraries are listed in dependencies. When i run repl using boot, - i can import my generated class which extends the library, - i can instantiate the class which i am extending as well - but when i try to instantiate my class (generated), it throws ClassNotFound and NoClassDefFound exceptions

java.lang.ClassNotFoundException: jnr.ffi.provider.jffi.NativeClosureProxy
             java.lang.NoClassDefFoundError: jnr/ffi/provider/jffi/NativeClosureProxy
                 java.lang.RuntimeException: java.lang.NoClassDefFoundError: jnr/ffi/provider/jffi/NativeClosureProxy
Now, today I tried to conver the project to leiningen project, and ran the same exact project in lein repl and i can instantiate my class just fine. How do i work and fix this issue? I feel like this is because of different classloader for compiling and running the classes, but i was assuming both would have access to the classpath and would have no problem loading the classes found in the :dependencies

micha20:12:22

@nux is there some kind of jni or something involved?

neupsh20:12:02

@micha yes, the jnr (alternative to jni and jna) is at the bottom of all those libs

micha20:12:22

so leiningen has a convention for importing native libraries

micha20:12:51

it looks inside jars and extracts the native implementation to the filesystem

micha20:12:24

there really isn't an established mechanism for this

micha20:12:27

in general

micha20:12:37

so boot doesn't try to do it automatically

micha20:12:07

if you extract the native libraries from the jars and put them in a directory

micha20:12:56

and add that to the java.library.path system property

micha20:12:24

one sec i will look at the jnr-fuse jar

micha21:12:02

can you paste a minimal example namespace with a genclass that shows the problem?

micha21:12:08

so i can test here

neupsh21:12:00

@micha let me write a small namespace

micha21:12:22

and your set-env! config please

micha21:12:38

i think native libraries is one case where the system classloader is maybe the right way to go?

micha21:12:51

because you get errors if you load a dll multiple times

micha21:12:14

so really each native library must be a singleton

micha21:12:35

you can't really load multiple instances in different classloaders

neupsh21:12:02

^^ this is what happens

richiardiandrea21:12:50

@bcbradley there is an attempt to build a clojure build tool in node.js here: https://github.com/eginez/huckleberry

bcbradley21:12:55

looks pretty interesting

richiardiandrea21:12:13

it is almost a clean slate now and might be worth pitching some idea there too, the author is always on #cljs-node. About the JVM vs Node thing, we cannot have a fair comparison until we actually build boot in node in terms of speed.

richiardiandrea21:12:26

I have noticed myself that what really triggers my frustration is when I try to execute boot tasks a-la-carte (clj), so basically when I need a (bash?) script instead of spinning up a JVM and pay for its startup time

richiardiandrea21:12:31

if you (like me) still want to be able to use cljs while scripting then I am going to suggest planck or lumo

richiardiandrea21:12:49

no startup cost upfront

bcbradley21:12:01

i just don't like the idea of creating specific file formats like .cljs.edn

bcbradley21:12:08

it smells like we are going down the wrong path with that

bcbradley21:12:38

converting one thing into another seems like the overall idea of boot

bcbradley21:12:46

which converts clojure files into jar files

richiardiandrea21:12:56

I hear you, I had exactly the same problem yesterday, where my boot-reload needed to know cljs options before boot-cljs runs

richiardiandrea21:12:02

but you can get around that

bcbradley21:12:06

if you want to convert clojurescript files into javascript files, it seems like that should be a thing tantamount to boot itself

micha21:12:09

the pom.xml is why maven is successful i think

bcbradley21:12:09

not a thinng within boot

micha21:12:28

since the pom.xml exists we can use maven in boot

micha21:12:43

the .cljs.edn is like the pom.xml

micha21:12:01

it lets you talk about a clojurescript application

micha21:12:18

without that you just have some collection of compiler options that mean nothing other than implementation details

bcbradley21:12:34

i think you are missing what i'm suggesting

richiardiandrea21:12:39

it is more powerful than a pom.xml actually, because you can change it at boot runtime

bcbradley21:12:01

is boot responsible for converting A to B in the general case

bcbradley21:12:08

or is it responsible for converting clojure to jar

bcbradley21:12:31

is it responsible for converting #{clojure, clojurescript, whatever, whatever, whatever} to {stuff stuf stuf stuf}

bcbradley21:12:42

the latter case is something i'd like to avoid

bcbradley21:12:53

boot shouldn't try to do everything

bcbradley21:12:54

and i certainly don't think boot should try to take on the goal of being able to convert arbitrary A into arbitrary B

bcbradley21:12:02

because that is what programming is

bcbradley21:12:35

instead, boot should focus on one single "goal" it should convert one X into one Y

bcbradley21:12:39

clojure to jar

bcbradley21:12:02

another thing "booty" can convert W to V

micha21:12:13

yeah i think the native things need to be extracted

micha21:12:17

from the ffi jar

micha21:12:10

i will try a thing

neupsh21:12:27

@micha can you eli5 that for me please πŸ™‚

neupsh21:12:10

sure let me know if you find a way to load it

neupsh21:12:19

@micha there is a stale directory in the target directory, which has a file named: leiningen.core.classpath.extract-native-dependencies and following content:

micha22:12:35

@nux which os are you on?

micha22:12:59

what happens if you do like find . -name 'libjffi-*.so' in the project dir after running lein repl

micha22:12:29

i imagine there will be a directory somewhere with DLLs

micha22:12:26

interesting ok

neupsh22:12:16

@micha, does boot use context class loader for isolating and loading the dependency classes at run time?

neupsh22:12:37

it uses SystemClassLoader, and i think system classloader in the jvm run by boot would not have those dependent libraries right?

micha22:12:30

@nux yes that's correct

neupsh22:12:19

@micha any suggestions on what can i do here?

micha22:12:45

i am experimenting now

micha22:12:54

adding the jar to the system classloader

neupsh22:12:26

@micha thanks a lot, let me know

micha22:12:53

having issues with clojure interop