Fork me on GitHub
#boot
<
2016-02-16
>
grounded_sage01:02:08

sorry for super nooby question but I can't seem to grok how this works. (deftask what "Specify a thing" [t thing THING str "An object" p pluralize bool "Whether to pluralize"] (fn middleware [next-handler] (fn handler [_] (next-handler {:thing thing :pluralize pluralize})))) (deftask fire "Announce a thing is on fire" [] (fn middleware [next-handler] (fn handler [thing-map] (let [updated-thing-map (next-handler thing-map) verb (if (:pluralize thing-map) "are" "is")] (println "My" (:thing thing-map) verb "on fire!"))))) :source http://www.flyingmachinestudios.com/programming/boot-clj/

grounded_sage01:02:15

I'm not sure what is being passed to what and since I can't simply print a line and have the actual function definition printed I'm finding it difficult to comprehend.

grounded_sage01:02:54

This is the command [ boot what -t "pants" -p -- fire ]

jethroksy02:02:51

anyone have success uberjar-ing with environ?

jethroksy02:02:49

hmm nvm i'll inject the variables when running the jar

richiardiandrea02:02:07

@jethroksy: yes I have had success

richiardiandrea02:02:53

yes of course you need export VAR= otherwise it won't work, I usually also merge with a default config

jethroksy02:02:51

@richiardiandrea: thanks!

jethroksy03:02:10

I have lots of environment vars... what a chore

richiardiandrea03:02:02

yeah that's true

richiardiandrea03:02:51

anyone getting any error using: (sift :to-resource #{#"version\.properties"}) ? Error:

clojure.lang.ArityException: Wrong number of args (1) passed to: task-helpers/sift-match/fn--1265

jethroksy03:02:55

that looks alright to me 😕

jethroksy03:02:31

sift-match looks like it requires invert as well

jethroksy03:02:42

so maybe try adding that option as well?

jethroksy03:02:57

there should really be a default value though

richiardiandrea03:02:13

yes that's right, but I will try

richiardiandrea03:02:10

clojure.lang.ArityException: Wrong number of args (1) passed to: task-helpers/sift-match/fn--1265
                                 ...                        
boot.task-helpers/sift-mv/fn/reducer  task_helpers.clj:   85
                                 ...                        
                     clojure.core/fn          core.clj: 6536
         clojure.core.protocols/fn/G     protocols.clj:  174
              clojure.core/reduce-kv          core.clj: 6562
             clojure.core/partial/fn          core.clj: 2500
                                 ...                        
                  clojure.core/apply          core.clj:  632
              clojure.core/update-in          core.clj: 5923
Looks like an error inside the reducer

jethroksy03:02:28

this is with :invert?

jethroksy03:02:22

do you know what some-fn refers to?

jethroksy03:02:29

(defn- sift-match
  [invert? regexes]
  (->> (map #(partial re-find %) regexes)
       (apply some-fn)
       (comp (if invert? not identity))))

jethroksy03:02:32

can't find it anywhere

richiardiandrea03:02:38

yes it is a Clojure standard function

richiardiandrea03:02:41

very tricky 😄

richiardiandrea03:02:54

I saw it first time here

jethroksy03:02:36

woah nifty! TIL

richiardiandrea03:02:31

will investigate tomorrow TIWE - today I worked enough 😄

jethroksy03:02:36

I'm still bad at reading code in the wild

richiardiandrea03:02:23

that one is tricky though

richiardiandrea03:02:19

generates a re-find function for all the regexes then passes it to some-fn

richiardiandrea03:02:39

the first who returns true short circuits

joshua.d.horwitz04:02:41

Are there any tutorials on how to build a blog site using Perun?

alandipert04:02:55

@joshua.d.horwitz: not that i'm aware of but there a few blogs on github, https://github.com/martinklepsch/martinklepsch.org is one

richiardiandrea04:02:07

But by the way does the above error works for you guys? is it just my problem?

grounded_sage04:02:18

Just reposting this question. sorry for super nooby question but I can't seem to grok how this works. (deftask what "Specify a thing" [t thing THING str "An object" p pluralize bool "Whether to pluralize"] (fn middleware [next-handler] (fn handler [_] (next-handler {:thing thing :pluralize pluralize})))) (deftask fire "Announce a thing is on fire" [] (fn middleware [next-handler] (fn handler [thing-map] (let [updated-thing-map (next-handler thing-map) verb (if (:pluralize thing-map) "are" "is")] (println "My" (:thing thing-map) verb "on fire!"))))) :source http://www.flyingmachinestudios.com/programming/boot-clj/ [11:08] I'm not sure what is being passed to what and since I can't simply print a line and have the actual function definition printed I'm finding it difficult to comprehend. [11:08] This is the command [ boot what -t "pants" -p -- fire ]

meow04:02:09

@alandipert: Is that enough info for you (or anyone) to help Wade?

meow04:02:34

@alandipert: Nevermind. Problem solved.

grounded_sage05:02:25

Feeling like I dove in way over my head with Boot. Will have to drop back to Leiningen for my projects until I am more familiar with Clojure.

alandipert06:02:10

@grounded_sage: the example is a little contrived - it's demonstrating the way boot works without doing anything related to building or running anything. what are you hoping to accomplish?

grounded_sage06:02:37

Was just looking at using Boot as it's just Clojure as a way of learning both Boot and continuing to learn Clojure. I am basically looking at building a simple static website using. Stasis, Hiccup, Garden, Mesh. Using live reloading of the browser when I make changes. Yes it's something really basic but I figured it would be the best way to get up and running with something that was still rather simple.

alandipert06:02:11

the big idea with boot is, the filesystem can be a value

alandipert06:02:20

once it's a value, you can pass and return it to/from functions

alandipert06:02:39

and then you're working with values and functions, and all of clojure can be brought to bear

alandipert06:02:55

so at a high level a boot "task" is a function that takes and returns a fileset

alandipert06:02:37

however it's convenient for tasks to be able to remember the values they saw so that they can cache/memoize... so boot tasks are actually functions that return functions

alandipert06:02:49

which is the nested fn stuff going on in the example

alandipert06:02:19

but it's also convenient to be able to parameterize this stack of 2 functions... so we do that with a function also. this what deftask is

alandipert06:02:42

in the example you pasted daniel is stringing together tasks with a map instead of a FileSet, which is the value passed between tasks in the real world

alandipert06:02:19

i hope that was helpful, but i gotta run now

alandipert06:02:34

for static websites, i would recommend checking out #C0JM5DUFL

jethroksy06:02:07

@onetom: I have a config that I'm very pleased with (with dev/user.clj) if you're interested I could show you

grounded_sage07:02:25

Thanks for the detailed response @alandipert most of that didn't make much sense to me haha. I'll come back to Boot another time when I am more skilled in Clojure

dominicm09:02:01

I'm trying to figure something out, fairly complex. And I think I'm 90% of the way there. I just need some advice on how to proceed.

dominicm09:02:52

I'm trying to track down why cider-nrepl isn't able to resolve the file location of symbols when the nrepl is run within a pod.

dominicm09:02:40

It looks like cider-nrepl is using fake.class.path to resolve the classpath, however fake.class.path doesn't include the classpath from within the pod. Any advice on where it should be pulling the classpath from?

dominicm10:02:38

I'm wondering if cider-nrepl middleware should infact be using clojure.lang.RT to get the classpath, and not get it from the system property directly? @micha @juhoteperi it looks like you guys are in charge of these tickets

onetom11:02:32

@jethroksy: of course i would be interested in it. thx!

mobileink14:02:24

Morning booters. I continue to have a very hard time understanding the Fileset API. This moring an insanely great idea for improving boot came to me. Here are some problems, as I see things. First, the Fileset API exposes way too much implementation detail. Take core/tmp-dir!, for example - it always creates an anonymous temp dir, so there is no reason it must be exposed to the developer. I just want my input stuff to turn into my output stuff, and I don't care what the mechanism is; provided I supply enough information, boot should be able to do the right thing however it pleases. Second, the API is not very clojurish. Take core/input-files for example. The beauty of the Fileset concept is that it treats the OS filesystem as just another database, and exposes functionality using defrecord TmpFileSet. That makes a TmpFileSet a clojure map - it should be possible to write (:input-files fs) rather than (core/input-files fs). Altering a FS should be done using ordinary Clojure idioms. With the API as it stands, the dev has to learn way too much about boot internals - make a tmp-dir!, make files, copy stuff, then do something like core/add-resource, etc. With a really Clojurized API it would be possible to use the standard map stuff. For example to copy foo/bar.clj to WEB-INF/classes/foo/bar.clj, something like:

mobileink14:02:28

(assoc fs :input-files (merge (:input-files fs) {"WEB-INF/classes/foo/bar.clj" "foo/bar.clj"}))

mobileink14:02:40

That gives boot enough info to do the right thing.

micha14:02:44

how would you have external tools create files?

micha14:02:15

by external tool i mean for example the cljs compiler

micha14:02:21

not external to the jvm

micha14:02:36

but code that doesn't know about filesets

micha14:02:11

the fileset api makes it possible to separate these concerns

mobileink14:02:44

Unfortunately this would require major rearchitecting - you'd have to switch from defrecord to deftype, and implement all the necessary interfaces. Fortunately, it turns out I've already done this for a very similar situation: the Google Appengine Datastore. I used defrecord to make gae datastore Entity objects behave exactly like Clojure maps.

micha14:02:55

you can totally implement an api on top of the fileset btw, i would be interested to see it

mobileink15:02:25

@micha: not sure yet how to deal with all case, but I'm sure with a little analysis a good solution would be found.

micha15:02:45

lol we've been analyzing it for years

mobileink15:02:51

Regarding implementation on top of the fileset: that's exactly what I want to avoid. Instead of an API layered on top of Fileset, better to define the Fileset API using protocols. That way the FS api is just the Clojure API - no special fns needed.

micha15:02:21

it's already implemented as protocols, which in hindsight i think was overengineering

onetom15:02:40

i just came across http://projects.spring.io/spring-boot/ and wondered if it has anything to do w boot... simple_smile i saw it mentioned in this rather aggressive remote job post: https://remoteok.io/remote-jobs/15499-remote-clojure-developer-restaurantzone is it any good!? how come ppl are using it together w clojure?

micha15:02:14

i think spring-boot is something completely different simple_smile

mobileink15:02:00

I think the knowledge you've gained analysing stuff would carry over. Protocols: I mean Clojure's interfaces, e.g. ILookup, IPersistentMap, etc. The goal would be to make Fileset look and act like any other map, with implementation details hidden.

micha15:02:43

it does already, since TmpFile is a record type

micha15:02:55

it implements those protocols

micha15:02:45

i think actual demo code would be useful

micha15:02:03

the current system is built on years of actual production use

mobileink15:02:20

Right, but the idea is to use a custom implementation so that standard operations like assoc can do specialized boot work. Using defrecord you only get a standard implementation. You can see an (unfinished) example at https://github.com/migae/datastore .

micha15:02:42

it's difficult to talk about a fictional thing when there are many subtle concerns that are being balanced

mobileink15:02:52

Understood 😉

micha15:02:12

so i recommend that you build a thing on top of the existing api, so we can compare in a productive way

mobileink15:02:33

a simple example: to support (:input-files fs), you have to implement containsKey, which is in the clojure.lang.Associative interface. Then you can return whatever you want, your deftype need not actually contain the :input-files key. The whole thing is virtualized. I'm not sure if you can override such interfaces with defrecord; I think you have to use deftype.

mobileink15:02:56

Not sure how soon I'll be able to write any demo code, just wanted to put the idea forward for consideration.

micha15:02:34

i don't see the advantage of this over plain functions

micha15:02:41

it's one more character to type, even

micha15:02:47

(input-files fs)

mobileink15:02:36

Sorry, I think its valAt, which is in ILookup. The advantage is that there is then no need to learn an API. You just use Clojure.

micha15:02:37

i'm kind of in the mindset where i think using protocols and record types in the first place was overengineering, i think it made things more complicated than they need to be

micha15:02:54

how do you know which magical key to use?

micha15:02:07

like how do you know that :input-files will do some special thing

micha15:02:19

you read the api docs, just like what we currently have

micha15:02:32

only now it's just a function, which you are free to use or not use

micha15:02:39

it's not all wrapped up in types

micha15:02:58

i think this is more lispy than using protocols

mobileink15:02:12

Fileset does not have an :input-files key. You implement valAt to return the same thing core/input-files returns. That's just a simple example.

micha15:02:24

right but i don't see the advantage

micha15:02:40

you still need to type the same thing, just with an extra :

micha15:02:01

only now there is an extra layer of infrastructure

micha15:02:08

instead of just functions that operate on data

micha15:02:34

like (input-files fs) vs. (:input-files fs)

micha15:02:44

i prefer the former, i think

micha15:02:26

with the magical key thing you will need to implement every permutation of selector in there

micha15:02:43

or you will have some hybrid thing where sometimes you use functions and sometimes special magical keys

micha15:02:55

again i think this would be worse than what we already have

micha15:02:02

since now it's all just functions

mobileink15:02:17

Matter of taste, I guess. I personally would very much prefer to be able to do everything I need to do with Filesets using nothing more that assoc, merge, cons, etc. - just the Clojure api, with the addition of some kind of commit! operation.

micha15:02:24

like what if i want all the input files that have the .cljs extension

micha15:02:48

then i need to use a combination of functions and special magic

micha15:02:12

so you'd need to know more internals to know which things are implemented magically and which things are just functions

mobileink15:02:51

(filter #(str/ends-with? # "cljs) (:input-files fs)), something like that.

micha15:02:02

basically every time i've used protocols in boot i've sort of regretted it, i think it could have been simpler if i hadn't jumped to overgeneralize

micha15:02:19

right well input-files itself is a filter

micha15:02:29

so now you have some filters that are implemented as special keys

micha15:02:36

and some which are implemented as functions

micha15:02:51

which is more complicated than just having functions, imho

mobileink15:02:06

keys are functions 😉

micha15:02:17

not when they're implemented via valAt or whatever

micha15:02:35

you can't call apply on it

micha15:02:39

for one thing

micha15:02:53

you can't use partial on it

micha15:02:11

it becomes a syntactic form almost like a macro

micha15:02:17

which i think is a regression

micha15:02:56

i guess i just fail to see how it's better than functions

micha15:02:49

like even if it's no worse, it would need to be better to justify adding another layer of infrastructure on top of simple data and functions

micha15:02:43

especially since it is totally feasible to implement all of this on top of the existing api

micha15:02:46

as a library

mobileink15:02:51

Keywords implement IFn. (apply :foo [:foo 9]) => 9

micha15:02:00

i concede that it may be no worse, for argument's sake simple_smile

mobileink15:02:36

In any case I don't mean to be argumentative, but I don't think what I have in mind could be implemented as a layer on top of Fileset, although I'd have to look into it a bit more to be sure.

micha15:02:00

you can use the proxy pattern i think it's called

micha15:02:06

you wrap the existing thing in your type

micha15:02:14

then you can do whatever you need to do

mobileink15:02:26

For me, it all comes down to ease of use for devs - as I mentioned, I'm finding it quite difficult indeed to understand the API.

micha15:02:00

(deftask foo
  [...]
  (with-pre-wrap [fs]
    (let [better-fs (wrap-better fs)]
      ...

mobileink15:02:07

I guess if I can find some time the thing to do would be to fork an experimental version of boot just for exploring ideas.

micha15:02:24

then you can have all the protocols implemented on the object that wrap-better returns

micha15:02:40

there is no need to fork boot to experiment

micha15:02:39

i meanthe existing implementation works, it's functionally sound

mobileink15:02:42

Sure but in that case you'd basically end up reimplementing the whole thing. I don't mean fork as in a competing implementation, I just mean one the I don't worry about keeping in sync with master.

micha15:02:52

if you want a different api you can do that via wrapper object

micha15:02:11

i don't see why you'd need to mess with the internals at all

micha15:02:22

what would you need to reimplement?

micha15:02:42

the existing thing is well factored and minimal

micha15:02:00

there shouldn't be any need to rip any existing functionality out, becuse every part is needed

micha15:02:18

so for adding additional functionality a wrapper is ideal

micha15:02:38

and it can be scoped to the task, so it's opt-in

mobileink15:02:39

ok for expermenting you're right, i could have my deftype just forward operations to a wrapped Fileset.

micha15:02:00

yes and you maintain compatibility with all the existing tasks that you don't want to have to rewrite

micha15:02:20

i don't see any downside at all

mobileink15:02:25

yeah, that's definitely the way to go.

micha15:02:34

you can even make your own with-pre-wrap macro that does the wrapping and unwrapping for you

micha15:02:52

or with-wrapped-fs macro

micha15:02:43

i think you will find that the simplicity of the existing thing will make this approach way easier than if it was already all wrapped up in protoocls

micha15:02:55

which is what we really want to achieve

micha15:02:22

boot should be a framework that is customized by the user to make it perfect for them

mobileink15:02:28

that way i could implement just bits of functionality, like the example i gave above for copying: (assoc fs :input-files (merge (:input-files fs) {"WEB-INF/classes/foo/bar.clj" "foo/bar.clj"}))

micha15:02:38

there is a cp function in bot.core to copy files

micha15:02:47

which you can use internally

micha15:02:01

or just see what it does at least

mobileink15:02:29

Glad you mentioned that, I wanted to ask about it. I don't understand the doc and tried without success to use it. Is using core/cp preferable to using io/copy?

micha15:02:47

i don't see how you would use io/copy

micha15:02:32

oh, that has nothing to do with the fileset really

micha15:02:42

that's for copying files to named directories

micha15:02:03

so the copies are not part of the fileset and not known to or managed by boot

micha15:02:38

the cp function is for copying fileset entries

micha15:02:58

it's similar to git cp or git mv

micha15:02:43

since the fileset is transactional, like git

mobileink15:02:52

what i find confusing is that the destfile is supposed to already be in the fileset.

micha15:02:33

if you want to copy to a new file you'd do that by copying the file in the temp dir and then adding that

micha15:02:01

which is the most common use case

micha15:02:53

i think the fileset is confusing in the same way git is confusing

mobileink15:02:01

if it's already supposed to be in the fileset how did it get there? copying is how you put something new in a directory; why is it differnt in a Fileset?

micha15:02:23

the cpfunction copies the contents from one tmpfile to another

micha15:02:09

like git is confusing, but it's extermely difficult to produce a better alternative when all concerns are considered

micha15:02:29

the fileset stuff has the same kind of stuff

micha15:02:39

so yes it's not as simple as io/copy

micha15:02:01

but when all concerns are considered i have not been able to simplify it any further

mobileink15:02:05

oy. how i hate java. ;)\

micha15:02:16

the fileset is different than the filesystem in the same way that the git tree is different from the working set

micha15:02:42

i'm not a git expert though so i am talking out my butt a little

micha15:02:49

but in order for git to provide the transactional commits etc it needs to do things differently than you'd do them with pure mutable filesystem objects

micha15:02:10

filesets are in the same boat there

mobileink15:02:04

what am I doing wrong here:

mobileink15:02:08

(core/with-pre-wrap [fs] (let [tgt (core/tmp-dir!) in-files (->> fs core/output-files (core/not-by-ext [".clj"]) (map (juxt core/tmp-path core/tmp-file)))] (doseq [[rel-path in-file] in-files] (let [out-file (doto (io/file tgt (str rel-path "/foo")) io/make-parents)] (println " in-file: " in-file) (println "tmp out-file: " out-file) (core/cp fs in-file out-file))) (core/add-asset fs tgt) (core/commit! fs))))

mobileink15:02:22

I get No implementation of method: :dir of protocol: #'boot.tmpdir/ITmpFile found for class: java.io.File

mobileink16:02:12

the mix of http://clojure.java.io and boot api stuff in examples and in the source is a little perplexing - i never know when to use what

micha16:02:31

you don't want to use core/cp there

micha16:02:57

the second argument to core/cp needs to be an existing tmpfile object

micha16:02:04

not a io.File

micha16:02:18

you want to use io/copy there

micha16:02:38

copy the file in the tgt dir

micha16:02:45

to another file in that dir

mobileink16:02:48

ok, but in-file was created by core/tmp-file, wasn't it?

micha16:02:08

core/tmp-file is just an accessor

micha16:02:12

a protocol, in fact

micha16:02:30

it extracts the anonymous io.File object for the contents of the file

micha16:02:53

you don't know or care where it is or what its name is

mobileink16:02:02

doesn't that make it - oops, just reread the doc, tmp-file creates a java.io.File - that's surprising, given the name of the fn

micha16:02:06

it's just an opaque File object whose contents you may read only

micha16:02:24

tmp-file does not create a file on the filesystem

mobileink16:02:31

so core/tmp-file really means core/tmpfile->javafile

micha16:02:59

it merely provides a io.File object corresponding to a file in some immutable, content-addressed storage area

micha16:02:04

which is why it's read-only

mobileink16:02:06

right, sorry, tmp-file returns the java.io.File associated with an existing Tmpfile

micha16:02:14

if you write to that file you will corrupt the fileset

micha16:02:49

you can't save that File object or do anything with it that spans multiple transactions

micha16:02:53

it's ephemeral

micha16:02:07

it's like a temporary handle for reading the contents

mobileink16:02:19

ok, do you have a simple example illustrating correct usage of core/cp? i remain thoroughly confused.

micha16:02:33

the next time you get the fileset and you do tmp-file on it, you will find a different io.File object

micha16:02:52

i think you don't want to use core/cp at all

micha16:02:05

it's only useful for some internal things that the fileset does

mobileink16:02:16

above yousaid cp copies from on tmpfile to another

micha16:02:25

the contents

micha16:02:49

so if you have two tmpfiles in the fileset and you want them to contain the same bytes, you can use that function

micha16:02:44

if you use io/copy to read from ont tmp-file and write to another you will corrupt the fileset

micha16:02:51

which is why that function exists

micha16:02:32

the basic rule is that the io.File objects you obtain from the fileset api are read-only

micha16:02:49

immutable, in fact

dominicm16:02:03

@micha I catch you. I'd love to pick your brain on fake.class.path a little later. I know you're busy now, and two conversations is too many 😛

micha16:02:30

@dominicm: i have been thinking about trying spacemacs again

mobileink16:02:40

do you have a simple example of how to copy stuff from source to target using just filesets?

micha16:02:56

@mobileink: it's not possible

micha16:02:32

that's like asking "how to copy stuff from a git repo to arbitrary directories using just git commands"

micha16:02:42

it's not for that

micha16:02:41

filesets are for a single specific purpose: providing a transactional, immutable interface on top of and alongside the actual filesystem, so you can control what is on the classpath

dominicm16:02:56

@micha: 😞 Vim is life.

micha16:02:42

named files on the filesystem are beyond the scope of the fileset api, as it only deals with anonymous files

micha16:02:16

i think this provides a very useful and necessary separation of concerns

micha16:02:25

however there are some functions for dumping the fileset to named places

mobileink16:02:51

let me rephrase the question: what is the cannonical way to write a task whose job is to copy stuff, in such a way that the task can be inserted in a pipeline of tasks? There must be an example somewhere. Can you recommend a third party lib that does things the boot way?

micha16:02:08

sure. the sift task does all those things

jethroksy16:02:35

btw since we're on sift

micha16:02:38

the target task copies to out-of-fileset places

mobileink16:02:05

yes, i've used sift, but what if i want to chain two copy tasks, e.g. one for css and one for jpeg. The first sift will remove the files needed for the second task, no?

micha16:02:45

it'll only remove what you tell it to?

micha16:02:13

@jethroksy: looks like a bug

micha16:02:16

i can't reproduce it though

jethroksy16:02:43

error occurs in the reducer

mobileink16:02:47

i'm looking for how to write copy tasks that can be chained, i don't see now sift can be used for that.

micha16:02:24

i think that might be a regression with the new sift code?

jethroksy16:02:25

cant reproduce either

mobileink16:02:36

anyway i've used up too much of your time. if you or anybody else can come up with sample code i would be grateful.

micha16:02:38

i know he was fixing bugs and improving the sift task

micha16:02:50

so maybe some regression surfaced

jethroksy16:02:09

hmm yeah maybe it's on his code

micha16:02:26

@mobileink: can you make a gist of the paste you pasted?

jethroksy16:02:30

@richiardiandrea: ^

micha16:02:34

i can show an example

mobileink16:02:56

@micha: not sure what you mean "the paste i pasted"?

micha16:02:09

oh, you pasted a task

micha16:02:19

i thought that was what you were talking about

micha16:02:28

if you gist what you've got i can help

micha16:02:44

like the css copying task

mobileink16:02:27

what i'm trying to do is break down the copying (for gae) to separate tasks, one for each file type. probably not necessary but i like small composable things

mobileink16:02:22

the copying i can manage, it's making everything pipeline-able that's giving me trouble

jethroksy16:02:17

I used to do that for my css files

jethroksy16:02:06

this was quite a while ago

micha16:02:14

@mobileink: commented on the gist

jethroksy16:02:26

i think i supplied :dir wrongly

mobileink16:02:30

@micha: on a completely unrelated note, I spent way too much time this morning finding out that I had written (core/by-ext ".clj") instead of (core/by-ext [".clj"]). So I added preconditions to the by-foo stuff in core.clj, e.g. {:pre [(vector? exts)]} for by-ext. Do you want a PR?

micha16:02:50

sure, although (seq exts) may be more flexible

micha16:02:09

or something, like it may not be a vector, it could be a list or sequence

alandipert16:02:04

probly sequential? for that, so maps would explode

micha16:02:33

but then you have sets

micha16:02:50

you should be able to provide a set there, too

alandipert16:02:46

(and (seq x) (every? string x))

micha16:02:47

basically if you call seq on it you should get a seq of strings

dominicm16:02:55

Schema anybody?

micha16:02:58

bingo i think that is a winner

dominicm16:02:28

@micha regarding fake.class.path, did you see my messages earlier?

micha16:02:44

i did, but i'm not familiar with cider internals at all

micha16:02:59

fake.class.path is a system property

micha16:02:17

so there is only one instance of it across the entire jvm

micha16:02:34

the classpath you want is in boot.pod/env

micha16:02:58

in the pod

micha16:02:30

@dominicm: you can use System/setProperty to make an even faker fake.class.path

micha16:02:56

i think that would be fine since its only purpose is to support things like cider

micha16:02:05

so clobbering it should be okay

dominicm16:02:52

Right, but for external tools, you guys were advocating the use of fake.class.path for external tools. Isn't a System property JVM-wide? Because there is other REPLs in that script, and they need access to their files also.

micha16:02:06

then you can't use system properties

micha16:02:20

you would need to change cider

dominicm16:02:29

I'm wondering if advocating fake.class.path might be the wrong approach here (in general for external tools), and instead using clojure's provided classpath is the better approach.

micha16:02:33

cider doesn't support more than one classpath at a time, basically

dominicm16:02:05

I agree, but in issue https://github.com/boot-clj/boot/issues/249 you guys advocate fake.class.path, which is why I'm bringing this up 😛

micha16:02:30

yeah i mean that was a hack for compatibility with existing tooling

micha16:02:05

boot is the only build tool that has pods, as far as i know, so nothing is designed with that in mind

dominicm16:02:22

Yeah, for sure.

micha16:02:55

you could use a template like fake.class.path.<podname> system property

dominicm16:02:58

I mean, I think the best hack (and stuart sierra caught onto this it seems, always ahead of us!), is to get the class loader from clojure, and derive the classpath from that.

micha16:02:12

then it would be relatively straightforard to parameterize the property cider uses

micha16:02:32

cider could even get the pods name from the runtime via the boot api

micha16:02:42

and igure out which property to use

dominicm16:02:53

That should theoretically always be correct, because even if people use a different "pod" library, clojure's classloader should be set correctly.

micha16:02:06

yes that's correct

micha16:02:29

but that doesn't tell you anything about where the source files are

micha16:02:32

necessarily

micha16:02:49

if you are just manipulating the source files directly then it's simple

micha16:02:02

but with filesets the two are decoupled

micha16:02:12

that's the issue with cider i think

micha16:02:23

it needs the fake classpath, not the real one

dominicm16:02:41

The fake classpath is the real src directory right? As opposed to the compiled one?

micha16:02:53

it's the one that belongs to the user, in their project

dominicm16:02:02

I'll double check, but I think that is set in the clojure.lang.RT classloader.

micha16:02:04

boot doesn't touch those except to read them and copy them into the fileset

micha16:02:19

boot has no facilities for doing anything to your files

micha16:02:28

only to its own files that it manages itself

micha16:02:55

like suppose you have :source-paths #{"src"}

dominicm16:02:58

Yeah. Which is good.

micha16:02:05

the files in that directory are not on the classpath

micha16:02:18

copies are made and impoted into the fileset

micha16:02:27

it's those files that are on the actual classpath

micha16:02:47

but it's possible to work backwards in some cases

micha16:02:53

or perhaps most cases even

micha16:02:13

to sort of draw correspondence between things in the fileset and the project files

micha16:02:26

but boot allows arbitrary modification of any file in the fileset

micha16:02:31

in fact that's something i do all the time

micha16:02:42

i have tasks that rewrite or add to my source files

micha16:02:51

so those will not map to anything in my project

micha16:02:21

basically the concept of banging on text files in editor plugins is not so compatible with boot

micha16:02:37

but boot provides the capability to abstract that out

micha16:02:44

and do it in the pipeline instead

micha16:02:51

leaving your source code out of it

dominicm17:02:08

For sure, I mean, the thing I'm trying to fix is libraries.

dominicm17:02:35

In fireplace/cider/misc, you can go to the source file of a library, but that is broken, because the library isn't on the fake classpath.

micha17:02:38

for something like refactoring it would make sense to me to do that in a pipeline

micha17:02:48

and emit altered source files to a target dir

micha17:02:52

where you can diff them

micha17:02:15

mangling files in place seems like you will end up with many hacks needed

micha17:02:54

yeah i think for the jump to file business you can just set the fake class path

micha17:02:07

so as you go from pod to pod you are ressetting it to the value in that pod

micha17:02:11

i mean as a workaround

dominicm17:02:41

http://pastebin.com/u6ggjNtw this is the result of clojure.RT.baseLoader. Yeah, that makes sense. I'm more than happy to contribute back to cider/fireplace/other tools.

dominicm17:02:06

I think it's important to decide on a solution that works for as many cases as possible for 3rd party tools to integrate with.

micha17:02:33

perhaps a function in boot.pod that will overwrite the properties

micha17:02:46

you can call it from the cider init code

micha17:02:07

so presumably it could be abstracted away and hidden in cider

dominicm17:02:36

I do agree that some of the refactoring make more sense elsewhere. But some minor ones make sense in the editor, so it's complicated really. And should you lose refactoring just because you're working on an ant/lein project now?

dominicm17:02:20

I don't think that you think we should, just, it's a hard problem really. And I think when it comes to editors, convenience is king, in practice

dominicm17:02:46

But anyway, the problem with the cider init code, is that it's lost in child pods, see: https://github.com/juxt/edge/blob/master/build.boot#L64 I have to resolve the default middleware in the parent pod, and pass it down to the child.

dominicm17:02:04

But yeah, my clojure.lang.RT has the original source paths. Which is what fake.class.path provides. I don't think children pods were picking up the source files unless I set the directories to include the source paths (which makes sense as children pods aren't part of a fileset really). But then that change would break parent pods, agh, difficult.

richiardiandrea17:02:16

@jethroksy: @micha it was a regression in my code, but it is weird because BOOT_VERSION=2.5.5 should not have my new changes...

laforge4917:02:39

But who still uses 2.5.5? simple_smile

micha17:02:18

@dominicm: how does clojure.RT have the actual project directory paths?

micha17:02:29

boot shouldn't be adding those to the classpath

micha17:02:46

it should be adding its own internal tempdirs instead

dominicm17:02:43

@micha so if you check the source I linked, I specify the directories. The source paths weren't included to the child pod, and I haven't seen an option to retrospectively run boot against a new list of source paths.

dominicm17:02:59

Maybe there is and I missed it

micha17:02:16

i mean i think you don't want to put project dirs on the classpath

micha17:02:25

that would cause problems

micha17:02:37

because there would be duplicated files in there

richiardiandrea17:02:55

How can I add a file the project root as resource? Is it idiomatic to use (add-resource fs (java.io.File. ".") :include what-goes-here?) ?

richiardiandrea17:02:14

ok the following works: (add-resource (java.io.File. ".") :include #{#"^version\.properties$"})

dominicm18:02:48

@micha the compiled directories aren't on the path though

micha18:02:26

right you aren't building anything in there

micha18:02:42

so there isn't any conflict with tasks that create files or anything

dominicm18:02:44

@micha Nope. But there would be if there were reprocessing in pods. I'd like to create a fully working system where you can isolate things within a task if you please.

dominicm18:02:43

I think that would be an ultimate display of power

dominicm18:02:18

I'm happy to fork projects and send PRs to make that happen.

micha18:02:29

the problem is that in order for boot tasks to be able to do what they do, they need to not need to know or care about the provenance of the files they're manipulating

micha18:02:43

that means that they treat all the files the same

micha18:02:02

once you have files owned by the user in there, the whole tower collapses

micha18:02:13

because now tasks need to know which files are which

micha18:02:29

so now you have many named places that all tasks need to recognize and handle specially

micha18:02:50

and some things simply become impossible, like tasks that modify .cljs.edn files for instance

micha18:02:13

those would be irreversible changes, which would break the fileset abstraction

micha18:02:22

i may be misunderstanding you though

dominicm18:02:25

I don't think I totally understand. I agree with what you're saying, but I don't understand how what I'm trying to achieve conflicts with that.

micha18:02:54

i mean adding the project directories to the classpath

micha18:02:08

that isn't really compatible with filsets i don't think

dominicm18:02:02

Essentially, I'm saying that it should be possible to isolate the dependencies of any task to it's bare minimum (as in, excluding the parent, instead of extending). This is obviously useful for clojurescript and clojure separation within a single jvm, but I'd love to make it generic.

micha18:02:32

haha i thought we were talking about the fake class path stuff

micha18:02:16

2.6.0 should have some version of with-env i think, how do you feel abot that?

micha18:02:08

or is it percolating into a more complete solution?

dominicm18:02:15

I do agree that having the source paths in the pods classloader is a little crude of a solution. It would be really awesome to load boot in boot, and have a new fileset running within a pod. I need to contribute a fix to the repl task for with env, because it has the delay in reading the environment, and therefore can't be podded directly yet.

micha18:02:10

i'll make a ticket about fixing all the places where that kind of problem may lie

dominicm18:02:20

But I may have discovered an issue with podding and the repl task in general, because the default middleware isn't passed in from the parent.

dominicm18:02:44

Because you get a new copy of boot (and it's state) in the pod.

micha18:02:01

i think the real problem is with the way nrepl is organized

micha18:02:14

the idea fix would be to run nrepl itself in a pod

micha18:02:30

such that it communicates with the session environment over a socket for example

micha18:02:49

it's the mixing of nrepl into your pod that is causing issues

micha18:02:04

like the repl client runs in its own pod

micha18:02:18

and that's fine because it communicates with the backend over a socket

micha18:02:34

it would be ideal to have the server be in its own pod too

micha18:02:46

then none of these problems would exist

micha18:02:22

not sure how to do it, or if it's even possible though

dominicm18:02:34

There will need to be a way to pass information into that pod (default dependencies and middleware), for the cider middleware to run

micha18:02:37

seems to be similar to a cljs repl though in a lot of ways

micha18:02:53

right totally

micha18:02:08

that would be simple because you'd just make some functions in boot.core for that

dominicm18:02:17

Well, I've got it running in a pod, so I know that it's possible.

micha18:02:27

no i mean in its own pod

micha18:02:36

like you have pod A where your code is

micha18:02:51

currently when you start the repl it loads nrepl into pod A

micha18:02:00

and creates pod B where the repl client is loaded

micha18:02:18

the repl client communicates with nrepl over a socket, so it doesn't need to be in pod A

micha18:02:34

reply has some hairy dependencies, so it's good that it's in its own pod

micha18:02:48

nrepl doesn't have dependencies but it would be good if it was in pod C

micha18:02:00

which could be separate from all of the other pods

micha18:02:16

if it could compile expressions and send them over a socket to be evaluated

dominicm18:02:21

Right, but check edge, I create a pod for the nrepl server to run in. In pod C. I have to subvert the repl task though

micha18:02:23

like via a clojure.main repl

micha18:02:35

whoa really?!

micha18:02:41

this is the holy grail of repls

micha18:02:55

i was banging my head against that for a long time simple_smile

dominicm18:02:17

Yeah! It's epic. I hope we are talking about the same thing now.

dominicm18:02:19

Yeah, have a quick look if you can. The server task creates a pod, and then starts a repl in it.

micha18:02:59

can you do (require 'nrepl.server) for instance in the repl?

micha18:02:19

like with the pod C approach, nrepl itself wouldn't be on the classpath

micha18:02:27

just like REPLy isn't

dominicm18:02:46

Oh, I see! Even more isolation. THAT would be cool.

micha18:02:00

because then you could have only one nrepl

micha18:02:04

and many clients

micha18:02:08

in different pods

dominicm18:02:32

Yeah, that would be awesome. I understand what you mean now. Yeah, socket repl should make that possible.

micha18:02:03

i am imagining the repl server generating expressions that it sends to clojure.main/repl in the pod where you want to compute

micha18:02:16

instead of calling eval itself

dominicm18:02:56

I wonder if nrepl can consume socket repl as a client maybe?

micha18:02:09

it can, but the problem is at the other end

micha18:02:22

i mean REPLy, the default client, uses a socket

micha18:02:36

but nREPL currently calls eval directly

micha18:02:59

which means the nrepl code needs to be on the classpath in the pod where you want to compute

micha18:02:57

i guess what i am imagining is moving all of the nrepl stuff into the client maybe

micha18:02:10

and the server is just clojure.main/repl

micha18:02:42

it seems to make sense to me to add middleware in the client anyway

micha18:02:45

not the server

micha18:02:55

because what if you have different clients with different mniddleware

micha18:02:02

that seems like a thing you'd want to do

micha18:02:31

and you want to store state per client most of the time, i imagine

micha18:02:47

because otherwise you can def a var or whatever

micha18:02:01

if you want state to be visible across all clients

dominicm18:02:13

I bet you could write some nrepl middleware that overrides ecal

dominicm18:02:35

Which calls to the socket repl

micha18:02:37

i wonder if it's feasible to just rewrite nrepl

micha18:02:02

make the middleware into a pipeline instead of a graph

micha18:02:17

simplify things

dominicm18:02:19

Problem there is being compatible with cider, losing cider would be a huge barrier to use in my opinion

dominicm18:02:00

I think most people would just use the built in repl without it.

micha18:02:07

i hear there is a simple inferior lisp situation that is becomig popular

dominicm18:02:41

I'm not sure I understand :p

micha18:02:55

in emacs, a simple repl

micha18:02:02

as an alternative to cider

micha18:02:40

i use vi myself, with fireplace

micha18:02:51

i'm very happy with it

dominicm18:02:53

Oh. I see. Fireplace uses cider.

dominicm18:02:59

I'm a vimmer too

micha18:02:18

i only use the most basic parts of it

dominicm18:02:29

Well, it uses cider nrepl middleware, which is what I really care about.

micha18:02:35

thngs you could do with a tmux inferior lisp thing in neovim

micha18:02:04

like eval an expression, paste the result somewhere, get docstring

micha18:02:18

view source

dominicm18:02:23

Ah, I find the ability to jump to symbol really handy, plus the namespace cleaning is mandatory on one project

dominicm18:02:38

I mean, jump to the source of a symbol.

micha18:02:40

that can all be done without needing access to source files, just from the classpath

micha18:02:56

i use ctrlspace for that

richiardiandrea18:02:03

sorry to put a question there, but is there a way to set BOOT_VERSION from build.boot?

micha18:02:27

@richiardiandrea: unfortunately, no, unless you are using boot-in-boot, in which case yes

richiardiandrea18:02:39

I am using it 😄

micha18:02:52

sure just set the ssytem property

micha18:02:05

hm actually no

micha18:02:11

that's nonsense

dominicm18:02:16

Doesn't work for libraries, plus in a 750 line file you're not familiar with, I hate figuring out what the alias is.

micha18:02:31

there can only be one boot.App on the classpath, and you can't put it in a pod

micha18:02:42

because it's the source of all pods

micha18:02:51

so it must be above them in the classloader tree

dominicm18:02:09

@richiardiandrea I may be missing something, but is there something wrong with boot properties?

micha18:02:21

however, you could make it work with some classloader shenanigans

richiardiandrea18:02:56

nothing wrong simple_smile now the way to force the version for tests in my patch is to use a boot.properties, but I was thinking of reading the version from the main version.properties

richiardiandrea18:02:09

in the boot's project root

dominicm18:02:34

Ah, I see. Higher level. Just wanted to double check you hadn't overlooked the obvious solution.

dominicm18:02:01

@micha, we got distracted by the ultimate isolation of the nrepl, I need to come up with a variable that matches these properties: - points to the fake classpath - always correct within a pod The latter is the blocked one currently. How is it best to solve this do you think? I'll patch cider nrepl to use it.

micha18:02:36

you can look at the set-fake-class-path! function (i think that's what it's called) in boot.core

micha18:02:15

i think something that does the same kind of thing can be added to boot.pod perhaps

micha18:02:55

in boot.pod/env i think you should have all the information you need

micha18:02:17

so perhaps add :fake-class-path to the env?

micha18:02:39

that seems like an ok place to put it, since it is fundamentally a description of the environment

micha18:02:59

maybe make-pod should do that

dominicm18:02:14

But it's it okay for the parent pod to set that in it's environment? Instead of as a property? That way getting it from the environment would be reliable (once get env is patched to use pod/env in boot)

dominicm18:02:45

That might work really well actually.

micha18:02:13

yeah totally

micha18:02:27

and we could set the property from that for backwards compatibility etc

dominicm18:02:07

Boot having the classpath set at the jvm level makes less sense when you consider the theory behind pods.

dominicm18:02:42

I'll put something on my organisation system to tackle these tomorrow.

dominicm18:02:48

Or at least begin to.

richiardiandrea19:02:09

What if I set boot.App/config("BOOT_VERSION", "2.6.0-SNAPSHOT") at runtime? will the next boot in boot see it?

micha19:02:32

yes but they share the same boot.App

micha19:02:48

a different version of boot will need a different version of that

micha19:02:07

if you really want to i think it can be done

micha19:02:17

but you'd need to get into some classladers and whatnot

richiardiandrea19:02:29

I wanted to be clever, but I think for now I will leave a static boot.properties

richiardiandrea19:02:50

and tests will be called in the Makefile I guess?

richiardiandrea19:02:49

I can generate a boot.properties in the Makefile

richiardiandrea19:02:59

before launching the tests

richiardiandrea19:02:23

I don't even need that, I will just set the BOOT_VERSION env var

laforge4920:02:27

If I add a .cljs.edn file to a library project and put build-jar in the build-boot file before build-jar, then a .js file is created and added to the jar file. I want to use this .js file with a web worker in an application. Separate compilation is not an issue here, as the web worker does is not running shared. Now I figure the .js file needs to be moved into a resources directory in the jar file so that it will be loaded? I'm a bit lost at this point.

micha20:02:59

@laforge49: you can do something like this

micha20:02:19

(require '[ :as io[)

(deftask foo
  [...]
  (let [tmp (tmp-dir!)]
    (with-pre-wrap [fs]
      (io/copy (io/input-stream (io/resource "foo/bar.js")) (io/file tmp "foo" "bar.js"))
      (-> fs (add-resource tmp) commit!))))

laforge4920:02:10

Looks like it is time I dig into boot. simple_smile

laforge4920:02:31

This task would be added to the library's dev I take it?

micha20:02:54

the sift task can also do it

laforge4920:02:17

I REALLY need to start reading up on boot! 😄

richiardiandrea21:02:11

Hello again guys, how can I see if a file in a fileset is a "resource"?

micha21:02:53

@richiardiandrea: you can do this:

micha21:02:36

((set/intersection (input-files fs) (output-files fs)) the-tmp-file)

micha21:02:48

if that's nil then it's not

richiardiandrea21:02:10

so source files are usually shown in input-files right?

micha21:02:31

well there are 4 permutations

micha21:02:40

of input/output roles

richiardiandrea21:02:45

ah yes i remember the table 😄

micha21:02:52

"source", "resource", "asset" etc are just names for those

micha21:02:14

resource files happen to be the ones with both input and output roles

micha21:02:26

but usually tasks don't care about that

micha21:02:39

they are either reading from input files or writing to output files

micha21:02:49

i mean packaging output files

micha21:02:58

not writing to them, they're read-only simple_smile

richiardiandrea21:02:40

yes but my tasks are peculiar 😄 I am trying to test to-resource or to-assets

micha21:02:36

ah, of course

micha21:02:58

so you can use the set operations to select the files with the roles you are interested in

richiardiandrea21:02:08

trying, let's see 😄

jannis21:02:08

When I use boot-reload and serve two HTML files with some ClojureScript, is there a way to run separate :js-onjsload functions for each of them?

jannis21:02:36

The reason I'm asking is that I have a devcards.html and an index.html (serving my app). When I update devcards, boot-reload keeps on calling the :on-jsload in my app, which then errors out because it tries to mount my Om Next app from app.js into devcards.html...

jannis21:02:58

The only quick solution I can think of is to have separate boot tasks for running the two.

jannis21:02:51

Hmm, :ids might do the trick?

juhoteperi22:02:42

A real solution would be to deprecate on-jsload and fix the reload logic to always reload dependent namespaces (if foo.a changes and foo.main depends on that, foo.main is always reloaded, and it would contain the app "reset" code)

jannis22:02:19

Sounds reasonable

jannis22:02:54

:ids #{"app"} simply disables all reloading for devcards.cljs.edn. That's certainly not what I want. 😉

jannis22:02:28

@juhoteperi: How hard would doing what you suggested be?

jannis22:02:32

Quick fix: in the app, check if the last path segment of JS window.location.pathname starts with devcards and don't run any reload logic if that's the case.

juhoteperi22:02:35

@jannis: It would be somewhat complex as proper dependent namespace reloading requires complete dependency graph from somewhere. I think the best solution will be to reuse Figwheel client code which reads the dependency graph from Closure files and fixes several other problems.

jannis22:02:33

Ok. Too complex for taking a quick shot at it then. No problem.

juhoteperi22:02:37

It is on my todo list but I'm busy with other stuff currently so it will be some time before I get around to it.

jannis22:02:21

Ah, cool. That's good to know though. I'm looking forward to it simple_smile

jannis22:02:26

But take your time

jannis22:02:12

Is there any way to "require"/load an EDN file with data from resources/ in CLJS code built with boot-cljs?

micha22:02:31

@jannis: you can use a macro for that

jannis22:02:51

Oh, yeah, that's a nice idea

jannis22:02:04

Because at CLJ compile time, it'll have access to the filesystem

micha22:02:05

(defmacro require-edn [path]
  (read-string (slurp (io/resource path))))

jannis22:02:15

Nice, thanks simple_smile

jannis22:02:16

What's the root directory io/resource will operate against? target?

micha22:02:30

no, the classpath

micha22:02:34

no files involved

micha22:02:47

nothing in boot knows about things in the target btw

micha22:02:56

it's write-only as far as boot is concerned

micha23:02:07

a place to put things for after boot exits

jannis23:02:16

So mostly for jar generation and so on.

micha23:02:19

while boot is running it's in an undefined state

micha23:02:28

even then only sometimes, but yes

micha23:02:43

sometimes you want boot to just emit files for the final state of the pipeline

micha23:02:50

but most of the time you don't care about files

micha23:02:58

like when i make a jar i don't want the file

micha23:02:17

i either install it in my local maven repo, or i deploy to clojars, or i upload it to s3

micha23:02:27

i very rarely want an actual file in a directory

micha23:02:32

same with web apps

micha23:02:39

either i serve from the classpath in local dev mode

micha23:02:52

or i push artifacts to s3 buckets for staging/production

jannis23:02:41

Right. So that's why the implicat target output was removed.

micha23:02:00

also like if you have two boot processes

micha23:02:09

and they both think they own the target dir

jannis23:02:58

@micha: The macro worked out of the box with data/seed.edn as the path. Perfect, thank you again simple_smile

micha23:02:10

sure, no problem!

jannis23:02:22

I shall remember this elegant way of "lifting" resources into ClojureScript.

micha23:02:49

the only thing you need to watch out for is that there is no explicit dependency between the macro file and the edn file

micha23:02:08

so if you change the edn file it won't necessarily change the macro expansion

micha23:02:29

there could be caching etc that will not be able to detect the dependency basically