Fork me on GitHub
#boot
<
2015-06-12
>
tsdh09:06:20

I'm just trying out boot for a project which uses leiningen currently. I set-env! :source-paths, :dependencies, and :resource-paths, define some pom task options (`:project`, :version, etc) and have a build task doing (comp (pom) (jar)). Then boot build creates a proper pom.xml and a jar, however that doesn't contain any source files, only meta-inf, META-INF, and the files from resources/. Why is that?

martinklepsch10:06:39

@tsdh: the meaning of source & resource paths is a bit different in comparison to leiningen

martinklepsch10:06:40

source = you need it in your build pipeline but don’t want it to be part of the compiled artifact resource = you want both things (build pipeline & artifact)

martinklepsch10:06:32

@tsdh: so the solution to your problem is basically to add your “src” directory to resource-paths

bcachet10:06:00

@martinklepsch: maybe @tsdh also need to add task/aot (https://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L529-L551) to his pipeline in order to compile Clojure files ? Or is it called automatically ?

martinklepsch10:06:40

bcachet: depends on what he wants to do I guess. If you’re packaging a library you usually do not aot things, do you?

bcachet10:06:53

@martinklepsch: Really I do not know. If we do not compile Clojure code before packaging, what will the JAR file contain ?

martinklepsch10:06:26

(I think most clojars just contain .clj files no?)

bcachet10:06:28

so this Clojure lib will only be usable inside a Clojure project

bcachet10:06:58

if I compile my Clojure file before packaging, Java user can maybe use it also

martinklepsch10:06:29

yeah, that sounds logical

bcachet10:06:30

but I have no experience in packaging Clojure. I am fearly new to Clojure(Script)/FP/LISP ..

bcachet10:06:22

I do not know what is the policy for libs available on Clojars

martinklepsch10:06:45

probably a good question to ask in #C03S1KBA2

tsdh10:06:50

@martinklepsch: Thanks! @bcachet I just want to distribute the source files so no need for the aot task.

bcachet10:06:10

@tsdh: when exploring my ~/.m2/repositories/ directory, seems that almost all libs only distribute source files. I think you take the good decision

tsdh10:06:52

@bcachet: Yeah, that's what I've done with this project/lib before with leiningen.

bcachet10:06:57

seems that I over engineer what packaging is 😀

tsdh10:06:31

Another thing: I want to define my own project-specific tasks (build, docs, test, etc.). How can I achieve that my own task don't have the chance to override default tasks and are still convenient to access (from the command line)? Basically, I'd be very fine with having to use boot/<task> for all tasks provided by boot and just <task> for my custom ones.

tsdh10:06:31

Right now, I go with naming my tasks <project>-<task> which is ok but not optimal.

martinklepsch10:06:00

@tsdh: you can put tasks in namespaces just like regular clojure functions

martinklepsch10:06:02

@tsdh: when you require them either refer single tasks or bind via :as. If you bind via :as you can do things like boot yourns/build-stuff

martinklepsch10:06:08

Oh and the namespace containing tasks requires some metadata

{:boot/export-tasks true}

tsdh10:06:35

@martinklepsch: I guessed so. But can I also define that my build.boot doesn't automatically use or refer all core boot tasks so that on the cmd line I'd call the pom task using boot boot.core/pom?

martinklepsch10:06:11

I’m not sure if that’s possible

martinklepsch10:06:22

are any of the tasks you want to write conflicting with the ones in boot.core?

tsdh10:06:17

@martinklepsch: No, at least not yet. But it's always possible that the next version of boot gets some build task which would then conflict with mine.

tsdh10:06:14

@martinklepsch: Also, I'd like to make my custom tasks more prominent, i.e., these are only the tasks which you will usually want to use for building, testing, and generation of docs.

martinklepsch10:06:07

@tsdh: so far @micha has been very conscious about choosing task names that are unlikely to conflict with user defined ones.

tsdh11:06:36

@martinklepsch: Yes, I believe that but my second reason is actually the more important one.

martinklepsch11:06:28

@tsdh: yeah, I agree about that one. You’ll have to come up with what works best for you. I’d probably go for documenting those in the Readme of a project.

tsdh11:06:06

@martinklepsch: I'm playing around a bit. build.boot is in the boot.user namespace. I can add a ns form and define my own stuff there, and then do (in-ns 'boot.user) and set it up as I see fit, e.g., remove all mappings from boot.task.built-in and alias that namespace instead.

martinklepsch11:06:33

@tsdh: cool. I’m thinking there probably might be a way not to import everything but I don’t know it 😅

tsdh12:06:18

How do I handle things like I would do with leiningen profiles? For example, my docs task has some more dependencies which are not required when using the lib, so I don't want to add it to the :dependencies of the environment globally.

tsdh12:06:08

@martinklepsch: Yeah, seen that. So I added a similar testing task which adds additional deps, and then a run-tests task which requires the namespaces of the new test-only deps and then tries to call them, e.g.,

(deftask testing
  []
  (set-env! :dependencies #(conj % '[adzerk/boot-test "1.0.4"]))
  identity)

(deftask run-tests
  []
  (require 'adzerk.boot-test)
  (adzerk.boot-test/test))

tsdh12:06:28

But then boot testing run-tests gives an error:

clojure.lang.ExceptionInfo: adzerk.boot-test
    data: {:file "/tmp/boot.user8833680531087528998.clj", :line 25}
java.lang.ClassNotFoundException: adzerk.boot-test
               ...               
boot.main/-main/fn  main.clj: 161
   boot.main/-main  main.clj: 161
               ...               
  boot.App.runBoot  App.java: 231
     boot.App.main  App.java: 338

martinklepsch12:06:07

oh yeah. That’s something that should be added to the wiki page

martinklepsch12:06:38

You have to wrap it in (resolve #’adzerk.boot-test/test) I believe

martinklepsch12:06:46

don’t exactly remember

martinklepsch12:06:59

but since you require things at runtime they can’t be found at compile time

tsdh12:06:12

Ok, require + ((resolve ’adzerk.boot-test/test)) works.

martinklepsch12:06:51

If you could add a small example snipped of that to the wiki page (or maybe even creating a new one) that’d be very awesome!

martinklepsch12:06:41

@tsdh: if you decide to create a new page, please send me the link, then I can integrate that into the sidebar thing simple_smile

tsdh12:06:25

I think I'll just update the current page.

alandipert12:06:40

@tsdh i may be missing context but you might also consider :scope "test"

alandipert12:06:20

e.g. (set-env! :dependencies '[[foo "1.0.0"] [adzerk/boot-test "1.0.4" :scope "test"]])

alandipert12:06:40

is this project is a library, foo will show up in the pom as a dependency, boot-test will not

tsdh12:06:33

Ah, that's indeed even better. What does :scope name? A task?

alandipert12:06:48

i tend to use only test, i also use it for deps i would like to be in boot repl but not in pom/uberjar

alandipert12:06:24

like using set-env! in tasks tho these "test" deps are not isolated in a pod, so there can still be conflicts

alandipert12:06:44

so if you really want a dependency to exist only for a particular task, it's a good idea to use a pod

jeluard12:06:50

@alandipert: Is it the best practice to always define task dependencies with :scope "test"?

alandipert12:06:42

@jeluard: in an application, maybe; in a library, it's always best for tasks to have no dependencies (deps are in a pod)

jeluard12:06:27

I mean the task itself.

alandipert12:06:49

oh - like if you depend on a library that has tasks in it?

jeluard13:06:53

Or simply declaring a task (say boot-test) you will use in your application. Not considering transitive dependencies yet simple_smile

tsdh13:06:17

So with the :scope thing, how do I define that my test task has to be run in that scope?

alandipert13:06:56

:scope "test" means the dependency will be available to all the tasks in that instance of boot

jeluard13:06:34

As a task user I think the :scope concept is a little too much implementation details. It's one of those areas boot could be more clear I think. Leiningen idea of separating plugin declaration in :plugins is easier to grasp for me.

bcachet13:06:47

I am developing a CLJS library. I create a dev task that represent my TDD environment (build/run tests on node-runner when code is updated). As I use an external JS library, I use :foreign-libs option of the cljs task

(deftask dev []
  "Generate JS from CLJS whenever src/test are modified"
  (set-env! :source-paths #{"src" "test"})
  (comp
    (watch)
    (cljs-test-node-runner :namespaces '[xml-cljs.core-test])
    (cljs :unified true
          :source-map true
          :foreign-libs [{:file "node_modules/sax/lib/sax.js"
                          :provides ["sax"]}]
          :optimizations :none)
    (run-cljs-test)))
I also have a noderepl task that fire a Node REPL. I also need to define :foreign-libs to this task
(deftask noderepl []
  (cljs.repl/repl (cljs.repl.node/repl-env)
    :watch "src"
    :foreign-libs [{:file "node_modules/sax/lib/sax.js"
                    :provides ["sax"]}]))

bcachet13:06:43

Is there a way to get this :foreign-libs information from one task and pass it to the other ?

tsdh13:06:52

@alandipert: Thanks, that's most helpful.

alandipert13:06:18

@tsdh: the idea is those dependencies are available when travis runs boot test, but don't show up as dependencies of the artifact we push to s3

jeluard13:06:20

@alandipert: I think what's confusing is that all tasks deps are defined at the same level as :dependencies. I understand why it works with :scope "test" but it's weird to have to define say bootlaces with it just because it's a task and not a regular dependency.

alandipert13:06:15

@jeluard: oh right, because of the meaning of "test"

alandipert13:06:26

agreed - not intuitive at all 😞

alandipert13:06:09

bcachet: above both tasks, perhaps (def foreign-libs [...])?

alandipert13:06:37

@bcachet: i'm not sure what you mean by "get from one task"

bcachet13:06:39

@alandipert: you are right. but if I want to write a task that I want to share with others for their own projects. They will also need to write cljs compile options for cljs task AND for my task. I thought that maybe I can extract this information from cljs task.

jeluard13:06:38

@alandipert: Sorry I am hijacking the discussion but I feel like it's important for boot in the long run. Right now the way boot tasks are loaded is almost a hack. Would love to see them more first class citizen.

tsdh13:06:44

I have the strange problem that at some point in time where I added another line to my build.boot, the output of boot -h has one empty line between each actual output line.

alandipert13:06:27

@bcachet: i think i see what you mean. well if task-options! is used, you can look that up in your task by looking at the metadata on the task function object

alandipert13:06:34

but that would be on the hackier side

alandipert13:06:59

another maybe better way is an edn file in the fileset with a particular extension

alandipert13:06:27

i haven't kept up with cljs tasks this approach is already used for module builds, and may also accomodate compiler options

tsdh13:06:29

Oh, no, has nothing to do with my build.boot...

alandipert13:06:03

@jeluard: i don't understand quite what you mean but id love to hear about other ways of loading tasks, if you have some in mind

alandipert13:06:05

i need to head to the office, back later 🚗

tsdh13:06:51

Ah, the problem is that every line in the boot output is as long as the longest line, i.e., it seems lines get padded with spaces. Well, and then all lines are wrapped as soon as there's one single long line...

bcachet13:06:52

@alandipert: good idea. I think cljs already define a .edn file to define per build compilation options

bcachet13:06:16

@alandipert: thanks . I think you point me to a very elegant solution.

micha13:06:57

bcachet: have you seen the .cljs.edn file format?

micha13:06:20

bcachet: this was developed to solve the problem you describe above

bcachet13:06:45

@micha I have seen it and forgot it. Thanks to @alandipert I remember this option.

micha13:06:10

bcachet: this is an important thing in boot i think

micha13:06:52

bcachet: it's superior to explicit options on tasks because this way tasks can be in the pipeline that you don't add explicitly

bcachet13:06:59

I just started with boot and boot-cljs. I never used this option for the moment

micha13:06:17

for example, a task can compose itself with other tasks, or create anonymous tasks, etc

micha13:06:51

in these cases you can't provide explicit options to those tasks

micha13:06:04

and we also don't want options to become "places"

micha13:06:24

if tasks look at each other's options the options become global variables

micha13:06:01

you can have tasks that generate or modify .cljs.edn files

micha13:06:09

which is what we do with the cljs task

micha13:06:11

for example

bcachet13:06:55

this .cljs.edn option file is consumed by boot-cljs. Not by cljs itself

micha13:06:19

but all the tools know about it, (or should know about it)

micha13:06:42

this is how the boot-reload task is able to add live reload without needing any code in your application

micha13:06:44

and the cljs-repl

bcachet13:06:08

ok. So I have to extract compilation information from this file and pass it to the cljs.repl.node

micha13:06:40

the .cljs.edn file has a :compiler-options key

bcachet13:06:10

is there any documentation or best practice in finding and reading content of this .cljs.edn file

micha13:06:45

that's the docs

bcachet13:06:48

because there can be any where in the project, as there location is used to define the name of the created namespace

micha13:06:32

you can look at boot-reload and boot-cljs-repl and boot-hoplon projects for examples of how to create, modify, and manipulate them

bcachet13:06:36

I have readen this part of the documentation, but doesn't explain how to use content inside another task.

micha13:06:53

how do you mean "use the content"?

bcachet13:06:09

@micha: thanks a lot for your time and your informations

bcachet13:06:34

I mean read the content in order to extract :compiler-options information

micha13:06:13

ah like how to find the file in the fileset and slurp it in?

micha13:06:18

here is a function that modifies .cljs.edn file

micha13:06:46

and here's how to extract files from fileset to pass to that function

bcachet13:06:50

thanks by-ext comes from boot.core ?

jeluard14:06:24

@micha: I'd love to have your feedback on that too.

micha14:06:57

@jeluard: sure, one sec!

alandipert14:06:52

@jeluard: thanks for writing this up! i will add my thoughts to the thread later

micha16:06:42

@jeluard: thanks for the writeup!

micha16:06:48

@jeluard: one thing to consider is that tasks should use pods to isolate their own dependencies from the project's dependencies

micha16:06:04

when this is the case :plugins seems redundant

micha16:06:28

leiningen has :plugins because it spins up two JVMs, one for tasks to run in and one for your project

micha16:06:55

:dependencies go into one JVM, and :plugins + :dependencies into the other

micha16:06:14

boot tasks, on the other hand, should not have any transitive dependencies

micha16:06:54

the problems with clojurescript i think are resulting from people adding transitive dependencies with incorrect scopes to their pom.xml

micha16:06:14

we depend on a version of cljs, but we scope it to "provided", so it's not pulled in transitively by projects that depend on javelin

jeluard16:06:57

I think the biggest difference with something like :plugins is about clarifying intents. It helps understanding when those dependencies are used. For instance it's not clear to me if tasks and/or its dependencies end up in an uberjar if I don't use the proper :scope. Also using :scope "test" for both tasks and test dependencies sounds a little confusing. But that might be due to my maven background.

jeluard16:06:51

Maybe those points could be addressed by some docs around using/writing tasks. e.g. it should be mandatory to use :scope "test" for tasks.

jeluard17:06:29

About the specific case of ClojureScript I am not clear how I can force tasks to use a specific version. Do I need to define it as part of the app dependencies? If so as :scope "provided" so that it's not packaged in an uberjar?

jeluard17:06:00

Also with :plugins boot could potentially do some pod magic and directly use the task dependencies as pod dependencies. Having a distinction between both looks complex, see what boot-cljs does for both Clojure and ClojureScript (declared twice).

micha17:06:12

@jeluard: it can be complex, but managing dependencies is sort of complex in that way by its nature. i agree about the intents for sure. i'm working on a response to your thing on discourse to explain what i mean a little better than i can here

micha17:06:29

@jeluard: i think we've been using magical tools for too long really

micha17:06:46

we expect things like :scope "test" to be magical, when they're not really

micha17:06:03

like :scope "test" doesn't have anything to do with test harness per se

micha17:06:22

just like :resource-paths doesn't have anything to do with JPG images per se

micha17:06:40

these are just ways to organize dependencies

micha17:06:15

the magical tools have used those things to complect them with testing and static assets and whatnot, but that's bad

micha17:06:48

it's much more powerful to use them in the abstract way where they do what they do and no more

micha17:06:35

scope "test" simply means "this dependency is to be ignored by projects that depend on this project when they resolve transitive dependencies"

micha17:06:09

scope "ignore" might have been a better name, perhaps

jeluard17:06:18

Right I agree with you about magical simple_smile Let's pretend I didn't use that term 😉

micha17:06:31

it's true though

micha17:06:50

tools don't ever expose the abstraction, instead they expose some specific workflow

micha17:06:52

like maven

micha17:06:02

boot does the opposite

micha17:06:10

all there is is the abstraction

micha17:06:43

we can make functions that take a different dependency format and emit the pomegranate format

mitchelkuijpers17:06:44

Just started playin with Boot, pretty impressed so far!

mitchelkuijpers17:06:35

I'm looking at boot because I am working on a project where we want to distribute a tar and I am currently having a very brittle leiningen project

micha17:06:37

mitchelkuijpers: :thumbsup:

mitchelkuijpers17:06:00

Just finished the simple boot tutorial, looks easy enough

mitchelkuijpers17:06:48

First thing i notice that it's a lot faster

micha17:06:49

should be easy:tm:

jeluard17:06:10

Maybe I'm reading too much in :scope "test". As it's also used in leiningen and maven I find it confusing. If it's intended to be used as an abstraction it probably deserves more doc love.

mitchelkuijpers17:06:14

Is there an example project somewhere with a clj and cljs repl?

micha17:06:06

jeluard: yeah, the unfortunate thing is that the :scope test business was intended to be only for consumers who are reading a pom.xml file to collect transitive dependencies

micha17:06:20

jeluard: it was never meant to be something that has any bearing on the project itself

micha17:06:27

or testing the project, etc

micha17:06:53

it's simply annotation in the pom.xml so that consumers can avoid downloading and adding useless dependencies to their classpath

micha17:06:56

however tools don't usually leave it at that, they build up towers of convention on those things, overloading it to do lots of unrelated things at build time

micha17:06:13

and the build gets brittle and hard to understand

micha17:06:27

there's a lot of baggage there i think

mitchelkuijpers17:06:12

Ok I am impressed, ill try to migrate my leiningen project sunday simple_smile

micha17:06:18

jeluard: i think boot complects the classpath with the <dependencies> in pom.xml, too

jeluard17:06:28

So are boot tasks supposed to be defined with :scope "test"? Sorry I am both less and more confused now!

micha18:06:11

jeluard: you should use :scope "test" for dependencies that are not needed by your project at runtime

micha18:06:53

jeluard: for example, if i use some generative testing libraries to generate seed data for my library while i'm developing it, that would be a good :scope "test" candidate

micha18:06:07

because someone can use my library without that dependency and the code will run just fine

micha18:06:26

i only use that generative testing dependency when i'm developing and testing my library

micha18:06:55

but for consumers who will be using my library via their own dependencies it's completely unnecessary

micha18:06:31

so using :scope "test" for that dependency in my project tells those consumers to ignore the dependency

micha18:06:37

and :scope "provided" is similar, but for dependencies that my library needs at runtime but will certainly be declared in the project itself

micha18:06:10

so for the "provided" case you're telling the consumer that it's so likely that there will be a conflict that you should just explicitly declare what you want in the application itself

micha18:06:18

like the case with cljs

micha18:06:43

the "provided" case is for framework dependencies mainly

jeluard18:06:09

I am talking specifically about packaged boot tasks, like bootlaces or boot-cljs. Any reason not to use :scope "test"?

micha18:06:23

where you want to prevent the stuff we see with cljs where the version that is chosen in the conflict resolution process is unpredictable and hard to understan

micha18:06:47

jeluard: yes you should totally use :scope "test" when you depend on boot tasks

micha18:06:20

if you use :scope "test" for bootlaces for example, and someone depends on your project in their own :dependencies, they won't get bootlaces in their project because that dependency won't be pulled in transitively

jeluard18:06:34

That's this very case I find confusing. I've seen both and because it seems it does not make sense not to use :scope "test" ..

micha18:06:57

yes, the distinction there is between application and library

micha18:06:15

in an application i don't bother using scopes, because no POM will be created

micha18:06:32

remember that scopes are only relevant in a pom.xml

micha18:06:57

so like when i make a web app or something it's not going to be uploaded to clojars and people won't be pulling it in via :dependencies

jeluard18:06:05

Yes transitive project is another can of worm. Right now I am more wondering about what is in classpath during test, jar, uber ..

micha18:06:18

oh that's easy

micha18:06:23

everything is on the classpath

micha18:06:31

because scopes do nothing in the project itself

micha18:06:51

all the scopes do is annotate the pom.xml file in clojars

micha18:06:13

it's the consumers who download that pom.xml from clojars when they depend on your thing

micha18:06:22

then it actually starts to do things

micha18:06:59

this is what i was talking about before, how boot complects the project classpath with project dependencies

micha18:06:05

the two aren't really the same thing

jeluard18:06:13

Surely my application that creates an uberjar doesn't package bootlaces?

micha18:06:42

well it depends on what you want

micha18:06:13

By default, entries from dependencies with the following scopes will be copied
to the fileset: compile, runtime, and provided. The include-scope and exclude-
scope options may be used to add or remove scope(s) from this set.

micha18:06:25

this is from boot uber --help btw

micha18:06:00

this is where the abstraction starts to be more powerful than the pre-packaged workflow

jeluard18:06:02

Ah ok so :scope play a role here too.

micha18:06:15

yes because uberjar isn't really a thing

micha18:06:20

it's sort of a hack

micha18:06:58

i don't think it was ever really considered in the JAR design or anything

micha18:06:24

so we expose those knobs so you can have it do what you need

jeluard18:06:56

ok. Things are clearer now. Still I believe the :scope concept does not map well to boot concepts. Now things are more complex when considering all potential scenario. I'll take a look at how I can improve docs in that regard.

jeluard18:06:01

Thanks for the help!

micha18:06:09

jeluard: np!

micha18:06:37

since set-env! is just a function, we can write macros and functions that implement other ways of doing it

micha18:06:14

like you could invent a different way to specify dependencies, and have a function that emits the pomegranate things with :scope and whatnot

micha18:06:03

(set-env! :dependencies (my-fn :plugins ...))

jeluard18:06:09

Yes that's a good way to experiment.

borkdude18:06:11

@mitchelkuijpers: I wrote a blog post about converting a leiningen project with clj and cljs repls to boot, if you're interested

borkdude18:06:32

@mitchelkuijpers: it was my first time using boot, I'm just starting out with it too

mitchelkuijpers19:06:48

Awesome @borkdude will be a nice reference

mitchelkuijpers19:06:37

Thnx @borkdude clear blogpost 👍

tsdh19:06:24

Does boot source some specific file where I can set its env vars or should I just set them in my ~/.profile?

borkdude20:06:13

@tsdh: I saw somewhere that boot has a profile file like in ~/.lein/profiles.clj but I can't find it now. Also you can do a lot with https://github.com/danielsz/boot-environ

tsdh20:06:31

@borkdude: Yeah, there's ~/.boot/profile.boot but that's a clojure file, nothing for setting environment variables.

alandipert20:06:50

@tsdh: BOOT_VERSION and BOOT_CLOJURE_VERSION can be set with a boot.properties file, the others are environment only

alandipert20:06:28

the short story is: setting environment variables on your own is the way currently

tsdh20:06:41

@alandipert: Perfect, I'm looking for setting BOOT_CLOJURE_VERSION. Is boot.properties in ~/.boot/ or can I have a one local to the project, too?

alandipert20:06:34

@tsdh: you can set it project-locally by doing boot -V > boot.properties

tsdh20:06:00

@alandipert: Works, thanks!

tsdh21:06:59

My clojure project uses a Java lib which compiles classes extending some base classes at runtime. For these in-memory compiled classes that Java lib uses custom classloaders. When I try running my tests with boot-test, the in-memory compilation won't work because the base classes the generated ones extend cannot be found.

tsdh21:06:14

I guess that's because boot-test uses pods.

tsdh21:06:45

Is there anything I can read up with respect to pods and classloaders?

tsdh22:06:32

Is it correct that (System/getProperty "java.class.path") returns /path/to/bin/boot? That could explain my in-memory compilation issues cause that's not really a classpath...

tsdh22:06:47

Ah, the real class path seems to be in the property "boot.class.path". But how do I get it so that some JavaCompiler called by some third-party lib actually sees that correct classpath?

tsdh23:06:43

Is there a better way to let boot use the clojure version declared in :dependencies other than to specify an additional BOOT_CLOJURE_VERSION in boot.properties?