@moxaj might make sense to use something like https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/from/me/raynes/conch.clj#L14 or or ProcessBuilder directly

@alandipert thanks i'll take a look

@micha I was finally able to give a shot to my idea of declarative declaration of tasks: https://gist.github.com/arichiardi/7e4c430ff4ead706392f22e77c85b16a

Any opinion is more than welcome, there is a custom macro behind of course 🙂

One thing is that defedntask does not declare parameters for now

but everything is data, which makes everything really flexible

the macro require the namespaces, resolves vars and composes the tasks

Can the pom task generate poms for maven plugins? I see an issue already, it does not have a parent

Uhm, working on the task above I wanted to add an option:

a parent SYM:VER=PATH       [sym str str]

java.lang.Thread.run              Thread.java:  745
java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  617
 java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1142
               java.util.concurrent.FutureTask.run          FutureTask.java:  266
                                               ...                               
               clojure.core/binding-conveyor-fn/fn                 core.clj: 2020
                                 boot.core/boot/fn                 core.clj: 1030
                                               ...                               
                         boot.core/construct-tasks                 core.clj:  986
                                               ...                               
            boot.from.clojure.tools.cli/parse-opts                  cli.clj:  500
   boot.from.clojure.tools.cli/parse-option-tokens                  cli.clj:  350
                               clojure.core/reduce                 core.clj: 6703
                                               ...                               
boot.from.clojure.tools.cli/parse-option-tokens/fn                  cli.clj:  347
                              boot.cli/assoc-fn/fn                  cli.clj:   79
                               boot.cli/parse-type                  cli.clj:   62
                               clojure.core/symbol                 core.clj:  579
java.lang.ClassCastException: clojure.lang.PersistentVector cannot be cast to java.lang.String
  clojure.lang.ExceptionInfo: clojure.lang.PersistentVector cannot be cast to java.lang.String

when using the (uber :as-jars true) task, how do I tell it to put jars in WEB-INF/lib/ while the classes in WEB-INF/classes ? right now, it's putting my *.jar files in WEB-INF/classes/ ... which I believe is causing problems

Boot-reload seems to have separate Slack webhook, I disabled that one now

@qqq: sift. or you could just use boot-gae/libs. 😉

@mobileink : I could not imagine missing out on the fun of working things out from scratch

maybe, or rather most likely, you'll come with better ways to do things.

i'm still far from having solid intuitions about how best to write boot tasks.

i tend to hack at them until they more or less behave.

@mobileink: I do have to give you props for one thing: the way you're using uber :as-jars + moving to lib/ results in FAR FAR FAR fewer class files than anything I've done in the past, and as a result, has resulted in FAR FAR FAR faster app_cfg.sh update .

ah, the creds there go to the boot devs, who seem to have thought about nearly everything, one way or another.

something else I'm trying at the moment is only having clojure, javax.servlet, and com.google.appengine/appengine-api as dependencies (so no ring, no compojure) and seeing how far this works

in theory it should be enough since: (1) all static files are handled by web.xml servlet, and (2) for all "REST" ish apis, I really only need one endpoint and have the request query encode everything

sure, you can get along fine w/out ring, etc., as long as you don't mind implementing the nitty-gritty stuff like dealing directly with the native request/response apis.

just defn -service.

if we throw out all of routing, all I need are just /static -- web.xml handles it, serves static stuff /api -- takes EDN as input, produces EDN as output

yeah, if you do not need routing in your app there's no point in using a routing lib. just put defn -service or defn -doGet or whatever in your clojure delegate ns.

you'll have to use java interop in your handler, but that's easy. once it's debugged you can aot the whole thing to get a little more speed.

a demo showing this in boot-gae-examples is on my backlist, maybe you'll contrib one? 😉

yeah (1) I think all I need are doGet, doPost, and doPut

and (2) as a result, my war only has 3 JAR dependencies

we should probably move this to a different channel, like #web or #google-cloud . i don't know how to do this automagically.

in any case, if your processing is simple enough you might be better off dropping clojure altogether in your backend app, just use java.

then you can lose the clojure jar and startup cost .

hrm, boot-cljs-test is adding a 2nd CLJS to my dependencies which causes warnings like:

Classpath conflict: org.clojure/google-closure-library version 0.0-20160609-f42b4a24 already loaded, ALSO loading version 0.0-20151016-61277aea
Classpath conflict: org.clojure/clojurescript version 1.9.456 already loaded, ALSO loading version 1.7.228
Classpath conflict: org.clojure/google-closure-library-third-party version 0.0-20160609-f42b4a24 already loaded, ALSO loading version 0.0-20151016-61277aea
Classpath conflict: org.clojure/tools.reader version 1.0.0-beta3 already loaded, ALSO loading version 1.0.0-alpha1

any idea to prevent it from injecting a second dep?

@anmonteiro I would check if boot show -d shows the one that it is including it and then use :exclusions

@anmonteiro Are you using version 0.3.0 I think it should only add deps if you don't already have them

I'm using 0.3.0

but I definitely have the deps

It should also log a message when it is adding new deps

right

Adding: ([org.clojure/clojurescript "1.7.228"] [doo "0.1.7"]) to :dependencies

but I have ClojureScript 1.4.456 🙂

That is strange, it should be checking current deps and not add deps that already exist

Are you dynamically adding deps in some tasks or something?

I don't believe so

that Adding clause, I have never seen it anywhere...looks like tasks are injecting something (maybe)

https://github.com/crisptrutski/boot-cljs-test/blob/master/src/crisptrutski/boot_cljs_test.clj#L33

And filter deps should remove already loaded deps: https://github.com/crisptrutski/boot-cljs-test/blob/master/src/crisptrutski/boot_cljs_test/utils.clj#L22

@juhoteperi the problem is that u/filter-deps is checking for the exact version to be loaded

but maybe I'm misunderstanding, I don't know how pod/dependency-loaded? works

OK turns out this may be a bug in Boot

found the problem

Version shouldn't matter: https://github.com/boot-clj/boot/blob/master/boot/pod/src/boot/pod.clj#L180

this is how I'm including CLJS:

[org.clojure/clojurescript   "1.9.456" :classifier "aot"]

if I remove the "aot" classifier, everything works as expected

maybe the comparison is by vector and it should be by symbol only ?

Probably dependency-loaded? breaks because the pom file is in different path

dependency-loaded? returns nil with AOTed CLJS

aoted jar doesn't contain pom.properties at all

Also, with aoted clojurescript, some packages are trying to load cljs 0.0-3165...

that's probably Doo

No, it is before I even gets to boot-cljs-test

It is piggieback

How do the classifiers work? Looks like the dependency with classifier doesn't replace normal one currently

Perhaps a bug in Boot?

maybe..

edge case that didn't come up until now? 🙂

but boot probably just forwards to aether

Is the aot dependency supposed to be used alone or together with the normal dep?

alone

AFAIK

yeah, definitely alone

after 5 exclusions I managed to get normal version exlucded

and reproduced the problem

🙂

right. this project has a bunch of exclusions

dependecy-loaded? is at least broken, and there is also something strange going on with normal version getting included even if there is aot version

Well, it is same with Lein, if I depend on aoted cljs, Reagent will bring on transitive dependency on normal dep

So I guess that is how Aether/maven is supposed to work

I think in java side they are often used for additional parts, and don't replace the main dependency, like sources or javadoc

in boot, is it possible for one "watch" to trigger two different compiles? I want to say: if this src directory cahnges, compile (:optimizations :none :source-map true) to THIS DIR compile (:optimizations :advanced :source-map false) to OTHER DIR

@juhoteperi Maven rules are weird indeed

https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html

I have this bookmarked for checking every now and then 😄

> NOTE: In two of these dependency references, we had to specify the <type/> element. This is because the minimal set of information for matching a dependency reference against a dependencyManagement section is actually {groupId, artifactId, type, classifier}. In many cases, these dependencies will refer to jar artifacts with no classifier. This allows us to shorthand the identity set to {groupId, artifactId}, since the default for the type field is jar, and the default classifier is null.

so classifier is actually part of the resolution process and I would not be surprised it took precedence when specified

@anmonteiro I think it is safe to depend on both normal and aot dependency

@qqq try to explore boot watch -h

You'll get the sources from both dependencies, but Clojure will anyway first use compiled classes

ah interesting

Aot contains all the same files as normal jar AND java classes

trying that now

@richiardiandrea : does that provide more info than (doc watch)? as (doc watch) only gives me options :help :quiet :verbose :manual :debounce :include :ezclude

from which it's not clear how to setup "one src directory, compile two times"

@qqq You could use multiple cljs tasks or one cljs task with multiple .cljs.edn

@qqq no more info than that, I would play with include, exclude and the compile tasks, but I have never tried myself

so here's the issue:

(1) I know how ot setup the unoptimized compile (2) I know how to setup the optimized compile (3) I can solve this via two boot sessions

@juhoteperi that definitely works, but still feels like a hack

thanks! 🙂

my issue is: "is there some "parallel task" which taks two tasks ad makes one task out of it

this doesn't seem like . 'watch' issue

it seems like a different primitive for 'run two tasks in parallel'

running tasks in parallel introduces some tricky situations

@qqq Boot-cljs supports parallel compilation of multiple cljs.edn files

since the purpose of tasks is generally to manipulate the classpath, and the classpath is generally related to the filesystem, and the filesystem is globally shared

boot-cljs can do that because each cljs application is compiling files that will go in different places

@micha: two different questions then: (1) should I just acept one boot one task, and avoid trying to "parallel" stuff in a single boot? (2) if I want to gen a optimized and unoptimized version -- do I run two separate boots, or do I use something else?

what is the objective?

if you make an optimized version that one will always take way longer than the unoptimized version i'm sure

for dev, I want the unoptimized version; occasionally I want to test on the optimized version too, without having to wait for a boot jvm startup

so doing them in parallel will only save you like 100ms out of 10000 or more

I see

@anmonteiro I wonder if Cljs is using classifiers like they are supposed to be used

even jvm startup is small compared to cljs compilation with advanced optimizations

I don't think dependency-loaded? can changed to take classified dependency into account

i mean clojure startup

okay, so you're saying "just fire up a new boot every time you need an optimized version" because the optimized compile process is so slow

@juhoteperi I think ClojureScript is using classifiers as they are meant, but I could be wrong. The problem with dependency-loaded? is that the AOTed artifact doesn't have a properties.xml or whatever

With Java libraries, they often have classifiers javadoc and sources, but those jars only contain docs or sources, so the normal dependency is also required

CLJS AOT doesn't seem to have .clj files, only cljs

@qqq i do all my development stuff without optimizations, and then test that optimizations don't break anything (like i forgot to include externs, whatever) before deploying to the staging environment

If it was enough for dependency-loaded? that there was any dependency with the id, adding only javadoc dependency would say that dependency is already provided though there are no classes

that makes sense

@micha: yeah, that soudns more reasonable

Based on my understanding, it seems to be reasonable to add normal dependency in addition to aoted one. Then one doesn't need exclude normal one from all the other deps that have transitive dependency on that.

Also, cold cljs compiles times seem about the same for me with normal and aoted cljs 😞

@juhoteperi right, AOT is not about making compile times better

it's about loading speed of Clojure namespaces

it would make e.g. entering a REPL faster

Yeah, but is should make loading Cljs namespaces faster which should make build faster

not by a significant margin I would say

So what is the usecase?

http://swannodette.github.io/2014/12/22/waitin

@juhoteperi I think ClojureScript started having AOTed artifacts when David was messing with REPL startup

is it just me, or are maven scopes insane? e.g. :scope "compile" does not mean scope is :compile, it means scope is :all. even worse, maven scopes conflate (classpath) scope and transitivity, and the official docs fail to say explicitly how scopes work in most cases. and even worster, they make no distinction between dev, test, and prod execution environments.

maven doesn't have dev test or prod environments

But yeah, they feel quite lacking as they are hardcoded and the names don't describe their uses very clearly

but in at least some cases we have limited control over execution envs (i'm looking at you, gae), so what is "provided" in prod may not be provided in dev.

i would like to propose that boot remedy this.

I don't think solution to this can be tied to one build tool

the thing with maven scopes is that they're designed for use with maven the build tool

maven was very successful and the dependency package stuff has been coopted in various ways

in clojure we use the maven repos without actually using maven (aether, the dependency resolution library was factored out)

@juhoteperi i would not even try to solve it everywhere, just thinking about boot. e.g. instead of :scope "compile", sth like :scope [:compile :test :runtime]

i haven't found "provided" or "runtime" scopes to be useful with clojure projects

but "compile" and "test" seem to provide all the functionality i need

like i use "test" if there is a dependency that i don't want to be transitive

"provided" can be used in libraries to depend on clojure core non transitively

@juhoteperi yeah i have experimented with that

but i don't think it's a good idea imo

oh

because when you declare a dependency as "provided" it doesn't show up in the dependency tree

Hmm, interesting

so my final thoughts we that if there is a contentious dependency, like org.clojure/clojure, that every library depends on, or the cljs compiler, etc

in that case the only sane thing is for the user to specify a version in their :dependencies

right but my bitch (let's be honest) is that a) the names are all wrong, and b) there's a big diif. between scope and transitivity. imho it would be better to make it all explict.

at least the user can use boot show -p for example to help decide which version might be the best

Well, we could probably provide something like Lein profiles to better control what dependencies use in what cases

But for the transitivity stuff, we can do nothing because that is implemented in aether

@mobileink the scope stuff is a maven thing, part of the POM spec

well, the issue i'm getting at is not just dependencies. it's more like dependency dependencies. ;)

also it's good to use the POM as correctly as possible because a maven user might want to use your clojure library

@micha: understood, my point is that does not impose requirements on the boot "api", so to speak.

@mobileink there is a thing called apache ivy, which might be interesting to look at

i haven't really used it, but i remember looking at it -- it has a more programmable resolution framework

maven says "provided". which is opaque if not inane. nothing stops us fron supporting sth less nutty, and translating it into maven-speak.

yeah i think "provided" is more for use with application containers, like tomcat etc

jboss

if you deploy uberjars (the usual thing) you don't need to worry about any of that

because there is no need for dependency resolution at runtime

the problem the tomcat guys had was that they needed to support hotloading jars into a running application

and they needed a way to organize the classpath in that weird situation

so there are some quirky things involved

nowadays we can deploy an uberjar to an EC2 (or GAE) instance and don't need to worry about application containers

yeah, but what "provided" means is not what it says. "compile" is even worse. imho we should be explicit: "provided: means on the compile and test classpaths, intransitive, that's all. so why not ":scope [:compile :test] :transitivity false"?

well boot doesn't have "compile and test classpaths"

boot is just a clojure program that runs and you program it what to do

maven has a built-in idea of "test mode" and "compile mode"

and other things, too

lein has similar concepts

but boot doesn't have any of that

@micha: imho that would be a problem, because boot does support :scope in depencencies.

apache aether supports it

which boot uses via pomegranate

but it works fine because the only things you care about when you consume a library in boot is whether a dependency is transitive or not

so boot is always working in what maven would call the compile classpath

well it's not even that simple

it sounds to me like there is a bit of a clash between boot and the underlying techs it uses.

i'm sure you saw the crazy matrix

i don't see an issue

^^ not trying to be a smartass, just want to know

fwiw in my experience figuring out what to do with :scope is a headache.

right, that's when you're crafting a POM

but when you consume a POM it's all very straightforward and automatic

like when you add a java dependency to your :dependencies, you don't need to ponder the meaning of scopes for it to work

aether takes care of that and it all works out as you'd expect

when you're making your own library thta will be used in your own or other people's projects as a :dependency, you need to make a POM and put your library in a repo

that's when you care about what the different scopes mean

and i personally believe that we only need to care about two of them: "compile" and "test"

crazy matrix? sounds interesting!

oh the maven POM spec has this matrix to "explain" how scopes work

but i've never met anyone who really understood it lol

i think you need to know a lot about maven the build tool to really get it

but i think if you're using boot to make libraries that you want to put in a maven repo you only need to know two scopes: "compile" and "test"

like the one at https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html? insane. 😢

"compile" (or no scope, it's the default) is for dependencies your library needs to function correctly when its used in someone else's project

"test" is for dependencies your library has in development, that the consumer of your library will not need when they use it

and that's all you need to know, imho

so for example, if your library uses JSON, you might bring in [cheshire "1.2.3"]

whoa!

that would be "compile" scope because if someone uses your library and they don't bring in that transitive dependency there will be a ClassNotFoundException

provided?

but suppose you use some tasks to build your library, like [adzerk/boot-cljs "4.5.6"]

in that case you'd use [adzerk/boot-cljs "4.5.6" :scope "test"], because when someone uses your library they won't have any use for that boot task

there is no need to use "provided" as far as i can tell

if my runtime env provides the libs i need?

i mean the prod runtime. e.g. gae.

like how do you mean?

which runtime are you referring to?

I think in those cases you would remove the dependencies when building the jar for the environment

so, i have a local dev/test env., and for that i need to include some jars that are already there in the (cloud) runtime.

the cloud runtime, is this like tomcat or somthing?

iow there is not just one "runtime", since we cannot always control eveything.

Yes, Google App Engine is similar to tomcat and such

ah ok

so in that case you deploy applications to it, not libraries

gae is a derived version of jetty.

well, the line between lib and app is pretty thin.

But yeah, scopes are not required in this case, it is your application packaging that must make the decisions which libraries to include in the package

^^ +1

but i don't 100% agree that libs and apps are a contimuum

continuum

i think there is a clean division

Though for this be easy, uber should probably allow blacklisting dependencies per artifact-id or something

@juhoteperi : ?? i list stuff in :dependencies, with :scope. am i missing sth.?

the difference is this: will the artifact be a dependency in another application?

if yes than it's a library, otherwise it's an appilcation

@micha: no. think servlets and micro-services.

if you make a library that needs [cheshire "1.2.3"] and you put that in a repo for people to use, you can't make it :scope "provided"

if you're loading jars into a classpath that already has stuff in it, there could be anything in there

so making scope "provided" in your special case doesn't work

we're no longer talking about our grampa's deps.

i would use exclusions in your case

if GAE already has dependencies on the classpath when it loads your code, you need to exclude those

when you make your application's POM

i csn write a service that will not be listed as a jar dep anywhere, but my app will depend on it. ok i'm splitting hairs, but i think theres sth there there.

it's not a dependency situation really

the POM is a way to declare dependencies

like the dependencies of your application, or the depenedncies of your library

applications are built from a collection of dependencies and some code that uses them

you wouldn't want to put your application POM in a maven repo probably because it would be broken if someone tried to use it outside of GAE

so imo it makes sense to cleanly delineate what is application and what is library

this works fine with micro services etc, in my experience

agreed, but there is no such thing as a mere dependency. they are always scoped. scoping is an essential property of a dependency, even if only implicitly.

how so?

what's the benefit of scope?

the classpath itself has no scope

you add jars and stuff to the classpath, and that's it

which classpath? compile, test, or runtime?

the JVM classpath

the JVM doesn't care about compile, test, runtime

no, but we do.

the scopes you see in the maven POM are just hints to some tool

it either adds the dependency to the classpath or it doesn't

but there are no actual scopes

like with lisp there are lexical scopes

those are a real thing

anyway, what i'm getting at is that you don't need scopes when you're making an application

and anything you're loading into GAE is an application

just make a POM that has the dependencies you need and not the ones that are already provided

or, alternatively, add the provided dependencies to the POM with :scope "provided"

but don't do that with libraries you have in maven or it will break when used outside of GAE

ok, we must be talking past each other. scopes do make a difference. i don't want mt test jars in prod so i give them :scope " test".

it is not the case that anythin i load into gae is an app.

yes transitivity does make a difference

i think i'm not getting my pt across. when developing a gae service, i need some stuff locally that i do not need in prod. not just my lical testing stuff, but stuff that is provided by prod gae but not dev gae. like the servlet api. i need that for compile and dev runtime but not prod runtime (i think).

do you make a war file?

how does GAE get your program

no, a war dir.

ok so when you build that, you can exclude any jars you don't like, no?

you include all your deps in a zip file or something that you upload to GAE?

sure, it's clojure +boot, i can do anything!

wouldn't that solve the problem?

no, you build a war dir, then run a cmd from the gae kit, passing the wardir.

boot has no way to know what the specific version of the GAE engine provides

i imagine it changes from time to time

but it's easy enough to tell it

when you build your war dir just add (set-env :exclusions '[foo.bar/baz baf/quux])

if those are the group/artifact ids of the jars provided by the runtime

then your war dir won't contain any conflicting jars or whatever

of course i can hardcode everything, but my originsl pt was more general: maven scopes are bonkers, and boot currently imports the craziness, but it doesn't have to.

boot doesn't support scopes

that's i think the disconnect here

maven supports scopes

boot just uses the aether library to resolve dependencies

maven also uses the aether library to resolve dependencies

of course it does! just muck about with the :scope kw in :dependencied.

but scopes have only one effect on resolution

all the :scope kw affects is how aether resolves dependencies

but that is extremely simple

that is, provided you don't mess about with anything but "compile" and "test" in your build.boot

and one common misconception about scopes is that they don't do anything to dependencies you specify that are not transitive

i mean that's the real deal, the misconception is that they do things to dependencies that are not transitive

no matter what scope you use in your build.boot, that dependency will be loaded into the classpath

like (set-env! '[cheshire "RELEASE" :scope "test"]) is the same as (set-env! '[cheshire "RELEASE" :scope "provided"])

there is no difference at all

where there is a difference is if you use that to make a POM that you put in a maven repo

and aether reads that POM while resolving transitive dependencies

that's all

in other words, aether, which is doing the resolving of dependency jars, completely ignores :scope for top level (non-transitive) dependencies

this is the world that boot lives in