boot has the feature of new project like leiningen

Hi! I'm a tool to get dependency graphs from Clojure projects and I would like it to be boot compatible. Given a repository I need to get the :source-paths as as found under set-env!. For leiningen projects I'm parsing project.clj with leiningen.core.project/read-raw. Is there an equivalent function in boot I could call from clojure?

@bensu: you'd need to run boot, because dependencies etc are dynamic

they can change at runtime

boot show -e will dump the basic environment settings

as an EDN map

but it's common to have tasks that add dependencies or directories etc

the best way to get dependencies is to look at the pom.xml provided with the project's jar

Hi @micha! I don't want the project's dependencies, I'm sorry if I wasn't clear. I only want the set for :source-paths

I can then feed that into tools.namespaces to get the dependency graph (how they depend on each other) for the different namespaces

without thinking about the external dependencies.

like something similar to cross-clj?

that kind of info is what you're looking for?

like this: http://www.tarind.com/depgraph.html

and all that data is in the ns declarations

so, from boot I only need that little set under (set-env! :source-paths #{"src/clj" "test/clj"})

yes, but those can be set dynamically as boot runs

@bensu: You probably want to try using (get-env :directories) (or source-paths) and fallback to java.tools.classpath if boot.core is not available

Hmh, I should have probably read the context

@bensu: read-raw is simply not possible with boot. consider some clojure.read-raw program that statically analyzes a clojure file and prints a map of all things it will do

this is impossible because clojure is dynamic

I understand that the parallel is not the same

and I also understand that the :source-path can be changed by the script itself

boot doesn't have "source paths" really, at least not like maven or lieningen

those are just the starting point for the fileset

boot tasks never see those directories

the tasks run on the fileset, which contains files from those dirs, but the dirs are not known by anything in boot

this is to ensure that tasks don't clobber your source files

another issue is that one boot program might build a number of different things in the same "project"

@bensu: I think ideally your program could be wrapped in a boot task. Then you can allow users to add it to their tasks and be (mostly) sure that all the dynamic bits the user expects have happened.

yeah that's the only way

Assuming your program is clojure and takes a set of directories that’s a fairly easy to write task.

it wouldn't help you to consume random repos from github though, because you would need to understand the boot script so you know where to inject your task

why not consume the jar file though?

you could find all the clj files in a jar

and build a graph from those

@martinklepsch: that wouldn't be desirable. Right now I have a "Paste a github ur;" which has to git clone and then use tools.namespace. Using a boot task and going through that process would imply getting all the dependencies and somehow "running" the project, which I don't want to do.

@micha: that might be a better idea, try to find a jar for the project and ignore boot entirely

@bensu: ah, I assumed it would be something users use directly

in that case using jar’s sounds pretty smart

@martinklepsch: haha, it's funny because it was you who suggested this

what did I suggest? writing a boot task for this?

No, we talked a little about dependency graphs and you said something browser-based would suit you better.

boot will never be amenable to static analysis, because build.boot is a program, not a specification

(and I'm adding boot because without it it wouldn't suit you at all)

@micha: got it

so there is no way to know what it's going to do without running it

ahhh! I remember now. What I meant though was to generate the data through some tool and then just visualize it in the browser. Not doing all in the browser/server.

jars on clojars/maven central have the scm attribute in their pom

maybe you can build a reverse index

so you can map github url to scm attribute of jars on clojars

@micha: I’m just toying with building boot with boot — have you experiences running the maven stuff from java directly?

@martinklepsch: not lately

i did some ant stuff before boot

like i made some java programs that called into ant programmatically

i don't think i ever tried to do maven from there

@bensu: I think you can get a pretty good flow with lein plugin + boot task and some semi automated way to get their results into a browser. (e,g, put stuff into users clipboard when done and automatically open website with textarea focused.)

It’s not the flow I’d expect of a commercial product but seems good enough for a fun side project type thing. + you would not need any servers, everything else could just happen in the browser.

right, but I'm very close to "Push to deploy"

and also, when it's this easy to paste a link and get a dep graph, you really do it

I wouldn't want to miss boot projects but figuring out the appropriate jar from the link sounds like a bummer. I'll investigate, maybe it's not

you could also have a server that takes incoming descriptions and renders them then the copy pasting wouldn’t be necessary at all

with a boot task it’s one invocation away: boot -d bensu/viz dep-graph no installing no anything

So boot figures out where bensu/viz is, downloads it, and executes it?

boot allows you to add dependencies at runtime via set-env! (in code) or —dependencies/-d at the command line. bensu/viz would be the artifact name on clojars or something

man, i think it was a mistake to merge https://github.com/boot-clj/boot/pull/269

i think we should revert it

why?

it was really nice to be able to do something like boot -d show -u and have it print the latest realease

i use that all the time

@micha: that's neat

really shows boot's flexibility

@bensu: what's the workflow for the dependency graph app?

like when would i use it? and what would my objective be?

i don't understand why i'd want to operate on a git repo and not a delendency coordinate?

I don't know what a dependency coordinate is 😞

I want to discuss architecture with my teammates and show then how/why things connect to each other inside the application.

and when there any big changes, see how they translate in those graphs

ah! ok

so if i understand correctly, you have a project, which is an application

and it has a build.boot file perhaps

that builds it and/or launches it etc

and you want to be able to generate a nice looking graph that shows all the modules in the application and how they are related?

yes! the confusion arises from the two uses of "dependency". What I actually want could be better described as a module's graph

yeah and applications don't necessarily have artifacts that are published in some repository like modules do

ok so you could definitely make a visualize task or something

so like if i have a build-application task for instance in my build.boot file

that builds my application (maybe makes an uberjar, whatever)

i would do boot build-application visualize

the visualize task would run after all the build tasks, so it would see the final state of the build environment

this would include all the source files, even ones that were generated dynamically by other tasks during the build that don't even exist in the github repo

it seems like boot is flexible enough to handle that, and even though it dynamically on the first try without installing.

I might make it my first boot task

but other tooling/environments don't have that (think non-clojure)

thanks everybody for the help, my take is "there is a much simpler way here" and "build.boot is a program, thus not amenable to static analysis"

haha :thumbsup:

i can imagine a show -n task in boot to analyze the dependencies between namespaces, maybe

i guess that's not really something that needs to be in boot itself

probably not but it is trivial: 10 lines of tools.namespace to implement and then you need to choose how to display it.

how does it handle multimethods and protocols, btw?

i guess that's impossible

like if namespace A defines a multimethod my-multi

and namespace B defines a dispatch for my-multi

and namespace C uses that dispatch

so C depends on B, but indirectly via A

it doesn't!

yeah it's impossible to statically determine that, i guess, since the dispatch fn can do anything

so there isn't any way to statically know that C uses the dispatch method defined by B

but if you are a good citizen and your application can handle it, you require B from C to ensure that the multimethod is loaded by the time you want to use it

so it would appear that nothing depends on B, but if you remove B then C will fail with "method not found"

usually the place where multimethods are used doesn't know about where the dispatches were defined

you'd have circular deps in a lot of cases

protocols have the same problem though

that's right, in many cases it's not clear where do things come from. My experience is that as the system gets bigger you want to start tracking such things, visualizing helps, even if it's not complete

yeah totally

multimethods are especially tricky, because they're mostly used when things are coming in from outside the type system, or going out of the system

like parsing http request bodies or something

so there is no way at all to map the dependency relationships exactly

@micha: What do you think about supporting metadata for task options? Like :deprecated flag

^:deprecated K gpg-keyring PATH str "The path to secring.gpg file to use for signing."

i like it

i still don't have a full idea of how to manage changes

but adding deprecation warnings is good i think

we could also have a way to suppress warnings of various kinds

i'm working on getting the boot.properties to work with all the env vars right now

and doing a code cleanup wherever we're directly referring to env vars in boot

basically i'm making a function in boot.App

would it be hard to replicate mvn install with boot? alternatively we could just move all files in one dir and then invoke maven there via dosh (target task would be handy there)

i think we could eliminate maven install, yeah

we can use boot javac task there

and uberjar etc

we still need to shell out for maven assembly in any case I assume

why do we need maven?

it was a long time ago but i think i was just using it for javac and stuff

with boot we have more control than i had with lein

mvn -q assembly:assembly -DdescriptorId=jar-with-dependencies

yeah that's a "fat jar" aka uberjar

oh ok

with lein it wasn't straightforward to make a pure java uberjar

so i used maven

I see

makes sense, thanks

i'd like to get away from AOTing everything at build time

why

because it messes with things like dunaj and makes building more of a pain

compiled classes aren't as flexible as the source

because you have to worry about binary compatibility then

it would be cool if we boot could compile itself and cache the classes

in boot cache area

with a separate cache for each version of java/clojure

so you'd only need to AOT the first time you run boot

after that it would be cached

@micha: the version.properties file in the base jar, I assume it carries the version information for runtime? Where is that modified?

getting close

i modify it manually for each release

all the modules read from that to configure their own versions

this way i don't have to edit all the project.clj files

it's a .properties file because of maven i think

no not that one.

the one in boot/base/src/main/resources ….

ah maybe that gets copied by the makefile?

boot/base/src/main/resources/boot/base/version.properties

maybe it's redundant

public static String
    readVersion() throws Exception {
        ClassLoader cl  = Thread.currentThread().getContextClassLoader();
        InputStream in  = cl.getResourceAsStream("boot/base/version.properties");
        Properties   p  = new Properties();
        p.load(in);
        return p.getProperty("version"); }

there we go

ah ok

and it has maven stuff in it

version=${pom.version}version=2.3.0

omg is it real ?!?!

@micha: ^ it’s a mess but pretty close, bit more cleanup and the exe/sh stuff and it’s done

Does it support watch? 😄

it's so small now

I'm quite fed up with running make while testing gpg stuff

awesome

@juhoteperi: yes you can certainly build sublibs with watch now

But probably not all libs with one boot

… or in the near future to be precise 😉

this is amazing work 💥

@juhoteperi: I think this could also work — why do you think it wouldn't

@martinklepsch: Because task for each lib is changing source-paths etc.

@juhoteperi: true but I’m thinking you can "reset the fileset” before each of them

ok, good night will finish this tomorrow & then we can do some testing

later!