Fork me on GitHub
#boot
<
2015-10-24
>
isaac02:10:30

boot has the feature of new project like leiningen

bensu18:10:58

Hi! I'm a tool to get dependency graphs from Clojure projects and I would like it to be boot compatible. Given a repository I need to get the :source-paths as as found under set-env!. For leiningen projects I'm parsing project.clj with leiningen.core.project/read-raw. Is there an equivalent function in boot I could call from clojure?

micha18:10:04

@bensu: you'd need to run boot, because dependencies etc are dynamic

micha18:10:10

they can change at runtime

micha18:10:27

boot show -e will dump the basic environment settings

micha18:10:35

as an EDN map

micha18:10:51

but it's common to have tasks that add dependencies or directories etc

micha18:10:35

the best way to get dependencies is to look at the pom.xml provided with the project's jar

bensu18:10:46

Hi @micha! I don't want the project's dependencies, I'm sorry if I wasn't clear. I only want the set for :source-paths

bensu18:10:20

I can then feed that into tools.namespaces to get the dependency graph (how they depend on each other) for the different namespaces

bensu18:10:32

without thinking about the external dependencies.

micha18:10:39

like something similar to cross-clj?

micha18:10:48

that kind of info is what you're looking for?

bensu18:10:43

and all that data is in the ns declarations

bensu18:10:17

so, from boot I only need that little set under (set-env! :source-paths #{"src/clj" "test/clj"})

micha18:10:38

yes, but those can be set dynamically as boot runs

juhoteperi18:10:41

@bensu: You probably want to try using (get-env :directories) (or source-paths) and fallback to java.tools.classpath if boot.core is not available

juhoteperi18:10:34

Hmh, I should have probably read the context simple_smile

micha18:10:44

@bensu: read-raw is simply not possible with boot. consider some clojure.read-raw program that statically analyzes a clojure file and prints a map of all things it will do

micha18:10:54

this is impossible because clojure is dynamic

bensu18:10:12

I understand that the parallel is not the same

bensu18:10:46

and I also understand that the :source-path can be changed by the script itself

micha18:10:57

boot doesn't have "source paths" really, at least not like maven or lieningen

micha18:10:07

those are just the starting point for the fileset

micha18:10:24

boot tasks never see those directories

micha18:10:53

the tasks run on the fileset, which contains files from those dirs, but the dirs are not known by anything in boot

micha18:10:09

this is to ensure that tasks don't clobber your source files

micha18:10:08

another issue is that one boot program might build a number of different things in the same "project"

martinklepsch18:10:09

@bensu: I think ideally your program could be wrapped in a boot task. Then you can allow users to add it to their tasks and be (mostly) sure that all the dynamic bits the user expects have happened.

micha18:10:20

yeah that's the only way

martinklepsch18:10:00

Assuming your program is clojure and takes a set of directories that’s a fairly easy to write task.

micha18:10:09

it wouldn't help you to consume random repos from github though, because you would need to understand the boot script so you know where to inject your task

micha18:10:35

why not consume the jar file though?

micha18:10:43

you could find all the clj files in a jar

micha18:10:50

and build a graph from those

bensu19:10:25

@martinklepsch: that wouldn't be desirable. Right now I have a "Paste a github ur;" which has to git clone and then use tools.namespace. Using a boot task and going through that process would imply getting all the dependencies and somehow "running" the project, which I don't want to do.

bensu19:10:48

@micha: that might be a better idea, try to find a jar for the project and ignore boot entirely

martinklepsch19:10:55

@bensu: ah, I assumed it would be something users use directly

martinklepsch19:10:12

in that case using jar’s sounds pretty smart simple_smile

bensu19:10:20

@martinklepsch: haha, it's funny because it was you who suggested this simple_smile

martinklepsch19:10:50

what did I suggest? writing a boot task for this? simple_smile

bensu19:10:22

No, we talked a little about dependency graphs and you said something browser-based would suit you better.

micha19:10:38

boot will never be amenable to static analysis, because build.boot is a program, not a specification

bensu19:10:47

(and I'm adding boot because without it it wouldn't suit you at all)

micha19:10:55

so there is no way to know what it's going to do without running it

martinklepsch19:10:31

ahhh! I remember now. What I meant though was to generate the data through some tool and then just visualize it in the browser. Not doing all in the browser/server.

micha19:10:50

jars on clojars/maven central have the scm attribute in their pom

micha19:10:59

maybe you can build a reverse index

micha19:10:16

so you can map github url to scm attribute of jars on clojars

martinklepsch19:10:32

@micha: I’m just toying with building boot with boot — have you experiences running the maven stuff from java directly?

micha19:10:58

i did some ant stuff before boot

micha19:10:13

like i made some java programs that called into ant programmatically

micha19:10:24

i don't think i ever tried to do maven from there

martinklepsch19:10:14

@bensu: I think you can get a pretty good flow with lein plugin + boot task and some semi automated way to get their results into a browser. (e,g, put stuff into users clipboard when done and automatically open website with textarea focused.)

martinklepsch19:10:20

It’s not the flow I’d expect of a commercial product but seems good enough for a fun side project type thing. + you would not need any servers, everything else could just happen in the browser.

bensu19:10:22

right, but I'm very close to "Push to deploy"

bensu19:10:50

and also, when it's this easy to paste a link and get a dep graph, you really do it

bensu19:10:04

I wouldn't want to miss boot projects but figuring out the appropriate jar from the link sounds like a bummer. I'll investigate, maybe it's not

martinklepsch19:10:16

you could also have a server that takes incoming descriptions and renders them then the copy pasting wouldn’t be necessary at all

martinklepsch19:10:25

with a boot task it’s one invocation away: boot -d bensu/viz dep-graph no installing no anything simple_smile

bensu19:10:51

So boot figures out where bensu/viz is, downloads it, and executes it?

martinklepsch19:10:36

boot allows you to add dependencies at runtime via set-env! (in code) or —dependencies/-d at the command line. bensu/viz would be the artifact name on clojars or something

micha19:10:50

man, i think it was a mistake to merge https://github.com/boot-clj/boot/pull/269

micha19:10:03

i think we should revert it

micha19:10:31

it was really nice to be able to do something like boot -d show -u and have it print the latest realease

micha19:10:42

i use that all the time

bensu19:10:51

@micha: that's neat

bensu19:10:07

really shows boot's flexibility

micha19:10:15

@bensu: what's the workflow for the dependency graph app?

micha19:10:27

like when would i use it? and what would my objective be?

micha19:10:20

i don't understand why i'd want to operate on a git repo and not a delendency coordinate?

bensu19:10:58

I don't know what a dependency coordinate is 😞

bensu19:10:42

I want to discuss architecture with my teammates and show then how/why things connect to each other inside the application.

bensu19:10:00

and when there any big changes, see how they translate in those graphs

micha19:10:27

so if i understand correctly, you have a project, which is an application

micha19:10:35

and it has a build.boot file perhaps

micha19:10:43

that builds it and/or launches it etc

micha19:10:08

and you want to be able to generate a nice looking graph that shows all the modules in the application and how they are related?

bensu19:10:42

yes! the confusion arises from the two uses of "dependency". What I actually want could be better described as a module's graph

micha19:10:22

yeah and applications don't necessarily have artifacts that are published in some repository like modules do

micha19:10:49

ok so you could definitely make a visualize task or something

micha19:10:09

so like if i have a build-application task for instance in my build.boot file

micha19:10:22

that builds my application (maybe makes an uberjar, whatever)

micha19:10:36

i would do boot build-application visualize

micha19:10:03

the visualize task would run after all the build tasks, so it would see the final state of the build environment

micha19:10:38

this would include all the source files, even ones that were generated dynamically by other tasks during the build that don't even exist in the github repo

bensu19:10:37

it seems like boot is flexible enough to handle that, and even though it dynamically on the first try without installing.

bensu19:10:58

I might make it my first boot task

bensu19:10:17

but other tooling/environments don't have that (think non-clojure)

bensu19:10:05

thanks everybody for the help, my take is "there is a much simpler way here" and "build.boot is a program, thus not amenable to static analysis"

micha19:10:01

haha :thumbsup:

micha19:10:55

i can imagine a show -n task in boot to analyze the dependencies between namespaces, maybe

micha19:10:14

i guess that's not really something that needs to be in boot itself

bensu19:10:59

probably not but it is trivial: 10 lines of tools.namespace to implement and then you need to choose how to display it.

micha19:10:48

how does it handle multimethods and protocols, btw?

micha19:10:55

i guess that's impossible

micha19:10:13

like if namespace A defines a multimethod my-multi

micha19:10:29

and namespace B defines a dispatch for my-multi

micha19:10:35

and namespace C uses that dispatch

micha19:10:45

so C depends on B, but indirectly via A

bensu19:10:52

it doesn't!

micha19:10:43

yeah it's impossible to statically determine that, i guess, since the dispatch fn can do anything

micha19:10:01

so there isn't any way to statically know that C uses the dispatch method defined by B

bensu19:10:32

but if you are a good citizen and your application can handle it, you require B from C to ensure that the multimethod is loaded by the time you want to use it

micha19:10:42

so it would appear that nothing depends on B, but if you remove B then C will fail with "method not found"

micha19:10:14

usually the place where multimethods are used doesn't know about where the dispatches were defined

micha19:10:24

you'd have circular deps in a lot of cases

micha19:10:35

protocols have the same problem though

bensu19:10:20

that's right, in many cases it's not clear where do things come from. My experience is that as the system gets bigger you want to start tracking such things, visualizing helps, even if it's not complete

micha19:10:48

yeah totally

micha19:10:35

multimethods are especially tricky, because they're mostly used when things are coming in from outside the type system, or going out of the system

micha19:10:51

like parsing http request bodies or something

micha19:10:10

so there is no way at all to map the dependency relationships exactly

juhoteperi20:10:56

@micha: What do you think about supporting metadata for task options? Like :deprecated flag

juhoteperi20:10:03

^:deprecated K gpg-keyring PATH str "The path to secring.gpg file to use for signing."

micha20:10:21

i like it

micha20:10:38

i still don't have a full idea of how to manage changes

micha20:10:59

but adding deprecation warnings is good i think

micha20:10:30

we could also have a way to suppress warnings of various kinds

micha20:10:51

i'm working on getting the boot.properties to work with all the env vars right now

micha20:10:16

and doing a code cleanup wherever we're directly referring to env vars in boot

micha20:10:35

basically i'm making a function in boot.App

martinklepsch21:10:22

would it be hard to replicate mvn install with boot? alternatively we could just move all files in one dir and then invoke maven there via dosh (target task would be handy there)

micha21:10:00

i think we could eliminate maven install, yeah

micha21:10:08

we can use boot javac task there

micha21:10:16

and uberjar etc

martinklepsch21:10:33

we still need to shell out for maven assembly in any case I assume

micha21:10:44

why do we need maven?

micha21:10:11

it was a long time ago but i think i was just using it for javac and stuff

micha21:10:27

with boot we have more control than i had with lein

martinklepsch21:10:30

mvn -q assembly:assembly -DdescriptorId=jar-with-dependencies

micha21:10:43

yeah that's a "fat jar" aka uberjar

micha21:10:06

with lein it wasn't straightforward to make a pure java uberjar

micha21:10:11

so i used maven

martinklepsch21:10:17

makes sense, thanks

micha21:10:49

i'd like to get away from AOTing everything at build time

micha21:10:05

because it messes with things like dunaj and makes building more of a pain

micha21:10:23

compiled classes aren't as flexible as the source

micha21:10:32

because you have to worry about binary compatibility then

micha21:10:54

it would be cool if we boot could compile itself and cache the classes

micha21:10:59

in boot cache area

micha21:10:11

with a separate cache for each version of java/clojure

micha21:10:33

so you'd only need to AOT the first time you run boot

micha21:10:40

after that it would be cached

martinklepsch21:10:32

@micha: the version.properties file in the base jar, I assume it carries the version information for runtime? Where is that modified?

martinklepsch21:10:58

getting close simple_smile

micha21:10:00

i modify it manually for each release

micha21:10:12

all the modules read from that to configure their own versions

micha21:10:25

this way i don't have to edit all the project.clj files

micha21:10:41

it's a .properties file because of maven i think

martinklepsch21:10:42

no not that one.

martinklepsch21:10:56

the one in boot/base/src/main/resources ….

micha21:10:08

ah maybe that gets copied by the makefile?

martinklepsch21:10:20

boot/base/src/main/resources/boot/base/version.properties

micha21:10:23

maybe it's redundant

martinklepsch21:10:51

public static String
    readVersion() throws Exception {
        ClassLoader cl  = Thread.currentThread().getContextClassLoader();
        InputStream in  = cl.getResourceAsStream("boot/base/version.properties");
        Properties   p  = new Properties();
        p.load(in);
        return p.getProperty("version"); }

micha21:10:34

and it has maven stuff in it

micha21:10:44

version=${pom.version}version=2.3.0

micha22:10:15

omg is it real ?!?!

martinklepsch22:10:44

@micha: ^ it’s a mess but pretty close, bit more cleanup and the exe/sh stuff and it’s done

juhoteperi22:10:50

Does it support watch? 😄

micha22:10:05

it's so small now

juhoteperi22:10:07

I'm quite fed up with running make while testing gpg stuff

martinklepsch22:10:31

@juhoteperi: yes you can certainly build sublibs with watch now

juhoteperi22:10:44

But probably not all libs with one boot

martinklepsch22:10:57

… or in the near future to be precise 😉

micha22:10:00

this is amazing work 💥

martinklepsch22:10:23

@juhoteperi: I think this could also work — why do you think it wouldn't

juhoteperi22:10:39

@martinklepsch: Because task for each lib is changing source-paths etc.

martinklepsch22:10:49

@juhoteperi: true but I’m thinking you can "reset the fileset” before each of them

martinklepsch22:10:34

ok, good night simple_smile will finish this tomorrow & then we can do some testing