Fork me on GitHub
#tools-deps
<
2020-02-14
>
andy.fingerhut00:02:34

I've added one section to the README for my haironfire repo here: https://github.com/jafingerhut/haironfire#some-leiningen-projects-can-depend-upon-multiple-files-andor-tools-to-determine-the-actual-dependencies. merely mentioning some challenges of calculating accurate dependencies for Leiningen projects, which I have not attempted to quantify in my work how often those tricky cases even arise in the projects I've analyzed. I don't have any plans to try to analyze those things, but others are certainly welcome to if they are interested, using any or all of the code in that repo if it helps. Perhaps the biggest hole in the 91.8% and 95.6% numbers I reported for EDN-readable project.clj files are: (a) Those were for repos with a project.clj file in the home directory. Outside of the 100% number is about another 10% of repos that have no project.clj nor deps.edn file in the home directory, many of which probably contain project.clj files in subdirectories that I haven't attempted to find or read. And (b) the difficulties in determining dependencies even after you read a single project.clj file, if there are tools like Lein-modules or similar things in use, again which I haven't tried to determine which projects might use such things.

andy.fingerhut00:02:54

Or in summary, I will now more fully understand if the decision is made not to have tools.deps read project.clj files 🙂

dominicm07:02:07

Something just surprised me while I was writing up a repro for 132, this dependency tree resolves without error, because somehow B and B compare as the same version, but I don't understand how because there's no compare-version for :local :local (yet, that's what my patch does).

deps.edn
+--A {:local/root "/A"}
|  |
|  +- B {:local/root "/B"}
+--C {:local/root "/C"}
   |
   +- B {:local/root "/B"}
It doesn't when there's a git in the way for A/C. I think the cause is https://github.com/clojure/tools.deps.alpha/blob/1feb495adfc5c72c9c5fc9a64917a6bab2203c1a/src/main/clojure/clojure/tools/deps/alpha.clj#L149 not using (= 0 (compare-version a c)) .

dominicm07:02:16

Nothing to do with git, this equivalence comparison just already exists. So my changes don't help. I could have sworn I tested it...

dominicm07:02:35

Oh, I did. But I was being very, very stupid (= (:local/root coord-x) (:local/root coord-x)) comparing x to x 🙂

dominicm18:02:25

I wrote a project to visualize the relationship between your project and it's dependencies. It currently relies on the deps.edn format in order to function. Here's the project: https://github.com/SevereOverfl0w/vizns

👍 20
andy.fingerhut19:02:48

Looks like a useful thing to add somewhere on this wiki page, if it is not already there: https://github.com/clojure/tools.deps.alpha/wiki/Tools

dominicm19:02:44

Thank you 😚

teodorlu11:02:27

This is really cool! Fantastic way to confirm that your code is all spaghetti after all. Some quick notes on usability: • I first naïvely ran it in a folder without a deps.edn file, and spent some time figuring out why it didn't work. • Then I reran it in an actual project, only to find out I was making it crash because I hadn't supplied any main args • Then I figured I needed to sudo apt install graphviz, because of the dependency on dot. Then all was fine!

dominicm12:02:36

I don't have the headspace to pick those up yet, but I'd welcome a PR if you have time. Thanks for the feedback, I appreciate it!

👍 4
Alex Miller (Clojure team)20:02:42

@dominicm the readme example has tools.deps.graph for the main class

dominicm20:02:52

D'oh, used that as inspiration. I thought I'd got it all, will fix, thanks.

Alex Miller (Clojure team)20:02:36

I assume that should be io.dominic.vizns.core

Alex Miller (Clojure team)20:02:00

also, it looks like you happened to rely on make-classpath-roots which I added in tda 0.8.666 (which I never announced) then made public in the release earlier today

Alex Miller (Clojure team)20:02:32

I didn't expect anyone would make use of it in interim :)

Alex Miller (Clojure team)20:02:50

seems like you could have just used make-classpath though?

dominicm20:02:01

I could have split the string, yes. It was nice to think of all the saved cycles of not doing that.

dominicm21:02:08

Oh, that's a better solution. Thanks for the PRs, makes it mergeable from my phone. The final one won't merge at the moment, but I can resolve from a keyboard :)

Alex Miller (Clojure team)20:02:27

there's some ongoing discussion between rich and I in that area about structured classpath data and I'm not ready to commit to anything there yet

borkdude20:02:13

maybe the tool should just deal with a classpath as input so users can decide which tool they use to create it (`clojure -Spath`, lein classpath, etc)?

dominicm20:02:20

I want to do that, but then I have to identify classpath entries back to their library name.

dominicm20:02:49

I'd need something slightly different to a classpath. Haven't figured out what that looks like yet from an agnostic point of view.

dominicm20:02:22

Git dependencies are pretty difficult to name.

borkdude20:02:39

(I haven't looked at any details so I don't know if that's going to make sense)

borkdude20:02:08

for the project.clj git dep problem, one maybe silly but effective solution could be to let tools.deps clone the project, then let it invoke lein classpath on that project, so lein can deal with itself, and write that classpath to a file which is cached, so next time tools.deps wants to know the classpath that that dep is going to produce, it doesn't have to invoke lein again

borkdude20:02:34

haven't really thought this through, feel free to dismiss it. have a great weekend. 🍻

Alex Miller (Clojure team)20:02:19

I think it's interesting to consider a dep type that could provide it's deps explicitly like that

Alex Miller (Clojure team)20:02:35

hadn't thought about it before

andy.fingerhut20:02:16

That still requires eval-level trust in projects you depend upon, at dep resolution time, yes? I do not know if it is any kind of clear bright line to avoid that -- just asking to confirm that the proposed idea still does do eval of project.clj files, even if you never require their code.

borkdude21:02:37

true, it will eval. it could potentially be a solution for lein projects with java sources as well btw.

borkdude21:02:38

I guess the user of a git dep can check the project at given sha if it's deemed safe

borkdude21:02:44

Running library code from the wild is going to be eval'ed anyway, so you should in theory always be checking the code of a dep before you run it?

andy.fingerhut21:02:54

It would be safest, yes. There is a lot of web of trust kinds of things going on, for sure, and the common practice that if something is discovered bad, it will be publicized. That all works with your idea, too, still. I'm only perhaps nit-picking that deps resolution time is another eval-time opening this creates, over and above require'ing the code and running it. To take advantage of it, a malicious party still needs to become trusted, i.e. depended upon by other projects, so in that sense it isn't a new opportunity for attack.