Fork me on GitHub
#tools-deps
<
2020-02-07
>
robert-stuttaford09:02:51

how do folks document the reason for a dep in their deps.edn? i was thinking we could just add a :doc "For feature x" key to the map alongside :mvn/version et al

vlaaad09:02:42

;; need this lib because x

danieroux10:02:57

I have this: org.clojure/tools.reader {:mvn/version "1.0.0-beta4" :dictated-by :datomic}

robert-stuttaford10:02:59

i like that! what do you do for deps that are for specific feature code, rather than to support another dep?

robert-stuttaford10:02:29

separate question: is there a dependency conflict diagnosis output in the tools.deps world, like the one lein has when you lein deps :tree?

dominicm10:02:42

I think there's a ticket for this

robert-stuttaford10:02:31

ok cool - this is probably the very last thing we'll still need lein for!

dominicm10:02:32

Lein handles conflicts very differently

dominicm10:02:39

I wouldn't compare the outputs at all

delaguardo10:02:06

there is a way to generate tracefile with https://github.com/clojure/tools.deps.graph right now there is no tooling to work with those traces and you have to analyze it yourself but good to know as an alternative

dominicm10:02:40

Personally, I've stopped having problems when using deps.edn. It has better conflict resolution behaviour.

dominicm10:02:02

Yeah ^^ trace file has the data.

dominicm10:02:29

You're creating 2 things you need to sync up. You won't remember when you need a new feature.

robert-stuttaford10:02:23

i agree, @dominicm -- the counter argument though is when upgrading dependencies, it's a lot of effort to dig around and understand how they are used. so i'm not planning reams of content. just enough to understand its importance and connection

dominicm10:02:53

On my list is the job to solve that through tooling :)

dominicm10:02:13

Kondo should make it fairly trivial to do at the namespace level

robert-stuttaford10:02:28

you're like Varys from Game of Thrones, with little birds telling you when people talk about kondo 😄

borkdude10:02:51

I will now crawl back under my rock

robert-stuttaford10:02:04

hahah i think it's great!

dominicm10:02:49

The trick is figuring out which jars have which namespaces defined within, then mapping that to the dependency

dominicm10:02:12

Kondo supports jars though, so it isn't too difficult.

borkdude10:02:26

kondo also reports in which files the namepace are defined. it also lists the jar when you use the option {:output {:canonical-paths true}}

dominicm22:02:18

Might be better to not use kondo actually, as then I'll miss datomic and other AOT libraries

borkdude10:02:38

the jar string is a little bit weird though, I want to make it more compatible with http://java.net, so it might change a little

dominicm16:02:20

It should probably be the same as what io/resource would give me

borkdude17:02:57

yeah, that's what I want to make it

borkdude17:02:05

so expect a small breaking change there

dominicm17:02:01

Okay, won't write my code this weekend then.

borkdude17:02:40

it's only a small change, you can still use it to compare if it came from a different jar, to e.g. discover namespace conflicts

borkdude17:02:19

jar:foo:bar will become something like <jar://file>://foo#!/dude or something, I can't remember these details

dominicm17:02:46

I do that with sed atm. I think you really want to compare jars for conflicts at the file level.

borkdude17:02:18

maybe you can just specify what you need, so I can take that into account

borkdude17:02:31

if something isn't working for that purpose right now

dominicm17:02:52

I could make what's there work, don't worry. I'm just patient enough. http://Java.net compatibility would be good. What I'm planning on doing is linking the jar back to its dependency (eg bidi/bidi), then I know which namespaces use which dependency.

souenzzo21:02:16

I just realized that {} is a enough deps.edn for some libraries

Alex Miller (Clojure team)21:02:29

if you have no deps but Clojure and typical src dir, you should be good :)

borkdude21:02:40

yep, it's awesome. and even if a library (like medley before not too long ago) doesn't have one, you can declare a manifest type

seancorfield22:02:06

@souenzzo or even no deps.edn at all 🙂

(! 1222)-> mkdir -p example/src/simple
(! 1223)-> echo '(ns simple.main) (defn -main [& args] (println "Hello, World!"))' > example/src/simple/main.clj
(! 1224)-> cd example/
(! 1225)-> ls
src
(! 1226)-> clojure -m simple.main
Hello, World!

souenzzo22:02:23

But without a file with a empty map you cant import as a git-dep

seancorfield22:02:21

Yes you can. You can specify a manifest type of :deps

borkdude22:02:43

You can: Local project coordinate: {:local/root "/path/to/project"} Optional key :deps/manifest Specifies the project manifest type Default is to auto-detect the project type (currently either :deps or :pom)

seancorfield22:02:08

Dang, you're fast! I was just about to paste that in! 🙂

borkdude22:02:49

It was still in my clipboard from the previous time someone asked (kidding)

seancorfield22:02:11

For example

(! 1234)-> clojure -Sdeps '{:deps {jedi-time {:git/url "" :sha "e3c9ab41267be7d9033a13e1d666a3411f53d3c6"}}}'
Cloning: 
...
Error building classpath. Manifest type not detected when finding deps for jedi-time/jedi-time in coordinate {:git/url "", :sha "e3c9ab41267be7d9033a13e1d666a3411f53d3c6"}
(! 1235)-> clojure -Sdeps '{:deps {jedi-time {:git/url "" :sha "e3c9ab41267be7d9033a13e1d666a3411f53d3c6" :deps/manifest :deps}}}'
Clojure 1.10.1
user=> 

😮 4
borkdude22:02:41

Would it make sense for tools.deps to default manifest type to deps.edn if a project has no pom.xml / deps.edn? Why not?

ghadi22:02:47

if there's no manifest, why default the manifest type?

ghadi22:02:17

there's an implication/assumption I don't get

borkdude22:02:00

in the case of git deps with only a src folder, that saves you one file or some config 😛 probably a stupid reason

ghadi22:02:19

sounds stupid (you said it 🙂 )

seancorfield22:02:19

I think defaulting to :deps would lead to confusing errors when someone tries to mistakenly depend on a Leiningen project that had other dependencies.

ghadi22:02:04

getting a magic ambient default seems like an undesired thing...

souenzzo22:02:26

there is only one default: paths ["src"]

borkdude22:02:27

the reason I brought it up is that this option is often not known to people, maybe because the guide isn't very elaborate about it, although it does document it

andy.fingerhut22:02:34

If http://Clojure.org folks are open to a PR enhancing the docs, and you are open to writing such an enhancement, I see magic occurring in the near future 🙂

ghadi22:02:28

I guess I don’t understand the problem enough to evaluate a solution

ghadi22:02:49

If you control the dir you can add a manifest easily

borkdude22:02:30

I was thinking the other way around: sometimes it's desirable to depend on a clojure project out of your control that only has a project.clj but no pom and no dependencies. what I did for medley is add a deps.edn to it: https://github.com/weavejester/medley/commits/master because that was more convenient to me than having to look up the exact option for manifest type again 😃

borkdude22:02:00

anyway, not that important I guess. have a good weekend y'all

ghadi22:02:32

It’s been a while since I tried but I though project.clj’s were read automatically

andy.fingerhut22:02:06

I don't think project.clj files are read if you depend upon a git commit SHA of a Leiningen library that has no deps.edn, is it?

andy.fingerhut22:02:49

maybe tools.deps has a way to do that I do not realize, but it seems like that would be kind of magical ambient default, wouldn't it?

andy.fingerhut23:02:19

I could be way off here, but my guess was that a Leiningen library deployed as a JAR, e.g. to http://clojars.org, typically contains a pom.xml that tools.deps would use.

Alex Miller (Clojure team)23:02:23

project.clj's are not read now

Alex Miller (Clojure team)23:02:31

but maybe they will be in the future

Alex Miller (Clojure team)23:02:15

I don't really want to change the defaults until that's sorted

andy.fingerhut23:02:37

I'm sure you know that project.clj files can contain arbitrary executable code...

borkdude23:02:23

so do sources files 😉

andy.fingerhut23:02:34

deps.edn files don't, though.

andy.fingerhut23:02:43

nor do pom.xml files

Alex Miller (Clojure team)23:02:42

less true than you think

Alex Miller (Clojure team)23:02:04

pom model resolution is remarkably f'ing complicated

Alex Miller (Clojure team)23:02:14

and the libs around it, more so

andy.fingerhut23:02:05

OK, news to me, but I am pretty sure you have looked at it 1000x longer than I have 🙂 I was assuming that pom.xml didn't have executable / evaluatable pieces to them.

Alex Miller (Clojure team)23:02:59

to actually resolve pom files the way mvn does, you need to talk to the web, download plugins + parent poms, etc

Alex Miller (Clojure team)23:02:16

and there are a lot of properties that need to be interpolated as well

Alex Miller (Clojure team)23:02:54

tools.deps is tapping into apis to do the right thing for most of that (and most people don't do anything very fancy)

borkdude23:02:24

maybe more people should get into the habit of checking in the generated pom.xml files if they want their non deps.edn libs to be used

andy.fingerhut23:02:22

FYI, I am not here asking for new tools.deps features -- just checking whether I am missing something in what it does now.

Alex Miller (Clojure team)23:02:43

I actually think it would be good to have project.clj support

Alex Miller (Clojure team)23:02:28

I am undecided whether it should be naive support (data only) that relies on no leiningen stuff or good (perhaps add-on) support that does the actually right thing

borkdude23:02:21

naive as in only read the :dependencies vector and be done with it?

andy.fingerhut23:02:43

I would guesstimate that over 90% of project.clj files start with (defproject and do not rely on code execution for the value of their top level :dependencies key, but haven't done any kind of wide survey.

borkdude23:02:43

would cover the first 80% probably

Alex Miller (Clojure team)23:02:18

yeah. honestly I would love to know the actual number.

borkdude23:02:58

sandboxed execution of project.clj could also work for the remaining 20%

borkdude23:02:04

yes. personally I would think it would be ok to only accept EDN-parsable project.cljs and ignore those with read-evals.

andy.fingerhut23:02:21

I am sure developers often download source of a Leiningen project and run Lein commands before looking whether project.clj does code execution, but at least they had to download the source for that to happen. Doing eval on the project.clj of transitive dependencies is another level of openness to exploits.

Alex Miller (Clojure team)23:02:23

there's a fairly easy path to do the right thing using the lein code base iirc, but it does require depending on that code (which pulls in a bunch of stuff)

Alex Miller (Clojure team)23:02:20

doing this the really really right way (like eval in clojail or something) is probably not something I personally am ever going to write :)

borkdude23:02:36

that's why I said "sandboxed", something like clojail or sci, so allow only a restricted set of functions. it sounds like a bit of a headache that may not be worth it for the 20% or less

andy.fingerhut23:02:59

think of the endless bug reports ...

borkdude23:02:59

same thoughts there

Alex Miller (Clojure team)23:02:05

like I said, I'd be interested in knowing a better estimate of the %

Alex Miller (Clojure team)23:02:27

if it's 99% then that's an easier decision

borkdude23:02:34

maybe it's more like 5% or 1%

andy.fingerhut23:02:42

Is the % number you are interested in mainly "is pure edn" vs. not? Or some finer distinctions would be interesting to know?

borkdude23:02:22

I guess you can retrieve a list of all clojars projects and then curl to their github repo and try to parse it with edn/read-string

borkdude23:02:28

I have swapped from read-eval to just templating an EDN-readable project.clj recently. the templating is done with clojure ironically 🙂

Alex Miller (Clojure team)23:02:36

just whether a naive slurp edn/read find-:dependencies would work

Alex Miller (Clojure team)23:02:46

if anyone wanted to get a believable estimate and write that code, I'm happy to consider it

andy.fingerhut23:02:49

and likely ignore any :profiles with their own :dependencies keys

Alex Miller (Clojure team)23:02:11

not doing anything down that line

borkdude23:02:19

just so we're clear, this is about using project.clj projects as git deps right?

borkdude23:02:40

that would probably exclude projects that have java sources in them

andy.fingerhut23:02:53

@borkdude I am happy to work on my other hobbies if this has set a fire under you. You would probably beat me to the answer, anyway 🙂 One perhaps additional interesting thing to check, if it isn't difficult, is whether the http://clojars.org pom.xml is "equivalent" to the top level :dependencies key.

borkdude23:02:40

@andy.fingerhut feel free to give this a go, I'm not doing this anytime soon, got lots of other stuff to do atm

andy.fingerhut23:02:06

Somehow reminds me of a scene from an old Simpson's show: https://www.youtube.com/watch?v=6qUQ94_MF2Y

borkdude23:02:56

I didn't even ask for project.clj support btw, hehe

borkdude23:02:09

Although I would not be against it

andy.fingerhut23:02:48

That's the beauty of it -- neither of us did, but Alex was already wondering about doing it 🙂

andy.fingerhut23:02:15

As a sanity check on whether people often deploy pom.xml files that are a different dependencies list than the top level :dependencies key

borkdude23:02:11

Usually when I need a project.clj as git dep really bad, I just fork it and add the deps.edn myself

borkdude23:02:05

Not directly related to this problem, but there is a repo which collects stats from clojars as edn files: https://github.com/plexus/clojars-stats Could be of use for some kinds of analyses

andy.fingerhut23:02:19

Thanks for pointing that out. If nothing else, it might be a good way to get a large list of projects, and sort them by download popularity or something like that.

andy.fingerhut23:02:48

and drive a program that downloaded more stuff about each project found there.