Fork me on GitHub
#tools-deps
<
2020-09-08
>
cfleming02:09:42

Is there any way with deps to manage source artifacts for non-Clojure dependencies?

cfleming02:09:25

Or rather, how could I specify that I would like them to be downloaded, and how would they be distinguishable from binary jars in the output?

cfleming02:09:00

This seems to be the answer:

{:deps {junit/junit {:mvn/version "4.13"}
        junit/junit$sources {:mvn/version "4.13"}}}

cfleming02:09:05

They do end up on the classpath, which is unfortunate. I guess they could go in an alias.

cfleming03:09:05

When using :local/root, the doc says: > If the jar has been packaged with a pom.xml file, the pom will be read and used to find transitive deps Is there a way to prevent this from happening? I would just like it to use the jar I give it and not look for transitive deps.

cfleming03:09:01

Looking at the code, it seems like not. My use case here is trying to use deps to manage the dependencies for Cursive. I need to depend on the IntelliJ framework, so I have a script which downloads and unpacks that, and I’m trying to create a deps.edn which refers to the jars within it. However a bunch of those are third party jars which include poms.

Alex Miller (Clojure team)03:09:26

you could use :local/root to a directory which has a deps.edn that includes the jar on the :paths

cfleming03:09:06

Oh, I can just put jar paths directly in :paths? That’s even better.

Alex Miller (Clojure team)03:09:16

kind of a hack, but it works

cfleming03:09:24

I guess that makes total sense, but never occurred to me.

Alex Miller (Clojure team)03:09:30

:paths are just things that get added to the classpath

Alex Miller (Clojure team)03:09:51

on the source stuff, tools.deps is about making classpaths so that's what you're going to get

cfleming03:09:02

Right. It’s actually much better for this use case since I don’t need to create a bunch of bogus dep symbols.

cfleming03:09:38

Ok. Using $sources works fine, so I’ll support that in Cursive, document how to do it and suggest putting them in an alias.

practicalli-johnny10:09:36

@alexmiller I'm confused about using -M with a project rather than a specific alias. In this example I use the hello-world example from https://clojure.org/guides/deps_and_cli#_writing_a_program A project deps.edn file was created containing the dependency for clojure.java-time and the source code from that page copied into src/hello.clj clojure -m hello runs the project and returns the time from running the -main function. However this gives a warning:

WARNING: When invoking clojure.main, use -M
clojure -M runs a REPL clojure -M -m hello runs the project and returns the time. But then I ask myself what is the purpose of -M Creating an alias to run the project seems an interesting idea, as I could also set default arguments. I can only get this to work with :main-opts and not :exec-fn This version of the alias works when calling clojure -M
:aliases
 {:run-project {:main-opts ["-m" "hello"]}} 
This version of the alias simply runs the REPL
:aliases
 {:run-project {:exec-fn hello]}}
Am I missing something from :exec-fn ? Edit: yes I am. The -X option is for running :exec-fn and not the -M function, oops. So clojure -X:run-project should work... except the hello-world project has an unqualified function and cannot be resolved. Moving the source code to src/practicalli/hello.clj and calling clojure -X:run-project gives an execution error, (ArityException) as the -main function does not take any arguments (defn -main [] ,,,) . Changing the -main function to (defn -main [& args] ,,,) fixes that and calling clojure -X:run-project works. From a user point of view, that seems quite a lot to extrapolate from the original warning message when running a project with clojure -m hello

Alex Miller (Clojure team)14:09:26

The guide hasn’t been updated for the new stuff yet - I’ll do that with the stable release of it

practicalli-johnny14:09:41

Great. That time frame wont impact me at all. Thanks If you want help with the docs, let me know (although you are the expert here)

practicalli-johnny11:09:50

Thinking about this further, it does feel that the -X flag would be the most important flag in the long term, especially with :exec-arg allowing default arguments that can be over-written on the command line. -M seems to be more of a (very useful) convenience function for the specific case of running -main-opts configurations. This does remove the need for command line args and bash quoting, so that is valuable. -M is also useful for those aliases that dont work with -X , eg rebel-readline I could only get working with -M . As more maintainers adopt the new version of Clojure CLI, there is scope to configure all projects and tools to work with -X and eventually make -M redundant. Using the -M flag seems to help a little when making the change from Leiningen to CLI tools, as setting the main namespace is a familiar concept. I am sure I am over-simplifying this and probably missing out on a lot of understanding as of yet.

hugod12:09:42

Would -M not continue to be useful for utilities that you expect to be invokable from the shell? How would you allow passing glob’ed files for example?

practicalli-johnny12:09:57

Yes, it seems the -M option is valuable when the :main-opts data is more intricate. Interesting to see what can be done with :exec-args verses :main-opts . I do like :exec-args as a means of using edn over strings. I dont believe I've have any use cases for using glob'ed files, so cant comment on that without specific examples. I assume something like *.jpg would just be a command line argument after the -X:alias-name and passed to the function.

hugod13:09:53

iuuc, -X expects arguments to be key-path edn-value pairs, so clojure -X:my-alias :x "1 2 3" ends up passing {:x 1} to the exec-fn.

hugod13:09:13

In general, if you want something that can be be composed with output from tools that don’t produce edn, I think -M is the easier (only?) option.

practicalli-johnny13:09:31

Ah, so I assume clojure -X:alias :x '["1 2 3"]' is required for :exec-fn , which fits in the concept stucture over strings. Although more specific options seem preferable. clojure -X:alias :x 1 :y 2 z: 3 . This is less ambiguous and ties into the general approach I use for Clojure arguments.

practicalli-johnny13:09:57

Still lots to learn, so interesting to hear these thoughts. Thanks.

hugod13:09:43

I’m still trying to grasp the implication of the changes too.

Alex Miller (Clojure team)14:09:52

@U05254DQM I think mostly what you’ve gleaned is correct. -M is for pass through to clojure.main. (And originally that’s the only thing clj did.) Calling with no exec opt is for repl. Right now that’s done with clojure.main but you should consider that an implementation detail. -X is for the new function execution style invocation.

Alex Miller (Clojure team)14:09:46

We expect most tools are better off with -X, but totally possible to support both of course

Alex Miller (Clojure team)14:09:18

For your own app, not sure yet how this will be taken up. Using -X means you are at least somewhat bound to using clj as your launcher (as the exec stub code is not available anywhere else (yet)). Rich seems to think that’s fine, I’m a little less sure about it. :)

Alex Miller (Clojure team)14:09:20

My biggest issue with the -X arg passing is that passing a string path as the v requires wrapping in multiple quotes which kind of ruins - expansion and other glob and auto completion stuff. I’d say that’s not totally finalized

practicalli-johnny14:09:58

Thanks for confirming Alex. I am far more confident in my understanding of the changes. Apart from updating practicalli/clojure-deps-edn with either -M or -X variants instead of -A, there seems no impact on my development projects. My notes from a project developer are shared on this gist: https://gist.github.com/jr0cket/ae7dd745eb45870109ace59fe835ce80

practicalli-johnny14:09:22

Updating the Practicalli books with the new approach is pretty easy and can run with the 'new' and 'classic' options for a while. I've planned screencasts of using Clojure CLI tools for Clojurists Together over the next 2 months. I would rather not re-do them as high quality screencasts are time intensive. I will need to decide when to do these videos, so any thoughts on a release date for the new features would help my own planning (I appreciate any comments on dates are no guarantee). If a release is not planned until December, I'll opt for some lower quality screencasts and recreate them running up to a release.

Alex Miller (Clojure team)14:09:31

I’m hoping to get to a new stable release in the next week or two

jeremys12:09:33

Hi, I have a couple questions about deps API. There is there a complement function to slurp-deps that also merges the 'user wide' deps.edn and the 'system wide' one ? Also the docs says that resolve-deps and make-classpath are supported. Then the question is: is make-classpath-map to be considered private api ? (and so subject to change?)

Alex Miller (Clojure team)14:09:29

You can use find-edn-maps to read and merge-edns function to merge edn maps once they’ve been read

Alex Miller (Clojure team)14:09:29

make-classpath-map is public but some details of the make-classpath details are going to change, probably this week

Alex Miller (Clojure team)14:09:06

(Regarding path ordering)

Alex Miller (Clojure team)14:09:02

The whole make-classpath area is in a bit of transition

jeremys14:09:26

Ok Alex, thanks a lot! I have been foolishly experimenting with the idea of making https://github.com/JeremS/mbt based on tools deps and I ran into those questions today. If I may I have one more concerning tools.gitlibs. Right now it uses jgit v4. Do you plan to use v5?

Alex Miller (Clojure team)14:09:12

Yeah, I’ve spent some time on it but not enough to get it over the line. It also plays into the jsch ssh stuff and look at their newer impl with Apache Mina.

Alex Miller (Clojure team)14:09:46

Just FYI, I have built a tools.build tool that will hopefully be making its way toward release soon

jeremys14:09:11

Oh man... my rationale was that tools.deps begs a tools.build... A proper tools.build is very good! The jgit question was because the clj-git project uses the v5 making it, I believe, incompatible with tools deps (breakage between jgit 4 and 5) Thank you again for the info.

seancorfield17:09:58

@U0FR867U1 If it helps you, take a look at https://github.com/seancorfield/clj-new/blob/develop/src/clj_new/helpers.clj#L60-L71 -- This uses the latest tools.deps.alpha (0.9.782) but still works with both CLI stable and CLI prerelease.

seancorfield17:09:48

Re: find/read/merge

jeremys19:09:06

Thank you @seancorfield the last 2 lines are almost exactly what I need!

borkdude15:09:34

Maybe aliases should also become fully qualified?

borkdude15:09:56

Then we can have a global (worldwide) alias registry =)

trollface 3
borkdude15:09:45

I wasn't trolling (at least, not consciously)

mpenet15:09:58

I didn't mean it that way 🙂

borkdude15:09:44

e.g. want to use seancorfield's aliases -A:org.seancorfield/rebl

borkdude15:09:43

I guess the problem with that is that aliases can change over time and effectively become a dep. Importing aliases from libs might be an idea, but maybe a bit too much.

Alex Miller (Clojure team)16:09:26

aliases can be fully qualified now

Alex Miller (Clojure team)16:09:58

people typically don't because more typing, but that's fully supported

rickmoynihan08:09:51

I’ve been doing this for a while with local overrides, e.g. :local/libname will be defined in ~/.clojure/deps.edn for when I have a local checkout of it. I think it works quite well, also to distinguish user wide aliases from project ones. cc @borkdude

borkdude08:09:29

yes, doing something something similar

Alex Miller (Clojure team)16:09:04

one thing we've done a bit of thinking about is - how could I specify a tool to use and have it automatically pulled from github without needing to declare an alias at all

borkdude16:09:54

yes, that was a bit was I was thinking with this

borkdude16:09:08

bit like lein new does maybe

borkdude16:09:52

Aliases can be fully qualified now, but will it ever be deprecated that they aren't - as with deps

dominicm16:09:55

Need some aliases procurers

Alex Miller (Clojure team)16:09:03

probably not going to fully tackle that for a bit, but I think it would be pretty useful. you need some kind of naming convention that takes you to a gh repo, that can be a deps.edn with deps etc

Alex Miller (Clojure team)16:09:20

I think just github support would take you pretty far, but yes all that needs to be thought about

Alex Miller (Clojure team)16:09:37

there is also some open space I've been working in around discovery with -X

borkdude16:09:38

this can probably be solved in a tool which you can put in your aliases

trollface 3
Alex Miller (Clojure team)16:09:17

like could -X have support for listing the functions available to call

borkdude16:09:27

I wasn't trolling! geez, what is it with people today ;)

Alex Miller (Clojure team)16:09:42

how can you support automatic documentation of a -X compatible function

borkdude16:09:00

maybe metadata?

borkdude16:09:07

^:x-callable

Alex Miller (Clojure team)16:09:56

all tbd, but with combination of mapping from alias to tool repo, and support for :ns-default (to find the default tool namespace), and ability to discover the -X functions, and help for each option....

borkdude16:09:31

yes, the mapping from alias to tool repo was what I was getting at earlier

Alex Miller (Clojure team)16:09:36

and then if only had a build system that could make use of those functions...

borkdude16:09:25

build system?

dominicm16:09:04

Metadata wouldn't work by itself, you'd have to require every namespace.

Alex Miller (Clojure team)16:09:05

anyhow, I'm on "vacation", so back to vacating...

dominicm16:09:18

Yes. Begone!

borkdude16:09:18

Maybe aliases aren't the right fit for name -> github repo and some other concept/option should be invented for that.

seancorfield17:09:56

Sort of related to this line of thinking: clj-new supports templates on GitHub via naming conventions, so maybe have a look at the readme for the usage and see if it is close to what you are thinking @borkdude?

borkdude17:09:19

@seancorfield You mean:

clj -A:new -M:new ...
?

borkdude17:09:49

or do you have some short form where you only mention the org + repo name, with automatically resolving the newest sha?

seancorfield17:09:25

Currently, you need the full URL but that's really only to detect what type of template you're using. If you know it's GH, you could take just the org/repo-name and build the rest. And my dot-clojure file has an example of using tools.gitlib to figure out the HEAD of master SHA.