Fork me on GitHub
#tools-deps
<
2021-07-28
>
Alex Miller (Clojure team)13:07:19

hey all, I've been out for the last week or so and I'm declaring Slack bankruptcy so if you posted something above, I didn't read it. as always, if you have a bug or feature request, please ask it at https://ask.clojure.org so I do not lose track of things. I get to everything there (eventually).

😄 9
🌴 21
seancorfield16:07:41

I think the only thing that hasn't yet made it into ask is a discussion about "multiple aliases" in the docs and that was partly because we haven't formulated what the problem really is that needs addressing...

seancorfield16:07:38

And that maybe belongs as one or more issues on the clojure-site repo anyway...

Derek20:07:03

1.10.3.933 was released? cool 😎

Alex Miller (Clojure team)20:07:11

still working on release notes, patience :)

Derek20:07:53

I saw the homebrew tap change

Derek20:07:54

just excited

Alex Miller (Clojure team)21:07:37

Clojure CLI https://clojure.org/releases/tools#v1.10.3.933 is now available. Changes since last stable release: • deps.edn ◦ https://clojure.org/reference/deps_and_cli#_git • If a git library name follows the repo convention names, the :git/url can now be inferred (`:git/url` can also be specified explicitly and takes precedence) • :git/tag and prefix :git/sha can now be specified instead of the full sha. Both must point to the same commit. ◦ :sha has been renamed to :git/sha but the original is still supported for backwards compatibility ◦ :deps/prep-lib - a new top-level key can be used to say how a source lib should be prepared before being added to the classpath. This key’s value is a map with :alias, :fn, and :ensures. See https://clojure.org/reference/deps_and_cli#prep for more info. ◦ :tools/usage - a new top-level key can be used to provide the :ns-default and :ns-aliases context for a tool • https://clojure.org/reference/deps_and_cli#tool_install - git-based programs that can be installed with a local name. Tools can provide their own usage context in deps.edn. ◦ Added new auto-installed tool named tools with https://clojure.github.io/tools.tools install, list, remove. See https://clojure.org/reference/deps_and_cli#tool_install#. ◦ Install a tool with clojure -Ttools install <lib> <coord> :as <toolname> ◦ Run a tool with clojure -T<toolname> fn (also takes -X style args) • https://clojure.org/reference/deps_and_cli ◦ New -T option is like -X (executes a function) but does not use the project classpath, instead uses tool classpath (and adds :paths ["."] by default). -T:aliases is otherwise same as -X. -Ttoolname resolves named tool by name and uses that tool lib. ◦ https://clojure.atlassian.net/browse/TDEPS-198 - -X and -T will not wait to exit if futures/agents have been used ◦ https://clojure.atlassian.net/browse/TDEPS-182 - Improve deprecation messages to be more accurate ◦ https://clojure.atlassian.net/browse/TDEPS-183 - Fix -Sdescribe output to be valid EDN on Windows ◦ https://clojure.atlassian.net/browse/TDEPS-179 - Fix incorrect classpath when :classpath-overrides removes path ◦ Delay computation of local-repo path (don’t compute at load time) ◦ Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.12.1003 • New -X:deps programs: ◦ find-versions - to find versions of Maven or git libs or tools ◦ prep - use to https://clojure.org/reference/deps_and_cli#prep source libs ◦ help/dir - to list available functions in a tool namespace ◦ help/doc - to list docs for a tool namespace or function

🎉 32
dpsutton21:07:55

does this mean 933 is a stable release?

seancorfield21:07:02

Thanks in particular for TDEPS-198!

vemv21:07:07

Congratulations! Unrelated question, I have in mind a program that would take a deps.edn file as input, and would compute a derived deps.edn as the output. We can consider it a pure defn written in vanilla clojure. (under Lein this would be a "plugin") Now let's say I want people to use my program idiomatically, having a good integration with other tools, IDEs etc. Is there a "tools.deps way" to do it?

seancorfield21:07:43

@vemv Sounds like a classic use of the new "tools" functionality!

seancorfield21:07:56

Write it as a defn that accepts a hash map of options. Put it up on GH and tag it. clojure -Ttools install libname '{:git/tag "v1.2.3"}' :as thing and then clojure -Tthing

seancorfield21:07:10

(assuming you want it as a public tool for others to use?)

seancorfield21:07:17

Otherwise, if it's just local, put it in an ns that's accessible via an alias and clojure -T:myalias (with :myalias specifying :exec-fn and :exec-args etc and the path to make the ns available).

seancorfield21:07:33

As for "IDEs", that's going to depend on the IDE. But "tools" can be invoked from the REPL easily enough if you go with the alias route.

vemv21:07:36

Thanks so far! Yes I suspected the new tools might be related however I never got the gist of them. I just see that one can add/remove them but what do they do?

seancorfield21:07:22

They are "installed" via an EDN file under ~/.clojure but don't affect your deps.edn.

seancorfield21:07:47

Otherwise they're just like any code that you would otherwise use via -X in earlier releases of the CLI.

seancorfield21:07:20

(with the key difference that -T doesn't include the project classpath when invoking the function -- just the tool itself)

👍 3
seancorfield21:07:26

I have both clj-new and depstar installed as "tools" locally so they don't need to be in my (user or project) deps.edn files and I can just say clojure -Tnew app :name myname/myapp to create a new application project, for example.

seancorfield21:07:55

And you can now easily find tagged versions of tools (and libraries) without needing to trawl Clojars or the source repo:

(! 700)-> clojure -X:deps find-versions :lib com.github.seancorfield/clj-new
Cloning: 
{:git/tag "v0.1.0"}
{:git/tag "v0.2.0"}
...
{:git/tag "v1.1.321"}
{:git/tag "v1.1.324"}

vemv21:07:05

Is there a particularly handy doc on tools ? I've tried on two different days to get them but I still don't for example clojure -Tnew app :name myname/myapp appears to run an arbitrary clojure program but what I want to do is to read the current project's deps.edn file (after applying any user-provided aliases, which I cannot know in advance), and compute in return another deps.edn value for tools.deps to use i.e. I want to affect how tools.deps itself works for a given project.

borkdude21:07:14

wouldn't it be "easier" if the output sorted the newest version first?

seancorfield21:07:45

@U04V15CAJ It would likely scroll off the screen -- better to have the newest last so by the time the command has finished outputting all versions, the most recent ones are still visible.

borkdude21:07:38

or perhaps it could just show a max number of version

borkdude21:07:57

I don't see why I would want to have all of the version back to the beginning, usually

seancorfield21:07:41

Yeah, it does seem a bit redundant for projects with long histories...

borkdude21:07:00

@vemv you want to transform the deps.edn in the local dir?

borkdude21:07:14

i.e. rewrite it?

borkdude21:07:50

what does a user provided alias have to do with the problem: the deps.edn doesn't change because of an alias?

borkdude21:07:18

or do you want to calculate a "basis" and then make tools deps use that?

vemv21:07:22

> @vemv you want to transform the deps.edn in the local dir? yes, but only at runtime read the computed deps.edn after aliases -> change it -> return it to tools.deps for it to continue whatever it was doing

borkdude21:07:42

ah so you won't actually change a file, I get it

vemv21:07:08

> what does a user provided alias have to do with the problem: the deps.edn doesn't change because of an alias? the computed value changes, I cannot just slurp deps.edn (I think you got that now. Posting for completeness)

borkdude21:07:41

what you maybe want is invoke a java process with a basis

borkdude21:07:52

once you get the basis from the tools.build create-basis function you can create a java process with that

vemv21:07:32

Sounds good. Would the final solution be idiomatic? An important requirement is that this should be concise and standard for users to do / IDEs to integrate Just like when one tells people "use Lein plugin foo", which is a well-understood thing to follow

Alex Miller (Clojure team)21:07:43

it sounds like you want to dynamically modify the deps that are being computed and used by the CLI, and that is not a thing

borkdude21:07:50

isn't -Sdeps something you could use for that as well?

👍 3
Alex Miller (Clojure team)21:07:26

there are a variety of ways you can dynamically compute/modify a classpath, but that's not available in a pluggable way in the CLI

borkdude21:07:41

(note: -Sdeps is merged, not a replacement for the local deps.edn)

borkdude21:07:37

perhaps describing a concrete use case would help find a better fitting solution

vemv21:07:42

Yes I checked out clojure --help and it seems to have the necessary building blocks. Can be done At the same time I was afraid of it being bit of a hassle for others to integrate. Hence the "easy as a plugin" remark

borkdude21:07:12

note that dealing with the CLI can be prone to OS/shell-specific issues

borkdude21:07:51

But invoking a java process using a basis is not.

vemv21:07:36

[...] > but that's not available in a pluggable way in the CLI has it been considered? should I ask on ask.clojure? Hope the use case is not a no-go

Alex Miller (Clojure team)22:07:07

not something I'm very interested in

Alex Miller (Clojure team)22:07:43

we have all kinds of affordances provided to modify the deps in common ways

Alex Miller (Clojure team)22:07:01

if it becomes dynamic requiring code to run, all tools and external users have to understand how to do that

Alex Miller (Clojure team)22:07:16

this is imo at odds with the notion of deps as data

Alex Miller (Clojure team)22:07:41

I haven't heard what you actually need to do - start there

vemv22:07:54

cheers. I want to make https://github.com/clojure-emacs/enrich-classpath easy/standard to use for deps.edn users It currently is both a vanilla clojure lib and a Lein plugin. It takes dependencies as data and returns a superset of those dependencies as data. For Lein users, I can instruct them to install a plugin. Not sure of what I should tell deps.edn users to do The functionality should work in arbitrary projects, and in face of user-provided aliases chosen at runtime. It should play fine and composably with any other core deps.edn functionality.

Alex Miller (Clojure team)23:07:11

so I think there are a few parts to this... 1. given deps/aliases etc, determine the full set of libs 2. for each lib, ensure that the javadoc or source jar is downloaded for the lib to the local repo 3. create a classpath that contains the original classpath + source/javadoc jars

👀 3
vemv23:07:09

thanks! The devil in the details resides in how to impl 3 . Would understand if it needs some hammock time. For example clojure --help gives me some building blocks, but the overall solution would seem excessively bespoke. Ideally people could just use a defcustom https://github.com/clojure-emacs/cider/blob/5aaaa3a01310f5f9118465f5371a75dfea35b856/cider.el#L160 (or whatever customization convenience the IDE provides) specifying a standard tools.deps flag with the would-be "plugin" id. So that it doesn't feel like an invasive or non-standard solution

Alex Miller (Clojure team)23:07:13

1. is well covered by tools.deps create basis stuff 2. is actually not directly supported by tools.deps at all (although you can target the lower level utils and probably get some use out of it for that) 3. creating the classpath is not hard once you have the set of things. there is not really any way to get in the loop right now to replace the classpath used by the CLI and I remain dubious that that's a good thing to do. The CLI does have -Scp which lets you just supply a classpath and replace everything. So you could create the classpath, then shell out to clojure -Scp ... or whatever, but that's a lot grosser than what you're looking for

Alex Miller (Clojure team)23:07:54

I don't think this is an area we're likely to change soon

vemv23:07:56

Thanks, yes it's useful to get a feel of how likely are things to change I think it's doable to craft my own script that does the right thing, then users would be customize this other defcustom https://github.com/clojure-emacs/cider/blob/5aaaa3a01310f5f9118465f5371a75dfea35b856/cider.el#L146 Then again a bespoke script doesn't necessarily compose well with other stuff (what if someone else also wanted to promote a similar script for a different feature?) But it can be OK to just release an "MVP", see if people like the ugly-ish script and then there could be a popular demand for something more standard

vemv23:07:19

For further context, the underlying feature has been desired for many years, I still see people asking about it on #cider every other week

Alex Miller (Clojure team)00:07:44

Is there some reason all these files need to be on the classpath?

Alex Miller (Clojure team)00:07:44

seems like for javadoc especially there are likely to be conflicting resources

Alex Miller (Clojure team)00:07:44

seems like mucking with the classpath is just asking for trouble. surely IntelliJ and Eclipse and other Java IDEs don't do that

vemv00:07:12

> Is there some reason all these files need to be on the classpath? yes, without them on the classpath CIDER can't possibly access the source artifacts, necessary for its Java jump-to-source integration to work. Similarly, other features work by javadoc parsing

vemv00:07:09

I've used this plugin for the entirety of 2021 without a problem, would be open to a specific insight though

vemv00:07:19

> surely IntelliJ and Eclipse and other Java IDEs don't do that it's a good observation. CIDER is fundamentally runtime-powered. Java IDEs can afford a more static approach

Alex Miller (Clojure team)00:07:08

I still don't see why "being on the classpath" is necessary for anything you just listed. it's just opening files and doing file stuff.

Alex Miller (Clojure team)00:07:08

another idea - if you either exploded or uberjarred all the files, you could just add one mega thing to the classpath

vemv00:07:36

> I still don't see why "being on the classpath" is necessary for anything you just listed. it's just opening files and doing file stuff. I agree, it's something I've considered but relying on files to be randomly opened breaks (reinvents?) the Resource abstraction. Arbitrarily many other tools can be based on Resources, i.e. they rely that if there's some interesting source or javadoc to parse, then it's on the classpath. I can't coordinate with everyone (including future tools!)

vemv00:07:55

> another idea - if you either exploded or uberjarred all the files, you could just add one mega thing to the classpath maybe. what I'm doing right now is intentfully placing all sources/javadocs at the tail of the classpath. It's one the few things that made the plugin reliable

borkdude21:07:19

you want to make a launcher it seems that injects some deps or whatever? you still haven't described it fully

vemv21:07:10

we're clear on the requirements now, let's keep it concise

borkdude21:07:17

actually the requirements still aren't that clear to me, but I'll go do something else ;)

vemv21:07:20

> ways you can dynamically compute/modify a classpath [in a pluggable way] these are them cheers

seancorfield21:07:15

I thought you were asking about a standalone process to produce a modified deps.edn file -- your requirements were certainly not clear to me 🙂

vemv21:07:14

That's slightly besides the point. A. Miller made a precise summary of what was being asked, so asking for a further use case seemed redundant from then on