Fork me on GitHub
#clojure-dev
<
2024-01-10
>
seancorfield17:01:58

I'd like to use clojure.repl.deps/add-lib in a tool (`deps-new`) so I can dynamically add new dependencies to the classpath. In order to do that, I need to: • set up a DCL (easy -- I've done that quite a bit in tooling before) • use Clojure 1.12 (I have a workaround for https://clojure.atlassian.net/browse/TDEPS-258 so I can run tools with different versions of Clojure) • bind clojure.core/*repl* (new in 1.12) to true so that add-lib lets me use it 🙂 The latter is public and documented, and wrapping the call to add-lib in (binding [*repl* true] ..) works but I feel like I'm using a "hack"... I think -- once we have a 1.12 version of the CLI and all tools can leverage new-in-1.12 features -- having tools be able to dynamically load new deps on the classpath could be very convenient for CLI tools: so the tool can adapt to different command-line options and load just the dependencies it needs, or even loading additional third-party dependencies that users specify as arguments (much shorter than requiring them to add -Sdeps '{:deps {group/artifact {coords}}}' prior to -Tthe-tool!). So I'd like to at least discuss additional contexts where folks might want to use add-lib et al that aren't REPL contexts.

seancorfield17:01:03

In the case of deps-new, it already depends on tools.build, so creating a basis and spawning a process with the extra dependencies is obviously a possibility but that becomes a more complicated approach (IMO) -- and not all tooling is going to be depending on tools.build or even tools.deps, which is kind of the lever that 1.12 provides via add-lib anyway (not having to depend on all that stuff).

Alex Miller (Clojure team)18:01:28

Why do you need dynamic deps?

seancorfield18:01:49

For deps-new specifically, because the invocation for 3rd party templates is:

clojure -Sdeps '{:deps {com.acme.project/cool-lib COORDINATES}}' -Tnew create :template com.acme.project/cool-lib :name myusername/mynewproject
and with dynamic deps it could be just
clojure -Tnew create :template com.acme.project/cool-lib :name myusername/mynewproject
which is more like what happens with clj-new (and Leiningen and Boot), where it uses the template name to deduce a dependency and loads the most recent version of it behind the scenes.

Alex Miller (Clojure team)18:01:06

> The latter is public and documented, and wrapping the call to add-lib in (binding [*repl* true] ..) works but I feel like I'm using a "hack"... you are receiving the intended message here - that being, while dynamic dep loading is useful for interactive work, we continue to think there is big value in declaratively stating your project deps as data.

Alex Miller (Clojure team)18:01:29

what you're trying to do here is beyond that normal use case, which is fine

seancorfield18:01:49

But I can see this being useful for clj-watson as well, where it could choose to load either the NVD NIST support stuff or the GitHub Advisory Database stuff dynamically based on command-line arguments.

Alex Miller (Clojure team)18:01:01

what are you getting from the git dep? is it just data files or actual code you load?

seancorfield18:01:16

Code. That has to be on the classpath.

Alex Miller (Clojure team)18:01:52

and what's the api for that?

seancorfield18:01:58

(templates are mostly just data which can come from the file system but some templates also leverage custom code to modify the template machinery)

seancorfield18:01:05

What do you mean "api"?

Alex Miller (Clojure team)18:01:18

I mean presumably you call some function in the template code

seancorfield18:01:31

Any function on the classpath, yes.

seancorfield18:01:17

Currently, users have to specify the 3rd party deps via -Sdeps which is ugly, error-prone, and often repetitive since the template name alone could be used to infer the git dep and add it dynamically.

seancorfield18:01:22

Also, the dynamic addition assumes "most recent" which is usually what you want for dev-related tooling: lein new (and clj-new) follow this same path.

Alex Miller (Clojure team)18:01:11

I think you should at least consider as an alternative using the new clojure.tools-deps.interop apis to invoke those functions via subprocess instead of dynamically loading them as libs. it's probably a lot cleaner, and would not require dynamic dep loading. it does have some constraints in that (like tools), it expects a function that takes a map argument

seancorfield18:01:11

Which also requires Clojure 1.12 🙂

seancorfield18:01:34

You're missing a lot of context...

seancorfield18:01:02

So deps-new directly invokes the functions for the template -- they can't be executed via a subprocess.

seancorfield18:01:45

deps-new processes data -- the template.edn file -- which may include symbols which represent functions to be called in-process to modify that data and derived data. If the 3rd party dep is on the classpath, that can all be in-process. If the 3rd party dep has to be handled out-of-process, deps-new has to marshal all that data back and forth and would end up running several out-of-process invocations which is slow and nasty.

seancorfield18:01:29

lein new and clj-new (and boot new) all handle this by dynamically adding deps to the classpath, e.g., https://github.com/seancorfield/clj-new/blob/develop/src/clj_new/helpers.clj#L12-L35 (uses pomegranate).

seancorfield18:01:16

I already have the 1.12 add-lib stuff working in deps-new (latest SHA in GH) but obvs. it's a pain with TDEPS-258 one way or the other: I can "force" people to workaround that issue by using an :override-deps alias in their user deps.edn but I don't want people to have to do that for the common case (built-in templates) so I plan to make deps-new adaptive to whether it is being run via 1.11 or 1.12 -- and if you want to use this new shortcut (`:template` used to infer and add the git dep dynamically) then you can use the alias workaround -- or you can specify the -Sdeps thing and suffer the ugly syntax and repetition. This is all about end user experience -- a number of people have complained about the arcane/verbose nature of the Clojure CLI and this area is one where those sharp edges poke through enough that my end users are creating tickets about it and complaining to me 🙂

Alex Miller (Clojure team)18:01:50

I get it, just thinking about possibilities

Alex Miller (Clojure team)18:01:09

I've got other stuff I need to work on atm, but I'll think about it

seancorfield18:01:49

NP. Just wanted to raise the context of tooling as a place where dynamic deps might be very useful to provide a better UX for CLI users.

Alex Miller (Clojure team)18:01:01

dynamic loading cases exist for sure and this is probably one that makes sense

seancorfield18:01:01

Me, personally, I really have no qualms about dynamically loading additional deps into all sorts of running programs but, hey, I understand the "guard rails" are there for a reason 🙂