Fork me on GitHub
#tools-deps
<
2021-10-04
>
cfleming01:10:58

Is there a reason why tools.build aliases use :deps instead of :extra-deps in its alias? Does this mean that if the build is invoked using that alias that the deps will be completely replaced rather than augmented as in :extra-deps?

cfleming01:10:58

Reading the doc, it looks like it but I’m not sure I’m interpreting that correctly.

cfleming01:10:31

If so, I don’t understand how this works. Thinking about the t.d.a API, I call (deps/combine-aliases deps-map aliases ) and then call (deps/resolve-deps deps-map alias-args) where alias-args is the result of the first call. But in doing that, I don’t get the dependencies from the :deps entry in the build alias map. I can’t see any way to actually achieve that using the API except manually merging those deps into :extra-deps before calling resolve-deps.

cfleming01:10:28

Backing up a bit for a problem statement, I’m trying to figure out how to make tools.build work with Cursive. Currently Cursive will sync dependencies from the deps.edn file to the project, so that symbols resolve and things generally work. When some of those deps are specified in an alias, the user selects which aliases they would like synced to the IDE, in order to make their dependencies available to Cursive. However this doesn’t work with dependencies under a :deps key, because t.d.a doesn’t return those to me.

cfleming02:10:48

It seems like the intention is to allow the user to maintain separate classpaths, one for the build tool and another for the project itself. There isn’t a good way to separate those in IntelliJ without some serious gymnastics, so my plan is to just provide a union of the two sets of dependencies that are available everywhere, and the user will have to take care where they use which ones. But for that to happen, I need t.d.a to return the :deps dependencies in addition to the :extra-deps ones.

cfleming02:10:37

Is there any way to use clj to output the classpath of a tool? Using e.g. clj -Spath -T:build gives me some very strange errors.

Alex Miller (Clojure team)03:10:11

:deps and :extra-deps are synonyms in aliases

Alex Miller (Clojure team)03:10:35

-T will not include the project :deps and :paths

Alex Miller (Clojure team)03:10:50

what errors are you getting with clj -Spath -T:build ?

seancorfield03:10:21

I thought :deps and :replace-deps were synonyms, not :extra-deps?

seancorfield03:10:32

@U0567Q30W Can you elaborate on "very strange errors" -- clj -Spath -T:build works and gives exactly the result I'd expect in all the projects I have a build.clj file...

seancorfield03:10:01

Example (with -Stree which is more readable):

seanc@Sean-win-11-laptop:~/oss/next-jdbc$ clojure -Stree -T:build
org.clojure/clojure 1.10.3
  . org.clojure/spec.alpha 0.2.194
  . org.clojure/core.specs.alpha 0.2.56
io.github.seancorfield/build-clj  2ceb95a
  . io.github.clojure/tools.build  599be6c
    . org.clojure/tools.deps.alpha 0.12.1048
    ...

seancorfield03:10:51

https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha.clj#L634 -- this is the context I was thinking of where :dep and :replace-deps are synonyms.

cfleming03:10:03

Ah, I see, clj -Spath -T:build works, but clj -Spath -Tbuild tells me: Error building classpath. Bad coordinate for library , expected map: nil

seancorfield03:10:01

The latter tries to run a tool called "build" and I doubt you have such a thing installed.

cfleming03:10:36

If :deps and :extra-deps are synonyms, how should I handle that in my API calls above? Currently I’m not seeing :deps returned.

cfleming03:10:08

@U04V70XH6 I see, I thought the alias was the name of the tool, but I guess I misunderstood.

seancorfield03:10:46

seanc@Sean-win-11-laptop:~/oss/next-jdbc$ clojure -Stree -Tpoly
Checking out:  at ad5693b5f0a7ec7bdbab56e94c02ca7df120f1a9
org.clojure/clojure 1.10.3
  . org.clojure/spec.alpha 0.2.194
  . org.clojure/core.specs.alpha 0.2.56
io.github.polyfy/polylith  ad5693b
  . poly/common /home/seanc/.gitlibs/libs/io.github.polyfy/polylith/ad5693b5f0a7ec7bdbab56e94c02ca7df120f1a9/components/common
  . poly/git /home/seanc/.gitlibs/libs/io.github.polyfy/polylith/ad5693b5f0a7ec7bdbab56e94c02ca7df120f1a9/components/git
  . poly/file /home/seanc/.gitlibs/libs/io.github.polyfy/polylith/ad5693b5f0a7ec7bdbab56e94c02ca7df120f1a9/components/file
    . me.raynes/fs 1.4.6
    ...
That's what you'd see with a tool you actually had installed.

cfleming03:10:46

What does it mean for a tool to be installed? I just installed one using the example from the doc, and it doesn’t touch the deps.edn. My mental model of this must be wrong, I thought that a tool being installed meant that I had an alias in my deps.edn file which would allow me to execute something.

cfleming03:10:28

I see, they’re installed under .clojure.

Alex Miller (Clojure team)03:10:53

what Sean said above is wrong

Alex Miller (Clojure team)03:10:23

-T:build runs the alias, as a tool. it does not run an installed tool

seancorfield03:10:40

I never said it ran an installed tool.

Alex Miller (Clojure team)03:10:01

sorry, I missed "the latter"

seancorfield03:10:13

Colin was confused about -Tbuild vs -T:build -- I was explaining they were different.

cfleming03:10:14

Right, so -T:build runs an alias from my deps.edn file, and -Tbuild runs one I have in my .clojure.

seancorfield03:10:35

clojure -Ttools install ... is how a "tool" is installed.

seancorfield03:10:38

And I think to answer Colin's other Q (or one of them), there's a tool function in t.d.a that produces the "tool" basis.

Alex Miller (Clojure team)03:10:40

running -Tsomething will find ~/.clojure/tools/something.edn which defines a lib/coord to use as the deps

Alex Miller (Clojure team)03:10:08

the tool function is confusingly different than tool at the CLI

Alex Miller (Clojure team)03:10:24

it pre-dates the existence of -T

cfleming04:10:01

So is it true that an alias in a deps.edn file designed to be run as a tool will not be used as a normal alias? I don’t think it’s true that :deps and :extra-deps are synonyms, because resolve-deps only looks for :extra-deps.

seancorfield04:10:03

Yeah, I seem to recall using it for quite a while...

seancorfield04:10:36

@U0567Q30W See the link I provided about :dep and :replace-deps (not :extra-deps).

Alex Miller (Clojure team)04:10:55

I am not on a computer with the code atm, but I think it's normalized during canonicalization

seancorfield04:10:16

In the confusing tool function:

(defn tool
  "Transform project edn for tool by applying tool args (keys = :paths, :deps) and
  returning an updated project edn."
  [project-edn tool-args]
  (let [{:keys [replace-deps replace-paths deps paths]} tool-args]
    (cond-> project-edn
      (or deps replace-deps) (merge {:deps (merge deps replace-deps)})
      (or paths replace-paths) (merge {:paths (vec (concat paths replace-paths))}))))

cfleming04:10:34

Thanks Sean, I hadn’t got around to clicking on that, was reading code…

Alex Miller (Clojure team)04:10:39

oh, sorry, yes - it's :deps and :replace-deps that are synonyms

Alex Miller (Clojure team)04:10:20

but in -T there are no project deps so really any of them will add deps

Alex Miller (Clojure team)04:10:59

sorry, I've been Strange Looping for the last week, my brain is out of Clojure world ;)

cfleming04:10:01

Ok. So: any tool alias will essentially create its own classpath. Is it true that tool aliases and other aliases are generally always going to be used in different ways? Would you ever have an alias that you would use as both?

cfleming04:10:11

No worries Alex, I’m jealous 🙂

Alex Miller (Clojure team)04:10:29

there is not a good public api function in tdeps that will do the real equivalent of what -T does on the cli. that stuff is buried in the build-classpath stuff atm. some reasons for that are still supporting some of the deprecated -R etc stuff that make that code not just a simple create-basis call, and some is some work that's just not done yet

cfleming04:10:57

I’m just trying to get straight in my head how to support all this in Cursive, and present it to the user in a way that will make sense.

cfleming04:10:17

Ok, thanks Alex. I’ll take a look at the code and see if I can figure it out, and come up with a plan.

Alex Miller (Clojure team)04:10:41

I think we also have a real gap here with getting a repl with the effective classpath for a tool

cfleming04:10:22

That seems like a reasonable thing to want to do.

Alex Miller (Clojure team)04:10:29

there's no combination of things to actually ask for that on the CLI

cfleming04:10:50

Separating the classpaths in IntelliJ is going to be either really complicated, or a bit crappy.

seancorfield04:10:22

@U0567Q30W For :build, can't you just run it in its own process?

cfleming04:10:29

I’ll play around with it to improve my mental model, and think about it some more.

seancorfield04:10:44

Or are you trying to do something inside Cursive with build.clj itself?

Alex Miller (Clojure team)04:10:07

it's helpful to separate • running an installed tool (-Twhatever) • running a local tool defined by an alias (-T:build)

Alex Miller (Clojure team)04:10:29

those aren't repl things but you might want a different kind of run config to do those

cfleming04:10:45

Well, the way IntelliJ works is by indexing things. In order to do that, it needs to know what the dependencies are, so that it can attach them to the project and index them. The issue is that here those dependencies are hard to determine, and it’s also hard-to-impossible to determine which files should use which dependencies. The easy-but-crappy option is just to have all the tool and project dependencies in one ginormous classpath, and let any file in the project use any of them, and put it on the user to keep that straight in their head.

seancorfield04:10:24

So you're trying to compute a classpath for build.clj in IntelliJ's world?

Alex Miller (Clojure team)04:10:37

and separately from above, I really want to get a repl with the classpath being used to run the tools scenarios above, particularly the local tool one, because I want to interactively develop my build

cfleming04:10:45

But assuming the entirely likely case that the user has a build.clj and a build/something.clj support ns, as well as all their src/**/*.clj files using a different classpath, I don’t think there’s a better option.

cfleming04:10:32

Yes, but build.clj is just a convention, right? Any tool can be used to run any clj file in the project root, as well as any other namespaces in subdirectories that they might pull in.

Alex Miller (Clojure team)04:10:36

what does intellij do for something like maven poms with optional deps?

cfleming04:10:10

The same as what Cursive does right now - you just select the aliases (or profiles) you want synced to the project, and it will sync all of them.

cfleming04:10:34

However here you have two distinct classes of dependencies, which ideally wouldn’t mix.

cfleming04:10:03

(it’s really n+1 classes of dependencies though, since each tool can have a totally separate set of deps).

seancorfield04:10:05

@U0567Q30W :build is the relevant alias tho' and it's typically :deps on tools.build (or my build-clj wrapper) and maybe some deploy stuff, and :paths ["."] -- so you already have the case that src.foo.bar inside build.clj is the same as foo.bar in the normal source context 🙂

cfleming04:10:51

Yes, but if someone is requiring src.foo.bar for their foo.bar ns, I have no problem telling them they deserve everything they get 🙂

cfleming04:10:25

But the fact that the alias is :build and the file is build.clj is just pure convention.

Alex Miller (Clojure team)04:10:43

yeah, you can't rely on that

seancorfield04:10:43

Haha... yeah, that's a pretty pathological case. I was more pointing out that when running -T:build, all of the src and test code is available, just with an extra segment on it.

cfleming04:10:57

No, it’s not.

Alex Miller (Clojure team)04:10:07

load is not going to succeed as the ns doesn't match the path

cfleming04:10:13

If you require foo.bar you won’t get that unless src is in your paths.

seancorfield04:10:29

You can load-file it and then use the namespace.

cfleming04:10:37

Unless you’re using load-file, in which case, see above about deserving what you get.

Alex Miller (Clojure team)04:10:50

but as colin says, you deserve what happens to you at that point

1
Alex Miller (Clojure team)04:10:09

that code presumably requires other code, which it will not properly be able to load

seancorfield04:10:35

And, to the earlier point, yes, :build could declare any default ns and the file could be anything, not just build.clj

Alex Miller (Clojure team)04:10:01

there seem to be separate problems with paths and with deps

cfleming04:10:18

Yes, the path thing is also a problem.

Alex Miller (Clojure team)04:10:28

with deps, it's more about finding the set of deps to "include" in the project right

cfleming04:10:50

Yes. But the tool files will need access to their deps too.

Alex Miller (Clojure team)04:10:16

yes, i was including all of the possible sets of deps

cfleming04:10:06

Right. The easy solution is to just mash all the project and tool deps into one, and give all Clojure files access to all of them, and the user will just have to be careful to not use project deps in their build files and vice versa.

cfleming04:10:31

That’s sounding like my v1 MVP to be honest.

cfleming04:10:45

The other, slightly more complex option, would be to just mash all the tool deps from the different tools into one, and have a tool classpath and a project classpath, and give the user some way of specifying which files should use which. It sounds horrible but it’s not so bad, I did this recently for Babashka, which is a similar issue.

cfleming04:10:19

Then the worst that might happen is the user might use the deps from one tool while editing another, which is at least slightly less terrible.

Alex Miller (Clojure team)04:10:18

and this is really a problem because you are not in the context of some specific deps execution (which would narrow this for you)?

cfleming04:10:59

Um, I’m not sure what you mean by “in the context of some specific deps execution”. The problem is that for a given Clojure file, I have to determine which dependencies it should be using. IntelliJ has a standard way of doing that, which is to have a classpath per module (which corresponds to a lein/deps/Maven project, i.e. managed by a single deps.edn/project.clj/pom). The issue I now have is that there are multiple sets of dependencies that code in a given module will use.

Alex Miller (Clojure team)04:10:33

right, that's what I'm saying - it's not relative to like a REPL execution that specifies an alias, it's independent of any such context

Alex Miller (Clojure team)04:10:23

seems weird this is not something that has come up elsewhere for intellij

seancorfield04:10:15

IntelliJ's model does not work well for Polylith-based repos either, which is something else Cursive has problems with because of this need to map to the underlying IntelliJ model. Right Colin?

cfleming04:10:47

Like I say, Babashka is a similar case which I did manage to deal with relatively cleanly. I might be able to do something similar here.

cfleming04:10:34

I’m not sure about that, I still haven’t sat down to look at Polylith seriously. I have some reports of problems, but I don’t think it’s at the level of “I can’t use IntelliJ for this project”.

seancorfield04:10:10

There are two separate issues. The only I'm referring to here is that projects all have their own classpath (and the :dev alias is the "development" project which is a separate classpath too)

seancorfield04:10:03

(I don't know how much it interferes with using Cursive with Polylith -- just pointing out it's a similar problem, but it may just not be one you actually need to solve)

cfleming04:10:10

In Polylith, you mean?

seancorfield04:10:14

Yes. Polylith.

seancorfield04:10:35

There is a more problematic issue with Cursive and Polylith but it's not related to this thread.

cfleming23:10:43

Still thinking about this - am I correct in thinking that I really can’t tell if an alias is intended to be used as a tool? I thought the presence of :ns-aliases or :ns-default would do, but that’s used for other things. I could look for :deps or :paths, but I guess a tool could also use :replace-deps or :replace-paths?

cfleming23:10:27

Also, is a tool required to use one of :ns-aliases or :ns-default?

cfleming23:10:15

Looks like not, since the user can always provide a fully-qualified sym to exec a function.

seancorfield23:10:55

An alias can be used for multiple, different contexts. -A, -M, -X, or -T. It's the usage that determines how the contents of the aliases contribute to the classpath. And by the time the process is running, you can't tell how it was invoked.

cfleming23:10:26

Right. So, I really need the user to tell me which aliases are intended to be used as tools.

Andrés Rodríguez21:10:17

Hey everyone! Got a bit of a messy situation over here. I've got a project with a docker-compose set-up for development with all the basic tools, an exposed REPL and mounts the source code as a volume so that the REPL can be refreshed with code as it's written. As I migrate this project to tools.deps, it has occurred that the .cpcache folder becomes invalid when the container is recreated (not to mention outside of the container) because it lives in the mounted volume but the cached deps live inside of the container's; ergo, recreating the container results in a class not found exception next time things are run. This would be easy to solve if I could set $CLJ_CACHE to a folder inside of the container but because I am in a project with a deps.edn, tools.deps ignores it, according to the docs in https://clojure.org/reference/deps_and_cli#_classpath_caching and a brief what-if-the-docs-are-wrong experiment. Mounting the host's .m2 cache as a volume is unfortunately out of question because the container has to run in root, plus I'm not sure how cross-plat that is. Any ideas?

hiredman21:10:35

this may not be helpful, but I kind of doubt there is a good reason to run your repl inside of a docker container to begin with, so if you just don't do that then no problem

1
hiredman21:10:02

like, I've get various services I need running for tests running in containers, but my code, repl, and actual development doesn't happen inside a container

1
Andrés Rodríguez21:10:53

I understand your point, but the whole team is used to being ready to work with a single docker-compose up, which prepares everything for frontend projects, Leiningen-based projects, Node.js projects, and I really would like to avoid altering their entire workflow just for one project using tools.deps.

Andrés Rodríguez21:10:36

I am making some strides in making that easier via Nix, but still not there 😅

hiredman21:10:38

in your dockerfile use sed to edit the clojure shellscript

hiredman21:10:09

∴ cat workspace/wsmain/build/clojure/bin/clojure|sed -e 316,+5d -e '315a cache_dir="$user_cache_dir"'|grep -C 3 'Determine whether'
  config_paths=("$install_dir/deps.edn" "$config_dir/deps.edn" "deps.edn")
fi

# Determine whether to use user or project cache
cache_dir="$user_cache_dir"
# Construct location of cached classpath file
val="$(join '' ${resolve_aliases[@]})|$(join '' ${classpath_aliases[@]})|$(join '' ${repl_aliases[@]})|$exec_aliases|$main_aliases|$deps_data|$tool_name|$tool_aliases"

hiredman21:10:29

if you want to be more flexible use regex addresses instead of line numbers

henrik4208:10:19

I'm using docker in a similar setup. I use a long-living maven-repo volume for ~/.m2 - not a bind-mount. That should be portable. Your colleagues just need to create the volume.

Andrés Rodríguez14:10:09

@U0NCTKEV8 I didn't know the clojure binary was a shell script. That certainly can come in handy either now or later. @U01GW6NKHP1 I might try this out, I don't know how this interacts with the container's own .m2 the base image already ships with but it sounds like it could be made to work one way or another. Thank you for the great ideas!

borkdude21:10:33

@hello525 Without going into details because it's getting late here, possibly using -Sforce can be used a temporary workaround, until you have a better solution.

Andrés Rodríguez21:10:36

That's what I'm doing as a workaround, by way of doing rm -rf **/**/.cpcache before execution, but thanks!