Fork me on GitHub
#tools-deps
<
2018-06-21
>
timgilbert15:06:03

Say, is there a tools.deps equivalent of lein deps, meaning "download everything you need to so I can cache it on my CI server"?

bronsa15:06:27

clj -Sforce < /dev/null should do it

🙏 8
sparkofreason15:06:48

I'm running most of my dev CLI stuff under WSL, but Cursive runs under Windows. So a tool like lein-tools-deps won't work correctly in the Cursive context. Anybody have any luck working around this?

richiardiandrea15:06:27

Oh. I hoped that would work for a couple of colleagues of mine. I have exactly the same setup for macos users

hagmonk16:06:57

I think lein-tools-deps works by invoking the clj command? If so you’d need to wait for Windows port.

sparkofreason16:06:26

I can get clj to execute under WSL using a batch script, but not sure that will be sufficient.

seancorfield16:06:30

@U06HHF230 would have more insight but I would have expected lein-tools-deps to work on Windows if you explicitly specify which deps.edn files you need to use in project.clj so it didn't have to rely on tools.deps.alpha's ability to shell out to the clojure command?

richiardiandrea16:06:01

I think lein-tools-deps calls clojure under the hood, but I saw that you can configure the location of it as well.

seancorfield17:06:46

Well, if it asks tools.deps.alpha for the (clojure-env), the shell out to clojure happens there (inside tools.deps.alpha itself). When I switch development machines (fairly soon, I suspect), I'll also be dealing with this since I use Atom/ProtoREPL on the host O/S. Right now, I'm on a Mac so the host for Atom is the same as the host for lein/`boot`/`clj` but I've already run into this on (Windows) laptop and my workaround was to start a REPL under WSL and then connect to it from Atom -- but there are still gotchas like file paths differing between Windows and WSL: when you "load file", ProtoREPL assumes that the REPL can be asked to load the file using the editor host path which fails unless you've been very careful setting up symlinks in WSL etc.

rickmoynihan22:06:44

Hi, sorry I don’t use Windows or have access to a machine I can test this setup on. Am I right that the problem is that you’re effectively running clojure in a (pseudo) linux environment, but Cursive in Windows; so the paths are resolving incorrectly? @U04V70XH6 is right that we call clojure-env and rely on tools.deps shelling out to clojure to get the env for us. Unfortunately it’s a slightly tricky problem just now and the current design is a bit brittle. I’d be open to disabling the call to clojure-env; though if we do that you’d also need to set the paths to your :install deps.edn and user one by hand. If this issue is effecting you though, please open an issue where we can discuss more.

cfleming00:06:18

I was actually planning to ask about this. It is possible to access WSL from Windows? If I try to open /usr/local/bin/clojure or whatever from Windows, does that work or not?

cfleming00:06:57

My currently plan for Cursive is to bypass the CLI stuff altogether, and just use the library. There are some outstanding questions about how to do that, such as how to get the system deps.edn - for an initial beta I’m going to bundle it with Cursive, but I’m hoping that it will eventually be available in the lib - Alex mentioned he was thinking about that.

cfleming00:06:24

Assuming I just use the lib, then it should all work on Windows.

cfleming00:06:47

Another complication is that the user will have to choose the version of t.d.a to use, and those versions don’t correspond to CLI versions, so there’s some potential for confusion.

cfleming00:06:45

And I guess if people are using the CLI in WSL and Cursive can’t access that, then that will also be confusing since you’ll end up with two parallel systems.

cfleming01:06:39

The red letters are: > DO NOT, under ANY circumstances, create and/or modify Linux files using Windows apps, tools, scripts, consoles, etc. And some following large black ones: > Creating/changing Linux files from Windows will likely result in data corruption and/or damage your Linux environment requiring you to uninstall & reinstall your distro!

cfleming01:06:05

So, I read that to mean that using the CLI from WSL will be totally inaccessible to Cursive anyway.

sparkofreason01:06:28

I think that guidance is somewhat out of date, and the interop between WSL and Windows file systems is much improved. I actually keep all the clojure project stuff on the Windows side anyway, but and do CLI stuff from WSL.

cfleming01:06:59

Right, but I think that WSL is aware of Windows but not the other way around.

cfleming01:06:15

Although I haven’t really followed it recently.

sparkofreason01:06:17

It is possible to execute clojure under WSL from Windows, bash -c "clojure". I haven't been able to make that work with lein-tools-deps yet, was planning on making a fork and investigating locally.

cfleming01:06:06

I’d be selfishly interested to know if you could execute it via ProcessBuilder from a Java app running on Windows.

cfleming01:06:15

If that is possible, then it should be possible to access it from Cursive. I’m still probably going to go lib-only unless there’s a really good reason not to because it’s much simpler, but it would be interesting to know.

cfleming01:06:58

@U066LQXPZ BTW that guidance is still in the WSL FAQ: https://docs.microsoft.com/en-us/windows/wsl/faq > One of the main limitations of using WSL is that changing Linux files with a Windows app or tool is not allowed.

cfleming01:06:15

Comment here: https://superuser.com/a/1111092 > Every file that resides on a Volfs folder (like /home) has Extended Attributes that store the Linux permissions of that file. If you edit that file on a windows editor then those attributes are lost and file will disappear from bash.

cfleming01:06:47

@U066LQXPZ If you’re running your dev in WSL, doesn’t that mean that your source files are also under WSL? How does IntelliJ access them? Or do you treat WSL as a server you deploy to?

sparkofreason01:06:01

All of the files are in the windows FS. The tooling (apart from the IDE) executes under WSL, and can work with the windows FS no problem.

cfleming01:06:42

Oh right, of course. So your Maven local repo etc is all under Windows? That makes sense.

cfleming01:06:50

So I guess the only issue would be accessing the installed deps.edn files (system and user). Do you use the CLJ_CONFIG env var to keep the user one under Windows too?

sparkofreason01:06:04

No, that is on the todo list.

sparkofreason01:06:00

It's how I have both lein and boot set up.

sparkofreason02:06:51

@U0567Q30W (clojure.java.shell/sh "clojure.bat" "-Sdescribe") works. I assume ProcessBuilder would as well.

cfleming02:06:18

@U066LQXPZ Ok, interesting, thanks.

cfleming02:06:52

I guess the hard thing is that the paths that -Sdescribe will return are all in linux-land. So even if you configure CLJ_CONFIG to /mnt/c/Users/dave/.clojure or something, that path is still invalid in Windows.

sparkofreason02:06:37

Right, you'd definitely want to move it to be under windows. You can configure WSL to behave as though Windows drives are mounted under root, so you'd get /c/Users/dave/.clojure, but obviously that still can't be used as-is.

cfleming02:06:16

I think it would be better to leave them under /mnt, since Cursive has to be aware it’s talking to WSL anyway.

cfleming02:06:33

It would make it easier to detect that it’s getting a Windows path back from Linux

sparkofreason03:06:59

Yeah, but other things like docker benefit from this (so they tell me, I haven't run into this myself). Maybe some sort of config option to indicate WSL?

seancorfield03:06:20

Microsoft Windows [Version 10.0.17692.1000]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\seanc>wsl.exe clojure
Clojure 1.9.0
user=>

C:\Users\seanc>wsl.exe clojure -Sdescribe
{:version "1.9.0.381"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/seanc/.clojure/deps.edn" ]
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/seanc/.clojure"
 :cache-dir "/home/seanc/.clojure/.cpcache"
 :force false
 :repro false
 :resolve-aliases ""
 :classpath-aliases ""
 :jvm-aliases ""
 :main-aliases ""
 :all-aliases ""}

C:\Users\seanc>

seancorfield03:06:39

Use wsl.exe to invoke Linux commands 🙂

seancorfield03:06:27

And just to be clear, when you do that, the Linux command executes in the context of Linux, e.g., user.home:

C:\Users\seanc>wsl.exe clojure
Clojure 1.9.0
user=> (System/getProperty "user.home")
"/home/seanc"
user=>

seancorfield04:06:49

(of course this totally doesn't help in terms of actually accessing those files from Windows, nor in starting/running a REPL that can accept "load file" commands from Windows)

cfleming04:06:09

@U04V70XH6 Thanks. So I’d need to make Cursive WSL-aware.

cfleming04:06:31

The problem is also still accessing the deps files from Windows.

cfleming04:06:42

(or, a problem, rather)

seancorfield04:06:17

Yeah, it's a tough nut to crack. I've dinked around a bit with Windows-Atom/ProtoREPL and WSL-REPL but not really gotten anywhere.

cfleming04:06:53

I think just using the lib and keeping everything in Windows-land looks like the cleanest solution.

seancorfield04:06:45

Well, once the lib knows how to -Sdescribe the environment...

cfleming04:06:42

Right, I’d have to hack that for the moment, probably bundle the system deps.edn in Cursive, and perhaps allow the user to edit it somehow.

cfleming04:06:23

And allow them to configure where their user one is, defaulting as required.

seancorfield04:06:42

My fallback with Boot has been to have Boot installed on both Windows and WSL and let Atom start a REPL via Boot and keep all the files in a project structure under the C: drive that I can easily map to Linux (symlink to make the same file paths without /mnt/c).

seancorfield04:06:47

When the PowerShell port of clojure hits, we'll all be a lot better off -- but we'll still have the weird "duplication" of the "home" deps.edn because that will be in a different place on Windows and WSL. I can't see a workaround for that since the user home folder has to be different.

cfleming04:06:52

You could use CLJ_CONFIG in WSL?

cfleming04:06:09

And point the linux one to where it is on Windows?

seancorfield04:06:50

I haven't tried CLJ_CONFIG...

seancorfield04:06:01

Oh right, so on WSL, CLJ_CONFIG could point to /mnt/c/Users/<whatever>/... yeah, sounds like a good approach once we have PowerShell clojure.

cfleming04:06:57

Although generally I think the best option for clients like Cursive and lein-tools-deps would be bundling the system deps.edn in the lib, and perhaps providing a method to locate the user deps.edn.

seancorfield04:06:13

When I first built boot-tools-deps I baked in the deps.edn from the brew-install GitHub repo so I didn't need clojure installed on the system. Alex regularly told me that was a bad idea 🙂 So I changed it to call (clojure-env) once t.d.a had that API. And now we're talking about the system deps.edn being baked into t.d.a anyway! :rolling_on_the_floor_laughing:

rickmoynihan10:06:29

Interesting discussion - I think I’d like to remove the call to clojure-env from lein-tools-deps and follow the bundled system approach; and just let people override the :system location if they need to.

cfleming19:06:46

@U06HHF230 Yes, I agree - this would make Cursive much easier too

cfleming19:06:03

The only issue is the need to specify the version to use.

Alex Miller (Clojure team)15:06:37

@timgilbert additionally, you could set :mvn/local-repo to a fresh directory, then capture that directory to know what you need.

timgilbert17:06:10

Am I crazy, or is tools.deps super slow at downloading versus lein deps? I'm seeing my download step go from about 40 secs to about 4 minutes on the same hardware

Alex Miller (Clojure team)17:06:26

I haven’t heard that before now

Alex Miller (Clojure team)17:06:36

Are you comparing with cold m2 repo in both cases?

timgilbert17:06:30

Yeah, they are both running from nothing on circleci servers. I do wonder if it might have to do with invalid artifacts in my tree, combined with a private S3 repo.

timgilbert17:06:31

I will do some poking around and file a ticket if I can come up with some numbers and a reproducible case

Alex Miller (Clojure team)17:06:59

Since most people have stuff in cache, most people wouldn’t notice

timgilbert17:06:42

Yeah. Does tools.deps cache misses? Somehow or another I've wound up with com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml in my deps tree, which doesn't exist, and it seems to be causing some of the latency along with several other similar artifacts

timgilbert17:06:50

(Obviously, these are things I am going to exclude from the tree once I've tracked them down)

Alex Miller (Clojure team)18:06:32

tools.deps uses the same Maven apis that lein (via pomegranate) does to download metadata and deps

Alex Miller (Clojure team)18:06:02

so I would expect it to generally do pretty similar things. there are some differences, notably that tools.deps uses its own transitive dep walk and can potentially choose different versions and may walk versions that lein deps doesn’t.

rickmoynihan22:06:41

I’ve been meaning to ask are the differences in resolution documented anywhere, or formally known? Asking because it’s relevant to leiningen projects moving deps into deps.edn and lein-tools-deps.

Alex Miller (Clojure team)00:06:51

Well it’s known to me but I haven’t gotten around to writing it up yet

Alex Miller (Clojure team)00:06:18

It’s somewhat similar to the new golang stuff

Alex Miller (Clojure team)18:06:33

tools.deps itself doesn’t explicitly do any caching of metadata or libs at all - Maven handles all of that

Alex Miller (Clojure team)18:06:11

several possible things to investigate there. if you file a ticket with a deps.edn to repro, I would eventually take a look at it

timgilbert18:06:54

Ok, cool. I'll see what I can do to come up with a reproducible case. Thanks!

kenny21:06:40

If both :sha and :tag are provided for a Git dep but :tag points to a different :sha, which SHA is used?

seancorfield21:06:46

@kenny I thought only :sha was used for dependencies and the :tag was there for use with the option that adds :sha based on :tag...?

seancorfield21:06:58

(it's easy enough to test the behavior!)

kenny21:06:48

This part from https://clojure.org/reference/deps_and_cli#_dependencies makes it a bit unclear > The :sha attribute should indicate the full commit sha > The :tag attribute is optional but can be used to indicate the semantics of the sha

kenny21:06:17

Also makes it sound like :sha is always required.

kenny21:06:09

Yes. Will test it out later today. Definitely could use some clarification in the docs though.

seancorfield22:06:34

That reads that :sha is used for dependencies and :tag is purely descriptive -- although there's an option to update deps.edn adding/replacing SHAs with the values that correspond to tags: clj -Sresolve-tags

seancorfield22:06:04

My interpretation of that is that if you have :sha and :tag and the latter doesn't match the former, it would not affect resolution. And if you ran clj -Sresolve-tags it would update the :sha to match what the :tag pointed at. @alexmiller can you confirm or deny?

kenny22:06:02

That interpretation makes sense to me. I'd still suggest adding that information into the docs 🙂

Alex Miller (Clojure team)22:06:50

Still considering whether to remove :tag and -Sresolve-tags entirely :)

kenny22:06:27

The reason I was thinking about using :tag is to convey breakage. With only :sha, determining breakage is a lot harder.