Fork me on GitHub
#tools-deps
<
2019-05-21
>
didibus16:05:27

When using a git repo, can you make it depend on a particulars branch head instead of a specific sha?

didibus16:05:34

Say I always want the latest commit

didibus16:05:13

My idea would be to have a release branch, where each commit is a new stable release

seancorfield16:05:50

@didibus Take a look in my dot-clojure file -- I give an example of how to resolve master HEAD to a SHA in there.

seancorfield16:05:15

Basically, tools.gitlibs provides tag/branch resolution

didibus16:05:11

I see, but that's basically the same as the user going to github to lookup the latest sha, just instead he can find it from inside a Clojure repl

didibus16:05:31

Or asking him to always dynamically inject the dependency

seancorfield16:05:19

You can hide it away behind a bit of code, but yes, it requires a lookup and a dependency load (my example is via add-lib).

didibus16:05:32

I was hoping for something where I can just say, add this alias and done. Now the user always gets the latest release, so my CLI is pretty much auto-updating.

didibus16:05:13

Like how mvn/version can be "RELEASE"

seancorfield16:05:54

Yeah, it would be nice if :sha (or an equivalent) could specify a label to resolve first -- but that goes against repeatability.

seancorfield16:05:29

And I believe even Maven's RELEASE is considered deprecated? Or at least somewhat bad practice...

ghadi16:05:48

tags are references, tools.deps only deals in values

Alex Miller (Clojure team)16:05:50

using a changing version destroys the ability to cache and select versions

Alex Miller (Clojure team)16:05:58

so it's intentionally not supported

didibus16:05:27

Yes, and I understand it in the Java world. I think Release is deprecated or even removed in Maven 3.x. But in the Clojure world, tools.dels is also used as a minimal package manager for Clojure build tools and CLI apps. For that, repeatability isn't a concern, but auto-update is.

didibus16:05:07

Your dot-file demonstrate that use case.

Alex Miller (Clojure team)16:05:00

auto-update is not a use case tools.deps is trying to cover right now

didibus16:05:00

Hum... Can't you still cache the sha of the head? Or you mean, the cache won't be as good since it will always need to check the repo for the latest commit sha?

Alex Miller (Clojure team)16:05:33

sha's don't change meaning

didibus16:05:59

The workaround I'm thinking is to wrap my CLI in another Clojure dependency that performs a dynamic add-lib and resolves latest. A kind of get-latest shim. But I kinda hate it

didibus16:05:49

That one would never need updating (hopefully), and would always just bootstrap my latest

Alex Miller (Clojure team)16:05:30

not saying this isn't a use case worth considering, just not one we are trying to cover right now

didibus16:05:32

I can live without it and patiently wait. Just thought I'd see if there was already something of that sort.

didibus16:05:43

I might be able to create an update command on the cli itself that edits the deps.edn file to its latest sha. That might be cleaner.

didibus16:05:13

I am very pro for reproducible builds. And now that I'm thinking about it, I can't see good reasons for local deps.edn to not be version specific, even in their version of build tooling, test runner, etc. Since you want everything exactly the same to reproduce. But for the global deps.edn, which is more for installing Clojure utils on your machine, which as I see it, is like a package manager for Clojure apps, such a feature would be great.

Alex Miller (Clojure team)16:05:45

Although you’d be pretty unhappy if your tool auto updated and broke stuff

lilactown16:05:09

yeah I like the fact that I can pin the tools to a specific sha. makes troubleshooting much easier

seancorfield17:05:41

I only use RELEASE for dev tools locally, although I made it the default in new projects created with clj-new -- on the assumption that when you start off, getting the latest versions is fine but you'd change it to a specific version as soon as you start to do any serious work.

seancorfield17:05:38

(and I didn't want to get into the situation of lein where it often lags way behind Clojure's releases and you need a whole new lein release just to bump the Clojure version in new projects)

lilactown17:05:06

good point seancorfield

didibus17:05:38

Hum... Ya. I'm wondering if a tools.deps tool could itself suffice. Like just have a -A:update-latest-deps

didibus17:05:09

Which would resolve latest maven and git releases and update your tools.deps file.

didibus17:05:26

So the user is in charge of when to update

didibus17:05:33

It could even have a rollback feature

didibus17:05:49

And what not, to make it more resilient in the case of breakage

didibus17:05:27

Also, another thought that crossed my mind, and I'm just brainstorming at this point. I've noticed my default usage of Clojure cli is -A. Would it make sense then to default all unknown commands to be treated as an alias?

didibus17:05:57

So clj new bla would be treated as clj -Anew bla

didibus17:05:57

Like when it's not a valid path to a script

seancorfield17:05:35

Most likely you would not want all Git deps updated, just certain ones.

didibus17:05:39

Ya, I'm sure it could be enhanced with some options to exclude or include. Or even made interactive so it asks you Y/n for each one you want to update

didibus17:05:47

Or to be honest I'd rather have to use say -s for running scripts, and aliases be the no option default

seancorfield17:05:35

It's kind of important to have the -R/`-C`/`-M`/`-O` alias options tho' right? -A is convenient but kinda lazy.

didibus17:05:38

Hum.. Ya I was wondering that. I guess I'm very much thinking here about the specific use case where tools.deps is used as an app launcher for Clojure and a build script launcher.

lilactown17:05:36

I won't speak for alexmiller / et. al. but it's my impression is that they are only interested in solving "resolving dependencies and constructing the classpath" right now

didibus17:05:24

Ya, I'm kinda just wishy washy throwing some ideas which I haven't put much thought into. So none of this is criticism or even feature request, since I'm not even convinced any of this is a good idea yet.

didibus17:05:39

It's just that, the more I think about "resolving deps and constructing the classpath" the more I realize that's basically all you need for being a package manager and app launcher for Clojure.

jmv17:05:12

hi all, i'm trying to get a coworker setup with a clojure project and we are hitting this error

org.eclipse.jgit.api.errors.TransportException: [email protected]:foo/bar.git: fromBase64: invalid base64 data
is this an ssh auth error?

jmv17:05:04

to follow up, we found the culprit. it was a bad entry in the known hosts file.

didibus17:05:34

The only real part missing is the idea of a "package". Its technically what an alias is. It's a preconstructed set of dependencies and main entry point which together consists of a working release of an app. But currently, aliases aren't easily shareable. Like you can't publish them, or update them easily. And launching them is a little ackward due to the use of -A.

seancorfield17:05:21

@didibus The one thing I really "miss" with aliases is the abilty to compose them in deps.edn. Otherwise you either need to duplicate stuff under multiple aliases or the user has to specify multiple aliases on the command line.

seancorfield17:05:44

(and I do generally use -A out of convenience rather than caring about whether I need -R or -M -- even tho' for things like test-runner, being able to use -R:test vs -R:test -M:test or -A:test would be "better practice" in a way)

didibus17:05:17

Hum.. I feel maybe it's that aliases are currently used as packages, where really they were meant as ways to optionally add more or less dependencies to your running code, for development purposes mostly.

didibus17:05:45

Maybe adding something like -P would make more sense for these use cases.

didibus17:05:16

Where its clear you won't be changing the set of dependencies, but fully using the one defined by it

seancorfield17:05:34

I don't see the "alias as package" similarity at all... puzzled

didibus17:05:33

So, I might want to inject a test runner and also have my src on the path and run that. So I create an alias to inject the test runner. That's one use case.

didibus17:05:00

But maybe I want to be able to run a command line editor implemented in Clojure. So I add an alias to it in my global deps.edn file and then to run it I do clojure -Acljedit bla.txt

didibus17:05:05

This command installs the cljeditor app on my machine, and launches it. On the next run, the install step is skipped (because of caching).

didibus17:05:55

In this case, I'm not interested in composing classpath dependencies. I'm not even coding. I want to install and run a Clojure app.

didibus17:05:29

In that way, the alias is defining a package in the sense of Clojure app.

didibus17:05:24

When tools.deps is used as a replacement for lein or as a build tool. Its kinds doing the same thing as well. In that you're adding all these convenient Clojure cli tools to your development environment. The difference is some of them also leverage tools.deps as the way to add your own code sources and resources which the build tool needs on the classpath as well.

didibus17:05:58

Which is where composing classpath comes into play I guess.

dominicm17:05:00

It would be really cool to have an upgrade bisect where an attempt is made to update your dependencies and then run the tests. If they still pass you're good, otherwise you remove the upgraded dependency which caused the tests to fail and try again

👍 4
didibus17:05:24

That's a great idea

Alex Miller (Clojure team)17:05:56

This does not need to be in tools.deps

👍 4
didibus17:05:02

Ya, most likely is best as a tools for it. Could be added to one of the existing test-runners

dominicm18:05:22

@didibus I was actually thinking that a tool which processed your deps.edn should take a "test" script to run that as long as it doesn't throw (or match whatever the clojure.test failure format is)

didibus18:05:34

That's even better, that way its decoupled from the concrete test-runner and might be extended to work for all of them.

dominicm18:05:10

Or even an arbitrary piece of code

dominicm18:05:17

"This code used to return 10, which dependency changed that?"

Alex Miller (Clojure team)18:05:45

would intersect well with some stuff Rich (in the form of codeq and REBL) and me (in the unreleased spec-aware testing tool) have worked on with respect to fine-grained code dependencies

seancorfield18:05:03

ears perk up... "spec-aware testing tool"? Mmm... sounds lovely!

Alex Miller (Clojure team)18:05:35

I've talked with you in the past about it, I'm pretty sure

seancorfield18:05:52

It's been a while tho'... glad to hear work is still ongoing...

Alex Miller (Clojure team)18:05:27

it's not. I mean it's not dead, but I haven't touched it in 2.5 yrs

seancorfield18:05:04

haz a sad Hopefully it will eventually see the light of day?

Alex Miller (Clojure team)18:05:13

but it was always part of the original spec idea and may eventually get revived

Alex Miller (Clojure team)18:05:57

it may make more sense to build on top of codeq 2 whenever I get back around to it (which is also not released, but is in active development)

Alex Miller (Clojure team)18:05:25

and starting to get real interesting when combined with some new things coming in REBL

Alex Miller (Clojure team)18:05:01

and Datomic, and Clojure itself, now that I think of it

seancorfield18:05:09

I'm always happy to see new features in REBL since it's a core part of my day-to-day workflow now.

Alex Miller (Clojure team)18:05:45

next release will have some great new stuff in it, Rich has been in the hammock :)

didibus17:05:20

Anyways, I'll think about all this some more and I'll stop with my thought dumping 🤤

Alex Miller (Clojure team)18:05:04

one of the goals with introducing clj was to encourage more "small" build programs. I think that's been pretty successful. I think it's also useful to step back with where we are now and look at how people are using those, what use cases/problems we could support better, and what we could do to help. Eventually, we'll get around to that.

👍 4
dominicm19:05:44

Running multiple programs seems to be the difficulty people keep stumbling over, from my observations at least.

Alex Miller (Clojure team)19:05:45

like a static set of things, or something where there is same input to all of them?

dominicm19:05:59

Figwheel, and then nrepl are the two I mostly see. And then combining them with rebel. It's not well suited to multiple mains because they need to background (except for rebel). But I also consider most of them individual choices rather than project ones.

seancorfield19:05:37

That's why we wrote our build script: it takes a series of arguments and runs clojure for each one (plus some alias manipulation).

seancorfield19:05:38

But the multi-main in one JVM is another use case that could be nice as a built in. It's no big deal to write your own -main to start up multiple things tho'...

seancorfield19:05:17

We have a dev.repl/-main that starts up various combinations of tooling for that.

Alex Miller (Clojure team)19:05:38

what if clojure.main took multiple -m's ?

dominicm19:05:19

@alexmiller it might work, I've done a little thinking, and I'm guessing that most tools would need to take a -d flag. The other side of this is "activation", the interface needs to be supportive to a user combining their own development programs with those of the project.

dominicm19:05:17

Demonize. Sorry.

Alex Miller (Clojure team)19:05:41

by "multiple -m"'s, I assuming each would have their own main + args

Alex Miller (Clojure team)19:05:10

so if foo or bar took -d, that would be fine

dominicm19:05:14

Me too. I just mean that eg nrepl would need a flag to indicate it should background itself.

Alex Miller (Clojure team)19:05:41

ah, unless they launched in parallel in different threads :)

Alex Miller (Clojure team)19:05:58

definitely things to work through, but seems non-crazy to me

seancorfield20:05:46

It would definitely be interesting if multiple -m options launched the first N-1 in threads and the last one ran directly -- and then it waited on all those threads to complete.

ghadi20:05:45

no thanks

ghadi20:05:03

sounds like that can be done in userspace

ghadi20:05:41

if there were multiple mains, I'd expect the -main functions to return before moving on to the next one

seancorfield20:05:15

Both options have their pros and cons. Maybe a -d option to run -main in a thread and -m to run -main sequentially is a good approach?

ghadi20:05:15

there are so many choices in this neighborhood

Alex Miller (Clojure team)20:05:50

I wouldn't leap to that conclusion

ghadi20:05:18

the conclusion i deleted? simple_smile

Alex Whitt20:05:24

Sorry to interrupt... is there a feature similar to lein clean in Clojure CLI?

seancorfield20:05:44

clj -Sforce is probably the closest.

seancorfield20:05:17

It ignores the cache, recalculates the classpath etc, and updates the cached values from that.

Alex Whitt20:05:46

Hmm, ok. Thanks!

seancorfield20:05:15

Both options have their pros and cons. Maybe a -d option to run -main in a thread and -m to run -main sequentially is a good approach?

seancorfield20:05:52

Although I guess you can get -d already with -e "(future ((requiring-resolve 'some.ns/-main)))" ... except for argument processing

hiredman20:05:44

of course then you have to decide who gets to call shutdown-agents

👍 4
dominicm20:05:01

I guess the thread needs to have interruption configured for shutdown

dominicm20:05:22

Or basically daemonization

Alex Miller (Clojure team)20:05:09

let's just reinvent pods! :)

seancorfield20:05:25

Or not... 🙂

dominicm20:05:09

Boot is coming for clj I believe, so maybe that will happen.