Fork me on GitHub
#tools-deps
<
2022-06-03
>
Sam Ritchie14:06:01

Question about git dependencies. if I have a repository with multiple projects in subfolders, is it possible to specify one of the subfolders as a git dep?

Sam Ritchie14:06:14

this is obviously not a clojure project but I’m looking to organize #sicmutils more like https://github.com/twitter/algebird, with algebird-core , algebird-generic etc as various modules. I know they can depend on each other with :local/root and picking up the root as a git dependency will correctly pull in the submodule. but can a user use git deps to grab a submodule directly?

Alex Miller (Clojure team)14:06:53

Yes, by specifying :deps/root you can pick out a sub folder of a repo

Alex Miller (Clojure team)14:06:21

I'm assuming you don't mean actual git sub modules (those don't work with tools.deps)

Sam Ritchie14:06:55

Correct just subfolders

pyr14:06:51

I mentioned this this morning, but we just changed our tool's repository's visibility: https://github.com/exoscale/tools.project. This tools tries to bring a certain number of convention to handle both standalone projects, and multi-module projects. This solves most of the gripes we had dealing with multi module projects in the leiningen world. The project itself isn't so big, but it serves as an aggregator for the many common things we rely on (antq, clj-kondo, cljfmt, slipset-deploy, deps-modules, and more)

metal 2
❤️ 2
Sam Ritchie15:06:09

Awesome I will look closely. So this provides conventions for local multi module development, and proper references between modules when you do a release?

pyr15:06:54

Amongst other things yes

mpenet15:06:53

@U017QJZ9M7W most things are built on a task runner that you can tell what to run tasks on (all modules or a subset) and when a command doesn't need that (when we detect you run that command from inside a module dir already) it will default to the corresponding standalone command

mpenet15:06:37

ex clj -T:project install from the root will run install for all modules in a single process, from a module dir it will just install that module

❤️ 1
borkdude15:06:39

I may have found a reasonable solution for shell quoting issues. If anyone ever runs into that again (e.g. on Windows) it'd be interesting if that solves the issue for the user. https://github.com/babashka/cli#usage-with-the-clojure-cli

Darin Douglass15:06:09

i just noticed that metadata annotation stuff, pretty neat! thoughts about making it :babashka.cli/cli so we can auto-namespace the key from the main babahska.cli namespace?

borkdude15:06:07

The metadata is designed to be used without the dependency, so that's not really a concern. I guess you could use :as-alias for this as a solution

borkdude15:06:41

Also the key can be used in deps.edn now and it will match the name of the library coordinate which is kind of neat ;)

Darin Douglass15:06:32

ah i skimmed over that part, makes sense 👍

borkdude15:06:49

I might have just added it :)

borkdude15:06:11

Yeah, I could move the namespace to org/babashka/cli.clj perhaps to fix this. Not sure if that's worth it... Also not consistent with the rest of the babashka libs

Darin Douglass15:06:27

eh, not really worth it imo babashka is a pretty unique initial namespace name

borkdude15:06:27

So I could consider :babashka.cli/opts

(require '[babashka.cli :as cli])
::cli/opts {...}
vs
:org.babashka/cli {...}
Hmm, not much difference I'd say

Darin Douglass15:06:10

i was mainly thinking of the use-case where babashka.cli was already required and aliased in the code

👍 1
Darin Douglass15:06:38

the only-exec-dep paradigm makes :org.babashka/cli make a lot more sense

seancorfield16:06:47

@U04V15CAJ I haven't looked closely at it yet, but (how) does it handle argument values that can be symbols or keywords?

borkdude16:06:34

Do you mean {:option 'foo} or {:option :foo} ?

seancorfield16:06:13

Yes. With standard -X, you just say foo or :foo -- because strings are the "odd one out"

borkdude16:06:38

There is no automatic coercion until you tell it to, either via metadata or via data in your deps.edn.

:org.babashka/cli {:coerce {:option :keyword}}
And you call it with --option foo which will then be turned into {:option :foo}

seancorfield16:06:33

I'm thinking of commands like clojure -Ttools install which takes a qualified symbol as a key, whose value is a hashmap (with one or more keyword keys and string values), and then :as as a key, whose value is a symbol.

seancorfield16:06:20

Ah, so you'd have to explicitly specify conversion to keyword, symbol, or even via EDN to a hash map?

seancorfield16:06:47

And you would need a specific structure to mimic/wrap the tools install (since it can have any arbitrary symbol as a key). You'd need to specify actual options for the libname and the coordinates. Just trying to get a good mental model of what it really is/does.

borkdude16:06:50

I haven't added the :edn option yet, but this can be added

borkdude16:06:12

> You'd need to specify actual options for the libname and the coordinates. Yes, that

borkdude16:06:11

It's not intended to focus on tools install itself, but to call exec functions like you would do with -X or -Tantq

borkdude16:06:24

So you get:

clj -M:antq --upgrade --skip github-action
vs.
clj -Tantq outdated :upgrade true :skip '["github-action"]'

seancorfield16:06:27

Understood. I was just thinking about different -X/`-T` argument formats that I've seen.

borkdude16:06:51

Yeah, I think what tools install itself does is kind of an exception to the rule

seancorfield16:06:11

(and I had looked at the clj-new example -- on my phone this morning, before I even got out of bed 🙂 )

seancorfield16:06:32

And how does it read the :org.babashka/cli data from the alias in the deps.edn file?

borkdude16:06:40

just a moment, dinner, back in a bit :)

seancorfield16:06:41

Just looked that up. Sneaky 🙂 Relies on reading the clojure.basis and getting the metadata from :resolve-args. I didn't realize the whole alias data ended up in there.

borkdude17:06:10

Yeah, I was happy I didn't have to do anything else. I was contemplating some project level config file but this seems the most natural place

borkdude17:06:10

Btw, you can use any function in :coerce and :collect metadata, the keywords are just for convenience (and .edn)

1
borkdude10:06:16

@U04V70XH6 From the next version (0.2.10) on, you can use clj-new like this in lein:

{:clj-1.11 {:dependencies [[org.clojure/clojure "1.11.1"]]}
 :clj-new {:dependencies [[org.babashka/cli "0.2.9"]
                          [com.github.seancorfield/clj-new "1.2.381"]]}
 :user {:aliases {"clj-new" ["with-profiles" "+clj-1.11,+clj-new"
                             "run" "-m" "babashka.cli.exec"
                             {:exec-args {:env {:description "My project"}}
                              :coerce {:verbose :long}
                              :collect {:args []}
                              :aliases {:f :force}}
                             "clj-new"]}}}
If clj-new would add that metadata, the :coerce, :collect and :aliases could be left out.

borkdude10:06:26

$ lein clj-new app --name foobar/baz --verbose 3 -f
Generating a project called baz based on the 'app' template.

borkdude10:06:04

I have some trouble deploying to clojars now (see #clojars) but this should be up once that is fixed.

seancorfield18:06:18

I'm afk but if you want to create an issue on the clj-new repo explaining the change that would make it easier for your lib...

borkdude18:06:52

Will do, have a nice weekend!

1
borkdude19:06:40

If that looks good, I'll create the PR

seancorfield19:06:03

Sure, go ahead and send a PR. Do you want to do a similar one for deps-new? I'd much rather encourage folks to use that instead of the older one with all the Leiningen/Boot baggage.

borkdude20:06:53

OK, both done now

1
borkdude07:06:31

Thanks for merging! Note that I mentioned a clj-new / deps-new version here: https://github.com/seancorfield/clj-new#babashka-cli and here: https://github.com/seancorfield/deps-new#babashka-cli but those have to be updated when the babashka CLI stuff is going to start working.

borkdude07:06:42

I've updated the docs here to use the newest SHAs: https://github.com/babashka/cli#clj-new https://github.com/babashka/cli#deps-new We could also just link there

seancorfield17:06:22

Ah, I didn't catch the version number had changed. I'll need to do new releases (and I'll update those references). I'll try to get to that tomorrow or over the weekend.

seancorfield05:06:03

New releases of deps-new and clj-new are available with support for babashka.cli! (v0.4.10 was broken, sorry, so please use v0.4.12)

🎉 4
pyr16:06:09

So quick one while I'm still looking up a few things wrt tools.project. I ended up adding a :deps/prep-lib key for tools.project itself, which does aot compilation of all namespaces at direct-linking true to speed it up. with this It's quite fast. This does make the tool usage a bit cumbersome though since you have to issue clj -X:deps prep :aliases '[:project]' before launching any command.

pyr16:06:31

Do I have another simple resort to make the tool available in AOT compiled fashion? I suppose I can't publish to a maven repo since there are dependencies to artifacts published as git only?

pyr17:06:28

Maybe building an uberjar and publishing that, which could be OK since this is a tool, but this feels quirky

seancorfield17:06:46

By "publishing" you mean putting it somewhere for folks to download, rather than using Clojars, right? Have you considered Graal and making native binaries?

pyr17:06:51

I did but some of what tools.project does doesn't work in graal. As for publishing I meant clojars

pyr17:06:18

I think for now I'll drop the :deps/prep-lib

Alex Miller (Clojure team)18:06:53

I would not (ever) publish an uberjar

👍 1
borkdude18:06:34

You can however distribute an uberjar and let people call that - but this probably not how you intend this to be used. It would be similar to distributing a graal binary though. There are also tools to make jars be wrapped in an executable wrapper

borkdude18:06:41

@U06V097TP What didn't work for you in graal? I have a native version of tools deps here (mostly experimental): https://github.com/babashka/tools-deps-native

borkdude18:06:08

What I often do with projects that need some Java compilation is publish the Java compiled stuff separately as there's usually not a lot of churn there and then do the rest as git deps / mvn deps more often

pyr19:06:46

The thing is, I want stuff to work out of the box for git clone + clj -T. We'll likely distribute a graal'd tools.project too but it should work without first having to go fetch stuff left and right which is what I'm focusing on atm

borkdude19:06:32

@U06V097TP kk - let me know if you want some help with the graalvm stuff - it's tricky with tools.build

tstout17:01:11

@U064X3EF3 Regarding never publishing an uberjar, how would you publish a java.lang.instrument agent jar?

Alex Miller (Clojure team)17:01:53

Never done it. I think the only time I’ve ever used agent jars (yourkit etc) it was part of an installation or download, not via maven

tstout17:01:16

Some locked down environments only allow maven central or corporate clones of maven central to be the source of jars. This is done to only allow jars which have passed security and license scans to be used.

tstout17:01:20

Related to this, I appreciate that core clojure is published to maven central. These locked down environments make using libs published to clojars a real pain.

Alex Miller (Clojure team)18:01:29

I would look at other agent jars in maven central and see what they do

Alex Miller (Clojure team)18:01:55

Might be kind of a special case because you’re never combining with other jars so you don’t have the dep conflict problem

Alex Miller (Clojure team)18:06:19

This is actually something I've thought about as I have the same problem with tools.build

Alex Miller (Clojure team)18:06:36

With tools.build you could easily make an alias at the root to do that for all subprojects - would that at least reduce the need a bit?

pyr19:06:13

Was this directed at me? If so I'm not sure I get the point you're making

seancorfield18:06:54

I tend to run tools.build/`build.clj` via a REPL to avoid startup costs: clj -M:build -i build.clj -r and then (build/some-task {:the "args"}) interactively as needed.

seancorfield18:06:32

It makes it easier to chain tasks as well: (-> {:projects '[api auth]} (build/run-tests) (build/uberjars))