Fork me on GitHub
#tools-deps
<
2021-07-10
>
seancorfield00:07:37

(this may be unique to clj-new because of the way it tries to dynamically resolve/load dependencies to find the specified template? the error EDN file does not mention the missing :sha failure at all)

Alex Miller (Clojure team)00:07:10

yeah, that's just the nature of it. you can still use :sha though

seancorfield00:07:33

I think you're missing the point: :git/sha is what is installed by the CLI, not clj-new.

seancorfield00:07:59

I think this will cause problems for any t.d.a-based tool that is installed via the new CLI but relies on an older t.d.a (that does not recognize :git/sha) to create an environment for executing code?

seancorfield00:07:18

The only difference between clj-new 1.1.314 which works and 1.1.309 which doesn't, is the version of t.d.a that they depend on.

seancorfield00:07:38

(and the new release now has :tools/usage in deps.edn but that's not relevant for this)

Alex Miller (Clojure team)01:07:51

doesn’t work at install time or at tool execution time?

seancorfield04:07:45

I showed exactly what doesn't work in the main channel... Let me know if I need to explain it better but it should be an exact repro.

respatialized13:07:55

I have a kind of open ended question about deps.edn and library/program design. Say I want to offer users of my library / program two different usage patterns that have different dependencies: the use case I'm thinking of here would be one version of a program that runs purely in memory and one that uses a database for persistence. For those users who want to run it only in memory, is there a way for them to pull in my library without pulling in the DB deps as well?

borkdude13:07:17

@afoltzm usually you make these dependencies optional, by just not making them dependencies of your lib

borkdude13:07:41

if you want to provide the storage-related functionality, you tell users to include extra deps

practicalli-johnny18:07:42

I guess I missed this, it seems the : is now not used between the flags , or is the : between the flag and first alias name optional (maybe it always was and I didnt notice it)?

clojure -X:project/new
clojure -Xproject/new
That's probably a clue to get the latest version and do some UX testing 🙂

practicalli-johnny18:07:47

Aesthetically I like the initial colon, especially when using multiple aliases clojure -M:fig:cider-cljs, feels more consistent. I appreciate I seem to be weird in that I like typing... :rolling_on_the_floor_laughing:

dominicm18:07:05

@jr0cket iirc, officially only -X:fig:cider-cljs is supported, the other happens to work and the code was intentionally written to work with that kind of input.

👍 3
Alex Miller (Clojure team)18:07:54

aliases should always use :

👍 3
Alex Miller (Clojure team)18:07:20

tool names are symbols and should not use a colon (with -T)

Alex Miller (Clojure team)18:07:54

in some cases, aliases without colons are tolerated (but don't do that)

practicalli-johnny18:07:31

Ah, so T is in the same style of flags as J and S it would seem. I should definitely get the latest and have a look at clojure --help . Thanks all.

Alex Miller (Clojure team)18:07:58

when you find it is lacking, please let me know :)

Alex Miller (Clojure team)18:07:16

I do have some todos in the syntax documentation area for next week

practicalli-johnny18:07:57

In the clj -h I think this could potentially be misleading, especially those new to Clojure CLI and the concept of aliases

exec-opts:
 -Aaliases      Use concatenated aliases to modify classpath
 -X[aliases]    Use concatenated aliases to modify classpath or supply exec fn/args
 -M[aliases]    Use concatenated aliases to modify classpath or supply main opts
 -P             Prepare deps - download libs, cache classpath, but don't exec
It assumes it is known that an alias is :new and not just the name new . It is stated that aliases are keywords earlier in the texts, but that can easily be missed Perhaps adding a specific example as with -X:deps section
Programs provided by :deps alias:
 -X:deps mvn-install       Install a maven jar to the local repository cache
 -X:deps git-resolve-tags  Resolve git coord tags to shas and update deps.edn
I have always found a few specific examples help clarify the general approach

Alex Miller (Clojure team)18:07:15

The help here is something I have struggled to make simultaneously small and useful :)

borkdude18:07:28

@jr0cket Did you know clojure has a man page as well? :-) You might in fact be the person who mentioned it here, don't remember.

practicalli-johnny18:07:15

I did know, but wasnt the one who mentioned it. I will read it (when I've installed the new version)

practicalli-johnny18:07:57

In the clj -h I think this could potentially be misleading, especially those new to Clojure CLI and the concept of aliases

exec-opts:
 -Aaliases      Use concatenated aliases to modify classpath
 -X[aliases]    Use concatenated aliases to modify classpath or supply exec fn/args
 -M[aliases]    Use concatenated aliases to modify classpath or supply main opts
 -P             Prepare deps - download libs, cache classpath, but don't exec
It assumes it is known that an alias is :new and not just the name new . It is stated that aliases are keywords earlier in the texts, but that can easily be missed Perhaps adding a specific example as with -X:deps section
Programs provided by :deps alias:
 -X:deps mvn-install       Install a maven jar to the local repository cache
 -X:deps git-resolve-tags  Resolve git coord tags to shas and update deps.edn
I have always found a few specific examples help clarify the general approach

practicalli-johnny18:07:56

The man clojure page looks very nice and much clearer I believe, e.g

clj-opts
       -Jopt  Pass opt through in java_opts, ex: -J-Xmx512m

       -A:alias...
              Concatenated aliases for REPL execution, ex: -A:dev:mem

       -X:alias... fn KPATH V
              Exec alias to invoke a function that takes a map, with keypath/value overrides. Function must either be supplied or be in :exec-fn argmap for alias

       -M:alias...
              Exec clojure.main, either from opts in alias or command line

Eddie18:07:46

I am learning about how aliases can be used as replacement for the maven “provided” dependency scope and I am having trouble understanding how to accomplish a very specific kind of compilation task. My AOT compiled application is deployed into an environment that supplies some pre-compiled deps. The application references classes in these pre-compiled deps and thus they must be on the compile-time classpath. However, I do no not want these deps (or their transitive deps) in the final uberjar. I understand how aliases allow for these deps to only be provided during the “build” task, but I have not been able to figure out how to declare these deps as being compile-time only. My current understanding is that this is https://ask.clojure.org/index.php/9110/scope-in-deps-edn-should-be-added-and-not-deleted-from-pom-xml?show=9117#c9117 not supported by tools.deps because aliases can be used to same the effect. However aliases don’t have any control over if a dep is included in an uberjar and it would be the responsibility of the assembly/build tool (for example https://github.com/seancorfield/depstar) to handle the exclusion of certain deps in the uberjar. Is this understanding correct?

Eddie19:07:27

In case it is helpful, my particular environment is a https://spark.apache.org/ that provides Spark as a pre-compiled dep at runtime.

Joshua Suskalo19:07:49

I believe you are right with the build tool handling the exclusion of certain deps. It turns out that depstar has a way to specify specific namespaces/classes not be included in the final uberjar. You may look at that to solve this particular issue.

Eddie19:07:14

Thanks! I played around with the :exclude option on depstar but (as far as I could tell) you would have to carefully identify all the unnecessary transitive dependancies and curate a set of regex that exclude them by name. I asked over in #depstar to see if I was missing something.

seancorfield21:07:32

Spark is a use case that depstar is specifically designed to work with. That is why the compile alias stuff was added @U7XR2PZFW

seancorfield21:07:16

I'm not at a computer to give you more details. You need to read the depstar docs closely about aliases.

seancorfield22:07:10

(so this was Storm, but the same thing applies for Spark, and the approach is to separate out "provided" deps into an alias so it doesn't become part of the JAR but specify that alias to the "compile" phase of depstar so it is available for AOT compilation)

Eddie01:07:21

Thank you @U04V70XH6! I was indeed missing the :compile-aliases option in depstar. I’m still mentally decoupling the notion of “scopes” used in maven into the independent parts that we prefer in Clojure. Based on everyone’s responses it seems like my mental model is correct: It is not the role of deps.edn to declare the separation between compile-time and runtime classpaths, but rather this is handled by an assembly/build tool (like depstar or tools.build). Furthermore these tools leverage aliases (which is a construct provided natively by deps.edn) to compose a classpath. This is clear to me now, and I like it a lot! Thanks all. Super helpful, as always. 🙂

3
Alex Miller (Clojure team)19:07:04

I can't speak to depstar, but the new https://clojure.org/guides/tools_build library allows you to construct a "basis" (a classpath execution context) in your build with whatever modifications you need. Additionally, you can construct a different basis for different parts of your build. So for example you could define an alias with the compile-time library:

(def compile-basis (b/create-basis {:aliases [:compile-libs]}))
;; use that compile-basis with b/compile-clj 
(def runtime-basis (b/create-basis {}))  ;; default basis, matches project with no aliases
;; use runtime-basis to b/write-pom (declares your dependencies in the pom from the basis for publishing)
;; use runtime-basis with b/uber to build the uber jar

Eddie19:07:21

Thanks @U064X3EF3! I will spend some time learning tools.build. It sounds like it has the level of control that I require.

seancorfield22:07:50

depstar has options specifically for this use case (well, originally added to support Storm). I explained in the other thread.

Alex Miller (Clojure team)19:07:11

the key here being that it's a program and you can do whatever you want :)

cfleming21:07:15

With the new tools.build stuff, has there been any improvement in constructing classpaths with deps that do not include Clojure, to allow it to be used for non-Clojure projects (or non-Clojure modules in mixed-language projects)? Last time I looked at this that was impossible since the system deps.edn always pulls it in and there was no way to override that.

borkdude21:07:52

@cfleming I've been doing this in bb for a while using classpath overrides

cfleming21:07:22

Do you mean overriding the Clojure jar with e.g. an empty jar or something?

borkdude21:07:29

:classpath-overrides {org.clojure/clojure ""
                                                      org.clojure/spec.alpha ""
                                                      org.clojure/core.specs.alpha ""}

borkdude21:07:40

(sorry for indentation, copied out of context)

borkdude21:07:02

I use this for babashka dependencies which do not need clojure on the classpath

cfleming21:07:36

No worries - that’s interesting, I’ll try that, thanks. I’m surprised that works, I must admit. I’d be interested in Alex’s view on whether that’s defined behaviour or a happy accident 🙂

borkdude21:07:57

I learned this from Alex :)

cfleming21:07:35

@borkdude I guess that means you always build them using an alias like :no-clojure or something similar, is that right?

cfleming21:07:41

Thanks, yeah, I was just looking at your repo. I see you also have :classpath-overrides {org.clojure/clojure nil ... } in one place, does that work similarly?

borkdude21:07:17

that's similar yes.

borkdude21:07:40

The call to deps/-main is calling into deps.clj which is my re-implementation of the Clojure CLI in Clojure itself

borkdude21:07:04

so I'm basically calling as if I'm writing -Sdeps ... on the command line

cfleming21:07:22

Interesting, thanks. I’ll play around with that and see if I can make it work for building Cursive.

👍 3