Fork me on GitHub

I'm having an issue getting depstar to work--I have tools.namespace in extra-deps but when i clojure -X:depstar it can't find tools.namespace. Details in thread


my :depstar alias in deps.edn:

{:exec-fn    hf.depstar/uberjar  
 :exec-args  {:aot        true
              :jar        "target/server.jar"}
 :extra-deps {org.clojure/tools.namespace     {:mvn/version "1.0.0"}
              thheller/shadow-cljs            {:mvn/version "2.15.0"}
              com.github.seancorfield/depstar {:mvn/version "2.1.253"}}}


the error:

$ clojure -X:depstar
Exception in thread "main" Syntax error compiling at (user.clj:1:1).
        at ...
Caused by: Could not locate clojure/tools/namespace/repl__init.class, clojure/tools/namespace/repl.clj or clojure/tools/namespace/repl.cljc on classpath.


Moving tools.namespace and shadow-cljs to my main :deps key solves the issue, but that's not ideal of course. Think I'm doing something silly here, would appreciate any insights


The depstar alias is for the dependencies of depstar


Adding dependencies there doesn't make them available to your code


Which is why adding them to your main deps key makes it work


Not sure why that is of course not ideal, if you depend on it, you need to specify it in your deps


that's right--I think I have other problems I need to figure out, actually... I don't think that I depend on shadow-cljs or tools.namespace anywhere in my code. My fried brain thought I depstar needed me to specify them but that can't be right obviously, it'd be pulling in whatever it needs on its own


total PEBKAC--was including my src/dev by mistake, which did use the dependencies i seemed to need to include šŸ™‚ thanks @U0NCTKEV8


For future reference, there's a #depstar channel so we can keep #tools-deps for general t.d.a / deps.edn questions.

šŸ‘ 2

I have some problems with the pre-release version of clojure cli. here is my ~/.clojure/deps.edn

āÆ cat ~/.clojure/deps.edn 
 {:socket-repl {:exec-fn clojure.core.server/start-server
                :exec-args {:name "server"
                            :port 5555
                            :accept clojure.core.server/repl
                            :server-daemon false}}}}
and when I try to start socket-repl in my project using clj -A:test -X:socket-repl :port 5557 it exits quietly with 0 status code


@delaguardo Have you also tried -X:test:socket-repl ?


yes, same result (


perhaps it starts a server and then quits immediately?


most likely this is what is happening


hmm, on my machine it works, so it does indeed seem to be happening with the newest one only?


yes, :version "" only that version behaves like this


btw, Iā€™m using mac os

Alex Miller (Clojure team)13:07:39

I assume this is due to changes in -X exiting where we now (System/exit 0)


sure, will do in few minutes

Alex Miller (Clojure team)13:07:09

actually, I made an update for this for the next release so don't need to file


A thought: threading functions with -X on the command line works best when functions: - Return the input map (let's call it context) with stuff added and don't return a completely new context - Use namespaced keywords for options to avoid conflicts between function inputs and outputs.


Considering this, I think you'd have to design most functions from scratch to work well with this pattern (i.e. you don't get a nice cmd line UX "for free" most of the time). But the namespaced keywords could make stuff boilerplatey as well.

Alex Miller (Clojure team)15:07:14

yes, that is the tension

Alex Miller (Clojure team)15:07:53

the key is that you very quickly are programming, so just put it in a program or use the repl

āž• 2
Alex Miller (Clojure team)15:07:45

the chaining/threading is useful in a relatively narrow set of cases

Joshua Suskalo15:07:16

It could also be implemented fairly easily here with an extra alias

clj -X:thread :aliases "[:alias-1 :alias-2]" :arg-map "{:some-arg 1 :other-arg 2}"
You could even put this in your user configuration so that you don't need to re-implement this alias for every project you'd want to do it in.

Joshua Suskalo15:07:30

Of course if you're using this as a part of the build process you probably would want to include it in every project.

Joshua Suskalo16:07:11

The only significant challenge here would be computing the correct set of dependencies based on what aliases are being loaded for this, which I imagine is part of why it doesn't seem like a great idea to add it to the Clojure CLI core.

Joshua Suskalo16:07:36

Just write it as a program like Alex says.


In the guide I see:

{:paths ["src"]
 {:build {:deps {io.github.clojure/ {:tag "TAG" :sha "SHA"}}
          :ns-default build}}}
Which can then be used with -T option. So any alias can also be a tool then, its just that the tool tool will not install things as an alias, but has its own separate global file to track what is installed?


-T:alias vs -Ttool-name


They are different. -T:alias is very similar to -X:alias except that it replaces the execution context implicitly (the docs explain this more accurately).


-Ttool-name is how you run a tool that has been -Ttools install'd and is not in your deps.edn file.


Is there a /.clojure/tools.edn file (or something analogous) that tracks installed tools? I enjoy being able to take my /.clojure/deps.edn file from machine to machine

āž• 2

Hum, kind of confusing, so the colon makes a big difference. It seems that for -X and -A the colon was optional and with or without did the same thing


With :replace-paths and :replace-deps, we've been able to run things as "tools" for some time -- it's how clj-new and depstar have both expected to be used, for example.


I feel it be nice if alias just had precedence on installed tools, and you could use either or


@dpassen1 The tools are installed into ~/.clojure under a tools folder.


> Is there a /.clojure/tools.edn file (or something analogous) that tracks installed tools? I enjoy being able to take my /.clojure/deps.edn file from machine to machine This was also something I expressed when I first heard about the tools idea, but I think the answer to this is to just make an alias in your deps.edn as always


(! 1015)-> ls -l ~/.clojure/tools/
total 24
-rw-r--r--  1 sean  staff  184 Jul 11 09:51 depstar.edn
-rw-r--r--  1 sean  staff  184 Jul  9 16:55 new.edn
-rw-r--r--  1 sean  staff  126 Jul  9 11:09 tools.edn


On a team, ideally no one has to install tools, and just git clone should have it setup so all needed tools are already there, so it be nice if I could have the tools all setup as an alias and ran with -T


(I haven't tried it, but I think you can just copy that folder from machine to machine)


@didibus just make an alias in your project's deps.edn as always?


Sure, but I think I should call out then that the new tools feature is not useful to my team


And that as it is, it seems we'll just continue to use aliases


But, I don't want to speak too soon, I've yet to play with it fully. But I was really expecting it to be more in-line with the declarative style of deps.edn to define apps


I suspect it would be useful to individuals on your team who want to install and manage their own additional tooling šŸ™‚


Ya, for that it would, but they might also find it better to add an alias so they can sync their tools install between desktop and laptop, and whenever they need to change laptop or rebuild their desktop instance


I also personally would have liked a way to mess with tools at a project level


I have ~/.clojure/ under version control and that's how I keep my laptop and desktop in sync.


The pain-point with alias was having to find the correct definition for it. Tools solve that, but it be nice to have something similar for alias. Like if I define an exported alias in my deps.edn for my lib


The same way you can define :tools/usage


I just tested on my laptop: I git pull'd the updated ~/.clojure repo from GH and ran clojure -Ttools list and the same tools/versions are installed there as on my desktop (without actually needing to install anything).

šŸ‘ 2

Ok, that's good, so it seems its just that tools don't pollute your deps.edn, but are also declarative.


Now I'm just concerned with reproducibility


But I guess those tools I'll just use aliases for


Well, the ~/.clojure/tools/<toolname>.edn file includes the git tag/sha so you can guarantee the version -- is that what you're concerned about?


I'm concerned with like, pulling an older commit of some project/application, and having even the tools be the exact same version, so I can get a complete repro of the setup


Oh, well, for project-specific tooling you should still use aliases in the project's deps.edn definitely.


That's why we vendor the whole Clojure CLI into our repo at work so we can reliably have the exact same version of the CLI on every machine.


<repo>/build/clojure/bin/clojure is what we use in all of our scripts.


Ya, that's a reasonable practice. Though I wish there was first class support for this. Like I wish in my deps.edn I could declare a dependency on the Clojure CLI, tools.deps and versions to use. I know it sounds like a weird bootsrap issue, but basically you use the globally installed clojure cli and tools.deps to pull down the defined version of clojure cli and tools.deps in the current project.


And same for all tools and aliases and everything


there is a JVM version of the clojure CLI


deps.clj, non-official, by yours truly

šŸ‘€ 2

Haha, of course, why am I not surprised!


Hum... but that might actually be useful. Maybe I could switch away from my custom launcher to using that.


clojure -Sdeps '{:deps {borkdude/deps.clj {:mvn/version "0.0.16"}}}' -M -m borkdude.deps -M -e '(+ 1 2 3)'


Though... couldn't the normal CLI also be bundled as a git lib or even in a Maven repo? As far as I know you are not restricted to java in them


It just needs to pull down the shell script


I guess it could, but running a shell script from a jar is a bit cumbersome


however, gitlibs might be more useful there

šŸ‘ 2

True, but with git libs, that shouldn't be a problem since it pull down the folders and files


Perhaps the new prep phase can install the script in a bin dir ;)


Yeah, that is definitely a new piece of OSS project's that we'll have to pay careful attention to -- since it could do "anything".


Ya, to me it seems tools -install could have just added a tool alias in your ~/.clojure/deps.edn along with the new -T option


And that way there could have also have a project tool install option, to do the same in the project deps.edn


Ok, so not trying to critique, I'm trying to wrap my head around on tools and all that's all. But, since they cannot include the project deps and path, it seems kind of limited what can actually be a tool no? Like you can't make clj-kondo a tool, or nrepl a tool, etc.

Alex Miller (Clojure team)21:07:54

you can do those things


@didibus How do you think depstar knows how to build JARs?

Alex Miller (Clojure team)21:07:31

those are all tools that require some "basis" (execution context) to run in

Alex Miller (Clojure team)21:07:50

functions are provided to compute a basis


do all tools have to depend on to be able to do that?

Alex Miller (Clojure team)21:07:42

yes, unless you are picking up the basis from an .edn file or something

Alex Miller (Clojure team)21:07:57

and that is something I'm thinking about


I guess one could combine a lib + to make an installable tool as a new project


so the lib doesn't have to depend on tools.*

Alex Miller (Clojure team)21:07:44

what is the reason to not depend on tools.*?


less deps, more good. graalvm compatibility, etc.


I would say "most people" don't care about that though... šŸ™‚

Alex Miller (Clojure team)21:07:48

less deps is theoretically better, but I'm not here an actual problem (yet) there

Alex Miller (Clojure team)21:07:05

and if you're building a tool to run via the CLI, not sure why graal matters for that use case


It helped depstar tremendously, in terms of ease of adding features, once I stopped worrying about "no deps" and just embraced t.d.a šŸ™‚


I don't know, seems it could cause incompatibilities both ways. Like if the tool depends on the installed tools.deps, if the installed one is too old or too new might break the tool. On the other hand, if the installed one supports new features to define paths or deps the tool might not pick up on those if it uses an old tools.deps

Alex Miller (Clojure team)21:07:21

an actual problem is a deps conflict between what a tool needs and what tools.* needs


well, e.g. clj-kondo can run both as a JVM tool and a graal-compiled tool. I'm extremely cautious with adding stuff to it which makes the binary bigger or causes incompatibilities for example

Alex Miller (Clojure team)21:07:39

I have seen one or two cases related to maven libs there


but I would probably make a "wrapper" for the clojure CLI tool case if that were a problem


(and I did run into that -- as I reported yesterday(?) -- where tools installs things and assumes :git/sha will be recognized, but I installed a version of clj-new that depended on an older t.d.a and when it tried to resolve deps, the tool runner failed because there was no :sha in the new.edn, only :git/sha)


I have since updated clj-new to the latest t.d.a and released that to Clojars so, hopefully, no one else will run into it


also loading less code could make things faster. loading when it's not necessary for all lib uses cases seems a bit wasteful


Do the tools deps merge with the project deps? Like could you override the tools deps?


That question doesn't even make sense to me... they are completely separate...


Tools are run in an isolated context, insofar as they ignore any project deps.edn file.


Ya, maybe I just find that weird haha


Why? They're tools -- they're not part of your project.


Because with aliases, you have full control, if the alias needs tools.deps.alpha, I can pin wtv version I want for the alias in my project


So if I use a newer tools.deps.alpha with git/sha support, I can try using it with the alias as well, and chances are it works


You can still use whatever you want with aliases. That's completely separate from tools.


I guess conceptually, I find it weird that a tool is not somehow related to a project. Like clearly I could have one project that has a deps.edn from 3 years ago, and another one using a new deps.edn with all the new features. So maybe I need the version of the tool compatible with both depending on the project


Like I said earlier: if you want a "tool" tied to a project, use an alias in the project's deps.edn file.


You can still run it "like a tool" with -T:the-alias.

Alex Miller (Clojure team)22:07:54

tools are a way to give a user-specific name to a lib/coord

Alex Miller (Clojure team)22:07:43

with support for install at the CLI (no file twiddling) and invocation by name


Ah, ya so that would solve it no? So I could have installed depstar as an alias and used override-deps in the alias to depend on the newer tools.deps.alpha and ran it with -T:depstar. Assuming you had not updated depstar to the latest, this would have been one workaround correct?


Yes, rather than install depstar as a "tool" and using -Tdepstar.


Ok, well at least there is that. So I guess one is convenient, the other solves edge cases. Which is good to have both


I'm already loving that I have -Tnew app and -Tnew lib and -Tdepstar jar and -Tdepstar uberjar available for any project now, without needing aliases in my ~/.clojure/deps.edn file šŸ™‚

catjam 2

Ok, but what are you loving about it over aliases? Just that you don't need to manually edit your ~/.clojure/deps.edn and find the correct alias from some readme file? Or is it that you like to keep the deps.edn uncluttered?


It will allow me, over time, to remove a lot of stuff from my deps.edn file -- and the tools install stuff makes it easier to find the right versions of tools and upgrade/downgrade them: I can do it from the CLI instead of editing my deps.edn file.


Ok, I guess I just don't see how that could not all have been done for aliases as well, just providing a cli for it


Which would have naturally extended to project specific uses as well


A lot of people don't want a CLI tool editing their files.


I guess it comes down to the "brew install" vs "nixos" way of doing things. Personally I'm leaning towards the deps.edn approach to keep things reproducible but it's nice to have options, I guess.


The nice thing is that we have both approaches available now: locally-installed tools (with a manifest that can be sync'd across machines) and alias-bound tools, either in the project deps or your own personal deps.


Ya, I prefer the deps.edn approach as well "nixos" style. But I guess I'll deal with it šŸ˜› I think it would have been fine to have like a convention of a aliases folder (or even the tools folder), but where you can just have more files that define more aliases.


Ya, but we don't have the thing I wanted the most, to have a way to install aliases from the CLI šŸ˜ž


As I just said "A lot of people don't want a CLI tool editing their files." šŸ™‚


@didibus That's just a script away


No its not, it requires full buy-in from the library authors


roll your own


You need to declare your alias in your lib in a way a cli could figure out what to install


That's why the tools approach pays attention to :tools/usage in your library's deps.edn file šŸ™‚


I don't have a lot of churn in my deps.edn anyway, I edit it probably maybe once a month or less


Right, but so that's for tools, would have been nice for aliases as well


I don't see why I would want to install depstar as a system-wide tool. I want a reproducible version of depstar in my project in CI.

āž• 6

For any tool really


We've just cut our project aliases way, way down (now that we have a build.clj script instead of trying to use bash and the CLI and aliases instead of actual functions).


That's unrelated no? Its just all the functionality of these disparate aliases are all merged into


@U04V15CAJ With depstar you may be right. clj-new makes more sense as a global tool.


@didibus No, that change was unrelated to


I usually write projects with my own bare hands without templates, but yes, that's a good example :)


Our build.clj existed before was released šŸ™‚


Ok, but still confused, what does that relate to tools or aliases?


Your build.clj is a new tool?


Sigh. I'm commenting on how few aliases we have in our project deps.edn and that they don't change very much -- unless we do wholesale project-level changes. And we made some changes earlier this year (which I blogged about) specifically to avoid having developer tooling in the project, i.e., so developers could use any tooling they wanted.


No, build.clj is "just" a Clojure script. It replaced our build bash script (and a few auxiliary bash scripts).


I guess you've missed folks here talking about switching over to using Clojure programs to do "build stuff" with instead of various non-Clojure scripts...


Oh I see, all my Clojure build stuff are already Clojure programs šŸ˜›


@seancorfield when can we expect the "monorepo part v" post? šŸ™‚


Soon @jasonhlogic soon! We're working on finalizing our build.clj stuff -- now that we can align it with now that's been released! -- and then I can write the next article, about how we switched to using Clojure for our test/build pipeline and how we adopted šŸ™‚


(I'll also have more to say about Polylith, since we're now up to 16 projects, 3 bases, and 23 components!)

polylith 4

should shadow-cljs be a tool under the new regime?


I thought Shadow-cljs was primarily installed via npm or something?


Right, but hardly anyone uses that directly I thought?


my deps.edn has

:main-opts  ["-m" "shadow.cljs.devtools.cli" "watch" "ui"]
where ui is a subproject in the monorepo


OK, so it's a -m invocation if you do that -- which is not how -X/`-T` work.


Tools invocation assumes the "exec fn" model, where you invoke a specific function, passing in a hash map.


could that not be a wrapper around shadow-cljs?


Someone -- theller or someone else -- could write a -X-compatible entrypoint/wrapper for it I guess.


prior to, i was trying to do too much in deps.edn. task-like aliases, logical deps sets, version defaulting for subprojects. a lot of aliases use shadow-cljs but only as an entrypoint to start a process. that feels more like a tool to me?


I don't do any cljs so I'm not familiar with it (I dabbled with figwheel-main a bit last year).


When I saw that shadow-cljs was so node-centric, I went looking elsewhere. I don't do JS/node and don't have any of that tooling installed (and don't want to).

ā¤ļø 2

But you would need a -X-compatible entry point in a GitHub project to use it as a tool (or in a Clojars artifact).


i understand your npm concerns but it allows js lib access just as maven coordinates allows java lib access. a useful hack.


i'll see if theller is interested


Would be cool to make shadow-cljs a tool I think. But since it requires npm anyways, and npm already supports local installs, it might still be best to keep it using npm


shadow-cljs does not make sense as a tool. it is not standalone and needs to be part of the project classpath


shadow-cljs is two parts right now, unfortunately I named them the same name. the shadow-cljs CLI tools published as an npm package, which handles only the CLI. and then the real thing shadow-cljs the CLJ library, where all compilation and stuff happens


so a "tool" could technically replace the npm part but that doesn't make too much sense IMHO. maybe there could be a namespace for an :exec-fn-like API though


well maybe it could be a tool. by the nature of the CLJS compiler it needs the project classpath, which I guess it could construct via deps.edn itself and manage its own secondary JVM much like the npm shadow-cljs tool does now


What I gathered is that tools are expected to depend on tools.deps.alpha and use its api to build the classpath for the folder it is run in


So a tool which needs a project classpath would still get one by using tools.deps.alpha as a library for getting it from the deps.edn file


well or it could just run a secondary clojure -M process similar to what the current shadow-cljs CLI npm thing does


but since you'll still need npm anyways for actual JS packages I'll probably keep the CLI package too. Starts fast and does everything you'd want. For clj purists a :exec-fn namespace seems useful, more so than a tool


Ya I guess it could do that too.

thheller05:07:44 like stuff is already fully supported and has been from the beginning so nothing to do there. meaning everything you can run from the command line you can also run from the CLJ REPL as regular functions

šŸ‘ 2

Ya, it does seem to possibly fit better as an alias with -X


Not sure that's even really useful, but if people are currently invoking it with clojure cli using -m as an alias, I guess it would probably be slightly better for them


most people using it directly today (without the npm CLI) use it in embedded mode though. so starting it as part of their user/start or so, together with their CLJ server


some have integrated it with libs like integrant, component or mount


What you said about I hadn't thought, ya I guess someone can just create themselves build tasks in build.clj that just call into shadow-cljs clojure API and then they can run shadow commands using the clojure cli with -T:build alongside their other commands they'd want


or lein of course. users will likely use the direct CLJ api a lot more than the CLI


Ya most likely


Hum, actually it seems now that I think of it, that all "build tool" or "build" related tooling will be better offered as a clojure lib to be used within While pure "tools" would be the kind that don't care about the project you are in and are more like generic Clojure tools you'd want installed globally, like if I wrote a grep tool in Clojure and you want to "install" it so you can use it.


yeah, thats actually how shadow-build (predecessor to shadow-cljs) started. only functions, no command-line API. just got too tedious to create the same few functions over and over again in each project, so a proper CLI and thus shadow-cljs was born šŸ˜‰


Interesting, I wonder if a similar thing will happen with, like if people will feel copy/pasting all the basic tasks will start to be tedious.


Though it has the advantage that each function becomes a "command" on the cli, even though a bit tedious: clj -T:build command


yeah, that is my concern with just the same functions in every project, likely generated by some template. we will see I guess. doesn't have to be a bad thing, makes things easiert to customize (if you know what you are doing)


@U05224H0W just for clarity, would you be against adding the introspection keys to shadow-cljs's deps.edn?


not at all but as I described you really cannot run it as a tool currently. shadow-cljs needs full access to the project classpath which tools don't get (AFAIK)


so adding those things really doesn't do anything


also shadow-cljs uses lein and the deps.edn is not really up to date currently


Tools can use tools.deps.alpha or, now, to calculate the project basis and that includes a project classpath, which you can then load dynamically, if you need to. But I agree that if you have a "tool" that really needs to execute in the context of a project's classpath -- and test runners are a good example -- then making them a "tool" makes little sense.


i concede the point šŸ™‚


but i do have a few apps which use the documented shadow-cljs deps setup, so it would be nice if that was up to date (which is a different discussion)


(and probably belongs in #shadow-cljs at this point šŸ™‚ )


whats wrong with :main-opts? that doesn't go away šŸ˜‰


:main-opts is great for some types of invocation (where argument processing needs to be *nix-compliant in some way or arguments can be kind of ad hoc -- and all derived from strings) but exec fn/args is nicer from a programmatic, composability p.o.v. and all exec fns take standardized arg formats (unrolled EDN hash maps, basically).


From a tool author's p.o.v., exec fn/args means that you don't have to write any argument parsing, and your entry points can (now) all have their help/documentation rendered from the command-line (via the new -X:deps help/doc stuff -- although I haven't quite gotten my head around how exactly it works).


shadow-cljs watch app is probably the most common command. kinda gets annoying with -X IMHO. so :main-opts looks better


but I'm open to adding stuff for CLJ purists so whatever makes sense I'll add


Also, from a user's p.o.v., exec args are great because they combine across multiple aliases -- :main-opts is always "last one wins" and does not combine across aliases.

šŸ‘ 2

So they both have their useful spheres.


(and, yes, simple commands like watch app do become more verbose with -X which is why some commands are simply better off with -M / -m)


I feel the word "tool" will confuse countless people going forward, that a "build tool" is not a Clojure "tool", but just an "alias".


i've never been happy with how it fit into the semantics of my deps.edn aliases