This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-11
Channels
- # announcements (16)
- # aws (4)
- # babashka (30)
- # beginners (58)
- # bristol-clojurians (4)
- # cider (9)
- # clj-kondo (2)
- # clojure (229)
- # clojure-europe (25)
- # clojure-gamedev (1)
- # clojure-italy (4)
- # clojure-nl (13)
- # clojure-sanfrancisco (1)
- # clojure-uk (97)
- # clojured (7)
- # clojurescript (27)
- # code-reviews (2)
- # cursive (30)
- # data-science (39)
- # datomic (14)
- # emacs (12)
- # events (2)
- # fulcro (6)
- # graalvm (8)
- # graphql (14)
- # jackdaw (10)
- # jobs (2)
- # lambdaisland (5)
- # malli (4)
- # off-topic (28)
- # protorepl (13)
- # quil (7)
- # re-frame (2)
- # reagent (1)
- # reitit (3)
- # remote-jobs (5)
- # ring-swagger (1)
- # shadow-cljs (72)
- # sql (4)
- # tools-deps (182)
- # uncomplicate (4)
- # vim (9)
- # xtdb (19)
Hello, I've been using the add-lib branch. Seems to be somewhat robust, but ran into an issue installing libpython-clj, which I figured I'd note:
user> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user> (add-lib 'clj-python/libpython-clj {:mvn/version "1.36"})
user> (require '[libpython-clj.require :refer [require-python]])
...
Error: failed to access class com.sun.jna.CallbackReference
Worked fine after restarting the repl with the dep.
Are there plans to ever merge add-lib into master?
it will probably be added in some form, but may change in api/impl
:thumbsup:
It's still super useful in it's current form, so thanks for the tool!
add-lib will not always be able to safely add libs as it's trying to resolve new libs in terms of the already loaded library set
I'm not sure how the native stuff interacts with any of that either
I see. I'm not well versed on how deps are resolved, but I can imagine there's all sorts of edge cases. Thankfully it does very often work.
Agree. I do all my dev work with Atom/Chlorine + REBL and the add-lib
branch of t.d.a. and I love being able to add new dependencies on the fly! Every now and then I add one and get false
back (indicating it didn't add it for some reason) but mostly it works exactly as expected.
It would be nice if, instead of just false
, add-lib
returned more information about why it didn't add a dependency.
For sure, at some point think it would be great to give it (the feature) some signal boosting, especially if we can squash more of the edge-case issues. It really helps with flow.
well, it doesn't actually know why
it takes all currently loaded libs + the new lib and then does a full resolution, then compares the result with the prior set
Is there a conceivable "brute force" / descructive approach?
Fair enough @alexmiller -- it can't report what it does know! 🙂
there are multiple possible reasons why you wouldn't end up with something new
because everything already loaded is "pinned", adding a different version of something already loaded isn't possible (starting from a real initial dep set doesn't have those constraints, so you can use the version selection to resolve)
I see. Sounds like the current form is about as good as we can do then.
@alexmiller Here's a repeatable case of failure, from an empty REPL with the add-lib
branch:
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'jedi-time {:git/url "" :sha "8e756ecb71bbfa0b081e00d71a21c47037f1eae4" :deps/manifest :deps})
Cloning:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
Checking out: at 8e756ecb71bbfa0b081e00d71a21c47037f1eae4
false
This works fine as a regular dep:
$ clj -Sdeps '{:deps {jedi-time {:git/url "" :sha "8e756ecb71bbfa0b081e00d71a21c47037f1eae4" :deps/manifest :deps}}}'
Clojure 1.10.1
user=> (require '[jedi-time.core :as jdt])
nil
Is it due to needing the :deps/manifest
override?
(I see the add-lib
branch is about two months behind master at this point)
(`add-lib` works if I use {:mvn/version "0.1.4"}
instead of the git coordinates)
OK, 13K GitHub repositories cloned, with URLs from Clojars projects, of which 12K have project.clj files in their home directories, ~500 deps.edn files in their home directories. More stats later.
@andy.fingerhut And the other ~500 are... boot? Or a mixture?
Don't know yet. What file name should I look for? Is it boot.edn?
(are you counting deps.edn
in projects that also have project.clj
?)
build.boot
No combinations checked for yet, but will do so.
That was just raw count of those files in root dirs.
Sweet! You're a rock star!
635 build.boot files. Again no stats on projects with different combinations of those. As I said, bigger more detailed report I will send a link later, after I find out how many of these project.clj files can be read with clojure.edn/read
And then I may want to get fancy and report not only the fraction, but also the weighted fraction, weighted by something like Clojars download counts.
Hmm, so CLI/`deps.edn` is already almost as popular as Boot. Interesting.
And of course many of these repos may not have been updated for years...
Good point. Year-by-year stats based on last update would be interesting if you're thinking of going that way 🙂
That may be interesting enough to spend the extra time to get it.
deps is way more than boot in the 2020 survey
Good to know @alexmiller -- did you see my add-lib
issue above (now scrolled off)
new deps-mgmt question was lein 84%, deps 49%, boot 6%
Wow! That's an impressive uptake!
and (obviously) that's multi-select, so presumably a lot of people using both lein and deps
I saw it, but not a priority for me right now - please file a question at https://ask.clojure.org
@alexmiller I went ahead and filed a JIRA on TDEPS: https://clojure.atlassian.net/browse/TDEPS-149
Gotcha. I'll see if I can narrow it down a bit first. Should git deps work, in general with add-lib
?
off the top of my head, sure?
it's using the same resolve-deps as normal deps, so can't think why it wouldn't
Thanks.
if i had an app project and a library project, and the library project had deps.edn aliases, could i invoke those from the app project?
thanks for confirming!
is there a canonical reference on how to use deps.edn git coordinates with private github repos?
thanks!
Hello
$ clj -Srepro -Sdeps '{:paths ["a" "b"] :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
a
b
/home/souenzzo/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/a
/home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/b
/home/souenzzo/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/home/souenzzo/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
a
b
paths from "root project" are "chained" to clj-kondo/lsp-server
. I think that is a issue, maybe related to TDEPS-52
https://clojure.atlassian.net/browse/TDEPS-52I expect /home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src
as the default :paths
is ["src"]
May also be nice to control :deps/paths
👀
what version of clj are you using?
clj -Sdescribe
will tell you
if you're not using latest (1.10.1.507) could you update and try again?
TDEPS-52 is long fixed, so I'm wondering if you're on an older version
this is my output with the newest (507):
$ clj -Srepro -Sdeps '{:paths ["a" "b"] :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
Cloning:
Checking out: at 4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5
a
b
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/a
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/b
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
dunno, must be a bug, please file either a jira or a question on https://ask.clojure.org
or I guess I'll just make a jira, nvm
atlassian login system is a pure mess I have 3 accounts (clojure, work1, work2) When I'm authed as ẃork1 and try to login to vote on clojure, it blows a auth error then I need to go to work1 workspace, logout, login in clojure workspace, then(.................) 😓
after create a new browser to login in jira, there is no vote option to vote on TDEPS issues?
the jira auth is for sure, not good, particularly cross accounts. identity is tied strongly to email but the gsuite tie-ins make everything pretty bad. I don't have a good solution to that other than private browser window
if you have a clojure jira logon, you should be able to vote. that said, they have been actively destroying the user experience with their "new ui" issue view. it could just be that it's buried a menu level or two down
I'm using the old ui, so not sure I can point to how to do it
I have an old account, but I'm not able to vote for a long time anymore, at least, can't find that option
okay after spawn 5 firefox isolated instances and login with all my credentials, I'm sure that voting is disabled.
My account is
with "raw password" (without microsoft auth).
there is no way in jira to disable voting for a user
voting is enabled in our jira product instance
whether you can vote is dependent on whether you can edit issues and I have it configured in all projects that if you are a logged in user, you can edit issues
your acct looks normal - you have jira product access and good default roles like other users
if you go to something like https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-132 - you don't see a link "Vote for this issue" by the votes field on the right?
it's a link at the top with the words "Vote for this issue" which seems pretty easy to find to me
(note that you can't vote for issues you create)
I have tried to find it multiple times, and apparently I'm not the only one. I gave up on it.
well there must be something I'm missing then
there is no intent for it to not be there
I see how to vote using my JIRA account when looking at a ticket using the old view, but not the new one
There is a link "See the old view" near the top center of the frame.
if you go to https://clojure.atlassian.net/secure/ViewPersonalSettings.jspa you can configure this
turn OFF "New JIRA Issue view" - when you do, it asks you why. tell them you can't vote. :)
But agreed without explicit instructions somewhere, most people won't bother hard trying to find it.
here's a jira issue in jira about this https://jira.atlassian.com/browse/JRACLOUD-71251
sorry, I turned this off so long ago I forgot about it
278 votes to enable voting
ok, I reverted to the old view of which I was unaware it existed and now I see the votes again
Those were some determined voters, who figured out how to vote on it 🙂
also, I as an admin have no ability to change the default - only users can change their own setting. there is another jira about that.
maybe we can migrate to clubhouse, I've heard they built it in clojure, so it must be good
I'm done migrating issue data for a while :)
it was nice that they added the new issue, with fewer features, and made it the default, and prevented admins from changing that default
many years ago, I had a boss that literally deleted the issue database with that theory
not great
if users have a stake in the issue that was posted, you could also trust on them to migrate the issue, if they found it important
mostly, they just felt like we didn't care about their problems
and we had no way to notify them when we did happen to fix stuff they'd logged earlier
there is some happy medium where you mark things as "won't fix" more aggressively than feels comfortable I think
@souenzzo As a workaround, this seems to work:
$ clj -A:lsp:dev -Srepro -Sdeps '{:aliases {:dev {:extra-paths ["a" "b"]} :lsp {:extra-deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}}}' -Spath | tr ':' '\n'
a
b
src
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
This one also works:
$ clj -A:dev -Srepro -Sdeps '{:aliases {:dev {:extra-paths ["a" "b"]}} :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
a
b
src
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
(at)alexmiller, as a continuation of the bugs I mentioned a bit last night related to nixpkgs. It seems as if this isn't related to lumo, but clojure-tools itself. Since I've been tagged a lot on various tickets on nixpkgs due to lumo, I
I'd love to find a workaround. (sorry pressed enter too early) https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/clojure/default.nix
so the problem is that the current clojure, and newer ones for that matter, will by default call maven when called. While a clojure-tools does provide all the classpath needed to run the clojure executeable. So it's basically bootstrapped as it is released. The way to workaround this, would be to force clojure to not check for updates, and just use the bootstrapped jar on runtime for its own environment, and if the user then later wants to resolve dependencies with another deps.edn, it will in those cases ofc continue it's normal behaviour. The issue here is only the runtime environment of clojure and not for another project.
If your deployed application uses a command like java -classpath your:class:path:here
and whatever other options you want, then it should not call Maven stuff, unless your application does.
Alex noticed last night that clojure -Scp your:class:path:here
might still call Maven and make network connections for that, which he considers a bug, I think.
to the default.nix I posted above, I added
doInstallCheck = true;
installCheckPhase = ''
CLJ_CONFIG=$out CLJ_CACHE=$out/libexec $out/bin/clojure -Spath
'';
this fails the same way, maven not found.Have you tried java
command instead?
might require changes to other command line options, too, so not necessarily a one-line change.
#! /nix/store/b85h3rw3y20hyj6s2aqgvlm26a8v61gy-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/rz4aba9s8895b69l3kmx8qb6ir8di2xv-clojure-1.10.1.507/bin:/nix/store/6xxk9hynq5q5nky6wh8l7k4y95y3aksv-rlwrap-0.43/bin:/nix/store/kqr55dy5zjmhklzkkwc85ghf7yxk11qq-openjdk-11.0.6-ga/bin'${PATH:+':'}$PATH
exec -a "$0" "/nix/store/rz4aba9s8895b69l3kmx8qb6ir8di2xv-clojure-1.10.1.507/bin/.clojure-wrapped" "$@"
did you read my message above, completely?
the one that mentioned Alex by name
yes, it allowes network calls, you just need to specify the endpoints by hand. This is by design ("purely functional" buzzwording)
So it seems you can wait for a bug fix to clojure
command, if one is forthcoming, or try java
command.
just thinking outside the box until a fix arrives, perhaps making some sort of .cpcache file to prevent clojure from attempting to call maven
Sure, if you want to run experiments like that, go for it. If you do not know clojure
internals, you are doing black box testing.
hence me asking, I'm also ok with solving this on a slow tempo, I'd just remove lumo temporarily in that case
You could also add https://repo1.maven.org to explicit whitelist for nixpkg, if you have such access, and then rinse and repeat as other unexpected host accesses arose.
that I can't do, but what I can do, is getting all the jars clojure is attempting to download from nix and put them anywhere visible to clojure. But I just worry if adding -Spath to the clojure wrapper will interfere with normal useage (like two -Spath would be then plausable input parameter to clojure if user provides one and nix one)
that is an alternative, but then again, all the files clojure needs to run are already downloaded, so I was hoping to do --offline-mode or something similar until going into more unknown terretory
It might work to run an experiment where you run a command like ps axguwww
on the deploy box while your failing clojure
command is running to see the full command line and options of all processes, and see what command line options are actually being used.
nice, if I find the root of the bug, there's always a possibility of patching (or using forked repo etc..). I'll try this! thanks!
I don't know your time tolerance level for experiments 🙂
the fix in clojure
is just extending the flag checks around the tool invocation
(and the equivalent in the windows script)
if someone wants to make a patch, I will likely be doing a release this week and would be happy to include
just reading trough the shell script quickly, everything makes total sense, and I like how the checksum is based on arguments and environment. Because it's hard to know if the current environment is "stale", so I'd be reluctant to want to make it less strict. Just a simple
if [[ "$stale" = true && "$describe" = false ]]; then
to
if [[ "$stale" = true && "$describe" = false && "$offline" == false ]]; then
or "$dont_recalculate_classpath" instead of $offlinedon't worry about the staleness check, just add it as a flag that affects the running of the tool
this line: if [[ "$stale" = true && "$describe" = false ]]; then
and not -Scp ...
whatever the bash is for that
I see, are there any use-cases of people partially wanting to add classpath but otherwise rely on cached (or yet to be cached) classpath?
there's extra-classpaths for that I guess, so -Scp would be total opt-out of make-classpaths mechanism
yes, that's it's purpose imo
makes sense to me, with extra-classpaths well documented, shouldn't come as a surprise
bug fixed https://github.com/NixOS/nixpkgs/pull/79868 easy 😛
are you going to make a patch?
Wow, just the raw numbers of issues, PRs, and commits on that project are mind-boggling. Sounds like a buzzing hive of activity: https://github.com/NixOS/nixpkgs
looks just like the homebrew core tap :)
When investigating which project.clj files are readable via clojure.edn/read, I have noticed a couple of projects that have both deps.edn and project.clj, and their project.clj is not EDN because it has non-EDN executable Clojure that reads the deps.edn file. I am guessing that if there is both a project.clj and deps.edn file in the root directory of a project's source code, then it doesn't matter whether project.clj is EDN or not?
I've also noticed @U04V15CAJ that you have joined "`#=` club" for several of your project.clj files. I have seen plenty that have an eval-able Clojure expression in place of a version string, that evaluates to a version string, that do not prefix that expression with #=
, but simply have the expression. I haven't verified myself whether that actually works, but might be worth testing to see if it does. Either that or plenty of projects left their project.clj files broken in that way, or it only worked in some versions of Leiningen, etc.
@andy.fingerhut I have joined that club for some projects but for e.g. this one I changed that to templating later: https://github.com/borkdude/clj-kondo.lsp/blob/master/script/update-project.clj which is I think the better thing to do. I haven't heard any problems about the read-eval approach tooling-wise though
There is no problem tooling-wise that I am aware of. This is just me noticing things while checking out ~12,000 project.clj files in projects, to see which are clojure.edn/read'able, which not, and an a few cases they are not, looking at why. Alex Miller mentioned the possibility of having deps.edn use clojure.edn/read for project.clj files if a project had no deps.edn, and I'm still doing the data collection....
I've never seen #=
in a project.clj
but I've seen plenty of ~
to evaluate inline expressions... which wouldn't read as EDN, right?
deps.edn is preferred
so if it has that, project.clj won't be used
Thanks for confirmation. Will use that in my summary of findings, and mention it.
Some projects will have both even if project.clj
isn't using machinery that reads deps.edn
since some people add the latter in response to folks complaining that their projects can't be used by the CLI as git deps 🙂 So they'll start off compatible in terms of general dependencies but may drift apart later if the maintainer forgets to update deps.edn
when they update project.clj
(in the same way that maintainers of Contrib projects added project.clj
files alongside pom.xml
files and then the POM was updated but not the project.clj
file!). So it's worth mentioning that they could differ but, hopefully, such projects would be a tiny proportion of the whole...
Understood they can differ, in the absence of such automated methods. Right now I'm partly documenting/understanding, and also wanting to make the results as accurate as I know how, by counting a project with both deps.edn and project.clj as "good for tools.deps to read project.clj file, in the last resort"
So I am planning to soon calculate which subset of these projects have each of the 16 possible combinations of files in their repository home directory: project.clj deps.edn build.boot pom.xml. Any others of interest to throw into that mix?
Then I plan to report stats for, within each of those 2^4=16 groups, which ones have EDN-readable project.clj files.
Those are the only four files I can think of that are interesting. It'll be interesting to see which notionally Clojure projects have none of those four (Gradle-based, maybe?).
I'm really concerned most about project that have only a project.clj (and no deps.edn or pom.xml)