Fork me on GitHub
#tools-deps
<
2020-02-12
>
penryu00:02:31

First, wanted to thank @seancorfield for his dot-clojure. šŸ˜„ Also wondering if tools.deps has anything similar to leiningen's :injections functionality?

Alex Miller (Clojure team)00:02:54

instead of running a repl, you could launch your own program that started its own repl that did something similar

penryu00:02:01

ack. are there any workarounds? something as simple as (require '[clojure.pprint :refer [pp]]) could save a lot of typing over time.

penryu00:02:19

True. I'm in the process of converting from emacs/cider to vim/conjure. I'll pursue that route.

penryu00:02:20

Thank you!

Alex Miller (Clojure team)00:02:26

like clojure.main effectively just does this:

(repl :init (fn []
              (initialize args inits)
              (apply require repl-requires)))

Alex Miller (Clojure team)00:02:11

the initialize part will already be handled by the main wrapper, so don't need that line

Alex Miller (Clojure team)00:02:58

you'd want to supply the :eval hook: #(do (injections) (eval %))

Alex Miller (Clojure team)00:02:12

something like that, the default is just eval

penryu00:02:17

... I'm afraid that's a little beyond my level of knowledge of the repl/clojure.main at the moment, but I'll take some time this week to dig in to how that would work in practice.

penryu00:02:36

Appreciated, nonetheless!

penryu00:02:09

the eval hook is defined in deps.edn?

Alex Miller (Clojure team)00:02:07

clj -e "(require '[clojure.main :as m]) (m/repl :init #(apply require m/repl-requires) :eval (fn [form] (do (require '[clojure.pprint :refer [pp]]) (eval form))))"

Alex Miller (Clojure team)00:02:16

which you can then fold into an alias if you like

Alex Miller (Clojure team)00:02:28

;; deps.edn:
{:aliases {:repl {:main-opts ["-e" "(require,'[clojure.main,:as,m])(m/repl,:init,#(apply,require,m/repl-requires),:eval,(fn,[form],(do,(require,'[clojure.pprint,:refer,[pp]])(eval,form))))"]}}}

Alex Miller (Clojure team)00:02:51

thanks to the Corfield Comma for serving as bash-safe whitespace... :)

8
Alex Miller (Clojure team)00:02:19

you could fold that tiny program into a tool that abstracted out the injections rather than hard-coding them and clean this up a bit, but that's the core of it

seancorfield01:02:16

Wait... pp is available directly in the REPL started by clj already...

seancorfield01:02:08

> clj
Clojure 1.10.1
user=> *clojure-version*
{:major 1, :minor 10, :incremental 1, :qualifier nil}
user=> (pp)
{:major 1, :minor 10, :incremental 1, :qualifier nil}
nil
user=> 

seancorfield01:02:56

@penryu What injections do you want in the Clojure REPL, specifically? (given that pp already seems to be referred in by default)

Alex Miller (Clojure team)01:02:23

it's only available in user

Alex Miller (Clojure team)01:02:35

injections makes it available in every ns

Alex Miller (Clojure team)01:02:02

I dropped the above in a blog in case it's helpful for posterity https://insideclojure.org/2020/02/11/custom-repl/

ā¤ļø 16
Alex Miller (Clojure team)01:02:14

and so I can just point someone to it next time :)

penryu01:02:40

Thanks for the explanation, @alexmiller!

penryu01:02:55

@seancorfield might also be something like tools.logging or spyscope (as recommended in a post), though Iā€™m having problems getting either of them to work right now.

seancorfield01:02:56

(I never used :injections in all the years I used Leiningen and I'm not sure not sure I'd want a bunch of symbols auto-inserted into every namespace while I'm working so...)

penryu01:02:50

I just found out about :injections today, but I'm getting a feel for the [mis]feature parity of deps vs lein.

Alex Miller (Clojure team)02:02:03

the philosophy of clj is much more to give you the tools to solve your own problems, not to exhaustively provide every possible feature

šŸ‘ 4
penryu03:02:34

Thanks again for your response. I know leiningen has accumulated a lot of feature-cruft, and I appreciate how clj factors that out. Especially, thank you for the workaround in your blog post!

penryu01:02:56

Yes, it would need to be sparse in the extreme. But certain things (like pp or #spy) might be reasonable. And I intended to bury it behind an alias (`-A:spy:rebel` or just fold it into rebel) anyway, never in a clean clj session.

ghadi02:02:31

What is the reason the Corfield comma is necessary in the first place?

seancorfield02:02:13

Round-tripping to text files and back into bash (and then into Clojure as command-line arguments).

seancorfield02:02:31

(shell escaping is... facepalm )

ghadi02:02:56

is there a technical reason it can not be done with full fidelity, or has no one dived into the GNU Bash Info page?

ghadi02:02:04

or whatever Windows uses for bash šŸ˜‚

seancorfield02:02:57

@ghadi The "official" clj-on-Windows uses Powershell -- which has different quoting rules from bash (you end up triple-quoting some things -- ugh!)

ghadi02:02:16

which I think is now... bash

andy.fingerhut02:02:37

I suspect it could be done with full fidelity, but the prevalence of multiple different versions of bash on Mac OS versus different flavors of Linux adds a level of difficulty to the problem. Also not just spaces, but single and double quotes, dollar signs, braces, ... It can make a grown person weep.

andy.fingerhut02:02:58

Not sure it is a hill I want to even try to climb, nor die on.

seancorfield02:02:11

@ghadi I'm sure Alex would welcome patches for it šŸ™‚

Alex Miller (Clojure team)02:02:53

There are patches for it. They are all scary

šŸ˜‚ 4
seancorfield02:02:06

"It's hard!" is the technical reason I've heard from several people.

andy.fingerhut02:02:17

I suspect it would be difficult to maintain the shell scripts, too, and lead to copious notes on what works and what doesn't across the platforms.

ghadi02:02:20

I'm thinking of tools that accept main arguments (like Docker) usually accept a canonical form: an array of strings (which is sort of punting)

seancorfield02:02:58

The complexity comes from clojure storing all that stuff in text files in .cpcache and then reading it back on subsequent runs.

andy.fingerhut02:02:50

I know reasonable people might disagree with this statement, but bash is just barely a programming language.

andy.fingerhut02:02:26

I will end my rant-ish statements on that topic for the evening šŸ™‚

andy.fingerhut02:02:38

I lied. As a starting point for consideration, bash --version on the penultimate macOS version (Mojave 10.14) gives this output:

GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

andy.fingerhut02:02:06

On Ubuntu 18.04 Linux:

GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 

andy.fingerhut02:02:16

There's 9 years of variation possible right there.

slipset18:02:31

Also, Catalina wants you over on zsh because the more modern bash have a license that doesnā€™t please Apple

Alex Miller (Clojure team)02:02:33

I actually have a recently raised alternative to the spaces issue that avoids all the escaping that Iā€™m exploring.

Alex Miller (Clojure team)02:02:31

Raised by rich and Stu for other purposes but happens to also cover this

seancorfield06:02:59

@deleted-user Then you should write up a patch for clojure and/or tools.deps.alpha that fixes these problems šŸ™‚

šŸ‘ 4
sogaiu07:02:34

but actually, may be that's a diff issue?

andy.fingerhut07:02:32

That shows differences in command line parsing for different command shells on Windows.

andy.fingerhut07:02:22

There are differences between command line parsing for different command shells on Linux/Unix/macOS, too, but bash is a pretty commonly used one, and what clj/clojure scripts are currently written in.

andy.fingerhut07:02:34

There are things involving single quotes, double quotes, white space, etc. that can be tricky to get correct even in bash, without considering the flavors of Windows command shells described at the link you gave.

sogaiu07:02:27

i guess if there ever is a cross-platform shell that gets adopted may be this issue will be less of a problem

andy.fingerhut07:02:33

bash exists on Windows in fairly easily obtainable form. That makes things more common with other platforms, if you use bash on Windows, but it doesn't eliminate the existing difficulties with bash

sogaiu07:02:22

i use bash on windows via the git for windows installation -- but i hesitate to use it as a shell all of the time. i think there's a chance that powershell core might at some point be a nicer alternative.

andy.fingerhut08:02:45

On Linux as well? And macOS? That seems unlikely to me, but stranger things have happened.

sogaiu08:02:20

first paragraph here seems to suggest some level of support for the 3: https://github.com/PowerShell/PowerShell#-powershell i've not tried it on macos, but have on linux

sogaiu08:02:29

i'm not a power bash user but have dealt on and off with shell scripts since the 90s, and though i've historically not been an ms fan, the powershell stuff (at least in core) seemed way more sane to me (though i've used it far less than bash, sh, zsh, etc.)

sogaiu08:02:04

it's easier to do so when you start over ofc

penryu08:02:37

At what point is it worth reading code from a file instead of a command line argument? It seems tools.deps has drawn that line in the sand.

andy.fingerhut08:02:38

I am not sure what you mean by the line in the send. The clojure command lets you give Clojure expressions on the command line after the --eval option (in the init-opts part of the command line args), and/or a file name to evaluate code from named path in the help output by clojure -h

andy.fingerhut08:02:01

By the line in the sand, do you mean it lets you pick what you want?

seancorfield18:02:37

Caching. clojure does two passes -- and invokes the JVM twice -- if the classpath/main/JVM options are not cached (in a file). It does just one pass with the cache in place. That's a big deal.

sparkofreason14:02:59

Expanded version of my earlier question about javac: suppose I want to get in between the resolve-deps and make-classpath steps before running clj, say to do some other processing on files fetched from git, add resulting artifacts to the classpath. Are there any examples of tools (or whatever) doing such?

delaguardo15:02:27

You can try https://github.com/EwenG/badigeon Iā€™m not sure this is exact solution for your specific task, but could provide some missing toolā€™s parts

sparkofreason15:02:21

Yes, assuming nothing else existed, that was going to be my starting point. Very useful stuff.

dominicm14:02:17

Not that I'm aware of, that's a new, separate tool.

dominicm14:02:42

However, I think that would be really useful for the javac case.

dominicm14:02:03

Other tools are generally shell scripts which you run first, but that reduces the smooth integration with cider, etc.

andy.fingerhut18:02:52

OK, I do not have stats weighted by number of http://Clojars.org project downloads right now, but I do have some stats showing that out of 11,736 Github projects I successfully cloned, and that is the subset of them that have a project.clj file but no deps.edn file in their root directory, 91.8% of those project.clj files were readable using clojure.edn/read , and had their defproject form first, and a reasonable-looking value for the :dependencies key (or no dependencies key, which is completely correct and fairly common). More data and steps I used to collect it here: https://github.com/jafingerhut/haironfire

Alex Miller (Clojure team)18:02:36

cool (and nice repo name :)

andy.fingerhut18:02:47

The most common reason that a project.clj file threw an exception while attempting to use clojure.edn/read to read its contents: regex literals are not valid EDN. Those are reasonably common in project.clj files, for things like jar inclusions/exclusions, or other kinds of file name selection sets.

Alex Miller (Clojure team)18:02:52

that's kind of interesting

Alex Miller (Clojure team)18:02:09

I wonder if you could hack the edn reader to read those into tagged literals or something

Alex Miller (Clojure team)18:02:54

how much additional % does that cover?

andy.fingerhut18:02:02

Good question, which I thought of attempting, but haven't yet. That was the first thing in the project.clj file that caused clojure.edn/read to throw, and there could be others after that, which would only be straightforward to calculate with such a modified version of clojure.edn/read

Alex Miller (Clojure team)18:02:27

such a thing could possibly even make its way into clojure's real edn reader

ghadi18:02:28

regexes aren't portable though

Alex Miller (Clojure team)18:02:43

I didn't say make it into edn

Alex Miller (Clojure team)18:02:05

there are lots of things the clojure edn reader does beyond the edn spec

Alex Miller (Clojure team)18:02:37

and I'm not suggesting reading it into java pattern, but into a tagged literal

ghadi18:02:44

same thing for the var dispatch I was talking to you about

Alex Miller (Clojure team)18:02:14

less sure on vars if that makes sense as you're then connecting to code

ghadi18:02:33

(tangent, but I want to have the choice of whether to connect to code or read as tagged)

dominicm19:02:52

Edamame can read regex I think

andy.fingerhut19:02:53

Based on the numbers I have seen, if the project.clj files that threw exception due to regex are edn-readable in all other ways except the regex literals, I think the most the numbers above could improve would be from current 91.8% up to close to 95% or 96%, but that would be noticeable, I think.

andy.fingerhut19:02:07

Sounds like a straightforward thing to hack up locally on my system. I will let you know if I try it out. If anyone hacks up a version of clojure.edn/read before I do, let me know.

ghadi19:02:54

@andy.fingerhut you could switch to tools.reader and monkey patch that to read regex + vars

andy.fingerhut19:02:20

Cool, that may be the quickest way.

ghadi19:02:48

naughty, but straightforward

andy.fingerhut19:02:48

Maybe I should take the opportunity to see if clojure.edn/read and tools.reader edn reader behave differently on any of these 11,736 files šŸ™‚

sogaiu21:02:33

on a tangential note perhaps, regarding regexes -- there's the regal thing that expresses regexes using plain clojure data: https://github.com/lambdaisland/regal

Alex Miller (Clojure team)21:02:16

this is about existing data in project.clj files, so I don't think that helps anything

sogaiu21:02:37

you're talking about converting regexes as well as whether the result is valid edn, no?

sogaiu21:02:45

anyway, for future cases, it could be useful for folks to express their regex using regal and having a file that is more likely to be edn?

andy.fingerhut21:02:52

This is experimental exploration here, wondering how we might use a secure safe reader, either existing, or perhaps slightly tweaked, to read existing project.clj files.

andy.fingerhut21:02:16

I doubt we could ever convince hundreds of people to update the project.clj files to use a new regex representation, for something that already works for them today.

sogaiu21:02:16

sure -- just musing about future possibilities. sorry for the noise.

andy.fingerhut22:02:25

No worries. Just hoping the context makes it clear why we're talking like we do šŸ™‚

sogaiu22:02:35

thanks for your ongoing patience and explanatory efforts šŸ™‚

andy.fingerhut21:02:02

Fraction of the 11,736 project.clj files that become 'good' in my experiments (readable using clojure.edn/read without throwing exceptions, defproject form is first, value of dependencies key looks reasonable) goes up from 91.8% to 95.7% with a hacked up version of EdnReader class that can read regex literals. Details here: https://github.com/jafingerhut/haironfire/blob/master/README-extended-edn-reader.md

andy.fingerhut21:02:53

As mentioned in #clojure-dev channel discussion, my 'good' evaluation here currently has no checking for whether the dependencies list is actually the one used by the project, e.g. no checking for or evaluating the effects of Leiningen plugins, e.g. Lein-modules. There are also approximately another 1,000 git repos associated with Clojars.arg artifacts that have no project.clj nor deps.edn files in their home directory. Most of those probably have some kind of hierarchical structure to their files, with one or more independent Leiningen projects in the same source repo. I haven't tried analyzing those.

deleted22:02:55

I mean, beyond the regex literals

andy.fingerhut22:02:06

The results I linked to show number of files that give exception messages of various kinds. The next most common non-EDN things, or at least the ones that occur first in the project.clj files, are mentioned explicitly in my commentary text here: https://github.com/jafingerhut/haironfire/blob/master/README-extended-edn-reader.md

andy.fingerhut22:02:52

Short answer appears to be "in-line eval'able code inside the defproject form"

andy.fingerhut22:02:39

One could pretty easily extend an EDN-style reader to read those things, too, but I suspect the return-on-time-invested is getting fairly low.

borkdude22:02:44

> One could pretty easily extend an EDN-style reader to read those things, too, but I suspect the return-on-time-invested is getting fairly low. @andy.fingerhut That's exactly what edamame already is.

borkdude23:02:18

e.g.:

(parse-string "#=(+ 1 2 3)" {:read-eval true})
;;=> (read-eval (+ 1 2 3))

borkdude23:02:44

but :read-eval can also be a function that transforms the value in any way you want

andy.fingerhut23:02:52

Understood. Part of the context of this discussion, I believe, is what dependencies outside of itself and Clojure that Alex is willing to take on for tools.deps.

andy.fingerhut23:02:06

I do not know his detailed thoughts on the matter, other than some nuanced approximation of "as few as possible".

borkdude23:02:23

Yeah, I understand. I guess you could also have a "forgiving" EDN parser that just skips over what it cannot read. If that leaves a valid :dependencies vector, that's probably all that counts.

andy.fingerhut23:02:00

I suspect the harder part of getting a valid dependencies sequence (one thing I learned is Leiningen accepts, and a few project.clj files actually use, lists or even a map for the value of :dependencies ), is the existence of git repos with a project.clj file in the root directory, and then several more in subdirectories, with interactions between their contents, e.g. via Lein-modules plugin or other mechanisms.

borkdude23:02:05

so what is the summary of your research, what's the percentage of EDN-readable project.cljs?

andy.fingerhut23:02:57

91.8% of 11,736 project.clj files that are in the root directory of git repos, and those were selected because they do not also have a deps.edn file in the root dir, are clojure.edn/read readable in Clojure 1.10.1. If you extended clojure.edn/read to not throw an exception on encountering a regex literal, it goes up to 95.7%

andy.fingerhut23:02:01

But as I mentioned two comments ago, the percentage that are EDN-readable isn't all that one cares about if one wants to get correct dependencies from a Leiningen source repo.

borkdude23:02:05

very cool. I assume you communicated this summary to Alex as well?

andy.fingerhut23:02:16

Earlier today in this same channel.

borkdude23:02:32

true, but maybe those multi-project.clj repos are also in the <10% range?

andy.fingerhut23:02:56

I haven't tried to determine that yet.

borkdude23:02:28

I haven't had time to read slack today. Nice progress!

andy.fingerhut23:02:02

Sure, it's been fun hacking away. Also makes me realize that the actual question is harder to answer šŸ™‚