Fork me on GitHub
#tools-deps
<
2018-12-04
>
borkdude12:12:11

does tools.deps support git deps that have an edn file in a subdirectory? do you have to reflect this in the git url?

borkdude12:12:37

e.g. my-project/lib2, my-project/lib2

dominicm12:12:42

@borkdude you can use :deps/root to specify a sub-directory

👍 8
shaun-mahood17:12:01

Specifically for use with REBL and openjdk, is there a way to tell REBL bring in the correct version of JavaFX as part of the extra-deps? I don't actually need it at the moment (I switched to the Java version to get it to work), but it seems like it would be useful to prevent problems (I ran into it on 2 of my machines that I use regularly for Clojure work).

seancorfield17:12:30

Maybe ask in the #rebl channel?

👍 4
dottedmag20:12:36

It's a pity that existing .main-files are lossy: both arguments are separated by spaces and spaces within arguments are written as spaces.

dominicm20:12:10

@dottedmag you mean via :main-opts?

dottedmag20:12:23

Any idea how to invalidate those? I could use readarray Bash builtin to read \0-delimited file into a Bash array.

dottedmag20:12:31

Hmm, I can change $ck, but that invalidate all existing cached files, no matter if :main-opts had had spaces in them or not.

dottedmag20:12:12

@alexmiller ^ would that be acceptable?

dominicm20:12:34

https://dev.clojure.org/jira/browse/TDEPS-56 🙂 some work is happening now. Ideas are being thrown around. \0 delimiters might be interesting, but can't parameters have \0 in them?

dottedmag20:12:05

@dominicm That's the ticket I'm working on. No, Bash cannot keep \0 in variables or arguments.

dottedmag20:12:44

There is even a hack in Bash to use '' as a "delimiter" to denote "use \0 as a delimiter".

dominicm20:12:55

Didn't realize you were down in the trenches with this 🙂

dottedmag20:12:28

Also, execve(2) API accepts NUL-terminated strings, so no way \0 can be passed through.

dottedmag20:12:52

Well, the patch itself is not so complicated. The only remaining issue is backward compatibility with previously written .main files.

dominicm20:12:17

On a personal note, it seems to me that updating clj may/should invalidate all my caches

alexmiller21:12:50

it is supposed to, as the install-level deps.edn is included in the cache calculation

alexmiller21:12:14

but “invalidate” != “delete”

alexmiller21:12:32

as the idea of writing a script that goes around rm’ing your dirs scares the crap out of me

alexmiller21:12:00

using a .main2 seems perfectly acceptable too

timgilbert21:12:30

I wonder if this would be usable for the string-splitting problem: http://mywiki.wooledge.org/BashPitfalls#hosts.3D.28_.24.28aws_....29_.29

timgilbert21:12:08

...or if it would be acceptable to fall back to a tool written to handle this kind of thing, like awk or something

timgilbert21:12:53

...or if the string could be written to .main separated by commas, and then read into the array on the bash side via readarray -d ,

alexmiller21:12:54

would prefer to a) not add any additional deps if at all possible and b) have the clojure program write something as dumb and obvious as possible to the .main file

dottedmag21:12:50

@alexmiller Only locations of install-level deps.edn files are included in the key of cache, not their contents, right?

pesterhazy21:12:12

i'd like to point out that I've put some thought into this and proposed a solution for quoting: https://github.com/clojure/brew-install/pull/3/files#diff-b3212e45e19f61de4754a755466b793f

pesterhazy21:12:24

(and submitted a first pass at a patch)

dottedmag21:12:57

@alexmiller How about this one? Or I can do .main2, it's even simpler.

pesterhazy21:12:00

if I'm not on the right track, happy if others take a swing at this, but I wouldn't want effort needlessly wasted if possible

dottedmag22:12:06

@pesterhazy Seems much easier to do \0-based output and read back using readarray like this:

-    main_cache_opts=($(cat "$main_file"))
+    declare -a main_cache_opts
+    readarray -d '' -t main_cache_opts < "$main_file"

dottedmag22:12:32

The only remaining change is to ignore .main and and read/write .main2 instead.

dottedmag22:12:27

@pesterhazy t.d.a change is also trivial:

--- a/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj
+++ b/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj
@@ -76,7 +76,7 @@
         (when (.exists jf)
           (.delete jf))))
     (if-let [main-opts (seq (get (deps/combine-aliases deps-map (concat aliases main-aliases)) :main-opts))]
-      (io/write-file main-file (str/join " " main-opts))
+      (io/write-file main-file (str/join "\u0000" main-opts))
       (let [mf (jio/file main-file)]
         (when (.exists mf)
           (.delete mf))))))

dottedmag22:12:18

@pesterhazy I have to go right now, so I can't finish it at the moment.

alexmiller22:12:06

sorry, I don’t have time to work on this atm but will hopefully have a slice of deps time in a day or two

dottedmag22:12:34

@pesterhazy I don't mind if you pick it up from here, if you think this approach is good enough.

alexmiller22:12:52

if there are two (or N) options, that’s fine, just describe the tradeoffs in a table (and separate the file impl aspect from the “not breaking existing stuff” aspect - those I think are orthogonal)

pesterhazy22:12:01

the obvious downside of readarray is that it's not available in bash3, which is what ships with macOS

pesterhazy22:12:40

I think that rules it out

dottedmag22:12:00

Oh, too bad.

pesterhazy22:12:48

also not sure how reliable reading NUL character in bash is - that's the kind of thing where bash if flaky

pesterhazy22:12:41

according to this https://stackoverflow.com/a/6571549/239678: "In Bash, you can't store the NULL-character in a variable."

dottedmag22:12:17

That's why they have readarray in first place.

pesterhazy22:12:25

many languages with C strings have trouble with NUL characters in strings, for the obvious reasons

dottedmag22:12:27

And special case for -d '' there

dottedmag22:12:09

read -r -d '' should work with an explicit loop.

dottedmag22:12:53

As of now all NUL-characters are stripped from .main when read by Bash, so no backward compatibility problems here.

dottedmag22:12:19

Unless someone relied on NULs being stripped 😄

pesterhazy22:12:20

well try it - I didn't have luck reading the "\\ " character sequence using read

pesterhazy22:12:54

no it's not backward compatible - it would turn potentially multiple arguments into a single argument, i.e. change the semantics

pesterhazy22:12:22

I don't think there's a way around using a different file extension

dottedmag22:12:38

@pesterhazy

$ echo '00000000: 310032003300' | xxd -r | while read -r -d '' i; do printf ">$i<\n"; done
>1<
>2<
>3<

dottedmag22:12:04

@pesterhazy By backward compatibility I meant backward compatibility for code that relied on \u0000 passed from deps.edn unchanged to the app. But that didn't work, so we don't need to care about it.

pesterhazy22:12:53

@alexmiller I've updated the ticket with a summary

dominicm23:12:26

The ticket doesn't mention back porting readarray, https://stackoverflow.com/a/49322365 has some hints. Would this work?