Fork me on GitHub
#clojure
<
2021-03-25
>
Michael Lan00:03:27

how do I use third-party dependencies in a single-file clojure execution

seancorfield00:03:33

@michaellan202 How are you running the Clojure file?

seancorfield00:03:16

(there’s a trick where you can turn a Clojure file into a shell script that can run on Linux/macOS and it can contain its dependencies and code to reinvoke itself via the Clojure CLI)

seancorfield00:03:26

Here’s an example I posted the other day in #tools-deps showing how a Clojure “shell script” could have functions invoked via the CLI -X option:

(! 962)-> ./example.clj -X example/test :foo '"bar"'
WARNING: test already refers to: #'clojure.core/test in namespace: example, being replaced by: #'example/test
[+] test successful {:foo bar}
(! 963)-> cat example.clj 
#!/bin/sh
#_(
   #_DEPS is same format as deps.edn. Multiline is okay.
   DEPS='
   {:deps {} :paths ["."]}
   '
   #_You can put other options here
   OPTS='
   -J-Xms256m -J-Xmx256m -J-client
   '
exec clojure $OPTS -Sdeps "$DEPS" "$@"
) ;; code goes below here
(ns example)
(defn test [arg-map]
  (println "[+] test successful" arg-map))

seancorfield00:03:28

And here’s another possibility:

(! 991)-> bin/time.sh 
Time is now 2021-03-25T00:38:56.623Z
Java version is 15
(! 992)-> cat bin/time.sh 
#!/usr/bin/env clojure -Sdeps {:deps,{clj-time/clj-time,{:mvn/version,"0.14.2"}}} -M
 
(require '[clj-time.core :as t])
(println (str "Time is now " (t/now)))
(println (str "Java version is " (System/getProperty "java.version")))

Michael Lan01:03:21

Wow thats interesting. the second looks a lot better. The command line options confuse me, like I don’t get how to execute the main method, or what -M or -X do exactly, and the docs I have found aren’t comprehensive

seancorfield01:03:55

Happy to answer any specific questions you have about the Clojure CLI and its options.

seancorfield01:03:08

TL;DR: -M means “execute clojure.main” and -X means “execute this specific function”

seancorfield01:03:05

clojure.main can invoke a Clojure script directly — the time.sh case is almost the same as this command-line:

$ clojure -Sdeps '{:deps {clj-time/clj-time {:mvn/version "0.14.2"}}}' -M time.clj
if the file was time.clj and did not include the #! line.

seancorfield01:03:09

The first example — with multiline deps and opts — is much more flexible but a little bit more setup.

Michael Lan01:03:20

Ah, thank you! Yea I found that link but it seemed to have glanced over the details. Thanks so much for your detailed reply, I will try this!

Michael Lan15:03:51

I had a lot of trouble and still can’t figure out how to get https://github.com/davidsantiago/hickory working with -Sdeps.

clojure -Sdeps '{:deps github-davidsantiago/hickory {:git/url "" :sha "ea248a6387f007dc4c4e8fcbbafbb1b9cbc19c78"}}' -M
gives the error:
Error while parsing option "--config-data {:deps github-davidsantiago/hickory {:git/url \"\" :sha \"ea248a6387f007dc4c4e8fcbbafbb1b9cbc19c78\"}}": java.lang.RuntimeException: Map literal must contain an even number of forms
I’d appreciate if you could give some pointers as to how to use this.

Michael Lan15:03:45

I’ve also changed the spaces to commas as you describe in this video https://www.youtube.com/watch?v=CWjUccpFvrg but I do get this error:

Error building classpath. Don't know how to create ISeq from: clojure.lang.Symbol
java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol

seancorfield16:03:21

@michaellan202 -Sdeps takes a hash map but yours is not valid: you have three items in it. It should look like this:

clojure -Sdeps '{:deps {github-davidsantiago/hickory {:git/url "" :sha "ea248a6387f007dc4c4e8fcbbafbb1b9cbc19c78"}}}' -M
However, that library is not set up to be used via a git dep as it is a Leiningen project (it has no deps.edn file). You can use it via it’s Clojars coordinates:
clojure -Sdeps '{:deps {hickory/hickory {:mvn/version "0.7.1"}}}' -M
The Hickory docs are outdated, it seems, and the coordinates for the dependency are buried way down in the README at https://github.com/davidsantiago/hickory#obtaining and it looks like the project is abandoned anyway.

Michael Lan16:03:42

Thank you for the detailed response!

seancorfield16:03:34

Also, if you are using :git/url to depend on a library that has Clojars (or Maven) releases, it is better to use the actual lib name of the released version so that the dependency machinery will know that you’re using a different version of, say, hickory/hickory, rather than a completely different library.

seancorfield16:03:42

That way, if you are working with code that also depends on that library, your explicit :git/url version choice will override the transitive dependency, rather than try to load both versions of the library and then having a conflict because you’ll have the same set of namespaces in two places.

Michael Lan16:03:01

Ah got it. That makes sense

seancorfield17:03:08

You can use non-`deps.edn` projects via :git/url by the way but you need to tell the CLI that you’re overriding the project type by adding :deps/manifest :deps into the coordinate map and then you also need to specify the project’s dependencies (from project.clj) yourself — which would be several additional things for Hickory, which is why I didn’t show that. If project.clj only has Clojure as a dep, you can use it via :git/url just by specifying the manifest (which is true for quite a few simple libraries out there). Most Leiningen-based projects tend to deploy to Clojars so it’s rare that you would need to depend on them via :git/url.

seancorfield17:03:46

Also worth noting that if a project contains any Java code, even if it is a deps.edn project, you can’t use it via :git/url because there’s no way to compile the Java code.

Michael Lan17:03:30

I’m struggling with figuring out how to fix this error:

Error building classpath. Don't know how to create ISeq from: clojure.lang.Symbol
java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol
my shebang line is as so:
#!/usr/bin/env clojure -Sdeps '{:deps {cheshire/cheshire {:mvn/version "5.10.0"}}}'

Michael Lan17:03:57

oh, the quote is being interpreted as a symbol

devn20:03:33

I have kind of a gross version because there are multiple calls to clojure, but: given a file foo.clj:

#!/usr/bin/cljit
'{:deps {cheshire {:mvn/version "5.8.0"}}}
(ns foo (:require [cheshire.core :as json]))
(println (json/parse-string "{\"a\": 1}"))
chmod +x foo.clj Drop this in /usr/bin as cljit
#!/bin/sh

set -e

if [ $# -eq 0 ]; then
    echo "usage: doclj <file> [args ...]"
    exit 1;
fi

FILE=$1

shift

/bin/clj -Sdeps "$(clj --eval '(fnext (read-string (slurp "'"$FILE"'")))')" -i $FILE -- "$@"
Then, ./foo.clj

Michael Lan22:03:01

ah that’s very interesting, i like your approach

nilern12:03:03

Wow. The first one is quite the trick but isn't the second one just posix #! (and an unregistered reader tag I guess).

restenb15:03:53

i have this long-running process that starts two go-loops to manage state while it runs. it's supposed to be started on demand by any client. is there a point to somehow "closing" the go-loops when each process completes?

restenb15:03:11

i honestly don't even know how one would "exit" a go-loop.

nilern15:03:20

Just stop looping. You can also request it to stop via a channel.

restenb15:03:12

slaps forehead so just (if finished? nil (recur))

ghadi15:03:36

lots of times people pass an input channel and a cancel channel to a go loop

borkdude15:03:42

(when-not finished? (recur))

ghadi15:03:58

if I signal is received on the cancel channel -> perform cleanup and exit

ghadi15:03:31

sometimes you arrange these processes in trees, so that a process can signal to subprocesses the need to clean up and exit

nilern15:03:40

But if the OS process is exiting there is not much that actually needs cleaning up. Threads, memory, even file handles will be wiped out by the OS anyway.

ghadi15:03:00

It depends on the use-case, can't make a generalization

ghadi15:03:36

e.g. you grabbed a lock, need to relinquish

nilern15:03:05

Some C++ programs take a looong time to exit because they are running a bajillion pointless destructors

mpenet15:03:23

it's not only destructors, sometimes it just matters to finish what you are doing cleanly

mpenet15:03:36

like draining some job queue or whatever

3
dpsutton15:03:04

(when-not finished? (recur)) this pattern requires a message after the "quit" signil though right? It's probably better to alt or some other mechanism that looks for a quit message in addition to a normal message

mpenet15:03:06

exit-ch is a good candidate for a promise-chan + alts!

mpenet15:03:01

inside your loop just alts! over your input-ch + exit-ch and do the right thing from there

nilern15:03:13

(when-not (<! quit-chan) ... (recur))

ghadi15:03:33

not that ^ because if your quit-chan doesn't signal anything you'll block

facepalm 3
mpenet15:03:35

that would park over your quit-ch

borkdude15:03:39

(when-not was just a syntactic rewrite of the (if finished? nil ..) expression, not an answer as such to handling core.async loops)

mpenet15:03:49

hence the alts!

ghadi15:03:25

closing the input channel is another implicit signal to quit

Alex Miller (Clojure team)15:03:37

when-some is probably better for most reading from channel cases btw

restenb15:03:53

(poll! quit-chan)?

mpenet15:03:59

or closed output-ch 🙂, there many ways to skin a cat

🙀 3
restenb15:03:00

assuming it will signal at some point

mpenet15:03:12

you can check the ret val of >! or <!

ghadi15:03:33

@restenb (alts! [input cancel])

mpenet15:03:08

(when-let [task (<! input-ch)] ... (recur))

6
mpenet15:03:11

"exit-ch" can be useful when you have to propagate that exit to other places, otherwise it's true than just checking ret vals of put/take can be enough

noisesmith16:03:22

you can also propagate the close, by closing all the chans you have for writing - then you get a nice modular protocol for shutdowns, and it's flexible enough that you can shut down a sub-tree of your forest of processes

mpenet16:03:13

yeah I meant more if you have various "components" that are not cooperating via chans otherwise

nilern15:03:14

when-some as alex said

9
🎉 3
🙌 3
raspasov21:03:14

I wrote this small lib to deal exactly with stopping/restarting go-loops. Uses a combination of atoms, poll! , and a loop-in-a-loop; Been using it happily for UI stuff in ClojureScript/React. Should work on JVM Clojure as well. https://github.com/saberstack/loop Note: A ss.loop/go-loop always exits on the very next (recur ...) call. It does not "die" automagically in the middle of execution.

raspasov21:03:44

It is very nice for the REPL specifically, when you have some go-loops that you’re writing and once in a while you’ll get a “runaway” go-loop and you need to restart your process in order to stop it.