This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-07-31
Channels
- # announcements (2)
- # babashka (145)
- # beginners (260)
- # calva (17)
- # chlorine-clover (7)
- # clj-kondo (9)
- # cljsrn (1)
- # clojure (88)
- # clojure-dev (65)
- # clojure-europe (31)
- # clojure-france (4)
- # clojure-nl (4)
- # clojure-uk (61)
- # clojuredesign-podcast (1)
- # clojurescript (31)
- # code-reviews (1)
- # cursive (32)
- # data-science (2)
- # datascript (9)
- # datomic (39)
- # docker (3)
- # events (1)
- # figwheel (95)
- # figwheel-main (4)
- # fulcro (17)
- # kaocha (2)
- # keechma (1)
- # malli (1)
- # meander (35)
- # nrepl (4)
- # off-topic (1)
- # pathom (8)
- # re-frame (4)
- # reagent (8)
- # reitit (3)
- # releases (1)
- # remote-jobs (2)
- # shadow-cljs (182)
- # sql (30)
- # tools-deps (89)
- # xtdb (31)
@cfleming babashka tries to be as compatible as possible with normal clojure, so you can run scripts with both the JVM or babashka. So by design there's no easy way to distinguish just by looking at the code, like @isak has also said correctly
Why is that necessary for Cursive and not say for CIDER, Chlorine, Calva and all the other IDEs that support babashka?
Because Cursive uses static analysis, so it has to understand the dependencies that are available. These are normally provided by the containing project, but in babashka’s case they’re implicitly available, right?
So I’ll need some way of knowing which deps babashka has available, and somehow providing them for these scripts.
I assume the ones that are available are from babashka’s deps.edn, correct? But that’s not actually published anywhere like Clojars?
The project.clj is probably the best source of truth. The available namespaces are listed here: https://github.com/borkdude/babashka#usage You can also see them with:
bb -e '(map ns-name (all-ns))'
Many babashka.* namespaces are also available on clojars, like babashka.curl and babashka.pods
I see the documentation is slightly out of date and e.g. babashka.curl is not mentioned in the list. I'll do something about that.
Yeah, I’d need the actual dependencies rather than the namespaces. Although I might be able to do something similar to Cursive’s existing stubs functionality, which introspects namespaces and creates function stubs which Cursive uses.
bb supports all-ns
, ns-interns
etc, so it's pretty easy to write out some edn file with stubs
Is it possible to use java.nio.charset? I have:
(ns random-string
(:require
[clojure.edn :as edn]
[clojure.string :as str])
(:import
(java.nio.charset
Charset)))
Which produces java.lang.Exception: Unable to resolve classname: java.nio.charset.Charset
. I don't see this included in https://github.com/borkdude/babashka/blob/master/src/babashka/impl/classes.clj#L143. I'm using the latest bb as provided by brewTo create strings of random characters. What I need is a list of all alphanumeric characters. What I have is:
(defn alphanum
[charset-name]
(let [ce (-> charset-name Charset/forName .newEncoder)]
(->> (range 0 (int Character/MAX_VALUE)) (map char)
(filter #(and (.canEncode ce %) (Character/isAlphanumeric %))))))
I'm definitely open to other solutionsI know that this library works with bb: https://github.com/clojure/data.generators You could maybe take a peek at how they do it first
btw,
user=> #(Character/isAlphanumeric %)
Syntax error (IllegalArgumentException) compiling . at (REPL:1:2).
No matching method isAlphanumeric found taking 1 args for class java.lang.Character
that might be a typo?(def ^:dynamic ^java.util.Random
*rnd*
"Random instance for use in generators. By consistently using this
instance you can get a repeatable basis for tests."
(java.util.Random. 42))
(defn uniform
"Uniform distribution from lo (inclusive) to hi (exclusive).
Defaults to range of Java long."
(^long [] (.nextLong *rnd*))
(^long[lo hi] {:pre [(< lo hi)]}
(long (Math/floor (+ lo (* (.nextDouble *rnd*) (- hi lo)))))))
(defn printable-ascii-char
"Returns a char based on *rnd* in the printable ascii range."
[]
(char (uniform 32 127)))
(print (printable-ascii-char))
(print (printable-ascii-char))
(print (printable-ascii-char))
This does what I need. Thanks!Someone just published a Babashka Github action: https://github.com/marketplace/actions/babashka-clojure
armed with the knowledge about ProcessBuilder
- which I just gained from the babashka readme! thx, @borkdude! 🙂 -
i thought i can just play with a datomic peer-server, directly from a REPL, like this:
(def peer-server-opts
["-h" "localhost"
"-p" "8998"
"-a" "myaccesskey,mysecret"
"-d" "hello,datomic:"])
(def peer-server-process
^java.lang.Process
(-> (into ["CURRENT/bin/run" "-m" "datomic.peer-server"] peer-server-opts)
(doto println)
(ProcessBuilder.)
(.inheritIO)
(.start)))
to my surprise, (.destroy peer-server-process)
only kills the run
shell script, but not the java process started by it!
so i went on a quest to see how can i send the equivalent of Ctrl-C
to the peer-server-process
.
typically very helpful sites couldn't answer this:
https://www.baeldung.com/java-lang-processbuilder-api
https://www.baeldung.com/java-9-process-api
searched more and i found... nothing, but endless questions for decades, with OS specific answers, to the effect of "just shell out to run kill -SIGINT <pid>
"
like this and a bunch more similar conversations: https://coderanch.com/t/676288/java/Wait-kill-interrupt-process-complete
there was one interesting variant, which was about trying to send byte 0x03 to the stdin of the process, to simulate Ctrl-C, which is the 3rd letter in the ABC, right?
https://garethrees.org/2015/08/07/ping/
but that assumes that the process is somehow listening to stdin and indeed interprets the byte 3 as ctrl-c and in turn sends a SIGINT to the process, which then also must have a handler assigned to it and handle it as process exit...
(i haven't tried this yet, but because it feels a little hacky and i hope there is some more generic/official/idiomatic solution)
so am i missing something? am i asking the wrong question?
how can i stop that peer-server scripts the same way as pressing ctrl-c would in a terminal?this btw works, but relies on this very specific situation:
(.destroy (first (iterator-seq (.iterator (.children ^java.lang.Process peer-server-process)))))
I start java processes all the time. E.g. this is in our dev script for work:
(defn less []
(let [cmd ["clojure" "-A:frontend:less/dev"]
pb (doto (ProcessBuilder. cmd)
(.redirectOutput ProcessBuilder$Redirect/INHERIT)
(.redirectError ProcessBuilder$Redirect/INHERIT))
proc (.start pb)]
(-> (Runtime/getRuntime)
(.addShutdownHook (Thread. #(.destroy proc))))
proc))
and it kills the java process reliably.The entire script is here: https://gist.github.com/borkdude/8f5dff7c2330ca520403eb44c9013a83
i want to initiate the shutdown manually from the repl session, not on the exit hook of the repl session
because i want to start the peer server with different -d
parameters, exposing different peers
What does the peer script look like btw? Does it fork the process into the background or something?
it's just the stock datomic run script:
#!/bin/bash
cd `dirname $0`/..
while [ $# -gt 0 ]
do
case "$1" in
-X*)
java_opts="${java_opts} $1"
;;
-D*)
java_opts="${java_opts} $1"
;;
*)
clojure_args="${clojure_args} $1"
;;
esac
shift
done
/usr/bin/env java -server -Xmx1g -Xms1g $DATOMIC_JAVA_OPTS ${java_opts} -cp `bin/classpath` clojure.main -i "bin/bridge.clj" ${clojure_args}
pretty generic wrapper scriptif that last line would just be an exec /usr/bin/env
, that might make it work, but regardless i want to know how to "simulate a ctrl-c" because that's a pretty common use-case
if im lucky, sending a 0x3 to its stdin might work, but i can't believe that java hasn't exposed the signal sending api of libc 😕
https://stackoverflow.com/questions/7835212/how-to-send-sigint-signal-from-java-to-an-external-process The answer is: there is no platform-agnostic solution.
and see what does the comment to that answer say: > That seems to send a SIGTERM signal, not SIGINT. See this question here: http://stackoverflow.com/questions/2950338/… – Lukas Eder Oct 20 '11 at 11:58
public abstract void destroy()
Kills the subprocess. Whether the subprocess represented by this Process object is forcibly terminated or not is implementation dependent.
im using nix to make sure that my base utilities are gnu not bsd, so i can rely on that to have the same kill
command, but only if i make sure im running kill
via a nix-shell
...
im promoting babashka in the company to adopt it for replacing makefiles and bash scripts
Ah I see. They added children in java 11! https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Process.html This may work indeed. I'd never seen that before.
in case of a shell script, i have no choice, i must rely on kill, which is btw often a shell builtin, eg:
➜ which kill
kill: shell built-in command
i tried destroyForcibly
too, that's why i have the type annotation in my examples; to allow autocompletion in cursive 🙂
but let me try it again
babashka doesn't promise anything about freeing people from complexity, it just brings Clojure with fast startup time, comparable to bash, with a couple of built-in libraries.
➜ pstree | grep datomic.peer
| | \-+= 17326 onetom /bin/bash CURRENT/bin/run -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic:
| | \--- 17330 onetom java -server -Xmx1g -Xms1g -cp resources:datomic-transactor-pro-1.0.6165.jar:lib/*:samples/clj:bin: clojure.main -i bin/bridge.clj -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic:
➜ kill -TERM 17326
➜ pstree | grep datomic.peer
\--- 17330 onetom java -server -Xmx1g -Xms1g -cp resources:datomic-transactor-pro-1.0.6165.jar:lib/*:samples/clj:bin: clojure.main -i bin/bridge.clj -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d hello,datomic:
just to confirm that SIGTERM leads to this detached java server processif i add an exec
in front of the /usr/bin/env
then the shell process is replaced by the java process and i can just use (.destroy peer-server-process)
BUT that assumes i have some code everywhere where im deploying the datomic distribution, which patches this script, so we just shifted the complexity then 🙂
im trying this from clojure first because i had some datomic.client.api code prepared there which i can use to test the started peer-server
i have no opinion about iterator-seq
or enumeration-seq
or the likes, because this is the first time i had to use it actually...
i didn't have to do much java interop in the past 5 years im using clojure
what's your decision process for deciding what goes into babashka from clojure.core
?
or is that documented in the development section of the docs?
for the record, i've just checked again, and .destroyForcibly
also just kills the bash process, not the java one started from it.
Made an issue to make this work in babashka: https://github.com/borkdude/babashka/issues/519
Do you know a good way to create a process with children?
./bb '(run! #(prn (.pid %)) (iterator-seq (.iterator (.children (.start (let [pb (ProcessBuilder. ["bash" "-c" "/usr/bin/env clojure"])] (Thread/sleep 3000) pb))))))'
This returns an empty stream.im experimenting with this now:
(def process-to-stop
(-> (ProcessBuilder.
["bash" "-c"
"echo sleep 20; (sleep 20 && echo done 20) &
echo sleep 10; sleep 10; echo done 10"])
(.inheritIO)
(.start)))
a more complete example:
(defn pstree [grep-for]
(->> (:out (clojure.java.shell/sh "bash" "-c" "pstree"))
(str/split-lines)
(filter #(str/includes? % grep-for))
(map #(subs % 0 (min 60 (count %))))
(run! println)))
(defn spawn []
(-> (ProcessBuilder.
["bash" "-c"
"echo sleep 20; (sleep 20 && echo done 20) &
echo sleep 10; sleep 10; echo done 10"])
(.inheritIO)
(.start)))
(defn stop-children [proc]
(run! #(.destroy %) (iterator-seq (.iterator (.children proc)))))
(pstree "sleep")
(def proc (spawn))
(pstree "sleep")
(stop-children proc)
(pstree "sleep")
With this script:
(let [depth (System/getenv "DEPTH")]
(prn depth)
(when-not (= "4" depth)
(let [cmd ["bb" "child.clj"]
pb (doto (ProcessBuilder. cmd)
(.inheritIO))
_ (.put (.environment pb) "DEPTH"
(str (inc (Integer. (or depth
0)))))
proc (.start pb)]
(if-not depth ;; the top process
(do
(Thread/sleep 500)
(prn (mapv #(.pid %) (iterator-seq (.iterator (.descendants proc))))))
(Thread/sleep 10000)))))
I get to see:
nil
"1"
"2"
"3"
"4"
[58990 58991]
with some modification in a branch process-children
so this shell script is expected to keep the backgrounded sleep 20
process running after Ctrl-C:
#!/usr/bin/env bash
echo sleep 20; (sleep 20 && echo done 20) &
echo sleep 10; sleep 10; echo done 10
so it's not a good example:
➜ chmod +x sigint-test.sh
➜ ./sigint-test.sh
sleep 20
sleep 10
^C
➜ pstree | rg sleep
| | \--- 22861 onetom rg sleep
\--- 22768 onetom sleep 20
➜ done 20
|-+- 59113 borkdude bb child.clj
| \-+- 59114 borkdude bb child.clj
| \--- 59115 borkdude bb child.clj
\-+- 59152 borkdude bb child.clj
\-+- 59153 borkdude bb child.clj
\--- 59154 borkdude bb child.clj
but prints:
[59153 59154]
the elements returned by the iterator-seq
are java.lang.ProcessHandleImpl
objects, so you would need to call .descendats
on those too.
> ok, brew installing it
you should just nix-env -i pstree
it ;)
or even call it as nix-shell -p pstree --run pstree
and it would download it on first use on any machine (which has nix installed)
i don't have clojure on my path by default, so if im in a directory outside of my clojure projects, i always run clojure as `nix-shell -p clojure --run 'clj ...'
this makes me very conscious about which version of clojure im using!
the answer to that is "who knows..."
which is exactly the same in case of a brew install clojure
.
which version is it?
"whatever is on the internet at that moment"
the projects im working on professionally, always have a pinned package tree - like i showed the other day - so it's guaranteed im getting the same clojure version on every machine
i was moving the current company over to nix (and clojure and now babashka 😉 in the past few month, so it became a reflex to take note of this
if it weren't for nix, i wouldn't be able to keep up with the latest clojure cli versions so easily and immediately make it available to all my colleagues.
all they have to do is git pull
then press an extra enter and the new clojure version is gets downloaded.
no extra communication needed!
literally this is the git commit for an upgrade:
index a2c3ce138..2b5ea1f12 100644
--- a/deps/clojure/default.nix
+++ b/deps/clojure/default.nix
@@ -2,11 +2,11 @@
stdenv.mkDerivation rec {
pname = "clojure";
- version = "1.10.1.561";
+ version = "1.10.1.590";
src = fetchurl {
url = "";
- sha256 = "0baf23npillrf7sy5piwb4brgv3cb1ij832kbq79rklgkz897ha6";
+ sha256 = "18x8xkxsqwnv3k1mf42ylfv7zzjllm7yiagq16b2gkq62j5sm1k7";
};
nativeBuildInputs = [
and that sha256
is the part which answers the "which version is it" part with extreme precision.which is just a copy of https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/clojure/default.nix in our repo
This seems to work fine now, it prints 4 pids:
(defn pids [^java.lang.ProcessHandle x]
(distinct (cons (.pid x) (mapcat pids (iterator-seq (.iterator (.descendants x)))))))
The entire script:
(defn pids [^java.lang.ProcessHandle x]
(distinct (cons (.pid x) (mapcat pids (iterator-seq (.iterator (.descendants x)))))))
(let [depth (System/getenv "DEPTH")]
(when-not (= "4" depth)
(let [cmd ["bb" "child.clj"]
pb (doto (ProcessBuilder. cmd)
(.inheritIO))
_ (.put (.environment pb) "DEPTH"
(str (inc (Integer. (or depth
0)))))
proc (.start pb)]
(if-not depth ;; the top process
(do
(Thread/sleep 100)
(prn (pids (.toHandle proc))))
(Thread/sleep 100000000)))))
Added a unit test now in the branch:
https://github.com/borkdude/babashka/commit/a7b3224a28e99d985c4944db8f18457a01ad57aa
This will be available in #babashka_circleci_builds in a few minutes. java.util.List
will also be in there.
@U086D6TBN There's no urgency to make a release from my side, but I can do a release whenever you need it.
now that i know that u have these ci builds available; that already helps. but if (or rather when) we adopt bb at work, we will make sure that we can build it ourselves too, preferably with nix and then i wont bother you this much 🙂
i was just looking those new tests.
you mixed a lot of concerns into one test.
i also can't see clearly the intent behind the names like
, handles
and run
.
i might comment on your commits to share my concerns, but i want you to know that im not expecting answers necessarily, neither do i expect to reach any agreement. i just just hope if u have time, u might consider them, but even if not, other bb developers later have a chance to understand the source better (if github is still around by then :)
http://child.bb -> killing child processes handles -> process handles (ProcessHandle is a class) run -> I needed to make this a function to trigger the reflection problem you had
> if github is still around by then in what type of dystopian future do you believe? 😉
a lot of tests are just a bunch of code proving by example that the interop works as expected in the binary image
well, i live in hong kong, so i don't have to go far to even experience elements of dystopia... 😉
and im not saying i don't understand what do those names refer to, but i only understand it because i was involved directly what that code was made.
some one else later, will not have this context.
for their sake, you should have better names, for example the way u explained those names can be the names themselves.
like killing_child_
or more like `spawning_child_http://processes.bb
handles
can be process-handles-of-all-decendants-recursively
it's stupidly long, but since you are only calling it once, why not? it's very explanatory.
alternatively u can put it into a function docstring...
as a heads up, you need to clone babashka with --recursive
, here are the instructions:
https://github.com/borkdude/babashka/blob/master/doc/dev.md#clone-repository
https://github.com/borkdude/babashka/commit/5a02faeb1f8da61b978a47a0fa74c9dc79fd167c
I think this problem was mentioned before here. Might be worth searching the Zulip archives for this channel: https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/babashka/search/kill Not sure if there was a proper solution
tl;dr for folks wanting to kill all subprocesses:
(defn handles [^java.lang.ProcessHandle x]
(distinct (cons x (mapcat handles (iterator-seq (.iterator (.descendants x)))))))
(run! #(.destroy %) (handles (.toHandle proc)))
It already works with the new binaries in the #babashka_circleci_builds channel, so testing is already possible
it didn't seem to work for me: https://clojurians.slack.com/archives/CSDUA8S6B/p1596221091003900
on a 2nd thought, i googled for java bash and signals and this came up: http://veithen.io/2014/11/16/sigterm-propagation.html which kinda suggests that leaving out bash from the equation is the best course of action probably :)
Interesting read. So what's your next course of action now, run the java process directly with bb and skip the bash script altogether?
something like that.
currently im experimenting with various ways to connect to datomic on-prem, so i need both the peer and the client api in the same process, which means i have to use clojure to get access to those.
but i also need things like wait/wait-for-port
, which i just quickly copied over from babashka's source.
this is what i have so far:
(defn datomic-classpath []
(str/trim (:out (shell/sh "CURRENT/bin/classpath" :dir (io/file "CURRENT")))))
(defn ^List peer-server-cmd [opts]
(into ["/usr/bin/env" "java" "-server" "-Xmx1g" "-Xms1g"
"-cp" (datomic-classpath)
"clojure.main" "-i" "bin/bridge.clj"
"-m" "datomic.peer-server"]
opts))
(defn ^Process make-process [cmd]
(println cmd)
(doto (ProcessBuilder. ^List cmd)
(.directory (io/file "CURRENT"))
;(.redirectOutput ProcessBuilder$Redirect/INHERIT)
;(.redirectError ProcessBuilder$Redirect/INHERIT)
(.inheritIO)))
we (or rather i?) decided literally only 2 days ago that the time has come to try using babashka.
to help my colleagues to get going, i want to make some examples.
my current work need clojure but i deliberately target parts of it for babashka.
that's why i was asking about aws integration the other day.
for example, we are using ssm to get rds credentials and for that i turned an aws
cli invocation into clojure a few months ago and used that as a reference implementation for the clojure implementation, which was using the cognitect.aws-api
:
(defn cli-ssm-get-param-via-profile [profile param]
(-> (sh "direnv" "exec" "."
"aws" "ssm" "get-parameter"
"--name" param
"--query" "Parameter.Value"
"--with-decryption"
"--output" "text"
"--region" "ap-southeast-1"
"--profile" profile)
:out
str/trim-newline))