Fork me on GitHub
#babashka
<
2022-02-14
>
Benjamin09:02:30

I'm thinking about a tool that queries stuff from a database and with the info gained queries some logs (aws cloudwatch). Let's say I make those processes bb tasks. I can depend the second task on the first. How would I have the output of the first task as input to the next task? Guess just writing to a file does it

borkdude09:02:03

@benjamin.schwerdtner If you write:

{:tasks {a 1, b {:depends [a] :task (+ 1 a)}}}
a is bound to the return value of task a in b

clojure-spin 2
Benjamin09:02:50

you already thought through it 😛

borkdude11:02:22

Just published a new version of babashka to master which has a massive performance improvement for loops (2-4x). Please try it out locally and report any regressions. I hope enough people will test drive it to find any problems before the next release!

$ bash <(curl ) --version 0.7.5-SNAPSHOT --dir .
$ ./bb  "(time (loop [val 0 cnt 10000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val)))"

✔️ 1
Michaël Salihi13:02:27

Just try on Linux and no regressions...it works great! 👍 1408 ms against 3435 ms 🔥

alexdavis11:02:59

fine for me, twice as slow as nbb which is surprising to me

alexdavis@Alexs-MBP ~ % ./bb  "(time (loop [val 0 cnt 10000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val)))"
"Elapsed time: 1235.124292 msecs"
10000000
alexdavis@Alexs-MBP ~ % nbb -e '(time (loop [val 0 cnt 10000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val)))'
"Elapsed time: 643.817084 msecs"
10000000

borkdude11:02:52

nbb runs on a JIT-ed environment which is probably the reason it works so well. Are you using nbb 0.1.6?

borkdude11:02:18

nbb 0.1.5 used to be 2x as slow as the old bb :)

alexdavis11:02:55

Yep just updated both, so is it just specific examples like this loop that nbb would do better at or do you think it will be faster in all cases? trying to choose between bb and nbb for a project at the moment, its probably not going to make much of a difference but still interested 😄

borkdude11:02:51

Hard to say. Parsing is way slower in nbb (subject to optimization improvements) so it really depends on the workload

borkdude11:02:46

also startup time of nbb is slower than bb

borkdude11:02:16

not that it really matters probably for most utils, but it's 180ms vs 20ms or so

alexdavis11:02:35

interesting, bb has always seemed like it had slower startup time in my tests

alexdavis@Alexs-MBP ~ % time ./bb  "(prn \"hello\")"
"hello"
./bb "(prn \"hello\")"  0.11s user 0.03s system 57% cpu 0.239 total
alexdavis@Alexs-MBP ~ % time nbb -e "(prn \"hello\")"
"hello"
nbb -e "(prn \"hello\")"  0.11s user 0.01s system 107% cpu 0.112 total
maybe my tests are flawed though

borkdude11:02:24

huh, that's weird! try to run it multiple times. the first time may be a little slower due to file system stuff.

$  time ./bb  "(prn \"hello\")"
"hello"
./bb "(prn \"hello\")"   0.01s  user 0.01s system 82% cpu 0.025 total

borkdude11:02:09

on my system, I get for nbb:

$ time nbb -e "(prn \"hello\")"
"hello"
nbb -e "(prn \"hello\")"   0.19s  user 0.02s system 108% cpu 0.192 total
Perhaps you are on m1 running bb with rosetta?

alexdavis11:02:20

Ah yes that would be the reason then 😄 forgot graal can’t do arm yet

alexdavis11:02:28

looks like it could be close though, you can build a version that works on m1 yourself now https://github.com/oracle/graal/issues/2666

borkdude11:02:56

I think trying the arm binary in docker on m1 in a linux container should also resemble what performance you could get. But it'd be cool if you could try the real m1 version too :)

borkdude11:02:29

It seems bb is now as fast as python3 in loops?

import time

start = time.time()

val = 0
cnt = 10000000

while (cnt != 0):
    cnt = cnt - 1
    val = val + 1

end = time.time()

print("Took:", end - start)
print(val)

Took: 1.1271159648895264
10000000

alexdavis11:02:38

node does it in 8ms, about the same time as jvm clojure

console.time("nbb")
var val = 0;
var ctn = 10000000;
while(ctn !== 0) {
	ctn = ctn - 1;
	val = val + 1;
}
console.timeEnd("nbb");

borkdude11:02:25

yeah, but this is because node is compiled and python is, like bb, interpreted.

alexdavis11:02:10

yeah I know, just surprising node is so fast, I always thought of it as slower than the jvm

borkdude11:02:23

yeah, it seems v8 has really been catching up on performance

alexdavis11:02:15

imagine if that much effort had been sunk into tools for a real language troll

👌 1
borkdude12:02:03

for comparison, ruby:

$ time ruby /tmp/test.rb
10000000ruby /tmp/test.rb   0.27s  user 0.02s system 98% cpu 0.288 total

borkdude12:02:26

With GraalVM ruby:

$ time $GRAALVM_HOME/bin/ruby /tmp/test.rb
10000000$GRAALVM_HOME/bin/ruby /tmp/test.rb   0.15s  user 0.06s system 145% cpu 0.147 total

borkdude12:02:10

Graal Python:

$ time $GRAALVM_HOME/bin/graalpython /tmp/test.py
Took: 0.21900010108947754
10000000
$GRAALVM_HOME/bin/graalpython /tmp/test.py   1.16s  user 0.19s system 278% cpu 0.483 total

alexdavis12:02:54

bash 😆 (I mean it is what babashka is meant to replace troll )

time ./loop.sh
./loop.sh  42.95s user 2.40s system 99% cpu 45.354 total

borkdude12:02:05

Btw in the browser (chrome) I get about the same time as with bb: https://babashka.org/xterm-sci/

borkdude12:02:43

Safari seems to be faster!

user=> (time (loop [val 0 cnt 10000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val)))
"Elapsed time: 735.000000 msecs"
10000000

borkdude13:02:03

@souenzzo you might also want to ask in #nbb

🆗 1
imre14:02:13

Is there any bb (debug/verbose) switch that makes (shell... & (clojure... calls made from tasks print the command they are executing?

imre14:02:26

sort of like set -x I believe

borkdude14:02:44

@U08BJGV6E babashka does have a --debug setting but the output is purely for debugging, not for daily usage. I think we could have a babashka.tasks global switch for this, like :command-print-fn or so in which you can do your own thing and also provide a default

borkdude14:02:44

for now you could also make your own shell and clojure wrapper

borkdude14:02:52

which just prints the args first and then calls the original

imre14:02:47

Makes sense, thank you. It would be a welcome feature for sure. Again, mostly for debugging:

; bb --debug run test

 ;; deps

(ns user-4b3db898-5cee-4604-8d36-839d2be6f97d )
(require '[babashka.tasks])
(when-not (resolve 'clojure)
  ;; we don't use refer so users can override this
  (intern *ns* 'clojure babashka.tasks/clojure))

(when-not (resolve 'shell)
  (intern *ns* 'shell babashka.tasks/shell))

(when-not (resolve 'current-task)
  (intern *ns* 'current-task babashka.tasks/current-task))

(when-not (resolve 'run)
  (intern *ns* 'run babashka.tasks/run))

nil
(def test (binding [
  babashka.tasks/*task* '{:name test, :doc "Run project tests via Kaocha. Args will be forwarded to Kaocha.", :task (apply shell "bin/kaocha" *command-line-args*)}]
  nil
(apply shell "bin/kaocha" *command-line-args*))) test


Loading namespaces:  (e

imre14:02:56

This prints (apply shell "bin/kaocha" *command-line-args*) which is what I'm invoking but what I'm interested in is whether I managed to invoke bin/kaocha as I had intended

borkdude14:02:18

yeah make sense. What about doing this in :init:

:init (do (defn shell [& args] (println (str/join " " args)) (apply babashks.tasks/shell args))

imre14:02:06

Yep, that should work for now although it still only tells me what args are being passed to shell and not whether that results in the process invocation I intended to do. I.e. it won't tell me if I'm misusing the shell api

borkdude14:02:07

I'm not sure if I get that. What else other than the above would you expect a built-in function to do?

imre14:02:09

I looked at examples https://book.babashka.org/#parallel and https://book.babashka.org/#tasks:clojure and I see that multiple args to the cli are sometimes given in one string and other times one arg per string. And I wasn't sure how quoting etc would work

borkdude14:02:35

right. the first argument to those functions are parsed with process/tokenize

borkdude14:02:08

which works similar to how shell strings work: foo bar becomes ["foo" "bar"] but 'foo bar' becomes ["foo bar"]`, etc

borkdude14:02:00

so (into (process/tokenize (first args)) (rest args)) should be the accurate representation

borkdude14:02:39

but for printing you need not worry about this probably

borkdude14:02:32

but for usability I think it'd be nice to have an optional function which gets to see the arguments before invocation

borkdude14:02:37

I"ll make an issue

imre14:02:33

Thank you. Yeah, really not a strong point from my end, just that it's probably more accurate if done closer to actual invocation

borkdude14:02:07

oh I think there's already a hook that you could abuse :)

borkdude14:02:03

$ bb -e '(babashka.process/process ["ls" "-la"] {:escape (fn [arg] (prn arg) arg)})'
"ls"
"-la"

borkdude14:02:32

This :escape hook is really there for Windows which has some odd corner cases with process arguments and defaults to some sane behavior. It's applied to every argument

borkdude14:02:52

but I guess we want to have something similar for all arguments

Benjamin16:02:12

what is the best way to fail all tasks in the chain of dependent tasks, but to keep going with the ones that do not depend?

Benjamin16:02:36

My tasks would all return an :anomaly object if they fail I guess I wrap everthing with a check

borkdude16:02:36

You can control failure with :continue true which will never throw but you will have to handle the result yourself

Benjamin16:02:29

can I do (pods/load-pod 'org.babashka/aws "0.1.2") before the requires somehow?

:tasks... 
:requires
  ([babashka.pods :as pods]
   [cos.iap.info.impl.dynamo :as dynamo])
  :init
  do
    (pods/load-pod 'org.babashka/aws "0.1.2")
    (require '[pod.babashka.aws :as aws])
doesn't work because I require aws in the dynamo namespace before I load

borkdude16:02:41

The best way to do this currently is probably to move your code to a file and then do load-pod + require in that file.

borkdude16:02:17

and then require that file in the tasks

kiranshila18:02:59

Hey everyone! Does babashka.fs have access to the Watch Service API?

borkdude18:02:47

Are you asking for usage within bb or with fs on the JVM?

kiranshila18:02:11

within bb, I'm trying to see how to watch for file changes

kiranshila18:02:24

Preferably with something like inotify

borkdude18:02:01

We have two file watcher for bb, one written in Rust and one written in Go: https://github.com/babashka/pod-registry Both should work fine.

kiranshila18:02:18

Oh excellent! Thank you

borkdude18:02:07

I have an example here where I watch sources from the babashka book and rebuild on change, and then refresh a browser to show the new renderings https://github.com/babashka/book/blob/master/script/watch.clj

kiranshila21:02:18

Can I have a pod dependency in bb.edn?

borkdude21:02:15

Not yet, this is on the TODO list

👍 1
escherize23:02:59

I’ve made a https://asciinema.org/a/ZnJ0kaaMr2FuRyDgpTqSbgLwA. Now, I am investigating the mechanisms around making a pod. My pod would run some js against a js library by shelling out to node, and thus has a dependency on a js library. Would it make sense to install the dep in the pod’s node_modules (via npm install … ), and just assume node is installed? To me it seems to be asking a lot of the pod user. Another option would be skipping pods entirely — I guess I can just distribute this feature as a directory (js file, and clj file), and let users install node and run npm install. But I feel like there must be a better way than that!

🤓 2
lispyclouds07:02:43

Maybe you could try writing it in #nbb too? that can load arbitrary js dependencies 😄

borkdude09:02:08

That's what I was going to say too :)