Fork me on GitHub
#babashka
<
2020-09-29
>
borkdude07:09:38

@jeroenvandijk Found a way to automatically update the path after install:

function glam-install {
    export PATH="$(clojure -M:glam --install $@):$PATH"
}

borkdude07:09:58

(and renamed the repo, since it conflicted with the C++ package manager: https://github.com/borkdude/glam)

jeroenvandijk07:09:51

Glam is also a nice find 🙂

danielgrosse10:09:46

Hello @borkdude, I wrote some small tools with Babashka an really like it. So thanks for the work you put into. I currently create a more complex project and would like to add some tests. But putting the script into a Clojure folder structure brings up errors when running it. Also I don’t know how to run tests, especially with pods. Do you have an example where I can look at?

borkdude10:09:21

@danielgrosse When putting it in a Clojure folder structure you will need to add src and test to the classpath: -cp src:test

borkdude10:09:56

@danielgrosse Here is a snippet that shows how you can run tests: https://github.com/borkdude/babashka#running-tests

borkdude10:09:06

And thanks.

jeroenvandijk11:09:39

I’m trying to write a Pod in babashka using pod-babashka-hsqldb as an example (https://gist.github.com/jeroenvandijk/55dc9005035bc506493ca279d43b2236). Something goes wrong in the communication while loading the pod, but I don’t see it how it is different from the example. This is the error

bb 
{format edn, namespaces [{name (quote calculate), vars [{name (quote +)}]}], opts {shutdown {}}}
[message {id #object[[B 0x167d0535 [B@10c7ae618], op #object[[B 0x10cb3ebd [B@10c7aec00]}]
{:id2 ab473001-1b2b-482f-abd8-c9b2a7f3a619, :op2 :describe}
----- Error --------------------------------------------------------------------
Type:     java.lang.ClassCastException
Message:  clojure.lang.PersistentVector cannot be cast to byte[]
Location: /Users/jeroen/Projects/Github/bb-pod/pod-usage.bb:3:1

----- Context ------------------------------------------------------------------
1: (require '[babashka.pods :as pods])
2:
3: (pods/load-pod ["bb" ""])
   ^--- clojure.lang.PersistentVector cannot be cast to byte[]
4:
5: (println "pod loaded")
6:
7: (require '[calculate])
8:

----- Stack trace --------------------------------------------------------------
user - /Users/jeroen/Projects/Github/bb-pod/pod-usage.bb:3:1

:EOF

borkdude11:09:00

Not sure, I've got a meeting coming up, so I'll answer later today.

jeroenvandijk11:09:07

o wait i see it i think nope

borkdude11:09:40

@jeroenvandijk it often helps to write to stderr in the pod in case of an error

jeroenvandijk11:09:11

Ok it was a quoting issue in describe map , now it works 🙂

jeroenvandijk12:09:01

I’ve updated the gist. Will make something more interesting later I hope 🙂

tzzh14:09:50

I’ve released v0.0.2 of https://github.com/tzzh/pod-tzzh-aws with improved pagination, better code generation and more services (happy to include more if anybody needs)

👍 6
borkdude15:09:26

@UJADF897F btw, I checked:

borkdude@DESKTOP-JN2UNTV ~ $ /mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe

Try the new cross-platform PowerShell 

PS Microsoft.PowerShell.Core\FileSystem::\\wsl$\Ubuntu-20.04\home\borkdude> clojure
Clojure 1.10.1
user=> (System/getProperty "os.arch")
"amd64"
;; Linux:
borkdude@DESKTOP-JN2UNTV ~ $ clojure
Clojure 1.10.1
user=> (System/getProperty "os.arch")
"amd64"
                                                                                                                                                                                                         ;; MacBook

$ clj
Clojure 1.10.1
user=> (System/getProperty "os.arch")
"x86_64"

borkdude15:09:03

so on macOS or maybe depending on which Java version you are on, os.arch responds with x86_64 or amd64 hmm

tzzh15:09:46

ah ok they are the same though right ? I just copied what had been auto generated in the binaries I created tbh

borkdude15:09:07

yeah, maybe the package manager should just deal with that

borkdude15:09:51

I'll fix that, thanks

👍 3
borkdude15:09:18

borkdude@MBP2019 ~ $ glam install tzzh/pod-tzzh-aws -g
borkdude@MBP2019 ~ $ pod-tzzh-aws
works :)

tzzh15:09:16

nice 🙂

jeroenvandijk15:09:57

Here is a proof of concept for loading arbitrary Clojure libraries via Pods https://github.com/jeroenvandijk/pod-babashka-clj/blob/master/examples/aws.bb.clj

😎 6
jeroenvandijk15:09:52

It needs some more work around keeping track of the started jvm processes, but the basics work

borkdude15:09:18

why not directly go with Clojure in this case, would be the obvious question?

jeroenvandijk15:09:28

True, i’m thinking you could have a process running in the background that can deal with the things that Babashka can’t. So if you use AWS a lot it only needs to start once

borkdude15:09:08

yeah. can also use the nREPL client for that

jeroenvandijk15:09:39

yeah maybe that would be an alternative. This is using the prepl so same idea i guess

jeroenvandijk15:09:01

Writing a pod in babashka itself was easier than i had expected

borkdude15:09:20

Have you already tested the tzzh aws pod, since you are interested in AWS stuff?

borkdude15:09:44

You can now install it using glam :)

jeroenvandijk15:09:19

No I haven’t. My main concern is not having to learn new libraries 😅 I think the tzzh aws pod is inventing a new API

jeroenvandijk15:09:03

i’ll check it out now before i say something stupid

borkdude15:09:40

I think a similar approach could work for Clojure. Just generate all the code at compile time instead of using reflection/dynamic requires

borkdude15:09:59

(at least, that's how I think the cognitect lib works?)

jeroenvandijk15:09:10

yeah I agree. pretty sure the aws-api can be made compatible with Graalvm

jeroenvandijk15:09:19

it just requires some work

borkdude15:09:42

the tzzh pod looks quite easy to use though.

borkdude15:09:06

although I don't know the Java AWS API, I can't imagine why they would re-invent a different API in Go or Python

borkdude15:09:21

it's all going to the same endpoints eventually I guess

jeroenvandijk15:09:31

no i mean the clojure api is not the same

jeroenvandijk15:09:39

the backend is definitely the same

jeroenvandijk15:09:11

so I mean the api of pod-tzzh-aws is not the same as cognitect aws-api. So i would have to learn both

jeroenvandijk15:09:19

cannot reuse my code

jeroenvandijk15:09:57

And personally I don’t want to debug go code if things go wrong

jeroenvandijk15:09:22

But that’s just me. it’s nice that pod-tzz-aws exists

borkdude15:09:52

I think the idea to use a small fast starting script to invoke something in a larger stateful context is similar to Datomic Ions with AWS Lambda + Datomic running on other machines

borkdude15:09:23

so you could have one Clojure REPL running on an EC2 instance permanently

borkdude15:09:35

and babashka in AWS lambda sending commands to its REPL

jeroenvandijk15:09:01

yeah i think you are right, pretty similar

jeroenvandijk15:09:00

It could also be useful if stateful means loading a really big library that would otherwise impact startup time

jeroenvandijk15:09:20

I’m not sure if the prepl should be used for concurrent commands, but the concept could be implemented via other protocols as well

borkdude15:09:18

nREPL supports this well via ids

jeroenvandijk15:09:04

sounds like its more suited for something like this then

borkdude15:09:25

A more full example can be seen in the tests of babashka.nrepl

👍 3
borkdude16:09:05

The babashka pod protocol is more or less the same as nREPL, with a few twists

jeroenvandijk16:09:58

cool, i haven’t really dived into that protocol yet. Not sure what the tradeoffs are compared with the prepl protocol

richiardiandrea17:09:05

is this a way to generically use any clojure/java library?

richiardiandrea17:09:31

(I am looking for a quick way to run the Mongo or the RabbitMQ clients in babashka 😄)

richiardiandrea17:09:33

Oh I see what you did there, kinda cool good job!

jeroenvandijk17:09:01

Yeah the idea is that anything that tools.deps can load, can be accessed via this pod

jeroenvandijk17:09:28

Currently there is no proper management of the backend process and the port is hardcoded right now, but this can be fixed

jeroenvandijk17:09:21

And maybe some error scenarios are hard to debug because of the limited communication between the pod and the babashka process right now. Probably needs another layer to get this right

richiardiandrea17:09:57

seems like this could be a good candidate for being embedded in babashka.. pods are nice for interop with other langs but if you already have a clojure libs it can be easier

jeroenvandijk20:09:19

Yeah maybe. Pods give us a nice way to experiment with this idea for now at least. If you want to try with Mongo and RabbitMQ I can help out if you run into problems

borkdude20:09:54

doesn't mongo support an http api btw?

Dig21:09:55

what would be equivalent in babashka to a common pattern $ cat file.txt | sort | uniq -c ?

nate21:09:30

one version:

bb '(-> (slurp "file.txt") (str/split #"\n") sort distinct)'

jeroenvandijk21:09:11

you are too quick 🙂 I can only add clojure.string/split-lines over str/split

nate21:09:23

ah, good one

nate21:09:45

I like to let bb to the input, so I would prefer this:

cat file.txt | bb -i '(-> *input* sort distinct)'

nate21:09:49

seems a bit cleaner

isak21:09:13

I think he needs frequencies , no?

Dig21:09:39

yep, that uniq -c is the tricky peace

nate21:09:07

ah, true, replace distinct with frequencies

nate21:09:17

cat file.txt | bb -i -O '(-> *input* sort frequencies)'

6
nate21:09:02

to make the number show up first:

cat file.txt | bb -i -O '(-> *input* sort frequencies (->> (map reverse)))'

👍 3
nate21:09:33

absolute closest to the original:

cat file.txt | bb -i -o '(-> *input* sort frequencies (->> (map reverse) (map (partial str/join " "))))'

nate21:09:42

ok, I'll stop now, bb is just too fun

🙂 3
borkdude21:09:50

Nicer formatting:

cat README.md | bb -io '(->> *input* sort frequencies (map (fn [[l n]] (format "%3s %s" n l))))'

Dig21:09:13

thank you! i've added couple of things and end up with

tshark -r /var/tmp/DNS.pcap00.gz -T fields -e dns.qry.name dns.qry.type==1 | bb -io '(->> *input* sort frequencies clojure.set/map-invert (into (sorted-map)) (map (fn [[k v]] (format "%6s %s" k v))))'

borkdude21:09:37

:thumbsup: clojure power!

nate22:09:20

be careful with map-invert, it'll collapse ties

nate22:09:00

(clojure.set/map-invert {:foo 1 :bar 1}) -> {1 :bar}

lucian30321:09:11

i'm getting an error trying to use drewr/postal with bb . is this something that should work or is it not supported? if the latter is there another way to send an email? i don't need anything fancy, it's just for error notifications. smtp or sendmail would be fine

----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: java.text.SimpleDateFormat
Location: postal/date.clj:24:1

----- Context ------------------------------------------------------------------
20: ;; WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21: ;; FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22: ;; OTHER DEALINGS IN THE SOFTWARE.
23:
24: (ns postal.date
    ^--- Unable to resolve classname: java.text.SimpleDateFormat
25:   (:import [java.util Date]
26:            [java.text SimpleDateFormat ParsePosition]))
27:
28: (defn make-date
29:   ([tmpl s]

borkdude21:09:22

@lucian303 Not yet supported.

borkdude21:09:19

@lucian303 I did in fact had one branch with postal in it once: https://github.com/borkdude/babashka/issues/366, but I ran into a JDK 11 bug

lucian30321:09:14

gotcha. thanks. are there any alternative libraries that you know of that might work? @borkdude

borkdude21:09:06

Probably not out of the box I think. The only alternative now is probably shelling out to sendmail or alike or using an HTTP mail service like sendgrid

lucian30321:09:47

gotcha. thanks. i'm probably just going to shell out then

borkdude21:09:49

Come to think of it, I think this can be perfectly implemented as a mailer pod

borkdude21:09:58

since an email is just plain data

borkdude21:09:35

e.g. drewr/postal could be compiled with GraalVM java 8 and then be used from babashka

borkdude21:09:04

it's a bit of work, but not too hard since there's already plenty of examples around

Darin Douglass21:09:45

The fix for postal in java 9+ is super easy https://github.com/drewr/postal/issues/107

borkdude14:09:54

I tried this, but I still get the same error

Darin Douglass14:09:44

hmm, shucks. that fixed postal for me in a normal JVM context. not sure what more graal will need 😕

borkdude14:09:13

I fixed that by adding a class to the reflection config. Now I'm looking at `javax.activation.UnsupportedDataTypeException: no object DCH for MIME type text/plain; charset=utf-8 `

Darin Douglass15:09:03

feels like we're going down the rabbit hole of javax.mail soon. (i WORK at a clojure shop that deal with email, so a rabbit hole it is)

borkdude15:09:04

Maybe writing it as a pod wrapping that other library could work

borkdude21:09:32

Ah nice. We can re-consider adding drewr/postal using that approach or implementing it as a pod first (depending on the extra size, how many people need this, etc)

borkdude21:09:12

Made an issue here: https://github.com/borkdude/babashka/issues/599 Feel free to leave feedback

borkdude22:09:09

btw, this is another lib in the SMPT space: https://github.com/toyokumo/tarayo

lucian30322:09:25

nice. i'll try tarayo as i'm a bit short on time

borkdude22:09:00

I don't think that will work with bb out of the box though

borkdude22:09:43

bb doesn't have the javax.mail.Transport etc classes currently

lucian30322:09:26

yeah, it won't