Fork me on GitHub
#babashka
<
2020-09-28
>
danieltanfh9505:09:34

is there a way to package bb + the clojure script for windows users?

danieltanfh9505:09:07

currently using cat config.edn | ./app.clj with #!/usr/bin/env bb -I

borkdude06:09:11

@danieltanfh95 I think only macOS takes args in env. For standalone scripts it might be better to just use edn/read

danieltanfh9506:09:49

lemme try with my windows machine later

danieltanfh9508:09:39

btw did anyone manage to integrate spec with babashka?

borkdude08:09:24

@danieltanfh95 For now there is: https://github.com/borkdude/spartan.spec There are some reservations about including spec into bb, although it can work. The hesitation is because spec is alpha.

danieltanfh9508:09:46

im using edn instead of command line args, it’s easier to do complex arguments with it, especially if spec is available

borkdude08:09:10

spec will be at one point part of babashka, but only when it's out of alpha probably

borkdude08:09:45

until then spartan.spec will be a drop-in replacement

jeroenvandijk09:09:07

Out of curiosity, I’m looking at the code of https://github.com/borkdude/cpm and wondering how it works. It says it’s piggiebacking on tools.deps, but in the code I don’t see this connection. Is this already there?

borkdude09:09:21

@jeroenvandijk My thinking was that dependencies between packages could be done via Git deps instead of implementing dep resolution ourselves. Also it uses the classpath to discover packages. But maybe it will just be simpler if cpm will be a standalone binary with a packages repo similar to brew

jeroenvandijk09:09:23

Cool, yeah I haven’t thought about it enough. I believe a package manager would definitely be useful to grow babashka. I’m guessing this is coming from a need to organize pods as well?

jeroenvandijk09:09:31

Anyway, I also read it is work in progress. So don’t mind me, was just curious 🙂

borkdude09:09:36

That was the initial inspiration yes

borkdude09:09:54

You can now also install a SNAPSHOT babashka version using CPM: https://github.com/borkdude/cpm/#babashka

borkdude09:09:15

which is kind of meta, since you need a SNAPSHOT version to run CPM with bb :)

jeroenvandijk10:09:04

I just noticed 🙂 Nice though!

➜  ~ bb -jar cpm.jar --install clj-kondo/clj-kondo --global --verbose
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Unable to resolve classname: java.util.zip.ZipInputStream
Location: /Users/jeroen/.gitlibs/libs/borkdude.cpm/borkdude.cpm/bf8b8e284382ee7c9fa180d6db15595b54f25bbe/src/cpm/impl/impl.clj:34:13
Phase:    analysis

jeroenvandijk10:09:23

And I did this

export BABASHKA_CLASSPATH="$(clojure -Sdeps '{:deps {borkdude.cpm {:git/url "" :sha "bf8b8e284382ee7c9fa180d6db15595b54f25bbe"}}}' -Spath)"

borkdude10:09:21

Yes, this is why you need the snapshot version :)

borkdude10:09:39

which you can install with clojure

borkdude10:09:36

Btw, do keep the feedback coming and do use it (maybe only for dev now), this is the only way it will improve over time

jeroenvandijk10:09:56

works!

clj -Sdeps '{:deps {borkdude.cpm {:git/url "" :sha "bf8b8e284382ee7c9fa180d6db15595b54f25bbe"}}}' -m cpm.main --install clj-kondo/clj-kondo --global --verbose

borkdude09:09:19

I guess I could also add a SNAPSHOT version of clj-kondo. Hmm, this could be nice for people who want to be on the latest bleeding edge ;)

borkdude09:09:25

To package things up nicely, use babashka's --uberjar option:

$ bb -cp src:packages -m cpm.main --uberjar cpm.jar
This uberjar contains all packages from the classpath. You can then run it from anywhere on your system:
$ bb -jar cpm.jar --install clj-kondo/clj-kondo --global --verbose

borkdude10:09:51

Hm, I guess cpm can also be used to install individual babashka (or clojure / whatever) scripts on the path

jeroenvandijk10:09:04

How would it work? Something like #/usr/bin/env bb + additions to babashka stdlib that can add stuff to the classpath?

borkdude10:09:43

Well, e.g. deps.clj is a standalone script with a bb shebang. If I would package that in up a zip, then it would be no different from installing a binary

borkdude10:09:02

It does depend on bb so maybe that would have to be added as a dependency as well

borkdude10:09:12

bb already has a stdlib thing to add stuff to the classpath, but I don't see how that is relevant here? for some specific scripts yes

borkdude10:09:12

So then the script would maybe add :package/depends [org.babashka/babashka] and that package will be installed as well

jeroenvandijk11:09:00

cool, sounds great 🙂

jeroenvandijk11:09:42

I guess a bit like how you can add binaries via ruby gems? (maybe also in python via pip, not sure)

borkdude11:09:41

clojure -M -m cpm.main --install borkdude/deps.clj --global --verbose
will effectively do:
clojure -M -m cpm.main --install borkdude/deps.clj org.babashka/babashka --global --verbose

borkdude11:09:34

if deps.clj depends on babashka

borkdude11:09:40

so then you will have bb on your global path as well

jeroenvandijk11:09:47

ah cool, so do i understand correctly that cpm would wrap deps + babashka? So calling cpm would allow fetching all deps and then run a main ?

borkdude11:09:33

no. it doesn't wrap anything. it can install deps and babashka, that's all

borkdude11:09:53

after that you can run deps.clj, the script.

borkdude11:09:55

cpm doesn't run programs, it's just a package manager that can put binaries (or scripts) on your path

borkdude11:09:20

just like brew basically

borkdude11:09:23

but cross platform

borkdude11:09:44

and possibly using clojure for scripting if you need to move things around after fetching the artifacts

jeroenvandijk11:09:21

ah i see. So you meant the executable could be a babashka script and because of it’s defined deps those will also be available in that shell session. I think i get it :relaxed:

borkdude11:09:25

so if you don't have bb installed, but you install deps.clj (as a bb script, not the binary deps), then it will also make sure bb will be installed, so you can run the script

jeroenvandijk11:09:59

I see now that’s more ~/.cpm/path gives binary paths. I think it’s clear now 🙂

borkdude11:09:33

yes, and you can add:

# cpm
export PATH="`cat $HOME/.cpm/path 2>/dev/null`:$PATH"
in your .zshenv or whatever and those executables will be available in all your shells

jeroenvandijk11:09:06

would that also pick up new binaries just after installing it via cpm? I’m guessing brew adds symlinks to /usr/local/bin for that reason

borkdude11:09:28

yes, if you use --global it will overwrite that path file

borkdude11:09:37

you will need to reload your shell though

sogaiu11:09:36

don't typical shells have a rehashing command usually so you don't have to reload? something like this: https://superuser.com/questions/490983/how-to-rehash-executables-in-path-with-bash

sogaiu11:09:51

it's not always called rehash iiuc

borkdude11:09:54

could be but the way cpm is set up now is that the path is added via a file

borkdude11:09:01

there are no global symlinks, etc

jeroenvandijk11:09:52

yeah i meant that part. Might not be an issue

borkdude11:09:57

I'm not sure about working with symlinks, since I also want to support Windows

jeroenvandijk11:09:16

True, probably better to wait for some feedback

borkdude11:09:26

it's a minor issue as I see it now

borkdude11:09:07

maybe a tiny bash function could be added and that can be called post-install

borkdude11:09:20

I'm sure there's a way to fix it

borkdude11:09:21

alias cpm_path='export PATH="`cat $HOME/.cpm/path 2>/dev/null`:$PATH"'
cpm_path
and then run cpm_path again post-install

borkdude11:09:13

(can be improved by first removing the old cpm_path from the path)

borkdude11:09:38

We could also make main return a bash command (using a --bash) flag so you can use source (and --powershell, etc), not sure if that's useful. Will wait with that.

jeroenvandijk11:09:26

If I understand correctly the current distribution method is via maven? Are you considering adding git repos? Maybe useful for being able to add (babashka) gists?

borkdude11:09:57

can you be more specific?

jeroenvandijk11:09:42

yeah i was thinking of creating a gist and being able to call (+ install) it as an executable? From the install instructions I’m thinking it’s not possible (yet). It seems the script is currently made to download a zipfile?

borkdude11:09:42

- cpm.edn files are currently read from the classpath. I am considering also supporting .cpm/packages/your-packages so you can git clone your package repository there. - yes, it currently only fetches zip files, I want to add tar.gz, possibly other things as well

borkdude11:09:03

maybe it's already possible to download a gist as a zip file from github

borkdude11:09:45

yep, there is a "Download zip" button

jeroenvandijk11:09:26

ah cool, so the current way would be to add it manually or add it via in repository dir of the cpm repo. And you are consider having personal packages

jeroenvandijk11:09:42

I’m thinking out loud 😊

borkdude11:09:14

the current way is to have a jeroenvandijk/my-cool-gist.cpm.edn file on your classpath (either bb classpath or clojure classpath)

borkdude11:09:54

cpm/packages is also on the classpath because packages is in :paths in deps.edn

jeroenvandijk11:09:16

It sounds like the sky is the limit 😎

borkdude12:09:41

Maybe for installing things like GraalVM we need to add some more stuff, e.g. when unzipping it, you will get a directory structure in which the binaries are not on the top level, so we need to expose a couple of paths instead of just the top level

borkdude12:09:30

And maybe having aliases for binaries would also be nice in case of conflicts

thomas.ormezzano13:09:28

Hey, I am trying to pass a function from a pod to another client side function from a pod but that’s causing issue as it can’t be serialized

(require '[pod.tzzh.a :as a])

(a/do-something-client-side a/some-pod-fn)
and I am getting Cannot JSON encode object of class: class babashka.pods.impl$bencode__GT_vars$fn__15573$fn__15577: [email protected]0

borkdude13:09:25

correct, you can't pass functions through pod borders

thomas.ormezzano13:09:43

ah ok I see thanks 👍 I can work around it but I would have thought that client-side functions didn’t need to serialize their args / speak to the pod

borkdude13:09:47

this example (a/do-something-client-side a/some-pod-fn) should work purely client side, if a/do-something-client-side was for example defined as (def do-something-client-side identity) but it appears like you are passing a function object back to the pod or vice versa

borkdude13:09:30

everything that goes back to the pod or comes from the pod has to be serializable

borkdude13:09:47

you could work around it using symbols or macros

borkdude13:09:17

'a/some-pod-fn or (defmacro client-side [fn-sym])

thomas.ormezzano13:09:53

OK cool I will take another look thanks

borkdude13:09:08

but I wouldn't know why you would pass back a pod function itself to the pod?

borkdude13:09:31

more concrete example will likely help

borkdude13:09:37

afk for an hour, will check back then

thomas.ormezzano13:09:14

OK yeah basically it’s for the aws pod I was trying to improve the pagination and lazily load pages so I have now a new namespace that exposes this client code:

(defn paginator-fn
      "Returns a fn that lazily fetches the pages for a given aws fn"
      [page-fn]
      (fn get-pages
        [input]
        (lazy-seq
          (let [page (page-fn input)]
            (if-let [next-continuation-token (:NextContinuationToken page)]
              (cons page (get-pages (assoc input :ContinuationToken next-continuation-token)))
              [page])))))
so you could do
(require '[babashka.pods])
(babashka.pods/load-pod ["./pod-tzzh-aws"])
(require '[pod.tzzh.paginator :as p])
(require '[pod.tzzh.s3 :as s3])

(let [paginator (p/paginator-fn s3/list-objects-v2)]
    (paginator {:Bucket "some-bucket"
                :Delimiter "/"
                :Prefix "some-prefix"
                :MaxKeys 50}))

borkdude13:09:34

ah yes, then simply make that (p/paginator-fn 's3/list-objects-v2) or write p/paginator-fn as a macro so you can pass it as a symbol and expand to (p/paginator-fn* 's3/list-objects-v2)

borkdude13:09:30

hmm, I'm not sure if I understand still

borkdude13:09:46

is (p/paginator-fn s3/list-objects-v2) running entirely client side?

borkdude13:09:19

yeah, that should work I think

borkdude13:09:46

running only (p/paginator-fn s3/list-objects-v2) does that work without error?

thomas.ormezzano13:09:25

no that’s the one that’s throwing an error

borkdude13:09:40

I need to see your pod branch for more info

borkdude13:09:43

back in an hour

borkdude13:09:29

so if you could post me the link by then, that'll be great

borkdude13:09:54

@thomas.ormezzano A test case: define a client side pod fn as (defn foo [f x] (f x)) and call it from bb as (foo inc 1). This should not communicate with the pod at all, and should return 2

thomas.ormezzano13:09:08

OK I’ll try that, I have also pushed my branch https://github.com/tzzh/pod-tzzh-aws/pull/2

thomas.ormezzano13:09:19

clojure.lang.ExceptionInfo: Cannot JSON encode object of class: class clojure.core$inc: [email protected]

thomas.ormezzano13:09:43

do I still need name when using code when describing the namespace ?

borkdude14:09:11

@thomas.ormezzano which bb version are you using?

borkdude14:09:26

that should be ok

borkdude14:09:18

can you push your branch with the foo function added?

borkdude14:09:56

@thomas.ormezzano I added a test for this at babashka pods and it works as expected: https://github.com/babashka/babashka.pods

borkdude14:09:59

Maybe it's a capitalization issue?

borkdude14:09:09

bb pods expect the json to be "code"

thomas.ormezzano14:09:14

OK thanks for checking I must have messed up somewhere I’ll try to have a deeper look when I have a bit more time

borkdude14:09:30

or wait, maybe it's an issue in JSON decoding in bb pods... I haven't tested that case

borkdude14:09:59

neh, it's in bencode, should be the same

borkdude14:09:46

Can you print the JSON that is generated by the go code?

borkdude14:09:09

bencode / json

borkdude14:09:59

hmm, you could try to run your pod using the babashka.pods library on the JVM and insert some debugging

borkdude14:09:03

to see what is coming through

thomas.ormezzano14:09:43

ok it’s definetely the serialisation not working, basically if I have Code string “code” then it works but the other Vars get an empty string

thomas.ormezzano15:09:46

ok fixed it now 🎉 thanks a lot for the help

thomas.ormezzano15:09:32

ok so for anybody that might be using the pod I have open https://github.com/tzzh/pod-tzzh-aws/pull/2/files which would allow to have better pagination as the pages would be loaded lazily, I also want to remove the current .*-pages fn, e.g s3/list-objects-v2-pages as the pod is still very new and they wouldn’t be useful anymore with the new pagination. If anybody is using the pod and is not happy with this let me know.

borkdude15:09:12

I think considering the age of this project, some breakage would be ok. As a safe step you could undocument the things you want to remove and then remove it later

borkdude15:09:01

Also keeping a CHANGELOG.md with breaking changes would help users

richiardiandrea16:09:54

Hi there i am a babashka newbie...I wonder what I can do to make a clojure lib working...for instance this one? http://clojuremongodb.info/ I tried to hook it up in the classpath but got an error back

borkdude17:09:02

@richiardiandrea bb is only able to interpret a selected set of Java classes (e.g. if joda time is used in that lib, bb can't handle it) and not all Clojure constructs (notably deftype is currently missing) So currently there's no way you can run that library from source in bb.

richiardiandrea17:09:49

oh i see, so the Java class needs to be in bb for it to work...basically any wrapper is a bit out of luck?

borkdude17:09:49

unless you write a pod and compile that library with GraalVM to something native, that could work, but that's work, not out the box

richiardiandrea17:09:19

I will inspect the pod documentation, thank you Borkdude for the prompt answers

borkdude17:09:33

@richiardiandrea Sure! Shelling out to another CLI is also an often utilized technique: https://github.com/DarinDouglass/clj-psql

richiardiandrea17:09:18

this is a good idea, in my case I would shell to either mongo or rabbitadmin 😉

borkdude17:09:59

But I do have a postgres sql pod as well: https://github.com/babashka/babashka-sql-pods/

mathpunk17:09:19

I'm not getting quite the output that I expect --- check this out:

$ echo "hello there lets repunctuate"|bb -io '(spit "foo" (str/replace *input* " " ":"))'
$ cat foo                                                                                           
("hello:there:lets:repunctuate")

mathpunk17:09:40

missing output flag maybe?

mathpunk17:09:19

hm, the presence or absence of the o flag has no effect on this

mathpunk17:09:29

$ echo "hello there lets repunctuate"|bb -i '(type (str/replace *input* " " ":"))'             
java.lang.String

borkdude17:09:36

@mathpunk

$ echo "hello there lets repunctuate" | bb -io '(type *input*)'
clojure.lang.Cons
$ echo "hello there lets repunctuate" | bb -io '(str/replace (first *input*) " " ":")'
hello:there:lets:repunctuate
*input* is a seq of lines, unless you use --stream

mathpunk17:09:37

huh! I guess I expected str/replace to throw an error, rather than returning something that looked a lot like it had been executed on a string

borkdude17:09:37

yeah, I guess that function is pretty forgiving:

user=> (clojure.string/replace 1 "1" "2")
"2"

mathpunk17:09:42

just a very chill, accepting function

mathpunk17:09:49

thanks for the help!

goomba20:09:01

ran into an uberscript error. I assume this is the same issue of only certain Java classes work right now?

$ bb -cp $(clj -Spath):src -m templatus.core --uberscript templatus.clj
rlwrap: warning: Your terminal 'dumb' is not fully functional, expect some problems.

warnings can be silenced by the --no-warnings (-n) option
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: java.util.regex.Matcher
Location: clostache/parser.clj:1:1

----- Context ------------------------------------------------------------------
1: (ns clostache.parser
   ^--- Unable to resolve classname: java.util.regex.Matcher
2:   "A parser for mustache templates."
3:   (:use [clojure.string :only (split)]
4:         [clojure.core.incubator :only (seqable?)])
5:   (:require [ :as io]
6:             [clojure.string  :as str])

borkdude20:09:49

@goomba which bb version is this

goomba20:09:35

good question

goomba20:09:59

$ bb --version
babashka v0.2.0

borkdude20:09:26

Are you trying to make an uberscript for non-bb usage?

borkdude20:09:20

The next error you will hit using this project is:

Message:  Could not resolve symbol: clojure.lang.Reflector/invokeConstructor
Location: clojure/core/incubator.clj:73:4
73:   (clojure.lang.Reflector/invokeConstructor
       ^--- Could not resolve symbol: clojure.lang.Reflector/invokeConstructor
74:    (clojure.lang.RT/classForName class-name)
So maybe forking the project and using bb reader conditionals may work

borkdude21:09:11

@goomba if you're looking for a templating library, this one works with bb: https://github.com/weavejester/comb

borkdude21:09:23

if your goal was to simply create an uberscript that you can run with clojure on the JVM: I can likely improve the uberscript feature by just ignoring classes in namespace forms

goomba22:09:30

@borkdude thanks for the feedback. The Babashka runtime is the goal or I could just run it with clj. I l'll look into comb, thank you for the feedback!