Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2020-05-15

Channels

#babashka
<
2020-05-15
>
Flo09:05:21

Just wanted to echo how amazing an expericene and UX babashka is. Recently had to work in a simple rest api in python. Created a simple bb for quick integration-testing (utilizing babashka.curl) and modified it on the go implementing each endpoint & subsystem. Worke like a charm. The system still needed some python-native test-cases on top, but for "live testing" they would have never been as efficient and fast in development/runtime as my lil bb helper. Thanks for the amazing work!

borkdude09:05:27

😄 thanks for sharing

Flo09:05:53

Was thinking about tooling that facilitates bb and pods to create new dev infrastructure. The gist of the idea is having tooling on top of bb/pods that allows for easily pushing & running bb scripts on dedicated cloud infrastructure. I always wanted to be able to simply push some small function somewhere and serve it as a, e.g. http endpoint. Think ease of php deployment. Have some thoughts around how to handle dependencies & version control in a (hopefully) painless way as well in such infrastructure. So, are there already any projects working in that direction that I should look into before starting from scratch?

borkdude09:05:43

@web20 pods are brand new shiny things, there's no dependency management for it, but maybe we can just piggie back on something existing like brew

Flo09:05:33

do you think it would be feasible to think in terms of data-defined dependencies/versions? The most intrinsic would likely be to version pods by some hash of a cleaned up version of its code. Or maybe based on its provided namespaces?

Flo09:05:53

With something as granular as bb/pods I'd love go get away from manual versioning

borkdude09:05:31

maybe pods can just implement a version function?

borkdude09:05:53

I'm not sure of how you would like to use these versions

Flo09:05:49

optimally i'd like to get away from thinking in terms of versions as much as possible. hence the idea of making versioning implicit based on some properties of the code, instead of explicitly chosen by the maintainer. As a user you would of course still want to be able to pin a certain revision of a dependency for it to remain consistent if re-run/-deployed elsewhere later.

Flo09:05:25

wondering how tooling could make this happen seamlessly, but transparently

Flo09:05:48

(not sure if I described it in an understandable way, lmk if I should elaborate)

Flo09:05:02

to be more specific how i'd use the versions: Say I develop some simple http endpoind handler that depends on a pod locally. when I'm finished, I want to push that script off into the magical new babashka-cloud to run there. How do I ensure, that the pod dependency I used locally is also used remotely

borkdude09:05:17

@web20 the pod protocol isn't versioned itself right now. pods are asked if they implement certain ops or not and will be called conformably

Flo09:05:46

without depending on things like commit hashes or other things that are not really a representation of the actual API the pod provides

borkdude09:05:07

maybe the script could install the pod itself if it isn't there?

borkdude09:05:55

like: just curl the binary if it can't be found?

Flo09:05:35

if the pod has some history and its API changed over the course of time, how'd the script know how to get the revision of the pod with the correct API that was depended on during development?

Flo09:05:25

think right now the solution would be to curl the binary at a certain commit hash or sth

borkdude09:05:28

I guess you would encode that in the pod install function? like: curl this or that release from github releases.

borkdude09:05:54

it's the same with any other binary you depend on for shell scripting

borkdude09:05:21

pods can come from any ecosystem, like pip install pod-foobar-sqlite

Flo09:05:51

I'd love to see if there's some way to remove the need for the maintainer to version it's library/pod with all the shortcomings (following some thoughts from the spec-ulation talk), but have the tooling be able to implicitly version the actual API of the pod, not the code itself

Flo09:05:05

e.g. if I decide to add documentation to all my functions, or refactor some code, but don't touch the API itself (i.e. any user wouln't notice a difference) I'd still need to bump the version in a pip-like ecosystem

borkdude09:05:28

if you follow the spec-ulation talk you should never make any breaking changes. but if you add a new pod function and come to depend on that in your script, then you'd want to have a way to know if you can call that function, right?

Flo09:05:08

exactly, but that's an API dependency in my opinion, not a "code dependency", do you agree?

borkdude09:05:47

yes, but how would you handle that? (resolve 'pod.foobar.sqlite/can-i-call-this-one)?

Flo09:05:37

that's what i'm trying to figure out 🙂

Flo09:05:58

like what property of a function/ns/pod can be used to define its revision, that works (well enough) in such a system

borkdude09:05:01

maybe just a version number works. never break, but at least you know that something is there if you use x version

borkdude09:05:43

Rich's libraries still use version numbers

borkdude09:05:48

The describe op expects a map back. You could use a hash of that for versioning. If you add functions, the version will change.

Flo09:05:52

true, yet they are created with such a discipline for accretion only, and no backwards changes, that its version numbers are (in my little experience) never something of concern when upgrading. I like to imagine an ecosystem that somehow enables/enforces a workflow that provides the same result in terms of no-breaking changes

borkdude09:05:57

However, if you add arities, the version won't change. So it's not perfect.

borkdude09:05:28

no-breaking-changes always takes discipline, imo that can't be automated

borkdude09:05:36

it's a culture, not something enforced

Flo09:05:12

also if we were to implicitly version the api by tooling?

Flo09:05:23

would never be perfect imo, but at least to a decent degree?

Flo09:05:34

so that it becomes culture of said ecosystem

Flo09:05:07

I like the thought of hashing the describe output, that could be a good start!

borkdude09:05:34

well, you could add extra information to the describe output about your arities, then the version will change

borkdude09:05:03

and since it's bencode, the representation is always the same

borkdude09:05:09

since dictionary keys are sorted, etc

Flo09:05:45

i really like that! also, that way (if it's handled by tooling) it could even be at the granularity level of each function

borkdude10:05:10

e.g.: 

$ bb -e '(bencode/write-bencode System/out {"namespaces" [{"name" "ns1" "vars" [{"name" "var1"}]}]})'
d10:namespacesld4:name3:ns14:varsld4:name4:var1eeeee%

borkdude10:05:42

$ bb -e '(bencode/write-bencode System/out {"namespaces" [{"name" "ns1" "vars" [{"name" "var1" "version" "1"}]}]})'
d10:namespacesld4:name3:ns14:varsld4:name4:var17:version1:1eeeee%

borkdude10:05:27

the describe output can also be propagated to the output of load-pod so the script can do whatever it wants with it

borkdude10:05:47

if that helps

borkdude10:05:36

then you could determine the version of a function yourself

Flo10:05:17

thanks a lot, that all really helped greatly. Will dive a little more into the bb codebase and work on some pods to get a grip of the developer UX. 🙂

borkdude10:05:12

Note that pods can be developed and tested outside of babashka using the JVM as well: https://github.com/babashka/babashka.pods

borkdude13:05:35

About pods: pods use the same tech as nREPL. If you can build an nREPL server or client in a language, you can also build a pod in it. Using this page you could find the necessary ingredients (e.g. bencode libraries) for the language: https://nrepl.org/nrepl/beyond_clojure.html cc @jeroenvandijk @sogaiu

jeroenvandijk13:05:34

@borkdude Yeah would be cool. Maybe in the browser Pods can interact more directly since it’s all js objects in the end and not random bytes. I’ll do some experimentation

jeroenvandijk13:05:24

I see Pods more as a concept here, but i guess the bencode approach gives guarantees that there is a possibility

borkdude13:05:34

Is your idea of letting compiled libraries talk to each other using EDN in a browser?

jeroenvandijk13:05:14

Ideally plain clojure objects, but then I need to make sure I control the order of compilation and fields associated are in sync with the multiple libraries. Maybe one core library should act as a middleman. I have a feeling there is something possible here. Otherwise edn or transit could work I guess

jeroenvandijk13:05:26

maybe that’s easier to start with

jeroenvandijk13:05:53

for performance reasons the other approach could be tried later

borkdude13:05:43

so you're not actually going to use the pod protocol, but more something that is inspired by pods?

jeroenvandijk13:05:26

I’m not sure yet. I think the Pods protocol like on the JVM might cause unnecessary overhead plus it requires more control over execution time. In the browser it would be nice if you would*n’t* have to add a layer of callbacks

borkdude13:05:00

what's the difference between calling libraries and "browser" pods then?

jeroenvandijk13:05:29

What I like about your Pods implementation for the JVM is that it doesn’t require anything different. It feels like the original library. But under water it relies on asynchronous operations, right?

jeroenvandijk13:05:04

In a javascript runtime asynchronous means callbacks. So you can’t use that library anymore like you are used too

jeroenvandijk13:05:42

if we can keep the synchronous nature of a library that would have my preference

borkdude13:05:47

yeah, you can't block a thread in a browser

jeroenvandijk13:05:17

But i think the api you have created with the command etc can be used

jeroenvandijk13:05:31

it would only have to return directly

borkdude13:05:59

by default all operations are sync unless a function declares itself async

borkdude13:05:19

but the message exchange is async, because you want to be able to call multiple functions at the same time

jeroenvandijk13:05:45

so the message exchange would have to be slightly different

borkdude13:05:52

for browser pods it can be slightly simplified

borkdude13:05:14

also the message format could be JSON since browsers already have that

jeroenvandijk13:05:23

it’s gonna be awesome 🙂

borkdude13:05:47

but then again, what is the difference between your idea and a normal library?

jeroenvandijk13:05:08

can you load a normal library on any webpage?

jeroenvandijk13:05:18

without compiling it beforehand

borkdude13:05:27

you mean a JS or CLJS lib?

jeroenvandijk13:05:31

think observablehq, where i loaded Sci

jeroenvandijk13:05:51

i mean you write cljs in Sci in a normal webpage

jeroenvandijk13:05:15

so anything that is pushed to npm can be dynamically loaded on a webpage on demand

jeroenvandijk13:05:43

how would you implement observablehq with normal cljs libraries?

jeroenvandijk13:05:49

it’s not possible

borkdude14:05:00

you can publish those libs to npm using a bundle target

borkdude14:05:25

like sci itself has done

jeroenvandijk14:05:26

but it would not interact in a clojure way. You would have to do the serialization yourself

borkdude14:05:02

yes, but in pods you are also doing that yourself

jeroenvandijk14:05:06

a clojure map from one advanced compiled library to another gives problems

jeroenvandijk14:05:12

no the library author is doing it 🙂

borkdude14:05:25

yeah, that's what I meant

jeroenvandijk14:05:26

not the library user

jeroenvandijk14:05:33

big difference

jeroenvandijk14:05:04

ok let me show you later 🙂

lukasz13:05:12

@borkdude regarding pods - I'm working on clj-kondo engine for CodeClimate, so I can either use bb + shell out to clj-kondo or use the clj-kondo pod. Which one is preferable/recommended?

borkdude13:05:58

@lukaszkorecki Up to you. Pods integrate a little bit nicer, but since clj-kondo can already spit out EDN the difference isn't that big.

borkdude13:05:29

If you use pods you can use more "core" functions, like merge-configs and print-findings!

jeroenvandijk13:05:53

@borkdude Pods should also have less latency for sequential commands, right?

borkdude13:05:28

yeah. the pod is left running during the script. so if you need to consult clj-kondo multiple times then pods are better

borkdude13:05:03

@lukaszkorecki Pods are also available on the JVM, if that adds anything useful

borkdude13:05:15

but then again, so is clj-kondo

lukasz13:05:30

Oh, merge-configs is actually useful, as I will have to hook into user-supplied config via codeclimate's yml file

dabrazhe14:05:56

Hi. When I open bb repl I can't do much with the input because it displays the codes for the arrow keys (^[[D etc). If there a way to fix this?

borkdude14:05:55

Yes, use rlwrap bb

borkdude14:05:13

@U96LS78UV Or connect from your favorite editor to a socket REPL or nREPL

dabrazhe16:05:02

rlwrap works, thank you! I do use calva but it's repl is a bit clunky compared to plain cli.

firstclassfunc17:05:18

Hello all, is it possible to consume a json string on the command line and parse it into edn from *input*

lilactown17:05:03

@gary001 you could use jet which does this for you

lilactown17:05:49

echo '{"asdf": [1, 2, 3]}' | jet --from json | ./your-script

borkdude17:05:00

@gary001 

$ bb '(json/parse-stream *in*)' <<< '[1,2,3]'
(1 2 3)

borkdude17:05:15

no json with *input*

firstclassfunc17:05:05

ahh that helps thanks!!

lukasz18:05:07

This is really cool: http://agdr.org/2020/05/14/Polyglot-Makefiles.html It means that this works, and I can write a bit more maintainable makefiles: https://gist.github.com/lukaszkorecki/4283f110cb423b25c297b9b349f98563 ;-)

borkdude18:05:54

I'm getting Makefile:8: * missing separator. Stop.

borkdude18:05:00

when I execute it with make

borkdude18:05:05

Something like this should work, but it doesn't for me: 

bb: .SHELLFLAGS := -e
bb: SHELL := bb
bb:
	(+ 1 2 3)

lukasz18:05:18

It's probably because it's missing actual tab characters

lukasz18:05:23

let me fix the gist

borkdude18:05:06

even when I use tabs

borkdude18:05:15

$ make
(+ 1 2 3)
java.lang.Exception: File does not exist: -c
make: *** [bb] Error 1

lukasz18:05:20

Then it's probably make version - it works with make 4.1

lukasz18:05:39

macOS make is v3.8 from 2006

lukasz18:05:45

(at least in my env)

borkdude18:05:48

yeah... let me upgrade make

borkdude18:05:19

$ gmake
(+ 1 2 3)
6

borkdude21:05:14

$ rlwrap ./bb
user=> (def f (io/file "/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.2-alpha1/clojure-1.10.2-alpha1.jar"))
#'user/f
user=> (def jf (java.util.jar.JarFile. f))
#'user/jf
user=> (def e (.getEntry jf "clojure/core.clj"))
#'user/e
user=> (subs (slurp (.getInputStream jf e)) 0 30)
";   Copyright (c) Rich Hickey."