Fork me on GitHub
#babashka
<
2020-04-25
>
kah0ona08:04:20

What is a sane way to develop babashka scripts in a REPL driven way, using Emacs/CIDER? My idea was to make the script(s) as much Clojure compatible as i can (for obvious reasons) and just use normal clojure REPL. I know nREPL is now supported, but would that give the same developer ergonomics?

dpsutton09:04:58

inf-clojure was just revamped and works very well with babashka

kah0ona09:04:07

thanks both!

kah0ona09:04:21

anyone tried to run bb on a raspberry pi?

kah0ona09:04:49

would be fun to try and build some home automation scripts to control my philips hue lights etc.

kah0ona09:04:34

ah well, only one way to find out 😉

borkdude09:04:34

There might be a link to a binary in there that might work. Not sure

sogaiu09:04:29

if you have a 64 bit kernel it may work

kah0ona12:04:17

thx for the info

kah0ona12:04:04

so with (ie.) the Ubuntu in the last link on the page above could work on my pi4 that arrives tomorrow? Will try

sogaiu12:04:18

yes, i think there's a chance it will work

borkdude10:04:43

I've got hsqldb working too with babashka. A few things I'm worried about though: - It grows the binary with almost 10MB. - Adding more libraries to babashka demands more from build environments like CircleCI and Github actions. Eventually we will run out of memory again? Maybe it makes sense to split out the sql stuff to a clj-jdbc command line that is a fork/branch of babashka that includes support for Postgres and HSQLDB? The downside of this is that it's nice to have everything in one tool, without having to download additional stuff and being able to execute your program all in the same process

borkdude10:04:02

(there is a hsqldb branch you can look at)

littleli10:04:56

I agree with your conclusion. HSQLDB also can run standalone and you can connect to it via "hsql" proprietary protocol or "http". So there are basically two modes of operations. • having a fork with hsqldb bundled in • having hsqldb operating as standalone (basically having native-hsqldb project) + in babashka only have client part either proprietary "hsql" or/and "http"

littleli11:04:38

It would be sad to see "babashka" to become a monster like docker

borkdude11:04:10

in what sense, can you elaborate?

littleli11:04:26

"a huge binary doing everything", it's a departure from more unix-like philosophy of cooperating utilities.

borkdude11:04:42

my feeling is that some libraries like yaml and jdbc aren't used that often for scripting, so it might be better to split those out into different bb forks with their own names?

littleli11:04:09

ok, to be fair, I just looked at docker binary and it's not that bad yet

littleli11:04:05

I feel bb case is more difficult because it's general scripting language there is this split between being barebone and minimalistic vs have some batteries of course. Having yaml/xml/json support is probably a good thing as they're used heavily in the wild. But with SQL i'm not sure. It's really cool to have it, once you really need it. But until that point... it's just waste.

littleli11:04:22

solution probably lies somewhere in good ecosystem, but that's so general statement I'm almost shy to tell it loud.

james11:04:54

Just thinking the same. Maybe one way to keep babashka lean is to think "could I do this with curl?" and have a mechanism for scripting to include extra deps. Something like "bb install postgres" would be very nice if you want to build an ecosystem around bb. I like the asdf model of adding packages, its very low tech bash scripting.

borkdude11:04:05

I don't know how bb install postgres would work though, since the postgres support would live in its own binary, it would be another version of bb, possible called clj-jdbc or something

borkdude11:04:15

but splitting JDBC out in its own util feels like a good choice. supporting data formats like xml and yaml feels essential to scripting

👍 8
borkdude11:04:22

and we can add support for mysql etc too, and not worry about bloating babashka

borkdude11:04:43

So, tentative conclusion: > Splitting out the JDBC part into clj-jdbc won't put any more pressure on the binary size and memory usage in CI environments. It allows for addition of more drivers for mySQL etc. without bloating babashka.

👍 4
james11:04:20

Could additions be downloaded and added to the class path? To a .bb subdir or something like that. Asdf is just a master repo that points to other user supported repos. I was really surprised how usable it is. https://asdf-vm.com/#/plugins-all

borkdude11:04:59

That's unfortunately not how it works. The classes / drivers for various JDBC drivers cannot be added at runtime. They have to be compiled along with the rest.

littleli11:04:56

That's true, native-image is doing closed-world analysis and binary is almost entirely static.

james11:04:04

Fair enough.

littleli11:04:44

Maybe bb can support some 'protocols' to speak to other programs, but outside of that options are limited.

borkdude11:04:05

Well you can load dynamic libraries, but then you would have to write a JNI wrapper around a clojure library

borkdude11:04:17

which would be a lot of work

james11:04:02

I guess the key thing is to keep the default size small.

littleli11:04:54

v0.0.89 is 51MB on Mac

borkdude11:04:02

zipped it's like 12MB

littleli11:04:13

btw hsqldb has some different license... so be careful there as well. I believe it's BSD derived so no violention should be happening, but still let's double-check.

borkdude11:04:42

if you could sort that out, that would be great for the future of clj-jdbc or whatever this awesome command line tool will be named 😉

borkdude11:04:48

Splitting out clj-jdbc will also allow adding more nice libraries like honeysql etc

kah0ona12:04:04

does splitting it out mean, it becomes some sort of ‘binary library’ that bb shells out to? Or is it a ‘bb with jdbc’ ‘distribution’

kah0ona12:04:30

that first part probably doesn’t even make sense 🙂

borkdude12:04:30

it's a new CLI tool called clj-jdbc or something else if I can make up a better name

👍 4
Darin Douglass12:04:13

bb-core (aka bb), bb-jdbc, et al

james12:04:26

bdbc the first of the enterprise babashka implementations. bbdb has a nice ring to it.

kah0ona12:04:28

bb-the-sequel 😉

Darin Douglass12:04:51

Maybe we’re thinking about it slightly backwards? IMO first-class db support feels right for bb. What about having a minimal bb build for environments that have space restrictions (pies, etc)?

borkdude12:04:00

One of the concerns is that stacking more and more libraries in bb will prolong build times and memory consumption in CI, eventually we'll have to make choices

borkdude12:04:29

Also adding hsqldb support adds 10MB, I don't think that justifies it if people will only use that seldomly

Darin Douglass13:04:43

Yeah the CI resources could be an issue, if it IS an issue I’d likely pitch in some money to help make it NOT an issue. As for the binary size, install size is far, FAR down my my personal list of concerns. I’d rather have a bigger core bb and have another minimal build for people whose use-case may also be more minimal than have support for tools (even ones that I don’t use like hsqldb) limited by “will this fit on super small machines?”. What is 10MB to the hundreds of GB likely available to the majority of users?

borkdude13:04:54

What about keeping postgres, but building other databases into the other CLI?

👍 4
borkdude13:04:34

And what are the options e.g. at Github actions to up the memory against what costs?

Darin Douglass13:04:29

To be clear: I’m not advocating precisely to keep Postgres support in. Though I do use it, going back to installing and parsing psql output wouldn’t be a problem. I’m arguing that, assuming we can build the thing, binary size (within reason) shouldn’t be a limiting factor on what gets pulled into bb. If people have a use case and it doesn’t take much to implement/maintain, why not support it?

borkdude13:04:47

I'm all for supporting things, even niche things. But eventually I'm afraid I will run out of memory again in CI. This happened before on CircleCI. They were kind enough to bump my memory to 8gb. But I think there will be another limit we're going to hit

nate13:04:19

This discussion reminds me that there are two builds of Hugo, the static site generator. hugo which works for most sites, and hugo-extended which does everything the other one does and adds scss support. I think a separate clj-jdbc makes sense.

nate13:04:57

Binary size doesn't matter to me much, but it makes me wonder at what point it impacts start up time.

borkdude13:04:50

@U0510902N Can you repeat that binary size doesn't matter to you in the issue? I might be optimizing for the wrong things, so I'd like to hear this clearly recorded over there.

borkdude13:04:07

So this is the idea: https://github.com/borkdude/clj-jdbc/blob/master/README.md @UGRJKK74Y Maybe you could try your script with this

nate13:04:12

Sure thing.

Darin Douglass13:04:13

Sure I can bring stuff to the issue. I’m pretty sure it’ll work assuming curl exists as well

borkdude13:04:38

babashka.curl isn't yet there, but clojure.java.shell is

borkdude13:04:12

but the idea would be to use babashka for every day scripting and then hand off the data to clj-jdbc for database interaction

borkdude13:04:23

so combine multiple smaller tools to achieve the goal

borkdude13:04:37

one thought: postgres seems popular enough to only support that in bb?

Darin Douglass13:04:02

IMO that sounds reasonable. I’ll have to look around for CI solutions. It doesn’t feel great having that being the limiting factor here.

Darin Douglass14:04:20

There still is the option of pure-clojure kind for dbs. bb-psql could be a lib that can interact with psql.

Darin Douglass14:04:55

Could have others for the other dbs. Would save on CI resources that way.

dominicm14:04:40

You could solve problems by piping between bb types and also by introducing a rpc system between instances perhaps. Then a master bb will run the others...

borkdude14:04:46

@UGRJKK74Y You mean like babashka.curl, babashka.psql?

nate14:04:11

@U09LZR36F I agree. Having multiple cooperating binaries hints at good rpc/ipc. Bash itself has an interop format: lines of text. Babashka could do that or something more (edn/transit/nippy?).

borkdude14:04:53

nippy could be useful to add (in the spirit of supporting data formats). transit is already there

👍 4
Darin Douglass14:04:16

Yeah something like babashka.psql et al. I like the idea of of cooperating binaries; though I wonder how cohesive a script that uses multiple of those binaries would feel.

Darin Douglass14:04:38

Though I use bb less like a bash replacement and more like a quick starting, lightweight clojure

nate14:04:16

Name suggestions: bbdb, jdbbc, dbb

Darin Douglass14:04:58

Interesting. That’s saying during build you're maxing out at about 5GB? Is circle constantly giving 8 GB?

borkdude14:04:20

yes. you should subtract some 1-1.5GB from that 8GB to give to the -Xmx of graalvm. So there's about 6.5GB available. For Github actions this is only 5.5GB.

borkdude14:04:41

maybe the build could do with less, it only takes much longer where there's just enough memory available, I guess a lot of swapping is happening

borkdude14:04:12

but at a certain point, it's going to throw an exception if the -Xmx value becomes too low. This has happened on CircleCI at 4GB

borkdude15:04:44

Interestingly, I'm trying to build with the 20.2-dev version with -Xmx=3000m, and this works on my local machine, whereas before with 19.3.1 this started to fail on CI. So maybe GraalVM improved something there as well.

borkdude15:04:27

I'm now trying -Xmx=3000m with 19.3.1 and it's taking a loooong time, so maybe 20.2 has some improvements in this area

borkdude15:04:46

And eventually it crashes. So conclusion: 3g wasn't enough anymore for 19.3.1. but it might be for 20.2

borkdude15:04:57

So maybe I panicked for nothing 🙂 There's still the question: do we want to pay 10 megabytes to include an embedded SQL database called hsqldb which can persist but also run in memory?

james17:04:19

Pity about sqlite not working, it has an impressive deployment footprint and it generally follows postgres syntax. hsqldb is more in the java space only l think. https://sqlite.org/mostdeployed.html

borkdude18:04:40

yeah, I'd love that to work. that's why it's maybe better to wait with hsqldb?

Roman Tsopin20:04:38

My 2 cents. I think it would be great to have all nice stuff (db, curl etc) under the current name but distribute stripped versions also if someone really cares about binary size. Something like brew install bb —no-curl —no-hsqldb etc. I know it doesn’t solve ci issue, but from ergonomic perspective it’s more intuitive

borkdude20:04:56

Can you also mention this in the issue? Would be appreciated, @UA6TBJPN0

👍 4
lukasz23:04:11

I've shared some thoughts to the issue, I hope they're helpful

borkdude11:04:32

I have collected some thoughts of the above thread in https://github.com/borkdude/babashka/issues/382

borkdude16:04:50

This would be the kind of thing that's possible with an embedded hsqldb in bb: https://github.com/borkdude/babashka/blob/hsqldb/examples/hsqldb_unused_vars.clj Pretty powerful, no need to shell out to sqlite anymore.

nate16:04:33

I do really like an embedded persistence solution. Yet another ability that opens up possibilities.

borkdude16:04:41

Yeah... I do like the idea as well. Although sqlite seems to be more popular, I can't get it to work with graalvm

borkdude16:04:14

sqlite is written in C, while hsqldb is pure Java

nate16:04:06

10mb of Java

Darin Douglass16:04:57

Seems like with Postgres we have support for the most used DB. And with hsqldb we have support for a slim in memory/local db. There are obviously more use cases for dbs, but I feel like those cover a good chunk of them

👍 4