https://github.com/borkdude/babashka/issues/241

There might be a link to a binary in there that might work. Not sure

if you have a 64 bit kernel it may work

thx for the info

so with (ie.) the Ubuntu in the last link on the page above could work on my pi4 that arrives tomorrow? Will try

yes, i think there's a chance it will work

hsqldb binaries are here: https://clojurians.slack.com/archives/CSDUA8S6B/p1587810750013300

cc @UBLU3FQRZ and @U0JEFEZH6

(there is a hsqldb branch you can look at)

I agree with your conclusion. HSQLDB also can run standalone and you can connect to it via "hsql" proprietary protocol or "http". So there are basically two modes of operations. • having a fork with hsqldb bundled in • having hsqldb operating as standalone (basically having native-hsqldb project) + in babashka only have client part either proprietary "hsql" or/and "http"

It would be sad to see "babashka" to become a monster like docker

in what sense, can you elaborate?

"a huge binary doing everything", it's a departure from more unix-like philosophy of cooperating utilities.

my feeling is that some libraries like yaml and jdbc aren't used that often for scripting, so it might be better to split those out into different bb forks with their own names?

ok, to be fair, I just looked at docker binary and it's not that bad yet

I feel bb case is more difficult because it's general scripting language there is this split between being barebone and minimalistic vs have some batteries of course. Having yaml/xml/json support is probably a good thing as they're used heavily in the wild. But with SQL i'm not sure. It's really cool to have it, once you really need it. But until that point... it's just waste.

solution probably lies somewhere in good ecosystem, but that's so general statement I'm almost shy to tell it loud.

Just thinking the same. Maybe one way to keep babashka lean is to think "could I do this with curl?" and have a mechanism for scripting to include extra deps. Something like "bb install postgres" would be very nice if you want to build an ecosystem around bb. I like the asdf model of adding packages, its very low tech bash scripting.

I don't know how bb install postgres would work though, since the postgres support would live in its own binary, it would be another version of bb, possible called clj-jdbc or something

but splitting JDBC out in its own util feels like a good choice. supporting data formats like xml and yaml feels essential to scripting

and we can add support for mysql etc too, and not worry about bloating babashka

So, tentative conclusion: > Splitting out the JDBC part into clj-jdbc won't put any more pressure on the binary size and memory usage in CI environments. It allows for addition of more drivers for mySQL etc. without bloating babashka.

Could additions be downloaded and added to the class path? To a .bb subdir or something like that. Asdf is just a master repo that points to other user supported repos. I was really surprised how usable it is. https://asdf-vm.com/#/plugins-all

That's unfortunately not how it works. The classes / drivers for various JDBC drivers cannot be added at runtime. They have to be compiled along with the rest.

That's true, native-image is doing closed-world analysis and binary is almost entirely static.

Fair enough.

Maybe bb can support some 'protocols' to speak to other programs, but outside of that options are limited.

Well you can load dynamic libraries, but then you would have to write a JNI wrapper around a clojure library

which would be a lot of work

I guess the key thing is to keep the default size small.

v0.0.89 is 51MB on Mac

zipped it's like 12MB

btw hsqldb has some different license... so be careful there as well. I believe it's BSD derived so no violention should be happening, but still let's double-check.

if you could sort that out, that would be great for the future of clj-jdbc or whatever this awesome command line tool will be named 😉

Splitting out clj-jdbc will also allow adding more nice libraries like honeysql etc

does splitting it out mean, it becomes some sort of ‘binary library’ that bb shells out to? Or is it a ‘bb with jdbc’ ‘distribution’

that first part probably doesn’t even make sense 🙂

the latter

it's a new CLI tool called clj-jdbc or something else if I can make up a better name

bb-core (aka bb), bb-jdbc, et al

bdbc the first of the enterprise babashka implementations. bbdb has a nice ring to it.

bb-the-sequel 😉

haha

One of the concerns is that stacking more and more libraries in bb will prolong build times and memory consumption in CI, eventually we'll have to make choices

Also adding hsqldb support adds 10MB, I don't think that justifies it if people will only use that seldomly

Yeah the CI resources could be an issue, if it IS an issue I’d likely pitch in some money to help make it NOT an issue. As for the binary size, install size is far, FAR down my my personal list of concerns. I’d rather have a bigger core bb and have another minimal build for people whose use-case may also be more minimal than have support for tools (even ones that I don’t use like hsqldb) limited by “will this fit on super small machines?”. What is 10MB to the hundreds of GB likely available to the majority of users?

What about keeping postgres, but building other databases into the other CLI?

And what are the options e.g. at Github actions to up the memory against what costs?

To be clear: I’m not advocating precisely to keep Postgres support in. Though I do use it, going back to installing and parsing psql output wouldn’t be a problem. I’m arguing that, assuming we can build the thing, binary size (within reason) shouldn’t be a limiting factor on what gets pulled into bb. If people have a use case and it doesn’t take much to implement/maintain, why not support it?

I'm all for supporting things, even niche things. But eventually I'm afraid I will run out of memory again in CI. This happened before on CircleCI. They were kind enough to bump my memory to 8gb. But I think there will be another limit we're going to hit

This discussion reminds me that there are two builds of Hugo, the static site generator. hugo which works for most sites, and hugo-extended which does everything the other one does and adds scss support. I think a separate clj-jdbc makes sense.

Binary size doesn't matter to me much, but it makes me wonder at what point it impacts start up time.

@U0510902N Can you repeat that binary size doesn't matter to you in the issue? I might be optimizing for the wrong things, so I'd like to hear this clearly recorded over there.

So this is the idea: https://github.com/borkdude/clj-jdbc/blob/master/README.md @UGRJKK74Y Maybe you could try your script with this

Sure thing.

Sure I can bring stuff to the issue. I’m pretty sure it’ll work assuming curl exists as well

babashka.curl isn't yet there, but clojure.java.shell is

but the idea would be to use babashka for every day scripting and then hand off the data to clj-jdbc for database interaction

so combine multiple smaller tools to achieve the goal

one thought: postgres seems popular enough to only support that in bb?

IMO that sounds reasonable. I’ll have to look around for CI solutions. It doesn’t feel great having that being the limiting factor here.

There still is the option of pure-clojure kind for dbs. bb-psql could be a lib that can interact with psql.

Could have others for the other dbs. Would save on CI resources that way.

¯\(ツ)/¯

You could solve problems by piping between bb types and also by introducing a rpc system between instances perhaps. Then a master bb will run the others...

@UGRJKK74Y You mean like babashka.curl, babashka.psql?

@U09LZR36F I agree. Having multiple cooperating binaries hints at good rpc/ipc. Bash itself has an interop format: lines of text. Babashka could do that or something more (edn/transit/nippy?).

nippy could be useful to add (in the spirit of supporting data formats). transit is already there

Yeah something like babashka.psql et al. I like the idea of of cooperating binaries; though I wonder how cohesive a script that uses multiple of those binaries would feel.

Though I use bb less like a bash replacement and more like a quick starting, lightweight clojure

Name suggestions: bbdb, jdbbc, dbb

More data: https://github.com/borkdude/babashka/issues/382#issuecomment-619389558

Interesting. That’s saying during build you're maxing out at about 5GB? Is circle constantly giving 8 GB?

yes. you should subtract some 1-1.5GB from that 8GB to give to the -Xmx of graalvm. So there's about 6.5GB available. For Github actions this is only 5.5GB.

maybe the build could do with less, it only takes much longer where there's just enough memory available, I guess a lot of swapping is happening

but at a certain point, it's going to throw an exception if the -Xmx value becomes too low. This has happened on CircleCI at 4GB

Interestingly, I'm trying to build with the 20.2-dev version with -Xmx=3000m, and this works on my local machine, whereas before with 19.3.1 this started to fail on CI. So maybe GraalVM improved something there as well.

I'm now trying -Xmx=3000m with 19.3.1 and it's taking a loooong time, so maybe 20.2 has some improvements in this area

And eventually it crashes. So conclusion: 3g wasn't enough anymore for 19.3.1. but it might be for 20.2

So maybe I panicked for nothing 🙂 There's still the question: do we want to pay 10 megabytes to include an embedded SQL database called hsqldb which can persist but also run in memory?

Pity about sqlite not working, it has an impressive deployment footprint and it generally follows postgres syntax. hsqldb is more in the java space only l think. https://sqlite.org/mostdeployed.html

yeah, I'd love that to work. that's why it's maybe better to wait with hsqldb?

btw hsqldb has postgres compatibility: http://hsqldb.org/doc/2.0/guide/compatibility-chapt.html

My 2 cents. I think it would be great to have all nice stuff (db, curl etc) under the current name but distribute stripped versions also if someone really cares about binary size. Something like brew install bb —no-curl —no-hsqldb etc. I know it doesn’t solve ci issue, but from ergonomic perspective it’s more intuitive

right

Can you also mention this in the issue? Would be appreciated, @UA6TBJPN0

I've shared some thoughts to the issue, I hope they're helpful

@U0JEFEZH6 thanks!