Fork me on GitHub
#babashka
<
2020-04-27
>
Ahmed Hassan07:04:43

I think babashka should keep database binary separate, in this way it will be easier to add SQL DSL libraries to it without worrying about size.

borkdude08:04:32

Most people do not seem to care about the binary size of babashka, as long as it has good startup time.

borkdude08:04:21

DSL libraries (transforming data -> data) do not add that much, it's more the drivers themselves (sqlite 20MB, hsqldb 10MB) that put extra pressure on size, but more importantly memory usage in CI.

borkdude08:04:34

Feature flags like @lukaszkorecki has suggested may be the way forward for very specific functionality and it can be included in the standard distribution when there is sufficient demand

borkdude09:04:05

I updated the goals of babashka here: https://github.com/borkdude/babashka#goals I added connectivity. Being able to connect to things like @lukaszkorecki is doing with UDP seems very useful. We already had sockets and http. Now we also have JDBC as a connectivity option (albeit only for Postgres at the moment). One other goal is multi-threading so from that perspective it makes sense to add some classes around the Executor stuff.

babashka 20
rovanion10:04:49

Any idea how hard it would be to get https://cljdoc.org/d/funcool/datoteka/1.2.0/doc/user-guide running in Babashka? When I try to import it I get the error message that its Unable to resolve classname: java.nio.file.OpenOption. java.nio.file.Path is resolveable though.

borkdude10:04:56

@rovanion extend-protocol and extend-type are not available, so that's where it currently breaks down. The java.nio.file.OpenOption class is just not mapped, but could be added, I don't see a reason why not.

borkdude11:04:14

I just pushed this class to master. If you want to experiment with new binaries, take a look at #babashka_circleci_builds (takes about 5 minutes for them to appear)

rovanion11:04:31

Right, thank you!

borkdude11:04:08

I do want to look at support for defprotocol because that seems the biggest incompatibility at the moment. But that could take a while 🙂

rovanion11:04:40

Seems like a lot of unknowns.

borkdude11:04:47

what do you mean with unknowns?

rovanion11:04:25

Looking from the outside there are a lot of things I do not know about how protocols work. It becomes hard to estimate the scope of a piece of work with a lot of unknown things about it.

borkdude11:04:04

the class is there now in #babashka_circleci_builds

jeroenvandijk07:04:09

My 2 cents, I don’t use Postgres at the moment, but I can imagine this addition is useful if you do. I also don’t care too much about the binary size. I can imagine things like this will put an extra burden on the maintainance of Babashka. I still hope there might be a future with some kind of native graalvm libraries that can be included without recompilation

borkdude09:04:31

That would be the most awesome.

borkdude09:04:54

But also very hard to pull I think, without writing a ton of boilerplate JNI code

sogaiu21:04:18

perhaps some day this will be usable and helpful? https://openjdk.java.net/projects/panama/

jeroenvandijk07:04:18

@U04V15CAJ Maybe babashka could emulate the JNI code specifically for a set of babashka libraries. Let’s say babashka is split up in modules, bb-core, bb-yaml, bb-psql etc, whatever make sense. Every module would have the same clojure core and serialization functions, meaning a script run by bb-core can be called with transit or just edn and we know bb-yaml will understand it because they share this serialization capability. Since bb-yaml has a very fast startup time just like bb-core and the other bb binaries, the overhead of shelling out is minimal. bb-core would have a utility function call-module that could be used like (call-module "bb-yaml", input-stream, optional-transformation-fn) and returns a promise or lazy sequence. All these libraries would need to be updated when the core functionality is upgraded (but this should become rarer over time). Individual libraries can be added and updated without touching any of the others

jeroenvandijk07:04:32

Does that make sense?

jeroenvandijk07:04:12

You could have a bb-bare that contains the minimal functionality, but is able to call other modules. People that care about binary size or want to add custom modules could use this one. A bb binary like you have right now would be a collection of modules that are nice to get started

jeroenvandijk07:04:21

Maybe this could be setup similar to how yada has it with: default , lean (https://github.com/juxt/yada/tree/master/bundles/lean)

borkdude09:04:01

Yes. I want to experiment with JNI directly as well. If that doesn't work out, it's going to be bb + bb-sql + bb-whatevers

jeroenvandijk09:04:29

Ok cool 🙂

jeroenvandijk09:04:16

Are you also thinking about the shared library idea that you did with Rust?

jeroenvandijk09:04:52

I guess you would have to not just be able to call Sci from Rust, but also Rust from Sci

jeroenvandijk09:04:40

Maybe you already did that, I lost track 😅

jeroenvandijk09:04:28

nice 🙂

borkdude10:04:31

The major thing I'm struggling with is how to return an Object from a native library to the caller.

jeroenvandijk10:04:51

Is it possible to use Java libraries like https://github.com/clojure/data.fressian ?

jeroenvandijk10:04:26

so same question would be can you return bytes?

borkdude10:04:38

well, I want to return just a hashmap with all the functions in it, so I can hook it to sci

jeroenvandijk10:04:52

Not sure if I follow. What kind of functions would be on the hashmap?

borkdude10:04:13

I would just return {:namespaces {'next.jdbc {'execute! ...}}} from the library and insert that into babashka's sci context

jeroenvandijk10:04:35

Ah I see and the actual function would be called via JNI I guess

borkdude11:04:32

I'm about to give up on this approach again.

user=> (clojure.lang.RT/loadLibrary "sci")
nil
user=> (babashka.modules.PgNextJDBC/init 1 2)
debug
{:namespaces {cheshire.core2 {generate-string #object[cheshire.core$generate_string 0x3cdf2153 "cheshire.core$generate_string@114305ce0"]}}}
debug2
Execution error (NullPointerException) at babashka.modules.PgNextJDBC/initPG (PgNextJDBC.java:-2).
null

borkdude11:04:21

When going down the serialization route (every argument must be serialized to and from edn, transit, or whatever): I'm afraid that quickly becomes limiting, since you can't pass any database connections or Java objects etc that way

borkdude11:04:40

So making another bb-sql is most flexible and easiest to maintain maybe

jeroenvandijk13:04:11

What if the thing that is hard to serialize stays in the module? E.g. the database connection would never exit the bb-sql module. You would only refer to it by an id? The results would be normal edn data?

borkdude14:04:11

I guess that would not be very different from just shelling out to another bb-sql CLI and use EDN to transfer data. And that would be a lot easier

borkdude14:04:23

I've created an issue about native libraries here: https://github.com/borkdude/babashka/issues/397

jeroenvandijk16:04:26

I added a comment about boot pods. Maybe it’s interesting to use that as an example

borkdude17:04:12

good analogy!