A little Christmas present and teaser for our pending 1.0 version: Datahike can now be compiled as a shared library with libdatahike/compile-libdatahike and we provide experimental Python bindings for it now here https://github.com/replikativ/pydatahike. There is also a fairly complete command line client that can be built with bin/build-native-image , which also should be easy to use from babashka/libsci (no pod needed, since it supports different edn serializations itself). We have tested the functionality so far on Linux, but would be curious to learn whether it works on other platforms as well (e.g. Mac, Windows, Android, iOS). The shared library is intended to be the default way to export our API to non-JVM/JS runtimes in the future, so help (bindings for other languages, stress testing, bug reporting, etc.) with it is generally appreciated. Also, as always, you can speed things up by supporting us on https://opencollective.com/datahike or https://github.com/sponsors/replikativ.

@whilo note that while a tool can dump EDN, it's still useful to have it implemented as a pod as this improved the UX in babashka. See e.g. how datalevin can be used as a pod. There is no additional install necessary of datalevin, this is automatic when you execute this code: https://github.com/babashka/pod-registry/blob/master/examples/datalevin.clj The pod and the CLI can be (and usually are) the same binary. E.g. clj-kondo also is able to run as a pod. But of course this work can be done later. Great progress :)

@borkdude I think it would be better to just have library (file) of sci code that wraps the CLI, it would be equivalent to sending messages back and forth with a pod, as Datahike is in effect stateless between RPCs.

A pod is just started once during the lifetime of a program, shelling out would start datahike for every query

Also it has to be installed separately

Starting Datahike everytime is not necessarily a problem, unless you need warm caches. The startup time itself is a few miliseconds, not much more than the serialization overhead, I guess.

I am happy to support pod development and care about babashka/libsci support, but I cannot prioritize it right now.

I think what would be nicer though is to compile Datahike directly into libsci without the pod middleware.

sure

libsci and babashka are two totally different things

Ok, I thought babashka uses libsci.

it uses SCI, but it doesn't talk to external shared libs. libsci is just a demo of how you can make a shared library with sci

Right, sorry, my bad. I meant sci.

Marshalling data at the interface into edn/transit when the database is implemented in the same language and has the same memory model as your interpreter is suboptimal in my understanding.

datalevin has also done that (they have a REPL which runs SCI code for example) but exposing it to babashka via the pod interface doesn't force people to write scripts in a different environment with a different set of libs. I understand it's not a priority, that's ok. This work has already done by datalevin and can be more or less copied. The pod can also be developed externally from datahike proper

Datalevin is not fully implemented in Clojure, but rather a stateful blackbox with a different memory model.

Anyway, this is just my intuition as a language designer/compiler builder. I need to look into the details.

> Marshalling data at the interface into edn/transit when the database is implemented in the same language and has the same memory model as your interpreter is suboptimal in my understanding. Sure, it's a trade-off to still get some features which you otherwise don't and this is still better than wrapping a CLI with repeated startup. Pod calls are sub-millisecond whereas startup of a graal binary is 10-20 ms

It's sweet that datahike is pure clojure, but pods are language agnostic, it could be implement in go or haskell, it doesn't matter If you want to remove this intermediate layer, you would have to build datahike as a built-in library into babashka

Would this be a problem?

not a problem if it compiles with graal, but it's a matter of 1) how much extra strain does it put on CI, 2) how much extra binary size, 3) how often do people ask for this, etc it can be done as an optional build flag for sure https://github.com/babashka/babashka/blob/master/doc/build.md#feature-flags

I see, you already provide support for DataScript there, nice! I guess it would be similar.

The nice thing for instance would be that you could also get lazy index iterators over the Datoms in this case.

It is fairly annoying to code that up with RPC.

yes

I noticed that datahike DataScript was quite intensive to compile in graal. Sometimes dynamic requires / resolve etc can make compilation a lot slower and the binaries bigger. Does datahike has a dependency on datascript or does it use its own copy of it?

It is a fork, so no dependency.

But compilation is somewhat heavy and not optimized yet.

@whilo it's worth trying this out before you load any other namespaces to see if you have any of these dynamic things: https://github.com/babashka/babashka/blob/68a6e2451624be3640b43c6c2f4ad915b9ba423f/src/aaaa_this_has_to_be_first/because_patches.clj#L6-L30

Thank you!

Did you derive a patch for DataScript from that?

No, I didn't, but I use it to scan stuff if I want to include a dependency in bb and the compilation takes much longer than necessary (and the binary grows from 80 to 110mb)

and occasionally I have to monkey-patch a library, like clojure.pprint which uses find-var which has a similar problem