Clojurians Log v2

In order to run performant queries you need to build some kind of index structure. In datalevin, those structures are very cheap to construct from the triples that you feed into it. If your desire is to construct some large clojure data structure and query it once or a few times, you might want to look to a library like redplanetlabs/specter, but datalevin will shine when you want to run many queries against the data, such that the time spent building the indexes and transforming the data into the triple format is paid for by fast access. That said, datalevin does already allow you to store arbitrary clojure edn data inside the values of the db, and the document store mode allows storing edn data and querying it as well. If your need is to allow querying some key using datalog queries but the value to fetch is some larger document or document fragment, maybe that would be appropriate for your usecase?

The analytics use case is a future direction that we hinted in the afterword of the book. It requires more than just handling different data sources, the query language itself needs extension as well. One thing that is weak for this kind of use case, is that JVM is still memory hungry, so it is not competitive against native solutions such as DuckDB, clickhouse, etc. That's why Datalevin currently focuses on OLTP use cases.

Basically, JVM is a good server technology, not the best for interactive analytics.

Thank you for the response, at the same time clojure HPC libs like dl-next seems very performant , also i used datalevin for the data-science calculation, it was also very performant as well (even better than TableCloth-API)

I always aim to be the best. If I were to do this, I would aim to beat duckdb, but I don't think we are ready to do that. At least I don't see a distinct advantage yet.

I knew I can beat PostgreSQL, because I saw a distinct advantage in triple stores. This is not the case with analytics use case.

Of course, I am still interested in advance the analytics capabilities in Datalevin, as you said, it is very performant right now, so it would be an ergonomics win for people already are using Datalevin.

One important thing in Data-science is not just processing of huge amount of data in the fastest time possible, but it is ergonomic of the data-scientist & datalog is much more ergnomocial option that pandas style data researching, or clojure Table-cloth So it can even be less performant to another technology from processing time of view but it is more performant from data-scientist time to do a certain task like joining between different datasets in Pandas/TableCloth is a nightmare!

Agree.

I would welcome feature suggestions to make it more convenient for people to use Datalevin as an analytics platform. E.g. lazy loading of different data sources. I would appreciate concrete proposals.

thank you