Thx

https://github.com/juji-io/datalevin/pull/156

I can add the project that uses datalevin via deps.edn as a project under examples/

wdyt?

Please do, thanks

sure

I use Calva and it formats deps.edn automatically. is it ok if I format the whole file? I don't know how you format the file. Did some efort to not break formatting but it's hard

pushed

it’s ok. I use spacemacs default format. the format is not consistent anyway.

ok 🙂 then we can close https://github.com/juji-io/datalevin/issues/98 . I saw github actions is done

merging did not close the issue https://github.com/juji-io/datalevin/issues/98

Thanks 🙏

More specifically, I am not sure we want to change the existing API. We will probably want to add a new one. Any ideas on how this should look? I am getting familiar with the API now

Right, we will add new functions for these functionality. For Datalog query, Datomic has qseq, so we should probably support that. Basically, query results are in a lazy seq. Similarly, for KV, we can have get-range-seq corresponding to get-range that returns a lazy seq, range-filter-seq , and so on.

Basically, implement ISeq protocols with iterative-kv

A naive implementation should work, we don’t need complexity such as chunk and so on.

Keep things simple and straightforward

ok, thanks.

We would normally discourage people to use the seq version of the functions, because it will hold a read transaction, the db will grow very fast if we hold too many of these

because, conceptually, each read transaction keep a copy of the db of its own.

these should only be used if the results set are too large to hold in memory.

I would go to expose the transaction semantics to user

so they can judge when and for how long to keep transactions opened

either sart-transaction / end-transaction or via (with-open txn (start-transaction params) )

Exposing transaction and lazy result sets are separate issues

They should not be considered together

In any case, we will not introduce a transaction object to users.

The goal is to not changing the existing functions

The need to have (with-transaction ..) is to ensure atomicity when there are both read/write.

The need for lazy query is to solve the memory problem.

A casual user should not be aware of the ideas of transactions at all. Simplicity is the goal of this project.

I hope I have made myself understood.

Using (with-transaction ..) instead of (start-transaction …), is to enforce the idea that a transaction should be something short and well contained.

range-seq is now implemented for embedded mode. It lazily loads data into memory in batches, implements Seqable , as long as one does not hold the head, process data as they come in, it should not blow up memory.

The eager version of get-range and range-filter now spill to disk when memory pressure is high. Hopefully, these together could alleviate some of the big data read memory problems.

thanks, I saw the releases and the changes - did not have time to check them out

hopefully will be able to do that soon