datalevin 2023-07-12 | Slack Archive

Namit Shah08:07:07

Does datalevin have a restriction of 2^20 just like in datomic or is it like xtdb where in each entity can have as many attributes as it wants.

Huahai14:07:19

2^32

Huahai19:07:44

basically, 32 bits integer id range

Kein16:07:36

[1] What are the best practices to scale dtlv after we exceed the total data size limits? [2] If I’d like to store relational/tuple/record data on the eav model for its elasticity, what are the performance gap when compared with relational DB? [3] In dtlv readme: The total data size of a Datalevin database has the same limit as LMDB’s, e.g. 128TB on a modern 64-bit machine that implements 48-bit address spaces. But it seemed to refer to the virtual memory of an OS. What is the actual size limit if I have say a 30GB SSD + 8GB memory server? Thanks.

Huahai14:07:42

1. Sharding the data to avoid exceeding?

✅ 2

Huahai14:07:13

2. Currently, large gap. Thats the issue we will solve for the query engine rewrite. Our plan is to bring the query speed to be on par with RDBMS. You can read about the plan and WIP in the query branch.

Huahai14:07:00

3.less than 30gb? We use LMDB, which use MMAP, so the limitation on data size is the same as the OS virtual memory size, if the disk size is unlimited, whichever is met first.

✅ 2

Kein02:07:30

For 2, do you currently have the results of such comparison?

Huahai02:07:26

Not formally

Kein02:07:39

That’s fine. What would the informal comparison you would subjectively conclude on read and write performance?

Huahai02:07:37

Really depends on data size and query. Small datasets wouldn’t be too different. Large datasets and complex queries, will be problematic.

Kein02:07:29

As you see, how does the existing products facing large datasets handle this issue?

Huahai02:07:08

Don’t write complex queries, use pull

Huahai02:07:31

Same thing people do with datomic

Huahai02:07:48

We do reorder query clauses automatically, so it’s better than datomic in this regard

👀 4

Kein02:07:53

I’ll keep considering. Thanks for reply.

2023-07-12

Channels