xtdb 2022-08-12 | Slack Archive

Athan09:08:06

Measuring Transactions Throughput with Datomic in-memory database and Datalevin storage engine Hi, I thought you might find this interesting. First https://clojurians.slack.com/archives/C03RZMDSH/p1660170818216469?thread_ts=1660170818216469&cid=C03RZMDSH below to see the experiment and the results. 1. It's important to explain quickly why I think there is such bad performance with Datomic in-memory client and I expect analogous results using https://github.com/tonsky/datascript. It's because of the data structures. To maximize the speed of writing in memory one has to use different memory management and data structures similar to those used in pyarrow, numpy. So, how about building an in-memory Datalog query engine on top of pyarrow ? 2. It was relatively easy to deploy, configure and test transaction throughput of a key-value storage engine (LMDB) of https://github.com/juji-io/datalevin. I would expect Datomic transactor on AWS DynamoDB Local or https://docs.xtdb.com/storage/rocksdb/ to have similar performance.

;; 5.2 sec for 23 cols x 10000 rows
;; 3.2MB books.csv file
;; Elapsed time: 5.2 secs
;; datoms inserted: 229,956
;; transactions throughput: 229,956/5.2 datoms/sec = 44222 datoms/sec

I have met similar problems in the past when I was testing the writing performance of redis KV storage engine. All these KV engines (redis, dynamodb, lmdb) are very good on point queries but they perform really bad when you want to write (import) a big volume of data. You may argue that writing performance is not critical for a transactional (OLTP) DBMS but it becomes super important when you want to import your data from another system/project, or you want to integrate a big volume of data from other sources, or you want to do analytics without adding another storage engine. In fact what we are discussing here is the price you pay for having a flexible universal data model based on EAV/RDF triplets. Which is a similar case when you try to construct a relational, tuple based data model on top of a KV storage engine or object like memory structure (Python/Clojure). The physical layout must be appropriate for such data model and the best candidate I found from my personal research and experiments is to use a columnar layout. Why not adding Datalog query engine on top of a columnar database engine based on LSM tree data structures, such as Clickhouse and Singlestore(MemSQL) ?

🙌 1

refset11:08:30

Hey @U03ET6PDHCK, the team here is also fully convinced by columnar memory layouts for all the reasons stated, and is currently betting heavily on a new execution engine than runs atop Apache Arrow (by the creator of Pandas). You can read more here: https://xtdb.com/blog/dev-diary-may-22/#:~:text=We%E2%80%99ve%20been%20listening.-,Behind%20The%20Scenes,-%E2%80%A6%E2%80%8Bbut%20we%20haven%E2%80%99t

🙌 1

refset11:08:05

You may also be interested to read about https://zed.brimdata.io/docs/formats/#3-the-data-model-and-formats which is centred around a similar columnar + schemaless intersection as XT (although I think Arrow already has all the necessary features and momentum, for our purposes)

Athan11:08:49

Oh boy, competition in the IT domain really drives things fast these days 🙂 Thanks for sharing these @U899JBRPF, I will find time to explore further your references

😄 1

refset11:08:40

Np, always happy to chat about these things!

Ferdinand Beyer11:08:26

I know that xt/db is not thread safe. I interpret this as: Don’t obtain a db value and access it across threads. However, I assume I can safely create independent db values in separate threads from the same node, right? Asking because I see regular segmentation faults from a background thread accessing my XTDB with both RocksDB and LMDB backends for all datastores. I attribute those to some unstable builds for Apple (M1) chips currently, just want to make sure I am using XTDB correctly.

refset11:08:27

> I know that xt/db is not thread safe. I interpret this as: Don’t obtain a db value and access it across threads. ✅ > However, I assume I can safely create independent db values in separate threads from the same node, right? Yep that should be totally fine. Do you have stack traces or log files for those segfaults you can share? I'd be happy to take a look

Ferdinand Beyer11:08:20

I do, would like me to send them to you? An excerpt containing the last Java traces is this:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 15526  org.lwjgl.util.lmdb.LMDB.nmdb_txn_begin(JJIJ)I (0 bytes) @ 0x000000011a8229d0 [0x000000011a822940+0x0000000000000090]
J 15525 c1 org.lwjgl.util.lmdb.LMDB.mdb_txn_begin(JJILorg/lwjgl/PointerBuffer;)I (30 bytes) @ 0x0000000112a9b334 [0x0000000112a9afc0+0x0000000000000374]
J 14755 c1 xtdb.lmdb$new_transaction.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (221 bytes) @ 0x000000011436dab0 [0x000000011436cf00+0x0000000000000bb0]
j  xtdb.lmdb$new_transaction.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
J 20284 c2 xtdb.kv.index_store.KvIndexStore.open_index_snapshot()Ljava/lang/Object; (83 bytes) @ 0x000000011ac1ec30 [0x000000011ac1eac0+0x0000000000000170]
J 16795 c1 xtdb.query.QueryEngine.db(Ljava/lang/Object;)Ljava/lang/Object; (513 bytes) @ 0x0000000113f44aa4 [0x0000000113f42fc0+0x0000000000001ae4]
J 16794 c1 xtdb.node.XtdbNode.db(Ljava/lang/Object;)Ljava/lang/Object; (159 bytes) @ 0x0000000113d4f970 [0x0000000113d4e340+0x0000000000001630]
J 16793 c1 xtdb.node.XtdbNode.db()Ljava/lang/Object; (57 bytes) @ 0x000000011433ffec [0x000000011433fb40+0x00000000000004ac]

Ferdinand Beyer11:08:35

(for LMDB, obviously)

refset12:08:18

anything at all you can send would be great

refset12:08:04

e.g. did those frames come with any kind of message above (about memory allocations or whatever)?

Ferdinand Beyer12:08:09

I stupidly and automatically deleted the log file :man-facepalming: Since I see them regularly, I will send the next one I got

😄 1

👍 1

refset12:08:17

feel free to DM if you prefer

✅ 1

Hukka13:08:26

LMDB was multithreading on reads only, but I'm not sure if xtdb handles that, or even lmdb itself

refset13:08:34

XT passes a read_only flag around for reads, and implicitly only has one writer to LMDB (for the index-store, at least). LMDB will throw an error if multiple threads attempt to open write transactions concurrently

2022-08-12

Channels