Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-03-26

Channels

#xtdb
<
2022-03-26
>
genekim15:03:24

I’d like to use XTDB for a real analysis project on an M1 MBP, and realizing that absence of M1 RocksDB may cause some headaches. Before I switch JVMs for this project, I wanted to try exploring using Xodus one more time. When I use Xodus 1.1.0, I get the following error when starting the JVM REPL: 

clojure.lang.ExceptionInfo: Conflicting data-reader mapping {:url #object[java.net.URL 0x62fad19 "jar:file:/Users/genekim/.m2/repository/juxt/crux-core/20.09-1.12.0-beta/crux-core-20.09-1.12.0-beta.jar!/data_readers.clj"], :conflict crux/id, :mappings {datascript/Datom #'datascript.db/datom-from-reader, datascript/DB #'datascript.db/db-from-reader, crux/id #'xtdb.codec/id-edn-reader, xtdb/id #'xtdb.codec/id-edn-reader, crux/base64 #'xtdb.codec/base64-reader, xtdb/base64 #'xtdb.codec/base64-reader}}
Is there an easy fix for this? (I’m resisting urge to blindly hack away with deps overrides. 🙂 Thank you!

genekim15:03:56

@U11SJ6Q0K I notice you mentioned you got Xodus working — any chance you can share how you did this? Thx! https://clojurians.slack.com/archives/CG3AM2F7V/p1629715955179600?thread_ts=1629713348.178600&amp;cid=CG3AM2F7V

genekim15:03:30

(Okay, I’ve just forked crux-xodus, and am changing deps from crux to xtdb…. REPL is loading now. Let’s see how far I get! 🙂

genekim16:03:11

Holy cow, I got Xodus KV store running on M1 Mac. https://github.com/realgenekim/crux-xodus So here’s a way to run XTDB on M1 without using Rosetta JVM! (I didn’t update the version in the POM or anything, mostly because I don’t know what I’m doing.)

🚀 2
❤️ 1
nice 1
alexdavis18:03:10

Fwiw you can use rocks db on m1, though evict currently doesn’t work properly (should be fixed by the time rocks is offficially updated though) see here for instructions on how to get it to work (its pretty straightforward, but the compile step can take a while, even on an m1 🙂 ) https://github.com/xtdb/xtdb/issues/1518#issuecomment-1051775671

richiardiandrea21:03:48

This is good news for us too - we had some problems with LMDB and definitely folks here use M1 for development...thanks for your work, I'll propose it internally

refset22:03:58

In related news, the RocksJava release with M1-compatibility finally landed maven a day or two ago, so we'll get an official solution out for Mac users very soon 🙂

refset12:03:39

Good news! 1.21.0-beta2 is now available, including native ARM RocksDB support for M1 Mac users and the ability to import tx-times. I appended to the previous beta1 release notes here: https://discuss.xtdb.com/t/ann-1-21-0-beta2s-out/77 /cc @avocade @henrik @genekim @richiardiandrea - anecdotal performance comparisons would be welcome(!)

❤️ 6
🍎 1
🎉 1
👍 1
Steven Deobald17:03:13

Beautiful and precise graphs... also welcome. 😉

genekim17:03:34

Fantastic news, @U899JBRPF! I’ll give it a shot in the next week or two — thanks for the heads up!!

🙌 1
🙇 1
henrik07:03:21

Here's my anecdotal performance comparison: Wroom! This applies to 1.20 with an extra org.rocksdb/rocksdbjni {:mvn/version "6.29.4.1"} dep applied. For beta2, I get, Index version on disk: 18 does not match index version of code: 20

😄 1
☕ 1
refset08:03:38

Cool! > For beta2, I get Index version on disk: 18 does not match index version of code: 20 Wipe the index-store directory and let it rebuild https://docs.xtdb.com/administration/1.20.0/configuring/#_index_rebuilding

👍 1
J20:03:39

Hi guys :), can somebody tell me what is there exactly in the query indices storage?

refset22:03:39

Hey @UHZPYLPU1 - the exact answer is always subject to change, but you can get a feel for the indices by looking here: https://github.com/xtdb/xtdb/blob/2f9f07c51d9fc04f7cf4edfbf7a859189a2b2b14/core/src/xtdb/codec.clj#L37-L56 (and studying the relevant PRs)

J06:03:41

Thanks @U899JBRPF I will read that.

👍 1
J09:03:36

Does the size of query indices is proportional to the document store and tx log?

refset11:03:35

Yes that's a fair way of describing the relationship, however the exact ratios greatly depend on the specific data set (width and size of documents) and workload (number of aborted transactions, lots of identical/similar documents)

👍 1