Fork me on GitHub
#xtdb
<
2020-03-18
>
Vishal Gautam00:03:14

Hello I am trying to get started with crux using rocks-db. Does someone has a sample project that uses these two things

refset10:03:42

Hello @UGMEQUCTV 🙂 there are a few examples of projects using Crux+Rocks in the wild at the end of this post: https://juxt.pro/blog/posts/crux-dev-diary-january-2020.html

refset10:03:29

Are you looking to use Rocks for the event log as well as the KV store (a.k.a. "standalone"? Or would you be able to use JDBC / Kafka for event log storage?

Vishal Gautam15:03:37

Hi I have a starter project. I am trying to start a crux node, but I get an error

(ns crux-example.sample
  (:require [crux.api :as crux]
            [ :as io]
            [crux-example.helpers :refer [easy-ingest]]))

;;
(def node
  (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                          crux.kv.rocksdb/kv-store]
                    :crux.kv/db-dir "data/db"}))

Vishal Gautam15:03:45

This is the error message I get

Syntax error (ExceptionInfo) compiling at (form-init455416676044078120.clj:8:3).
Spec assertion failed
[crux.standalone/topology crux.kv.rocksdb/kv-store] - failed: (fn [id] (and (or (string? id) (keyword? id) (symbol? id)) (namespace (symbol id))))

Vishal Gautam15:03:25

@U899JBRPF thank you for your help 🙂

Vishal Gautam15:03:02

Never mind, I manage to solve the issue 🙂

Vishal Gautam15:03:18

Please disregard the above post

refset16:03:40

🙂 awesome, and for the record: I'm always happy to be a rubber duck!

Jacob O'Bryant20:03:25

Does crux have a limit on string length like Datomic does (4096 characters)? Would storing, say, the contents of a bunch of blog articles in Crux be ok?

refset22:03:31

Hey, good question. I don't believe there are any such limits made explicit in the Crux code, so it's a case of looking through the Kafka / RocksDB / LMDB docs and config. For instance, Kafka by default is optimised for messages <1MB: > Out of the box, the Kafka brokers can handle messages up to 1MB (in practice, a little bit less than 1MB) with the default configuration settings, though Kafka is optimized for small messages of about 1K in size. (https://ibmstreams.github.io/streamsx.kafka/docs/user/ConsumingBigMessages/)

refset22:03:27

Rocks is apparently happy with 3GB though 🙂 > In general, RocksDB is not designed for large keys. The maximum recommended sizes for key and value are 8MB and 3GB respectively

refset22:03:17

The way Crux works means that large things get hashed automatically and so all the joins and lookups happen fast regardless of the size of the data stored

refset22:03:10

Therefore, blog-length large strings should be fine!

refset22:03:40

Strings longer than 128 bytes have a slightly different code path, but I don't think you would notice any performance impact: https://github.com/juxt/crux/blob/master/crux-core/src/crux/codec.clj#L79 https://github.com/juxt/crux/blob/master/crux-core/src/crux/codec.clj#L204

Jacob O'Bryant22:03:26

very informative, thanks!

🙂 4
Jacob O'Bryant22:03:24

That was a significant pain point for me with Datomic in one of my projects

ordnungswidrig08:03:44

Regarding the hashing what is Crux using? SHA-3 or something?

refset10:03:02

@U054UD60U right now Crux uses SHA-1 (160 bits), so the same as git, but we are open to making this pluggable at some point. It can probably be done without too much change although it's not a priority for us currently https://github.com/juxt/crux/blob/master/crux-core/src/crux/hash.clj

refset10:03:39

The document store in Crux is intentionally very similar to git's object store.