This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-12
Channels
- # aleph (22)
- # aws (7)
- # babashka (17)
- # beginners (69)
- # chlorine-clover (9)
- # cider (2)
- # clj-kondo (3)
- # cljdoc (30)
- # clojure (113)
- # clojure-dev (30)
- # clojure-europe (11)
- # clojure-italy (2)
- # clojure-nl (16)
- # clojure-spec (1)
- # clojure-sweden (3)
- # clojure-uk (17)
- # clojurescript (77)
- # cryogen (12)
- # data-science (5)
- # datomic (27)
- # duct (2)
- # emacs (37)
- # fulcro (24)
- # graphql (2)
- # kaocha (1)
- # lambdaisland (27)
- # leiningen (4)
- # off-topic (15)
- # onyx (1)
- # other-lisps (3)
- # re-frame (94)
- # reagent (2)
- # reitit (20)
- # ring (1)
- # shadow-cljs (66)
- # spacemacs (5)
- # sql (59)
- # tools-deps (140)
- # vim (1)
- # xtdb (17)
I tried crux with a hosted confluent kafka but I’m a little lost regarding the settings. What’s the recommended KV-store? I had the experience that sometimes queries would not return any results although the transactions where published successfully. I’m pretty sure I’m missing something fundamental here. Is the kv-store required to be persistent? I used memdb for simplicity.
hey @U054UD60U 🙂
re 'recommended KV store' - it's possible to persist the memdb, although we'd recommend other stores (e.g. RocksDB, LMDB) in production usage - are you able to paste your node config?
re missing transactions - once you've submitted the transaction it's then indexed asynchronously. to ensure your node's up-to-date with the transaction you've submitted, you can call sync
using the transaction-time returned - e.g. (sync node (:crux.tx/tx-time submitted-tx) timeout)
I see! Maybe that was the issue.
(the latest documentation, showing await-tx
, unfortunately isn't yet released - 1.7.0 will hopefully be coming in the next week or so 🙂 )
(crux/start-node {:crux.node/topology 'crux.kafka/topology
:crux.node/kv-store 'crux.kv.memdb/kv
:crux.kafka/replication-factor 3
:crux.kafka/kafka-properties-file "kafka.properties"})
For a freshly started node, how can I away for the latests tx to be processed? sync
with current time?
or, yes, if you know which transaction you want to sync to, you can provide that tx-time
(if you provide the current time, it'll wait for a transaction at that time, which likely doesn't exist)
about the persistence - memdb will persist if you supply a :crux.kv/db-dir
property in that map, although I wouldn't recommend its use in production. RocksDB is also perfectly suitable for local dev, we use it a lot here
I see. Trying this now.
Trying to put
~3,000,000 docs into Crux, where Kafka is running on Docker hosted on a local machine.
So it’s not the speediest thing in the world. But any examples of how you’re doing something like this? Thanks
Hey @U3SG7RX7A one place to look would be the crux-bench code, e.g. https://github.com/juxt/crux/blob/master/crux-bench/src/crux/bench/ts_devices.clj#L59