This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-01-10
Channels
- # announcements (1)
- # babashka (9)
- # beginners (372)
- # calva (3)
- # cider (9)
- # cljsrn (71)
- # clojure (57)
- # clojure-australia (2)
- # clojure-europe (26)
- # clojure-france (11)
- # clojure-nl (11)
- # clojurescript (5)
- # core-typed (1)
- # fulcro (33)
- # meander (3)
- # off-topic (1)
- # pathom (3)
- # re-frame (4)
- # reagent (6)
- # sci (1)
- # shadow-cljs (34)
- # xtdb (14)
Hi, I want to use crux in production on a kafka cluster. Is there a best practice to configure the topics? Do I need infinite retention for all topics? I can't find any wiki page for this.
Hi 🙂 Crux automatically creates its own topics with all the appropriate configs - it would definitely be best to let Crux do this as there are (currently) no checks to confirm that any manually configured topics are valid. However, if for some reason you do need to set up your topics manually you can see how its all managed in this ns: https://github.com/juxt/crux/blob/master/crux-kafka/src/crux/kafka.clj (the key items being: infinite retention on both, partitions=1 for the tx log, and cleanup.policy=compact for the doc log)
Thanks for the info. I tried configuring it manually and it threw exceptions until I configured the number of partitions correctly. At first I had set the transaction log to compact as well, which threw exceptions on write operations. I tried starting crux as a client on confluent kafka and it didn't create it's topics. Thats why I created them manually
Interesting, I can think why it wouldn't have created the topics automatically :thinking_face: can you tell me which version that happened with please?
Were you following these steps? https://opencrux.com/blog/crux-confluent-cloud.html (or the ones on the http://juxt.pro site?)
I adapted the older version of the tutorial from the juxt blog. I used the current version of crux from clojars. It's very possible that I did something wrong though, probably not an issue with the code.
Is there a way to enable compaction on a topic that has already been created? The retention policy is set to infinite.
alternatively, is there a way to dump and restore the complete crux database?
I'm fairly sure you can change the compaction setting easily enough for an existing topic - do you have access to a UI?
> alternatively, is there a way to dump and restore the complete crux database? Crux doesn't provide any specific utility for this right now. The best option is to use a regular Kafka-compatible backup tool (e.g. https://stackoverflow.com/questions/47791039/backup-restore-kafka-and-zookeeper)
If I want to run crux in prod using sqlite3 as my backing store (kafka overhead is a bit much for me, all that stateful stuff all over), can I tell crux to use the same database for documents and transactions? I'm worried about backing up the document store then the transaction log in different processes and as such possibly making it impossible to restore from a given backup (race conditions)
In general I'd avoid recommending sqlite3 in prod for Crux - you may as well use a single Rocks instance for docs+tx instead (+ a second Rocks instance for the index), as it will give higher throughput. However, production probably should be using something with a distributed backend anyway, for high-availability & durability ...unless you can afford to risk having unhappy customers! You'll probably have a better time with a single Postgres for docs+tx 🙂