This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-11-29
Channels
- # adventofcode (20)
- # announcements (6)
- # asami (13)
- # babashka (9)
- # beginners (80)
- # calva (53)
- # cider (16)
- # clj-kondo (24)
- # cljs-dev (40)
- # clojure (13)
- # clojure-australia (9)
- # clojure-europe (117)
- # clojure-india (3)
- # clojure-italy (4)
- # clojure-nl (5)
- # clojure-sg (1)
- # clojure-spec (4)
- # clojure-uk (6)
- # clojurescript (6)
- # cursive (41)
- # datalog (5)
- # datomic (11)
- # emacs (9)
- # events (1)
- # fulcro (46)
- # integrant (31)
- # jobs (1)
- # kaocha (1)
- # lein-figwheel (3)
- # lsp (2)
- # meander (3)
- # missionary (4)
- # pathom (6)
- # portal (84)
- # re-frame (3)
- # remote-jobs (1)
- # reveal (2)
- # shadow-cljs (36)
- # tools-build (3)
- # xtdb (17)
anybody happen to have ballpark estimates on xtdb backends' disk footprint? using single node so all components matter. using lmdb at the moment and it seems rather disk hungry, wondering how others compare before i implement
lmdb doesn't compress it's indexes on disk afaik. RocksDB does however, so anecdotelly it is slightly slower but uses less disk-space...
compared with rocks and for my usecase (lots of tiny records with 2-10 attrs each) it looks like rocks uses about an order of magnitude less disk space. very good to know, thanks for the pointer!
update: actually after indexing was all done and settled lmdb appeared to be bigger only by a factor of 2
Hello ppl. I've been playing around xtdb using kafka for transaction log, I have an AWS MSK cluster with 2 brokers, and I can only transact stuff when I override the default producer configuration for idempotence. I have other services using this same cluster, so I'm not sure if this is a misconfiguration on MSK or something wrong with xtdb. It looks like I cant get all acks, as soon as I override the default properties, it works
:properties-map {"acks" "1"
"enable.idempotence" "false"}
Hi @U01UAG3P57B - are you using a Kafke .properties file as well? If so, please can you share it (with secrets redacted)?
@U899JBRPF Im not, the only config other than this is bootstrap-servers
hmm :thinking_face: well I know we've not thoroughly validated the impact of setting "enable.idempotence"
to false
before...so I can't officially recommend it, but it seems strange that your producer isn't working otherwise. Are there any errors shown anywhere?
adding a debug line somewhere in here might be revealing https://github.com/xtdb/xtdb/blob/aa933a6d22b5e5b7b88d494c349307285c1f027f/modules/kafka/src/xtdb/kafka.clj#L175-L180
@U899JBRPF I think I may have narrowed down the issue to: for some reason I'm not getting acks from all brokers, there are 2. The event still gets through even tough the transaction returns a timout error on produce, I know this because I get to see it in tx_event table in Postgres
hmm :thinking_face: does is topic look like it's being replicated across both brokers? I wonder if the replication factor has some impact. Also, and this may well be completely unrelated to your issue, but I believe 3 brokers is a recommended minimum number https://stackoverflow.com/questions/58761164/in-kafka-ha-why-minimum-number-of-brokers-required-are-3-and-not-2
do you see any of the errors outlined in https://towardsdatascience.com/10-configs-to-make-your-kafka-producer-more-resilient-ec6903c63e3f ?