This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-10-26
Channels
- # announcements (7)
- # aws (1)
- # babashka (15)
- # beginners (144)
- # calva (8)
- # chlorine-clover (15)
- # clara (4)
- # clojure (65)
- # clojure-europe (131)
- # clojure-france (1)
- # clojure-nl (6)
- # clojure-nlp (3)
- # clojure-spec (10)
- # clojure-uk (52)
- # clojuredesign-podcast (2)
- # clojurescript (28)
- # cryogen (1)
- # datomic (17)
- # events (2)
- # figwheel-main (2)
- # fulcro (8)
- # hugsql (2)
- # jackdaw (4)
- # jobs (1)
- # leiningen (8)
- # lumo (1)
- # malli (4)
- # off-topic (23)
- # parinfer (3)
- # pathom (3)
- # pedestal (5)
- # re-frame (9)
- # reagent (26)
- # reitit (13)
- # reveal (25)
- # shadow-cljs (45)
- # spacemacs (7)
- # sql (7)
- # tools-deps (40)
- # vrac (2)
- # xtdb (22)
Hi, are there any performance implications to using JDBC (Postgres) instead of Kafka as the document store?
Hi 🙂 invariably yes. Kafka, when properly configured, will be faster for accepting writes, especially under high concurrency, and provide more efficient horizontal read scalability. I don't have any specific benchmark numbers to offer for direct comparison but would be happy to help you investigate.
Hi, maybe a stupid question: is there a "real world" demo project using crux, that maximally tries to capture the "idiomatic" way of using crux in a (e.g. web) project?
I'm not sure my project fits with the "idiomatic" way of doing things, but I am building a real world application with crux + rocksdb. It will be open source once the code gets a bit cleaner, but in the meantime I could invite your github user to the repository and you could take a look if you want
Hey @UEJ5FMR6K that'd be great, it's 'tmoerman'
@U052A8RUT Hey we don't have any significant demos in the wild I can point to, but there might be something of interest in the "soak environment" example project: https://github.com/juxt/crux/tree/master/examples/soak/src/crux/soak
Great, thanks
@U052A8RUT invite sent! File of interest: https://github.com/victorb/instant-website/blob/master/src/instant_website/db.clj If you see something funky, please shout 🙂
Thanks guys! I'm working on a project using Fulcro, Pathom and Crux. Unfortunately not open source so I cannot share anything at the moment. Maybe in the future I'll blog about some lessons learned, caveats, etc...
i'm not really looking for a technical answer, more like a "flavour" example
e.g. how are entity attributes named ideally: :my.user/name
:vs :generic/name
cfr mixin-style logic etc..
i realize this is a vague question 😅
I'm curious how the disk space usage of the document-store, index-store and tx-log grow relative to each other? I have s3 setup for the document store and am wondering how long I can get away with a local rocksdb for the other two. I'm attempting to store batches of 100-1000 messages in each document, so the documents will be relatively large. Do all documents go through the transaction log, and thus take up disk space, but eventually get "garbage collected" if they aren't used frequently?
As the other reply from Dominic mentioned, the tx-log only contains references, so would typically be much smaller (unless all your documents and transactions are tiny). The indexes could be as much as 3-4x the size of your documents, but it really depends on the shape of your documents as if the majority of data is nested below the top level then the indexes will also stay very small.
Are you planning to move the tx-log to something with high-availability (e.g. JDBC) for the prod setup?
Got it! I wasn't sure if I need to move to something else yet. I don't think i particularly need high availability. I really just want to store a bunch of event batches as documents on s3 with a bit more queryability just listing the s3 "directories". I'm not too worried about query or insert performance each batch is ~1000 messages and I'll have a few thousand batches per day and I was trying to see how long a rocksdb setup might last with just the disk space on a small ec2 instance
Ah okay, so more to the point, durability & write-loss might be a more significant concern. As in the disk underlying Rocks could fail and you would lose all the transactions since the last backup. You could mitigate it by using EFS as the disk though