xtdb 2020-06-07 | Slack Archive

lispyclouds08:06:34

Hello! 👋:skin-tone-3: Im new to crux and trying to set it up for the following scenario: - I have a bunch of services as a part of bigger single service which depend on the data produced by each other - Currently its based on plain postgres but is getting increasingly complex to manage history and shared state - These services synchronize based on RabbitMQ and also use it for durable backpressure - These are mostly clojure but some in kotlin and have a possibility of more diverse tech coming in - Should be easy to be deployed on some scalable managed provider using RDS etc Given these requirements I was thinking using a temporal immutable store would be a better fit and also I always wanted to give crux a try 😄 I tried setting up a standalone crux HTTP node(as multiple non clojure/non jvm things need to talk too) backed by jdbc/postgres, but it always seems to choose the im memory KV store. Also I wanted to know if this is right thought process of setting it up like this without Kafka? Possibly there is a better way of setting up a Crux cluster? Any help is much appreciated!

👋 3

lispyclouds08:06:52

my postgres+http config:

(crux/start-node {:crux.node/topology          '[crux.jdbc/topology crux.http-server/module]
                  :crux.jdbc/dbtype            "postgresql"
                  :crux.jdbc/dbname            db-name
                  :crux.jdbc/host              host
                  :crux.jdbc/port              port
                  :crux.jdbc/user              user
                  :crux.jdbc/password          password
                  :crux.http-server/port       http-port})

lispyclouds08:06:12

http 
HTTP/1.1 200 OK
Content-Type: application/edn
Date: Sun, 07 Jun 2020 08:50:59 GMT
Server: Jetty(9.4.22.v20191022)
Transfer-Encoding: chunked

{:crux.index/index-version 6, :crux.doc-log/consumer-state nil, :crux.tx-log/consumer-state nil, :crux.kv/kv-store "crux.kv.memdb.MemKv", :crux.kv/estimate-num-keys 1, :crux.kv/size 0}

lispyclouds09:06:11

this is the diagram if it helps

dvingo15:06:50

hi! that sounds like a nice setup. I'm curious about using crux with non-clojure langs as well. I'm not sure if this is a typo but the topology vector you posted above is quoted, so the vars would not get eval'ed

dvingo15:06:02

oops nvm

dvingo15:06:23

you most likely need to include the kv store in the topology: https://opencrux.com/docs#config-properties

lispyclouds16:06:49

hey, do i need to include it? as far as i understood the doc, i need to include the modules i want to use in the vector. the kv-store is specified by the other things i include?

From version 20.01-1.7.0-alpha-SNAPSHOT the kv-store should be specified by including an extra module in the node’s topology vector. For example a rocksdb backend looks like {:crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]}

lispyclouds16:06:53

ah! could it be possible that the one from http server is overriding the one from the db?

dvingo17:06:56

i'm pretty sure the http module is orthogonal to specifying the transaction logs and document store

dvingo18:06:46

hmm, so it seems like specifying the jdbc topology should have txs and docs going to the sql db (https://juxt.pro/blog/posts/crux-jdbc.html) . Maybe you can debug first without the http interface to see if jdbc is working as expected

lispyclouds18:06:19

yes i had tried to run just the postgres backed node from the REPL and works as expected. just that when i add the http module it doesnt seem to work?

lispyclouds20:06:52

seems to work now! i just got confused by the status of :crux.kv/kv-store "crux.kv.memdb.MemKv" i guess Set each component up properly and it works! 😄 Thanks for the help

🙂 4

dvingo20:06:01

sweet! glad the battle is over ha - you mind posting your working node config? I've only been running a solo rocksdb setup so far and I'd like to try the jbdc setup

lispyclouds20:06:45

sure thing its more or less the same thing i started with, just was misreading the messages it seems. here is the component im using: https://github.com/bob-cd/bob/blob/queue/db/src/db/system.clj

🙏 3

lispyclouds20:06:38

starts up the HTTP server on 7778 by default and you check out the default postgres connection values

dvingo21:06:35

awesome, thank you!

➕ 3

2020-06-07

Channels