Fork me on GitHub
#xtdb
<
2021-01-26
>
ordnungswidrig13:01:51

Is there a way to run crux from the docker build connecting to kafka? I tried to come up with a crux.edn file yesterday and I failed miserably 😞

refset16:01:35

Do you mean with the juxt/crux-in-memory image that we publish to Dockerhub? If so, then no, you will have to craft a crux.edn to create your own image. If you share the crux.edn you started on I might be able to fix it for you 🙂

refset16:01:17

Or just let me know which modules you were looking at and I can cook up an example

refset16:01:04

We also have various configs within the repo that might help, e.g. https://github.com/juxt/crux/blob/master/crux-bench/src/crux/bench.clj#L220-L305

ordnungswidrig16:01:03

I tried something like this:

{:crux.node/topology crux.kafka/topology
 :crux.http-server/server ...
 :crux/tx-log ...
 :crux/document-store ...}
But it fails with a “weird” error “Error locating module”

refset16:01:42

Ah, so we moved away from the pre-built topology system altogether, so you shouldn't really be seeing/using that word anywhere - where were you referring to?

refset16:01:50

If it was the original Confluent blog post, I rewrote the steps when we cross-posted it to https://opencrux.com/blog/crux-confluent-cloud.html but I never did update the original (sorry if that was it 😅 )

ordnungswidrig16:01:26

I found it somewhere on a blog or docs. I tried a lot honestly 😛

ordnungswidrig16:01:45

ah, cool that blog post escaped me.

refset18:01:55

no worries, I know a set of common examples in the docs somewhere could help a lot. I've also been in favour of creating a "kitchen sink" build and Docker image for a while now that could remove a few steps. I'll make sure we give it some thought soon 🙂

ordnungswidrig18:01:30

I'm fine with creating a custom docker image. I was trying to setup a minimal but solid 3 node cluster with zookeeper and kafka orchestrated by cloud-init. And I just wanted to point some crux at it to verify its working as expected. mabye i should have used the benchmarks instead :thinking_face:

👍 3
ordnungswidrig20:01:58

@U899JBRPF thanks I’ve got it working now.

ordnungswidrig20:01:37

One thing though, with this config:

{:crux.kafka/kafka-config {:bootstrap-servers "10.0.0.2:9092,10.0.0.3:9092, 10.0.0.4:9092"
                           :num-partitions 1
                           :replication-factor 2
                           :create-topics? true}

 :crux/tx-log {:crux/module crux.kafka/->tx-log
               :kafka-config :crux.kafka/kafka-config
	       :tx-topic-opts {:topic-name "crux-transaction-log"} }

 :crux/document-store {:crux/module crux.kafka/->document-store
                       :kafka-config :crux.kafka/kafka-config

	               :doc-topic-opts {:topic-name "crux-docs"}
	               :local-document-store {:kv-store {:crux/module crux.rocksdb/->kv-store
	                                                 :db-dir "/var/lib/crux/rocksdb"}}}
The http console reports: :crux.kv/kv-store "crux.mem_kv.MemKv"

ordnungswidrig20:01:05

Shouldn’t that be rocksdb? I see the rocksdb files created in /var/lib/crux/

refset09:01:38

Cool, glad to hear 🙂 I think crux.api/status may be misreporting, as that's what the console uses, I'll double check at this end and then file a bug

Richard22:01:51

Hi. I'm struggling to write the correct syntax for joining two documents together with a crux/q query. Say I had the following in my node: (c/submit-tx node [[:crux.tx/put {:crux.db/id :test.request/id-1, :test.request/foo "Foo"}] [:crux.tx/put {:crux.db/id :test.result/id-1, :test.request/id :test.request/id-1, :test.result/bar "Bar"}]]) I want to query my node to return ["Foo" "Bar"] where :test.request/foo = "Foo" and the :crux.db/id for "Foo", is the same as "Bar"'s :test.request/id Can someone suggest the right form for the query, please? Or how I can better shape my data? Thank you for any suggestions.

jarohen23:01:28

Hey 👋 shape of the data looks fine Something like this?

'{:find [?foo ?bar], 
  :where [[?res :test.request/id ?req] 
          [?req :test.request/foo ?foo]
          [?res :test.result/bar ?bar]]}

Richard07:01:09

Hi jarohen. Thanks for the reply. That's helped a lot in understanding it. I've written a lot more variations on the query and extended the data in the node to understand it more. It's not completely clicked yet, but you've put me back on track. Thank you.

🙏 4