Fork me on GitHub
#crux
<
2020-05-09
>
fugbix08:05:17

Greetings everyone. I am experimenting with Crux and Kafka, trying to get a feel for what a production system would sound like. Could anyone kindly tell me where to find juxt/kafka-connect-crux:20.04-1.8.2-alpha (as instructed https://opencrux.com/docs#confluent-quickstart): this specific version (20.04-1.8.2-alpha). The latest available version on Confluent Hub is 19.12-1.6.1-alpha .

refset08:05:08

Hi @U5URYMRV2 we've not released the 1.8.2 connector to Confluent Hub (it's a quite a manual process) but I can create a build or you can build it yourself from the repo

fugbix08:05:03

Cheers @U899JBRPF. I’ll try and build it from the repo and install it manually on my confluent local

refset08:05:39

Cool, happy to help, and if you're still lost in a couple of hours I can create a release for you πŸ™‚

refset08:05:03

I've made a note to publish to Confluent Hub more regularly!

fugbix08:05:58

thanks Jeremy

fugbix11:05:04

Ok πŸ˜¬β€” so lein package keep trying to pull 20.04-1.8.3-alpha-SNAPSHOT artifacts from clojars. I am on the 20.04-1.8.2-alpha tag (but it’s the same with master .

refset11:05:48

right, so you can either roll back the repo commits to the 1.8.2 release commit and try again, or you can lein sub install from the root and work with the SNAPSHOT versions in your app. I'd recommend the first approach

dvingo13:05:50

hi πŸ™‚ I'm trying to setup https://github.com/crux-labs/crux-console pointing at my local crux node. It looks like I have to first start an http-server for my crux node, so I've added the dep:

juxt/crux-http-server               {:mvn/version "20.04-1.8.2-alpha"}
and have this config:
(defn rocks-config [data-dir]
  {:crux.node/topology                    '[crux.standalone/topology
                                            crux.kv.rocksdb/kv-store
                                            crux.http-server/module]
   :crux.http-server 8099
   :crux.kv/db-dir                        (str (io/file data-dir "db"))
   :crux.standalone/event-log-dir         (str (io/file data-dir "eventlog"))
   :crux.standalone/event-log-kv-store    'crux.kv.rocksdb/kv
   :crux.standalone/event-log-sync?       true
   :crux.kv/sync?                         true})
but when I start the node I get:
Execution error at crux.topology/resolve-id (topology.clj:15).
No such var: api/open-q
anyone have an idea of what's wrong here?

refset14:05:56

I don't see what would obviously be wrong with open-q not being found. But still thinking about it. This though should be more obvious if you can share your curl command

dvingo14:05:19

this is happening when starting the node

dvingo14:05:46

(ns space.matterandvoid.server.crux-node
  (:require
    [crux.api :as crux]
    [ :as io]
    [mount.core :refer [defstate]])
  (:import [crux.api ICruxAPI]))

(defn rocks-config [data-dir]
  {:crux.node/topology                    '[crux.standalone/topology
                                            crux.kv.rocksdb/kv-store
                                            crux.http-server/module]
   :crux.http-server 8099
   :crux.kv/db-dir                        (str (io/file data-dir "db"))
   :crux.standalone/event-log-dir         (str (io/file data-dir "eventlog"))
   :crux.standalone/event-log-kv-store    'crux.kv.rocksdb/kv
   :crux.standalone/event-log-sync?       true
   :crux.kv/sync?                         true})

(defn start-crux-node ^ICruxAPI [storage-dir]
  (crux/start-node (rocks-config storage-dir)))

(comment (start-crux-node "crux-store"))

 ; repl output =>
(start-crux-node "crux-store")
Syntax error compiling at (crux/http_server.clj:189:16).
No such var: api/open-q

dvingo14:05:22

When I start the repl and inspect the crux.api ns - i see that there's no open-q var in there

dvingo14:05:49

these are my deps:

juxt/crux-core                      {:mvn/version "20.04-1.8.1-alpha"}
juxt/crux-rocksdb                   {:mvn/version "20.04-1.8.1-alpha"}
juxt/crux-metrics                   {:mvn/version "20.04-1.8.2-alpha"}
juxt/crux-http-server               {:mvn/version "20.04-1.8.2-alpha"}

dvingo14:05:39

ha, those 1.8.1 should be 1.8.2 woooops πŸ™‚

dvingo14:05:30

so the server is working now:

curl 
{:crux.index/index-version 6, :crux.doc-log/consumer-state nil, :crux.tx-log/consumer-state nil, :crux.kv/kv-store "crux.kv.rocksdb.RocksKv", :crux.kv/estimate-num-keys 39, :crux.kv/size 623497}b
but the connection from the console is not working:

dvingo14:05:11

ah, it's a cors error, I think the docs mention that...

refset15:05:33

Yeah, you need to enable cors in your http server config

refset15:05:23

Although we may have now prevented that from being possible πŸ€”

refset15:05:17

The reason being that the crux-console project is essentially mothballed atm, as we're working on an entirely new UI which is really a content-negotiated PWA that lives entirely on replacement http routes. Not long to go until we're ready to show that off πŸ˜‰

dvingo16:05:55

just saw the handler is private, i tried a hack to use the var, but that didn't work. It's not essential at the moment, just wanted to try it out. Very cool! I'll just wait a bit then πŸ™‚

refset16:05:07

sorry about that, I should have foreseen the issue! Building your own fork of crux-http-server is another option

refset16:05:01

what kind of use-cases for a console do you have in mind? e.g. something to help understand an existing data set? debugging?

dvingo16:05:23

i'm going through the churn phase of the data model for my app so i want to make ad-hoc queries against the data - deleting some data that got in a bad state, updating a doc to put it into the state i want - i'm doing it via some function helpers through the repl at the moment. I thought it may be faster to have a UI to dig through the data instead

dvingo16:05:22

and no problem! - i'm going into all of this with the expectation that there will be some hurdles to jump over

refset19:05:13

cool, glad to hear you've got some kind of repl flow figured out already. The crux-console SPA is pretty nice for certain things, but it's not as handy as a repl πŸ™‚