This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-12-20
Channels
- # admin-announcements (1)
- # bangalore-clj (4)
- # beginners (176)
- # boot (38)
- # cider (9)
- # clara (1)
- # clojars (9)
- # clojure (290)
- # clojure-belgium (25)
- # clojure-berlin (2)
- # clojure-dusseldorf (10)
- # clojure-italy (1)
- # clojure-russia (141)
- # clojure-sg (1)
- # clojure-spec (40)
- # clojure-uk (38)
- # clojurebridge (19)
- # clojurescript (148)
- # code-reviews (37)
- # community-development (7)
- # cursive (27)
- # datomic (71)
- # editors-rus (3)
- # events (1)
- # heroku (1)
- # hoplon (16)
- # jobs (5)
- # lambdaisland (3)
- # lein-figwheel (211)
- # luminus (3)
- # off-topic (52)
- # om (18)
- # onyx (49)
- # overtone (3)
- # pedestal (48)
- # protorepl (7)
- # rdf (2)
- # re-frame (61)
- # reagent (3)
- # timbre (2)
- # untangled (69)
It’s really not that bad. Docker is acceptably quick.
Got another error on cheshire: :kafka/deserializer-fn :onyx.tasks.kafka/deserialize-message-json
I mean, when I use that deserializer. cheshire is on onyx-kafka :dev dependencies. Am I not supposed to use that deserializer?
Did you require the namespace that the function is in? (:require [onyx.tasks.kafka])
(I assume the error was that the symbol couldn’t be resolved.)
At first I didn't but then I did and it still didn't work until I added cheshire to my own dependencies
Does it matter where I require oinyx.tasks.kafka? I required it in the namespace where I define the job map
@yonatanel yeah, you might be submitting your job from a namespace that is different to the one that the peers boot up
It needs to be required on the classpath of the JVM that is running the actual peer.
Sure, I required onyx.plugin.kafka in my core namespace instead of dev namespace and I was using the REPL. Probably that was the problem. Thanks!
onyx-kafka reader task has :kafka/offset-reset values of :earliest or :latest according to its readme, but the user guide says "and you’ve also set :kafka/offset-reset to :largest".
I'm getting a rocksdb error that the tmp file is not found. I didn't see examples configuring rocksdb so I wonder if I should set it up somehow or maybe my task name confuses it: org.rocksdb.RocksDBException: IO error: /tmp/rocksdb_filter/e268acd9-2230-48b6-b79c-2dfd74c86bed_:my.task/process-commands: No such file or directory
Yep, I changed the task names to be non-qualified keywords like :process-commands instead of :my.ns/process-commands and now it works. Is it something you want to fix? I like using qualified keywords :(
A fix for that would be very welcome.
I believe we’d just have to ensure that task-id
was something acceptable as a file path https://github.com/onyx-platform/onyx/blob/75a48ce99f7d9cc829ea27922d987617abc7c5ac/src/onyx/state/filter/rocksdb.clj#L61
@gardnervickers Don't you think task-id should be any keyword, and implement escaping for file name?
I believe it’s just used for keying Rocksdb buckets to each leaf aggregation in a job, so it’s probably suitable to just drop anything that would make it an invalid path, or even take the hash of the key and use that.
The former would be preferable incase you ever wanted to go inspect the buckets yourself
I vote for hashing task-id 😄
rocksDB db file is deleted on close filter
When using embedded zookeeper and kafka I'm getting this error in onyx.log. Any ideas as to why? Configuration is pasted after the error:
clojure.lang.ExceptionInfo: Could not locate any Kafka brokers to connect to.
recoverable?: true
zk-addr: "127.0.0.1:2188"
clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: Could not locate any Kafka brokers to connect to.
job-id: #uuid "31d71252-1104-4718-b337-e530cfa138db"
metadata: {:job-id #uuid "31d71252-1104-4718-b337-e530cfa138db", :job-hash "3c3ec58f7e276b81cbbea96a8992741bb92a9409a83e4e8424276e275e147"}
peer-id: #uuid "d66319a4-8624-40a3-b3e9-d15867c55ca3"
recoverable?: true
task-name: :read-commands
zk-addr: "127.0.0.1:2188"
(def tenancy-id #uuid"58582e9e-673e-4646-a07c-9fdd26d3d2eb")
(def peer-config
{:zookeeper/address "127.0.0.1:2188"
:onyx/tenancy-id tenancy-id
:onyx.peer/job-scheduler :onyx.job-scheduler/balanced
:onyx.messaging/impl :aeron
:onyx.messaging/peer-port 40200
:onyx.messaging/bind-addr "localhost"})
;; And later in component system map:
:onyx-env
(onyx-env
{
:zookeeper/server? true
:zookeeper/address "127.0.0.1"
:zookeeper.server/port 2188
:onyx.bookkeeper/server? true
:onyx.bookkeeper/delete-server-data? true
:onyx.bookkeeper/local-quorum? true
:onyx.bookkeeper/local-quorum-ports [3196 3197 3198]
:onyx/tenancy-id tenancy-id})
:embedded-kafka
(component/using
(embedded-kafka
{:hostname "127.0.0.1"
:port 90922
:broker-id 0
:num-partitions 1
:zookeeper-addr "127.0.0.1:2188"})
[:onyx-env])
:onyx-peer-group
(component/using
(onyx-peer-group peer-config)
[:onyx-env])
:onyx-virtual-peers
(component/using
(onyx-virtual-peers 5)
[:onyx-peer-group])
:onyx-job
(component/using
(onyx-job peer-config (em-job))
[:onyx-virtual-peers :embedded-kafka])
Can you connect to it from some other tool like kafkacat
?
I saw an error that the port was illegal, so I tried 9094. I still can't connect to the embedded kafka. I tried using kafka's cli
Yea sorry, I don’t know much about embedded Kafka, it’s definitely better running in a container we’ve found
Right now I can't use docker. Do you know how I make the real zookeeper installed on my machine to forget its onyx state?
For some reason I changed the job map, restarted, and it still runs the old one even though I don't submit an ID.
Onyx namespaces itself by tenancy id, so you can either delete that in ZK or change your tenancy id
Yea if you don’t cancel the job it’ll keep trying to run
Or use the with-test-env macro which clears out in-mem ZK for you
Just by restarting the in-mem ZK component
I'll try in-memory zk for onyx and real kafka for plugin, and using my own env component to do the shutdown.
Hmm I’m actually not sure if in-mem ZK allows external connections.
If you can’t use docker then maybe try and replicate it’s functionality by having a set of setup/teardown scripts for local ZK and Kafka?
Also, the kafka plugin uses zookeeper to find brokers, and my real kafka uses real zk
Yes, you need to use the same ZK
I would just add a hook to kill the job
All good now. Thanks @gardnervickers
You can supply it with a log4j file
@yonatanel It buckets the segment into the nil
group.