This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-12-23
Channels
- # adventofcode (92)
- # beginners (12)
- # boot (3)
- # boot-dev (38)
- # cider (9)
- # clara (26)
- # cljs-dev (26)
- # cljsrn (3)
- # clojars (3)
- # clojure (48)
- # clojure-france (6)
- # clojure-greece (2)
- # clojure-spec (3)
- # clojurescript (7)
- # datomic (3)
- # fulcro (6)
- # hoplon (19)
- # jobs-discuss (1)
- # leiningen (4)
- # lumo (9)
- # off-topic (8)
- # onyx (43)
- # planck (10)
- # powderkeg (4)
- # precept (44)
- # re-frame (4)
- # shadow-cljs (4)
- # sql (13)
- # uncomplicate (1)
- # unrepl (3)
When firing up i) onyx ii) with onyx-kafka , iii) on a single machine, onyx.log
spits out this message.
17-12-22 23:39:14 6c4ec751622d INFO [onyx.peer.virtual-peer:17] - Starting Virtual Peer a8fad2b9-f0d9-e015-1c76-62ab461c076d
17-12-22 23:39:14 6c4ec751622d INFO [onyx.log.zookeeper:151] - Stopping ZooKeeper client connection
17-12-22 23:39:16 6c4ec751622d INFO [onyx.log.commands.submit-job:91] - Job ID 76d33ae1-79df-8d07-daac-1d556c3346a0 has been submitted with tenancy ID dev, but received no virtual peers to start its execution.
Tasks each require at least one peer to be started, and may require more if :onyx/n-peers or :onyx/min-peers is set.
If you were expecting your job to start now, either start more virtual peers or choose a different job scheduler, or wait for existing jobs to finish.
Seems like the job was either submitted on a different tenancy id, or the peers haven’t started up fully (possibly because they can’t connect to Aeron - it will warn you about this)
@lucasbradstreet This is my :peer-config
, which is what I use when calling onyx.api/start-peer-group
then onyx.api/start-peers
, then onyx.api/submit-job
.
{:onyx/tenancy-id "dev"
:zookeeper/address "zookeeper:2181"
:onyx.peer/job-scheduler :onyx.job-scheduler/balanced
:onyx.peer/zookeeper-timeout 5000
:onyx.messaging/impl :aeron
:onyx.messaging/bind-addr "0.0.0.0"
:onyx.messaging/external-addr "0.0.0.0"
:onyx.messaging/peer-port 40200
:onyx.messaging.aeron/embedded-driver? true}
Hmm, that looks fine. No, you don’t need to wait, but it may give that warning if the peers haven’t started up or you don’t have enough yet. Did you start enough peers for the job?
Well, that’ll be it. It needs enough for all the tasks as described in the warning
Nope, but the onyx.api/start-peers call needs to start enough to run the job.
I mean virtual-peers not peers, sorry
So a peer-count of 1
is not enough then. B/c for the moment, I’m just firing 1 onyx.api/submit-job
once.
(let [{:keys [zookeeper-url] :as config} (read-config (io/resource config-file-string))
peer-config (:peer-config config)
peer-group (onyx.api/start-peer-group peer-config)
peer-count 1
v-peers (onyx.api/start-peers peer-count peer-group)]
(for [[the-workflow the-lifecycles the-catalog] [[psc/workflow
(psc/lifecycles :kafka)
(psc/catalog zookeeper-url "scanner-command"
peer-count :kafka)]]]
(do (println "the-catalog: " the-catalog)
(let [job {:workflow the-workflow
:catalog the-catalog
:lifecycles the-lifecycles
:task-scheduler :onyx.task-scheduler/balanced}
{:keys [job-id task-ids] :as submitted-job} (onyx.api/submit-job peer-config job)]
submitted-job))))
You only need one submit job, but yes, you need a higher peer count there.
You just need to add up max-peers/n-peers for all the tasks
Ah ha, so if my catalog has 4 “things” (with :onyx/max-peers
) in it, then the peer count is 4.
If one is min-peers 2, one is n-peers 1, and another max-peers 2, it will need 2 + 1 + 1
Hmm, using onyx / onyx-kafka, I’m not seeing any output from my workflow (onyx.logs here: https://pastebin.com/ktQrFDdJ).
Besides setting :onyx.log/config
what other ways are there to troubleshoot / inspect job execution?
If you’re using 0.12 you can try out “onyx.api/job-state. Plays back the log for a given tenancy-id and job-id and returns the current state of the job.”
Looks like it did start though. If you use onyx-http-peer-query you can also query /metrics and find out what it’s up to
Ok yeah, I don’t think onyx.api/job-state
or onyx-http-peer-query
have made it into the cheatsheet yet… Let me take a look at the source.
Http query requires https://github.com/onyx-platform/onyx-peer-http-query
Gotta run - Xmas travel. Good luck
The thing I’d most be interested in is the epoch_Value metrics - those should be increasing over time.
@lucasbradstreet Have a basic workflow happening with onyx and kafka.
Also working for multiple jobs and workflows. So 2 birds with 1 late night in. Many thanks :)
Any idea what the problem was?
@lucasbradstreet After I got i) a correct peer-count, ii) I realized that my :kafka/key-deserializer-fn
had the wrong arity.
I also upgraded to org.onyxplatform/onyx "0.12.0"
and org.onyxplatform/onyx-kafka "0.12.0.0"
. But that probably tangential.
I figured out both errors, while watching onyx.log. So that was the breakthrough, after getting stuck watching console logs for zookeeper, kafka, my app. I think what threw me off were those NoNode
exceptions in zk. I spent a lot of time digging into that, which was a non-issue.
Cool, thanks
I’m using the onyx-seq plugin to read a file from disk. how can I fire a trigger when I’ve reached the end of the seq? I want to do a batch computation over all the data and only emit once, when all the data is processed
figured out a solution: use a punctuation
trigger with a pred checking if (= event-type :job-completed)
Am I correct that these segments cannot be used by downstream task due to https://github.com/onyx-platform/onyx/issues/779?