Fork me on GitHub
#onyx
<
2018-05-18
>
eoliphant15:05:35

Hi I have a pretty 101 question lol. I’ve been through the learn-onyx, etc stuff (great material BTW), but one thing that’s not clear is the best, or just a simple way to set stuff for a ‘typical’ deployment. When deploying and uberjar one would conceivably just set everything up then kick off the job. The onyx-app template seems like more of a generic job runner. trying to hack it down to just startup and run the configured job. Is something like this ‘right’?

(defn -main [& args]
  (let [onyx-id (java.util.UUID/randomUUID)
        env-config (assoc (-> "config.edn" io/resource slurp read-string :env-config)
                     :onyx/tenancy-id onyx-id)
        peer-config (assoc (-> "config.edn"
                               io/resource slurp read-string :peer-config) :onyx/tenancy-id onyx-id)
        env (onyx.api/start-env env-config)
        peer-group (onyx.api/start-peer-group peer-config)
        peers (onyx.api/start-peers 5 peer-group)
        job-id (:job-id
                 (onyx.api/submit-job peer-config
                                      (onyx.job/register-job "basic-job" nil)))]
    (print job-id)

    #_(onyx.api/await-job-completion peer-config)
    #_(assoc component :env env :peer-group peer-group
                     :peers peers :onyx-id onyx-id)))

Travis15:05:54

The way it used to work was by using CLI params. So one set of params would essentially just spin up your peers. If you run it with a job name it would it would submit the job to the peers

Travis15:05:18

this main looks like it starts your peers and then immediately runs your job

eoliphant15:05:03

yeah I’m trying to figure out the best approach if say this guy is in a docker container, etc in my case this is a ‘forever’ job, part of a cqrs-style system. So it should just come up and start processing commands

jasonbell16:05:10

We ended up combining the startup of the peers and the job in the main function, not perfect but it handled the job in hand perfectly.

jasonbell16:05:45

Wrapped in a Docker container and there's also a Marathon config template too so you can deploy via DCOS

jasonbell16:05:58

It's a couple of years old now but you'll get the idea.

eoliphant16:05:50

ah sweet, that looks exactly like what I need

jasonbell16:05:35

There is a heartbeat server baked in there too so marathon could check the state of the container.

eoliphant16:05:40

cool, this is going into kube, but will just add something comparable

lucasbradstreet16:05:03

The only gotcha if you submit when you startup the peers is you should use a stable job id to make the job submission idempotent

eoliphant16:05:35

ok that makes sense

lmergen20:05:57

i have a cli tool myself, that has options to start peers, submit jobs, kill jobs and perform gc

lmergen20:05:07

it’s also exposed as a web interface

lmergen20:05:16

works pretty well tbh

lmergen20:05:36

i handle job progress monitoring on a higher level

Travis20:05:51

sounds pretty cool

eoliphant22:05:54

hi, do you normally set the consumer group.id via the consumer opts with onyx-kafka or is it doing something internally ?

michaeldrogalis22:05:25

@eoliphant Onyx is managing the Kafka consumer state on its own. You can set it if you want, but if I recall right, it’s not doing anything.

lucasbradstreet22:05:31

It’s mostly useful if you’re monitoring consumers on the kafka side

eoliphant22:05:49

ah ok, so say separate jobs, that are reading from the same topic, onyx is taking care of offset mgmt, etc

michaeldrogalis22:05:06

Correct. You can set it with :kafka/group-id in the catalog if you like.

michaeldrogalis22:05:12

It defaults to "onyx".