Fork me on GitHub
#onyx
<
2017-01-02
>
chrisblom13:01:10

whats a good way to automate updating onyx jobs?

chrisblom13:01:27

i have a job that is updated sometimes, at the moment i simply kill the old job and start the new version of the job manually

michaeldrogalis16:01:00

@chrisblom What tool do you use for deployment orchestration?

chrisblom16:01:44

cloudformation

michaeldrogalis16:01:40

CFN is tough, it’s not one of the more expressive tools. You can maybe have an update path in your CFN template call out to an external bash script to pull a Git repo with your new job and do the kill/submit cycle?

Travis16:01:16

yeah, Cloudformation is tough for this kind of thing

michaeldrogalis16:01:31

What is the difficult part right now? Retaining the old job ID, or acquiring the new job data?

chrisblom16:01:04

retaining the old job id

michaeldrogalis16:01:01

@chrisblom Can you take the output of submit-job and store it in a file on the box where successive submissions take place?

Travis16:01:37

Would you need the same job-id? Assuming your reading from Kafka would the group id ( i think thats the key ) kind of give you restart capability. When the new job starts up it will have the same group id therefore starting were the previous job left off? Little rusty on this part as we haven’t worked out upgrade scenarios yet but will be shortly

michaeldrogalis17:01:20

@camechis He needs to kill off the old job.

chrisblom18:01:06

@michaeldrogalis yeah, i could store the job id somewhere

chrisblom18:01:22

though sometimes we deploy with the the repl, so i'd have to make sure the stored job id stays in sync with what is actually running

chrisblom18:01:24

is it possible to query the log to retrieve job ids by job name?

michaeldrogalis18:01:40

@chrisblom I’d highly recommend having only 1 method of deployment - it really decreases the chances of something going wonky because a step was added or forgotten. The log isn’t really directly queryable. You just play it forward and accrete to your local replica. You can write a function that takes a replica and returns job name -> ID though.

chrisblom19:01:22

@michaeldrogalis thanks, that should do the trick