Fork me on GitHub
#onyx
<
2017-02-15
>
jcf16:02:33

Anyone know if Onyx is supposed to balance peers across multiple jobs if you set the task scheduler to :onyx.task-scheduler/balanced? Let's say I submit a job and it gets all the peers assigned. Then I submit another job an hour later… Onyx will rebalance the peers to make sure my new job gets some of the available vpeers, right?

Travis16:02:07

I believe that is the case

jcf16:02:36

@camechis me too, but I can't see it confirmed in the docs.

jcf16:02:09

Oh, wait. Found it in the docs! > If a job is submitted while this scheduler is running, the entire cluster will be rebalanced. For example, if job A has all 100 peers executing its task, and job B is submitted, 50 peers will move from job A to job B.

Travis16:02:29

yep, was just going to send that to you

jcf16:02:32

Question answered. Thanks @camechis!

Travis22:02:52

Hey guys, I have an interesting question for you. I am going to try and explain this. We are attempting to use spandex ( an elasticsearch lib ) in a bulk task. What is interesting about this library is that the ES bulk task is built around using core.async channels. We are wondering how this might work inside of a bulk task or if it will?

Travis22:02:21

Currently we are going to setup the channels in a lifecycle which get passed in to the bulk task using a param

gardnervickers22:02:08

You can use lifecycles to inject any state you may need, including core.async channels.

Travis22:02:24

ok, I think its just feeling wierd to put stuff on a channel and then deal with the results on an ouput channel in a single task

Travis22:02:38

but thats the way the bulk ops work on this library

Travis22:02:02

i think we will have to do a blocking take on the output and check for errors

gardnervickers22:02:45

Depending on what you need for reliability guarantees, it might be a better choice to use a plugin.

Travis22:02:24

yeah, i don’t think we have enough time right now to build one out

Travis22:02:30

for our needs