Fork me on GitHub
#onyx
<
2018-03-16
>
sparkofreason01:03:33

I'm curious how onyx handles restart with stateful tasks. I understand why the number of peers assigned to a stateful task can't change while the job is running, but what happens when you kill and restart. Can it redistribute the tasks to a different number of peers?

lucasbradstreet01:03:22

It does not currently have the ability to re-partition state. It’s been on our backlog for a while.

lucasbradstreet02:03:46

It wouldn’t really be all that hard to implement a feature which would use the resume-point functionality to recover and repartition state.

sparkofreason02:03:59

That would be a great feature.

lucasbradstreet02:03:25

Agreed. Our current suggestion is to over-partition your stateful peers to begin with. We know it’s not great.