Fork me on GitHub
#onyx
<
2018-09-22
>
djjolicoeur14:09:38

thanks @lmergen , that helped a lot! one other question, it appears the the onyx GC fn is being called on every replica in the cluster. is that the correct approach? From the docs, it sounds like you want to call the fn on one replica and it would trigger GC on all the peers. I also see some errors in our logs originating from the GC fn in the stack trace related to ZooKeeper

lmergen14:09:01

i believe the GC is mostly about cleaning up the distributed log -- since all peers should have the same "view" of the log, it makes sense that it's executed on all peers

lmergen14:09:08

what - i think - actually happens is that a simple entry is written in the log, "perform gc" -- all peers read this, and since GC is completely deterministic, at the end of the operations they all still have one consistent view of the (now GC'ed) log

lmergen14:09:31

this is a bit of a consequence of the masterless design

djjolicoeur14:09:27

I think I get it, it would get executed on all peers by onyx itself, but should only be triggered once in the cluster, is that right? I think we may be writing the special “perform gc” to the log several times, if that makes sense

djjolicoeur15:09:32

@lmergen appreciate the help. we are scaling up and this system that has been somewhat reliable for about a year is starting to have issues as we scale.

lmergen15:09:45

yes that's correct

lmergen15:09:33

i don't think it's too much of a problem to write too much GC, apart from wasted CPU cycles

djjolicoeur15:09:15

ok. thanks. Either way, we have an easy way to get that to run on one replica, so might as well get eliminate that as a cause.