Fork me on GitHub
#onyx
<
2016-02-05
>
robert-stuttaford13:02:42

@lucasbradstreet @michaeldrogalis if i need to clear ZK’s checkpoints, i can rm /tmp/zookeeper/* right?

robert-stuttaford13:02:21

i used a new checkpointing key instead

lucasbradstreet14:02:44

@robert-stuttaford: you could also use datomic/reset? on the next run to clear the checkpoint, or just delete the checkpoint node in zookeeper via a client library

lucasbradstreet14:02:58

I wouldn't recommend trashing zookeeper for something like that

lowl4tency14:02:12

robert-stuttaford: don't delete /tmp/zookeeper/myid (if we talk about cluster)

michaeldrogalis15:02:13

Right, what they said.

michaeldrogalis15:02:37

@lowl4tency: @robert-stuttaford Did you end up getting multi-node connected?

lowl4tency15:02:31

michaeldrogalis: as far as I know, guys are finishing the feature. It's on dev env atm

lowl4tency15:02:06

michaeldrogalis: atm we have aeron ext driver and zk cluster. Need to finish app part

michaeldrogalis15:02:08

Also FYI since I know some from your group were asking about it, the new Task Constraints feature is ready and will be out in a few days: https://github.com/onyx-platform/onyx/blob/develop/doc/user-guide/scheduling.md#tags

robert-stuttaford20:02:58

thanks for the info guys! super helpful simple_smile @michaeldrogalis you must be stoked to be landing the constraints stuff finally simple_smile

michaeldrogalis20:02:17

Only took a year to get around to 😛

robert-stuttaford20:02:38

another sanity check: if i want :read-log to redo the same work over and over (like in a dev setup) would ZK checkpointing prevent work from happening the second + time?

robert-stuttaford20:02:05

man, you guys set a crazy pace. can not keep up with all the cool stuff you’re adding

michaeldrogalis20:02:07

You'd use :datomic/reset? to force it to start from the beginning. As long as the ZK state hasn't been fiddled with, it will prevent duplicate work from being replayed in the presence of :reset? being false.

robert-stuttaford20:02:45

aha! is that documented somewhere?

michaeldrogalis20:02:05

Er, yeah. Sorry, had the name wrong simple_smile

robert-stuttaford20:02:31

i feel quite ashamed to admit how much of my own time i’ve wasted by bashing my head against this

robert-stuttaford20:02:22

i dig your new avatar

robert-stuttaford20:02:35

“i’m in ur kodez"

michaeldrogalis20:02:42

It looks pretty judgy, specifically at my own comments. XD

robert-stuttaford20:02:54

"did i just say that?"

robert-stuttaford20:02:20

how’ve you been?

robert-stuttaford20:02:36

did my blog post move the needle for you in any meaningful way?

michaeldrogalis20:02:04

It did actually - thank you. Things have been well. Commercial support has picked up

michaeldrogalis20:02:48

Definitely thankful for that. Onyx is gigantic, it would be too hard to do this part time much longer

robert-stuttaford20:02:20

the universe has just snapped back into focus

robert-stuttaford20:02:48

i’m super glad to hear that, man. you deserve to get to do this full time

michaeldrogalis20:02:12

Ha! Is that running on your prod cluster, then?

robert-stuttaford20:02:27

naw that’s just in test

robert-stuttaford20:02:55

so, this force thing, that’ll tell the plugin to ignore checkpointing info, but won’t alter any that might exist, right?

robert-stuttaford20:02:03

so i’d keep it there until i’m done testing

michaeldrogalis20:02:26

I believe it will blow away any existing checkpoints and start from scratch, as if it had never run before.

robert-stuttaford20:02:48

ok. i’ll disable and see if it still runs on a subsequent pass

michaeldrogalis20:02:53

Once you force reset, you can't undo that, if thats what you mean

robert-stuttaford21:02:00

my homebrew ZK takes a long time to warm up all the peers it needs

robert-stuttaford21:02:07

can i do anything to speed that up?

robert-stuttaford21:02:35

16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.messaging.aeron] - Starting Aeron Messenger
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.peer.virtual-peer] - Starting Virtual Peer ddfa18dd-b0b3-440f-8f89-600d7c3dda1a
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.

michaeldrogalis21:02:44

Are you seeing slowness only when you run ZK on your machine? Hard to say whats happening there.

robert-stuttaford21:02:07

yeah prod does this pretty quickly, but locally it takes a while

michaeldrogalis21:02:51

Im really not sure tbh, perhaps check the config file on your local installation for something wonky.

robert-stuttaford21:02:51

like 2-5 secs per zk connection

michaeldrogalis21:02:10

Huh, Ive never seen that before. That smells like ZK configuration

michaeldrogalis21:02:29

Yeah, let me know what you find. Thats annoying

robert-stuttaford21:02:37

it’s stock homebrew zookeeper

robert-stuttaford21:02:47

goodness knows what it does

robert-stuttaford21:02:09

appears the force thing is like a clutch pedal - works only when it’s on

robert-stuttaford21:02:56

when deciding whether to write or force write

robert-stuttaford21:02:14

i’m guessing other plugins are a little more mutative-y

robert-stuttaford21:02:28

update-in-place-y

michaeldrogalis21:02:38

Yep, correct assumption

robert-stuttaford21:02:20

thanks man - have a great weekend!