@lucasbradstreet @michaeldrogalis if i need to clear ZK’s checkpoints, i can rm /tmp/zookeeper/* right?

i used a new checkpointing key instead

@robert-stuttaford: you could also use datomic/reset? on the next run to clear the checkpoint, or just delete the checkpoint node in zookeeper via a client library

I wouldn't recommend trashing zookeeper for something like that

robert-stuttaford: don't delete /tmp/zookeeper/myid (if we talk about cluster)

Right, what they said.

@lowl4tency: @robert-stuttaford Did you end up getting multi-node connected?

michaeldrogalis: as far as I know, guys are finishing the feature. It's on dev env atm

@lowl4tency: Nice, good to hear.

michaeldrogalis: atm we have aeron ext driver and zk cluster. Need to finish app part

Also FYI since I know some from your group were asking about it, the new Task Constraints feature is ready and will be out in a few days: https://github.com/onyx-platform/onyx/blob/develop/doc/user-guide/scheduling.md#tags

thanks for the info guys! super helpful @michaeldrogalis you must be stoked to be landing the constraints stuff finally

Only took a year to get around to 😛

another sanity check: if i want :read-log to redo the same work over and over (like in a dev setup) would ZK checkpointing prevent work from happening the second + time?

man, you guys set a crazy pace. can not keep up with all the cool stuff you’re adding

You'd use :datomic/reset? to force it to start from the beginning. As long as the ZK state hasn't been fiddled with, it will prevent duplicate work from being replayed in the presence of :reset? being false.

aha! is that documented somewhere?

perhaps you mean :checkpoint/force-reset? ? https://github.com/onyx-platform/onyx-datomic/blob/766f0ff15b8898765c7b7d97d43b55b633ff652f/src/onyx/plugin/datomic.clj#L229

Er, yeah. Sorry, had the name wrong

hehe

feel the learn!

i feel quite ashamed to admit how much of my own time i’ve wasted by bashing my head against this

Now you know

hells yeah

i dig your new avatar

“i’m in ur kodez"

It looks pretty judgy, specifically at my own comments. XD

"did i just say that?"

Heh

how’ve you been?

did my blog post move the needle for you in any meaningful way?

It did actually - thank you. Things have been well. Commercial support has picked up

Definitely thankful for that. Onyx is gigantic, it would be too hard to do this part time much longer

the universe has just snapped back into focus

i’m super glad to hear that, man. you deserve to get to do this full time

Ha! Is that running on your prod cluster, then?

naw that’s just in test

so, this force thing, that’ll tell the plugin to ignore checkpointing info, but won’t alter any that might exist, right?

so i’d keep it there until i’m done testing

I believe it will blow away any existing checkpoints and start from scratch, as if it had never run before.

ok. i’ll disable and see if it still runs on a subsequent pass

Once you force reset, you can't undo that, if thats what you mean

Sure

my homebrew ZK takes a long time to warm up all the peers it needs

can i do anything to speed that up?

16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.messaging.aeron] - Starting Aeron Messenger
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.peer.virtual-peer] - Starting Virtual Peer ddfa18dd-b0b3-440f-8f89-600d7c3dda1a
16-Feb-05 22:59:51 MacBook-Pro.local INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.

this cycle

Are you seeing slowness only when you run ZK on your machine? Hard to say whats happening there.

yeah prod does this pretty quickly, but locally it takes a while

Im really not sure tbh, perhaps check the config file on your local installation for something wonky.

like 2-5 secs per zk connection

Huh, Ive never seen that before. That smells like ZK configuration

* 30 peers

cool. i’ll RTM

Ew

Yeah, let me know what you find. Thats annoying

it’s stock homebrew zookeeper

goodness knows what it does

appears the force thing is like a clutch pedal - works only when it’s on

which makes sense - in the onyx-datomic plugin it only appears once in the source - https://github.com/onyx-platform/onyx-datomic/blob/766f0ff15b8898765c7b7d97d43b55b633ff652f/src/onyx/plugin/datomic.clj#L229

Right

when deciding whether to write or force write

i’m guessing other plugins are a little more mutative-y

mut-y?

update-in-place-y

Yep, correct assumption

thanks man - have a great weekend!

You too!