Morn'

Howdy

morning

happy lunch time one and all

so this morning zookeeper died of dysentery or something and totally shat all over its snapshots 😢

kill zookeeper with fire

unpleasant memories

first time it's ever un-recoverably borked for me

it looks like the ubuntu 14 current version has several critical bugs in it which are fixed in the following point releases

though the number of "critical" bugs in their release notes really doesn't inspire confidence

@mccraigmccraig: Do you use zookeeper directly?

@dominicm: indirectly - mesos, kafka and onyx all use it

sweeet: JSON CRDTs http://arxiv.org/abs/1608.03960

That looks like you might have just opened up the rabbit hole I've been looking for ...

looks interesting, I’ve spotted a mini typo We introduce a single, general mechansim

kevin_e: maybe it isn't a typo and it is their neologism o_O

I like that this looks like it will apply to an awful lot of clojure style data structures too

haha 🙂

I have a question that may relate to CRDTs: - User updates username to A - User very quickly updates username to B Due to distribution, B hits the database first. How do I prevent A from becoming the username?

@dominicm: in that case you could supply ordering information from the user's client to allow the server to discriminate with something like update users set username=X where id=ID and timestamp<Y

if there are two or more clients making concurrent changes with no obvious priority then you may want CRDTs

update user set name=A where name=old is possibly the most general approach

then if you do the same update twice, all is good

even better @glenjamin

I think I see. I thought it might be. That was the best idea I had too. Could that break if a user tried A->A->B->A, and ended up with A->A->A->B ?

yeah, the most robust is probably using the revision number, what to do when you actually do hit conflicts in interesting

Something else I'm trying to figure out is how to make idempotent writes. Onyx can retry a message if it's not sure it succeeded. Datomic is pretty easy if you create a custom transactor function which looks for transactions with a particular message-id. Something more traditional like postgres confuses me a little still though.

In general, datomic is great for distributed systems, because even if a write happens during a process, your reads are consistent.

Distributed systems just seem hard for updating state. Better to just store a history of messages and reprocess them on-demand.

dominicm: any ACID database should be consistent during reads (tho many are not configured that way)

Maybe I meant persistent. Especially for multiple reads.

If something doesn't process for 10s, I still want it to operate with the way the world was 10s ago, I think?

@dominicm: your history of messages is also state - all participants have to agree on the history, otherwise their notions of eventual state may be different

most databases throw away context, the more you can keep, the more options you have to reconcile conflicts

😭 that's it. Distributed systems are dead to me. 🙂

Write last wins. Overwriting state like an atom. With no history.

In seriousness, is there a way I can learn to handle all of this stuff? Resources you guys could recommend?

I reckon it’s entirely contextual, depending on what you’re building you may or may not need any of this

for single-user-writing, configuring a majority quorum for writes should be plenty

and multiple-users editing the same resource is a people problem 😄

One of my interests is replayable events, as in, you can rebuild your database from scratch from the list of events. But that means you must be truly consistent, you can't have out of order changes screwing with things.

So I guess that's my context really 😛

@dominicm: Martin Kleppmann’s blog is a good resource and his book is really good https://martin.kleppmann.com/

@minimal: Ooh, thankyou!

oh he is a co-author in the json crdt paper!

https://twitter.com/martinkl/status/765451802213027840

dominicm: generally finding books with animals on the cover that talk about computery things is a good start

there are a few other books out there from other publishers too (like Henry Garner's book)