Fork me on GitHub
#clojure-uk
<
2021-03-18
>
dharrigan07:03:45

Good Morning!

joetague09:03:05

Morning... 🤞 my copy of Software Design for Flexibility arrives this morning. I might need to go buy more coffee grounds to set myself up for reading it

dharrigan09:03:11

Oooh, would like your review of it when you are ready

mccraigmccraig10:03:04

oof - so one of our kafka-streams app ran out of capacity this morning - the 60 partitions i'd assigned to the topic a while back (thinking that it would be enough for a long time in the future) are not nearly enough

mccraigmccraig10:03:52

do any of y'all have any experience with this - is it safe to add partitions to the existing topic, or is it better to create a new topic and migrate everything to there ?

danm10:03:59

New topic, 100%, IMO

dharrigan10:03:44

not a good idea to add partitions to an existing topic if you want to maintain partition assignment for keys

danm10:03:09

If you add additional partitions to an existing topic, you lose guarantees about ordering if reprocessing in future or similar, because the message key -> partition algorithm is based on the number of partitions, so changing that means keys will move to different partitions

danm10:03:15

If you're not going to reprocess old data, and you're all caught up, or you have some other means of ordering/deduping which means you don't care if a given key suddenly changes partition, or you just don't care about processing order in general, then OK

danm10:03:21

But you want to be really sure about that

mccraigmccraig10:03:35

we need the ordering guarantees of partitions - otherwise users will see out of order messages, badly ordered lists of conversations and general inconsistency

mccraigmccraig10:03:15

so it sounds like i should [1] create a new topic, [2] switch routing over to send to new topic, but don't start processing from the new topic [3] drain old topic [4] start processing on new topic

danm10:03:45

'drain the old topic'? I have to admit I've not used Kafka streams specifically. But in the past when i've wanted to do this my pattern has been [1] Create new topic, [2] Create 'duplicator', which runs through the old topic consuming all messages and writing them to the new topic (per-key order is maintained), [3] When the duplicator has caught up and is basically copying messages as soon as they appear on the old topic, switch off the app, wait for the duplicator to process potentially the last few messages (literally milliseconds, mostly likely), and switch the app back on now pointing at the new topic Streams having state in rocksDB and such may change how viable that is, it might mean it takes a lot longer to start back up on the new topic or whatever

dharrigan10:03:31

Yes, I've done what Dan has done in the past too

mccraigmccraig10:03:36

that process would work for us... what do you use for duplicating topics ? confluent replicator, or something else ?

dharrigan10:03:33

I wrote a bit of code in Clojure 🙂

danm11:03:38

Likewise. It wouldn't surprise me if there was a Confluent tool to do it for you, that comes as part of the paid enterprise package, but at the time we were using the free OSS version

mccraigmccraig13:03:38

thanks @carr0t @dharrigan - all moved over to a new topic with 1440 partitions (vs 60 on the old) and everything is running smoothly again

3
danm13:03:31

Is this your own install, or MSK/similar?

mccraigmccraig13:03:53

it's strimzi on k8s

danm13:03:58

👍:skin-tone-2:

mccraigmccraig13:03:11

strimzi has been great

mccraigmccraig13:03:53

especially the topic management nouns

mccraigmccraig18:03:31

well that was a "good" bug - an unexpectedly unbounded recursive call in some promise-based code ate the world