morning

Morning

Anybody watched this talk from Clojure/West yet? https://www.youtube.com/watch?v=eRq5UBx6cbA&list=PLZdCLR02grLq4e8-1P2JNHBKUOLFTX3kb&index=17

Daniel talks about parallelism and the fork/join framework

Which from what I can tell is an implementation of the ExecutorService interface in Java. Docs are here https://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html

Anyone used this or looked into it before?

didn’t watch it, but it looks interesting, cheers

haven’t had time yet to watch that one… but it does look interesting.

if you do get round to watch it please let us know.

It was really good

I watched a few talks from the conf last night

Daniel is quite comical in his presenting style too

i really liked his book as well

+1 🙂

This talk on Kafka and Clojure was also pretty interesting https://www.youtube.com/watch?v=VC_MTD68erY&index=23&list=PLZdCLR02grLq4e8-1P2JNHBKUOLFTX3kb

yes, I watched that one @yogidevbear , very good. and very relevant to some of the work I have been doing the last few years.

Really interesting concepts that they both chatted about

are there any kafka gurus here who could point me in the right direction towards diagnosing a problem i've just had ?

i've done quite a bit... hesitate to say guru though 🙂

@tcoupland: i just observed a problem where a pair of brokers contrived between them to truncate the logs for a bunch of topics, so that consumers were unhappily left with offsets which were off the end of the topic. all topics were replicated (1 replica), so wondering what sort of failure modes to be looking for

this seemed to happen around the same time ZK and mesos were having brainfarts too - though i have no idea why ZK elections would cause kafka log truncation and it might have been a tipping-factor rather than a fundamental cause

hmm, don't think i can be too much help, but removing elements from the front of the logs is something that really shouldn't happen. Therefore, you'd hope the circumstances when it should would be documented quite carefully.

@tcoupland: i think it removed elements from the end of the logs, rather than the front

my, so far unsupported, working hypothesis is that replicas were behind the leader for some reason, the replica became leader and on doing so set something in ZK causing the erstwhile leader to throw away the front of its log

ah, so the consumers were behind. Then when the log got truncated (which is normal [ish]) they dropped off the end

no, the consumers were caught up, and ended up with an offset which was higher than the highest offset in the remaining log...

ok, hmm that sounds fiddly. I'd be interested to hear what you find out 😄

thanks @tcoupland - reading around, i'm guessing i need to be fiddling with minimum in-sync-replica config, so i get early failure rather than data loss

sounds like the right track. Good read of the replication strategy and configuration options does sound in order.

is the offset supposed to go down when truncation happens?

not too my mind. Truncation removes the tail of the log. The message ids/offset don't need to reflect that at all

that would be my expectation too, do you have any monitoring of max-id / offset over time mccraigmccraig ?

@glenjamin: not yet

i found https://github.com/quantifind/KafkaOffsetMonitor worked quite nicely for a basic overview

oh, nice

thanks @glenjamin

if you have a time series DB around you’ll probably want to move to pumping the values into there at some point

but that’s a single jar that can get you something useful right away

@mccraigmccraig: Given I've spent 6 weeks just changing pom files and arguing with 'architects' about versioning strategies I envy your kafka problems! I look back on the issues I was having with HBase last year with great fondness.

It's time to hack on the train again! Been reading up about spec. I see from conversations in this channel it has had a mixed reception but it does appear to be a step in the right direction. I can see the argument for separating definitions of map/seq structure from the value predicates.

I heard an interesting suggestion from someone yesterday

what if you could do something like type inference on specs

and then combine that with runtime checking and some sort of value tagging, to apply type checks when values pass through a spec boundary, but not recheck them if they stay within the specced portion

@glenjamin: not sure my small brain can understand the implications of that but sounds interesting. Also I can't visualise what the type inference would look like. Would that imply some static type checking but bounded by the application of the spec?

I do miss some of the advantages of compile time type checking but I would certainly not want the rigidity of something like the type system in Java so type inference would be useful but what little I've read and watched about it suggests it's not an easy balance to get right.