This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (27)
- # boot (49)
- # cider (51)
- # cljs-dev (29)
- # cljsjs (1)
- # cljsrn (19)
- # clojure (59)
- # clojure-austin (2)
- # clojure-belgium (19)
- # clojure-china (1)
- # clojure-dev (14)
- # clojure-dusseldorf (7)
- # clojure-russia (8)
- # clojure-spec (115)
- # clojure-uk (45)
- # clojurescript (118)
- # css (6)
- # cursive (8)
- # datascript (20)
- # datomic (32)
- # emacs (5)
- # events (2)
- # flambo (21)
- # hoplon (58)
- # incanter (8)
- # jobs-rus (1)
- # jobs_rus (1)
- # off-topic (3)
- # om (22)
- # om-next (9)
- # onyx (5)
- # other-languages (79)
- # re-frame (126)
- # reagent (6)
- # ring (7)
- # specter (1)
- # untangled (119)
- # yada (38)
Anybody watched this talk from Clojure/West yet? https://www.youtube.com/watch?v=eRq5UBx6cbA&list=PLZdCLR02grLq4e8-1P2JNHBKUOLFTX3kb&index=17
Daniel talks about parallelism and the fork/join framework
Which from what I can tell is an implementation of the
ExecutorService interface in Java. Docs are here https://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html
Anyone used this or looked into it before?
It was really good
I watched a few talks from the conf last night
Daniel is quite comical in his presenting style too
This talk on Kafka and Clojure was also pretty interesting https://www.youtube.com/watch?v=VC_MTD68erY&index=23&list=PLZdCLR02grLq4e8-1P2JNHBKUOLFTX3kb
yes, I watched that one @yogidevbear , very good. and very relevant to some of the work I have been doing the last few years.
Really interesting concepts that they both chatted about
are there any kafka gurus here who could point me in the right direction towards diagnosing a problem i've just had ?
@tcoupland: i just observed a problem where a pair of brokers contrived between them to truncate the logs for a bunch of topics, so that consumers were unhappily left with offsets which were off the end of the topic. all topics were replicated (1 replica), so wondering what sort of failure modes to be looking for
this seemed to happen around the same time ZK and mesos were having brainfarts too - though i have no idea why ZK elections would cause kafka log truncation and it might have been a tipping-factor rather than a fundamental cause
hmm, don't think i can be too much help, but removing elements from the front of the logs is something that really shouldn't happen. Therefore, you'd hope the circumstances when it should would be documented quite carefully.
@tcoupland: i think it removed elements from the end of the logs, rather than the front
my, so far unsupported, working hypothesis is that replicas were behind the leader for some reason, the replica became leader and on doing so set something in ZK causing the erstwhile leader to throw away the front of its log
ah, so the consumers were behind. Then when the log got truncated (which is normal [ish]) they dropped off the end
no, the consumers were caught up, and ended up with an offset which was higher than the highest offset in the remaining log...
thanks @tcoupland - reading around, i'm guessing i need to be fiddling with minimum in-sync-replica config, so i get early failure rather than data loss
sounds like the right track. Good read of the replication strategy and configuration options does sound in order.
not too my mind. Truncation removes the tail of the log. The message ids/offset don't need to reflect that at all
that would be my expectation too, do you have any monitoring of max-id / offset over time mccraigmccraig ?
@glenjamin: not yet
i found https://github.com/quantifind/KafkaOffsetMonitor worked quite nicely for a basic overview
if you have a time series DB around you’ll probably want to move to pumping the values into there at some point
@mccraigmccraig: Given I've spent 6 weeks just changing pom files and arguing with 'architects' about versioning strategies I envy your kafka problems! I look back on the issues I was having with HBase last year with great fondness.
It's time to hack on the train again! Been reading up about spec. I see from conversations in this channel it has had a mixed reception but it does appear to be a step in the right direction. I can see the argument for separating definitions of map/seq structure from the value predicates.
and then combine that with runtime checking and some sort of value tagging, to apply type checks when values pass through a spec boundary, but not recheck them if they stay within the specced portion
@glenjamin: not sure my small brain can understand the implications of that but sounds interesting. Also I can't visualise what the type inference would look like. Would that imply some static type checking but bounded by the application of the spec?
I do miss some of the advantages of compile time type checking but I would certainly not want the rigidity of something like the type system in Java so type inference would be useful but what little I've read and watched about it suggests it's not an easy balance to get right.