This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-03-16
Channels
- # aatree (1)
- # admin-announcements (6)
- # alda (3)
- # beginners (66)
- # boot (41)
- # cider (4)
- # cljsjs (3)
- # cljsrn (18)
- # clojure (146)
- # clojure-android (2)
- # clojure-nl (1)
- # clojure-russia (35)
- # clojure-sdn (2)
- # clojure-sg (5)
- # clojure-uk (41)
- # clojurescript (116)
- # datomic (12)
- # dirac (40)
- # docker (2)
- # editors (2)
- # hoplon (85)
- # immutant (19)
- # jobs (1)
- # keechma (2)
- # lein-figwheel (8)
- # mount (33)
- # off-topic (1)
- # om (114)
- # onyx (159)
- # parinfer (24)
- # proton (3)
- # reagent (4)
- # ring-swagger (15)
- # uncomplicate (7)
- # untangled (93)
- # yada (30)
@greywolve: I merged your onyx-metrics fix. Would you like me to cut a release? We’re on 0.9.0 alphas now but I can cut one from the 0.8.x branch if you need to deploy it
Looks like there is a link between onyx and https://github.com/ymilky/franzy ? Just wondering what it is exactly, if this would be the recommended kafka lib for use with onyx, and also if you think something about https://github.com/pyr/kinsky ?
@ymilky created it out of a personal need, along with Onyx’s need for a better client than clj-kafka
kinsky is too opinionated for our needs
Awesome write up on the Jepsen testing you folks have been doing, really enjoyed reading it http://www.onyxplatform.org/jekyll/update/2016/03/15/Onyx-Straps-In-For-A-Jepsening.html
Thanks @mrjaba. It took a while to get right, but it was totally worth it.
Now that we have the framework right, we’ll be able to continue to add tests to it pretty easily
I'm only working through the tutorials still at the moment, but really looking forward to getting onto the more advanced examples
Great . Give us a shout here if you have any questions
I hate just dumping STs, but... https://github.com/onyx-platform/onyx-dashboard/issues/48
I just replied to your issue :)
I'll be pushing out a new release very shortly so let me know if you hit any troubles with that version :)
@lucasbradstreet: trying it now is there a preferred browser?
To be honest I’ve only tested it in chrome. I’ll actually give it a go in Firefox now
Nice. Works fine in Firefox
Sorry IE, don’t really care about you
I just closed 10 dashboard issues
hmm me too
https://cdnjs.cloudflare.com/ajax/libs/react/0.9.0/react-with-addons.min.js seems to work
Must be
I actually don’t see any 500s in my network activity when I load it in chrome though
uBlock origin is blocking it
Oh wait
I clicked the min link sorry
Yeah, still 500s
Does anything work, or is it completely broken?
Could you try a lein clean and then try the instructions again?
Hate to suggest that 😕
I shall, but I think if that react link is bust then it makes sense that everything else falls over
Hmm yeah. At least it should work with our advanced mode compiled jars, since I think they'll have everything in the one JS file
lucasbradstreet: I had an Onyx app running but I've torn everything down and tried to use the sample job. Starting Onyx test environment
looks like it might be hanging though
You mean via the lein run example in the README?
OK, let me know how you go with that first
I think it’s probably just ZK conn string problem. I just made a couple changes so you can supply that at the REPL for dev. I’ll point you to it once you’ve confirmed that this works
Not with onyx.peer.dag-test. That has everything you’ll need to try the dash
Weird. Stuck at zookeeper still? CI is releasing a new version now if you want to try out a jar
I'll let you know when it's ready
No bother at all. Good to know about these issues and it'll be good to have someone try out what I just released :)
@acron it's ready. Could you follow along with the instructions here: https://github.com/onyx-platform/onyx-dashboard
@lucasbradstreet: sure thing
Link to releases.md is at the top. Note that you'll need 0.8.11
hopes the release process worked (I'm on my phone at the moment
grabbing https://s3-us-west-1.amazonaws.com/onyx-releases/onyx-dashboard/onyx-dashboard-0.8.11.7.jar
Don't run the jar from your onyx-dashboard repo. Clojure will actually complain about mixed mode compiles. Every time this happens I WTF
error {
:cause KeeperErrorCode = NoNode for /onyx
:via
[{:type org.apache.zookeeper.KeeperException$NoNodeException
:message KeeperErrorCode = NoNode for /onyx
:at [org.apache.zookeeper.KeeperException create KeeperException.java 111]}]
Could be a better error message, but it's probably just not able to connect to a zookeeper which has had onyx connect to it
Figwheel started? Oops haha. That shouldn’t happen from the jar
Ok cool 😄
Awesome. Let me know if you have any requests. I’ve been in triage mode with it, but I really want to add some features to it in the future
By the way, try out the time travel feature and you can step through the cluster state over time
@lucasbradstreet: Super cool, thanks a lot We are impressed
Glad you like it, and I’m really happy that it’s been triaged. It was somewhat incorrect, and quite hard to use before.
@nha I'm asking people to try Franzy before I proclaim it should be the Kafka client the should use, but I'm currently using it without issues. As Lucas mentioned, I created it out of a real need. I had a lot of trouble with other Kafka clients, especially on 0.9 ranging from not working to inaccurate results/bugs to being slow. The reason I think it is a good fit with Onyx is that all the data to/from is maps and records. This should make it easy to not only feed records to the producer/consumer in the conventional way, but also to use metadata, metrics, and other info you can get back form the API with onyx. Moreover, since there is also an admin api, at least for my purposes, that helps, especially if I need to bootstrap a topic or add partitions or something before letting data flow out to Kafka from an onyx job.
@otfrom: I hope to be on this list someday 😄
ymilky: I can more than likely give franzy some production use in the next few months. Similarly to you, I find clj-kafka kinda wonky
@acron sounds good, I haven't encountered any bugs from Franzy itself, but surely the underlying Java client parts have some bugs, so test it well first
Ok, newbie question. Is it possible to run multiple peers on the same machine? I notice peer-ports
and peer-port-range
were deprecated in favour on a single port number
@acron: Yes, you want to run roughly the number of peers per number of cores in the machine. They'll multiplex efficiently through one port. You only want one peer group per machine, though
A peer group is essentially a set of shared resources, which includes the Aeron networking layer. Aeron only wants one of these on the box. So yeah, add more virtual peers, keep it to 1 peer group.
To get the terminology correct: 1 peer contains n virtual peers, and 1 peer group contains n peers ?
This is a result of some poor naming at the outset of the project. A "peer" is meant to basically be physical machine, a "virtual" peer one of N workers on the machine, and a "peer group" a set of resources shared locally on the machine for the N virtual peers.
Will probably look into clearing up the naming for 1.0, but hopefully that makes sense.
@otfrom: Honored.
@michaeldrogalis: That's cool, thanks for clearing it up
@michaeldrogalis: exceptions coming out of handle-exception are qualitatively worse now because we’re dropping the original stack trace, to be replaced by the handle-exception one
fault lies in the ex-info returned by merge-error-keys
I suspected that - I saw a weird stack trace last night and meant to look into it, thanks for the confirmation
This ex-info becomes the new stack trace that gets logged
@michaeldrogalis: stacktrace issue fixed https://github.com/onyx-platform/onyx/commit/cb2667a7f818615c2c3f68388e39e8699e868d96
Dynamic types hum
@lucasbradstreet: Thanks. Upgrade it to master when you're ready.
I’m also going to improve that “no bookkeeper” exception, which was the main reason I encountered it
Will do
Unsolved problems: automated tests for "good error messages".
@ymilky: "You didn't read the code, comments, documentation, or anything else I told you to" 😆
@ymilky: Is there a mechanism within franzy for discovery of Kafka brokers from ZK? Or should one use zookeeper-clj
for that?
@acron what kind of discovery do you need? You can use franzy-admin to list brokers and their associated metadata if you want
there's also a few methods that will return you info about the replicas, leaders, in-sync replicas, etc.
there shouldn't be anything that is in clj-kafka that isn't in franzy as far as I am aware, functionally speaking
specifically, you can use https://github.com/ymilky/franzy-admin/blob/master/src/franzy/admin/cluster.clj#L19
this will read from zookeeper directly, so no need to know about another kafka broker yet
@michaeldrogalis: I’m checking my python version
Python 2.7.11 is greater than 2.7.8 I assume
you can get the ssl channels this way for example if they are different from plaintext (or non-existent)
I suppose I should add a convenience method that will tranduce the results into a valid bootstrap.servers connection list
@acron working on some other stuff today, but I'll try to throw in something like I described in a commit tomorrow so unless you are desperate, don't bother writing tons of code to manipulate the map returned. Good to hear some real-world feedback for potential feature requests, thanks.
@lucasbradstreet: That's correct. You can always go to https://www.python.org/downloads/ when in doubt of versions. Thankfully the versioning is sane, the real trouble is v2 vs v3 versions.
Thanks ymilky. Seems like it was a bug in engraver or it’s pip package
It's unlikely something is happening in Franzy itself as that call generally just proxies
additionally, you could try send-async! with a callback and you'll at least get notified when kafka thinks it has ack'd your record. Your actual behavior will be dependent on the ack settings in your config for the producer/broker
If you need a crappy log4j example, there's one in https://github.com/ymilky/franzy-examples/blob/master/resources/log4j.properties
I haven't had time to tweak, it's just a quick one i had for seeing what kafka is sending back. Obviously you'll also have to be sure you can turn on all the log4j stuff for your clojure project, which in some cases may mean pulling in some of the deps
I've tried sending 1 million+ entries in a single tight loop without issue, but who knows, there could be some issue with the java client itself