This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (24)
- # boot (80)
- # braid-chat (11)
- # cider (89)
- # clara (11)
- # cljsfiddle (5)
- # cljsjs (9)
- # cljsrn (63)
- # clojure (114)
- # clojure-austin (1)
- # clojure-berlin (5)
- # clojure-brasil (4)
- # clojure-dusseldorf (5)
- # clojure-hamburg (17)
- # clojure-india (1)
- # clojure-new-zealand (3)
- # clojure-poland (1)
- # clojure-russia (91)
- # clojure-taiwan (1)
- # clojure-uk (54)
- # clojurebridge (3)
- # clojurescript (170)
- # core-matrix (1)
- # cursive (14)
- # datomic (8)
- # emacs (13)
- # hoplon (96)
- # immutant (20)
- # jobs (9)
- # jobs-rus (13)
- # kosmos (3)
- # off-topic (8)
- # om (111)
- # onyx (41)
- # parinfer (116)
- # pedestal (2)
- # proton (4)
- # re-frame (46)
- # reagent (7)
- # ring-swagger (24)
- # slack-help (2)
- # testing (1)
- # untangled (8)
@richiardiandrea: you can use the embedded libraries I have, but I prefer them more for unit testing personally. If you want something running closer to a production environment but on your dev machine, I'd go with a docker image
This one isn't the best image ever, but it certainly is quick to get started with - https://github.com/wurstmeister/kafka-docker
if not those, then as I mentioned https://github.com/ymilky/franzy-embedded and https://github.com/ymilky/travel-zoo together work. I haven't pushed a real version to clojars yet of anything but travel-zoo, but I plan on doing so when I get more testing from users.
If you use franzy-embedded, be sure to use the startable versions unless you want to write some extra code. I might put out one or two additions to help with that, but honestly the startable versions are good enough for 99% of people
nice...well if you run into problems, let me know. I'd say also make sure you grab https://github.com/ymilky/franzy-admin to make your life easier for dealing with creating topics, partitions, fixing stuck things, etc.
It's not too hard to get something going, but it takes some more time to develop your data flows with consumers and producers. For example you probably will want to spend some time on writing some threads and go blocks with core.async. I usually do this by wrapping things in components and starting the threads inside there, much like you see in the onyx source as well. I've also used manifold in conjunction with all and the rest of the jvm concurrency options. I might put out some examples of this kind of stuff, but my feeling is everyone's flow is very different that copying one isn't really smart anyway.
yes you are right, I was actually thinking of trying
pulsar, concurrency (with clojure) is usually fun for me so I don't mind that 😉
ok sounds good, just make sure you are careful with the shutdown. I've helped a few people with problems and usually they're doing something like tight looping on a consumer that can never get shut down properly because the loop can never be exited. In core.async, this is very easy if you just close a channel or send a value on a dropping buffer or promise chan
the consumer should be polling in its own thread, and shutdown with wakeup! from another, while pretty similar for the producer. I'm not sure why so many people keep doing things like creating and destroying consumers/producers in loops. Not something with Franzy, just something I've seen in all languages and real-life code in Kafka. Most of the time you want something running a long time.
I wonder what happens in
core.async if you throw an
InterruptedEvaluationException, just went to check for a sec
Why do you need to throw that exception? Or do you mean catching it inside a thread or go block that is running something else?
In general, I've never heard that you can/are supposed to interrupt core.async threads or go blocks except by closing them yourself, i.e. you can't interrupt all threads on exit without doing it manually
in practice it's actually not a problem I've found, especially using component or mount
usually you've done something super wrong if you need to interrupt all threads at a low-level and you're not writing an os or something
most of the time for a loop, I also/instead just check a when on some channel if it's closed or not when I put, which means you should stop recuring/looping
yeah this protocol of
nil-ing for closing the chan really makes everything super smooth
ok good, this is very useful and thanks! I will try your
franzy packaging, I am actually excited about it 😉
Careful with closing a channel from the consumer side. Channels are for multi-writer->multi-reader concurrency and there’s no guarantee that in between calls to
closed? and your
>!!, the channel is actually open.
I actually avoid ever closing anything from the consumer side, I just let the loop break due to putting on to a closed chan or close it from the creator/producer, often a component that owns the chan in its stop method, perhaps with a take from the go block itself to block until it's done. Perhaps I'm explaining bad, but essentially I never use closed?, just closed in the sense of the standard when/if check on the op. From what I've seen in onyx, ex: lifecycle/peer files, this is the same approach.
In order to improve test coverage and reduce the amount of code there, I’m looking at potentially merging them since it doesn’t seem like I ever want one without the other
@lvh: In production, you'll likely want to run Aeron in a separate JVM to avoid thread contention. We gave it its own namespace so that you can have a separate
-main, which would easy the transition.
Aeron is pretty sensitive to context switching if there are too many active threads. Certainly for development you can put them together, though.
It's also so that if you get a big GC pause in your main JVM, you don't take out your media driver
@michaeldrogalis: just a quick Q, is there a plan to move from zookeeper to etcd, and other more nimble components like that for the other services, to move to a more light dependency stack?
btw thanks for onyx, I loved storm but was blown away by the data pureness and comprehensivess of your project
@avocade: Not at the moment, no. Etcd can't act as a substitute for ZooKeeper. It lacks a number of primitives - the most prominent being ephemeral nodes, which we in turn use as observation triggers.