This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-01-24
Channels
- # beginners (19)
- # boot (118)
- # capetown (4)
- # cider (37)
- # cljs-dev (69)
- # cljsjs (23)
- # clojure (212)
- # clojure-austin (10)
- # clojure-india (3)
- # clojure-italy (2)
- # clojure-mke (1)
- # clojure-nl (1)
- # clojure-russia (5)
- # clojure-spec (52)
- # clojure-uk (86)
- # clojurescript (31)
- # core-async (9)
- # cursive (123)
- # datomic (91)
- # emacs (22)
- # events (3)
- # hoplon (68)
- # klipse (4)
- # lambdaisland (10)
- # leiningen (2)
- # off-topic (14)
- # om (14)
- # onyx (44)
- # perun (14)
- # proton (20)
- # re-frame (15)
- # reagent (10)
- # ring-swagger (9)
- # specter (18)
- # untangled (3)
- # vim (26)
- # yada (4)
newbie question: how do you pass handle IO state per peer (stuff like cassandra connection to be used at one step)
@mpenet Have a look at Onyx Lifecycles http://www.onyxplatform.org/docs/user-guide/0.9.10-beta1/#lifecycles
You could use :lifecycle/before-task-start
to setup a Cassandra connection.
:lifecycle/before-task-start
will be run once per peer setup.
:lifecycle/before-batch
will be run once per task invocation.
For initializing some state like a connection where it’s expensive and you only want to do it once, use :lifecycle/before-task-start
Yup np
@mpenet i return :onyx.core/params
with an alia session from an :lifecycle/before-task-start
noted, I am still at baby steps stage, I am trying to see if onyx could replace some of the stuff we use at the moment
@mpenet You want :lifecycle/before-task-start
to initialize connections. That’s once at task boot up time. before-batch
is per batch of segments received, and will be invoked multiple times during the task.
Heh, sorry. Typo’ed 🙂
got it. The docs are top notch actually, I just need to get used to the terminology I guess
@mpenet Thanks 🙂 Another thing that might help you get up to speed is learn-onyx if you haven’t seen it.
is there any "large"ish deployement of onyx in the wild that could give me an idea of its scalability?
@mpenet Performance under large clusters and high message volumes is always subject to the specific workload. I would recommend benchmarking it yourself with your own data set and studying the architecture. Anything short of that won’t give you a clear enough picture.
Yes for sure. I was just wondering if you had already deployments with 10s of peers, and the battle-stories that go with that
Make sure you’re monitoring is good. 😉
@michaeldrogalis: Learn onyx is fun so far - I started going through it last night and it feels more like a puzzle than work so far, so kudos on that! No idea if onyx will be the right fit in the end for what I need to do but the thought process behind it has already got me reevaluating a bunch of previous work so it's been well worth it already. I was skimming through the beginners guide and saw the bit about high latency workflows (order processing etc. with multi day delays) - would you generally just treat it as separate onyx workflows running on separate logs or data sets and split it up that way?
@shaun-mahood Thanks! My point about tasks that have very high latency is that they will best be served by another tool that checkpoints progress at a fine grained level. Onyx checkpoints periodically - not per message. That’s how we get high performance. But if the cost of a roll-back during recovery is too high, you’re better off with something else.
Good rule of thumb is something like higher than 15-30s per message is probably too high.
Some people do it though if the cost of recovery is low. Depends on the application.
@shaun-mahood Your comment about it not feeling like work made me think of this: Hhttps://twitter.com/GonzoHacker/status/465865268112420867 Hah.
Just to clarify my understanding, is that 15-30s for each segment to move through a single task? So then Onyx would checkpoint after X number of segments have been processed through that task? Either way I'm pretty sure anything I would be doing would fall well within the performance bounds.
15-30s to pass through the entire workflow
Yeah, you’re fine then — nothing to worry about if you’re below that margin.
someone has implemented Kafka in Golang… https://github.com/travisjeffery/jocko
Ok, so if I understand correctly then your main scaling options once you get past that point would either be breaking it into multiple workflows or scaling your machines / changing your workflow to make it fit. The best time to learn JavaScript is when you need CLJS interop 🙂
@shaun-mahood Ha 🙂 And yes. When you need to scale, add more peers. Onyx will automatically parallelize the work.
I'm getting an error trying to add onyx-local-rt
to a new project using clojure 1.9.0-alpha14
, works fine with 1.8.0
- want me to open an issue or just wait until things mature a bit?
@shaun-mahood What error are you encountering? clojure.future dependency problems?
I think you need to bring in the clojure.future dependency. We’re working around not having Spec in Clojure 1.8.
Bring in clojure.future to a project using 1.9
?
Onyx core is on 1.8.
Oh yeah, that makes more sense. Want me to PR the readme with a note once I get it working?
@shaun-mahood Sure, thanks 🙂