Clojurians
#onyx
<
2016-03-07
>

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

acron11:03:35

lucasbradstreet: I am also looking at integrating Onyx with input from an HTTP endpoint - I was going to go down the route of using an async chan to shovel the requests into an Onyx workflow, but would using onyx-seq be a better sln do you think?

lucasbradstreet11:03:13

@acron: placing the requests on an async channel is fine but not very fault tolerant / clusterable as you will have to do it on the node that is allocated the task. You could do this by pinning that task via tagged constraits but you're probably better off just using onyx-seq. Do you have a way to re-fetch results in a node goes down? I ask because I need to know whether it might be possible to write a more fault tolerant input plugin

lucasbradstreet11:03:10

If you could explain a little more about how the web service works it might help

acron11:03:16

lucasbradstreet: you're right, my explanation was bad sorry. And I actually liked your idea about using Kafka as an input - so I'd have a web service take 'CQRS' commands, write them to Kafka and then an Onyx node can pick them up from there.

lucasbradstreet11:03:22

@acron: I think that approach would be fine. You just need to make sure to monitor your service that transfers from the web service to Kafka, since it’s the lynchpin

acron12:03:00

cheers lucasbradstreet

acron14:03:10

I'm trying to use the kafka plugin but it doesn't seem to read messages off my topic. No complaints during startup and kafkacat sees the messages, but the logging around my task doesn't seem to be kicking in which I presume means there's nothing being read..

gardnervickers14:03:22

@acron: Are you seeing the peers start their jobs?

gardnervickers14:03:41

in onyx.log something like 16-Mar-07 09:48:34 Gardners-MacBook-Pro.local INFO [onyx.peer.task-lifecycle] - [732a9b65-1674-46db-b80e-dcc493d9234c] Enough peers are active, starting the task

gardnervickers14:03:53

Are you using the lein template?

acron14:03:59

I'm not, no, I am just evolving one of the onyx-examples. I had a go with the lein template but it opted to go for something contained in a single file, just so I can understand it better whilst I'm still toying

acron14:03:27

I'm also using my own kafka+zk rather than the embedded servers

acron14:03:40

I tried the embedded Kafka with my Zookeeper but still no dice

lucasbradstreet14:03:24

@acron are you using the same ZooKeeper that Kafka is connected to with Onyx?

lucasbradstreet14:03:56

Maybe Kafka is writing its information to a different ZooKeeper than the one that the Kafka plugin is using for discovery

acron14:03:08

Yeah, I set :zookeeper/server? false and provided local creds (`127.0.0.1:2181`)

acron14:03:55

I might tear it all down and try a tiny example with embedded servers first then

lucasbradstreet14:03:57

that might be a good idea

lucasbradstreet14:03:15

anyway, I suspect it’s something ZK related

acron14:03:29

I would like to use the lein template, it's just a bit scary :wink:

gardnervickers14:03:45

Yea understandable

gardnervickers14:03:49

we’re working on that

acron15:03:35

Wow, ok, it was me. Forgot to add the lifecycle snippet for onyx-kafka :facepalm:

michaeldrogalis15:03:06

@acron: S'all good. Gotta bump your head a few times before you learn your way around. :simple_smile:

lucasbradstreet15:03:20

that'll be one upside of the new task bundles, as it'll be impossible to forget that

acron16:03:23

Just noticed something curious. I defined my workflow as [[:a :b] [:b :c] [:b :d] [:d :e]] where :c is an output/leaf. I added a flow-condition {:flow/from :b :flow/to [:d] :flow/predicate :pred?} and this seems to prevent any flow to :c even though I'd assume the :b->:c relationship is unaffected by the flow condition.

acron16:03:35

If I add a flow condition like this, am I obliged to be explicit about everything else that :b flows to?

lucasbradstreet16:03:59

I always forget about how this works, but the idea is that enabling flow conditions for that task will default to flowing out to the flow/to tasks you define, unioning the to tasks as predicates are matched for all the flow conditions you define

michaeldrogalis16:03:20

@acron: Yes, it requires you to be explicit.

acron16:03:06

Cool, thanks

lucasbradstreet16:03:16

Doing it this way is a lot more composable, though it does force you to be more explicit

acron16:03:10

I've seen a couple of examples of Onyx workflows and catalogs that involve writing to a Kafka topic and then reading from the same topic as the next step (http://yuppiechef.github.io/cqrs-server/, half way down the page). Other than for the purposes of visualising and maintaining a single workflow, is this even a good idea?

acron16:03:16

Are there advantages or disadvantages to maintaining two workflows at this point?

gardnervickers16:03:01

@acron: Independent starting/stopping is one advantage of having two separate jobs (workflows).

acron16:03:10

I think I'd like the idea of two separate jobs in this scenario

michaeldrogalis16:03:19

@acron: I tend to prefer it as well. Just a heads up, that CQRS project you linked to is super old. Hasn't been updated in about a year.

acron16:03:02

@michaeldrogalis: thanks for the warning. I haven't actually looked at the code, but trying to get inspiration for methodology

gardnervickers16:03:09

@acron: A ES/CQRS project based on Onyx today would look quite a bit different.

gardnervickers16:03:32

We have a stateful aggregation model now that did not exist back then.

gardnervickers16:03:07

Ah gotcha cool. Let me know if you have any observations/suggestions. Its always good to get a new set of eyes on this stuff.

acron16:03:35

Cheers guys, this is really helpful

michaeldrogalis16:03:39

Yw :simple_smile:

robert-stuttaford21:03:06

@lucasbradstreet, @michaeldrogalis: our new cluster is live :simple_smile: 3 x ZK, 3 x Onyx. finally!

bridget23:03:10

Congratulations!!