@michaeldrogalis Okay, so I have verified that if I start two instances of the app with the same onyx id, none of the tasks in the workflow actually start

If I go back to each instance having their own :onyx/id then one of them executes tasks as normal, but the other one never gets any tasks

I read through your docs above about the architecture, which was helpful

So I wonder if there is something I need to do to trigger the second peer joining the cluster?

And is it the :onyx/id that assigns a node to a particular log in zookeeper?

So specifically, segments are pushed onto the channel but never read off the channel

(in the case of two instances with the same :onyx/id)

This is with a straightforward workflow, no jobs submitting jobs or anything

@spangler: You said channel - are you trying to use core.async in a distributed environment?

The core.async plugin is for dev only and shouldn't be used in a multi-machine environment since core.async is a local runtime thing in general

I am trying to use the onyx-template in a distributed environment yes

Ah, so the template does not work in a distributed way...

Hmm, do you have an example that does?

I was trying to get this to work before going over to kafka, but maybe I need to switch over to kafka before this will work?

I believe the production profile of the template reads from an HTTP endpoint and dumps its output to a core.async channel - which is fine but kind of useless since no one can read that.

Its reading from a channel which doesnt really make sense

You could do that. Its hard to say without seeing your logs, you might have a more basic connectivity problem

Are you referring just to onyx.log?

Like, if the peers arent receiving tasks, at all, period, its the latter

Yeah

Or the zookeeper log?

onyx.log

Okay

Do you want to see them?

Sure, I can check it out

It is fairly straightforward

Okay, here are the logs for the latest run:

So from the log it looks like it tries to start all the tasks, backs off for awhile then starts them

But it never gets to any of the actual functions

Is this from 1 machine?

Yep

Seems like all the tasks started fine, nothing is wrong there.

Do you see similar messages in the other log?

Right, except I have println's in all the tasks right when they start, and none of them fire

When you say "start", where do you mean?

When I am using just one instance, it works fine

At the top of the function

the task calls

Yeah - okay, so my guess is that Aeron's addressing is misconfigured, and the two instances can't send messages to each other. Basic network configuration hiccup

I'd investigate with standard netstat usage to make sure ports are open and Netcat or whatever to see if data is coming through. Same as you'd do for any distributed system's pieces that cant talk

Hmm, I am using the aeron setup from the template

(defn -main [& args]
  (let [ctx (doto (MediaDriver$Context.))
        media-driver (MediaDriver/launch ctx)]
    (println "Launched the Media Driver. Blocking forever...")
    (<!! (chan))))

Is there somewhere I need to configure that?

In your peer config, you specify a hostname and port

http://www.onyxplatform.org/cheat-sheet.html#:onyx.messaging/bind-addr && http://www.onyxplatform.org/cheat-sheet.html#:onyx.messaging/peer-port

The media driver supports the Aeron process, those params configure how the peers talk specifically in terms of network protocols

Ah, sorry those links are broken

Those params are under "peer configuration".

Hmm.... so does it matter what port I choose?

Or just that I have a port?

It doesnt matter, as long as its free. I think bind-addr is burning you. Again just a guess. But thats the host name that the peer advertises how to contact it

So if you're using localhost, obviously that cant work

Ah, so

If I have two instances on the same machine, I need to give them different bind-addrs?

We probably shouldnt have that in the prod template, but there's not really a good default

Yeah I see

No - you need to supply a hostname that any other node in your cluster can use to talk to it.

Its like the, "This is my IP, use it to talk to me" param

So, that makes sense for when they are deployed on different machines... but if I want to test it on my development box, how can I do that?

From the same machine I mean

Use localhost for that, but use different peer ports so they dont collide.

Ahhhh

okay

That is helpful, thank you!

What is the relationship between peer-port and peer-port-range then?

I have a peer-port-range but no peer-port currently

I don't see it in the cheat sheet

not sure where I got it

Our bad on this one. See the first note in the upcoming 0.8.0 release: https://github.com/onyx-platform/onyx/blob/master/changes.md#080

Ahhhh

Keep using peer-port-range for now, switch it to peer-port when we release. Tiny change, big efficiency gains

So will peer-port even work right now?

Not in any 0.7.x version. Its added in 0.8.0-SNAPSHOT though.

Or do I need to set them each to different peer-port-ranges to make it work currently?

Use non-overlapping ranges, I cant recall if its smart enough to not collide

It might be, give it a shot I suppose

Alright I gotta run, sounds like you're in good shape. Catch ya tomorrow!

Yep, thanks for your help!

Anytime!

@spangler: if you're going to test locally with two JVM instances on the same machine, using different Aeron ports, look up how to turn off short circuiting via the peer config. If you don't it'll lead to lost messages.

Hi

:onyx.messaging/bind-addr

How should I configure this for non-AWS environments?

cluster setup

You need to get the IP address of the interface you’re binding to. This is a good discussion of it http://stackoverflow.com/questions/9481865/getting-the-ip-address-of-the-current-machine-using-java

Alternately you could do some ifconfig shell magic and pass that in via an environment variable or command line arg

oh

got it. thanks

I have to assoc ip manually after loading peer config.

Yep

is there a job template to test out whether the cluster is setup correctly ?

Not really. We’d probably need something durable like Kafka setup so that we could actually try pushing some data between nodes and out to somewhere where we could check

I’ll put it on my list of things to consider doing though

I started out form onyx-starter

*from

trying to expand my job to multiple nodes .

Ah. You may want to shoehorn your code onto the onyx-template

https://github.com/onyx-platform/onyx-template

it’s all setup for multiple dev/prod modes

Even if you don’t try to port it over, it’d be worth creating a new project with it and have a look how you do things

ok. thanks

I'm about to write a "going to production" check list. I'll paste it in here when it's done.

That will be nice. fingers crossed.

Hi all, we finally created a production ready (or multi-node) checklist that you can run through before going to production https://github.com/onyx-platform/onyx/blob/develop/doc/user-guide/environment.md#multi-node--production-checklist

@devll see above

@yusup rather

wow ,that was quick .

👍

@lucasbradstreet: it seems to me like nearly all of these points could be checked automatically by a linter?

or at least many of them…

That’s a good point. Maybe half of them could be

Probably a bit less than that. In addition, understanding of what’s actually going on is important

We’re having a lot of users go to production recently and it’s better to get a doc up before we consider that

It’s made especially hard because we need to check a number of conditions from multiple nodes

You’re right though, there are several settings that should be configured a certain way when used in production/multi-node. I think it’d require an extra peer-config/env-config setting to be enabled before it gets linted.

I'll end up asking this a few times in the next few weeks, but can you speak up here/PM me if you're using Onyx in production or are using it internally? Compiling my Conj slides.

i reckon you’re aware by now that we’re using it

@robert-stuttaford: Indeed, got'cha 😉

@michaeldrogalis: we're in test... customer pilots in the next few weeks, full production sometime in december or january probably

@mccraigmccraig: I can stick your company's logo on the slide if you'd like.

SWEET JESUS you guys are doing everything right!

@erichmond: Hahah thanks man!