Definitely write your workflow as a test using mock data and inputs/outputs first

Or if you have access to your live data source, you can just read it in through the appropriate Onyx plugin locally while you write your test.

sorry feel like i asked this before but forgot to record it down (that and some other bits). Does it make more sense to start a project that intends to end using kuberentes with onyx-template, onxy-starter or something else. Im reading over the project files now to get a sense on how things are setup.

I would definitely go with the Onyx Template. It gets you started with a Dockerfile and docker-compose.

On top of that, I am currently working on the docs to go along with a full Kubernetes deployment. Things have been rather busy lately so it’s slow going. It’s being developed in the open here https://github.com/onyx-platform/onyx-twitter-sample/tree/master/kubernetes Most everything works as far as K8s manifests go, it’s just that to use it requires a fairly comprehensive understanding of both Onyx and Kubernetes at this point.

It mostly just needs a final good push to tidy up some loose ends and piece together the disparate documentation. Expect something more concrete in the very near future.

@drewverlee: I sense you’re asking that question to try and be as future-proof as possible, if that’s the case then going with the Onyx Template as it stands now, and following the task bundle pattern will get you there 😄

@gardnervickers: Thanks a ton for the insight. futureproofing was one aspect of it. My question was mainly because of my nativity with the to-production story. I suppose, that story depends greatly on your needs though. Its been helpful to read. Just to see someone specific example and thought process https://dataissexy.wordpress.com/2016/07/31/using-onyx-template-to-craft-a-kafka-streaming-application-part-1-clojure-onyx-kafka-streaming-data/

Ah great good to hear!

Is there a strong reason that :onyx/params uses keywords into the catalogue? It's an interesting idea.

As opposed to a vector of args?

@lucasbradstreet: Yeah.

I think you'd have to ask Mike, but my guess would be to make the data in the catalog entries more composable, e.g. you can just merge task map opts in to your task map, without having to check whether you have to mess with an args vector

It also allows tasks to be supplied or without without a fn, where the args are named, which is a kind of documentation

That seems to make sense. I guess it makes it easier to add arguments in varying orders too.

Do you know where the idea for the catalogue came from? I'm interested in implementing something similar in a library.

For a very different domain. But it's just such a nice idea.

You'd have to ask Mike :)

Haha, okay. Thanks 🙂

@dominicm: Once you separate structure from the equation (the workflow), you have to do something with all the parameterization that it used to be joined with. Thus born, the catalog.

@michaeldrogalis: It's an awesome structure. It's something I've been grasping at mentally. Seeing it realized in Onyx made it finally click for me.

I'm currently building a validation library built on a similar split.

@dominicm: Cool. 🙂

I just think it's such a good way to bring declarative programming to Clojure.

@dominicm: One of the upshots is that, since our entire API is described in a big Clojure map, we were able to generate a Datomic schema pretty quickly, then load in tasks exactly as they are.

@michaeldrogalis: Not dissimilar from the benefits I'm seeing in this domain. I hadn't even considered it's application for datomic schema generation yet.

Clojure.spec generation, too, for client & server.

Yep. That's my favourite one. Especially as the equation is serializable, the client can take the equation, and figure out what it needs from it.

It's been good stuff. :thumbsup:

@vijaykiran: Upgrading the docs may be a weekend activity. Lots to do today. Haven't forgotten about it. 🙂

It looks like Onyx yells at me if I specify a :onyx/type :function task with no outbound edge in the graph. Is there a way to specify a function task with no out edge, whose only purpose is some side effect with no result to pass downstream?

There’s a good example here: https://github.com/onyx-platform/learn-onyx/blob/master/src/workshop/challenge_6_0.clj Onyx calls them “Leaf Functions"

you can just set :onyx/plugin :onyx.peer.function/function

ah yes

it’s not fantastically documented unfortunately. We should make a better error message

ah awesome, thanks guys!

Any ideas on why onyx test env would hang? Also no onyx.log is being produced

hmm, tried putting in that plugin but still getting:

------ Onyx Job Error -----
There was a validation error in your workflow for key :handle-lifecycle
[
   [:read-input :dispatch-events]
   [:dispatch-events :handle-lifecycle]
                     ^-- Intermediate task :handle-lifecycle requires both incoming and outgoing edges.
...

@aengelberg: See "Leaf functions" for an exact spec - http://www.onyxplatform.org/docs/user-guide/latest/functions.html

@camechis: No idea, sorry. Doesnt sound like it actually got off the ground if the log isnt writing

oh I see, it has to be a :onyx/type :output.

@michaeldrogalis: Yeah thats my thought, I know didn’t give you a lot to go on, lol

So my onyx job with the onyx test env is running ( although having issues somewhere in the middle ) but no onyx.log is being created. Any ideas on what would cause the log to not be produced at all?

What does your config look like?

which part?

peer or env or both?

onyx.log/config

ah, its set to warn? Is there any warn output?

Try not setting it

use the defaults

just switched it to trace in the config. So looks like this

:onyx.log/config #profile {:default {:level :trace}
                             :docker {:level info}}}

for both peer and env

That forces the default timbre appenders, writing to stdout

set :default nil

and then you’ll get logging out to onyx.log

gotcha

we set {:level info} there so when using docker we write to stdout as is customary

yep

that makes sense

hmm set it to nil but still nothing in the log

:onyx.log/config #profile {:default {:level nil}
                             :docker {:level :info}}

:default nil

ah!

lol

there we go

thnx!

just upgraded to 0.9.9.0 and i'm seeing this from onyx-kafka on startup - https://www.refheap.com/121943 - does that look familiar ?

I’m waiting for it to load

but is it possible you’re using kafka 0.8

yep

you need to use plugin onyx-kafka-0.8

oh, lol - duh!

i've upgraded production, just haven't upgraded my laptop

We probably should’ve done a better job making sure that the upgrade threw an error there

I don’t know if we even mentioned it in changes.md. Sorry

it is a bit non-obvious 🙂 - thanks for the help @lucasbradstreet

Is there an easy way to tell if multiple peers are properly communicating?

I just scaled up on mesos but I am not sure if the peers are connecting to each other or if they think there each running standalone

Have your nodes each spawn one or two peers and run a job that requires more than one or two peers.

But tbh, as long as your bind_addr is routeable from all other nodes in the cluster you should be fine

I scaled up 3 nodes with 5 virtual peers each

I see that in the dashboard it is showing 15 peers

What are you setting BIND_ADDR to?

or :onyx/bind-addr

hmm, i am using the template i need to check how that is getting set

It defaults to hostname

yeah, i just looked at my scripts

so I am running inside Docker via marathon

trying to think of what that should be

6215de8a6d98        peerimage:1.0   "/init opt/run_peer.s"   24 minutes ago      Up 24 minutes       3196-3198/tcp, 40200/udp, 0.0.0.0:7250->40200/tcp

Does Marathon have any tools for service discovery? Container identity?

yeah it has mesos-dns which gives you srv record, There is also VIP which allows you to set an IP:PORT combo that can be used ( it will be load balanced with this )

potentially could also set the networking to Host instead of Bridged

Kubernetes provides an ip address through the “Downward API”, which is what we use. They also allow creating services that provide srv records to container’s based on a selector system. You’ll need something like this for Marathon.

right, just so I make sure i understand if these were physical boxes how do the peers discover each other and then communicate?

bind_addr needs to be set to something that other peers in the cluster can talk to. So for bare metal that would be an IP address or a hostname

trying to get all the networking straight in my head

do they talk on the aeron port ?

@camechis: Peers advertise their address through a log entry, the address specified by bind_addr & the Aeron port. The host and port of each peer then gets stored in the replica.

ok thats kind of what i was thinking

is there any easy way to see what the peers advertised? Then i might be able to figure out how to fix it

{:tags nil, :group-id #uuid "edf849e8-1ede-4890-a096-a40629e6c8d2", :id #uuid "5a9c313e-87c8-494d-8646-79570da5f95c", :peer-site {:aeron/port 40200, :aeron/external-addr "localhost”}}

If you’re not setting BIND_ADDR when launching your containers, it is whatever hostname is in your container.

from the dashboard

my guess is

aeron/external-addr “localhost

Yea I’m not sure what you’re exact setup is

you know what, I can probably query the mesos-dns to get what its srv record is

inside of the docker container

and then set the BIND_ADDR

@camechis: If you use localhost, all peers will try to talk to themselves since they're advertising their own address.

exactly

@mccraigmccraig: You may have the title for longest running Onyx production deployment. Not sure if @robert-stuttaford got off the ground before you. But it's been a while. 🙂

it has indeed been a while @michaeldrogalis - i started cutting code july 15 and iirc we hit our first production version in dec 15. it's been a pretty straightforward experience too - onyx pretty much always does what it says on the tin :D

@mccraigmccraig: Glad to hear its been smooth 🙂

So we have 3 peers deployed with 6 virtual peers and we can see a log where it connected to kafka however nothing happens. If we launch the job with 1 peer and enough virtual peers it starts processing data. Anyone have any ideas on what to look at here? Still wondering if our aeron communication is working right with some new changes

@camechis: Almost certainly sounds like an Aeron problem. Is the UDP port open?

If it works for 1 but not for > 1, there's a network communication issue.

@michaeldrogalis: thnx will look into it more. Think we are having issues getting the ephemeral IPS and ports for docker mapped right through mesos