Fork me on GitHub
#onyx
<
2017-06-29
>
eelke07:06:53

@michaeldrogalis how many people use the amazon s3 plugin 0.10.x. Are there good experiences with it?

lucasbradstreet07:06:37

@eelke hard to know how many people are using it at the moment. We've used it without too many issues in the past, but for pretty restricted use cases

eelke09:06:45

We are putting up a load test now, checking if we can reproduce the situation under heavier load

matan20:06:09

Hi, new to onyx here 😊

matan20:06:34

Can I use Onyx for a message-driven architecture a la what Akka is?

matan20:06:58

Does it have a special fault-tolerance philosophy/architecture? can extra nodes be added on-the-fly to share on workload, without restarting the entire application cluster?

lucasbradstreet20:06:40

Yes, you can absolutely use it for a message driven architecture.

lucasbradstreet20:06:22

It uses asynchronous barrier snapshotting to take state snapshots that are consistent over the cluster. The process is described here https://github.com/onyx-platform/onyx/blob/0.10.x/ABS_RELEASE.md

lucasbradstreet20:06:09

Extra nodes can be added on the fly, though not currently for any peers that are processing state, as we do not support dynamic re-partitionining yet. The input, output and transformation peers can be dynamically resized, plugin dependent.

matan20:06:07

well @lucasbradstreet thanks, I think this gives me the right pointers to explore more now! 🙂

matan21:06:10

I really like what I'm seeing in the docs

matan21:06:05

Just not sure how adding a peer node to an existing cluster would look like ― the entire cluster needs to be restarted, if it's not an input or output peer but a processing peer? I am not sure where the different peer types are discussed in the user guide (http://www.onyxplatform.org/docs/user-guide/0.10.x/)

Travis21:06:08

you essentially turn on the new node that uses the same tenant id I believe

matan21:06:57

mmmmm I'm not quite seeing how elasticity works here.

michaeldrogalis21:06:12

@matan You can add machines to the cluster dynamically. What @lucasbradstreet was saying is about adding more resources to a running job.

michaeldrogalis21:06:43

If a job is doing stateful operations, more available resources won’t get allocated to tasks that accrete state at the moment. You’re probably not going to run into that case much.

matan21:06:09

@michaeldrogalis Thanks for putting it in perspective

matan21:06:32

I should probably start with better understanding workflow definitions, in http://www.onyxplatform.org/docs/user-guide/0.10.x/#_workflow

michaeldrogalis21:06:40

To be a little more direct: peers get added to the cluster by connecting to a common ZooKeeper address, and, as @camechis says, using the same tenancy ID. They’ll start picking up work automatically from there.

michaeldrogalis21:06:21

Also, regarding this particular topic, almost all the content in this talk is still accurate as of the latest release: https://www.youtube.com/watch?v=KVByn_kp2fQ&amp;t=2s

matan21:06:23

That's easy to understand, thanks. I guess the tenancy ID is like a cluster ID (or application ID)

matan21:06:35

Will watch the video first thing in the morning

matan21:06:46

Just one question till then, regarding elasticity. in http://www.onyxplatform.org/docs/user-guide/0.10.x/#_workflow, do the nodes of the edges represent a "worker type" or a specifically named "worker instance"?

michaeldrogalis21:06:54

Nodes in the workflow graph represent tasks. Tasks are fanned out across peers, which are closer to a worker. Each task can run on one or more peers concurrently depending on how you configure the scheduler.

matan21:06:16

Great, thanks, I think I'm all set!