@hugesandwich I believe what you refer to as "workflow" is not necessarily an onyx workflow.

What's the preferred way to continuously feed onyx input with DB records that are yet "unhandled" and keep growing, like reading from kafka but instead from a table? Should I write a plugin or is it better done inside a task?

More specifically, I have an event store that I keep adding events to, and I should be able to say to onyx "start from this snapshot and keep going forever"

Are there any working examples of the S3 plugin for writes on github anywhere. A lack of docs on things like credentials and whether the client needs starting (ie like it is in the unit test) is making things a little frustrating. Thanks in advance. J

@jasonbell We did some work on the S3 plugin in the last week or so, changes waiting to go out. on [org.onyxplatform/onyx-amazon-s3 “0.9.15.1-SNAPSHOT”]. The output task is reasonably straightforward: https://github.com/onyx-platform/onyx-amazon-s3/tree/master#output-task Credentials are discovered via the AWS SDK: http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

The client is only getting booted up in the test so that we can go and retrieve the files on S3 and make sure the writes are in tact.

@yonatanel Where do the records concretely live? It’s the job of the input plugin to partition the corpus of data into zones that can be read, and Onyx core takes care of backpressuring and invoking the plugin when it needs more input.

@michaeldrogalis Yeah it was the creds stuff that I couldn't see so I backtracked through the AWS client. Cheers.

Sure

@michaeldrogalis The records live in Couchbase, and they are sequenced (have a sequence number) per "aggregate" and also timestamped.

@yonatanel Would be most appropriate to use an input plugin then, but hold off until the ABS technical preview is out since the interface changed substantially. @lucasbradstreet is rewriting the Kafka and S3 plugins against the new interface today or tomorrow. Not sure how far he got last night.

I see, then I might have to rely on kafka for now

@jasonbell: thanks for the heads up on the s3 plugin. I didn't document how to add the credentials because it tries all of amazon's supported methods but I can see how it's non obvious still

@lucasbradstreet Do you use franzy for the new kafka plugin?

yes, but I think we may switch to the raw java client, as franzy doesn’t seem to be being maintained

I see. Thanks

franzy is well written, but I think it’s more trouble than it’s worth for onyx-kafka given the maintenance issue

so my plan for lambdas and Onyx is still on, I think Onyx's abstractions are in the right place, especially regarding workflows and data passing from function to function

of the architecture I picture in my mind I think Onyx adds two three things I might not need at the moment: Aeron, ZooKeeper and Java

I was actually wondering if an onyx-lite version would make sense 🙂

@richiardiandrea None of those 3 things are present in onyx-local-rt (if you’re running in JS). What do you envision being different?

Would this lambda version be distributed?

Or would it be local-rt running on one process as the result of a lambda trigger?

true, so in my head each task in onyx-local-rt is actually a lambda

so I would probably need an external coordinator for sending data (json, transit) back and forth to lambdas

SQS might work for you, I know it can trigger lambdas.

declared using the vector of vector format for building the graph

yep

it still need some external trigger on events

i mean someone sending a "go" message on SQS right?

and it looks like my coordinator is very similar to onyx but without ZooKeeper, only the ability to trigger workflows

Your peers can only be alive for 60s too

mmm...yes lambdas are limited in time, this means I need to consider failure, I guess this is also the external coordinator's job...

ok it definitely became onyx at this point 😉

I think the main selling point of Lambda would be eliminating the need for Onyx to handle resource scheduling. The larger idea would be to compile and Onyx job to a set of entrypoints in a jar or js file, then package those entrypoints into containers/Lambda slugs. That way Lambda/Kubernetes/Mesos could handle resource management and scheduling. For the JS versions you’d obviously need to provide an Aeron substitute like SQS.

Interesting to think about

yep definitely, the "trigger" usually comes from an external event like a hit on the Gateway API

also, you could use Kinesis instead of ZooKeeper for the execution log of the peers

with the right abstraction in the right places, onyx could definitely run on AWS (leader election would be implemented in onyx though)

but ok that's a lot of work

😄

Kinesis isn’t a viable substitute for ZooKeeper. Onyx uses ZooKeeper for things other than it’s sequential znode facility. We’re using watches for failure detection and znodes < 1 MB for small storage of job metadata.

@lucasbradstreet no worries at all, I’m still in the explain like I’m five (eli5) camp of documentation. 🙂

ok that makes sense, I only watched your videos Mike and don't have the whole picture, the AWS tools are so many though and I would not be surprised if they had similar features grabbed from here and there (probably using ZooKeeper under the hood)

@richiardiandrea It comes down to whether or not the goal is to use Onyx itself for distributed activity. If that’s the case, you’re unlikely to substitute anything for ZooKeeper or Aeron that’s an adequate match. If the goal is to use Onyx’s model you have more flexibility, at the expense of choosing the right tools to cover as much fault tolerance as you need.

Thanks yes, that's right...and for sure not after having investigated better the implementation which you guys for sure have put more thoughts in than me 🙂

Distributed systems really don’t want to work properly.

Hi all, what's the best way to catch peers start error? like zookeeper offline, etc

@yonatanel What I was describing normally works just fine in Onyx, I have tried it. The problem is just in any feature added on related to mutations. Most things like toggling a task on/off can be done, but the implementations are hacky. My goal is try to avoid reinventing the wheel and to also save time, so I'm trying to actually eliminate that Onyx can do some of these things. Regardless, I use Onyx for plenty of other things, so the more I can actually do it the better since the cluster and maintenance is already in place.

To be clear, I mean mutating the order of the workflow, the tasks in the workflow (adding/removing), and the parameters of each task. Some of these can be done or work for certain kinds of tasks, or there are otherwise workarounds but there is often a cost even when the answer is "yes, it will work."

@lellis Run your peers under a process supervisor such as Monit, or use a container and an orchestration system like Kubernetes to restart failed containers.