@aaelony: We haven't used Onyx to read data out of S3 in a serious manner. I really cant speak to that. The plugin is defunct for the mean time though

@michaeldrogalis: okay, thank-you. would that mean that bringing down an s3 file locally then using onyx-seq is the way to go for now?

@aaelony: That would work great, yes.

super awesome

thanks

No prob. Let us know if you need anything. 😄

Hello everyone, I am new to using http://onyx.It seems it seems i have misunderstood something , I found if functions which defined in catalog at :onyx/fn key raise any exception , job would stop. Is there one entry can caught all of Exceptions in the job? thanks

Hi @michael.wong, try using the flow conditions functionality to handle exceptions in your task https://github.com/onyx-platform/onyx/blob/0.8.x/doc/user-guide/flow-conditions.md#exceptions

@lucasbradstreet: thanks for your help , it seems just can catch the exceptions in the defined workflow. Is there one entry to caught all ?

@michael.wong You can use flow condition from :all to whatever task you want to handle your exceptions

thanks , @lucasbradstreet: this remind me : ) and I will try whether it is possible to use from :all to :all

Why would you want all to all? There might be something I'm missing

that is this , I figure out that once any of my tasks in a job raise a exception, the job will stop. I wonder this is onyx normal behavior ? And I guess if I can catch every exception at upstream in the job , it would not crash.

That is normal onyx behaviour, but flow conditions all to a task or many tasks will allow you to send the segments causing exceptions on to another task. All to all would mean sending all exceptions and corresponding exceptions to all tasks, which wouldn't make much sense.

Hmm, what I've said isn't so clear

It won't kill the job if you use flow conditions exception handling in all your tasks

comm on scene if raise a exception , I just want to send to me log (eg: send email etc) and ignore,not crash.

if I use flow condition exception handing , I think I have to write handle in every flow.

This why I want handle exception in one place : )

That's what flow/from :all is for.

Oh thanks for you great help , I will try in my project : )

Then use flow/to :none to send them nowhere :)

No problem :)

thanks for bring us this great project !

guys - are there any plans to enable Onyx beyond CLojure?

we are looking at hooking together Docker processes and this looks like a very similar problem

@raymcdermott: Absolutely. We’re iterating a bit on onyx-java which will provide a more natural interface for Java (and other JVM languages). Internally we support java plugin implementations and function implementations, but we want to make sure the API is right

ideally we would like a stackless approach where we can just send data around (via Kafka)

@michael.wong: hmm, that might be a bug

but great that you are looking at Java … if you also did JS I would pretty much be sold at this point 😉

@raymcdermott: ultimately converting some json to clojure data should be pretty easy, so that’s something we will definitely have some libraries around in the future

ok, the other aspect is how to call into node.js modules, yes?

@lucasbradstreet: you mean probably bug in my code or in onyx ?

Yeah that part is going to be super tricky. It’d be easier to use something like nashorn to evaluate, but I hear there are some issues there too

maybe via CLJS?

You still need the onyx/fns to evaluate within the context of the distributed environment, which is going to be JVM based for the forseeable future.

@raymcdermott: I'm interested in pursuing it earlier than expected if someone wants to help out. It'd be a good foil to AWS Lambda.

@michael.wong: That might be an exception in Onyx. I'll take a look in a few minutes.

thanks

ha - that’s right Michael - that’s the exact model that we were thinking around

@michaeldrogalis: would love to help

@michael.wong: Confirmed bug, I'll get a fix out tonight: https://github.com/onyx-platform/onyx/issues/464

@michaeldrogalis: where could I start (I also have a few buddies at work who could be willing to join in)

@raymcdermott: Let me put a plan together and I'll ping you on Monday. Step 1 is going to be some sort of a design proposal. It'd be great having ya'll helping with this part, I'm out of my league with Javascript engines.

ok - we’re not mega deep on the internals but know enough node.js to be dangerous!

Excellent. Even something we can just play around with and explore would be a great step.

@michaeldrogalis: super… I will let you have some holiday and we can chat again on Monday! It’s a great validation of our thinking to be honest that you quickly brought up AWS Lambda, so I’m sure our thinking is aligned

@raymcdermott: I look forward to it. Gotta run, have a good Tuesday everyone!

@michaeldrogalis: great ,excellence work : )

good night, everyone

@lucasbradstreet: @michaeldrogalis just jumped on the clojurescript channel and one call call node programs from CLJS quite easily, so I guess the ‘problem’ is a provisioning issue for node.js programs?

the nodyn project suggests that this can be done on the JVM http://nodyn.io

oops http://nodyn.io is dead 😞

bad idea anyway to host node.js on a JVM - probably just done to prove a point

@raymcdermott: off the top there are a number of difficulties. We could probably spin up / manage node processes but we'd likely still need messaging between the JVM and node or some kind of way to use Aeron directly from node

With the new C++ Aeron implementation that may be possible to do in the future. It'll likely be way too much work to be feasible for us to work on though. @michaeldrogalis may have other ideas about how this could be done though.

Nashorn JS functions are probably the easiest short term bet but compatibility with node may be a bit hit and miss.

I didn’t realise that you had such a strong dependency on Aeron

ok, let’s see if Michael has some other ideas as you say

@lucasbradstreet: I dont want to go so far as reimplementing Onyx in JavaScript. In that respect, there will always be a JVM in the equation.

It would be enough, in my mind, to be able to call JS functions where we currently either call Clj or Java functions. Coupling that with first class lifecycles and plugins would do the trick. Not to say that that's easy, but it's about as far as I think we should take it.

@michaeldrogalis: @lucasbradstreet would be nice to understand what the functions need to support in terms of in / out params and error handling

Yeah, this is where more planning is needed. I can't name everything off the top of my head. Ideally, you'd be able to write everything you could in Clojure but in JavaScript, and still rely on the Onyx Clojure implementation to do all the heavy lifting. So I guess that's the goal.

I agree with all of this. Re-implementation is definitely not a priority. In practice I think this approach limits us to Nashorn, but if anyone has ideas later I’m all ears

Yeah, I was wondering if that's the case. This is where my JS knowledge runneth dry.