This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (31)
- # boot (5)
- # cider (1)
- # clara (1)
- # cljs-dev (9)
- # clojure (118)
- # clojure-nl (2)
- # clojure-russia (90)
- # clojurescript (344)
- # cursive (4)
- # datascript (1)
- # datomic (41)
- # emacs (5)
- # hoplon (48)
- # ldnclj (13)
- # lein-figwheel (13)
- # leiningen (1)
- # off-topic (1)
- # om (146)
- # omnext (1)
- # onyx (65)
- # re-frame (22)
@aaelony: We haven't used Onyx to read data out of S3 in a serious manner. I really cant speak to that. The plugin is defunct for the mean time though
@michaeldrogalis: okay, thank-you. would that mean that bringing down an s3 file locally then using
onyx-seq is the way to go for now?
Hello everyone, I am new to using http://onyx.It seems it seems i have misunderstood something , I found if functions which defined in catalog at :onyx/fn key raise any exception , job would stop. Is there one entry can caught all of Exceptions in the job? thanks
Hi @michael.wong, try using the flow conditions functionality to handle exceptions in your task https://github.com/onyx-platform/onyx/blob/0.8.x/doc/user-guide/flow-conditions.md#exceptions
@lucasbradstreet: thanks for your help , it seems just can catch the exceptions in the defined workflow. Is there one entry to caught all ?
@michael.wong You can use flow condition from :all to whatever task you want to handle your exceptions
thanks , @lucasbradstreet: this remind me : ) and I will try whether it is possible to use from :all to :all
that is this , I figure out that once any of my tasks in a job raise a exception, the job will stop. I wonder this is onyx normal behavior ? And I guess if I can catch every exception at upstream in the job , it would not crash.
That is normal onyx behaviour, but flow conditions all to a task or many tasks will allow you to send the segments causing exceptions on to another task. All to all would mean sending all exceptions and corresponding exceptions to all tasks, which wouldn't make much sense.
It won't kill the job if you use flow conditions exception handling in all your tasks
comm on scene if raise a exception , I just want to send to me log (eg: send email etc) and ignore,not crash.
if I use flow condition exception handing , I think I have to write handle in every flow.
we are looking at hooking together Docker processes and this looks like a very similar problem
@raymcdermott: Absolutely. We’re iterating a bit on onyx-java which will provide a more natural interface for Java (and other JVM languages). Internally we support java plugin implementations and function implementations, but we want to make sure the API is right
ideally we would like a stackless approach where we can just send data around (via Kafka)
but great that you are looking at Java … if you also did JS I would pretty much be sold at this point 😉
@raymcdermott: ultimately converting some json to clojure data should be pretty easy, so that’s something we will definitely have some libraries around in the future
Yeah that part is going to be super tricky. It’d be easier to use something like nashorn to evaluate, but I hear there are some issues there too
You still need the onyx/fns to evaluate within the context of the distributed environment, which is going to be JVM based for the forseeable future.
@raymcdermott: I'm interested in pursuing it earlier than expected if someone wants to help out. It'd be a good foil to AWS Lambda.
@michael.wong: That might be an exception in Onyx. I'll take a look in a few minutes.
@michaeldrogalis: where could I start (I also have a few buddies at work who could be willing to join in)
ok - we’re not mega deep on the internals but know enough node.js to be dangerous!
Excellent. Even something we can just play around with and explore would be a great step.
@michaeldrogalis: super… I will let you have some holiday and we can chat again on Monday! It’s a great validation of our thinking to be honest that you quickly brought up AWS Lambda, so I’m sure our thinking is aligned
@raymcdermott: I look forward to it. Gotta run, have a good Tuesday everyone!
@lucasbradstreet: @michaeldrogalis just jumped on the clojurescript channel and one call call node programs from CLJS quite easily, so I guess the ‘problem’ is a provisioning issue for node.js programs?
@raymcdermott: off the top there are a number of difficulties. We could probably spin up / manage node processes but we'd likely still need messaging between the JVM and node or some kind of way to use Aeron directly from node
With the new C++ Aeron implementation that may be possible to do in the future. It'll likely be way too much work to be feasible for us to work on though. @michaeldrogalis may have other ideas about how this could be done though.
Nashorn JS functions are probably the easiest short term bet but compatibility with node may be a bit hit and miss.
It would be enough, in my mind, to be able to call JS functions where we currently either call Clj or Java functions. Coupling that with first class lifecycles and plugins would do the trick. Not to say that that's easy, but it's about as far as I think we should take it.
I agree with all of this. Re-implementation is definitely not a priority. In practice I think this approach limits us to Nashorn, but if anyone has ideas later I’m all ears