Fork me on GitHub

hey! I'm writing here only when I have problems, but still: my tests (on Onyx 0.9 still, hehe) have a weird problem: One task generates a list of combinations, and another one fetches them from ES to put as cache in Cassandra. The thing is, because of some change we have 1167 combinations instead of 248, which causes test to retry infinitely. I added 5 min :onyx/pending-timeout to input task (which is onyx-seq from a file), and didn't help. It takes about 50 seconds for lein test my-test to start up and go through 500 combinations, but if I remove that restriction, it just re-runs and re-runs. Anything I should look at?


ok, so 990 combinations are ok, and 1020 are not ok already


anything related to 1000 somewhere? 🙂


@asolovyov: what are you using as an output?


Are you not draining the buffer?


(def default-channel-size 1000)


that's what happens when you blindly use templates!


I swear I'm going to re-read it all


Oh heh, yea they're mostly made for testing with small-ish data sets and assume that to be the case.


yeah, well, we've stretched it a bit


I usually just write something quick to drain the channel to a log or file or something.


yeah, we're doing thingie from test example still:

(helper/feedback-exception! peer-config job-id)
        (let [segments (into [] (take-segments! (:write-filters-cache chans)))


Just checkin in on the current state of Java interop. I’m trying to get Onyx used in NOAA (specifically in the climate data corner) which would require java interop (onyx->Java->C++->FORTRAN actually). There is this: But not a lot of other convo other than saying its a low priority. Is the above a good place to start? What’s missing since we will put in some work on this if its not too daunting


@georgek “onyx->Java->C++->FORTRAN” I got dizzy reading that sentence.


We haven’t made Java integration much of a priority, and the interface bindings are lagging behind for 0.10 — mostly because of a lack of demand tbh.


I hear that! I guess I’m wondering about the scope needed to take the above repo and get it up to date


Specifying :onyx/language :java in a catalog entry should still work. If you need Java bindings for plugins that lifecycles, that’s a larger task.


There are 2 of us (potentially 3) that could bang it out if its not terribly painful


Yeah, I suspect we may need lifecycle support


I can answer that, but backing up - which pieces of Onyx do you need to use from Java? That’ll help me answer your question.


Okay, yeah - that’s a bigger task then.


For starters basically its going to just be triggering tasks that are backed by java (and native underneath)


Anything that does resolution from a keyword into a function will need to be able to specify a Java method rather than a Clojure fn and resolve it correctly.


So, I guess that for basic usage as task handlers we are pretty close to good to go then?


Yeah, should be.


Kew kew. Just out of curiosity then, how big of a job to complete a full java API ?


(starting with the above repo)


Probably a week IMO.


It’s going to be easy once we figure out the general pattern - it’s all grunt work after that.


Kewl, that’s totally doable given the advantages to us even just from a selling this project point of view


Sweet, how should we go about that? Suggestions on where to start?


Hmm, here’s another question. Is the intent to also build the Onyx job’s in Java? Because another thing you can do is write the job itself in JSON and write a shim to turn it into EDN.


It’s really a matter of whether you want to use Java to specify the behavior/functions or the job itself, which is just data.


Right, we may end up doing runtime adjustment’s of the workflow in Java (our business logic is going to be in java that will use Onyx to run jobs) so while this would work for job construction we will want to have high-level control over the jobs running in the system which, I’m assuming, would need a Java API?


Basically we can’t use straight up Clojure (too new and scary for the gov’t) so we are emulating the declarative approach in a Java system that triggers native (legacy) code


Gotcha. Yeah, I mean in the case, doing what I started doing in onyx-java is probably a viable route. All it’s doing is making a fully static API to build up data structures on each method call.


Once you land at a fully formed job, make a call out to :onyx.api/submit-job — which should work okay if it’s casting step-by-step as you go


Sweet, lemme talk it over with the team and we will give it a go!


That seems like the most attractive thing internally. We def. want to avoid not really having a clean interface layer when we talk to the workflow system!


(Given the whole Java thing)


Cool, keep me posted. Thanks for the interest, been looking to put this one to bed for a very long ago. 😛


Will do! I’m excited