Fork me on GitHub
#onyx
<
2017-02-14
>
asolovyov12:02:03

hey! I'm writing here only when I have problems, but still: my tests (on Onyx 0.9 still, hehe) have a weird problem: One task generates a list of combinations, and another one fetches them from ES to put as cache in Cassandra. The thing is, because of some change we have 1167 combinations instead of 248, which causes test to retry infinitely. I added 5 min :onyx/pending-timeout to input task (which is onyx-seq from a file), and didn't help. It takes about 50 seconds for lein test my-test to start up and go through 500 combinations, but if I remove that restriction, it just re-runs and re-runs. Anything I should look at?

asolovyov13:02:47

ok, so 990 combinations are ok, and 1020 are not ok already

asolovyov13:02:57

anything related to 1000 somewhere? šŸ™‚

gardnervickers13:02:36

@asolovyov: what are you using as an output?

gardnervickers13:02:39

Are you not draining the buffer?

asolovyov13:02:45

(def default-channel-size 1000)

asolovyov13:02:15

that's what happens when you blindly use templates!

asolovyov13:02:31

I swear I'm going to re-read it all

gardnervickers13:02:50

Oh heh, yea they're mostly made for testing with small-ish data sets and assume that to be the case.

asolovyov13:02:40

yeah, well, we've stretched it a bit

gardnervickers13:02:47

I usually just write something quick to drain the channel to a log or file or something.

asolovyov13:02:03

yeah, we're doing thingie from test example still:

(helper/feedback-exception! peer-config job-id)
        (let [segments (into [] (take-segments! (:write-filters-cache chans)))

georgek17:02:55

Just checkin in on the current state of Java interop. Iā€™m trying to get Onyx used in NOAA (specifically in the climate data corner) which would require java interop (onyx->Java->C++->FORTRAN actually). There is this: https://github.com/onyx-platform/onyx-java But not a lot of other convo other than saying its a low priority. Is the above a good place to start? Whatā€™s missing since we will put in some work on this if its not too daunting

michaeldrogalis18:02:55

@georgek ā€œonyx->Java->C++->FORTRANā€ I got dizzy reading that sentence.

michaeldrogalis18:02:20

We havenā€™t made Java integration much of a priority, and the interface bindings are lagging behind for 0.10 ā€” mostly because of a lack of demand tbh.

georgek18:02:50

I hear that! I guess Iā€™m wondering about the scope needed to take the above repo and get it up to date

michaeldrogalis18:02:04

Specifying :onyx/language :java in a catalog entry should still work. If you need Java bindings for plugins that lifecycles, thatā€™s a larger task.

georgek18:02:16

There are 2 of us (potentially 3) that could bang it out if its not terribly painful

georgek18:02:38

Yeah, I suspect we may need lifecycle support

michaeldrogalis18:02:38

I can answer that, but backing up - which pieces of Onyx do you need to use from Java? Thatā€™ll help me answer your question.

michaeldrogalis18:02:50

Okay, yeah - thatā€™s a bigger task then.

georgek18:02:18

For starters basically its going to just be triggering tasks that are backed by java (and native underneath)

michaeldrogalis18:02:24

Anything that does resolution from a keyword into a function will need to be able to specify a Java method rather than a Clojure fn and resolve it correctly.

georgek18:02:31

So, I guess that for basic usage as task handlers we are pretty close to good to go then?

michaeldrogalis18:02:49

Yeah, should be.

georgek18:02:28

Kew kew. Just out of curiosity then, how big of a job to complete a full java API ?

georgek18:02:48

(starting with the above repo)

michaeldrogalis18:02:10

Probably a week IMO.

michaeldrogalis18:02:33

Itā€™s going to be easy once we figure out the general pattern - itā€™s all grunt work after that.

georgek18:02:19

Kewl, thatā€™s totally doable given the advantages to us even just from a selling this project point of view

georgek18:02:46

Sweet, how should we go about that? Suggestions on where to start?

michaeldrogalis18:02:35

Hmm, hereā€™s another question. Is the intent to also build the Onyx jobā€™s in Java? Because another thing you can do is write the job itself in JSON and write a shim to turn it into EDN.

michaeldrogalis18:02:59

Itā€™s really a matter of whether you want to use Java to specify the behavior/functions or the job itself, which is just data.

georgek18:02:14

Right, we may end up doing runtime adjustmentā€™s of the workflow in Java (our business logic is going to be in java that will use Onyx to run jobs) so while this would work for job construction we will want to have high-level control over the jobs running in the system which, Iā€™m assuming, would need a Java API?

georgek18:02:19

Basically we canā€™t use straight up Clojure (too new and scary for the govā€™t) so we are emulating the declarative approach in a Java system that triggers native (legacy) code

michaeldrogalis18:02:10

Gotcha. Yeah, I mean in the case, doing what I started doing in onyx-java is probably a viable route. All itā€™s doing is making a fully static API to build up data structures on each method call.

michaeldrogalis18:02:50

Once you land at a fully formed job, make a call out to :onyx.api/submit-job ā€” which should work okay if itā€™s casting step-by-step as you go

georgek18:02:44

Sweet, lemme talk it over with the team and we will give it a go!

georgek18:02:23

That seems like the most attractive thing internally. We def. want to avoid not really having a clean interface layer when we talk to the workflow system!

georgek18:02:56

(Given the whole Java thing)

michaeldrogalis18:02:04

Cool, keep me posted. Thanks for the interest, been looking to put this one to bed for a very long ago. šŸ˜›

georgek18:02:23

Will do! Iā€™m excited