This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-14
Channels
- # aws (1)
- # bangalore-clj (1)
- # beginners (48)
- # boot (65)
- # braveandtrue (1)
- # cider (1)
- # clara (15)
- # cljs-dev (7)
- # clojure (179)
- # clojure-austin (1)
- # clojure-denmark (2)
- # clojure-greece (68)
- # clojure-italy (7)
- # clojure-russia (41)
- # clojure-serbia (9)
- # clojure-spec (44)
- # clojure-uk (27)
- # clojured (15)
- # clojureremote (20)
- # clojurescript (70)
- # community-development (2)
- # core-async (10)
- # cursive (14)
- # datomic (36)
- # defnpodcast (3)
- # emacs (13)
- # events (13)
- # hoplon (33)
- # immutant (18)
- # instaparse (2)
- # jobs (29)
- # jobs-discuss (71)
- # klipse (38)
- # lein-figwheel (4)
- # leiningen (1)
- # mount (34)
- # off-topic (36)
- # om (3)
- # onyx (51)
- # pedestal (5)
- # perun (8)
- # proton (2)
- # rdf (8)
- # re-frame (33)
- # reagent (24)
- # remote-jobs (1)
- # rum (6)
- # spacemacs (2)
- # specter (14)
- # sql (5)
- # testing (6)
- # untangled (1)
- # vim (10)
- # yada (3)
hey! I'm writing here only when I have problems, but still: my tests (on Onyx 0.9 still, hehe) have a weird problem:
One task generates a list of combinations, and another one fetches them from ES to put as cache in Cassandra. The thing is, because of some change we have 1167 combinations instead of 248, which causes test to retry infinitely. I added 5 min :onyx/pending-timeout
to input task (which is onyx-seq
from a file), and didn't help. It takes about 50 seconds for lein test my-test
to start up and go through 500 combinations, but if I remove that restriction, it just re-runs and re-runs.
Anything I should look at?
@asolovyov: what are you using as an output?
@gardnervickers core.async channel
Are you not draining the buffer?
@gardnervickers thanks!
Oh heh, yea they're mostly made for testing with small-ish data sets and assume that to be the case.
I usually just write something quick to drain the channel to a log or file or something.
yeah, we're doing thingie from test example still:
(helper/feedback-exception! peer-config job-id)
(let [segments (into [] (take-segments! (:write-filters-cache chans)))
Just checkin in on the current state of Java interop. Iām trying to get Onyx used in NOAA (specifically in the climate data corner) which would require java interop (onyx->Java->C++->FORTRAN actually). There is this: https://github.com/onyx-platform/onyx-java But not a lot of other convo other than saying its a low priority. Is the above a good place to start? Whatās missing since we will put in some work on this if its not too daunting
@georgek āonyx->Java->C++->FORTRANā I got dizzy reading that sentence.
We havenāt made Java integration much of a priority, and the interface bindings are lagging behind for 0.10 ā mostly because of a lack of demand tbh.
I hear that! I guess Iām wondering about the scope needed to take the above repo and get it up to date
Specifying :onyx/language :java
in a catalog entry should still work. If you need Java bindings for plugins that lifecycles, thatās a larger task.
I can answer that, but backing up - which pieces of Onyx do you need to use from Java? Thatāll help me answer your question.
Okay, yeah - thatās a bigger task then.
For starters basically its going to just be triggering tasks that are backed by java (and native underneath)
Anything that does resolution from a keyword into a function will need to be able to specify a Java method rather than a Clojure fn and resolve it correctly.
So, I guess that for basic usage as task handlers we are pretty close to good to go then?
Yeah, should be.
Probably a week IMO.
Itās going to be easy once we figure out the general pattern - itās all grunt work after that.
Kewl, thatās totally doable given the advantages to us even just from a selling this project point of view
Hmm, hereās another question. Is the intent to also build the Onyx jobās in Java? Because another thing you can do is write the job itself in JSON and write a shim to turn it into EDN.
Itās really a matter of whether you want to use Java to specify the behavior/functions or the job itself, which is just data.
Right, we may end up doing runtime adjustmentās of the workflow in Java (our business logic is going to be in java that will use Onyx to run jobs) so while this would work for job construction we will want to have high-level control over the jobs running in the system which, Iām assuming, would need a Java API?
Basically we canāt use straight up Clojure (too new and scary for the govāt) so we are emulating the declarative approach in a Java system that triggers native (legacy) code
Gotcha. Yeah, I mean in the case, doing what I started doing in onyx-java is probably a viable route. All itās doing is making a fully static API to build up data structures on each method call.
Once you land at a fully formed job, make a call out to :onyx.api/submit-job
ā which should work okay if itās casting step-by-step as you go
That seems like the most attractive thing internally. We def. want to avoid not really having a clean interface layer when we talk to the workflow system!
Cool, keep me posted. Thanks for the interest, been looking to put this one to bed for a very long ago. š