This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-04-21
Channels
- # admin-announcements (2)
- # beginners (22)
- # boot (223)
- # cider (161)
- # cljs-dev (19)
- # cljsrn (4)
- # clojure (186)
- # clojure-austin (6)
- # clojure-beijing (1)
- # clojure-boston (3)
- # clojure-china (1)
- # clojure-czech (1)
- # clojure-france (1)
- # clojure-greece (10)
- # clojure-russia (17)
- # clojure-uk (154)
- # clojurebridge (3)
- # clojurescript (82)
- # component (12)
- # cursive (12)
- # datomic (71)
- # dirac (3)
- # editors (2)
- # emacs (29)
- # flambo (31)
- # hoplon (21)
- # immutant (11)
- # instaparse (17)
- # jobs (2)
- # jobs-discuss (2)
- # jobs-rus (1)
- # lein-figwheel (12)
- # leiningen (2)
- # off-topic (44)
- # om (78)
- # onyx (38)
- # parinfer (1)
- # re-frame (34)
- # reagent (32)
- # spacemacs (56)
- # untangled (74)
- # vim (12)
- # yada (2)
This will be useful "Java 8u92 includes new VM switches: ExitOnOutOfMemory and CrashOnOutOfMemory http://www.oracle.com/technetwork/java/javase/8u92-relnotes-2949471.html"
We should setup an unhandled exception handler for at least our tests https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions
is the lein template for creating plugins (https://github.com/onyx-platform/onyx-plugin) still safe to use? I notice it's a couple of versions behind
Hi, i’m trying to write test where i want to have guarantee that the previous segment was processed before the next arrives to the job. How can I achieve this?
Your going to need to use the windowing features of you want to get order/exactly once semantics.
The idea is that in real world these events are not ordered, and i don’t care if they come simultaneously, i just want to test situation when they are distributed in time
Hmm I’m a bit confused by what you mean by “test situation when they are distributed in time"
If you could elaborate on what you’re trying to do I think I could offer better insight
ok. Imagine that some data arrives to job and after its processing it is stored in the database. Now if another segment arrives, it is processed, but data that was stored in the database perviously can affect results of this processing. If both segments are processed simultaneously it might happen that they will not affect each other. So I want to test the situation when the second segment is processed after the first, the case when data from database will affect processing of the second segment.
Yes thanks, if you need ordering guarantees then you will need to use Onyx’s windowing features.
@acron: I need to upgrade it and get it under our auto-release process. It's mostly okay, but make sure you're using the latest version.
Not much has changed with respect to the plugin interface in the last couple of releases.
@rasom: In general, Onyx doesn't provide any guarantees about the ordering of your data when it's in flight - otherwise performance would be crippled. You can use windowing, as @gardnervickers said, to maintain some state over time and get a view into that state - but it sounds like you'd be better off with unit or integration testing for what you're trying to do.
@rasom, I should have elaborated, using windows will not provide ordering by itself, but they will allow you to order segments before flushing state to a database.
@gardnervickers: ok, i see. Thanks
@michaeldrogalis: thanks
If I set :onyx/batch-size
to 1, does it mean that segment will be processed exactly when it comes to task, without waiting for other segments?
@rasom: Correct.
@michaeldrogalis: I'm playing with onyx-redis
and noticing that write-batch
is called repeatedly. Is this normal for 'output' plugins?
@acron: The plugin is receiving no input, and the batch timeout is running out - so the lifecycle executes with an empty batch.
@michaeldrogalis: ok, that makes sense. So it's okay to just do nothing at that point?
ack-segment will get called when all of the segments generated from the initial root segment have been processed
Root segment being the segment read from the input plugin. By segments generated I mean any segment in the tree of segments resulting from that original segment
In the overall job
So if a segment is read at input task T1, and it is sent to T2, which created two segments from it, then those are sent to an output task, where it is written out. An ack is sent for each segment at each stage of the process, and when all of those acks are received the plugin will call ack-segment on the input task
Sorry for all the qns @tcoupland and I are actually trying to shoe-horn some loops into a workflow using external storage so...this is really testing our understanding of Onyx and the plugins
Ok, that makes sense. So, regards 'pending messages' (a pattern in a lot of the plugins), is the logic that at any point the plugin could be asked to 'retry' a segment and should retrieve it from the pending list?
Exactly, yes.