Fork me on GitHub
#onyx
<
2016-09-07
>
Travis01:09:36

Awesome job on the defn podcast guys. Enjoyed it. Shout out to @lucasbradstreet for doing this at 2am and staying coherent. lol

michaeldrogalis02:09:44

Did ya'll move into production now that you found your performance problem?

Travis02:09:09

We are definitely moving forward, we hope to start getting there with in a month or two to our in house users and expand from there

Travis02:09:48

Also are plan will be to deploy on top of DCOS so I hope to be able to aid any other mesos users

michaeldrogalis02:09:03

That's pretty slick. 🙂

Travis02:09:40

Thnx, it's been awesome working on this

Travis02:09:51

I wish I had started working on this back in November so I could have had some good conversations with you at clojure conj

michaeldrogalis02:09:06

I don't think I'll be at the Conj this year, but I'm sure @lucasbradstreet and/or @gardnervickers will. 🙂

Travis02:09:57

Yeah, not sure I will be either. I was at the last two

michaeldrogalis02:09:51

Always a good time, though.

Travis02:09:02

Definitely

michaeldrogalis02:09:30

Anyhow, off into the night to work on the Next Big Thing! Looking forward to unveiling it in a few months -- every Onyx user will immensely benefit from it.

Travis02:09:12

Awesome looking forward to it. Hope to be able to contribute back in some way to onyx. Have a good one

yonatanel07:09:41

@camechis Is there a link link to the defn podcast you mentioned?

lucasbradstreet07:09:49

I suggest you add "defn" to your podcast listener and get it there. You can listen to the specific episode here https://soundcloud.com/defn-771544745/episode-9-onyx-with-mike-and-lucas

v.solovyov07:09:29

heh, I just un-subscribed yesterday from defn, guess I'll have to subscribe again

lucasbradstreet07:09:26

We had to balance bringing new people in to Onyx vs some more advanced discussion

v.solovyov07:09:04

I find that sometimes it's useful to read/listen some stuff aimed at beginners, even if I'm already familiar with the thing for a time. Can help to straighten my thoughts about it

lucasbradstreet07:09:31

Definitely. You can usually pick up a few things that aren't quite what you expected

lucasbradstreet07:09:43

Or a different way of thinking about something

vijaykiran08:09:05

@v.solovyov any reason why you unsubscribed ?

v.solovyov08:09:46

@vijaykiran I listened to a few episodes (5, 7 and 8, I think), and after that I found out that I can't remember anything that stick into my mind. I'm usually listening while driving a car and I'm not a native English speaker, so there is that. I'm not sure I can provide more specific feedback, sorry

vijaykiran10:09:40

okay, thank you 🙂

vladclj12:09:58

Hi guys, I set :http-output/url via catalog and pass segment like {:args {:headers {"content-type" "application/xml"} :body xml}} to onyx-http and get an error. Can I combine task-map params with segment's?

aspra15:09:59

Hi there! I created a job with a kafka consuming task. I noticed that I couldn’t get around the fact that the number of kafka topic partitions should be equal to the task max and min peers. So for instance, could I not have smaller number of consumers than partitions?

michaeldrogalis15:09:18

@aspra onyx-kafka doesn't support multi-partition consumption yet, but it wouldn't be hard to add if someone has some time to do a patch for it.

michaeldrogalis15:09:30

@vladclj What error do you see?

aspra15:09:24

@michaeldrogalis cool, thx for your quick answer

michaeldrogalis15:09:57

@aspra Np. I know Franzy, the underlying library, easily supports it. Should just be a few lines of changes

vladclj15:09:23

I get something like this clojure.lang.ExceptionInfo: Thaw failed: Decryption/decompression failure, or data unfrozen/damaged

michaeldrogalis15:09:57

@vladclj That would indicate that you are receiving a segment or task map that isn't EDN deserializable.

vladclj15:09:53

Hmm:slightly_smiling_face: And what about: can I combine task-map params with segment's?

michaeldrogalis15:09:31

@vladclj I don't understand the question. Can you be more specific?

michaeldrogalis15:09:52

Task map parameters as in :onyx/params [...]? What do you mean by combine them?

vladclj15:09:54

segmennt:
{:args {:headers {"content-type" "application/xml"} :body xml}} 
and
task-map:
{:http-output/url "" 
 :http-output/args {:as :json
                    :headers {"content-type" "application/json"
                              "content-encoding" "gzip"}}}

michaeldrogalis15:09:39

Okay. What exactly are you trying to do with those values?

vladclj16:09:18

Can I combine, set some params in segment and other in task-map? For exemple

in segment
{:args :body xml}}
and in catalog
{:http-output/args {:headers {"content-type" "application/xml"}
:http-output/url "" }

michaeldrogalis16:09:41

Oh, you're asking specifically about the onyx-http plugin?

michaeldrogalis16:09:54

I'm not sure to be honest, let me take a look.

michaeldrogalis16:09:38

Pretty sure not -- @aspra can you confirm?

aaelony18:09:29

Question about https://github.com/onyx-platform/learn-onyx/blob/master/src/workshop/challenge_6_0.clj#L74-L75

(defn inject-reader-ch [event lifecycle]
{:core.async/chan (u/get-input-channel (:core.async/id lifecycle))})
What would this look like to read from file? I've been using
(-> base-job (add-task (s/buffered-file-reader :read-segments filename batch-settings))) 
but confused as to how this would work in the inject-reader-ch function

aaelony18:09:31

ah nevermind, I think I've found it

aaelony18:09:47

in the onyx.plugin.seq docs

michaeldrogalis18:09:53

Yup. onyx-seq is quite good for that.

aaelony19:09:18

cool, I'm almost there. Just wrapping my head around a few errors... I guess I'm casting something (I need to track it down).

actual: java.lang.IllegalArgumentException: No implementation of method: :coerce-key of protocol: #'onyx.windowing.units/ICoerceKey found for class: nil

michaeldrogalis19:09:38

@aaelony Seems like something that supposed to be a unit is nil.

michaeldrogalis19:09:54

Units being the term for things like [5 :minutes]

aaelony19:09:31

ah, thanks... I haven't specified that... fixing it. (my first foray into windowing/aggregation in onyx.. )

aaelony19:09:12

still not clear what is missing. Would that be in the windows or the triggers? What's a good way to track it down?

:windows [{:window/id :collect-segments
                             :window/task :count-events
                             :window/type :fixed
                             :window/aggregation :onyx.windowing.aggregation/count
                             :window/window-key :event-time
                             :window/range [1 :hour]
                             :window/doc "Count events in one hour fixed windows."
                                }]
                  :triggers [{:trigger/window-id :collect-segments
                              :trigger/refinement :onyx.refinements/accumulating
                              :trigger/on :onyx.triggers/segment
                              :trigger/fire-all-extents? true
                              :trigger/threshold [10 :elements]
                              :trigger/sync ::deliver-promise!
                              :trigger/doc "Fires against all extents every 10 segments processed."
                              }
                             ]

michaeldrogalis19:09:00

Can you give me a little more context on the error you're seeing? Is it job submission time, or runtime? Would expect to see a much better error message at submission time. If its at runtime, my guess is :window/window-key in your segment is nil.

michaeldrogalis19:09:30

Gotta run out to handle something -- will be back later if you still have questions.

aaelony19:09:52

np, will tinker on... thanks

aaelony19:09:04

I think it might be a :done ...

michaeldrogalis19:09:38

The sentinel value is never present in the stream of segments.

michaeldrogalis19:09:55

It is intercepted and played back at the outputs.

aaelony19:09:56

I'm sure it's something I'm doing... I'll study it more

mariusz_jachimowicz20:09:43

@aaelony you have this error because you have some segments without :event-time key or value is nil

aaelony20:09:52

@mariusz_jachimowicz, I wish it was, but that's actually not the case. This is a small test file and all are populated and correct. I've confirmed this with the same functions but without windowing

mariusz_jachimowicz20:09:10

very strange because it seams that it is only used by windowing

aaelony20:09:06

it's something subtle though. It'll be satisfying once I've figured it out:)

mariusz_jachimowicz20:09:04

I am very curious also

michaeldrogalis21:09:16

@aaelony What's the full stack trace? I can take a good guess based on that

michaeldrogalis21:09:19

Hmm, yeah. My money is still on @mariusz_jachimowicz's guess.

michaeldrogalis22:09:37

@aaelony Drop the braces around your count aggregation if the aggregation doesn't work against a single key.

michaeldrogalis22:09:03

Not sure if that's your error -- if it is it needs a patch for a better message.

aaelony22:09:25

I've tried that both ways, same error either way

aaelony22:09:00

I should be able to get away with a bare minimum of id, task, type, aggregation, and window-key as in this example, https://github.com/onyx-platform/onyx-examples/blob/0.9.x/aggregation/src/aggregation/core.clj#L65-L70, right?

michaeldrogalis22:09:30

Correct, yeah. I'd highly recommend doing a local install of Onyx and printing at the call site where you see a nil though. Somewhere along the way your task is losing day, and that's a really fast way to figure out whats happening

michaeldrogalis22:09:57

Thanks @mariusz_jachimowicz, nice work ^^ We need to catch up on our backlog of your excellent PRs 🙂

aaelony22:09:51

the culprit is somewhere here:

(defn deliver-promise! [event window
                        {:keys [trigger/window-id] :as trigger}
                        {:keys [lower-bound upper-bound] :as state-event}
                        state]
  (println "Inside deliver-promise!, lower-bound: " lower-bound ", upper-bound:" upper-bound )
  (let [lower (java.util.Date. lower-bound)
        upper (java.util.Date. upper-bound)
        ]
    (println "Trigger for" window-id "window")
    (swap! fired-window-state assoc [lower upper] (into #{} state))))
which yields:
Inside deliver-promise!, lower-bound: Inside deliver-promise!, lower-bound:   -Infinity-Infinity  , upper-bound:, upper-bound:  InfinityInfinity

aaelony22:09:34

narrows it down

michaeldrogalis23:09:49

A global window only has one instance -- with bounds negative infinity -> positive infinity

aaelony23:09:38

the lower and upper bounds aren't the dates they should be.

aaelony23:09:35

which I believe is causing a Error in process state loop

michaeldrogalis23:09:31

Is it a global window?

aaelony23:09:47

it is, I'll switch it back to :fixed

michaeldrogalis23:09:04

Those bounds are correct then.

aaelony23:09:39

ok. I'm learning slowly by iteration.

michaeldrogalis23:09:32

This might help a bit if you're trying to figure out how things get partitioned - https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/windowing/window_id.cljc

michaeldrogalis23:09:54

Only applies to Fixed and Sliding windows, but this is where the notion of bounds comes from. Its pretty neat reading.

aaelony23:09:19

this is what I'm seeing from my lein test with a println call in the deliver-promise! function.:

Starting Onyx test environment
Inside deliver-promise!, lower-bound:  1471454160000 , upper-bound: 1471454279999
Trigger for :collect-segments window. lower( #inst "2016-08-17T17:16:00.000-00:00" ), upper( #inst "2016-08-17T17:17:59.999-00:00" )
16-09-07 23:22:42 MacBook-Pro.local ERROR [onyx.peer.window-state:306] - Error in process state loop.