Awesome job on the defn podcast guys. Enjoyed it. Shout out to @lucasbradstreet for doing this at 2am and staying coherent. lol

Thanks @camechis 🙂

Did ya'll move into production now that you found your performance problem?

We are definitely moving forward, we hope to start getting there with in a month or two to our in house users and expand from there

Also are plan will be to deploy on top of DCOS so I hope to be able to aid any other mesos users

That's pretty slick. 🙂

Thnx, it's been awesome working on this

I wish I had started working on this back in November so I could have had some good conversations with you at clojure conj

I don't think I'll be at the Conj this year, but I'm sure @lucasbradstreet and/or @gardnervickers will. 🙂

Yeah, not sure I will be either. I was at the last two

Always a good time, though.

Definitely

Anyhow, off into the night to work on the Next Big Thing! Looking forward to unveiling it in a few months -- every Onyx user will immensely benefit from it.

Catch ya

Awesome looking forward to it. Hope to be able to contribute back in some way to onyx. Have a good one

@camechis Is there a link link to the defn podcast you mentioned?

I suggest you add "defn" to your podcast listener and get it there. You can listen to the specific episode here https://soundcloud.com/defn-771544745/episode-9-onyx-with-mike-and-lucas

heh, I just un-subscribed yesterday from defn, guess I'll have to subscribe again

We had to balance bringing new people in to Onyx vs some more advanced discussion

I find that sometimes it's useful to read/listen some stuff aimed at beginners, even if I'm already familiar with the thing for a time. Can help to straighten my thoughts about it

Definitely. You can usually pick up a few things that aren't quite what you expected

Or a different way of thinking about something

Exactly

@v.solovyov any reason why you unsubscribed ?

@vijaykiran I listened to a few episodes (5, 7 and 8, I think), and after that I found out that I can't remember anything that stick into my mind. I'm usually listening while driving a car and I'm not a native English speaker, so there is that. I'm not sure I can provide more specific feedback, sorry

okay, thank you 🙂

Hi guys, I set :http-output/url via catalog and pass segment like {:args {:headers {"content-type" "application/xml"} :body xml}} to onyx-http and get an error. Can I combine task-map params with segment's?

Hi there! I created a job with a kafka consuming task. I noticed that I couldn’t get around the fact that the number of kafka topic partitions should be equal to the task max and min peers. So for instance, could I not have smaller number of consumers than partitions?

@aspra onyx-kafka doesn't support multi-partition consumption yet, but it wouldn't be hard to add if someone has some time to do a patch for it.

@vladclj What error do you see?

@michaeldrogalis cool, thx for your quick answer

@aspra Np. I know Franzy, the underlying library, easily supports it. Should just be a few lines of changes

I get something like this clojure.lang.ExceptionInfo: Thaw failed: Decryption/decompression failure, or data unfrozen/damaged

@vladclj That would indicate that you are receiving a segment or task map that isn't EDN deserializable.

Hmm:slightly_smiling_face: And what about: can I combine task-map params with segment's?

@vladclj I don't understand the question. Can you be more specific?

Task map parameters as in :onyx/params [...]? What do you mean by combine them?

segmennt:
{:args {:headers {"content-type" "application/xml"} :body xml}} 
and
task-map:
{:http-output/url "" 
 :http-output/args {:as :json
                    :headers {"content-type" "application/json"
                              "content-encoding" "gzip"}}}

Okay. What exactly are you trying to do with those values?

Can I combine, set some params in segment and other in task-map? For exemple

in segment
{:args :body xml}}
and in catalog
{:http-output/args {:headers {"content-type" "application/xml"}
:http-output/url "" }

Oh, you're asking specifically about the onyx-http plugin?

I'm not sure to be honest, let me take a look.

Pretty sure not -- @aspra can you confirm?

Question about https://github.com/onyx-platform/learn-onyx/blob/master/src/workshop/challenge_6_0.clj#L74-L75

(defn inject-reader-ch [event lifecycle]
{:core.async/chan (u/get-input-channel (:core.async/id lifecycle))})

(-> base-job (add-task (s/buffered-file-reader :read-segments filename batch-settings))) 

ah nevermind, I think I've found it

in the onyx.plugin.seq docs

Yup. onyx-seq is quite good for that.

cool, I'm almost there. Just wrapping my head around a few errors... I guess I'm casting something (I need to track it down).

actual: java.lang.IllegalArgumentException: No implementation of method: :coerce-key of protocol: #'onyx.windowing.units/ICoerceKey found for class: nil

@aaelony Seems like something that supposed to be a unit is nil.

Units being the term for things like [5 :minutes]

ah, thanks... I haven't specified that... fixing it. (my first foray into windowing/aggregation in onyx.. )

still not clear what is missing. Would that be in the windows or the triggers? What's a good way to track it down?

:windows [{:window/id :collect-segments
                             :window/task :count-events
                             :window/type :fixed
                             :window/aggregation :onyx.windowing.aggregation/count
                             :window/window-key :event-time
                             :window/range [1 :hour]
                             :window/doc "Count events in one hour fixed windows."
                                }]
                  :triggers [{:trigger/window-id :collect-segments
                              :trigger/refinement :onyx.refinements/accumulating
                              :trigger/on :onyx.triggers/segment
                              :trigger/fire-all-extents? true
                              :trigger/threshold [10 :elements]
                              :trigger/sync ::deliver-promise!
                              :trigger/doc "Fires against all extents every 10 segments processed."
                              }
                             ]

Can you give me a little more context on the error you're seeing? Is it job submission time, or runtime? Would expect to see a much better error message at submission time. If its at runtime, my guess is :window/window-key in your segment is nil.

Gotta run out to handle something -- will be back later if you still have questions.

np, will tinker on... thanks

I think it might be a :done ...

The sentinel value is never present in the stream of segments.

It is intercepted and played back at the outputs.

I'm sure it's something I'm doing... I'll study it more

@aaelony you have this error because you have some segments without :event-time key or value is nil

@mariusz_jachimowicz, I wish it was, but that's actually not the case. This is a small test file and all are populated and correct. I've confirmed this with the same functions but without windowing

very strange because it seams that it is only used by windowing

it's something subtle though. It'll be satisfying once I've figured it out:)

I am very curious also

@aaelony What's the full stack trace? I can take a good guess based on that

Hmm, yeah. My money is still on @mariusz_jachimowicz's guess.

@aaelony Drop the braces around your count aggregation if the aggregation doesn't work against a single key.

Not sure if that's your error -- if it is it needs a patch for a better message.

I've tried that both ways, same error either way

Ah, figured

I should be able to get away with a bare minimum of id, task, type, aggregation, and window-key as in this example, https://github.com/onyx-platform/onyx-examples/blob/0.9.x/aggregation/src/aggregation/core.clj#L65-L70, right?

please review https://github.com/onyx-platform/lib-onyx/pull/11

Correct, yeah. I'd highly recommend doing a local install of Onyx and printing at the call site where you see a nil though. Somewhere along the way your task is losing day, and that's a really fast way to figure out whats happening

ok

Thanks @mariusz_jachimowicz, nice work ^^ We need to catch up on our backlog of your excellent PRs 🙂

the culprit is somewhere here:

(defn deliver-promise! [event window
                        {:keys [trigger/window-id] :as trigger}
                        {:keys [lower-bound upper-bound] :as state-event}
                        state]
  (println "Inside deliver-promise!, lower-bound: " lower-bound ", upper-bound:" upper-bound )
  (let [lower (java.util.Date. lower-bound)
        upper (java.util.Date. upper-bound)
        ]
    (println "Trigger for" window-id "window")
    (swap! fired-window-state assoc [lower upper] (into #{} state))))

Inside deliver-promise!, lower-bound: Inside deliver-promise!, lower-bound:   -Infinity-Infinity  , upper-bound:, upper-bound:  InfinityInfinity

narrows it down

A global window only has one instance -- with bounds negative infinity -> positive infinity

the lower and upper bounds aren't the dates they should be.

which I believe is causing a Error in process state loop

error

Is it a global window?

it is, I'll switch it back to :fixed

Those bounds are correct then.

ok. I'm learning slowly by iteration.

This might help a bit if you're trying to figure out how things get partitioned - https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/windowing/window_id.cljc

Only applies to Fixed and Sliding windows, but this is where the notion of bounds comes from. Its pretty neat reading.

this is what I'm seeing from my lein test with a println call in the deliver-promise! function.:

Starting Onyx test environment
Inside deliver-promise!, lower-bound:  1471454160000 , upper-bound: 1471454279999
Trigger for :collect-segments window. lower( #inst "2016-08-17T17:16:00.000-00:00" ), upper( #inst "2016-08-17T17:17:59.999-00:00" )
16-09-07 23:22:42 MacBook-Pro.local ERROR [onyx.peer.window-state:306] - Error in process state loop.