Fork me on GitHub
#onyx
<
2016-09-03
>
michaeldrogalis00:09:47

@vladclj A workflow is just a plain Clojure data structure. You can make as many stages in your workflow as needed using Clojure's sequence functions.

asolovyov08:09:45

oh, it seems like :window/window-key cannot be a path to the window key

asolovyov08:09:07

so if I generate a {:message {stuff here}} for kafka, I can't aggregate that...

lucasbradstreet08:09:59

Yeah, unfortunately. I think we also need window/window-fn

lucasbradstreet08:09:27

We'd accept a PR. The new unstable development is really far off develop/master, or I'd just add it for all the window key lookup options

lucasbradstreet08:09:34

I would accept a PR, and eat the cost of merging it myself later though

asolovyov09:09:05

@lucasbradstreet: also, still about this ordering stuff (I keep being distracted, so it moves slowly): so I'll aggregate with fixed window, then I need to put a trigger on that window, and then I'll need to write those message to kafka myself, right? No other way to pass those to a regular output?

lucasbradstreet09:09:44

@asolovyov: afraid there's no way currently. Unstable has a way but it's a bit far off being ready

asolovyov09:09:06

I wonder if I've decided on a wrong way to do things

lucasbradstreet09:09:34

Not really. Doing in order processing is just way harder than it seems :/

asolovyov09:09:45

but it'll be tough for consumers of those topics if it's out of order

lucasbradstreet09:09:45

Maybe there is a way to rethink the problem though

asolovyov09:09:33

well, those entities are orders (in a shop) and statuses of those orders; and if status arrives before the order, it's a tough call for consumer - has to keep track of those statuses or something

asolovyov09:09:51

more than that, some statuses just have to be in order, like 'cancellation requested' and 'cancelled'

asolovyov09:09:15

(those shouldn't be very close to each other though)

lucasbradstreet09:09:52

Yeah, I mean what you're trying to do makes sense to me.

asolovyov09:09:00

I want to keep both orders and statuses in a single kafka topic just because it's way easier for consumer - if it starts lagging behind new orders (and that happens to our warehouse system), it knows it will never get statuses for orders it still has no idea about

asolovyov09:09:19

so yeah, ordering seems like a way to solve this stuff, but it's a bit more complex than I expected 😄

asolovyov09:09:48

not that I have a lot of experience with this stuff - obviously introducing async messaging inside processing makes ordering harder

lucasbradstreet09:09:57

Group by key and then maybe write out to keyed kafka message for each message, then compact the topic, so you only get the last one? You would have to maintain all of the messages in memory in Onyx though, at which point you need to think about when to discard

lucasbradstreet09:09:38

I'll have a bit more of a think about it and research a little more. It's something I want to know the best answer to, since it does come up

lucasbradstreet09:09:52

As I said before the story will be better once we get the next messaging model done

asolovyov09:09:28

heh, well, I want to write all statuses, so warehouse system knows all the details 🙂

asolovyov09:09:47

hm, next messaging model will be ordered? or something else?

lucasbradstreet09:09:43

Next messaging order will maintain order unless you distribute to multiple peers in which case the order is maintained but will be separate but also in order

asolovyov09:09:45

I wonder if it's just better to leave current state of things like it is right now (with a custom http api) and wait a bit for a new release

asolovyov09:09:15

riight, I can distribute to those peers based on order id, which means all statuses will be in order for the single order! 🙂

lucasbradstreet09:09:20

Eg 1 2 3 4 might become 1 3 and 2 4 before being sent to an output

lucasbradstreet09:09:42

As long as there aren't some intermediate tasks that might mess up the order

lucasbradstreet09:09:48

But you could group them too

asolovyov09:09:12

tasks to mess up the order is something that reads from the database, for example?

asolovyov09:09:22

because then time is not exactly the same for them?

lucasbradstreet09:09:14

Well say you read from Kafka, then you send to an ungrouped task split over multiple peers, then it goes to a grouped task, well you've just split up the ordering and need to re-zip

lucasbradstreet09:09:39

But you could just using grouping on the intermediate task and all is fine

asolovyov09:09:36

do you have any ideas about time frame in mind? 🙂

lucasbradstreet09:09:06

I can't make any promises about that. It's taking longer than we hoped

lucasbradstreet09:09:18

Hoping for technical preview in a couple weeks

asolovyov09:09:23

I would love to switch to a new API before end of November, since it brings us a lot of orders 😄

asolovyov09:09:33

ah, well, couple of weeks is nothing to me ;))

lucasbradstreet09:09:13

It won't even be close to prod ready though.

lucasbradstreet09:09:43

But you can dev against it. It'll take time to do the plugins too, but I'll get Kafka done for the preview

asolovyov09:09:14

ok, I'll try to look at window-key in the next week or so, would love to contribute a bit :-))

lucasbradstreet09:09:04

Shouldn't be hard. We do similar things for stuff like group-by-key

mariusz_jachimowicz12:09:29

@lucasbradstreet do you use index.html from onyx-visualization?

lucasbradstreet12:09:29

@mariusz_jachimowicz: just the JS and some css I think

mariusz_jachimowicz12:09:00

It seems that purpouse was to use it for testing code compiled with advanced mode, right ?

lucasbradstreet12:09:46

I think it may have just been what I was using before I add the devcards

mariusz_jachimowicz13:09:45

Ok. Could I update index.html so it could be used for testing code after advanced optimization?

mariusz_jachimowicz13:09:25

I want to be able to quickly test is the code running in production mode properly

lucasbradstreet13:09:48

Go for it, please

lucasbradstreet14:09:21

@mariusz_jachimowicz onyx-visualization 0.4.0 released. Tested with onyx-dashboard :thumbsup:. Thank you!

michaeldrogalis19:09:14

@aaelony Can you open a PR or issue please? Happy to get to it, can't do it now though.

mariusz_jachimowicz21:09:00

Thx. It's a pleasure. Eh, unfortunately I have to return to commercial work soon. I am searching for job in the meantime. Would be great to find Clojure job. Onyx is great project and codebase.

michaeldrogalis22:09:06

Thanks! Hoping to open our doors for hiring soon, but we still need some time to get the product in order.