This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-09-03
Channels
- # admin-announcements (1)
- # beginners (4)
- # boot (1)
- # chestnut (2)
- # clara (1)
- # cljs-dev (8)
- # cljsjs (50)
- # clojure (40)
- # clojure-austin (3)
- # clojure-brasil (3)
- # clojure-canada (1)
- # clojure-gamedev (2)
- # clojure-italy (3)
- # clojure-russia (19)
- # clojure-spec (14)
- # clojure-uk (1)
- # clojurescript (60)
- # core-async (4)
- # cursive (4)
- # datomic (3)
- # editors-rus (2)
- # emacs (4)
- # events (1)
- # figwheel (2)
- # flambo (4)
- # hoplon (94)
- # jobs (4)
- # leiningen (3)
- # om (9)
- # onyx (64)
- # re-frame (86)
- # reagent (52)
- # spacemacs (4)
- # test-check (1)
- # yada (31)
@vladclj A workflow is just a plain Clojure data structure. You can make as many stages in your workflow as needed using Clojure's sequence functions.
Yeah, unfortunately. I think we also need window/window-fn
We'd accept a PR. The new unstable development is really far off develop/master, or I'd just add it for all the window key lookup options
I would accept a PR, and eat the cost of merging it myself later though
@lucasbradstreet: also, still about this ordering stuff (I keep being distracted, so it moves slowly): so I'll aggregate with fixed window, then I need to put a trigger on that window, and then I'll need to write those message to kafka myself, right? No other way to pass those to a regular output?
@asolovyov: afraid there's no way currently. Unstable has a way but it's a bit far off being ready
Not really. Doing in order processing is just way harder than it seems :/
Maybe there is a way to rethink the problem though
well, those entities are orders (in a shop) and statuses of those orders; and if status arrives before the order, it's a tough call for consumer - has to keep track of those statuses or something
more than that, some statuses just have to be in order, like 'cancellation requested' and 'cancelled'
Yeah, I mean what you're trying to do makes sense to me.
I want to keep both orders and statuses in a single kafka topic just because it's way easier for consumer - if it starts lagging behind new orders (and that happens to our warehouse system), it knows it will never get statuses for orders it still has no idea about
so yeah, ordering seems like a way to solve this stuff, but it's a bit more complex than I expected 😄
not that I have a lot of experience with this stuff - obviously introducing async messaging inside processing makes ordering harder
Group by key and then maybe write out to keyed kafka message for each message, then compact the topic, so you only get the last one? You would have to maintain all of the messages in memory in Onyx though, at which point you need to think about when to discard
I'll have a bit more of a think about it and research a little more. It's something I want to know the best answer to, since it does come up
As I said before the story will be better once we get the next messaging model done
heh, well, I want to write all statuses, so warehouse system knows all the details 🙂
Next messaging order will maintain order unless you distribute to multiple peers in which case the order is maintained but will be separate but also in order
I wonder if it's just better to leave current state of things like it is right now (with a custom http api) and wait a bit for a new release
riight, I can distribute to those peers based on order id, which means all statuses will be in order for the single order! 🙂
Eg 1 2 3 4 might become 1 3 and 2 4 before being sent to an output
Yes true
As long as there aren't some intermediate tasks that might mess up the order
But you could group them too
Well say you read from Kafka, then you send to an ungrouped task split over multiple peers, then it goes to a grouped task, well you've just split up the ordering and need to re-zip
But you could just using grouping on the intermediate task and all is fine
I can't make any promises about that. It's taking longer than we hoped
Hoping for technical preview in a couple weeks
I would love to switch to a new API before end of November, since it brings us a lot of orders 😄
It won't even be close to prod ready though.
But you can dev against it. It'll take time to do the plugins too, but I'll get Kafka done for the preview
ok, I'll try to look at window-key in the next week or so, would love to contribute a bit :-))
Shouldn't be hard. We do similar things for stuff like group-by-key
@lucasbradstreet do you use index.html from onyx-visualization?
@mariusz_jachimowicz: just the JS and some css I think
It seems that purpouse was to use it for testing code compiled with advanced mode, right ?
I think it may have just been what I was using before I add the devcards
Ok. Could I update index.html so it could be used for testing code after advanced optimization?
I want to be able to quickly test is the code running in production mode properly
Go for it, please
@lucasbradstreet Please review https://github.com/onyx-platform/onyx-visualization/pull/8
@mariusz_jachimowicz onyx-visualization 0.4.0 released. Tested with onyx-dashboard :thumbsup:. Thank you!
Thanks @mariusz_jachimowicz. Nice work. 🙂
@aaelony Can you open a PR or issue please? Happy to get to it, can't do it now though.
Thx. It's a pleasure. Eh, unfortunately I have to return to commercial work soon. I am searching for job in the meantime. Would be great to find Clojure job. Onyx is great project and codebase.
Thanks! Hoping to open our doors for hiring soon, but we still need some time to get the product in order.