Fork me on GitHub
#re-frame
<
2015-08-02
>
jhchabran12:08:51

How would you handle some events source that streams data at a really high rate ? We tried doing it as you’d do with ajax, using a handler and dispatch, but it fills the channel that re-frame uses under the hood and raises a lot of errors. We thought about updating app-db directly, as it worked properly before we started re-frame but we're not sure that’s the right way of doing it. We could of course do some operations to slow things down, but I’m still curious how we’d do it if we were to display these events in our logview, where you’d typically want to see everything simple_smile For some context, we’re building a drone control panel, which typically have things like a map showing current position, various flight data, etc … The drone itself sends us loads of flight data, some of which are required for example, to graph things or reproducing flight controls.

jhchabran12:08:09

( if anyone is interested we already open sourced our code that transforms drone’s data to a websocket that streams json https://github.com/openskybot/skybot-router )

jhchabran12:08:45

btw, the error we’re getting as it is now

Uncaught Error: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
(< (.-length puts) impl/MAX-QUEUE-SIZE)
cljs.core.async.impl.channels.ManyToManyChannel.cljs$core$async$impl$protocols$WritePort$put_BANG_$arity$3	@	channels.cljs?rel=1438379373829:81
cljs$core$async$impl$protocols$put_BANG_	@	protocols.cljs?rel=1438379373727:17
cljs.core.async.put_BANG_.cljs$core$IFn$_invoke$arity$2	@	async.cljs?rel=1438379373509:107
cljs$core$async$put_BANG_	@	async.cljs?rel=1438379373509:101
re_frame$router$dispatch	@	router.cljs?rel=1438379372614:79
(anonymous function)	@	handlers.cljs?rel=1438512081521:17
callHandlers	@	Client.js:134
handleMessage	@	Client.js:307
bound

jhchabran12:08:51

and our current code

(register-handler
  :subscribe-uav
  (fn [db [_ c uav-name]]
    (bot/on-uav-update c uav-name #(dispatch [:store-uav uav-name %1]))
    db))

(register-handler
  :store-uav
  (fn [db [_ uav-name response]]
    (let [data (u/uav->map response)]
      (assoc-in db [:uavs uav-name] data))))

jhchabran12:08:20

hum, I wrote a wall of text, if this isn’t the right place to post this, please tell me simple_smile

darwin12:08:11

I would throttle dispatching to the frequency of preferred UI updating frequency, say 10 times per second

darwin12:08:14

I assume the data is only sampled state of some measurements and dropping samples does not affect other parts of your app

jhchabran12:08:52

clearly, we can drop things and that won't cause any issues

jhchabran12:08:47

and what about updating app-db directly ? not saying it's the right solution, throttling clearly is but I'm curious of the consequences of a such approach

darwin12:08:36

updating app-db directly will still trigger all watchers on the ratom, which might be slow if your app is more complex

darwin12:08:51

all reactions and subscriptions add watchers on app-db

jhchabran12:08:36

so if I were to add some logviews, the best would be too buffer our events and then only update app-db in chunks right ?

darwin12:08:19

not buffer, just drop them

darwin12:08:06

if you want 10 times per second, drop all but last message in each 100ms time window

darwin12:08:36

so you call dispatch max 10 times per second

jhchabran12:08:43

I wasn't clear, when I said a hypothetic logview, I meant something where you can see everything that had been received

darwin12:08:11

ah, ok, then you would have to keep this data outside app-db

jhchabran12:08:07

but if I update app-db only at an acceptable rate, like ten per seconds but this time updating it with a big chunk it'd be acceptable right ?

jhchabran12:08:40

mm, in fact you're right

jhchabran12:08:55

in we're going to have some view that hold huge logs, they have nothing to do with app-db

jhchabran12:08:18

thanks for your help simple_smile

darwin12:08:00

you are welcome, and yes, I think updating app-db with bigger chunks of data would be acceptable, you trigger reaction watching machinery just once for each chunk

jhchabran12:08:37

btw, your project felt like golden for us, we intend to build an atom plugin to edit code that is send to a drone

darwin12:08:20

simple_smile cool, but it is still long way to go, it is not yet suitable even for experimental use

jhchabran12:08:38

so having some code that shows us the ropes of writing an atom plugin in cljs is really great

jhchabran12:08:49

we're still beginners with clojure

darwin12:08:25

just a tip, if you are not using it already: https://github.com/binaryage/cljs-devtools

jhchabran12:08:03

yeah I stumbled on it yesterday while browsing some chan in on clojurians

jhchabran12:08:10

looks awesome

jhchabran12:08:44

for now we're just floating in pure bliss thanks to figwheel / reagent / reframe / clojure / repl

mikethompson13:08:28

@jhchabran: I'm late to the party here, but I'll try to provide a new piece of info ... > Uncaught Error: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer. means that a huge volume of dispatches have filled up the core.async channel before the events could be handled (by event handlers). So possible solutions: 1. dispatch less often, by either dropping data or, better, doing less dispatches but making each one carry more data. 2. Increase the size of that core.async buffer. Well, I'd have to give an API for that. So that's not an immediate solution.

jhchabran13:08:12

@mikethompson: thanks for the insights

jhchabran13:08:31

that's exactly what we did, basically wiring our callbacks to dispatch, wasn't a good idea 😄

jhchabran13:08:47

we'll go for route 1

jhchabran13:08:32

route 2, while interesting in our case would just push the problem a bit further, we'd hit the same wall once we'll start handling more things on the drone

mikethompson13:08:49

We have run into the same issue with websokets WHEN the app is in the background. Tons of updates flying down the websocket, and because the app is in the background, the javascript is throttled. You aren't getting 60 fps. Annimation frames are much loooooonggggger

jhchabran13:08:45

ok, dully noted 😄

jhchabran13:08:58

I'm not very familiar with what happens when app is in the background

mikethompson13:08:33

Neither were we. simple_smile

mikethompson13:08:52

Hash lessons!! Is there no end to the pain?? simple_smile

jhchabran13:08:43

well for us, we felt relieved when we started doing clojurescript 😄

jhchabran13:08:19

if you could have seen the number of times my friend who started cljs a week ago was like "OH BOY THIS IS EXACTLY WHAT I NEEDED" while he was reading your re-frame's readme 😄

jhchabran13:08:59

sent you a PM on reddit before noticing you're here on clojurians btw simple_smile

mikethompson13:08:44

Hey, just read it. Thanks for the kind words!!

meow13:08:11

I've been exploring these issues of channels as streams and dealing with large volumes of inputs and while I can't give you a definitive answer I think I can share a few things that are worth considering.

meow13:08:43

First, its good to understand the inner workings of core.async channels a bit. Every channel has two queues: one for putters and one for takers, each 1024 in size. This is a given and is independent of buffering.

meow13:08:50

If you want to deal with back pressure then that's where the buffer comes in. You can size the buffer and pick a type that drops values either as fifo or lifo.

meow13:08:03

@mikethompson: for the re-frame main channel I don't think it makes sense to add a buffer - I think what you have is fine as is. When someone runs into the 1024 puts limit then they need to rethink their app.

mikethompson13:08:49

@meow: music to my ears!! Thanks.

meow13:08:16

If the events in question aren't terribly important and can be dropped in response to back pressure then adding a buffered channel between the event source and the process of feeding those events on to re-frame is the way to go. That way the buffering and dropping is handled by core.async and is independent of timing and other browser issues.

meow13:08:43

That's the easy case.

meow14:08:06

If you want to deal with all the source data then you need to control that in a loop that is independent of re-frame or any other frame based mechanisms because those typically use a js requestAnimationFrame mechanism and browsers will slow down or stop the RAF loop for tabs that don't have the active focus.

meow14:08:39

Leading to increased back pressure on your channel.

meow14:08:40

If timing is really important, say to make a game appear smooth and at an appropriate speed given that the browser won't always be able to operate at 60fps or even necessarily at a fixed, slower rate, like 30fps, well now you are in a whole other world and will have to know quite a bit more about how timings really take place in the browser.

meow14:08:13

That's where I'm still getting up-to-speed. Google Closure has some tools for this in their async module.

Pablo Fernandez14:08:20

I read through https://github.com/Day8/re-frame and there’s no mention of URLs and how to change them and dispatch and so on. Does re-frame work in a single URL?

meow14:08:14

@mikethompson: I hope that all makes sense. This is something I'm still learning but thought it might be useful in general.

meow14:08:56

This wiki page might be useful in terms of lessons learned, such as how awesome transducers are for channel data transformations: https://github.com/clojure/core.async/wiki/Sieve-of-Eratosthenes

meow14:08:51

@jhchabran: I hope that helps some. Your kind of situation where you have a firehose of data streaming in is the kind of thing I have in the back of my mind as I've been exploring this issue. Would love to see how your app shapes up.

meow14:08:55

In short, the browser event loop is never faster than 60 frames per second so you can't feed it more events than it can handle in that timeframe of 16.667 ms for each frame. That applies to the event loop in re-frame as well. So the main event loop needs to be fast and primarily focused on what you want to display in the next frame update.

darwin14:08:03

@pupeno: SPAs == Single Page Applications, they have no routing to different URLs

Pablo Fernandez14:08:24

darwin: well, many SPAs frameworks update the URL so that if you pass the URL to someone else it’ll load the same part of the application. For example, EmberJS makes that straightforward and easy. Without that, apps behave like they flash apps. And yes, that makes SPA a misnomer, but there are very few real SPAs out there. Look at the one that started it all, Google Maps, it constantly updates the URL to represent your current view as best as possible.

darwin14:08:06

call it whatever you like, you could use something like secretary lib, but that is not routing to different urls, same url with subset of app’s state encoded in url hash

Pablo Fernandez14:08:50

I know about secretary and silk and bidi.

Pablo Fernandez14:08:44

But it looks like reframe is not doing anything with the URL, so all that needs to be manually wiring and the way reframe handles state, it looks like it would be a horrible hack. This is my question.

meow14:08:08

@pupeno: I'm not sure if any of the react-based libraries in clojurescript do any kind of automatic url updating: om, reagent, re-frame, quiescent, freactive, etc.

meow14:08:36

I could be wrong, and am curious to know.

darwin14:08:10

why hack? keep routing state in your app-db, when url changes by user action, call re-frame’s dispatch to change that state, from the opposite direction: subscribe to that routing info in app-db, when it changes, update the hash, without triggering dispatch

Pablo Fernandez14:08:52

@meow: yeah, I’m curious too. I wish there was an emberjs-like URL handling built on top of react.

Pablo Fernandez14:08:15

darwin: well, I never used reframe before, so I don’t have a good mental model of how a finished app looks like. An all the documentation I seen there’s not a single mention of URLs which makes me thing I might be going against the grain of the library by using URLs.

darwin14:08:23

IMO re-frame does not want to care about “outer world”, it is your job to map app state changes to changes of the world around and dispatch proper commands to modify app state in reaction to world events, re-frame even does not care about react/reagent/whatever-you-use to present your app state to the user

meow14:08:42

@jhchabran: You might find this useful:

(defn listen-next-tick!
  [callback]
  (letfn [(step [] (if (callback) (goog.async.nextTick step)))]
    (goog.async.nextTick step)))

jhchabran14:08:49

mm, I’m having some trouble understand what it does, could you provide an example ?

meow14:08:57

I believe that listen-next-tick! is the fastest looping mechanism for js that plays nicely with the js event loop

meow14:08:37

Think of it like an event listener - you pass in your callback, which gets called every time there is an opening in the js processing loop. (Sorry for my sloppy language, the details are in the Google Closure library).

meow14:08:15

Your callback needs to return true to keep the loop going.

meow14:08:12

Your callback could then, for example, read data off the drone and log it.

jhchabran14:08:06

I see what you mean simple_smile well in our case, the drone is just sending stuff permanently, we can’t read from it

meow14:08:04

Ok, well, you're going to read from some data source that's being populated by the drone, yes?

meow14:08:35

Or the drone is going to supply data in some fashion.

jhchabran14:08:43

(I’ve just finished reading what you wrote earlier, thanks for the clear explanation ! )

meow14:08:23

And I'm just focusing on being able to do everything in the browser but that's just my thing and obviously this kind of problem could be split into a client/server solution.

jhchabran14:08:13

actually, since we’re also the ones building the software that connects to the drone, we could do some handling/buffering directly in there

jhchabran14:08:11

but for now, we tried to not drift too far away from how it’s internally done by the drone’s controller (Taulabs)

jhchabran14:08:37

we’re reading from a ws that streams json

jhchabran14:08:43

productivity aside, yeah doing everything in the browser in a damn fun challenge simple_smile

jhchabran15:08:42

typically, we receive objects that represents some drone subsystem, like current altitude, position, etc ..

jhchabran15:08:29

so, since our SPA is just a dashboard where you can see what’s happening and adjust some settings, dropping most of these objects coming from the WS doesn’t create any issue on our side

jhchabran15:08:02

presently, we’re just wiring the callbacks, but as things progresses, it clearly hints we should add in core.async in there

jhchabran15:08:33

populating some array in low level fashion, then periodically reading from it could be an even simpler solution

jhchabran15:08:12

it’s just we’re still beginners in clojure so that’s a lot to digest 😄

meow15:08:43

if it were me (and keep in mind you know far more about your app than I do) I would wire up a callback for handling the json stream and put that data onto a core.async channel, using a transducer, if necessary, to transform the data into something easier or more appropriate for the rest of your app to use, and I would give that channel a buffer size and type to deal with the back pressure

meow15:08:31

then you'd have some other loop that takes the data off that channel to do whatever you want with it

jhchabran15:08:48

yeah I thought about doing things that way (not clearly expressed as you just wrote) and I feel like it’s the direction we should take

meow15:08:50

you can think of a channel as just a queue

jhchabran15:08:24

ah there he is @stant is the other developer on this project simple_smile

meow15:08:28

but you get back-pressure logic and async concurrency

meow15:08:30

clojurescript has great tools for this sort of thing

stant15:08:37

hi got a lot to read here oO

stant15:08:56

yes indeed just discovered it thanks to @jhchabran very cool so far

meow15:08:58

and re-frame is a very good choice for the UI

jhchabran15:08:56

yeah re-frame made things much clearer for us, with solely reagent, we would have been a bit lost

stant15:08:59

yes actually using some kind of channel might be a good idea in our case. For exemple the thing that broke is the update of the quaternion decribing the actual attitude of the drone, which means that it’ll certainly feed a 3d representation of the drone

meow15:08:14

to follow up on my looping suggestion, if you look at the source code for core.async you'll see that they use goog.async.nextTick to schedule the work done by the go macro, so if you use core.async you're already using goog.async.nextTick whether you knew it or not. simple_smile

stant15:08:32

and in any case a data that arrives at a such data rate will be ending in the DOM

meow15:08:32

And since you folks are new to clojure you might not be that familiar with Google Closure yet, but it is the compiler used by ClojureScript so it becomes readily available for use in cljs apps, and by "it" I mean a large library that is part of the Google Closure compiler+library tool set.

meow15:08:14

Google Closure has many modules and goog.async is one of them.

meow15:08:28

My poly library is an attempt to pull in bits and pieces of Google Closure that can be enhanced or reshaped for easier use in cljs, but Google Closure can also just be used as is since it's just js code. But you might see some interesting things in poly where I'm taking js events and turning them into clojure maps and supplying them via channels.

jhchabran15:08:25

yeah the more I read about cljs, the more it feels I should get familiar with google closure simple_smile

meow15:08:00

Yes, there are definitely good things in goog and dnolen is always telling devs to look there.

jhchabran15:08:37

gonna spend some time reading poly tonight then 😄

jhchabran15:08:51

damn, the whole clojure/script world is so refreshing

jhchabran15:08:03

no fancy homepages, but awesome readme and great code everywhere

meow15:08:56

poly is definitely alpha code and an exploration for me so keep that in mind, but I would definitely recommend reading it to get an idea of what is available in goog and how one person is using it, especially the async stuff.

jhchabran15:08:15

yeah no worries, you clearly stated it in the readme simple_smile

meow15:08:29

not sure I even have a readme for poly...

meow15:08:39

If you want a readme, @mikethompson has got a readme simple_smile

jhchabran15:08:43

hum, you’re right

jhchabran15:08:49

I messed up projects in my head 😄

jhchabran15:08:59

we’re gonna get back at our code and take some time to digest the huge amount of feedback we got here

jhchabran15:08:12

thanks guys, really simple_smile

danielcompton19:08:58

Just following up on the discussion about 1024 pending puts

meow19:08:24

1024 pending puts on the wall, 1024 pending puts, <! one down, pass it around, 1023 pending puts on the wall...

danielcompton19:08:06

You run into the 1024 pending puts issue because dispatch uses put! to put events on the channel. put! runs asynchronously so it has to go into the pending queue

danielcompton19:08:58

If you had a reference to the re-frame channel then you could “dispatch” inside a go block. This would park the “dispatcher” until there was space in the channel to put the event

danielcompton19:08:17

I also ran into this recently working with web sockets and streaming events

danielcompton19:08:58

I have a feeling there is a performance gain to be had by removing https://github.com/Day8/re-frame/blob/v0.4.1/src/re_frame/router.cljs#L43

danielcompton20:08:28

but it needs to be replaced with a call which is smart enough to know when to yield to the next animation frame but continue processing events until then

danielcompton20:08:13

@jhchabran or @stant, are you able to share a Chrome profile of your code running? A screenshot should be enough for a first pass

danielcompton20:08:01

The problem with (async/timeout 0) is that it isn’t really a 0 second timeout. It’s browser specific, but usually takes between 2-15 ms to run

danielcompton20:08:20

What @meow said about the using dropping channels is a good idea. Another option is to have the server debounce events and merge every 1/10th of a second so you’re not processing too much data

danielcompton20:08:44

It really depends on your data model though.

meow20:08:49

(defn request-animation-frame!
  "A delayed callback that pegs to the next animation frame."
  [callback]
  (.start (goog.async.AnimationDelay. callback)))

(defn listen-animation-frame!
  [callback]
  (letfn [(step
           [timestamp]
           (if (callback timestamp) (request-animation-frame! step)))]
    (request-animation-frame! step)))

meow20:08:54

@danielcompton: if you removed that line of code then the go loop would never yield to the js process

danielcompton20:08:27

@meow: exactly, you want to run as much as you can before the next animation frame needs to run

meow20:08:14

the browser is a mess, but I feel like I'm getting closer to understanding how to make it behave

danielcompton20:08:47

Totally agree. We’re lucky in our project to only need to target Chrome which makes life a lot simpler

meow20:08:57

@danielcompton: can you explain what you mean about (async/timeout 0) behavior?

meow20:08:47

or were you thinking of js timeout being flaky

danielcompton20:08:25

that calls js/settimeout

danielcompton20:08:51

setTimeout 0 won’t actually execute in 0 msecs. Test it on your browser at http://javascript.info/tutorial/events-and-timing-depth

danielcompton20:08:06

On the section "Measure your setTimeout(.., 0) resolution"

danielcompton20:08:33

I got around 5 ms on Safari, Chrome, and Firefox on OS X

danielcompton20:08:55

So every time we call settimeout 0 in the router loop we have to wait at least 5 ms before we process the next event

danielcompton20:08:24

What would be nice is to alt! over two channels, the re-frame dispatch channel, and a channel which told you when to yield to next animation frame

meow20:08:16

Ah, right - hadn't seen that.

meow20:08:29

I think goog.async has a fix for that as well.

meow20:08:58

I think the problem is even worse than you describe - gory details inside goog.async source.

danielcompton20:08:24

The plumbing is there to support it with setImmediate but it looks like it’s mocked out when used in actual browsers

meow20:08:09

As you can see from the code I've posted for listen-next-tick! and listen-animation-frame! I'm heading in the same direction of having multiple channels independent of rAF

danielcompton20:08:59

I didn’t quite understand what the purpose of that code was?

danielcompton20:08:09

What will that allow you to do?

meow20:08:53

They treat these like registering event listeners like you would do for say a mouse-move event.

meow20:08:21

listen-animation-frame will call its callback for each frame

meow20:08:54

listen-next-tick will call whenever there is a break in the js processing context

meow21:08:16

goog.async.nextTick does what we'd like (async/timeout 0) to do

meow21:08:03

I just took Google's improved handling of animation-frame and "do this right away", which expect callback handlers, and wrapped them in ClojureScript and then added a simple looping option in the same form as listening to events. Then its trivial to register a callback that would turn these into event channels.

meow21:08:05

I already have code that takes mouse and keyboard events and turns them into channels.

meow21:08:24

I'm sure that's all clear as mud. Sorry.

danielcompton21:08:11

I think I follow mostly

danielcompton21:08:31

enough to get the general gist though. Thanks simple_smile

meow21:08:34

the reason I had to add my own looping mechanism is that goog treats these like listenOnce things if that makes sense

meow21:08:18

the final result looks like this, which is my own rendering where I'm not using re-frame:

(defn render! [timestamp state]
  ...)

(defn render-cycle! [timestamp]
  (render! timestamp @state)
  (get-in @state [:app :rendering?]))

(defn start-rendering! []
  (console/info "start rendering")
  (swap! state assoc-in [:app :rendering?] true)
  (poly/listen-animation-frame! render-cycle!))

(defn stop-rendering! []
  (console/info "stop rendering")
  (swap! state assoc-in [:app :rendering?] false))

meow22:08:57

So, to put this back into the context of re-frame. I created what I did so I could understand how to optimize cljs rendering from the ground up, using goog tools to fix browser issues and avoid unnecessary delays in the browser or in core.async. Looking at the code for router-loop I would be concerned about the timing issues, as @danielcompton has mentioned.

danielcompton22:08:20

@meow: since you’ve been thinking about this a lot, how else could you structure the router-loop?

meow22:08:27

The events being dispatched, these are re-frame events, right?

meow22:08:43

These aren't like mousemove events and such.

meow22:08:06

Keep in mind that I'm not actively using re-frame and have only toyed with it.

meow23:08:32

Is it correct to say that these events have been dispatched by handlers that expect them to take place before the next frame?

meow23:08:12

Or rather dispatched by something that expects handlers to be called before the next frame...

meow23:08:42

If so I would look at doing something like my listen-animation-frame! loop since that relies on Google Closure's clever code to peg it to the next frame request - it is Google providing the ultimate RAF polyfil.

meow23:08:48

Then inside that loop I would have a go-loop with a when-let / recur that took every event off and called (try (handle event ...

meow23:08:32

I'm sure I've got some of that wrong because I don't know re-frame that well. I just don't think async timeout channels provide the kind of control that you want for an event loop like this.

meow23:08:59

I might take a closer look at this tomorrow.

meow23:08:50

I do agree with you, @danielcompton, that time is being lost that could be used to process more re-frame events in each frame.