Fork me on GitHub
#react
<
2021-05-26
>
lilactown01:05:50

@martinklepsch I’ve been working on https://github.com/lilactown/flex/tree/main/packages/flex-core when I have time which is v advanced, v alpha but sounds more like what you’re looking for. it might inspire you

lilactown01:05:56

mostly it is v slow 😅

martinklepsch09:05:41

@lilactown interesting, thanks! that looks very similar to derivatives in that they’re both attempting to implement some kind of incremental dataflow. I haven’t thought about scheduling much at all tbh but a lot of this makes sense to me.

martinklepsch09:05:06

What makes it so slow? the basic design would make me think it should be fast?

martinklepsch09:05:22

The https://github.com/martinklepsch/derivatives/blob/master/src/org/martinklepsch/derivatives.cljc is 120 lines if you ignore some of the fringes but it’s all synchronous(?) via add-watch! and I am not doing anything special to order/batch updates. I think the ordering should mostly work though since the low level computations are updated first, then the ones that depend on those etc.

lilactown17:05:46

There are multiple slow parts I think. The part that stands out to me, if memory serves, is the calculate! function used to calculate and set the new state and dependents of each computation

lilactown17:05:56

that and the heap algorithm I use to order updates

lilactown17:05:29

the heap is important for ordering because even if you do what you’re saying (breadth first) you can end up in diamond dependencies that require extra work

lilactown17:05:45

there’s an inherent cost to the way flex stores state in a dynamic variable that will always be slower than storing the state on the computation object itself

lilactown17:05:06

i.e. every dereference requires a lookup in this dynamic var

lilactown17:05:25

my hypothesis is I can get it Fast Enough:tm: that the flexibility of being able to use multiple environments for e.g. testing, or reusing computations between different parts of a system, is worth it

Roman Liutikov18:05:41

at Pitch we are using re-frame, but it shows its weakness due to the nature of dataflow in the app: since Pitch is collaborative realtime app we are sending data events between clients, the whole state of user’s workspace is calculated out of atomic objects (folder, presentation, user, permission, font, style, etc. records). When there’s a new object in memory re-frame has to recalculate related indexes from scratch. While that still works to prevent change propagation in subscriptions graph, recalculating indexes is still quite expensive.

Roman Liutikov18:05:52

in our case ideally we want some sort of incrementality when calculating derived state

💯 3
Roman Liutikov18:05:59

we are also exploring a completely diff approach: event based state updates, basically Kafka on the frontend where instead of randomly writing into an app db and relying on subs graph to figure out what changed, we would update state atomically based on events signalling what exactly is about to change (what type of an object is added/updated/removed)

Roman Liutikov18:05:51

my conclusion so far is that change inference via subs graph doesn’t scale well in terms of performance, but it’s very easy to use

martinklepsch18:05:24

The Kafka/change feed approach is also something I’ve been thinking about

martinklepsch18:05:08

One problem with it is a bit that it’s nice for lists of similar entities but probably a bit trickier for “streams” that combine data from multiple sources. Like ad-hoc subscriptions become harder

lilactown18:05:14

the tricky part of using an event ledger is that your events need to be fine grained enough to allow change detection to be precise

martinklepsch18:05:29

Would be very interested to have a chat about this with some folks some time if you’re down 🙂

lilactown18:05:34

ah I think we just said similar thing :D

martinklepsch18:05:56

We can upload it to youtube afterwards or something 😄

lilactown18:05:16

I wish I had time today

lilactown18:05:48

I think the timely-dataflow / differential-dataflow stuff is the state of the art here

👀 3
lilactown18:05:49

IIRC it combines a log of changes a la kafka with a dataflow graph to allow far more efficient (CPU time) of incremental computations at the cost of additional memory

lilactown18:05:56

it’s been a little bit since I read the paper

danieroux18:05:13

If I was to start a multi-user UI collaboration project, I would have a hard look at https://croquet.io/sdk/docs - it use to be a 3D peer-to-peer world implementation in Smalltalk - https://web.archive.org/web/20090220083519/http://www.croquetconsortium.org/images/2/2b/2003_Croquet_Collab_Arch.pdf

danieroux18:05:23

Croquet is a synchronization system for multiuser digital experiences. It allows multiple users to work or play together within a single shared distributed environment, and it guarantees that this distributed environment will remain bit-identical for every user.”

danieroux19:05:28

Yay, finding myself in a new/old rabbit hole: https://www.youtube.com/watch?v=9xWDTwdqfno