Fork me on GitHub
#re-frame
<
2017-01-02
>
qqq00:01:10

in re-frame, my app has a single source of truth; and the gui is a function of the db if we take this a step further, my collaborative app has one global source of truth (the aws db), and each client has a projection of this global truth (based on user permissions) then, all of 'collaboration / server requests" is about (1) sending motificatikons on changing this global db [subject to permissions] and (2) getting updates from the db [also subject to permissions] is there a toolchain for setting this up with re-frame? so it handels not only client side, but client + server + distributed ?

mikethompson00:01:04

Nothing publicly available that I'm aware of. You'd need a reactive database (one that can tell you when it has changed). You'd need to handle *latency compensation* (when the user does action X, how long before the effect of that action is reflected in the GUI). Etc. perhaps these links will help: https://yogthos.net/posts/2016-11-05-LuminusPostgresNotifications.html https://github.com/Day8/re-frame/blob/master/docs/MentalModelOmnibus.md#full-stack

qqq00:01:54

@mikethompson : I'm willing to "optimistically" do stuff on the client side, and in case of "merge conflicts", push it back to the user (like git commit conflicts)

mikethompson00:01:20

Yeah, and its hard to do properly :-)

mikethompson00:01:42

We've had some experience with rethinkdb

mikethompson00:01:54

And you'll need to think it through VERY carefully

qqq00:01:57

will look into comander pattern / datalog all the way down, I ctually already have a poor man's eav / datalog impl going on

qqq01:01:46

@mikethompson : what if I made the following simplification: everything is either (1) only one person can write to it or (2) append-only [like slack/irc channels]; there should be no 'merge conflicts' in this case

mikethompson01:01:54

If you want background on the "latency compensation" side of things, google for terms like "write fencing" etc.

qqq01:01:30

it seems very weird that (1) on the client side, I have this beautiful data sturcutre; then to colalborate with another client, I have to shove it to the server, shove it to some db, then send it back to the server, then send it to the client, where the db is often not datomic and doesn't match well with the clikent data structure

mikethompson01:01:54

Yeah, I promise it is almost certainly more complicated than what you realise. At least that was my experince, and I was already expecting it to be hard. Consider: you have a "subscription for a certain query" and then your user updates one of the items, but then someone else also changes another item (not the one that your user changed), and then the change that your user made gets rejected on the server (but not the change made by someone else). So your "optimistic change" has to be undone, BUT not the one successfully made by the other party.

mikethompson01:01:59

@qqq the difference is this: on the client side you don't have shared mutatable state (the root of all evil). Whereas that's EXACTLY what a shared database is.

mikethompson01:01:32

That's why within the client it is all "easy". Introduce shared mutable state AND distributed state and, oh boy, you just increased the complexity through the roof.

qqq01:01:13

1. I don't disagree it's hard. 2. Git seems to somehow allow people to "work locally then merge globally" 3. Can we decompose the server problem into 3 parts: (1) data format (storing clojure data in the database), (2) latency, and (3) merge conflicts? Or does this split not make sense?

qqq01:01:47

There's something called CRDTs also right? Where everything I can force into CRDTs won't have merge conflicts, and for the other things, I have to be very careful with.

mikethompson01:01:54

Just to be clear: the problem is not too hard if you don't do latency compensation.

qqq01:01:05

right, user make saction

qqq01:01:06

you wait on server

qqq01:01:10

then all is synced up

qqq01:01:25

but everything shoul.d feel 'immediate'

qqq01:01:30

so we will have potential merge conflicts

mikethompson01:01:42

The latency compensation is what makes it hard: because you are suddenly dealing with distributed state.

mikethompson01:01:05

Remember the old quote: the two hard things in CS are naming things and cache invalidation. The "cache invalidation" bit should really be "keeping distributed state consistent"

mikethompson01:01:19

The moment you bite off "latency compensation" you are actually biting off "keeping distributed state consistent"

qqq01:01:46

Here's the thing:

qqq01:01:56

I think I can split all actions into "CRDT-able" and not "CRDT-able"

qqq01:01:05

the CRDT-able ones, will be immediate and merged (without conflict)

qqq01:01:15

the not-CRDT-able ones, the user will have to wait on

mikethompson01:01:27

That certainly works

qqq01:01:59

I don't even know how CRDTs work in theory.

qqq01:01:03

Any idea where to start?

qqq01:01:25

There's a bunch of clj libraries, but I feel like I should get the theory first.

qqq01:01:44

"the only thing worse than deterministic bugs are non-deterministic bugs" <-- CRDT debugging fun

mikethompson01:01:59

While I'm aware of CRDT, my knowledge is shallow

martinklepsch03:01:05

This may also be interesting wrt CRDTs: http://replikativ.io/

qqq03:01:00

https://arxiv.org/abs/1410.2803 <-- supposedly the paper Phoenix Presence is based on

danielcompton07:01:52

There's a bunch of good talks on YouTube for crdt's

danielcompton07:01:20

I went down the same thought path as you @qqq but it's not trivial.

qqq07:01:31

As a personal quirk, for these things, I prefer to read an in depth review paper

qqq07:01:42

that says: here are the standard algorithms; here are how they work; here is their limitations

danielcompton07:01:47

Sure, search google scholar for crdt

qqq07:01:53

Youtube talks, in general, seem too high level for me.

danielcompton07:01:57

I think inra has a review paper

qqq07:01:05

Going to read the Phoenix Presence one first. 🙂

qqq07:01:15

Need to understand one well enough as a reference point for others.

qqq07:01:28

@danielcompton: having gone thorugh the pass, what's your current view on CRDTs, and do you use them?

danielcompton07:01:35

That's a good starting point I think

qqq07:01:44

ah right, have taht one, haven't printed it yet, it's 50 pages 🙂

danielcompton07:01:54

I think they're incredible and need to be adopted further

danielcompton07:01:14

But there is a bunch of low level stuff that is cutting edge research, eg garbage collection

danielcompton07:01:49

In theory it's awesome, in practice, you'd have to do a lot of your own plumbing and deeply understand the research to get something reliable

danielcompton07:01:07

Because if you messed it up, debugging it would probably be really hard

danielcompton07:01:10

Another big problem is ehat you do on the server side. How do you do permissions?

danielcompton07:01:21

How do you share state?

danielcompton07:01:54

Does everything have to live in memory on one server, or do you cluster the state across multiple backend servers

qqq07:01:00

Yeah, I feel like for CRDTs, if I can't rewrite the proofs, I'm not ready to implement it yet.

qqq07:01:13

I was hoping, for the server side, to push everything into AWS RDS with TRansactions

qqq07:01:27

so "how to scale server state = amaxon RDS's problem"

qqq07:01:37

permissions = "does usedr have right to run this transaction / op?"

qqq07:01:47

and I'm concedrned only with CRDT in terms of "making client feel snappy and not laggy"

qqq07:01:51

by optimisticalyl evaluating stuff

danielcompton07:01:59

Sure, then how do you translate a relational db to something on client side?

qqq07:01:20

not too concerned, I have a simple datomic (more like datascript) impl on client side

qqq07:01:32

so I think I'm confident I can match it up with amaxon rds

qqq07:01:37

it's crdts that cohncern me at the moment

danielcompton07:01:56

How do you optimistically update on client side and make sure that the same logic runs on the server?

qqq07:01:11

CRDTs are associative

qqq07:01:27

I'm looking for ops that are transiaitive, associative, and imdeompotent

danielcompton07:01:46

Sure, i know that :) as others have said, when the rubber meets the road it's a bit more complicated than that. But if you come up with something that works with reframe I'll be keen to see it

qqq07:01:20

I have no doubt I'm way too optimistic, but if I wasn't, I'd have given up long ago 🙂

Oliver George11:01:11

Is there an example of how to use cursors with reg-sub type 2?

Oliver George11:01:59

The first example simply assess the db. That's effectively a cursor.

Oliver George11:01:13

The two function version (input-signals-fn then computation-fn) doesn't seem to allow cursor like data access. Neither take db as an arg.

Oliver George11:01:16

Perhaps the recommended solution is to register a :path sub.

mikethompson11:01:48

@olivergeorge I'm seemingly the self appointed president of the anti cursor movement. Please see the "Finally" point under "Guiding Philosophy" in https://github.com/Day8/re-frame/blob/master/docs/MentalModelOmnibus.md#guiding-philosophy

mikethompson11:01:57

Are you perhaps a refugee from OM? :-)

mikethompson11:01:39

Kidding aside, I think what you are asking is .... should you create a create a generalised level 2 reg-sub which is given a path and which extracts out the part of app-db at that path. In effect, you want a generalised READ ONLY cursor for an arbitrary path?

elahti11:01:22

it's not kidding. there's a lot of frustration towards om(.next)

mikethompson11:01:39

If so, I'd urge against it. I know it seems attractive. But it means the view (where the subscription is made) has to provide the path which means it has to "know" about the structure of app-db.

mikethompson11:01:23

We'd prefer the subscription to be "declarative". It should simply state what it requireds, and not where to get it.

mikethompson11:01:36

That way, when the inevitable restructuing of app-db happens later, you don't have to scoot around all the views finding references to paths and altering them.

Oliver George20:01:23

Hi Mike. Thanks. Yes, I'm only interested in read only. I agree paths are unstructured.

Oliver George20:01:01

So lots of "hardcoded path" subs make sense.

Oliver George20:01:03

Happy new year

mikethompson20:01:26

Yeah, so you will likely create a bunch of reg-subs which are very, very similar except for the path they contain. And you will have to actively resist the urge to "refactor" them. This is not a place where refactoring out the common pattern (accessing a path within app-db) will be useful.

mikethompson21:01:35

So this issue has come up enough times that I've added it to the docs: https://github.com/Day8/re-frame/blob/master/docs/SubscriptionsCleanup.md#a-final-faq

mikethompson21:01:56

Now I can point people to something :-)