Fork me on GitHub
#datascript
<
2019-05-13
>
danielstockton09:05:36

@tonsky What do you consider state of the art these days for reactive UIs? I found an old tweet about precept, which you seemed to suggest at that time was better than things built on datascript. Anything new?

lilactown14:05:41

I’m trying to get ahead of optimizations I know I’ll want to do later… is there an understood way to detect whether a particular query has changed based on tx-data returned from a transaction?

danielstockton14:05:49

@lilactown Currently looking into similar things. You might find https://github.com/comnik/clj-3df interesting, as well as https://github.com/CoNarrative/precept

danielstockton14:05:43

This is actually what motivated my above question too.

lilactown14:05:07

precept does look cool but I’m looking for something uncoupled from reagent

tonsky14:05:36

I don’t follow the field of reactive datalog, unfortunately. Can’t recommend anything

👍 1
lilactown15:05:22

yeah I’m not even really interested in the reactive part, more just trying to see if there’s a way to detect a query has changed that’s cheaper than executing the query itself

lilactown15:05:34

pull-based is a-ok for my use case

lilactown15:05:53

I’m just worried that as my app grows and runs over time, the DB will grow and number of queries will start to slow down my app

danielstockton15:05:53

There is also https://github.com/mpdairy/posh which claims it now works with multiple frontends.

danielstockton15:05:04

"Posh is also very fast because the in-component data queries only run when the database is updated with relevant data (found by pattern matching on the tx report)."

lilactown15:05:54

Yeah I actually rediscovered an issue I had filed like 6 months ago 😛 https://github.com/mpdairy/posh/issues/35

lilactown15:05:20

it’s made more complicated by the fact that React really doesn’t want to let be in control of when computation should occur

metasoarous16:05:13

^ Yes, however, Posh is doing rather primitive pattern matching and gets things wrong sometimes. Also, when a new datom matches a query, it still has to rerun the query to get complete results. So, better than nothing, but not ideal. 3df is very promising, and ultimately differential dataflow is probably the right solution to the problem (this was our leaning for datsync, before seeing the 3df talk at the conj this year). However, the down side there is that since its implemented in rust, you can't run clojure functions in reactive queries. Apparently, the rust part has been compiled to web assembly, but I don't know what the front-end deployment scenario looks like there; likely somewhat nontrivial.

metasoarous16:05:38

Would be great to have our own solution in cljc, but means implementing the underlying differential dataflow logic, which is not trivial.

metasoarous16:05:16

Precept is a rules engine, so a little different in scope, and RETE also doesn't support the full range of expressiveness as datalog (recursive rules, for example).

lilactown17:05:31

yeah I’m worried that in order to get a truly performant + ergonomic solution I would have to build something that integrates a little more tightly with React

lilactown17:05:46

which I’m not smart enough to do yet

metasoarous18:05:20

Nah; I don't think that's necessary. Are you a Reagent user? Reactions should be all you need. Honestly, if we're able to get 3df running on the browser, we may be good with a sort of hybrid solution where simpler queries that work on 3df get run through that, but those that need clj functions go through something more heuristic, like Posh (or just get rerun every time).

lilactown20:05:42

I'm not. I'm using just React and Hooks

metasoarous20:05:31

Well, there's your problem right there 😉

metasoarous20:05:28

j/k; Posh is designed to be fairly modular, and bindings have been written for both Reagent and Rum. I'm guessing someone could create a minimal set of bindings directly upon React + Hooks.

lilactown20:05:16

I’ll have to look deeper. my concern is that I’m trying to use the new React Concurrent features, which makes relying on external subscriptions problematic

metasoarous20:05:44

Ah; interesting. Thanks for sharing that context.

lilactown20:05:44

I think ideally I would have just some API that I can call:

(if (query-changed? tx-info query)
  ;; execute new query
  (q new-db query)
  ;; use cached
  cached-value)

lilactown20:05:14

this way I could store the DB in React state, and components that queried it could check to see if any relevant datoms updated / were added

lilactown20:05:19

this might not be feasible tho 😕

lilactown20:05:00

how does posh determine a query has changed? you said it does some kind of pattern matching?

metasoarous23:05:25

Datascript has a tx-listener/callback feature. So basically, every time a tx hits the db, it fires any attached listeners. Posh attaches such a listener that looks at the datoms to see if any of them match patterns associated with each of the queries to which you're subscribed. If there's a match, the queries are rerun, and if there's a change, (in the case of the reagent plugin) an r/atom representing the result is updated, and everything flows from there according to the standard Reagent machinery.

lilactown23:05:44

so it sounds like if I were to do a similar solution as posh (not fully incremental) that would work with Concurrent React, I would want to get rid of the subscription mechanism. I want something that I can synchronously transact, match, and query.

lilactown23:05:46

the listener/subscription sounds like basically a way of optimizing the computation of what has changed for the (possibly many) listeners on one query

lilactown23:05:02

I might be able to get back some of that performance using memoization

metasoarous23:05:45

You could probably use Posh as a lib, and run the checks against it that way. You wouldn't have to reimplement it necessarily.

lilactown23:05:18

that would be what I want yes 😅

lilactown23:05:36

I need to have time to sit and look at posh’s source

lilactown23:05:37

the differential data flow stuff sounds really interesting, but I don’t really get it… yet. do you think it would also work in this synchronous + immutable way?

lilactown23:05:39

actually if it’s wasm, it would I think be running on another worker so it would have to be calling me back anyway

metasoarous23:05:21

Yup; 3df will have to be async I'm afraid.

lilactown23:05:17

that would probably be OK for my use case. I’m slowly giving up on the idea of having “one DB to rule them all” when it comes to React applications

metasoarous23:05:33

Is there not a way to plug in asynchronous updates via React Hooks? I thought that was part of the whole point? (Disclaimer: I know very little about this newfangled Hooks business)

lilactown23:05:44

there is, you just lose out on certain things

metasoarous00:05:05

It depends on how much data you need.

metasoarous00:05:31

And how much querying you need

metasoarous00:05:39

Gotta jet; good luck!

lilactown00:05:50

😄 thanks for the help! ttyl

lilactown00:05:46

to complete my thought: Concurrent React implements async rendering (like Reagent) but also a scheduler to prioritize updates (which Reagent doesn’t have). In doing so, if you want something to be e.g. top priority (button click, keyboard event) then you need to tell React to re-render inside of the event handler

lilactown00:05:34

so if you are, for example, storing the state of a text input box that lives in an external data store that updates things via subscription, then you run the risk of blocking your text input from updating as the user types because some other updates come in that are the same or higher priority. async updates are automatically scheduled lowest priority.

lilactown00:05:40

there’s also the chance of your application state drifting from what’s being rendered on the screen. if you let React completely control the state of your application, your state updates are prioritized and executed the same as the UI updates and so they are always in sync

metasoarous23:05:49

The problem with this is that the pattern matching that Posh is doing is 1/2 way there towards a 1/2 baked implementation of a Datalog engine (Datalog is, after all, a pattern matching language). So you can pretty quickly see how much nicer it would be if everything was just operating in a real incremental/differential update flow.

Daniel Hines23:05:38

Re: 3df Just to be certain, the software we need to run in the browser is the 3df server, right? The clj-3df lib doesn’t do anything for us? Or am I misunderstanding?

metasoarous23:05:08

You're more or less right

metasoarous23:05:20

The server is what does the actual differential dataflow evaluation

metasoarous23:05:27

And is the part that's in rust

metasoarous23:05:37

The clj-3df lib is just making calls out to that

metasoarous23:05:38

@niko963 - Am I recalling correctly about the feasibility of getting the server part running on the browser? I seem to remember you saying that someone had already tried compiling to web assembly, but that there would be more work needed for a smooth deploy path. Do I have that right, and is there anything else you can add about the path forward there?