Fork me on GitHub
#hyperfiddle
<
2023-09-06
>
nivekuil00:09:16

electric is working as expected here, but I thought it'd be interesting to share a problem I ran into with using odoyle for highly dynamic (ui-triggered, conditional, fine-grained, not subscribing to every database update) subscriptions: 1. button changes some data 2. data triggers subscription, which changes UI rendering path, creating a new child subscription 3. that new subscription has no data, because the data which would have triggered it was inserted in 1. odoyle has no way of asking for data that was inserted before the subscription was added, as RETE is strictly for forward inference. My plan is to duplicate writes to asami for backward inference, and when the continuous flow is initialized it queries asami first (conveniently with the same datalog triple syntax), solving the empty initial state problem and allowing ad-hoc queries in general

👍 2
Dustin Getz00:09:38

that’s what materialized does as well, an initial batch query and then differential subscription from there

nivekuil00:09:55

yeah, it took me some thinking to realize that these are two completely different problems

telekid13:09:57

Ah, that's such a smart pattern. It steps around the event sourcing rehydration problem in a very elegant way.

Dustin Getz14:09:27

the other solution is the rama / golem.cloud / dataflow solution, where you make the language state itself durable and all changes to prod are "over the air updates". I dont know how Rama does it; i know Golem intends to restrict the set of legal code changes to those that don't require historical replay, and otherwise force you to write a state migration

👍 2
👀 1
nivekuil21:09:45

asami not supporting upserts was a little annoying, but it still was pretty elegant, coming out to ~70loc including debug lines. now to figure out how to replicate writes across the network and have server->client reconcillation

Dustin Getz15:09:08

yeah we know, we tried to fix it when we merged your PR but failed and ran out of time

Garrett Hopper15:09:32

Ok, cool At least it's on the radar 🙂

Dustin Getz15:09:40

we will try again soon, unless you know the answer

Dustin Getz15:09:51

i believe we tried :skip-doc or some such

Garrett Hopper15:09:12

I do not unfortunately; may take a look at some point, but it'll be a few weeks

pez16:09:45

It thinks I should get started supporting Electric better in Calva. 😃

teodorlu18:09:02

Interesting! I wonder if it's picking up on your enthusiasm - and if that's why it's asking follow up questions. If the same content was provided without joy, would it still want to know more?

pez18:09:55

Could be, but I’m a joyful kind of person. If this would trigger it’s curiosity I should have been interviewed before. Could be a combo. I remember it was also very impressed with Missionary, when I told it about that the other day. But there I fed it the README and various documents and code examples, so it draw it’s own conclusion from that instead of relying on mine (which are probably terribly flawed anyway).

👍 2
jjttjj18:09:01

I was actually impressed with how well chatGPT caught on to electric once I basically just told it I was using a "clojure dom library called electric" and copy pasted the code i had then asked it for changes. It was probably the best results I've had with "I'm using a clojure library to do X, here's some examples and here's what I want to do" It's probably just that it understands html/the dom really well and electric is close enough that it doesn't struggle with the conversion. But I've definitely noticed it struggling more with trivial conversions in all other domains.

pez19:09:37

I should start a thread where I feed it all I can find about Electric, like I did with Missionary.

telekid18:09:32

low-importance question, what are the differences between current electric and upcoming differential electric?

leonoel19:09:28

that's not a low-importance question !

🙌 6
leonoel19:09:14

The general principles will stay the same. New features are : • ability to inject an arbitrary differential collection. If the database supports it, that means zero runtime diffing all the way through browser. • all expressions are multiplexed by default, so e/for-by is not bound to a single peer anymore. • better performance due to reduction of network round-trips

Dustin Getz19:09:03

Did that answer your question @UABU2MMNW?

Garrett Hopper19:09:41

> If the database supports it Curious if there's any known options at the moment that would support this?

nivekuil20:09:32

if they're prioritizing this I wonder if they're planning to build a db on top of electric

Dustin Getz20:09:29

Rama, Materialized, anything streaming

👍 2
telekid13:09:34

> all expressions are multiplexed by default What does it mean for a collection to be multiplexed? > ability to inject an arbitrary differential collection I'll take a stab at this one. In this case a collection might be a result set from a database, and a differential collection might be a result set from a database that can report fine-grained changes to the result set reactively – basically, a notification that members have been added or removed from the result set.

telekid13:09:42

> build a db on top of electric A logical next step, but also more of a leap than a step 😆

leonoel13:09:37

> What does it mean for a collection to be multiplexed? Multiplexed expressions. Each expression implicitly works on items of a list, as if it returned multiple values.

(e/client
  (e/render-table
    (e/render-row (e/server (e/inject result-set)))))
In this example e/inject takes a differential collection and returns all items in the collection. All derived computations also run on every item, it can seamlessly traverse the network, the ui layer can translate it to dom operations, etc.

Dustin Getz13:09:57

"multiplexed" is basically amb from sicp

👍 2
leonoel13:09:51

yes, amb is the special case where collection has fixed size

xificurC13:09:03

I'll tell you a story! When designing the next ui namespace we wanted to tackle a seemingly simple thing - render a todo list. When you have a todo-list UI you have an input field and a list. The list is server data. You e/for-by over it on the server and stream the diffs to the client. Now the client can do fine-grained updates to the list. You see this pattern all the time

(e/server
  (e/for-by key-fn [v (query)]
    (e/client
      (dom/div ...))))
What happens when you type in a new todo item? .... Hmm, that's client data. The value goes to the server over the wire. The server transacts it to the db. The db acks the write. The server e/for-by streams the diff over. Now the client can finally render your new item! But ugh, the latency! What can we do? Ideally we'd like to optimistically render the in-flight items in the same e/for-by list. But it's on the server! So what semantics are we looking for exactly? Well, we want to optimistically include a client value in a server collection and when the server responds with success/failure the value stays or gets removed or gets a red X with a retry button or ... Up to the user. The information should be available for you to decide. So, can we add a client value to a server collection? No.. Differential electric to the rescue! If we operate on diffs, we can fold over them in any way we like. We can take client and server diffs and fold them into a single e/for-by-like loop. This is what we mean when we say e/for-by is not bound to a single peer anymore. Of course this solves many more use cases and opens up many new possibilities. Hopefully @U053XQP4S will correct any wild inaccuracies 😉

👍 4
telekid14:09:07

Thanks for the explanations! really cool

telekid15:09:26

took a few readings, but now makes perfect sense and seems almost obvious in hindsight

telekid15:09:42

it's funny, I bet everyone working on, say, a REST API has at some point had the realization that it is much easier to design all of your queries to return collections even if you happen to be building a user endpoint rather than a users endpoint. It makes perfect sense to me to lift that all the way back to the expression level.

telekid15:09:32

(obviously the motivations are different)

telekid15:09:22

I do wonder what electric's learning curve will look like eventually. "if you think you had to unlearn things when you started writing Clojure, wait until you try electric..."

telekid15:09:39

I guess that will work itself out though

Dustin Getz16:09:05

the differential stuff is internals from the beginner's perspective, e/for-by should look the same as it does today, it's the implementation of it that changes. The big idea is that today e/for-by is a userland macro and that macro does diffing in userland; after differential electric, this logic is absorbed into the electric runtime and e/for-by is just syntax sugar over the amb multiplexing stuff Leo was talking about. The advantage of having the electric runtime own the diffing is that the network planner can be aware of it and take advantage of it.

👍 2
Dustin Getz16:09:33

When you use a streaming datasource like Materialized, you don't even want e/for-by, you want "the second stage of it" (the reduction over diffs), not the diffing part since Materialized gives us pre-diffed output. So differential electric also unbundles this better.

👍 2