This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-09-06
Channels
- # announcements (14)
- # babashka (12)
- # beginners (61)
- # biff (2)
- # calva (16)
- # clj-kondo (22)
- # cljdoc (7)
- # clojure (131)
- # clojure-europe (52)
- # clojure-losangeles (9)
- # clojure-norway (54)
- # clojure-spec (5)
- # clojure-uk (4)
- # clojurescript (18)
- # cursive (14)
- # datomic (19)
- # deps-new (14)
- # emacs (8)
- # events (7)
- # fulcro (6)
- # graphql (3)
- # hyperfiddle (42)
- # instaparse (5)
- # lsp (10)
- # malli (21)
- # nbb (1)
- # off-topic (3)
- # pathom (3)
- # polylith (7)
- # reagent (14)
- # releases (2)
electric is working as expected here, but I thought it'd be interesting to share a problem I ran into with using odoyle for highly dynamic (ui-triggered, conditional, fine-grained, not subscribing to every database update) subscriptions: 1. button changes some data 2. data triggers subscription, which changes UI rendering path, creating a new child subscription 3. that new subscription has no data, because the data which would have triggered it was inserted in 1. odoyle has no way of asking for data that was inserted before the subscription was added, as RETE is strictly for forward inference. My plan is to duplicate writes to asami for backward inference, and when the continuous flow is initialized it queries asami first (conveniently with the same datalog triple syntax), solving the empty initial state problem and allowing ad-hoc queries in general
that’s what materialized does as well, an initial batch query and then differential subscription from there
yeah, it took me some thinking to realize that these are two completely different problems
Ah, that's such a smart pattern. It steps around the event sourcing rehydration problem in a very elegant way.
the other solution is the rama / golem.cloud / dataflow solution, where you make the language state itself durable and all changes to prod are "over the air updates". I dont know how Rama does it; i know Golem intends to restrict the set of legal code changes to those that don't require historical replay, and otherwise force you to write a state migration
asami not supporting upserts was a little annoying, but it still was pretty elegant, coming out to ~70loc including debug lines. now to figure out how to replicate writes across the network and have server->client reconcillation
for others, i had to look it up https://github.com/threatgrid/asami
FYI, cljdoc build failing due to missing core.async
: https://app.circleci.com/pipelines/github/cljdoc/builder/45561/workflows/e16a6237-979e-40af-b1b1-6cff4ff2608d/jobs/61936
https://cljdoc.org/d/com.hyperfiddle/electric/v2-alpha-428-g22937f75/doc/readme
yeah we know, we tried to fix it when we merged your PR but failed and ran out of time
Ok, cool At least it's on the radar 🙂
we will try again soon, unless you know the answer
i believe we tried :skip-doc or some such
I do not unfortunately; may take a look at some point, but it'll be a few weeks
When ChatGPT got electrified 😃 https://twitter.com/pappapez/status/1699450096063029537
Interesting! I wonder if it's picking up on your enthusiasm - and if that's why it's asking follow up questions. If the same content was provided without joy, would it still want to know more?
Could be, but I’m a joyful kind of person. If this would trigger it’s curiosity I should have been interviewed before. Could be a combo. I remember it was also very impressed with Missionary, when I told it about that the other day. But there I fed it the README and various documents and code examples, so it draw it’s own conclusion from that instead of relying on mine (which are probably terribly flawed anyway).
I was actually impressed with how well chatGPT caught on to electric once I basically just told it I was using a "clojure dom library called electric" and copy pasted the code i had then asked it for changes. It was probably the best results I've had with "I'm using a clojure library to do X, here's some examples and here's what I want to do" It's probably just that it understands html/the dom really well and electric is close enough that it doesn't struggle with the conversion. But I've definitely noticed it struggling more with trivial conversions in all other domains.
I should start a thread where I feed it all I can find about Electric, like I did with Missionary.
https://github.com/hyperfiddle/electric/blob/master/src-docs/user/electric/electric_compiler.clj#L10-L41, can't believe I hadn't come across it before
low-importance question, what are the differences between current electric and upcoming differential electric?
The general principles will stay the same. New features are :
• ability to inject an arbitrary differential collection. If the database supports it, that means zero runtime diffing all the way through browser.
• all expressions are multiplexed by default, so e/for-by
is not bound to a single peer anymore.
• better performance due to reduction of network round-trips
Did that answer your question @UABU2MMNW?
> If the database supports it Curious if there's any known options at the moment that would support this?
if they're prioritizing this I wonder if they're planning to build a db on top of electric
> all expressions are multiplexed by default What does it mean for a collection to be multiplexed? > ability to inject an arbitrary differential collection I'll take a stab at this one. In this case a collection might be a result set from a database, and a differential collection might be a result set from a database that can report fine-grained changes to the result set reactively – basically, a notification that members have been added or removed from the result set.
> build a db on top of electric A logical next step, but also more of a leap than a step 😆
> What does it mean for a collection to be multiplexed? Multiplexed expressions. Each expression implicitly works on items of a list, as if it returned multiple values.
(e/client
(e/render-table
(e/render-row (e/server (e/inject result-set)))))
In this example e/inject
takes a differential collection and returns all items in the collection. All derived computations also run on every item, it can seamlessly traverse the network, the ui layer can translate it to dom operations, etc.I'll tell you a story! When designing the next ui namespace we wanted to tackle a seemingly simple thing - render a todo list. When you have a todo-list UI you have an input field and a list. The list is server data. You e/for-by
over it on the server and stream the diffs to the client. Now the client can do fine-grained updates to the list. You see this pattern all the time
(e/server
(e/for-by key-fn [v (query)]
(e/client
(dom/div ...))))
What happens when you type in a new todo item? .... Hmm, that's client data. The value goes to the server over the wire. The server transacts it to the db. The db acks the write. The server e/for-by
streams the diff over. Now the client can finally render your new item! But ugh, the latency!
What can we do? Ideally we'd like to optimistically render the in-flight items in the same e/for-by
list. But it's on the server!
So what semantics are we looking for exactly? Well, we want to optimistically include a client value in a server collection and when the server responds with success/failure the value stays or gets removed or gets a red X with a retry button or ... Up to the user. The information should be available for you to decide.
So, can we add a client value to a server collection? No..
Differential electric to the rescue! If we operate on diffs, we can fold over them in any way we like. We can take client and server diffs and fold them into a single e/for-by
-like loop. This is what we mean when we say e/for-by
is not bound to a single peer anymore.
Of course this solves many more use cases and opens up many new possibilities.
Hopefully @U053XQP4S will correct any wild inaccuracies 😉took a few readings, but now makes perfect sense and seems almost obvious in hindsight
it's funny, I bet everyone working on, say, a REST API has at some point had the realization that it is much easier to design all of your queries to return collections even if you happen to be building a user
endpoint rather than a users
endpoint. It makes perfect sense to me to lift that all the way back to the expression level.
I do wonder what electric's learning curve will look like eventually. "if you think you had to unlearn things when you started writing Clojure, wait until you try electric..."
the differential stuff is internals from the beginner's perspective, e/for-by should look the same as it does today, it's the implementation of it that changes. The big idea is that today e/for-by is a userland macro and that macro does diffing in userland; after differential electric, this logic is absorbed into the electric runtime and e/for-by is just syntax sugar over the amb
multiplexing
stuff Leo was talking about. The advantage of having the electric runtime own the diffing is that the network planner can be aware of it and take advantage of it.
When you use a streaming datasource like Materialized, you don't even want e/for-by, you want "the second stage of it" (the reduction over diffs), not the diffing part since Materialized gives us pre-diffed output. So differential electric also unbundles this better.