Clojurians Log v2

We have successfully raised money with Electric in our tech stack. And we have completely disregarded “aimed at internal tools, not rich UX”

@shkertik I would say the main consideration is the number of concurrent viewers. Electric is not marketed as a solution for a landing page that gets millions of views, but well-suited to authenticated dashboards and rich real-time applications. LLMs have or will solve hiring risk (IMO). An electric instance can only handle N viewers (@dustingetz can confirm on which hardware), so you need to scale edge nodes and deal with distributed data issues. The edge scaling is straight-forward, but dealing with distributed data syncing can be tricky, esp. if you are subscribing to real-time data feeds. E.g. if you're using Datomic tx-listeners, backend will re-query on every transaction, for every user, if you don't shard your subscriptions. There are ways to materialize these views, but it's not "out of the box." I suspect there exists a nice consulting/mini-product business for Hyperfiddle if they want to licence incremental view maintenance components for popular databases.

The ideal Electric experience for me would feel like https://github.com/sixthnormal/clj-3df, where you can write Datalog queries (or SQL queries? ew), an incremental dataflow component analyses those queries and maintains optimal real-time query subscriptions for every active user. Maybe Rama is the answer here.

Rama can provide fine-grained updates to the level of granularity of the transform used on the data. I.e., if you replace a map with another map, you’ll just get a blob update. But if you do something like [(keypath :hello) (termval :world)] in the same map, you’ll get a smaller diff.

The diff language looks like this: https://redplanetlabs.com/javadoc/com/rpl/rama/diffs/Diff.html, so it does cover reordering/adding/removing in sequences, which I think is what Electric is primarily concerned with when it comes to diffing. As far as I know, Electric doesn’t diff to a more granular level than that at the moment.

the electric diff model is not less powerful than rama's, iiuc Electric's representation is more powerful because it supports dynamic changes to DAG topology, and this higher level of expressiveness is why Rama and Electric diff models are slightly different. IIUC, when Rama presents a single diffed document with KeyDiffs at granularity of leafs, Electric would represent that as nested differential sequences. i.e., in Electric you represent a reactive document in layers through composition and each layer is its own sequence

Rama doesn’t diff documents though. A pstate is mostly a combination of ordinary data structures. When you open a proxy (ie., a reactive query) against it, you specify a Specter path. The result of querying that data structure with that path gives the diffs you receive. If you point it at a map, you’ll get keys added, values changed, etc. If you point it at something sequential, you’ll get sequential diffs, and diffs on the things contained in the sequence. So you could get diffs like “item at index 4 has moved to index 0”, but you can also get eg. a https://redplanetlabs.com/javadoc/com/rpl/rama/diffs/SequenceIndexChangesDiff.html with “key and value added to map at index 1" or “key removed in map at index 4” and so on (if it’s a sequence of maps).

Electric gives you the primitives you need to get fine grained reactivity at any point or path in a document. There will be a mechanical translation between the flattened Rama API and the non-flattened Electric API which does not smush "move index 4 to 0" and "assoc 42 at path [0 :foo]" into the same operation

Rama has a very spectre-centric document oriented approach, it's weird

I am sure there are great reasons, it does look like the same structures that many hyperscale databases offer

Alright, I guess I don’t understand what a document is. Either way, the diffs describe a tree of changes. I’m not arguing for better or worse here, but there’s something more than changes at leaf nodes going on.

it would be better to have this conversation after you have spent some time with electric v3

I think in most cases, the tree is going to be pretty flat. From Electric, it’s also convenient to break up queries and colocate them with views for cleaner code. I.e., get the sequence, iterate in Electric, then get the values in each call.

the only reason i commented at all is because people read these threads anonymously/silently and make assumptions about us that are not true

Ah, sure. What did I write that was incorrect specifically? (Note: I don’t feel attacked or anything like that, I want to update my understanding)

As far as I know, Electric doesn’t diff to a more granular level than that at the moment.this sentence is misleading. It's true specifically that the e/diff-by operator performs shallow diffing. BUT we claim that deep diffing is equivalent to shallow diffing at multiple layers, treating each layer of a nested associative document separately. And that teasing apart the layers is more expressive, which means Electric can meet Rama's API with no loss in fidelity or granularity, but i dont think Rama can meet Electric computations in the same way. To be clear it does not matter, "UI" and "Backend" are different problem domains that model different computational structures (similar but not identical), so it is unsurprising that the abstractions are not identical

Ah. No, I just meant that my understanding is that Electric doesn’t diff anything that you don’t treat as a sequence. Ie., in a sequence of maps, the maps would be considered atomic unless you iterate them by k/v.

also different performance reqs, different dataset sizes, different memory consumption, different hardware requirements - client web browser vs cloud cluster

Ie., updating the value of the key in a map doesn’t generate a key diff in this case.

You are still "thinking in Rama", i really would prefer to have this conversation after you have spent more time with electric v3

Alright

Your investors care about that level of technical detail?

@markaddleman not themselves, but the question might come up if they ask for a third-party review at some point

Interesting. If they do, I would just confidently say that your team has reviewed your requirements and the technology landscape and you’ve concluded that Electric is most effective technology to build the UI quickly. I don’t see how they would argue.

~@styles

🤦 Thanks - actually it was the & which had crept in from a previous version. I still feel a bit dirty using a macro for this - is this the right approach to compose props? (an aside, I know hiccup has been discussed and doesn’t make sense till the fundamentals are all in place, but it would be nice if we could assume that a first arg map was props, and that a bare string was text, at which point I probably wouldn’t care too much about jumping to hiccup)

quick answer - we considered a sugar like this a couple years ago (as we also were annoyed by the verbosity, everyone is) but it had two major limitations, i'd need to reload context to state exactly what they are with any accuracy. (1) There is a very subtle performance issue with respect to whether you are passing map literals (comptime static), or runtime map objects. (2) I recall an expressiveness issue (i.e. abstraction leak) but i forget the details

I think we do have a design hypothesis written up that has good syntax and resolves the issues but it hasn't been a priority to implement

is your macro the right approach? I'm not sure, we will need to think about it

hi Tim yes indeed

> Does Electric Clojure support runtime changes to the flow (new queries, ad hoc queires, etc)? yes

Re glitch-free propagation - if you ignore Electric's network distribution bits, Missionary provides glitch free propagation guarantees for a single process reactive computation (including the case of runtime changes in topology), Leo gave a talk about the propagation alg https://www.youtube.com/watch?v=xtTCdT6e9-0. The differential structures are more or less orthogonal to this

perfect, I'll give that a look, thanks!