Clojurians Log v2

I think it would be helpful to have an upfront cheatsheet for all the matrix fns. Just a list of the main functions, what the name stands for, a brief description, and maybe a link to a more detailed doc. I might also consider using all of the matrix functions via an alias like matrix/mget even if that's not idiomatic usage. The main reason is that for someone new to the library, it's not easy to tell which functions are part of the example code, which are part of matrix, or even which functions might be builtins/utilities. It's also helpful to have a typical set of :requires for your example code. For some of the examples, the explanation is after the code. I might consider a more literate form and moving the comments inside of the code. As a reader it makes it easier to follow since I don't have to jump back and forth between the explanation and the code it explains (which I have to figure out). It also helps if I copy and paste code locally to have the explanations in place. I was curious about asynchrony. I'm glad that's covered! I might also include a section on exceptions.

Another thought: The https://github.com/kennytilton/web-mx/blob/main/src/tiltontec/example/quick_start/README.md#review covers a bunch of cool features in matrix. As a hook, it could be interesting to summarize the review section in the beginning and build your way back to it by the end. The classic 1, 2, 3 of 1. tell them what you're going to say 2. say it 3. tell them what you said

Thanks for the detailed response, and valuable perspective. Just what I was looking for. I will take a shot at embedding the commentary. I have done that before and felt it made the D/X seem more tortured than it is, but lemme try again. It may well turn out better by eliminating the postcript commentary. Thx! I will also add a prelimary glossary of the sundry APIs used. The requires? Ugh. I have had separate threads on how my "small file" coding preference leads to hell-ish requires. I plan a major source overhaul to get them down to one per Cells, Model, and Web/MX. But I'll toss them in now as is. Exceptions are a tough one: none of the APIs do anything special about those. Do you have an example of exception handling in mind? Thx much!

> It may well turn out better by eliminating the postcript commentary. Thx! In some cases, it might be useful to have both. My take is that it's ok to be a little redundant in docs, tutorials, and guides. In some cases, it's even helpful. Especially, if it makes sections standalone and supports skimming and jumping around. > Do you have an example of exception handling in mind? If you have a formula and it throws an exception. What happens? Obviously, that's a situation you want to avoid, but it's going to happen in development, and eventually, production.

Agreed on the redundancy. Don't know how many times I have missed things by reading too fast. I'll start by commenting the code, and leave the postscripts mostly alone, for folks not looking at the code. As for exceptions, that's a tough one. Based on how Cells works, I do not think it can help make an exception survivable -- too much internalese would be broken. So for now it is up to the dev to code their own exception handling. Do you think some general handling would be feasible?

I don't know enough about how matrix works to say if general exception handling makes sense. Error handling can be really tricky and becomes more important as an application becomes more complex. One realization just popped to mind is that my perspective of matrix is to view it as a kind of database. Is that the right perspective? If so, I then try to think about stuff like atomicity, consistency, and isolation (from ACID without durability) which leads to questions like: • Are there transactions? • Can you update multiple cells "at the same time"? • What happens when transactions "fail"? • Are there asynchronous transactions? • Is there a convenient way to specify data model constraints? • Can you take a snapshot of the current db?

@smith.adriane wrote: "my perspective of matrix is to view it as a kind of database. Is that the right perspective?" No, but you remind me that I once used multiple inheritance between MX and Franz's persistent CLOS metaclasses to make an AllegroStore DB reactive. That was a huge success. But there AllegroStore provided the rollback. MX does have a tight definition around state change: https://github.com/kennytilton/web-mx/blob/main/src/tiltontec/example/quick_start/README.md#data-integrity. I think that covers atomicity and consistency, and MX internals enforce single-threading, so isolation is trivially delivered. Transactions are implicit: if I change X, before the change operation even returns it has propagated the change fully. But I do not have to tell Matrix when to update things together, because it knows the dependencies, and it refreshes everything at the same time I used to have with-one-propagation macro that would support multiple cell updates to be made/propagated together, but that looks like it did not survive the Common Lisp to CLJC port. I will go look for that. It was dead simple, IIRC. If a change hits an exception, I think nothing can be guaranteed because MX mutates cell internals as it propagates. Back in the day I often got way with Common Lisp exception restarts after fixing bugs, and the app almost always carried on successfully, but I would never say that was guaranteed. Async processing is surprisingly easy, but MX started as a GUI library where one never knows when the next user input will come along, so maybe I should not be surprised. The trick is to have an input cell where an async request handler can inject the response, as just a normal state change. Then the above data integrity guarantees are met as usual. In the Review example, https://github.com/kennytilton/web-mx/blob/f8f39c4a998efce56782af8d3e2ad0e3a607f650/src/tiltontec/example/quick_start/lesson.cljs#L420, we battle async "wind drag" triggered by a js/interval with the "+" control that adds speed. Again, single-threading of updates makes this straightforward. As for a snaphot, as something that could be loaded and resumed? No, not without a ton of new code. ps. Re asynch: in my CLJD port, because everything in Flutter/Dart seems to be async, I hacked an :async? option into MX. I plan to port that back to CLJC RSN.

For atomic, I was thinking "a transaction will either completely succeed or completely fail". It seems like transactions aren't in the atomic in the presence of exceptions. For consistency, I was referring to enforcing constraints. These are database concepts, so may not directly apply. However, it might be worth keeping in mind since matrix plays a critical role in managing state. In principle, it's possible to have async an asynchronous transaction that reads data, takes an asynchronous action, and then writes data to the database, but only committing the transaction if the read data hasn't changed in the meantime. Again, more of a database concept and may not apply here. > Transactions are implicit: if I change X, before the change operation even returns it has propagated the change fully. There's a few motivations behind asking if you could update multiple cells at the "same time": • efficiency: batching can sometimes be more efficient than propagating individual changes • effects: if you have a side effect that is triggered due to an input change, it's common to be able to group inputs into one "logical" input to make sure there's a correspondence to the number of output side effects with the number of logical inputs. (eg. if you mark all todo items as complete, how do trigger a single notification rather than a notification for each todo that was changed?)

Oh, OK. Since MX does not handle exceptions, not atomicity. As for multiple mutations in one go, agreed. I found this in my CL code:

(defun call-with-one-datapulse
    (f &key
      (per-cell (lambda (c prior-value prior-value?)
                  (unless (find c *the-unpropagated* :key 'car)
                    (pushnew (list c prior-value prior-value?) *the-unpropagated*))))
      (finally (lambda (cs)
                 (print `(finally sees ,*data-pulse-id* ,cs))
                 (loop for (c prior-value prior-value?) in (nreverse cs) do
                       (c-propagate c prior-value prior-value?)))))
  (assert (not *one-pulse?*))
  (data-pulse-next :client-prop)
  (trc "call-with-one-datapulse bumps pulse" *data-pulse-id*)
  (funcall finally
    (let ((*one-pulse?* t)
          (*per-cell-handler* per-cell)
          (*the-unpropagated* nil))
      (funcall f)
      *the-unpropagated*)))
  

Btw, here is a similar exercise I did around my JS version. Note the glossary on each tab. Might be the way to go. https://codepen.io/kennytilton/pen/mXQNYR?editors=0110

I guess I got a little off topic, but the answers are interesting regardless 😄. Mostly I was just trying to think out loud a bit about my initial impressions. Here's a more generalized summary: • My impression of matrix was a "database-like" thing which seems to be the wrong model. It's probably the hardest part, but if there is a elevator pitch for a matrix mental model, that would be helpful. • When I evaluate libraries, I like to know what hard problems they help me solve (which is why I was thinking about the tangentially related db questions). • The other question in my head when checking out a new lib is "Where's the data?"

This might help, @smith.adriane: https://tilton.medium.com/the-cells-manifesto-b21ed10329f0 Key excerpt: "{Cells] is useful for any application involving an interesting amount of long-lived state and a stream of unpredictable inputs. Two examples are any GUI-based application and a RoboCup client (trying to simulate a soccer player given a stream of sensory data fed it by the game server)." The hard problem is the one that made Facebook give up on MVC and invent Flux: they were forever struggling with keeping the displayed "unread chat" count in sync with the actual number of unread messages. This is also the problem Fred Brooks identified in "No Silver Bullet": as the number of inter-dependent states grows, the difficulty of keeping them consistent increases exponentially. Facebook would agree. 🙂 The mental model? I hesitate because all us reactive folk use the analogy and it always fails, but these kinds of reactive systems are exactly like spreadsheets for program state. They solve automatically the problem faced by an accountant using a paper spreadsheet in 1960 after they changed a cell: what other cells now need to be recalculated, and in what order. Systems like Matrix and MobX not only solve that problem, but unlike the Flux pattern, they do so without any subscribe/publish by automatically detecting dependencies. Now the analog is a spreadsheet application, where I write my formulas using any other cells I like and the software puts it all together. The other thing a spreadsheet does is naturally guide us into solving large problems in small pieces, with no one cell being all that hard to calculate. Divide and conquer and all that.

Found the video where FB discussed the chat problem. You can feel their pain. https://www.youtube.com/watch?v=nYkdrAPrdcw&t=783s

> {Cells] is useful for any application involving an interesting amount of long-lived state and a stream of unpredictable inputs. This problem description screams database to me. Why not use a database?

"Why not use a database?" I want the color of the border of an input field to be red if the value is incorrect. And changing color as they type, after every keystroke. Where does a persistent database come into play? How does it help? Let me try. So I write the bad value to some database, hopefully in memory -- what kicks off the assessment of the new value, which is our key requirement here in terms of consistent state (the border matching the correctness)? And what kicks off the color change, depending on how the assessment goes? These are the questions any reactive framework addresses, property-granular or Flux. Does our database know about state dependency, and know how to order propagation to avoid glitches? Glitches arise when state dependency gets too complicated. Hmmm. I guess the crux of the reactive matter lies in dealing with derived data. In my example, the color does not reflect the entered string, it reflects the correctness assessment of the entered string. And chains several times longer. Does that sound like a static database? Perhaps a stronger viewpoint: different elements will have a different assessment of the same data. How do clients express different requirements of the same data in a database? Sounds like we have to move individual GUI elements (in some form) into this imagined database, so dynamic change propagation can get to the color property of the style attribute of a DIV. I am OK with that if we have an in-mem database with change propagation built-in based on formulaic specification of DB entities, but then that would be Matrix with the persistence. But what does persistence add to a GUI consistency challenge? Or by "database" did you mean sth that supported dynamic transaction rollback?

Generally speaking, a database allows you to accrete or insert new information and provides tools to leverage the available information. Not every database is durable. Many of the new dbs are in memory or have an in-memory flavor (datomic, asami, datascript, relic, etc). Many dbs offer tools for both derived data and responding to newly available information (eg. materialized views and triggers). I'm not particularly focused on durability. I'm mainly interested in dbs for providing semantics for inserting new information, maintaining consistency, and potentially, triggering effects. I think https://github.com/wotbrew/relic is a particularly interesting example.

Relic does look cool! But now we have a separate store, a la Flux. Working on a widget, coding up the color property of the style, I stop and go look in my Relic (or whatever) database, and there is a problem: after a quick glance at the Relic doc, I do not see how application validation of the field value gets into the relic DB. But even if that could be arranged to run automatically every keystroke, I am still at a distance, doing extra work to pipe the separate store info back into the widget style color. In Web/MX, MX manages the state "in place". I agree it sounds like a database, indeed I like to say a w/mx app is its own database. The important thing is having the style color an actual "database" datum, intimately bound with the rest of my code, such as the field validation. To me the hallmark of a useful library is that it is invisible. The whole "separate store" idea makes its bet on separation of concerns, meaning taking the database implicit in a web app and making it explicit. The problem being, great, now I have to wire my app and store together, which is how we get the hell of boilerplate. I love to say "divide and conquer", but not if I am now going to be forever running back and forth building bridges between the two divisions. Does that make sense? As clearly as you see a database, I see the value of managing state in place.

> The problem being, great, now I have to wire my app and store together, which is how we get the hell of boilerplate. I think this is a false dichotomy. You could have an API identical to what #matrix provides that uses a db to store the data. Just because you have convenient API tailored towards a specific use case does not imply that you can't build that API on top of a simpler, more general API.

Also, I'm not trying to imply that matrix doesn't have a simpler, more general API underneath. I'm only saying that you could implement the same API with a db underneath.

So whether or not there's a db in the mix is a separate question from boilerplate or convenience.

"You could have an API identical to what #matrix provides that uses a db to store the data." How will that DB get designed? btw, Have you used re-frame or, heaven forbid, Redux. re-frame is vastly nicer, but it is still a distinct DB with its own schema. I once fixed a bug that arose because the app evolved to have two different subscriptions for the same collection. One got loaded at start-up, so things worked fine until we implemented user additions to the collection. The person implementing that fed the new item into a different subscription, one the app was not even watching, although the name of the subscription was temptingly appropriate. In Web/MX, the "DB" is just the interface I am building. ie, There is no DB. I mine the interface structure for data. There is no separate effort, which is why there is no friction connecting the interface to the DB. The next level of this comparison comes with the inevitable, never-ending revision of the interface. As the interface gets shuffled, there is nothing to do, the state moves with the components. I think original MobX offered the same "in place" state directness.