Clojurians Log v2

Do the deletion events clearly identify the deleted event[s] somehow? ie, A unique ID of some kind?

ps. 👋 back. Esoteric but important Q: are you using observers to mutate the outside world in response to MX data? Those are hard to take back. I am thinking meanwhile about the general case of altering history. 🤔

yes the deletion events contain an id that matches the id of the input event we’d want to delete

re the observer question, if I understand you correctly I don’t think that’s a concern. We’re essentially streaming events into matrix and then periodically querying it for the output state that we’re accumulating and passing that along

I think this is like git commit history, if you will. To change history, we need to delete the event[s] and replay the survivors. Two issues: 1. If memory is tight, we want to have the obsolete story line GCed. If not we can just flag it as a dead branch; and 2. When replaying survivors, from where do we start? The beginning would be easy, but if that is too expensive we need to figure out a world state to which we can start feeding the rewritten history. Aside from git analogies, my mind wanders to those goofy physics conjectures in which the particle both decays and does not decay, in two different universes spawned just to handle that uncertainty. Do we copy the whole Matrix...well, it's too late, an MX is stateful. In a case like this, I like to try the easy, slow fix and see exactly how slow it is. Sometimes it screams. If not, one possibility would be to continue processing the now contaminated history in one thread while catching up in a second, with the app switching to the second thread when it has caught up. This also lets us just toss the second thread if another deletion comes along. Let's pin this down some more, if you like, and I can get more definite on approaches. Feel free to take to DM for anything proprietary, ps. "MX is stateful" but Clojure is efficient with its immutability. Can we take efficient snapshots of an MX by recording the contents of every model and cell?! Might just be possible, as costly as it sounds.

thanks, that’s very interesting. I think your suggestion of trying to replay from the start is a good one, and seeing how that goes performance-wise. My hope is that the deletion events are pretty rare so a full refresh is not something we’d need to do all that often.

Yeah, I was thinking deletions might not be super common, which gives me hope for the replay all solution. That solves the "rewrite history" problem (by not rewriting) but the implementation might get interesting around the mid-stream cutover. In my Algebra app I used brute force to find optimal solutions, and this involved a branching solution space that had to be dynamically pruned to not crush the CPU. Getting branches to stop growing once they were clear losers was tricky, as was getting the solution tree to grow breadth-first. I do not think this case is as bad, but you might hit some headscratchers. Ping me if you get stuck.

I should have mentioned that the moral with the Algebra app was that functional/declarative is great fun, but then we need to do something imperative, which is usually easy, but now we need to find a functional/declarative way to stop the beast, without losing the declarative nature. Fortunately, MX supports MSET! et al in observers, aka watches, so if we detect a need to kill off a thing growing off events, we just give it an "abort" input cell and MSET! that from an observer. This would be trivial, and more performant than a pure solution that might, on each event received, scan all events received for a "deletion", to decide if each new event should be processed.