Fork me on GitHub

Hi! I am looking for some advice on something. I am working on an application which involves an entity for which I want to track all changes as a sequence of events (aka that entity is event-sourced). When the user performs actions on the frontend I want them to be immedialy reflected on the state of the entity (aka added to the event log and reduced), but I also want them to be synced with the backend. My question is: how do I ensure that these events are synced in order?


If I simply send every event to the backend I believe there won’t be any order guarantee


@hmaurer so, Fulcro guarantees that mutation order is preserved (and goes before reads). This is per individual client.


Are you asking a distributed systems question? i.e. are you asking how to get a total order among all users?


Fulcro is quite good at normal event sourcing, though, because of the client guarantees, and the fact that the mutations are already "event-like"


@tony.kay no. Perhaps I should give you a little bit of context. I am building an exercise platform for students. I want to store the sequence of actions that lead to a particular state in an exercise, as this is useful to later analyse and see where the students went wrong, etc


@hmaurer for investigating there is also . It's built-in so not much work to add it.


@U3LP7DWPR that’s… amazing


Exercise states aren’t shared; they’re per-student


then you're fine. Fulcro will preserve order of mutations based on when the events occur


Amazing. thank you 🙂 How does it do that?


The network plumbing has a queue. Each mutation is processed locally, then each is processed of remotes (in the order they appeared in the tx expression). When remote-processed, they are placed on that outbound queue in that order


this preserves can be disabled, but it is on by default because otherwise things get hard to reason about


And just to be clear: you are telling me that if I have the exercise state on the frontend, which I update on every action, and I have the exercise state on the backend, which I also update on every action (sent over the network), the frontend state and the backend state won’t go out of sync? (well, at least events will never be applied in the wrong order?)


correct...though you will have to deal with error recovery yourself


oh I see, so there is only ever one in-flight mutation


e,g, network outage


again, you can disable that, but it is the default so that reasoning is logical


if there is a network outage the worst case scenario is that the backend state would be out of date, right?


no, the worst case scenario is lost mutations


your UI has state that the back end does not


So if I have mutations A, B, C (in that order) on the frontend, it’s possible (due to network outage) for the backend to see “A, C”?


Yes, unless you add recovery logic for errors


Ok. Thanks!


If you use Fulcro's websocket support with auto-retry and make sure your mutations are idempotent, then that will auto-recover for you as well (and allow for easy server push back to the client)


idempotent is relatively easy: Give each mutation a UUID (generated when you call it, and sent as a parameter) so the server can know whether it has applied it or not...if it sees it twice, just ignore the repeat.


Ah good point. I was going to say that it could be a bit annoying because some of my mutations are “add a line”


but your UUID solution sounds simple enough


I think the websockets with auto-retry is the easiest, since the recovery is detected and managed at the network layer.


You could write a similar auto-recovery remote for normal xhr, but I have not done so...still, it's not that hard


Great, I’ll try this then. I have a limited amount of time for this project and have never used Fulcro before but I want to build my frontend in Clojure and it seems like the best choice


(great documentation by the way)

tony.kay14:10:22 sure to watch the youtube video is really helpful for beginners


I watched a few! Great as well 🙂


I’m impressed by the amount and the quality of documentation around Fulcro

👍 1

It’s especially nice to see a lot of focus (both in writing and in the videos) on the “why” and not the “how”


Well, it’s a healthy balance of both, but I found the documentation and videos did a good job at explaining some of the underlying ideas


Which is great because that’s transferable knowledge


I'm a "why" person more than "how"...I want to know how, but why matters a lot. Glad it helps you too.


I'm curious: Does it come across that the UI ends up being much more decoupled and such? I think this is one of the big differences in Fulcro: React and many libraries encourage you to "push shared state" up the UI tree, which creates tight coupling. Also, wondering if it is fairly obvious that instead of event-like systems where code navigation becomes impossible (where is this thing being changed?) you have much more directly traceable logic? I've been thinking a lot lately about what distinguishes Fulcro in the world, and I'm wondering if those things are visible enough, or if people are just lost in the shuffle.


Of course, the combo of Pathom wizardry I think takes it to the next level, too...but I hope that becomes clearer when Wilker gives his take at the upcoming Conj

☝️ 3

BTW, everyone should watch his talk when he gives it:


I will be able to answer those questions better once I’ve dipped by toes a bit more into Fulcro; I haven’t written anything with it yet (beyond a hello-world type application).


> React and many libraries encourage you to “push shared state” up the UI tree, which creates tight coupling.


What do you mean by this?


What I can tell you is this: the way you introduced Fulcro as a way to render UI on top of a graph database spoke to me a lot. Perhaps this is because I’ve dabbled with Datomic and generally like thinking about graphs (I study math…), but it felt very natural.


A question that came to mind while I was watching early videos is “what if your UI doesn’t map 1-1 to data in your graph database?”


It seems like Fulcro provides a very nice API for rendering and mutating data that maps 1-1 to UI components


At the time I couldn’t come up with good examples where my UI wouldn’t map 1-1 to data, and for all I know it might be very easy to handle those cases with Fulcro anyway, so it’s not really a concern.


i have used GraphQL pretty extensively as well, so I think that also meant I was familiar with some ideas before reading about Fulcro (i.e. declarative data queries colocated with components)


> Also, wondering if it is fairly obvious that instead of event-like systems where code navigation becomes impossible I am not 100% sure what you mean by this; could you elaborate a little?


@hmaurer i'll attempt to answer some of your questions, though i'm sure Tony would do a much better job: 1) React encourages composition of data (props) from parent to child; this means that the parent must know precisely what each child needs, and so data that is shared between children gets pushed up the UI tree, and couples the parent tightly to each child. in Fulcro, each child specifies what it needs itself, without having to know about the parent, and the parent doesn't need to know specifically what the child needs, it can just ask for it (e.g get-initial-state, get-query). 2) not quite sure what you mean by 1-1 mapping, but: you can have components that don't interact with the DB at all, and you can have multiple components that take a view on the same piece of data in the DB 3) in systems like DOM events, a change can come from anywhere, so finding out where a given change has come from / why it's happened is difficult, since - at the limit - you need to check each and every one of your event handlers. with Fulcro, there's a much stronger correspondence between given changes and mutations, and given mutations and components, so it's much more feasible to infer where a change is coming from, and why a given mutation has been fired


That actually answers my seems obvious once you use it, but not as an external new user 🙂


on the 1-to-1: it is rare your UI perfectly matches your schema...graphql shares this problem: you expose a graph, and that does not necessarily match your UI shape. Fulcro addresses this in two ways: post-mutations of loaded data, or processing data through query parsing engines like pathom. The combo of Datomic and Pathom on the back end make it extremely easy to make a server API that can morph with your UI...pretty amazing, actually.


On event systems (any kind of system where the logic "happens" as the result of something else, but isn't connectable via code navigation tools)...In many systems you hook up something to a component via a separate mechanism (e.g. reducers, event listeners, etc.) and evolve application state that way. When you go to trace such a thing your code is hard to navigate, because the "side effecting" bits are so disconnected from your UI that you have trouble finding them. With Fulcro, you evolve things with mutations, which you can just "jump to" directly from the UI element in question. E.g. you click a button and it runs a mutation: Pressing CMD-B in IntelliJ can jump you to the mutation definition. If you instead have, for example, "Fire event X" and some number of things can register to "handle X" at runtime, now you have the problem of figuring out "who registered for X?"....which no tool can help you with because it is a non-deterministic run-time problem.


so you end up with a "run it and see" kind of development experience just to figure out what code to even look at


Yeah, it's sort of weird, because up front it's quite frankly 'a lot' (and as others have said, kudos to the depth and breadth of docs). my personal experience was that on my first pass through I sort of 'got' it, but still kind of felt "but reagent/re-frame is so much easier", and it is until lol. Until you start banging out code for your client/server interaction story, until after reading up on best practices for organizing your re-frame db (indexed entities, etc) you code your version of normalizers, etc, until you want to make your components some flavor of "tree aware" and code that. etc etc. So in the end you still end up with 'a lot' whatever you use, but it seems more the tradeoff between initial learning curve and having to 'fill in the blanks' and maintain those bits as well


yeah might be worth trying to market the full stack story more?


@fiyaz right, so (1) is what GraphQL enables in the vanilla JS land. (2) I mean having a component that interaction with different parts of data tree? if that makes sense. instead of a specific node


(1) right, Om Next & Fulcro were inspired by Relay and Falcor (2) well, you can query for more than one piece of data from the level the component's at, as well as more than one ident from the graph in general (e.g. having the query (`[[:person/by-id 1] [:fruit/by-id 1]]`), or from the root (`[[:root/key _]]`)


@eoliphant Good description. That's the feedback I've heard from others as well...I hope to make it as easy as possible for people to see why you'd take the time to learn it, when other things "look" easier. It's a bit daunting, because initial developer convenience is often the biggest criteria ppl look for.


@tony.kay “The combo of Datomic and Pathom on the back end make it extremely easy to make a server API that can morph with your UI...pretty amazing, actually.” => out of curiosity, have you used Datomic yourself? And do you have a good story regarding access control? Back when I dabbled with Datomic it was my biggest question-mark


(I haven’t used Pathom/have no knowledge of it)


I use Datomic, yes. I talk a bit about that in the book.


Any graph system is going to have similar problems with do you keep people from walking somewhere you don't want them to go?


> With Fulcro, you evolve things with mutations, which you can just “jump to” directly from the UI element in question. E.g. you click a button and it runs a mutation: Pressing CMD-B in IntelliJ can jump you to the mutation definition. Is there a fundamental difference between this, and, say, dispatching an action with Redux?


the answer is the same as all other systems: you have to implement something.


@tony.kay yep the problem is by no mean limited to Datomic, but since you seem to know and think a lot about graphs I’m wondering if you have insights on how to tackle it 🙂


I suggest putting security checks in the server-side to look at graph edges, and set permissions on those


If you're using on-prem you can do a lot with d/filter


(and thanks for your in-depth reply!)


or, in the case of more specific queries, you write the query yourself and that self-limits where they can go (e.g. you assume a query starting with X only needs Y because that's all that is allowed, and don't trust the client)...all sorts of ways


Where do you write the action in Redux? It's in a swtich-case somewhere, isn't it? And it's a string, right? So, how do you find it when your code gets big??? Grep?


as far as access control. In some cases, you can keep access control out of your queries, by just passing them a pre-filtered db. We're doing new stuff with Cloud/Ions, and it's an annyoing omission from the API at this point


my point is that as you scale apps up, having tools that can instantly navigate your code become more important, and a developer is less able to "keep it all in their heads"


maybe the tools and techniques have improved...I've not looked at Redux in a while, but from what I've seen it's a mess to keep track of stuff in most systems...and it can be in Fulcro, too...nothing keeps you from making a mess, but the central design is one that is meant to try to be better at this


it's the primary reason there's a defmutation you can have tools treat mutations like functions, even though they are implemented as multi-methods...multimehthods are non-navigable with the always get taken to the defmulti line

👍 1

yeah on that front, pretty much any clojury equivalent is cleaner and less crufty than Redux.


I can't imagine trying to scale Redux to a 100KLOC


put a fork in my eye first, I think

😄 1

we've been moving to fulcro, for some of the reasons I described, but to me, say even re-frame's events/subs are far easier to manage than redux's reducer soup that you end up with as the app grows


From my perspective scaling is about a bit of decent design, and then a lot of bookkeeping, organization, and navigability. You want decoupling and flexibility, but some forms of decoupling also sacrifice navigability


I love it that in Fulcro I can write any component, even with full stack behaviors in a dev card, and then drop it anywhere in the app, and then pick it up later and move it. I just wrote a file upload resource manager in a dev card, complete with full-stack data tracking, upload, image serving, and management...and at another job wrote a UI designer (with drag and drop, schema nav, etc.)...both cases wrote serious full-stack parts of applications in cards, with full knowledge that context in the main app didn't matter.


throw in the component-local css, and my stylings travel along...


and with pathom resolvers, I don't even have to worry about where I compose it in, since the server can adapt


I was just looking at reason react and apollo stuff yesterday. Looks like in those graphql systems you kind of have to tie a component to a, you wrap the component in a thing that can get the data. OK, but then what if two components want to share that data? I guess you "push it up the tree"...but that means a lot of crap gets pushed to (or very near) root. But that only solves server shared data...what about client app state you go to solve that some other way...most people punt and it seems like a lot of apps then shove a lot into component-local state (and push a lot of that up the tree), but now you've got hidden mutable state everywhere, and a lot of tight coupling. Fulcro's got some shortcomings, too, but so far I'm pretty happy with the general solution compared to the other options.


@tony.kay I am using React with Apollo on a project at the moment actually! Queries compose in the same way as fulcro (get-query), but it’s true that for UI-only concerns, such as “is editing”, I tend to put that in component-local state (even though Apollo technically supports client app state it’s not that straightforward to use iirc)


Out of curiosity what are the shortcomings of Fulcro ?

👀 1

@tony.kay is there a simple way to add a UUID to all mutations?


@hmaurer Not 100% sure it's the right approach but you can use the request-middle ware to add a UUID to your post request.


@hmaurer I'd just write a function that transforms txes:

(defn idempotent-transact! [component tx]
  (prim/transact! component (mapv (fn [ele]  (let [ast (prim/query->ast1 ele)  ast2 (update-in [:ast :params] assoc :uuid (random-uuid))] (prim/ast->query ast2)) tx)))


Any reasoning other than being more explicit behind the fn vs. middleware choice?


that's almost certainly not quite right, but it is the general idea


then use your function instead of transact...also, you'll want to make sure you only transform the mutations, and not the follow-on reads (ast :type = :call, I think)


Fulcro shortcomings: 1. Choosing clj/cljs as a tool chain has some big bonuses, but the speed of the compiler can get in the way as projects get large unless you're careful. 2. It's a novel model, which can be hard to adapt to, and hard to get a team to coalesce around unless they are motivated to do so. 3. It stresses that you put most everything in app state in order to get debugging and other features, but that approach can lead to performance issues unless you use things like union queries (e.g. defrouter) to make sure your queries stay tight. 4. Union queries are confusing to people, and tend to clutter you app state in ways that I find unappealing 5. Loads always start out putting data in the root, which can cause accidental collisions. The load system couldbe redesigned so that targeted stuff just goes to the targeted location, but I have not had time. 6. Routing isn't as full-featured as it needs to be, but really should have been a separate library (or libraries), since it is an area where ala carte would have been better. You can still do that, of course, but people see it in the lib and assume it's the way to do it. 7. The "read after write" problem isn't as well-solved as I'd like. ptransact is decent, and @wilkerlucio’s additions in fulcro-incubator db-utils are possibly a good add-on, and I've not had time to further refine that into a mass-consumable story. As a result, when the problem comes up, there's usually a bit of struggle. It's doable, but could be a lot better. There are times when you actually need to transact in things like a post mutation, but I discourage it, which is confusing 😜 For example, I just wrote a file upload facility where you need to upload, see progress (possibly cancel the upload), and on upload complete create a resource (which should be remote transacted and can fail) and then allow them to edit the resource attributes (e.g. is the thing you just uploaded for X or Y, what are the tags on it, etc.). So there is a file upload mutation that triggers another remote interaction, that triggers a UI interaction....In js you do this will callback chains. In fulcro, you use ptransact, combined with embedded calls to ptransact. And all through that you want some error handling support. It's not pretty anywhere in js-land, but I think we can actually do a lot better than the current implementation.


I could probably go on. Obviously I still use it for everything and think it is great...but there's always room for improvement.


good question. depends on networking...if he was using the websockets for auto retry, then middleware doesn't apply

👌 1

Makes sense, thanks! And is that due to some technical limitation or it's just not implemented yet (or, perhaps, not wanted)?


to do it in middleware you'd have to implement the (xhr) networking to do the retries, and then you might as well do it there 🙂


although, now that I think about it, you could make a protocol wrapper for the remote-with-retries, and use middleware for the uuids


feel free to contribute any of this, by the way, anyone who cares to 🙂


just not implemented. I had a consulting client that wanted the auto-retry in websockets

👌 1

so it exists there 🙂


Oh, I think I expressed myself poorly. By "that" I meant "no middleware for websockets"


I see, yeah, middleware wasn't part of the original Untangled anywhere, and the http remote got it because people were using it and often having to write a new remote to get what they needed (esp talking to things non-Fulcro) it was a critical hole


websockets isn't nearly as heavily used, and no one has made any noise at all about needing mroe from it. And I have limited time.


so, no squeak, no grease 🙂

👍 1

@hmaurer and OMG just trying to read the apollo docs to figure out what the "state of the art" is, and I see:

<Mutation mutation={updatePageNameMutation}>
        {updatePageName => (
          // outer div elements
          <li className=“sidebar-item” onClick={() => updatePageName({ variables: { name: ‘React’} })}>React</li>
          // other list items and outer div elements


I mean, people complain about syntax quoting???


I'm also curious, if anyone knows because I don't use Apollo and don't see it in the docs: besides "refetching" and "polling", how would one do an update to a remote thing and see the change...e.g. optimistic updates?


@tony.kay do you personally use the websocket remote more often?


and for optimistic updates.. hang on


I use both kinds of remotes


depends on the project


there is a way to specify an “optimistic response” in a mutation


well, in “mutate”


I do this in a few places in my application


I see, but it still does a refetch it seems


to ensure it went ok...that's the error handling story I guess


How so? I don’t think it does a refetch (but I haven’t diged into the internals)


It says in the docs on that page


"Optimistic UI provides an easy way to make your UI respond much faster, while ensuring that the data becomes consistent with the actual response when it arrives."


I suspect it uses this optimistic response to update the store until the server replies, then just substitutes it for the actual response when it arrives


and it breaks encapsulation as have to talk about optimism at the UI level


ah I see what you mean; it’s not a refetch. I don’t know how much experience you have with GraphQL but basically mutations have outputs. You can specify what data you want to return from the mutation


that’s what the optimistic response simulates


Yeah, you can do the same in Fulcro


but there is no need if you know it went well


Mmh unless some data needs to be computed on the server, no?


that's the case where you want to ask for it, yes


but you're right, it sounds like perhaps they let you say "trust me" on the optimistic response


By the way, completely unrelated but have you heard of Postgraphile?


oh, it's like that other one...


I have been using it on a project lately. In short, it generates a graphql API by reflection over a postgresql schema




nah, different


It’s really neat and useful; if/when I have the time I would like to build something similar on top of Fulcro


walkable is a lib like that for Fulcro/PostgreSQL integration


it has been a life-saver for the project I am currently working on (internal platform for an NGO, very CRUD-y)


but I highly recommend you try out Pathom


I think it's the bees knees


Pathom Connect and any back-end storage infrastructure is amazing


Definitely will use it as the backend for my upcoming Fulcro project. It looks similar to how I am used to write graphql servers; haven’t digged into it enough yet to see if there are any differences and where it shines


@tony.kay one thing that confused me a little bit in the doc is how you seemingly pass UI component instances to functions to mutate data, i.e. This makes sense in the context of binding data 1-1 to UI components, but it still threw me off a bit at first


Again, this might be due to the fact that I haven’t yet carefully read the doc, and if that’s the case don’t waste time answering 🙂


Another quick question: is there a preferred way to handle backend validation / “soft errors” on mutations with Fulcro? i.e. “email already in use” for a signup, etc. Any error that should be reported to the user and which won’t occur if he takes some action as a result


@hmaurer That particular case is (first question) leverages the fact that the component is known to the mutation to do local-only kinds of data updates, but is really for that specific kind of case (as opposed to a general mutation).


transact! is also called on component instances, right? (when it isn’t called on the reconciler directly)


yes. Using a component will cause ref to be set inside of the mutation env, which is the ident of the component.


unfortunately legacy name...really, "ident" should have probably been "lookup ref"


early versions of Om Next mixed the terms in kind of arbitrary ways before settling on "ident", but this particular spot never got updated to the final term


you can also call transact with an extra arg which is an explicit ref, to set the "context" of a mutation (e.g. set ref in env to an explicit value)


In general, I tend to make sure that mutations are not component-centric, nor tie them to ref, so that they are not context-specific (which can be confusing)...but in some cases a mutation only makes sense when used in a context, and that is the case this stuff is for.


(e.g. I might want any component to trigger a UI routing change, but a checkbox control is the only thing that should be messing with the local :ui/checkbox prop)


@tony.kay that makes sense; thank you!


the function itself just calls transact, and is wrapped in a function for concision


mutations returning a result (second question)


in Fulcro, a mutation can return any need not be tied to the component query