Fork me on GitHub

@hmaurer I have a few more question to really get to the bottom of this, but really don't want to stretch your kindness too far, so please don't feel pressured to answer if you're not inclined. 1. Cache invalidation configuration for the Optimistic Updates feature: For a UI component that is fully snappy, I would need to pull in a lot of data from the server in the manner of a Query on the component level. as far as I understand it, the Query is only run when the component is rendered, so say when the app first loads, this particular UI component is not yet in active state, so no Query will be sent. so here It really matters to me how the cache works: the UI component can be mounted and unmounted willy-nilly as the user decides to enable a widget and disable it. In this case, is it configurable by me to decide when the cache is invalid? i.e. I get to configure when the UI component can use the cache (if the data it requires was last fetched within the last 60 seconds) and when it invalidates the cache and decides to fetch the whole data again. the point here is that fetching the data on every mount/re-mount of the UI component is going to feel sluggish. 2. Performance considerations of memory space in the browser: how many entity nodes that refer to each other in a more or less Dense graph can I store in-memory? this is probably something that changes from browser vendor to another and is highly variable. 3. Performance considerations of entity node access: since an entity db (the value in the app db map that belongs to the entity key (e.g. :Person { 1 {...} 2 {...} ... })) is a flat map as described in 2.8. The Secret Sauce – Normalizing the Database. from my understanding, lookup performance of read from a persistent clojure map is O(log32N) only when the map is a balanced branched tree. is that understanding incorrect somehow? well the implication in my use-case is that lookup can be quite costly, meaning that looking up a node in the entity db might hang the UI while looking it up. am I wrong somehow?


@icyiceling184 no worries at all; any specific questions are most welcome 🙂 I will reply to (1) tomorrow morning as it’s getting late here. The other two answers are shorter: 2. I wouldn’t worry about that. Since the in-client db acts as a cache, it should only contain roughly what appears (or appeared) on screen. This should be a relatively small amount of data; nothing to cause memory concerns. Of course there might be exceptions, but I doubt your use-case is one of them. 3. I am not sure about the lookup performance of maps in Clojure, and even less so in ClojureScript, but this isn’t something I would worry about. The cost of map lookups in the in-client db likely has a very minor impact on rendering performance. Worrying about this is well beyond micro-optimisations, imho.


@hmaurer sure! looking forward to that reply on (1) Regarding (2) the thing is that this widget is actually somewhat special use case: it's a search bar that traverses a graph starting either from some root node or a specific node (depending on the search bar mode).


so really the amount of data that this widget needs to traverse is non-trivial


It should be north of 100K nodes at the very least.


the idea is to make this search bar trigger traversals on the graph interactively as the text changes on the search bar, and output the collected results of the traversals in a list underneath.


I guess since it's quite a rare use case I will have to give it an actual try and test the performance for myself.


@hmaurer I had a blast learning from you, thanks again and hope to get to talk to you once more. 🙂


wow the fulcro-todomvc example repo is so cool! following the code flow from client_setup to client_main to ui to api helped me learn a lot about the architecture. in particular, seeing the mechanism for a parent component to refer to what a child component requires for its :initial-state and :query clarifies for me the comparison David Nolen made between the way things like Redux / Re-frame / Elm architecture does state management in waterfall data flow (data being drilled from above through props, being managed at some component higher up on the UI tree), and the way things like Om.Next / Relay / Falcor (i think?) do things which is that every parent component specifies the data it needs (initial state or query) but also makes sure to consider the data that any of its child components need and passes it to them via the props mechanism, basically data flow from the leaf components to the root component, maybe ground-up data flow i'm still far from having a good understanding of things, but at least feeling that i'm making progress with the help of a working example. 😀


@icyiceling184 hmmm I would need more information on exactly what you are trying to do to give precise advice, but my feeling is that if you are trying to search through a graph with 100k nodes in real-time in the browser you will have performance issues unless you use some sort of index.


as in, if you shove 100k nodes in the denormalised database and try to traverse that on every keystroke I wouldn’t expect it to be smooth. If you build a smart index and use that to perform search queries then perhaps it will be fast enough to be smooth. If it’s not fast enough even with an index and you absolutely must search that graph in that specific way on the client then I would look into using a service worker to do the search asynchronously and display a nice progress indicator to the user.


And yes you seem to have a good grasp of the way initial states / queries compose 🙂 Have you watched David Nolen’s “Demand-Driven Architecture” talk? If you haven’t, I recommend it? (it was recommended to me by @wilkerlucio)


And now replying to (1) from yesterday: > as far as I understand it, the Query is only run when the component is rendered, so say when the app first loads, this particular UI component is not yet in active state, so no Query will be sent. Incorrect. Roughly speaking, when the Root component is rendered its query (which composes all of it children’s queries) is ran against the in-client db. This does not send any query to the server. It is your responsibility, as a developer, to call (df/load ...) when you see fit to send a query to the server and load the data in the in-client db.


> In this case, is it configurable by me to decide when the cache is invalid? The most obvious way in which the cache would become invalid is when you perform a mutation (say, mark a todo item as completed on a todo list). You would then update the cache in the mutation’s code. The cache could also become invalid over time, if you expect the data to change on the server regardless of whether the user performs mutations from his currently opened client. There are many ways to deal with this. You could trigger a new load on every page change for the page’s data, regardless of whether it is already in the cache or not. You could trigger a new load only if the data in the cache is “outdated” (e.g. has been there for longer than X minutes). You could trigger a new load every X seconds (a technique called “polling”). You could push notifications from your server when data changes over websocket and update the cache then, etc…


As they say, cache invalidation is one of the two hard problems in computer science 😬


I feel like this question got buried, so asking again. How does Fulcro correlate a client side mutation to a server side one? Is it just by the mutation name? I have a rather unconventional, as in not using lein new fulcro, application directory structure. Right now mine more like src/client/${project_name}/mutation/ or src/server/${project_name}/mutation/. So if there is some directory magic or namespace magic going on where the client “knows” where the server is, I’d like to try and set that up


@njj ah sorry, I thought you got an answer to this. It’s correlated by qualified name (mutation name + namespace) afaik.


in the lein template you’ll see that the same namespace is used for client and server mutations, even though they’re placed in different files (clj and cljs)


ok interesting, so they are both (ns project.model.user) - but because one is clj and cljs they don’t have a collision


and then fulcro recognizes these as being related


So will I be able to namespace my mutation files in a similar manner, even if they are in different directories?


@njj no; the directory has to correspond to the namespace. There might be another way to correlate client and server mutations, but that’s a question for @tony.kay I am afraid


cool thanks


@hmaurer thanks for the detailed answer, really helps grok the machinery at play here. I can already see how I'll be going back and re-reading that answer a few times more before fully making use of it. @njj sorry for accidentally burying your question 😅 thanks for raising it again and giving me the chance to see how that first assumption I had on it was verified by the more experienced @hmaurer to be the way it goes, i.e. the clj, cljs files containing the mutations are both using the same namespace as a convention.

👍 4

@icyiceling184 out of curiosity, what are you building and what brought you to Fulcro? 🙂


@hmaurer I'd be happy to tell you about it 🙂 This is a really important project for me.


The vision I have is a web application that has widgets very much similar to filesystem programs like File explorers displaying files / metadata about the files in various forms.


The trick is that there is no traditional filesystem here. All files are stored in a document db and all their metadata is stored in a graph db.


This allows me to present a graph-based filesystem through the web app, and eliminate the traditional restrictions filesystems impose (particularly non-graph organization)


@icyiceling184 hmm, so folders would be nodes in the graphs, and files would be contained in those nodes?


and what benefits do you see to a graph organisation of a filesystem vs a tree?


@hmaurer yep, however files won't really be contained on the nodes because that would probably be too expensive in practice. What I intend to do instead, is give files some ID which file nodes will store, and have the actual files contents stored in a dedicated document DB by that file ID.


@hmaurer the benefits of a graph organisation of a filesystem over a tree are huge, that is basically the crux of my motivation behind doing this


@hmaurer there are so many applications to this, but one that is closest to my heart is this: user-curated filesystem based on made-up semantics as they user sees fit.


basically imagine yourself being able to create something akin to Wikipedia that you can arbitrarily compose according to your understanding of the things you catalogue.


this is a power-tool for the modern information worker


tree-based filesystems are very limited, since a file can only reside under a single parent directory, but graphs don't have this restriction.


hmm I see, interesting


I could talk about this for days to be honest, but I'll go straight to what brings me to Fulcro


I want to get this to a working prototype so bad 😁


well tying back to your old question on performance: I wouldn’t load the whole graph in the browser for this; I would offload all the work to a server


well yes so I'm not a Clojure programmer, I'm mostly a fullstack JavaScript developer, but I've also been keeping my eyes on Clojure ecosystem and there are two things that captivated my attention: 1. DataScript on the browser 2. Om.Next and a better alternative to REST-based communciation


The first thing I tried actually before going to Clojure is use Neo4j on the server and querying it for my graph.


But then I wanted more than just slow server queries! I wanted to see if it's possible to do all my interactions on the client and propogate those to the server as just a persistence mechanism.


so that had me go down the rabbit hole into the Clojure ecosystem: I first thought DataScript would be a perfect fit for this, and synchronizing DataScript on the client with a Datomic / Neo4j db on the server might already have a good stack that lets me do this quickly.


Turns out I was wrong, and DataScript is not yet as flushed out as I would like in the sync story between client<->server 😕 that was a huge bummer for me.


So I almost gave up on the idea of client-side driving the interactions, just before I found out that Fulcro has both: 1. Graph-like db on the client 2. a complete stack story for Optimistic Updates


and here I am trying to see how to make the most out of this existing cutting-edge stack


@icyiceling184 would your files be stored on the user’s machine or in the cloud?


the files would be stored in the cloud only. only fetched by demand


but the metadata on the files is stored in the graph, and that is the most important data i need for the interactions.


@icyiceling184 and what sort of queries do you need to run on the metadata graph?


Search bar with results: - Get me all the graph nodes that have this name attribute match my provided text - Get me all the graph nodes that have this name attribute match my provided text but are also a child of this provided node File explorer: - Get me all the graph nodes that are either parents or children of this provided node


Yeah you’ll need a full-text search index for this if you are working on a big graph; you can’t do a naive graph traversal and expect good performance.


Parent or child is super cheap if the DB is in the right format


@hmaurer I've heard this before in the Neo4j docs, but please could you elaborate what full-text search index means? just to clarify, I'm not interested in grep-like search in the files (remember, those are not stored in the graph at all), I only need file-like search through the file names.


I just assumed you know what grep / file are like, but in case you're not familiar please let me know. those are the tools I know of from Linux (UNIX?)


at least going by the Wikipedia definition: >Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases (such as titles, abstracts, selected sections, or bibliographical references).


well, say you have a simple relational table (e.g. a table in postgres) which contains all your files metadata. And say there is a column name in that table, containing the filename. A query to get all the files whose name matches a certain string (naively) would look like:

SELECT * from files_metadata WHERE name LIKE '%foo%'
Now, if your table is large, this will become very slow. Why? Because the database has to scan through every row and check query the filename contains “foo” in order to return the results you’re asking for. You could make it faster by enabling an “index” on the name column of the table. This will instruct the database to construct a special data-structure optimised for querying the data in that column in specific ways. I am suggesting you do the same: construct a data-structure optimised to look for files by name.


@hmaurer wow that makes sense! every time you go into details I am blown away by how well you know these domains of knowledge. I've been grinning for the past 15 minutes straight just out of sheer joy of being able to talk about these things like this 😁 the Internet is awesome.


@icyiceling184 😄 you’re welcome. If you’d like a basic example of what an index is, google “binary tree”


or more specifically, “binary search tree”:


this is so cool! the most I know about this topic is some graph traversal algorithms I picked up in a Data Structures course. wish I had kept better notes on that 😅


@icyiceling184 😄 are you still a student?


@hmaurer nope, I actually didn't like studying that stuff at all (mismatch between curriculum and my interests)


I didn't pursue CS any further after that course


I've gone on to pursue web development after that as a better fit for my interests.


Ah, I see 🙂 Fair enough


actually Neo4j offers an API for full-text search that I've already been able to make use of on the server side. if I understood you correctly, it should be possible to construct a data structure on the client-side as well, that basically acts as an index for particular node metadata. what do you think about this? is this worth pursuing implementing that? obviously just driving the interactions from the server is easier...


@icyiceling184 I would personally opt for driving these interactions from the server, unless you have a very good reason to want to do everything in the browser.


If you do want to do it in the browser you could write a clojurescript wrapper for something like


hmm regarding the option of driving the interactions from the server: so basically the way this would work is say: 1. have some text in search bar, search button is pressed 2. search bar text sent to server 3. query is run with that text on the server graph db 4. results of the query are sent back to the client and listed below the search bar


this is actually not that bad for the search bar widget when I think about it. can have a spinner run on that for say 100ms-150ms or so after the button is pressed.


the user might not even feel it actually. of course that's a different interaction from having a text change trigger a query, which is a bit of a richer interaction.


but the problem that is more concerning to me is the other widget: file explorer. one way to do it is: 1. start at some predefined node (either Root or specific chosen node) 2. send query for the children of that node to the server 3. server runs the query on the graph db and returns the children 4. display the resulting children in the file explorer widget Sorry kind of thinking out loud here 😅


@icyiceling184 no worries at all 😛 Well, first of all, you mentioned 100k nodes, but that sounds like a looot of nodes for a filesystem. Are you sure it won’t be mor ein the ballbark of 1k nodes for typical use?


if it’s a knowledge organisations ystem for a single user


think of it as the url bar in your browser... it goes to google and back, and doe not feel slow at all (at least here it doesn't!)


i've been thinking on such a system as you describe, not only for files but for all kind of document/data


@carkh google is likely super optimised though, with edge servers, etc. But point taken


and my conclusion was that just a graph of node doesn't cut it ... some kind of tag cloud might be better... but then again that's not a new concept, and it didn't exaclty take over the world ...


maybe nobody ever managed to present it well enough


@hmaurer anything below 70ms will be good enough


@hmaurer @hmaurer it's a good question but unfortunately there is no going around the 100k nodes as a very realistic ballpark. remember. this is a power-tool. the vision is much grander than just user manually creates a file and saves it to per's filesystem. the idea is that data can be aggregated into a user's filesystem from more sources than just through the app directly. consider bookmarks on the user's web browser. that's something extremely useful to a power user! being able to keep track of URLs and metadata about them. something like Evernote's WebScraper extension is a great example of the kind of integration i'm looking for in the long run.


sorry that's WebClipper*


@icyiceling184: i'm with @hmaurer on this, the actual query should take place on the server, in that cloud you're already using


@carkh yeah true, anything under 100ms would be just fine. Then you can optimise further later if necessary


I already have some filesystem right now that stores bookmarks for me through a web browser extension, and I have about 1K bookmarks saved from just the last 3 months. my browser probably has at least 10K nodes from the last time I checked.


@icyiceling184 damn, bookmarks were my first thought. let me tell you what path i was envisioning toward a similar goal. I was going to first do a small chrome extension for bookmarks in a graph (mine would be acyclic but with "diamonds"), all client side with datascript and indexedb persisting. Then extend it to other data kinds (notes or whatever). Then finally, if the model seems good, make a server and migrate the data and query logic there


thanks for weighing on into this as well @carkh ! I'm happy to hear your thoughts.


that's how i came to investigate fulcro, looks like it has a good story not only for local and server, but also for easy migration from one to the other


i couldn't resist, funny how close our thoughts were


and that we ended up in the same place


@carkh wow seriously?! damn the Internet is so awesome. this couldn't be a coincidence


this is convergence of good ideas.


but as for Fulcro I haven't actually used it much at all yet... I would very much like to but it's not an easy learning curve for me (got better yesterday after Tony Kay fixed the TodoMVC example that allowed me to study its workings)


anyway so I think I'm really going to tackle this with the server driving the interactions of the app. @hmaurer the conversation with you has been invaluable for me and I cannot stress enough how much of it helped me mentally wrap around the problems I'm facing.


Glad I could be of help @icyiceling184 😛

Robin Jakobsson14:05:21

hi! do you know of any examples on how to generate a "production build" from a Fulcro project? basically for testing the generated JS with Chrome Lighthouse Audit.


The README in your generated project should cover how to build an uberjar. lein uberjar should do it. That includes the production js.


@jakobssonrobin hello! are you using the latest lein template? (with shadow-cljs), or something else?

Robin Jakobsson14:05:12

yes with shadow-cljs 🙂


shadow-cljs release <build-id> I think

Robin Jakobsson14:05:43

thanks, I'll try that 🙂 :thumbsup:


@njj you can give a defmutation a fully-qulified symbol if your nses on client and server are not the same. defmutation adds the current ns if you don’t ns the kw, so it acts like defn. Doing so lets you tell the IDE that defmutation works like defn, and then you get source nav. defmutation just outputs a defmethod. Mutations all run through on multimethod named mutate in the mutations ns…you can just write multimethods if you wanted (as described in the book), but multimethods lack decent nav support, which is why the macro exists.


Ok, so in my case I’d want to use defmethod m/mutate and namespace is accordingly. This way my nses don’t need to be the same on client/server? For example, I have a mutation named listing/initial-load. So I would do (defmethod m/mutate 'listing/initial-load) and the same on the server side as well?


I guess I’m confused as to what the server side counter part would look like for a defmethod m/mutate, since m/mutate doesn’t exist on the server side. @tony.kay


same on the server, but it is a diff multimethod/defmutation.

(ns does-not-matter
    [fulcro.server :refer [defmutation]]))

(defmutation x/y [params] 


just like on the client


the API is identical


while the client looks like (defmethod m/mutate 'x/y ...)


If you happen to use the same ns (e.g. vs cljc), then you don’t need to specify the ns in the defmutation


yeah they aren’t the same ns


server is like app.api.listing, client is app.mutation.listing


I often do this: CLJC file:

(ns app.something
    #?(:clj [fulcro.server :refer [defmutation]]
      :cljs [fulcro.client.mutations :refer [defmutation]])))

(defmutation do-thing ...)
:cljs (defmutation do-thing ...))


or put a clj/cljs file with same name in same package


@tony.kay Would server-mutate be the counterpart? defmethod server-mutate 'listing/initial-load


Doesn’t seem to work, but just wondering…


Use the source 🙂


that’s part of why it’s open


Haha yeah thats where I found this


(m/defmutation x/y [params]
  (action [env] ...))

;; same as

(defmethod m/mutate 'x/y [env k params]
   {:action (fn [] ...)})

;; same as

(ns x ...)

(defmutation y ...)


I am going through the and using Cursive as my editor. I am having an issue with reader macros. Specifically,... #?(:clj [fulcro.client.dom-server :as dom] :cljs [fulcro.client.dom :as dom]) Intellij wants the files containing this to be named *.cljc which seems reasonable. So I made that change and now the build does not find the namespace e.g. The required namespace "flechar.ui.root" is not available, it was required by "flechar/client.cljs". The "flechar.ui.root" is in .../src/main/felchar/ui/root.cljc. This is under the control of shadow-cljs.


Sometimes renaming the extension requires a compiler restart due to caching.


other times that message happens because some other require in the file you just renamed is not, itself, CLJC, so one side or the other (when compiling it) fails to find the referenced file


hi, it’s not intellij but rather clojure(script) itself that dictates needing .cljc files if you’re going to have code for more than one platform in it. Are you doing SSR? You typically can just use .cljs unless this is the case. In any case, can you paste in your full ns declaration


server side rendering


where you’re basically pre-rendering the page


if not, you probably don’t need to bother, and can leave the file as a .cljs


and only require [fulcro.client.dom :as dom]


no, not at all. SSR is something you may do to optimize your app. if you’re just getting familiar with fulcro


don’t bother


just leave the files as they were generated by the template


is there a way to have a hiccup-like dsl on client side with fulcro ?


ok, I will revert it. thanks @eoliphant


I guess it’s possible, but not sure how much it buys you @carkh. Also, you wont be able to take advantage of some of the shortcuts for localized CSS and what have you


@eoliphant been using the like since my common lisp days, but that whole react thing makes it a bit harder i guess


no big deal, just curious


Though there is undeniable power in just manipulating data structures and have these interpreted later


How to send multiple (fm/returning state components) in a defmutation?


(each one with a fm/with-target)


yeah I used reagent/re-frame a lot prior to fulcro @carkh, was missing it for a little bit, but after a while, not so much so 🙂


@eoliphant thanks for answering !


I sometimes end up with my own little data dsl for forms and what have you


i'm yet to investigate that part of fulcro, usually doing server validation with a roundtrip as the user types (with a bit of debouncing) yes that requires a bit of trickery =)


yeah. I’ve done a few experiments. Nice thing about clojure(script) is that they tend to be less expensive. RIght now, I’m tending towards specs in .cljc files.


mhh i dislike the idea of have the client doing any kind of validation, it just doesn't have enough data, then you need to do it again at server anyways


and that's not the 100ms delay max that will make a huge difference to the user experience


just establish a protocol for form validation, then use it everywhere is my motto =)


well yes and no, there’s frequently ‘basic’ stuff you can do like say SSN’s have a variety of structural rules that can easily be checked on both client and server. But then I might need to call the actual validation web service on the server


and to your point, if your latency, etc is low


you can even do ‘as you type’ server validation


true i never had to call a service outside my "world"...that would be a good reason for that


I've tried defining 1 action and multiple remotes inside the defmutation, but it didn't workout


Are you trying to use the same remote? If so you just need two diff mutations. A mutation join is what is sent, and there is only one kind of thing that can be returned. If you’re using two diff remotes, then I would think it should just work.


but both against the same: no support…you can’t change a single mutation into multiple on the same remote


I was misunderstanding the documentation. Thanks for the support.


@carkh I have often wished to have some sort of magical validation framework where you can specify your validaion rules declaratively and it automatically splits what can be executed on the client and what has to be executed on the server (e..g because it needs db access)


@hmaurer haha well let's see if someone comes up with something.... i'm usually a bit wary of those "does it all by itself" things though


never tried that before. You’re trying to call two services @patrickcms ?


@carkh yeah I am too, but it seems like something that a machine should be able to do. Maybe in a couple of years with AI advances 😄


can't wait for it though =)


Well strictly speaking, reader macros sorta give you that magic for free, we’re using some of it on a few projects, something like

(defn validate-ssn 
   [context ssn]
   (:clj "check the string format, call a web service, db")
   (:cljs "check the string format"))


of course it lacks the ‘automatic’ part 😉