This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-12-06
Channels
- # adventofcode (71)
- # aleph (1)
- # announcements (6)
- # aws (1)
- # babashka (27)
- # beginners (60)
- # biff (7)
- # calva (3)
- # clj-kondo (3)
- # clj-yaml (1)
- # clojure (60)
- # clojure-europe (43)
- # clojure-nl (3)
- # clojure-norway (75)
- # clojurescript (16)
- # code-reviews (7)
- # css (4)
- # cursive (47)
- # datascript (4)
- # events (5)
- # fulcro (37)
- # gratitude (5)
- # hyperfiddle (4)
- # introduce-yourself (4)
- # joyride (23)
- # juxt (4)
- # malli (4)
- # membrane (64)
- # nbb (8)
- # off-topic (12)
- # other-languages (6)
- # pathom (6)
- # polylith (9)
- # random (3)
- # rdf (66)
- # reitit (3)
- # releases (2)
- # shadow-cljs (18)
- # tree-sitter (10)
@tony.kay I noticed that I was having really bad performance with attributes using RAD’s automatic resolvers that return large maps I traced the issue to:
secure-resolver (fn [env input]
(->>
(resolver env input)
(auth/redact env)))
where auth/redact is doing:
(defn redact
"Creates a post-processing plugin that redacts attributes that are marked as non-readable"
[{attr-map ::attr/key->attribute
:as env} query-result]
(p/transduce-maps (map (fn [[k v]]
(let [a (get attr-map k)]
(if (readable? env a)
[k v]
[k ::REDACTED]))))
query-result))
I believe the problem is that pathom’s transduce-maps
will walk the entire returned data structure
(defn transduce-maps
"Walk the structure and transduce every map with xform."
[xform input]
(walk/prewalk
(fn elide-items-walk [x]
(if (native-map? x)
(with-meta (into {} xform x) (meta x))
x))
input))
I believe transduce-maps should be checking for ::p/final true
in the metadata before prewalking, but it currently does not, so using the pathom ::p/final
trick doesnt help us here.
However, I’m not sure if thats true for all uses of transduce-maps
in pathom (and I suspect not). However, We could write a custom transduce-maps
that does in the RAD code if you’re open to that.
Let me know if you have any thoughts on a generic solution here, or if one should just not use RAD for large attributesPathoms elide-not-found
, which is included in the RAD parser as a plugin:
(p/post-process-parser-plugin p/elide-not-found)
exhibits the same behavior, but will only effect large items returned in the final outputI'm happy to take performance improvements. The authorization redact isn't even desirable in my opinion. Coding that at a resolver level that has security wrappers is a better way to go, but it was a quick and dirty demonstration of how you could do things. It had not occurred to me that it would be a performance problem, although it should have, because I've had similar issues with the merge routines. Those leverage the query, and that allows me to stop traversing early when there's a large value that doesn't need walked.
Leveraging the query is the best solution, because when there's a prop in the query you can just stop, and when there's a join, you can pull the subquery and continue. I'm not sure that final would be an answer, because a resolver can technically return nested structure; however, if you're talking about the auto generated resolvers, then having an algorithm that only walks a single level at the resolver would be the best solution, and eliminate the recursive walk all together
I’d be fine with removing the redact as the default as well.
as for an alternative fix, I was thinking of not traversing values with ::p/final
, as that seems to be how pathom generally tackles this problem, but open to just a single-level traversal as well
should i start with a PR to simply remove the redact call in auto-resolvers?
ahh yeah sorry
the final idea doesn’t work…because a resolver can return nested data that MIGHT need to be affected
The whole idea of final is to be able to return any arbitrarily complex value of any depth without further processing…but post-processing must apply to that value in this case
(e.g. if I send a password back 3-levels deep in a final result, I’d want the password redacted still)
ahh yeah fair point
Are mutations within a transaction guaranteed to be ran synchronously? (I think yes, but I need confirmation)
You can get local synchrony if you want, but mutations that are full stack can't. If you use a single remote, then order is guaranteed.
And optimistic actions also guarantee order. Independent of the number or type of remotes.
I think the question might be: will mutations run in the order they are defined in a transaction?
For the optimistic parts, yes. For any remote parts: they will be submitted in order, but if more than one remote is involved there's no way for it to guarantee the return order, or even the processing order for that matter.
What about pessimistic transactions? If I understood it right, than this will run and return in order. The next transaction is only run after the last transaction finished. So if you are concerned about the return order then you would have to run pessimistic transactions, wouldn’t you?
Yes I actually wanted to know if they’re run in order. So any mutation involving mutations on the client database will be run in order, right? Things like merge/merge-component!
, dr/target-ready
, uism/begin!
, uism/trigger!
?
correct. There is a guarantee that (the optimistic part, for local client database) mutations are run in the order they are submitted.
They are also submitted to remotes in the order transacted…but Fulcro cannot make guarantees about the arrival order of remote mutations to multiple remotes, or the completion order.
Ok. Coming from re-frame, this is actually very enjoyable 😁 thanks for your answers
@U023LKF2PQV A transaction in Fulcro is a vector. Vectors imply a required order. So, a single call to transact!
has the semantic of an in-order sequence of operations:
[(f) (g)]
means that you’d like f
to run before g
. Fulcro is in control of two things; The client database, and the network queues. So, the best it can do is run the local (optimistic) operations against the client db (in order) and submit the remote behaviors in that same order (see also pessimistic transactions, which, if both operations have remote operations, will wait for one to finish before doing any part of the following one).
If the requests are to the same remote, then Fulcro guarantees that the server will see the identical transaction. Atomic an in-order behaviors are then a function of your server code. Fulcro’s out of the picture.
If the mutations talk to multiple remotes (e.g. f
talks to remote 1, and g
remote 2) then each of those remotes has a separate network queue. The order in which the operations are sent to remotes is not well defined because it doesn’t matter. Distributed systems simply cannot guarantee anything about the ordering of those two.
The same thing holds true for a single mutation that talks to multiple remotes:
(defmutation f [params]
(remote1 [_] true)
(remote2 [_] true))
A (transact! this [(f)])
will send off requests to both remotes. Network delays, processing queues, etc. will then define the total ordering of the operations, and the ok
and error
actions of f
will trigger twice (once for each remote) in an undefined order.Two calls to transact!
will always be run and queued in the order they are run, but they will be separate transactions as far as the server is concerned. In the case of a single transact!
it is therefore possible for you to code an atomic processing of a sequence of mutations in your server code as long as they are run as a single unit. Communication between mutations on the server (e.g. seeing result of the first one in the second) is completely up to you.
(e.g. you could make middleware that starts a transaction on a thread-bound SQL connection, and commit/rollback at the end of the Pathom processing, and have all the Pathom mutations use the thread-bound connection. Intermediate db changes would be visible on that shared connection)
Thanks for that detailed explanation. That makes totally sense. Fulcro is awesome. It handles so much in a sane well designed way :)
Thanks. The most recent fun “remote” I made might be of interest on this thread as well. I made a WebWorkerRemote. It does just what it sounds like. It lets you hook a mutation up to an operation that runs in a WebWorker. So now, given a separate js file that has code I want to run on a thread, I can (transact! this [(some-bgop)])
and it works just like any other Fulcro mutation that has async (remote) operation…but is in fact just running in a thread in the browser. Parameters go to the web worker, and the web worker can respond with progress and final result, which of course merges according to the normal rules of mutations.
@tony.kay I wanted to read more about the WebWorkerRemote, but couldn’t find anything. Is this included in Fulcro? Or is there a snippet, which I could use as a starting point?