Fork me on GitHub
#fulcro
<
2022-12-06
>
Tyler Nisonoff00:12:33

@tony.kay I noticed that I was having really bad performance with attributes using RAD’s automatic resolvers that return large maps I traced the issue to:

secure-resolver (fn [env input]
                   (->>
                       (resolver env input)
                       (auth/redact env)))
where auth/redact is doing:
(defn redact
  "Creates a post-processing plugin that redacts attributes that are marked as non-readable"
  [{attr-map ::attr/key->attribute
    :as      env} query-result]
  (p/transduce-maps (map (fn [[k v]]
                           (let [a (get attr-map k)]
                             (if (readable? env a)
                               [k v]
                               [k ::REDACTED]))))
    query-result))
I believe the problem is that pathom’s transduce-maps will walk the entire returned data structure
(defn transduce-maps
  "Walk the structure and transduce every map with xform."
  [xform input]
  (walk/prewalk
    (fn elide-items-walk [x]
      (if (native-map? x)
        (with-meta (into {} xform x) (meta x))
        x))
    input))
I believe transduce-maps should be checking for ::p/final true in the metadata before prewalking, but it currently does not, so using the pathom ::p/final trick doesnt help us here. However, I’m not sure if thats true for all uses of transduce-maps in pathom (and I suspect not). However, We could write a custom transduce-maps that does in the RAD code if you’re open to that. Let me know if you have any thoughts on a generic solution here, or if one should just not use RAD for large attributes

Tyler Nisonoff01:12:06

Pathoms elide-not-found , which is included in the RAD parser as a plugin:

(p/post-process-parser-plugin p/elide-not-found)
exhibits the same behavior, but will only effect large items returned in the final output

tony.kay14:12:36

I'm happy to take performance improvements. The authorization redact isn't even desirable in my opinion. Coding that at a resolver level that has security wrappers is a better way to go, but it was a quick and dirty demonstration of how you could do things. It had not occurred to me that it would be a performance problem, although it should have, because I've had similar issues with the merge routines. Those leverage the query, and that allows me to stop traversing early when there's a large value that doesn't need walked.

tony.kay14:12:28

Leveraging the query is the best solution, because when there's a prop in the query you can just stop, and when there's a join, you can pull the subquery and continue. I'm not sure that final would be an answer, because a resolver can technically return nested structure; however, if you're talking about the auto generated resolvers, then having an algorithm that only walks a single level at the resolver would be the best solution, and eliminate the recursive walk all together

Tyler Nisonoff18:12:02

I’d be fine with removing the redact as the default as well. as for an alternative fix, I was thinking of not traversing values with ::p/final , as that seems to be how pathom generally tackles this problem, but open to just a single-level traversal as well

Tyler Nisonoff18:12:22

should i start with a PR to simply remove the redact call in auto-resolvers?

tony.kay18:12:39

Breaking change. Need a new option to disable it instead.

tony.kay18:12:13

the final idea doesn’t work…because a resolver can return nested data that MIGHT need to be affected

tony.kay18:12:04

The whole idea of final is to be able to return any arbitrarily complex value of any depth without further processing…but post-processing must apply to that value in this case

tony.kay18:12:21

otherwise you’re overloading the meaning of final in a way that limits flexibility

tony.kay18:12:47

(e.g. if I send a password back 3-levels deep in a final result, I’d want the password redacted still)

Tyler Nisonoff18:12:59

ahh yeah fair point

Quentin Le Guennec09:12:41

Are mutations within a transaction guaranteed to be ran synchronously? (I think yes, but I need confirmation)

tony.kay14:12:03

No way to do that

tony.kay14:12:04

You can get local synchrony if you want, but mutations that are full stack can't. If you use a single remote, then order is guaranteed.

tony.kay14:12:49

But there is no way to do remote ops synchronously in js browser

tony.kay14:12:44

And optimistic actions also guarantee order. Independent of the number or type of remotes.

Joe R. Smith00:12:28

I think the question might be: will mutations run in the order they are defined in a transaction?

tony.kay02:12:09

For the optimistic parts, yes. For any remote parts: they will be submitted in order, but if more than one remote is involved there's no way for it to guarantee the return order, or even the processing order for that matter.

Hendrik06:12:20

What about pessimistic transactions? If I understood it right, than this will run and return in order. The next transaction is only run after the last transaction finished. So if you are concerned about the return order then you would have to run pessimistic transactions, wouldn’t you?

Quentin Le Guennec08:12:28

Yes I actually wanted to know if they’re run in order. So any mutation involving mutations on the client database will be run in order, right? Things like merge/merge-component!, dr/target-ready , uism/begin!, uism/trigger! ?

tony.kay14:12:42

correct. There is a guarantee that (the optimistic part, for local client database) mutations are run in the order they are submitted.

tony.kay14:12:52

They are also submitted to remotes in the order transacted…but Fulcro cannot make guarantees about the arrival order of remote mutations to multiple remotes, or the completion order.

Hendrik14:12:52

Do you refer to the case, where one mutation has multiple remotes?

Quentin Le Guennec08:12:05

Ok. Coming from re-frame, this is actually very enjoyable 😁 thanks for your answers

tony.kay15:12:35

@U023LKF2PQV A transaction in Fulcro is a vector. Vectors imply a required order. So, a single call to transact! has the semantic of an in-order sequence of operations:

[(f) (g)]
means that you’d like f to run before g. Fulcro is in control of two things; The client database, and the network queues. So, the best it can do is run the local (optimistic) operations against the client db (in order) and submit the remote behaviors in that same order (see also pessimistic transactions, which, if both operations have remote operations, will wait for one to finish before doing any part of the following one). If the requests are to the same remote, then Fulcro guarantees that the server will see the identical transaction. Atomic an in-order behaviors are then a function of your server code. Fulcro’s out of the picture. If the mutations talk to multiple remotes (e.g. f talks to remote 1, and g remote 2) then each of those remotes has a separate network queue. The order in which the operations are sent to remotes is not well defined because it doesn’t matter. Distributed systems simply cannot guarantee anything about the ordering of those two. The same thing holds true for a single mutation that talks to multiple remotes:
(defmutation f [params]
  (remote1 [_] true)
  (remote2 [_] true))
A (transact! this [(f)]) will send off requests to both remotes. Network delays, processing queues, etc. will then define the total ordering of the operations, and the ok and error actions of f will trigger twice (once for each remote) in an undefined order.

tony.kay15:12:46

Two calls to transact! will always be run and queued in the order they are run, but they will be separate transactions as far as the server is concerned. In the case of a single transact! it is therefore possible for you to code an atomic processing of a sequence of mutations in your server code as long as they are run as a single unit. Communication between mutations on the server (e.g. seeing result of the first one in the second) is completely up to you.

tony.kay15:12:59

(e.g. you could make middleware that starts a transaction on a thread-bound SQL connection, and commit/rollback at the end of the Pathom processing, and have all the Pathom mutations use the thread-bound connection. Intermediate db changes would be visible on that shared connection)

Hendrik15:12:56

Thanks for that detailed explanation. That makes totally sense. Fulcro is awesome. It handles so much in a sane well designed way :)

tony.kay15:12:40

Thanks. The most recent fun “remote” I made might be of interest on this thread as well. I made a WebWorkerRemote. It does just what it sounds like. It lets you hook a mutation up to an operation that runs in a WebWorker. So now, given a separate js file that has code I want to run on a thread, I can (transact! this [(some-bgop)]) and it works just like any other Fulcro mutation that has async (remote) operation…but is in fact just running in a thread in the browser. Parameters go to the web worker, and the web worker can respond with progress and final result, which of course merges according to the normal rules of mutations.

Hendrik05:12:05

That sounds like a super useful abstraction to utilise web workers in your web app 🙂

Hendrik18:12:33

@tony.kay I wanted to read more about the WebWorkerRemote, but couldn’t find anything. Is this included in Fulcro? Or is there a snippet, which I could use as a starting point?

tony.kay17:12:24

it’s something I did for a client. Nothing to it really. Write a js web worker. Write a fulcro remote that sends/receives messages to/from it.

tony.kay17:12:36

A lightweight web worker in vanilla js can just serialize JSON, and the remote you write just translates that to/from EDN.