Fork me on GitHub
#fulcro
<
2021-11-20
>
cl_j02:11:34

Hi all, how to initialize UI state calculated from initial db state?

cl_j02:11:06

I think i could try the computed function when creating the element

👍 1
Jakub Holý (HolyJak)11:11:17

It's just data and functions.

sheluchin13:11:14

Using RAD with XTDB, I want to call some of the queries without going through Pathom. Is there some way I can get an env value to pass to my backend queries? I think I need to pass the app atom to to application/current-state, but since the app atom is in CLJS, I'm not quite sure how to do that.

Piotr Roterski14:11:55

hey, I'm trying to grasp what you're trying to do. What type query do you want to call without going to Pathom - you mean xtdb query or fulcro component query (against client db)? What exact data you want from the env/current-state? Passing those big objects might be a bit overkill.

sheluchin15:11:15

I have some server functions that need to query the XTDB DB. I have all my query functions following the format in https://github.com/fulcrologic/fulcro-rad-demo/blob/develop/src/xtdb/com/example/components/database_queries.clj and want call those to get the data I need.

Piotr Roterski15:11:43

I'd still go through Pathom with this - you could declare a https://blog.wsscode.com/pathom/v2/pathom/2.2.0/connect/resolvers.html#GlobalResolvers and call the xtdb from there with env being accessible

sheluchin15:11:58

Why? It seems like an unnecessary level of abstraction added in the middle when all I want is some data from the DB.

sheluchin15:11:10

(not saying you're wrong, just want to understand your reasoning 🙂 )

Piotr Roterski15:11:49

yeah, but that's what pathom is there for - getting data from the DB 🙂 with all the context provided i.e. env that you need

sheluchin15:11:29

I usually craft my queries in the REPL something like this in a rich comment when working on them:

(let [node (:main xtdb-nodes)
        db (xt/db node)]
  (xt/q db
   ...

Piotr Roterski15:11:30

alternatively, you could just define another endpoint in the ring middleware that you could serve outside of pathom, but that's kindof missing the point of this architecture

Piotr Roterski15:11:10

that's what I do as well in the REPL 😉 but assuming you want to call this query from the client, the idea is all queries go through /api endpoint served by pathom - so you have single point of entry

sheluchin15:11:25

I do not want to call this query from the client.

sheluchin15:11:37

Just a backend function working with some scraped data.

Piotr Roterski15:11:10

ah, ok - so you could just call xtdb directly like you do in the repl right?

Piotr Roterski15:11:13

unless you want env which is related to http request (that you're not making in this case)

sheluchin15:11:55

But then I need a new query function for a thing that I already have a query function for 🙂 I may require it from the client, so I do expose it there, but I also require it outside the client, and I'd prefer not to re-write the query.

sheluchin15:11:46

I think what I'm trying to get at is I prefer the query functions to be client agnostic.

Piotr Roterski15:11:07

ah, I get it now - you could just modify those query functions to take in xtdb node as an arg instead of the whole env

Piotr Roterski15:11:23

and you could reuse them in both contexts - you would need to only change how you call them from pathom resolvers (so you pass node instead of env)

sheluchin15:11:40

Can you suggest how I can get the DB atom stored in the [xo/databases :production] path in the env that Pathom passes in?

Piotr Roterski15:11:15

in the same way as you do in the REPL I'd say:

node (:main xtdb-nodes)
just replace :main / :production depending what you need

sheluchin15:11:52

I think I need the atom... or at least its value at the time of invocation. I think I've had REPL cases where the state of (:main xtdb-nodes) differs from the current state the rest of the application is working with.

sheluchin15:11:53

On second thought that may have been caused by some inline def debugging hackery.

sheluchin15:11:55

Thanks for your input @UHA0AQZ2M. Much appreciated. Do you have any thoughts on changing the demo repo to structure the query functions in a slightly more client agnostic manner as I've described? I think I'm going to try to refactor my query namespace to work that way, and to intern the query maps separately from the functions actually doing the work. I could see about porting a similar change in a PR if you think it would offer users enough benefit.

Piotr Roterski15:11:23

no problem, thanks for digging in it and asking questions - my reasoning could be wrong so it's good to try it

Piotr Roterski15:11:43

as for queries in demo... that's just a demo to get started, so I'm not sure it's intended to serve all the use cases - it's relatively easy to modify it in whatever direction you need,. In my project I've already abandoned those queries and just call xtdb directly in resolvers (loosing db portability). I also wouldn't want to diverge from other dbs in the demo so such change would need to be applied to all of them

sheluchin15:11:18

True. All good points. Thanks again!

sheluchin16:11:39

(some-> (get-in this-env [xo/databases :production]) deref)
=> #<XtdbDB {:xtdb.api/valid-time #inst "2021-11-20T15:52:35.516-00:00", :xtdb.api/tx-id 1601}>

(:main xtdb-nodes)
=> #<XtdbNode>
:thinking_face:

sheluchin16:11:09

Ah, (xt/db (:main xtdb-nodes)) does it.

💯 1
Piotr Roterski21:11:33

yeah, so semantically the difference between XtdbNode and XtdbDB is xtdb.api/db https://docs.xtdb.com/clients/1.19.0/clojure/#_db that takes node and valid-time so it's a node instantialized as point-in-time snapshot that you then query

Piotr Roterski21:11:44

I tried to follow that distinction between xtdb database and node in the fulcro-rad-xtdb code but it's not strongly typed so prone to mistakes - I'd be to happy to fix inconsistencies if you find them

Piotr Roterski21:11:55

however I'm not sure if that database atom should exist on the env - updating it might be flaky and lead to desync surprises. I copied that pattern from fulcro-rad-datomic but it might not make sense anymore in context of xtdb

Piotr Roterski21:11:14

maybe keeping node and valid-time in the env would be enough and more robust - as long there's no significant performance loss from calling xtdb.api/db multiple times vs keeping it in the atom

Piotr Roterski21:11:37

putting valid-time and node in the env is enough to share the point-in-time view of the world across the http request/EQL query by calling (xtdb.api/db node valid-time) each time you want to query data

Piotr Roterski21:11:00

there's no need to update any atom when mutating the db state which eliminates the risk of desynchronising it

Piotr Roterski21:11:47

escape-the-hatch beauty of this solution is that in single http request you can use the env's valid-time and mix it with other arbitrary timestamps (e.g. derived from params) - making use of xtdb's bitemporality

Piotr Roterski21:11:20

bothering with env's database atom makes it more confusing and does not really belong to plugin's responsibility - xtdb.api https://docs.xtdb.com/clients/1.19.0/clojure/ around syncing nodes and establishing point-in-time state from valid-time and tx-time - there's no reason to mirror it in plugin's atom management

refset12:11:26

> not sure if that database atom should exist on the env I'm not following everything discussed here, but just wanted to chime in to point out that XT's db value aren't thread-safe (in case that's relevant!)

Piotr Roterski18:11:00

hey @U899JBRPF 👋 thanks for chiming in. I'm trying to understand the implications. Let me sum up the dilemma: • each /api request gets wrapped up with env variable that gets passed around through each pathom resolver (from where xtdb is called) • https://github.com/roterski/fulcro-rad-xtdb/blob/b28fcb411bcda3b88589f2a4a4e460b7cbd9b376/src/main/roterski/fulcro/rad/database_adapters/xtdb/pathom_plugin.clj#L29-L30 both node and database are kept in atom in env • that means database-atom needs to be updated whenever we https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L148 to the db during that request • this adds a risk of desynchronisation if a write comes from any other place than in save-form!https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L127-L153 • this translates to weakly enforced constraint that all writes in the request must go through save-form! which requires specific delta format and might not cover all the cases (mutation coming from anything other than a form) • We want to preserve atomicity/consistency of db state in each request because we want to prevent the situation in which we would get different results for querying the same data at different points within single request • With assumption that each request is processed serially thread-safety needs to be considered from perspective of what happens when other request modifies the db in the middle of processing the request • https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L148 in env does not eliminate this problem because it still refetches the db state that could be modified by other request and db value is not thread-safe (as noticed by @U899JBRPF above) • what I'm considering is dropping database atom from env and keeping only node , tx-time and valid-time there. Database would need to be initialised with (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time}) each time it's needed to query the data • by using fixed tx-time from env throughout the request for fetching db we could expect the more or less the same db state as much as eventual consistency of xtdb allows for it • relying on valid-time only wouldn't give the same guarantee because data can be entered at arbitrary valid-time points while tx-time is fixed to the system time of the data insertion - so ideally we would use both tx-time and valid-time (in most cases they both would be equal to the same timestamp of the beginning of the request) • To my understanding, maintaining database atom in env doesn't buy us anything in terms of thread safety as long as it doesn't use fixed tx-time for refetching the db - it should be added there • in the end, whether or not should we keep database in env comes down to convenience of not having to call (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time}) manually vs risk of desynchronisation when db gets updated outside of save-form!

sheluchin18:11:07

@UHA0AQZ2M Great summary and far more detail that I was aware of. In particular: > this adds a risk of desynchronisation if a write comes from any other place than in save-form! handler Now I'm wondering if my backend-only operations that are writing to XTDB are going to introduce some desync problems as you have described. > in the end, whether or not should we keep database in env  comes down to convenience of not having to call (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time})  manually vs risk of desynchronisation when db gets updated outside of save-form! What about structuring things such that regardless of where the write operations are coming from, in the end it passes through a function that updates the database in env or some other central location? I think this ties to what I was talking about earlier regarding decoupling the DB operations from Pathom. At the lowest level, write operations to the DB should be possible from anywhere as long as a valid xtdb node is accessible. It should be the responsibility of the client - whether Pathom or anything else - to make sure to retrieve and pass in the xtdb node to complete its operation.

Piotr Roterski18:11:43

> What about structuring things such that regardless of where the write operations are coming from, in the end it passes through a function that updates the `database` in `env` or some other central location? I think that's the idea behind already existing https://github.com/fulcrologic/fulcro-rad/blob/develop/src/main/com/fulcrologic/rad/form.cljc#L581-L583 mutation. The thing is using it is not strictly enforced, not visible enough and not nearly as convenient as just using xtdb.api directly. It's just too easy to accidentally omit going through this central save-as-form for it to be robust.

Piotr Roterski18:11:03

Nevertheless going through this central point should be a recommended pattern anyway because it could do more than that, but without database atom the consequences of omitting it are less significant

sheluchin19:11:48

Could some of that code be further decoupled from Fulcro/Pathom conventions such that it would have no connection to forms or idents and still be useful in the scope of a Fulcro application? I understand that we're talking about a Fulcro plugin here 😛 but could it be useful to give it some independence at some level of the abstraction?

Piotr Roterski19:11:01

so it'd be more of some convenience wrapper around xtdb api? I imagine maintaining it and keeping up to date would be a hassle. As of now, plugin aims to be just a lean connector between fulcro-rad and xtdb. I'm not sure there's enough value added yet to justify abstracting it into separate entity :thinking_face:

sheluchin19:11:53

Yep, that's about what I'm thinking. Something to sit between what exists in the current plugin and the xtdb API. I don't think that performing DB operations from the backend is a rare use case. Just think of any application that does lots of web crawling and scraping. It might help to clarify the model that the RAD XTDB model uses as a secondary benefit. Of course, I'm merely trying to think through all the pieces and this is particularly relevant to my project right now... I do not insist that this is the one true way :)

Piotr Roterski19:11:03

> don't think that performing DB operations from the backend is a rare use case. yeah, I agree - but for it direct use of xtdb.api (or any other convenience wrapper) is good enough and outside of the fulcro-rad-xtdb 's responsibility. The plugin should not require only one way of interacting with xtdb and should not break when xtdb is accessed outside of it

Piotr Roterski19:11:27

I can totally imagine that if the plugin evolves and offers more value on its own, then some of it could be extracted to a separate wrapper. It's just that, as of now, there's not enough value that could be extracted.

sheluchin19:11:54

I understand your point. Thanks for engaging on this @UHA0AQZ2M. Appreciate you explaining some of the internals and considering my input.

🙏 1
roklenarcic19:11:47

Was there a Datascript RAD plugin mentioned sometimes? I recall there was some sort of datascript integration mentioned on Slack at one point?

Piotr Roterski19:11:39

anyway, I was in similar position looking for xtdb (crux back then) plugin few months ago and I didn't find one so I wrote one myself - it comes down to https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb.clj#L16-L24: start-databases, pathom-plugin, generate-resolvers, wrap-save, wrap-delete

Piotr Roterski20:11:09

I'm thinking maybe if RAD exposed some kind of interface for plugins to implement, developing more plugins and ensuring their compliance could be more seamless - I'm not sure what would be the best mechanism for that ( https://clojuredocs.org/clojure.core/defprotocol ?)

Björn Ebbinghaus10:11:00

Sounds great. Well-defined plugin points are always great. I use datahike in my projects for example.

tony.kay17:11:25

So, I intentionally did not use a protocol for various reasons: 1. If we end up with client-side databases, protocols break dead code elimination 2. Protocols define “required” things and also place unnecssary contraints (method signatures) on the plugin. I want plugins to be able to define what they want and expand/contract that set on their own terms. 3. The reliance on save middleware, resolvers, and form diff concepts make it possible to be quite flexible in what/how you define a plugin, and how you later refine/expand it. 4. The existing plugins are pretty short, and can easily be copied/adapted to new things if you want to use the existing patterns. A datascript plugin would probably take an afternoon based on the current Datomic one. It could make sense to further refactor some of the useful utility functions out to RAD itself or something so that plugins could share them.

👍 1
tony.kay04:11:14

Yeah, as long as they don’t depend on any particular plugin, I could just put them in RAD itself.