This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-11-20
Channels
- # announcements (1)
- # babashka (4)
- # beginners (8)
- # cider (4)
- # clj-kondo (10)
- # cljdoc (1)
- # cljsrn (7)
- # clojure (3)
- # clojure-europe (20)
- # clojure-france (3)
- # clojure-sg (2)
- # clojurescript (16)
- # clojureverse-ops (3)
- # community-development (5)
- # core-async (35)
- # cursive (1)
- # datahike (14)
- # datomic (7)
- # events (5)
- # fulcro (59)
- # graphql (11)
- # lsp (33)
- # meander (1)
- # off-topic (33)
- # polylith (23)
- # portal (33)
- # re-frame (1)
- # reagent (10)
- # reclojure (7)
- # reveal (14)
It's just data and functions.
Using RAD with XTDB, I want to call some of the queries without going through Pathom. Is there some way I can get an env
value to pass to my backend queries? I think I need to pass the app atom to to application/current-state
, but since the app atom is in CLJS, I'm not quite sure how to do that.
hey, I'm trying to grasp what you're trying to do. What type query do you want to call without going to Pathom - you mean xtdb query or fulcro component query (against client db)? What exact data you want from the env/current-state? Passing those big objects might be a bit overkill.
I have some server functions that need to query the XTDB DB. I have all my query functions following the format in https://github.com/fulcrologic/fulcro-rad-demo/blob/develop/src/xtdb/com/example/components/database_queries.clj and want call those to get the data I need.
I'd still go through Pathom with this - you could declare a https://blog.wsscode.com/pathom/v2/pathom/2.2.0/connect/resolvers.html#GlobalResolvers and call the xtdb from there with env being accessible
Why? It seems like an unnecessary level of abstraction added in the middle when all I want is some data from the DB.
yeah, but that's what pathom is there for - getting data from the DB 🙂 with all the context provided i.e. env that you need
I usually craft my queries in the REPL something like this in a rich comment when working on them:
(let [node (:main xtdb-nodes)
db (xt/db node)]
(xt/q db
...
alternatively, you could just define another endpoint in the ring middleware that you could serve outside of pathom, but that's kindof missing the point of this architecture
that's what I do as well in the REPL 😉 but assuming you want to call this query from the client, the idea is all queries go through /api endpoint served by pathom - so you have single point of entry
ah, ok - so you could just call xtdb directly like you do in the repl right?
unless you want env which is related to http request (that you're not making in this case)
But then I need a new query function for a thing that I already have a query function for 🙂 I may require it from the client, so I do expose it there, but I also require it outside the client, and I'd prefer not to re-write the query.
I think what I'm trying to get at is I prefer the query functions to be client agnostic.
ah, I get it now - you could just modify those query functions to take in xtdb node as an arg instead of the whole env
and you could reuse them in both contexts - you would need to only change how you call them from pathom resolvers (so you pass node instead of env)
Can you suggest how I can get the DB atom stored in the [xo/databases :production]
path in the env that Pathom passes in?
in the same way as you do in the REPL I'd say:
node (:main xtdb-nodes)
just replace :main / :production depending what you needI think I need the atom... or at least its value at the time of invocation. I think I've had REPL cases where the state of (:main xtdb-nodes)
differs from the current state the rest of the application is working with.
I think [xo/databases :main]
and (:main xtdb-nodes)
point to the same thing
https://github.com/fulcrologic/fulcro-rad-demo/blob/develop/src/xtdb/com/example/components/xtdb.clj#L7-L9
https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/start_databases.clj#L72-L77
Thanks for your input @UHA0AQZ2M. Much appreciated. Do you have any thoughts on changing the demo repo to structure the query functions in a slightly more client agnostic manner as I've described? I think I'm going to try to refactor my query namespace to work that way, and to intern the query maps separately from the functions actually doing the work. I could see about porting a similar change in a PR if you think it would offer users enough benefit.
no problem, thanks for digging in it and asking questions - my reasoning could be wrong so it's good to try it
as for queries in demo... that's just a demo to get started, so I'm not sure it's intended to serve all the use cases - it's relatively easy to modify it in whatever direction you need,. In my project I've already abandoned those queries and just call xtdb directly in resolvers (loosing db portability). I also wouldn't want to diverge from other dbs in the demo so such change would need to be applied to all of them
(some-> (get-in this-env [xo/databases :production]) deref)
=> #<XtdbDB {:xtdb.api/valid-time #inst "2021-11-20T15:52:35.516-00:00", :xtdb.api/tx-id 1601}>
(:main xtdb-nodes)
=> #<XtdbNode>
:thinking_face:yeah, so semantically the difference between XtdbNode and XtdbDB is xtdb.api/db
https://docs.xtdb.com/clients/1.19.0/clojure/#_db that takes node
and valid-time
so it's a node instantialized as point-in-time snapshot that you then query
I tried to follow that distinction between xtdb database
and node
in the fulcro-rad-xtdb code but it's not strongly typed so prone to mistakes - I'd be to happy to fix inconsistencies if you find them
however I'm not sure if that database atom should exist on the env - updating it might be flaky and lead to desync surprises. I copied that pattern from fulcro-rad-datomic but it might not make sense anymore in context of xtdb
maybe keeping node
and valid-time
in the env
would be enough and more robust - as long there's no significant performance loss from calling xtdb.api/db
multiple times vs keeping it in the atom
putting valid-time
and node
in the env
is enough to share the point-in-time view of the world across the http request/EQL query by calling (xtdb.api/db node valid-time)
each time you want to query data
there's no need to update any atom when mutating the db state which eliminates the risk of desynchronising it
escape-the-hatch beauty of this solution is that in single http request you can use the env
's valid-time
and mix it with other arbitrary timestamps (e.g. derived from params) - making use of xtdb's bitemporality
bothering with env
's database
atom makes it more confusing and does not really belong to plugin's responsibility - xtdb.api
https://docs.xtdb.com/clients/1.19.0/clojure/ around syncing nodes and establishing point-in-time state from valid-time
and tx-time
- there's no reason to mirror it in plugin's atom management
> not sure if that database atom should exist on the env I'm not following everything discussed here, but just wanted to chime in to point out that XT's db value aren't thread-safe (in case that's relevant!)
hey @U899JBRPF 👋 thanks for chiming in. I'm trying to understand the implications.
Let me sum up the dilemma:
• each /api
request gets wrapped up with env
variable that gets passed around through each pathom resolver (from where xtdb is called)
• https://github.com/roterski/fulcro-rad-xtdb/blob/b28fcb411bcda3b88589f2a4a4e460b7cbd9b376/src/main/roterski/fulcro/rad/database_adapters/xtdb/pathom_plugin.clj#L29-L30 both node
and database
are kept in atom in env
• that means database-atom
needs to be updated whenever we https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L148 to the db during that request
• this adds a risk of desynchronisation if a write comes from any other place than in save-form!
https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L127-L153
• this translates to weakly enforced constraint that all writes in the request must go through save-form!
which requires specific delta format and might not cover all the cases (mutation coming from anything other than a form)
• We want to preserve atomicity/consistency of db state in each request because we want to prevent the situation in which we would get different results for querying the same data at different points within single request
• With assumption that each request is processed serially thread-safety needs to be considered from perspective of what happens when other request modifies the db in the middle of processing the request
• https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb/wrap_xtdb_save.clj#L148 in env
does not eliminate this problem because it still refetches the db state that could be modified by other request and db value is not thread-safe (as noticed by @U899JBRPF above)
• what I'm considering is dropping database
atom from env
and keeping only node
, tx-time
and valid-time
there. Database would need to be initialised with (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time})
each time it's needed to query the data
• by using fixed tx-time
from env
throughout the request for fetching db we could expect the more or less the same db state as much as eventual consistency of xtdb allows for it
• relying on valid-time
only wouldn't give the same guarantee because data can be entered at arbitrary valid-time
points while tx-time
is fixed to the system time of the data insertion - so ideally we would use both tx-time
and valid-time
(in most cases they both would be equal to the same timestamp of the beginning of the request)
• To my understanding, maintaining database
atom in env
doesn't buy us anything in terms of thread safety as long as it doesn't use fixed tx-time
for refetching the db - it should be added there
• in the end, whether or not should we keep database
in env
comes down to convenience of not having to call (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time})
manually vs risk of desynchronisation when db gets updated outside of save-form!
@UHA0AQZ2M Great summary and far more detail that I was aware of. In particular:
> this adds a risk of desynchronisation if a write comes from any other place than in save-form! handler
Now I'm wondering if my backend-only operations that are writing to XTDB are going to introduce some desync problems as you have described.
> in the end, whether or not should we keep database in env comes down to convenience of not having to call (xtdb.api/db node {::xt/valid-time valid-time ::xt/tx-time tx-time}) manually vs risk of desynchronisation when db gets updated outside of save-form!
What about structuring things such that regardless of where the write operations are coming from, in the end it passes through a function that updates the database
in env
or some other central location? I think this ties to what I was talking about earlier regarding decoupling the DB operations from Pathom. At the lowest level, write operations to the DB should be possible from anywhere as long as a valid xtdb node is accessible. It should be the responsibility of the client - whether Pathom or anything else - to make sure to retrieve and pass in the xtdb node to complete its operation.
> What about structuring things such that regardless of where the write operations are coming from, in the end it passes through a function that updates the `database` in `env` or some other central location?
I think that's the idea behind already existing https://github.com/fulcrologic/fulcro-rad/blob/develop/src/main/com/fulcrologic/rad/form.cljc#L581-L583 mutation. The thing is using it is not strictly enforced, not visible enough and not nearly as convenient as just using xtdb.api
directly. It's just too easy to accidentally omit going through this central save-as-form
for it to be robust.
Nevertheless going through this central point should be a recommended pattern anyway because it could do more than that, but without database atom the consequences of omitting it are less significant
Could some of that code be further decoupled from Fulcro/Pathom conventions such that it would have no connection to forms or idents and still be useful in the scope of a Fulcro application? I understand that we're talking about a Fulcro plugin here 😛 but could it be useful to give it some independence at some level of the abstraction?
so it'd be more of some convenience wrapper around xtdb api? I imagine maintaining it and keeping up to date would be a hassle. As of now, plugin aims to be just a lean connector between fulcro-rad and xtdb. I'm not sure there's enough value added yet to justify abstracting it into separate entity :thinking_face:
Yep, that's about what I'm thinking. Something to sit between what exists in the current plugin and the xtdb API. I don't think that performing DB operations from the backend is a rare use case. Just think of any application that does lots of web crawling and scraping. It might help to clarify the model that the RAD XTDB model uses as a secondary benefit. Of course, I'm merely trying to think through all the pieces and this is particularly relevant to my project right now... I do not insist that this is the one true way :)
> don't think that performing DB operations from the backend is a rare use case.
yeah, I agree - but for it direct use of xtdb.api
(or any other convenience wrapper) is good enough and outside of the fulcro-rad-xtdb 's responsibility. The plugin should not require only one way of interacting with xtdb and should not break when xtdb is accessed outside of it
I can totally imagine that if the plugin evolves and offers more value on its own, then some of it could be extracted to a separate wrapper. It's just that, as of now, there's not enough value that could be extracted.
I understand your point. Thanks for engaging on this @UHA0AQZ2M. Appreciate you explaining some of the internals and considering my input.
Was there a Datascript RAD plugin mentioned sometimes? I recall there was some sort of datascript integration mentioned on Slack at one point?
I don't recall it but you can try scanning for in clojure-verse slack archive https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fclojurians-log.clojureverse.org%2Ffulcro+rad+datascript
anyway, I was in similar position looking for xtdb (crux back then) plugin few months ago and I didn't find one so I wrote one myself - it comes down to https://github.com/roterski/fulcro-rad-xtdb/blob/main/src/main/roterski/fulcro/rad/database_adapters/xtdb.clj#L16-L24: start-databases, pathom-plugin, generate-resolvers, wrap-save, wrap-delete
I'm thinking maybe if RAD exposed some kind of interface for plugins to implement, developing more plugins and ensuring their compliance could be more seamless - I'm not sure what would be the best mechanism for that ( https://clojuredocs.org/clojure.core/defprotocol ?)
Sounds great. Well-defined plugin points are always great. I use datahike in my projects for example.
So, I intentionally did not use a protocol for various reasons: 1. If we end up with client-side databases, protocols break dead code elimination 2. Protocols define “required” things and also place unnecssary contraints (method signatures) on the plugin. I want plugins to be able to define what they want and expand/contract that set on their own terms. 3. The reliance on save middleware, resolvers, and form diff concepts make it possible to be quite flexible in what/how you define a plugin, and how you later refine/expand it. 4. The existing plugins are pretty short, and can easily be copied/adapted to new things if you want to use the existing patterns. A datascript plugin would probably take an afternoon based on the current Datomic one. It could make sense to further refactor some of the useful utility functions out to RAD itself or something so that plugins could share them.
Yes on the utility functions. Currently the one I'm responsible for has a duplicates ns: https://github.com/fulcrologic/fulcro-rad-kvstore/blob/master/src/main/com/fulcrologic/rad/database_adapters/key_value/duplicates.cljc.