I've thought that it is "up to you" to make sure that you don't have tx functions that call code that changes with time

^ That's correct, XT doesn't attempt to analyse a transaction function for purity, and instead relies on the user to make this determination before using it.

Perhaps worth mentioning in https://docs.xtdb.com/language-reference/1.21.0/datalog-transactions/#transaction-functions ?

I was actually a bit surprised that the function cannot for example generate new entities with random uuids

I kinda assumed that they are more direct counterparts to stored procedures in SQL dbs. While of course they are run on the nodes, I hadn't realized that they might be run many times, on multiple nodes, and they need to return the same result

> Perhaps worth mentioning in [the docs] Agreed, that is definitely a good idea - I'll make a note to the board so we/I don't forget. And yep, generating a UUID inside a function would definitely count as an impure side-effect that undermines the overall consistency.

Maybe more concretely, if we delete an instance of type A, and want to delete all links too, how can we make sure that nobody added more links between querying for existing links, and issuing the delete transaction?

Hm, perhaps this is the wrong way to look into it. In my code handling type A, I have no idea what all other places might be linking to A. I cannot do the deletion reliably

Did you try transaction functions?

How would the code handling the deletion know what all functions to "call"?

You would write your own transaction function from which you would return xt/put calls that would delete your docs.

Yes, but how would the namespace that handles creating and removing type A know what all transaction functions to call, to delete all the other documents that refer to type A?

It's starting to look like that cascades is a fools errand, and every place that cares about the links has to check if the linked documents exist, not just rely on the link field

I didn't even think that you would want to have more than one transaction function for deletions. I thought that you would walk the DB graph inside of this single function. Which essentially would mean "has to check if the linked documents exist".

Even that would require knowing what the name of the link field is, for doing the query

Sure, it could be a convention, but that is very different level of checks compared to a DB enforced foreign key

Interesting :thinking_face: I think despite my reply in the other thread about the importance of keeping transaction functions pure, this may be a valid exception to that rule given XT's current APIs. You should be able to achieve what you want by enumerating the output of attribute-stats (which unfortunately necessitates referring to the node somehow!) - this code may help: https://gist.github.com/refset/25d8ae03c69993831c9c23cf7e12a537

Given how transaction functions are executed during indexing, this should be completely safe. I wouldn't recommend doing it for general querying though, as the output from attribute-stats can change even with an otherwise stable db basis (because indexing happens concurrently)

I would say that since the db is an input, it is still pure. With identical db, the resulting output of operations should not change

Another workaround could be to enumerate your list of system-wide attributes in an additional entity (i.e. stored explicitly in userspace)

What is $ in that gist?

Ah, found it in https://docs.xtdb.com/language-reference/1.21.0/datalog-queries/#custom-functions

It's the implicit db basis for that query. In Datomic/DataScript I think it always has to be passed in via :in but I might be wrong

And yeah, attribute-stats would work. A bit heavy, but works

Another workaround is to hard code the list of all known attributes directly into the transaction function, and then just create new functions each time your schema evolves

That certainly works, but makes a bit too much spaghetti. Trying to find ways where the namespaces don't need to know about internals in other namespaces

Hmh, althought that might be a bit overreaching

So for comparison, sometimes in SQL you would refer to another table without a foreign key, to avoid the cascades

fwiw, I would not create a tx function that indiscrimnately walks the db and deletes stuff... rather would have something per entity type like :customer/delete which would know statically which subentities it needs to cascade to

but if you really have such free form data that you don't know which attributes in which entities are links, then you may need to walk

Yeah, some kind of convention where it's easy to test that the cascading delete really happens, and is hard to trigger accidentally when just wanting a "normal" link

in a deeply nested structure that is shredded to the database as multiple documents, there might be some documents that are links that should not be deleted and some that should

Exactly

{:order/id 1 :order/items [{:order-item/id 1} ...] :order/customer {:customer/id 1}]

same as patching, no perfect completely generic solution exists...

Yeah, although in this case the question was particularly about many-to-many link documents, that is, the document doesn't serve any other purpose except to make the many-to-many relationship indexed. Then it's pretty clear that if one of the linked documents is deleted, all links should be too. But indeed, how to recognize link documents from the rest…

Something like :customer/delete-me would work as the field name 😉 Though have to think a bit more about the naming.

yeah, consistent naming of the link attributes would help

Such lovely queries

'{:find [(pull order [*])]
  :where [[link :do-not-delete-this-customer customer-id]
          [link :delete-this-order order]]
  :in [customer-id]}

Well, not even that, since you would not delete an "order" if only one of many "customers" would be deleted, just the link. Names are hard

:delete-me-not-this-customer

if you do all entities that need cascade delete with backlinks and have a single name for the link, then when you are deleting customer c you can just query for [to-delete :customer/link c] to find what needs to be deleted

Yup

Could be global name too, something like :our-app/link, and use generic cascading delete functions for all kinds of entities

Well, needs at least two of them to make the many-to-many links work.

you mean a link doc is {:our-app/link1 doc1 :our-app/link2 doc2} ?

Yeah

why not {:our-app/link [doc1 doc2]}?

would be found when either is deleted

I guess, if you can generate the docs in the right order 🙂

Well, same problem with two keys, which is which

I've been using the strategy @U11SJ6Q0K suggests--having a tx function per entity type that implements its own deletion policy for related records

>> why not {:our-app/link [doc1 doc2]}? >> I guess, if you can generate the docs in the right order 🙂 XT is also happy with {:our-app/link #{doc1 doc2}} (p.s. great discussion!)

> XT is also happy with {:our-app/link #{doc1 doc2}} But how would you query "all links to/from doc1"?

[link :our-app/link doc1]

or if you mean "all docs linked to/from doc1", then:

[link :our-app/link doc1]
[link :our-app/link other-doc]
[(!= doc1 other-doc)]

Oh, I had no idea that the normal triple clause will also match values inside sets. Is it just sets, or all kinds of collections? But it isn't indexed, right?

Then again, reading https://docs.xtdb.com/language-reference/1.21.0/datalog-queries/#_maps_and_vectors_in_data again I notice that even with vectors I could query without caring about the order

Because if the vectors actually would be indexed, I could just put the many to many relationship directly into a field of doc1

top-level sets and vectors get decomposed into triples, fully indexed 🙂 https://docs.xtdb.com/language-reference/datalog-transactions/#indexing

Ok, that's explicit. We had read > The same thing should apply for maps so instead on https://docs.xtdb.com/language-reference/datalog-queries/#_maps_and_vectors_in_data which sounded like maps and vectors work the same: they need to be manually flattened

Ah, yes that could/should definitely be clearer :thinking_face: As an aside, taking that idea to the extreme looks like... https://gist.github.com/refset/ead5cc5ab36c0726b85c1713f0455509

But how to name them with nested vectors 😉 Well, if you assume that maps don't have keys like :0 then you can just put numbers. And of course better hope that the maps don't have string keys 😬

the http-server module has a GUI component to it: https://docs.xtdb.com/extensions/http/ I don't know if the docs do a super good job of advertising it but I've been using it in my own projects--if you navigate in a browser to the :port specified in :xtdb.http-server/server you should see it

We lost momentum to get the http server GUI to a 'finished' state, but it works for some things 🙂 There's also @@tatut's inspector https://docs.xtdb.com/ecosystem/ (which is newer, and arguably a much slicker architecture than the http server's cljs-heavy re-frame app)

FWIW We are using the console atm but it feels unfinished indeed. We are planning to move to tatut's work soon-ish

I'm happy to take any suggestion on how to improve the XTDB inspector... so far it has been mostly me scratching my own itch

Hi @U11SJ6Q0K if I want to bring inspector into my project using deps what would the package name be?:

<<what.goes.here?>> {:git/url ""
                             :git/sha "c3f80a545ab095184a13fc2445a96459de56e26f"}

tatut/xtdb-inspector

for example

Hi @U11SJ6Q0K - I was able to bring in tatut/xtdb-inspector as a dependency into my own project by creating a ring adapter that checks the URL and either dispatches to my default handler or to xtdb-inspector.core/inspector-handler. All “static” URLs work fine, but I’m having trouble getting the __ripley-live?... URLs to work properly. The WS connection is closed every time with a 404. It doesn’t look like the ripley.live.context/connection-handler is even being used. Could this have something to do with me using the jetty server in my project, whereas xtdb-inspector may rely on http-kit for the WS connection to work properly?

hi, sorry for the late answer, just now back at a machine… I haven’t tried with jetty, but you could raise an issue in github describing the setup