Fork me on GitHub
#asami
<
2022-05-25
>
folcon15:05:42

I'm trying to mimic :db.fn/call behaviour A bunch of my existing functions generate [:db.fn/call] datoms and I've noticed there doesn't seem to be db-with. So, should I be trying to put a processor before the transact and then: 1) check each datom in a transaction for a :db.fn/call 2) if it exists split the transaction coll on that point 3) collect the datoms upto that point 4) call transact on the collected datoms 5) then call the :db.fn/call on the resulting db, dropping that datom from the transaction 6) Then continue processing the transaction, repeating 4-5 every time a new :db.fn/call comes up? Not sure how to rollback if 5 throws an error either, maybe save the db before calling the function and then use that if there's an exception? Is there a better way that I'm missing? Sorry, still really new to asami 😃

folcon16:05:00

Ok digging into it further it's discussed here: • https://github.com/threatgrid/asami/discussions/158https://github.com/threatgrid/asami/issues/190https://github.com/threatgrid/asami/issues/98: Not sure if this approach has memory implications, do I need to explicitly call delete-database for example, or will it just get GC'ed?

quoll17:05:49

I’m guessing that you're trying to do this from outside Asami, and just using it through the public APIs. Is that right?

folcon17:05:48

Well I was hoping to do it from the inside, but I'm not clear where to start from there, so yes, I'm looking at the public api's first 😃... If this is already on your roadmap or there's a really simple way to support it, I'm all ears!

quoll17:05:51

If so, db-with is going to be difficult to do without

quoll17:05:29

(Sorry, slow typing. I’m on my phone in a Dr office right now, so I'll go quiet soon)

quoll17:05:00

I have db-with ready for testing, but not in yet

quoll17:05:22

Hopefully very soon?

folcon17:05:11

That's fine 😃... Happy to discuss further, when it's more convenient for you. I'm basically looking to use something datascript like in a game context, so asami is really interesting from a perf perspective. Happy to test stuff if that's useful!

quoll17:05:10

Are you staying with an in-memory db, or using durable storage (ie disk)?

quoll17:05:33

If it's in memory, then it should be easy to work with

folcon17:05:48

I'm sticking to in-memory for now, at some point in the future, durable will become useful, ie save files, but for now, in-memory is perfectly fine 😃

quoll17:05:07

OK. Will definitely follow up

1
quoll17:05:49

Dr just left for a minute… A Database is just a thin wrapper around a Graph object, that makes it look like a Datomic database. So you can access it the graph directly if you want. (Look in asami.graph for the protocol. It's implemented in asami.index). This is just like any other immutable structure, so to wind back you can just use the previous version.

quoll17:05:43

All the functions in asami.core get the latest Database from a Connection, and then the Graph from the Database, then everything happens on the graph

folcon18:05:49

Ok, I'll dig around in that and see what I can figure out, just a quick question, is there anything about reverse relations? I remember seeing in one of your talks that you had bidirectional relations, but when I searched the repo I couldn't see any mention of it. Figured it out, I was putting my entities in a vector so I had to first walk that relation >_<...

folcon01:05:57

Would you be looking to do something similar to what datascript does here? Or a different approach? • Repeated recur inside the transact when a :db.fn/call is present? https://github.com/tonsky/datascript/blob/55af41d6d7d18f12358957a056a6cd1bb484fee7/src/datascript/db.cljc#L1429

folcon13:05:04

Ok, having looked at the source, what's update-fn used for? It seems to be in the correct place to do what I want, but it's working at the transaction level instead of at the datom level. So it seems like I can broadly get the behaviour I want by splitting tx's into :db.fn/call's and non-db.fn/calls and then just calling transact multiple times. Not sure what the performance of that looks like, but it's a good test point.

quoll14:05:58

Hi, I’m just looking at this now. I’m unaware of Datascript’s operations here. I’m expecting that it’s the same as Datomic’s transactions functions? Any idea why it’s not used the same way?

quoll14:05:31

update-fn does a few things. It allows data to be passed in as a lazy seq, since it could be gigabytes. It also lets me return all the actual insertions, deletions, and generated ids that occurred during the transaction

quoll14:05:06

Once upon a time it was a lot simpler, but to cover each of the various requirements it got into this horrible callback thing

quoll15:05:43

Oh, and also, transactions can be done as either: • a stream of add/retract statements • a function that accepts a graph, and returns a new graph transact-update is asked to do both. The transact-data function takes a function that generates add/retracts or a pair of add/retract seqs and wraps them into the update-fn function that transact-update receives

quoll15:05:21

Also… I see that asami.durable.store/transact-update* needs a try/catch to do a rollback. Right now, nothing (should) throw an exception, but if transaction functions are happening then this will need to happen

folcon15:05:41

When you say: > Any idea why it’s not used the same way? What are you referring to? I was under the assumption that datascript correctly modelled datomic's behaviour when it came to :db.fn/call's (I've not used datomic itself in a while)

quoll15:05:39

After creating functions, they get called by their own name, mapped as a keyword

folcon15:05:48

Oh yes, that bit, I think the current behaviour was deemed sufficient? To be honest I wasn't sure if asami was going to go down the route of having transaction functions.

folcon15:05:07

They have their benefits

quoll15:05:53

I planned to, but hadn’t got to it. I mean, I plan to do lots of things! 🙂 Things have been a bit slow in recent months, because of my new job, but I’m hoping to start improving my pace again

quoll15:05:47

I’m thinking that a rollback on a transaction is the best approach, though there IS a new operation coming that could be used instead

quoll15:05:00

I’ve written (and not fully tested) a “graph wrapper” graph. It wraps an existing graph, plus a graph full of new assertions, and a graph or retractions. Querying it gives a concat of the wrapped graph with the assertion graph, filtered by the retraction graph. This is the basis for 2 things: • db-with • fast transactions when executed on storage

quoll15:05:16

I’m thinking that you might want me to get this committed asap

folcon15:05:36

Oh, well if you're already doing that, I would love to kick the tires 😃

quoll15:05:38

(incidentally, the fast transaction approach will launch a background thread to merge them, transparently switch over to the merged version, then inform subsequent wrapping graphs that it’s their turn to merge)

quoll15:05:37

Mind if I DM to coordinate with you?

folcon15:05:21

No problem 😃

folcon15:05:22

One question that came up when I was testing btw:

(let [things
        [{:name :wheat-farm
          :kind :farm
          :fields [{:kind :field :crop :empty}]}]]
    (-> @(d/transact conn {:tx-data things})
      :tx-data))
=>
(#datom[:a/node-58405 :name :wheat-farm 1 true]
 #datom[:a/node-58405 :kind :farm 1 true]
 #datom[:a/node-58407 :kind :field 1 true]
 #datom[:a/node-58407 :crop :empty 1 true]
 #datom[:a/node-58405 :a/owns :a/node-58407 1 true]
 #datom[:a/node-58406 :a/first :a/node-58407 1 true]
 #datom[:a/node-58406 :a/contains :a/node-58407 1 true]
 #datom[:a/node-58405 :a/owns :a/node-58406 1 true]
 #datom[:a/node-58405 :fields :a/node-58406 1 true]
 #datom[:a/node-58405 :db/ident :a/node-58405 1 true]
 #datom[:a/node-58405 :a/entity true 1 true])
(let [db (d/db conn)]
    (d/q '[:find ?kind
           :where
           [?e :crop :empty]
           [?e :kind :field]
           [?v :a/contains ?e]
           [?farm :fields ?v]
           [?farm :kind ?kind]] db))
=> ([:farm])
(let [db (d/db conn)]
    (d/q '[:find ?e ?e2
           :where
           [?e :crop :empty]
           [?e2 _ :farm]
           [?e ?a* ?e2]]
      (d/db conn)))
=> ()
I was under the impression from the docs that the second query should be able to find the term

quoll15:05:50

I’m in a meeting, but give me a few minutes to read 🙂

folcon15:05:03

No problems

folcon15:05:07

I'm not in a rush 😃

folcon15:05:26

Oh and if it wasn't clear, feel free to DM

quoll15:05:58

I would expect that second query to return an empty result. But if the final term was the other way around: [?e2 ?a* ?e] then it should return: [[:a/node-58407 :a/node-58405]]

quoll15:05:59

I could understand if I messed up and it didn’t return that though. The “zero step” when using * rather than + is not always intuitive

folcon15:05:16

Hmm, ok. I'll need to do an OR then to create a "find me a path" from X to Y.

quoll15:05:40

Why is that?

quoll15:05:39

Your original query tries to find a path from the embedded object back up to the top level object. There is none

quoll15:05:07

(Since paths are directed)

folcon15:05:37

Oh, well my intention was to try and write a query so I can take two arbitrary nodes and find a path between them, that way I can try and use that path in further subqueries. (This may be the entirely wrong way to think about this 😃...)

quoll15:05:38

There’s a path from the top level object (:kind :farm) to the embedded object (:kind :field)

quoll15:05:04

Oh, if you want to go either way, then yes. Use an OR

folcon15:05:43

Yep, that way I can have an entity in my world, say a :field and I can ask questions like, give me the farmer, or the lord who owns this etc.

quoll15:05:04

That’s only one direction though?

folcon15:05:34

Sure, but I can also take the farmer and ask, what fields do you own, so I can render them

folcon15:05:43

Or display them in a UI to the player 😃

quoll15:05:31

In that case, an OR. But it sounds like you’re using them for different things, so different queries would be better?

folcon15:05:46

A lot of the time the game engine goes up, whereas the UI rendering layer goes down.

folcon15:05:29

Sure, but one problem is visibility, I'm not yet clear how to introspect the graph and find out what nodes/edges are available from a particular node

quoll15:05:27

Well… if you have a node n then: • outbound edges are: [n '?edge '?node] • inbound edges are: ['?node '?edge n]

folcon16:05:14

A lot of the time when I'm coding at the repl I've got some reference to something and I'm trying to work out, where does this fit into the wider world, do I need to do a query or can I walk links to get what I want? This approach my not work in asami as I believe we get back values directly, instead of what datascript gives you which is an entity which when you call a key, if there is a link available, it will give it to you. If that workflow is possible in asami, that would be really helpful, but completely understand if not 😃 Actually, does the graph give you this? I should be able to call the values I get back from the query on the graph :thinking_face:.

quoll16:05:59

If you’re asking Asami for entities, then it just returns a map for you, so that’s the outbound values only. “Smart” entities are possible… but they’re not implemented. They would need a map to the graph that they’re from, and they would need to implement the map interfaces

quoll16:05:39

Yes, the graph has everything you need

quoll16:05:50

The query API gets everything it needs from the graph

folcon16:05:43

Ok, I'm going to try working with the graph and the query then and seeing if that gives me the info I need to build better queries. Sorry, there's a bit of difficulty in trying to work out how to use asami in a repl context 😃...

quoll16:05:33

inbound edges are (asami.graph/resolve-triple the-graph '?node '?edge your-node) outbound edges are (asami.graph/resolve-triple the-graph your-node '?edge '?node)

folcon16:05:11

Thank you, I'll give it a go!

quoll16:05:37

At a repl, I usually do the same, but with a query: e.g. for inbound edges, I’d do: (q '[:find ?node ?edge :in ?n :where [?node ?edge ?n]] your-db your-node)

quoll16:05:33

That query gets converted into the resolve-triple above