Clojurians Log v2

Ok, having looked at the source, what's update-fn used for? It seems to be in the correct place to do what I want, but it's working at the transaction level instead of at the datom level. So it seems like I can broadly get the behaviour I want by splitting tx's into :db.fn/call's and non-db.fn/calls and then just calling transact multiple times. Not sure what the performance of that looks like, but it's a good test point.

Hi, I’m just looking at this now. I’m unaware of Datascript’s operations here. I’m expecting that it’s the same as Datomic’s transactions functions? Any idea why it’s not used the same way?

update-fn does a few things. It allows data to be passed in as a lazy seq, since it could be gigabytes. It also lets me return all the actual insertions, deletions, and generated ids that occurred during the transaction

Once upon a time it was a lot simpler, but to cover each of the various requirements it got into this horrible callback thing

Oh, and also, transactions can be done as either: • a stream of add/retract statements • a function that accepts a graph, and returns a new graph transact-update is asked to do both. The transact-data function takes a function that generates add/retracts or a pair of add/retract seqs and wraps them into the update-fn function that transact-update receives

Also… I see that asami.durable.store/transact-update* needs a try/catch to do a rollback. Right now, nothing (should) throw an exception, but if transaction functions are happening then this will need to happen

When you say: > Any idea why it’s not used the same way? What are you referring to? I was under the assumption that datascript correctly modelled datomic's behaviour when it came to :db.fn/call's (I've not used datomic itself in a while)

https://docs.datomic.com/on-prem/reference/database-functions.html#using-transaction-functions

After creating functions, they get called by their own name, mapped as a keyword

Oh yes, that bit, I think the current behaviour was deemed sufficient? To be honest I wasn't sure if asami was going to go down the route of having transaction functions.

They have their benefits

I planned to, but hadn’t got to it. I mean, I plan to do lots of things! 🙂 Things have been a bit slow in recent months, because of my new job, but I’m hoping to start improving my pace again

I’m thinking that a rollback on a transaction is the best approach, though there IS a new operation coming that could be used instead

I’ve written (and not fully tested) a “graph wrapper” graph. It wraps an existing graph, plus a graph full of new assertions, and a graph or retractions. Querying it gives a concat of the wrapped graph with the assertion graph, filtered by the retraction graph. This is the basis for 2 things: • db-with • fast transactions when executed on storage

I’m thinking that you might want me to get this committed asap

Oh, well if you're already doing that, I would love to kick the tires 😃

(incidentally, the fast transaction approach will launch a background thread to merge them, transparently switch over to the merged version, then inform subsequent wrapping graphs that it’s their turn to merge)

Mind if I DM to coordinate with you?

No problem 😃

One question that came up when I was testing btw:

(let [things
        [{:name :wheat-farm
          :kind :farm
          :fields [{:kind :field :crop :empty}]}]]
    (-> @(d/transact conn {:tx-data things})
      :tx-data))
=>
(#datom[:a/node-58405 :name :wheat-farm 1 true]
 #datom[:a/node-58405 :kind :farm 1 true]
 #datom[:a/node-58407 :kind :field 1 true]
 #datom[:a/node-58407 :crop :empty 1 true]
 #datom[:a/node-58405 :a/owns :a/node-58407 1 true]
 #datom[:a/node-58406 :a/first :a/node-58407 1 true]
 #datom[:a/node-58406 :a/contains :a/node-58407 1 true]
 #datom[:a/node-58405 :a/owns :a/node-58406 1 true]
 #datom[:a/node-58405 :fields :a/node-58406 1 true]
 #datom[:a/node-58405 :db/ident :a/node-58405 1 true]
 #datom[:a/node-58405 :a/entity true 1 true])
(let [db (d/db conn)]
    (d/q '[:find ?kind
           :where
           [?e :crop :empty]
           [?e :kind :field]
           [?v :a/contains ?e]
           [?farm :fields ?v]
           [?farm :kind ?kind]] db))
=> ([:farm])
(let [db (d/db conn)]
    (d/q '[:find ?e ?e2
           :where
           [?e :crop :empty]
           [?e2 _ :farm]
           [?e ?a* ?e2]]
      (d/db conn)))
=> ()

I’m in a meeting, but give me a few minutes to read 🙂

No problems

I'm not in a rush 😃

Oh and if it wasn't clear, feel free to DM

I would expect that second query to return an empty result. But if the final term was the other way around: [?e2 ?a* ?e] then it should return: [[:a/node-58407 :a/node-58405]]

I could understand if I messed up and it didn’t return that though. The “zero step” when using * rather than + is not always intuitive

Hmm, ok. I'll need to do an OR then to create a "find me a path" from X to Y.

Why is that?

Your original query tries to find a path from the embedded object back up to the top level object. There is none

(Since paths are directed)

Oh, well my intention was to try and write a query so I can take two arbitrary nodes and find a path between them, that way I can try and use that path in further subqueries. (This may be the entirely wrong way to think about this 😃...)

There’s a path from the top level object (:kind :farm) to the embedded object (:kind :field)

Oh, if you want to go either way, then yes. Use an OR

Yep, that way I can have an entity in my world, say a :field and I can ask questions like, give me the farmer, or the lord who owns this etc.

That’s only one direction though?

Sure, but I can also take the farmer and ask, what fields do you own, so I can render them

Or display them in a UI to the player 😃

OK

In that case, an OR. But it sounds like you’re using them for different things, so different queries would be better?

A lot of the time the game engine goes up, whereas the UI rendering layer goes down.

Sure, but one problem is visibility, I'm not yet clear how to introspect the graph and find out what nodes/edges are available from a particular node

Well… if you have a node n then: • outbound edges are: [n '?edge '?node] • inbound edges are: ['?node '?edge n]

A lot of the time when I'm coding at the repl I've got some reference to something and I'm trying to work out, where does this fit into the wider world, do I need to do a query or can I walk links to get what I want? This approach my not work in asami as I believe we get back values directly, instead of what datascript gives you which is an entity which when you call a key, if there is a link available, it will give it to you. If that workflow is possible in asami, that would be really helpful, but completely understand if not 😃 Actually, does the graph give you this? I should be able to call the values I get back from the query on the graph 🤔.

If you’re asking Asami for entities, then it just returns a map for you, so that’s the outbound values only. “Smart” entities are possible… but they’re not implemented. They would need a map to the graph that they’re from, and they would need to implement the map interfaces

Yes, the graph has everything you need

The query API gets everything it needs from the graph

Ok, I'm going to try working with the graph and the query then and seeing if that gives me the info I need to build better queries. Sorry, there's a bit of difficulty in trying to work out how to use asami in a repl context 😃...

inbound edges are (asami.graph/resolve-triple the-graph '?node '?edge your-node) outbound edges are (asami.graph/resolve-triple the-graph your-node '?edge '?node)

Thank you, I'll give it a go!

At a repl, I usually do the same, but with a query: e.g. for inbound edges, I’d do: (q '[:find ?node ?edge :in ?n :where [?node ?edge ?n]] your-db your-node)

That query gets converted into the resolve-triple above

Thanks!

Ok digging into it further it's discussed here: • https://github.com/threatgrid/asami/discussions/158 • https://github.com/threatgrid/asami/issues/190 • https://github.com/threatgrid/asami/issues/98: Not sure if this approach has memory implications, do I need to explicitly call delete-database for example, or will it just get GC'ed?

I’m guessing that you're trying to do this from outside Asami, and just using it through the public APIs. Is that right?

Well I was hoping to do it from the inside, but I'm not clear where to start from there, so yes, I'm looking at the public api's first 😃... If this is already on your roadmap or there's a really simple way to support it, I'm all ears!

If so, db-with is going to be difficult to do without

(Sorry, slow typing. I’m on my phone in a Dr office right now, so I'll go quiet soon)

I have db-with ready for testing, but not in yet

Hopefully very soon?

That's fine 😃... Happy to discuss further, when it's more convenient for you. I'm basically looking to use something datascript like in a game context, so asami is really interesting from a perf perspective. Happy to test stuff if that's useful!

Are you staying with an in-memory db, or using durable storage (ie disk)?

If it's in memory, then it should be easy to work with

I'm sticking to in-memory for now, at some point in the future, durable will become useful, ie save files, but for now, in-memory is perfectly fine 😃

OK. Will definitely follow up

Dr just left for a minute… A Database is just a thin wrapper around a Graph object, that makes it look like a Datomic database. So you can access it the graph directly if you want. (Look in asami.graph for the protocol. It's implemented in asami.index). This is just like any other immutable structure, so to wind back you can just use the previous version.

All the functions in asami.core get the latest Database from a Connection, and then the Graph from the Database, then everything happens on the graph

Ok, I'll dig around in that and see what I can figure out, just a quick question, is there anything about reverse relations? I remember seeing in one of your talks that you had bidirectional relations, but when I searched the repo I couldn't see any mention of it. Figured it out, I was putting my entities in a vector so I had to first walk that relation >_<...

Would you be looking to do something similar to what datascript does here? Or a different approach? • Repeated recur inside the transact when a :db.fn/call is present? https://github.com/tonsky/datascript/blob/55af41d6d7d18f12358957a056a6cd1bb484fee7/src/datascript/db.cljc#L1429