Clojurians Log v2

We have • a Generator, which relates to a set of Batch, • each Batch relates to a set of REC, • each REC relates to a set of Receipt • a Receipt may or may not have an attribute The query looks like:

[?b :batch/generator ?g]
[?r :rec/batch ?b]
[?receipt :receipt/recs ?r]
[(missing? $ ?receipt :receipt/transfer-id)]

As part of the query, we want to scope to Generators which have some Receipt down the entity graph path that is missing a :receipt/transfer-id. This is a 99% + case that the condition holds true, so it would be nice if we didn't have to explode the relationships and could "bottom out" once we detect that condition is satisfied.

So, "find the set of receipts that are not in the set of receipts which have a :receipt/transfer-id?" To confirm, 99%+ of :receipts do not have :receipt/transfer-id?

our short hand for receipts without :receipt/transfer-id are called "mint receipts" in English what we want would be described as: • only include generators with a mint receipt and 99% of our Generators have Batches where 99% of their RECs have a single mint Receipt

For the query results, we really don't care about the full collection of Receipts (that come from the multiplication of Batches x RECs x Receipts)

How many :receipt/transfer-in datoms are there total? e.g. what does d/db-stats say?

Receipts: :receipt/transfer-id -> 277686 :receipt/id -> 818165 RECs: :rec/id -> 778023 Batches: :batch/id -> 357959

It is thinking about reducing the scope of what the query needs to hold onto, when the result we want to use in the query is a boolean that could be "short circuited" once we hit true, a la some or reduced(true)

Have you considered writing a query function to do that by scanning :aevt, calling some ? (might be :vaet, still trying to understand the model)

They are nested relationships, so it would be multiple lazy-cat of :aevt scans if I am thinking about it correctly

It should hopefully be an edge case of our query, but I wanted to check if there are any ways to help guide Datomic about "bottoming out"

The goal is the clauses that happen before should (eventually) help narrow down the set of Generators we are getting batches for and reduce the working set needed

Datalog is set-based (as I'm sure you know), so if you wanted that short-circuiting capability you would need to create a predicate-fn which took a ?batch and quickly scrubbed through the indexes returning truthy.

That is what we were thinking. Mainly wanted to check that there wasn't some not-empty function that could allow Datomic to short circuit a set of relationships 😄

Your clauses would then look like:

:where
 [[?b :batch/generator ?g]
  [(my-batch-checker-that-shortcircuits $ ?b)]]

If these were modeled as tuples, you would be able to short circuit with a scrub

But because each clause is loading sets, I don't think there is any short-circuiting we can do with just the clauses.

What is "scrub"?

(some (d/datoms db :avet :my/tuple [?generator ?batch ....]))

ah

But you'd be maintaining a secondary index for this. Might be worth it but, alas, tradeoffs 🙂

We are on Datomic Cloud, so I don't think we get that secondary index feature anyway...

Thanks for confirming our suspicions

?? Cloud has tuples just fine

In cloud you would write the query fn as an ion, then it gets allowed in the query clauses

I was thinking Secondary Indexes were not part of Datomic Cloud I knew we could support a query function and tuples

what is the desired output of this query? (the :find, if that expresses it, or somethimg simpler like true/false?)

nm, I don't think it matters. if you want to do less IO, you need a predicate function that uses more direct index access, as Joe said.