I think you may just want to use rules to organize your query?

There are some things difficult to express in datalog eg https://stackoverflow.com/questions/43784258/find-entities-whose-ref-to-many-attribute-contains-all-elements-of-input but this sounds like you just don’t like binding a new name? A rule would hide that in its scope

Yeah, that’s what I was going for, but if ?inv goes into a rule, then it’s going to get unified, right? Why means a rule can’t do the GROUP BY/HAVING type logic to scope the ?cust values, or it will reject some invoices I want. (Let me know if this is unclear.) So I think the best I can do is to roll up my initial invoice constraints (in this case (not [?large-inv :invoice/void?])) into a rule so it can be cleanly used twice (what I called “A” and “B” in the initial text). But this rule needs to be used from inside the ?cust selection logic, so it seems impossible to have a general (cust-with-large-invs ?cust ?inv) type rule that would compose.

Ha, thanks. I actually had this exact page open but it’s a bit challenging.

You would unify with the rule arg, but bind to another name for the cust-inv search

Specifically, I don’t understand what this is doing: `

[(identity ?groups) [?group ...]]

Right, but if I’m binding to a new name, how can I say that ?rule-inv ⊆ ?inv ? This is where I get stuck.

I don’t think that’s what’s going on here. You want invoices whose customers have invoices where any one of them is > 100?

Or do they all need to be over 100?

“all” is the hard part; “any” is trivial

No, just one. While preserving some pre-existing set of constraints over the invoices. In this case, only dealing with non-void invoices.

Any is what I’m struggling with, so please enlighten me 🙂

Your first query gives you the right behavior?

It does

I’m wondering if there is any way to write it something like this and still get back all of the invoices for the matched customers: (d/q '[:find ?cust ?inv :in $ % :where [?inv :invoice/customer ?cust] (not [?inv :invoice/void?]) (high-bal-cust ?cust ?inv)] test-data rules)

(sorry, typos fixed now)

Extract clauses for A and for B to separate rules which share ?inv

Im sorry i can’t spell this out further because I’m on a phone

haha, no trouble. I appreciate the help either way. Just to make sure we’re on the same page, the high-bal-cust rule would need to embed the invoices-we-want rule, right? Meaning they don’t really compose.

(High-bal-cust ?cust)(non-void-inv-for-cust ?inv ?cust)

Order doesn’t matter for correctness just performance

Your ?large-inv is inside the high-bal-cust impl

You’re saying the rules look like this, yes?

[[(high-bal-cust ?cust)
  (non-void-inv-for-cust ?inv ?cust)
  [?inv :invoice/balance ?bal]
  [(> ?bal 100)]]]

I gave the query not the rule

To be a high bal cust you need a non void inv with bal > 100

Correct

Then you just want all the non-void inv for those customers right?

Yes

So I’m not seeing the nesting?

Ah ok you are trying to DRY harder

Yeah that could serve as a high-bal-cust impl

Sorry, trying to type up a working example of what I think you’re proposing.

You don’t want to repeat the not-void check

Right. I think you’re saying something like this: (d/q '[:find ?cust ?inv :in $ % :where (non-void-inv-for-cust ?inv ?cust) (high-bal-cust ?cust ?inv)] test-data '[[(non-void-inv-for-cust ?inv ?cust) [?inv :invoice/customer ?cust] (not [?inv :invoice/void?])] [(high-bal-cust ?cust) (non-void-inv-for-cust ?inv ?cust) [?inv :invoice/balance ?bal] [(> ?bal 100)]]]) (I’m not immediately seeing why but this is giving me an OutOfBounds exception, probably just some typo I haven’t spotted yet). I think this kind of thing should work fine, but the high-bal-cust is married to the constraints on the invoices (`non-void-inv-for-cust`). I was hoping to be able to bring any constrained set of invoices and then apply something like high-bal-cust to it.

Oh, I see the typo now…

Fixed. And yields expected results:

(d/q '[:find ?cust ?inv
       :in $ %
       :where
       (non-void-inv-for-cust ?inv ?cust)
       (high-bal-cust ?cust)]
     test-data
     '[[(non-void-inv-for-cust ?inv ?cust)
        [?inv :invoice/customer ?cust]
        (not [?inv :invoice/void?])]
       [(high-bal-cust ?cust)
        (non-void-inv-for-cust ?inv ?cust)
        [?inv :invoice/balance ?bal]
        [(> ?bal 100)]]])
;; => #{[1 3] [1 4]}

You can make high-bal-cust accept an inv, and derive cust from it. But again it’s not the same inv as the Invs you are checking for high balances

Right… this is what I wish I could write:

(d/q '[:find ?cust ?inv
       :in $ %
       :where
       (non-void-inv-for-cust ?inv ?cust)
       (high-bal-cust ?cust ?inv)]
     test-data
     '[[(non-void-inv-for-cust ?inv ?cust)
        [?inv :invoice/customer ?cust]
        (not [?inv :invoice/void?])]
       [(high-bal-cust ?cust ?inv)
        [?large-inv ∈ ?inv]
        [?large-inv :invoice/balance ?bal]
        [(> ?bal 100)]]])

Basically ?large-inv is a new name for ?inv that can be separately unified.

I don’t understand why you think ?large-inv and ?inv should be related

Because ?inv has been scoped to a subset of invoices (the non-void ones, in this case). Now I want only invoices from that narrowed scope, if their associated customer has an invoice from that narrowed scope with a balance > 100.

It may sound convoluted, but I think it’s pretty common. Show me all of the invoices owed by customers who owe us a lot. This sort of thing. (Really owe some large invoice, since I’m not working off a total in this case, but either variant would be interesting to me.)

(voided invoices don’t count)

The candidate ?inv you are filtering is not the same as the ones you are inspecting for cust high-bal test

You can do this, but you need to supply all the invs as a single binding, and you still need to destructure twice

I’m a bit lost on both counts. What do you mean by “destructuring”? (I’m well-acquainted with the term, just not sure what you mean in this context.)

Sounds like you’re talking about binding the variables to the set of candidates in a rule, yes?

Unpack ?invs to individual ?inv

Not necessarily in a rule

What you are describing is soothing I would normally only attempt if I discovered a perf problem

I’m actually trying to go the opposite direction… Not exploring this for performance reasons. In fact, ignoring that fully for the moment and thinking about how I can write a series of queries in terms of rules that compose so that I can ensure they behave in similar ways.

Didn’t we just do that?

Haha, I don’t think so. 🙂 Let me type up something else to look at…

(Non-void-inv ?inv)(filter-inv-high-val-cust ?inv ?cust)

Where the vars are the same among rule invocations, that has the effect of a filter

Unification ensures you on out get the tuples left where all rules matched

“On out”=>only

But filter-inv-high-val-cust is going to reject the invoices with balance < 100

So we only get filter AND high balance AND inv cust has some high balance inv

No, because it won’t unify on ?inv for the high Val balances, only on the cust

?inv happens to be a superset of ?large-inv in this case in this order but not necessarily

Datalog unification means order doesn’t matter

(Again except in datomic for performance)

I’m not suggesting order matters. I’m just not sure how you’re going to pass the constrained set of invoices to filter-inv-high-val-cust , then query those invoices for high balance ones without causing unification on ?inv. That’s exactly the crux of my issue.

You will make my day if you tell me what I can put in filter-inv-high-val-cust to make this happen 🙂

You did it in your first query via another variable name

?large-inv

Right, but that only works if the logic for constraining the set of invoices if also embedded there. Which means that function is not actually reusable / composable. It’s a mirror of what was in the original query.

It’s not reusable if, say, we wanted non-void invoices for some particular product, for example.

I see, you want goodness to be parameterizable

Voidness

But that is introducing a third concept separate from ?inv and ?large-inv

?legal-invs or ?checked-invs something like that

Another option is to filter the data source itself

Yeah, that is an interesting option.

I just want to mimic this:

(->> [{:invoice/balance 40M, :invoice/customer 1, :db/id 3}
      {:invoice/balance 150M, :invoice/customer 1, :db/id 4}
      {:invoice/balance 50M,
       :invoice/customer 1,
       :invoice/void? true,
       :db/id 5}
      {:invoice/balance 40M, :invoice/customer 2, :db/id 6}
      {:invoice/balance 150M,
       :invoice/customer 2,
       :invoice/void? true,
       :db/id 7}]
     (remove :invoice/void?) 
     (group-by :invoice/customer)
     vals
     (filter #(some (fn [{:invoice/keys [balance]}] (> balance 100)) %)))
;; => ([{:invoice/balance 40M,  :invoice/customer 1, :db/id 3}
;;      {:invoice/balance 150M, :invoice/customer 1, :db/id 4}])

Filter some invoices, then group and filter the group.

I’m not sure this would be fast enough but you could also do [(non-void-inv ?inv) [(identity ?inv) ?large-inv] (large-bal-cust ?inv ?cust ?large-inv)]

It seems that I have to tie the code of first and second filters together because I can’t just continue processing the intermediate value

Ok, so that’s what I was wondering about earlier. So [(identity ?inv) ?large-inv] basically lets me bind a new name… I’m sure we lose all set-related optimizations, but it’s possible…

Identity is a Clojure function that returns its argument

(Identity x) => x

I definitely know that 🙂

So this is just an idiom for rebinding without unifying

Not datalog syntax

Right, I follow.

But then we’ve dropped down to dealing with individual values… generate and filter

Which works. But my broader question is, is there a way to get a fresh binding for a collection and not have it unify. I think the answer is ‘no’.

Sub query or :in

So…as far as I can tell, you can only pass scalars to a subquery, correct?

I don’t think there are any docs on this, so this is just what I’ve seen in my prodding.

No

Ok, so let’s say we wanted to write (large-bal-cust ?cust ?inv) with a subquery that checked to see if any of the invoices in ?inv for that ?cust had a high balance; is there a way to do that?

I’m unsure how to pass in the relation (or collection of ?inv), or however that works exactly…

All I have discovered is how to pass in one ?cust and one ?inv. Specifically talking about subqueries here.

Simplest is :in ?set-of-inv-ids

Then just don’t destructure

Sub query is same concept

Datalog unification is hiding the whole set behind ?inv for you (because it is not knowable until all constraints are satisfied), so there’s no way to “collection-use” all ?inv values in the middle of your query

“Collection-ize “

So…maybe you can tell me what I’m doing wrong here…

(d/q '[:find ?cust ?inv
       :in $ %
       :where
       (non-void-inv-for-cust ?inv ?cust)
       (high-bal-cust ?cust ?inv)]
     test-data
     '[[(non-void-inv-for-cust ?inv ?cust)
        [?inv :invoice/customer ?cust]
        (not [?inv :invoice/void?])]
       [(high-bal-cust ?cust ?inv)
        [(datomic.api/q
          '[:find ?i .
            :in $ ?c ?i
            :where
            [?i :invoice/customer ?c]
            [?i :invoice/balance ?bal]
            [(> ?bal 10000)]]
          $ ?cust ?inv)]]])

But you see how “what Invs should filters inspect” is itself a filter parameter. Conceptually it will always be a different thing from “does this particular inv unify”

Oh, you’re saying it is not possible to do this. Ok.

I’m saying it’s just not how datalog works. You’re confusing the columns for the rows in datalog

Right? Can’t “collection-ize” in a subquery.

You can return a collection from the sub query

And bind to a new name that is the whole collection

Ah.

Then you can unify with other vars later via destructuring the collection or even via (contains? ?coll ?x) predicates

Where ?coll is what I returned. Because if it’s a logic variable then we don’t know what will be in it.

But the point is datalog sees it as a single value not a collection. It’s not participating in unification

Yep, I gotcha.

So I think I understand the mechanism, and I appreciate you explaining it. But I’m not sure if it gets us any further. Say we wrote a subquery to find high-balance invoices for a customer; if we intersect that with the ?inv values (via (contains ?high-val-invs ?inv)), then we’re back to unifying ?inv in an undesirable way. We can’t give ?inv (as a collection) a new name without knowing the rules that were used to build that collection. Or at least this is my thesis…

Basically I want something like the identity idiom but for a collection in the query engine so I can say “duplicate ?inv to a new name and don’t unify back to it”. I don’t think such a mechanism exists, but I’m not aware of a reason why it couldn’t.

Because the true set of ?inv is what ultimately pops out of the :find

It’s not the “non void invoices” of your first clauses

Datomic happens to evaluate clauses in order (mostly) but datalog semantics don’t

So you need a new name for that set

That was the example earlier using identity

It’s interesting, I’m curious what I’m saying that has you thinking I care about order. I’m not trying to speak to order. But I must sound like I am.

What the set members are of ?inv depends on what clauses were evaluated

Thus order matters

The set gets smaller as you evaluate more clauses

From a performance perspective, sure.

And, perhaps, if we do more non-Datomic Clojure stuff in query

So you can’t say at a filter rule boundary “just whatever is in my ?inv now use for ?largeinv checks”

Oh.

Because that is an impl detail

I’m not actually trying to say “now”.

You need a new name to make it stable, thus another argument to the rule

I mean capture the constraints on this variable (at all times) then further add to them in a new name.

Maybe that’s not logically consistent but it sounds reasonable to me at first blush.

Ok, syntax you want is something like (not-void ?inv) (bal-over 300 ?inv) (high-bal-cust ?inv ?cust)

Bal-over I added as a new “filter” to illustrate the problem

No, I don’t think you’re suggesting what I’m looking for.

How is high-bal-cust going to know that ?inv means only not-void and doesn’t fail to check ?inv with bal < 300

I want to say: (let [?x (fresh-var)] [?x ⊆ ?inv] [?x :invoice/balance ?bal] [(> ?bal 100)]) Syntax not withstanding. I want introduce a new name, with additional constraints, but not apply those constraints to ?inv.

Ok that’s [(identity ?inv) ?new-inv]

But has to be done at the level of individual values.

The key is how do you communicate to a filter what ancillary values it may inspect vs the thing it is filtering

I see no automatic way of communicating this, because it is core to the semantic of the filter

You need to name the two things, not infer it from the filtered thing

So…let say we have some set, S. X = {x ∈ S: x > 0} Y = {x ∈ X: x > 100} Y doesn’t impose any constraints on X, right? S is analogous the DB. X is analogous to ?inv. Y is what I’m looking for. I’m probably missing something because I don’t have a complete mental model of how the query resolver works, but it’s not obvious to me why we can’t have Y without having to define it directly in terms of S.

I’m not sure what ancillary values you’re referring to. I assume you must mean the relationship between ?cust and ?inv?

The ancillary values are the invoices of the customer whose balances must be inspected

Distinct from the ?inv being filtered

In your example, X is not inv except directly after the non-void-inv binding and before any other filtered are applied

?inv is actually the query result itself, the set of non void inv whose customers have high balance non void invoices

The set of invoices where the ?inv binding is first established is not the set of ?inv

Datalog is solving a constraint problem, and doesn’t say anything about set members that don’t satisfy the constraint

But it can only shrink in size from there, correct? We add additional constraints on the set.

Yep, I think I follow all of that

And you’re suggesting I want to apply constraints to the set members that don’t satisfy the constraint. Is that right?

I’m saying that “can only shrink from here” is an impl detail

Borne out of the clause ordering

And the particular way datomic evaluated datalog

And what if you have multiple filters ?

Yes, but I don’t think I’m trying to rely on that implementation detail. I’m just describing how I understand it to work.

Which one is the “this is the whole thing” rule and which is the “filtered down” one?

I’m suggesting that filtered values would have to have new names

You seem to want a way for ?inv to “remember” the largest set it ever had

the non magical way to do that is to give that set another name and pass it as an additional param to rules that need it

That’s what I’m trying to do…without having to rebuild the value from scratch

I’m not trying to change the value of ?inv in (only) some places.

I’m saying I want to bind ?large-inv to be the members of ?inv with large balances.

Ok, but the members of ?inv is not all non-void inv

Depending on where you do it

Ahhhh. So you’re saying if we don’t unify it “back” (as it were), we’re imposing order-dependent results.

Because we’re not ultimately getting the intersection of all of the constraints anywhere (necessarily)

I’m saying the fact that the rule is invoked with non-void inv that aren’t in the final :find is an impl detail

Non-void invoices and ?inv only happen to be the same in this particular datalog rule time at particular spots in it’s evaluated

“Rule time “=>runtime

I follow you. The ?large-inv rules could run on all invoices, or could run on non-void invoices, yielding potentially different results. But if it added a unifying constraint to the overall query then it wouldn’t matter which order it ran in.

Mathematically ?inv never “was” any set other than the set that satisfies the entire query

That it “changes” is an impl detail

Yes, I follow.

I appreciate you walking me through that. It’s a good point. I recall these kinds of questions coming up in miniKanren and in Prolog… and that people often reach for impure solutions.

So (non-void-impl ?all-inv)(non-void-impl ?inv)(has-largebal-cust ?inv ?cust ?all-inv) would do it

But may not be best perf

Managing ?all-inv as a single coll value may be faster

Where this is really annoying actually is negation

I imagine so

I have to step away for a bit. I’ll check back when I can. Thanks again for your help!

(or-join [?x] (cond ?x) (and (not (cond ?x)) (other-cond ?x))

(Cond ?x) seems to evaluate twice needlessly, and there really is no workaround, and it seems like a sufficiently smart datalog could avoid that

Or at least memorize it

*memoize

great discussion! i was also struggling with a similar problem and i was not sure how to introduce a 2nd set of the same kind of entities. it's reassuring to hear that the [(identity ?e) ?other-e] is a common pattern to break unification.

I’m impressed you made it all the way here, haha. 🙂

… or maybe has to do with “Everything about query-stats is alpha and subject to change in future releases.“? Still seems worthwhile to expose it in a more explicit way, even if in alpha.

Yikes! Could have sworn it was there. I'll double check and see if it got reverted.