Fork me on GitHub
#datomic
<
2019-12-18
>
Quest00:12:14

Could the tuple attribute limitation of "must have at least 2 elements" be removed in a future release? I've been using it but padding with nil to reach >= 2 length. This creates awkward code like below.

(defn pad-tuple-nils
  [v]
  (let [length (count v)]
    (cond (>= length 2) (vec v)
          (= length 1) (conj (vec v) nil)
          :else [nil nil])))
And the inverse, making sure any frontend display runs a (remove nil? tuple) so as not to render empty elements. I do want to note that the convenience of getting vectors back from DB queries is a great addition & I hope this can be upgraded to bring greater parity between plain Clojure vectors <-> Datomic tuples.

camdez15:12:17

Hi all…struggling with a query that I feel shouldn’t be too hard to write…perhaps I’m wrong…

(def animal-db
  [[1 :name "Bear" 1]
   [1 :kingdom :mammal 1]
   [2 :name "Dog" 1]
   [2 :kingdom :mammal 1]
   [3 :name "Snake" 1]
   [3 :kingdom :reptile 1]
   [4 :name "Frog" 1]
   [4 :kingdom :amphibian 1]])

;; Find all animal names belonging to the kingdoms passed in
(d/q '[:find [?name ...]
       :in $ [?target-kingdom ...]
       :where
       [?animal :name ?name]
       [?animal :kingdom ?target-kingdom]]
     animal-db #{:mammal :reptile})
;; => ["Snake" "Dog" "Bear"]

;; How can I find the names of the animal NOT in the kingdoms passed
;; in?

;; ???

;; => ["Frog"]
(1) How can I write the desired query, excluding values for a collection passed? (2) Is this a case of “dynamic conjuction”, as described in this post? https://stackoverflow.com/questions/43784258/find-entities-whose-ref-to-many-attribute-contains-all-elements-of-input

camdez15:12:31

FWIW, I know it can be done the following way, but I assume it’s inefficient:

(d/q '[:find [?name ...]
       :in $ ?target-kingdoms
       :where
       [?animal :name ?name]
       [?animal :kingdom ?kingdom]
       (not [(contains? ?target-kingdoms ?kingdom)])]
     animal-db #{:mammal :reptile})

favila15:12:10

why assume it’s inefficient?

camdez15:12:26

Since predicate expressions can contain arbitrary code, they must not be factored into the query planner…I’m deducing that they just run over the full set of matches, and filter it. In this case that would be every single entity (“animal”), so I’m basically doing a full tablescan.

camdez15:12:44

Rather than taking advantage of the indexes.

favila15:12:57

that’s true either way

favila15:12:38

[?animal :kingdom ?kingdom] if it were the first clause and :kingdom were indexed would benefit from an index

favila15:12:05

if it’s not, it’s a filter; and using datalog pattern-matching may be slower than checking against a set

favila15:12:00

[?animal :kingdom ?kingdom][?animal :name ?name] is the fastest if ?kindom is known and :kingdom is indexed

camdez15:12:55

Totally agreed that the clause ordering would have been a fundamental issue for performance. But that’s trivially fixable. With that out of the way, is there a way the query can be written?

favila15:12:28

[?a :kingdom ?k](not [(contains? ?kingdoms ?k)]) [?a :name ?n]

favila15:12:44

will scan :kingdom, but filter quickly

favila15:12:18

alternatively if you know all kingdoms, you can find the set-difference and convert your negation into a positive match

favila15:12:50

that would be faster if computing the set-difference is faster. if kingdom is an open set you will need a query to find all kingdoms anyway, so it may not be faster

camdez15:12:28

Yeah, the contains? approach is the same as what I supplied earlier, but with an improved clause ordering (I do know this is significant from a performance perspective). Call it an intellectual exercise if you want, but is it possible to write the query with a collection binding, similar to the way the original query matching desired kingdoms (rather than their complements) was written?

favila15:12:56

using (not [?a :kingdom ?kingdom])

favila15:12:32

(where ?kingdom is destructured from your list of kingdoms not to include)

favila15:12:27

The difference is only that this will evaluate every ?kingdom, but the set containment test will only run once

camdez15:12:40

I believe you’re suggesting the following:

(d/q '[:find [?name ...]
       :in $ ?target-kingdoms
       :where
       (not [?animal :kingdom ?target-kingdoms])
       [?animal :name ?name]]
     animal-db #{:mammal :reptile})
;; => ["Frog" "Snake" "Dog" "Bear"]
Which returns incorrect results. I suspect because it’s finding all animal where there exists a kingdom not in the target-kingdoms set.

favila15:12:11

no, I’m suggesting :in $ [?target-kingdom …] and (not [?animal :kingdom ?target-kingdom])

favila15:12:21

that clause should also be second

favila15:12:41

> (where ?kingdom is destructured from your list of kingdoms not to include)

favila15:12:02

By this I mean [?kingdom …]

favila15:12:51

(d/q '[:find [?name ...]
       :in $ [?target-kingdom ...]
       :where
       [?animal :name ?name]
       (not [?animal :kingdom ?target-kingdom])]
     animal-db #{:mammal :reptile})
;; => ["Frog" "Snake" "Dog" "Bear"]

favila15:12:03

putting it all together

camdez15:12:05

I appreciate you putting it all together. But the results are still not what we’d want (I just ran it). Desired output would be ["Frog"].

camdez15:12:03

I do understand that if I had a closed set (i.e. I know the full set even before going to the database), then that would make things much easier.

favila15:12:11

oh, yes, you’re right, the semantics of not don’t help here because every possibility is evaluated

favila15:12:30

so at least one of the kingdoms will not-match, thus the not clause will succeed

favila15:12:12

or I think? I’m still a bit puzzled by this result honestly

camdez15:12:30

Bingo, that’s what I’m thinking. And what I poorly tried to explain above. I think it expands to something like:

(or (not (= kingdom :mammal))
    (not (= kingdom :reptile))
    ,,,)
And it’s always not at least one of those values. So, somehow, those need unify…

favila16:12:09

I can’t think of anything which doesn’t force evaluating an item at a time (e.g. first+rest plus recursive rule) or uses sets

favila16:12:41

I doubt anything is faster than set membership testing

camdez16:12:39

I appreciate the input. This SO post has some suggestions that feel like they might apply, but I haven’t yet managed to adapt them to fit this problem: https://stackoverflow.com/questions/43784258/find-entities-whose-ref-to-many-attribute-contains-all-elements-of-input

dvingo18:12:59

I've been playing around with this, set up an in-mem db:

(def schema
  [#:db{:ident :animal/kingdom :valueType :db.type/ref :doc "" :cardinality :db.cardinality/one}
   #:db{:ident :animal/name :valueType :db.type/string :doc "" :cardinality :db.cardinality/one}
   #:db{:ident :kingdom/name :valueType :db.type/keyword :doc "" :cardinality :db.cardinality/one}])
(def data
  [{:kingdom/name :mammal :db/id 1}
   {:kingdom/name :reptile :db/id 2}
   {:kingdom/name :amphibian :db/id 3}

   {:db/id 4 :animal/name "Bear" :animal/kingdom 1}
   {:db/id 5 :animal/name "Dog" :animal/kingdom 1}
   {:db/id 6 :animal/name "Snake" :animal/kingdom 2}
   {:db/id 7 :animal/name "Frog" :animal/kingdom 3}])

(d/transact conn schema)
(d/transact conn data)
This is the only way I could get a query with the correct result:
(d/q '[:find ?kingdom
       :where [?kingdom :kingdom/name ?name]
       (not [?kingdom :kingdom/name :mammal])
       (not [?kingdom :kingdom/name :reptile])
       ] (d/db conn))
the (not [(contains?... version did not work for me.

dvingo18:12:58

This seems in line with Val's SO post along the line of generating a query.

onetom15:12:09

I think u can do something with (complement #{:mammal :reptile})

stijn19:12:01

is there a way to see which version (rev, i.e. git sha) of an ions application is deployed on a give compute group?

dvingo19:12:23

if you know which CodeDeploy you want, click on the deploy and then there is a section "Revision details"

stijn19:12:37

ok, it's for script automation, but I think I can get to it with the aws cli, thanks!