datalevin 2023-03-10 | Slack Archive

Hello everyone, Would it be possible to mix a Datomic DB and a Datalevin DB in the same (Datomic) query ?

(require '[datomic.api :as d])
(require '[datalevin.core :as dl])

(def dl-conn
  (dl/create-conn "/tmp/datalevin/dl_db" {}))

(dl/transact dl-conn [{:user/email ""
                       :user/email-hash "e2e6d7a977f2ca2b3900e22c68655b30"}])

(d/q '[:find ?fullname .
       :in $ $dl ?email
       :where
       [$dl ?datalevin-id :user/email      ?email]
       [$dl ?datalevin-id :user/email-hash ?email-hash]
       [$   ?datomic-id   :user/email-hash ?email-hash]
       [$   ?datomic-id   :user/fullname   ?fullname]
       ]
     (d/db datomic-conn) @datalevin-conn "") ;; => nil

The idea would be to join around the ?email-hash value and use Datalevin as a mutable store where critical GDPR values can be deleted. I tried it and got nil as a result.

jjttjj14:03:33

Instead of using the db position in the where clauses, could you call a nested query on the datalevin db? Something like this (untested)

(d/q '[:find ?fullname .
       :in $ $dl ?email
       :where
       [(datalevin.core/q [:find ?email ?email-hash
               :where
               [$dl ?datalevin-id :user/email      ?email]
               [$dl ?datalevin-id :user/email-hash ?email-hash]]
          $dl)
        [?email ?email-hash]]
       [$   ?datomic-id   :user/email-hash ?email-hash]
       [$   ?datomic-id   :user/fullname   ?fullname]
       ]
  (d/db datomic-conn) @datalevin-conn "")

I forget off hand how what's required to invoke arbitrary functions like this on datomic, but I know something like this should work if you instead invert things and call datomic from a datalevin query

lambdam14:03:35

:thinking_face: I don't get exactly what kind of Datomic clause is the following code:

[(datalevin.core/q [:find ?email ?email-hash
                    :where
                    [$dl ?datalevin-id :user/email      ?email]
                    [$dl ?datalevin-id :user/email-hash ?email-hash]]
                   $dl)
 [?email ?email-hash]]

From https://docs.datomic.com/on-prem/query/query.html I can see:

pred-expr                  = [ [pred fn-arg+] ]
fn-expr                    = [ [fn fn-arg+] binding]

Is it one of the two?

jjttjj14:03:55

the latter

jjttjj14:03:27

It's surprisingly hard to find documentation of this ability, across datomic/datalevin/datascript

lambdam14:03:26

I'll try this on the branch with my wip code and let you know what happens.

lambdam14:03:57

Yes I'm looking for information and struggle a bit

jjttjj14:03:04

I think in datomic you may need to register the function or something, and it might be different behavior across cloud and on prem

jjttjj14:03:39

but on datascript at least, and i think datalevin, you can just use any fully qualified symbol within a query

Huahai17:03:40

In your code you are using datomic as the query engine, maybe do the reverse? datascript/datalevin can query on any data source that looks like a seq of triples (triple could be a vector or a map), i.e. something like [[1 :a "a"] [2 :b "b"]] is a valid data source, even [[:a "a"] [:b "b"]] works. Not sure about datomic.

Huahai17:03:29

and as jjttjj said, you can use whatever fully qualified function in the query in datalevin.

lambdam14:03:14

I was taking inspiration from this article: https://tgk.github.io/2012/11/creating-datomic-data-sources.html

jjttjj15:03:33

Is datalevin.core/v supposed to work on the results of get-range?

(->> (dl/get-range kvdb jobs-table [:all] :instant :data)
       (mapv dl/v))

Is giving me:

IllegalArgumentException No implementation of method: :v of protocol: #'datalevin.lmdb/IKV found for class: clojure.lang.PersistentVector

Huahai16:03:29

k and v are there for working with IKV, which is the raw point query result. Range query results are already compatible with Clojure data structures, so you shouldn’t need to use these.

Huahai16:03:51

k and v is needed if you are writing a predicate, for example, where you need to deal with raw IKV

Huahai16:03:59

get-range etc return a seq of Clojure vectors or scalars, depending on your arguments

Huahai16:03:32

k and v return ByteBuffer, that you need to use b/read-buffer on them.

Huahai16:03:28

So they are very low level, and are used in function such as visit, range-filter etc to write predicates.

jjttjj16:03:28

gotcha, thanks!

Huahai17:03:03

Updated the doc to make this clear. Thx.

🙏 2

2023-03-10

Channels