Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2023-09-08

Channels

#datomic
<
2023-09-08
>
steveb8n11:09:02

AWS is about to release aurora serverless with vector support. Feels like the right tool for the job despite how we already use datomic

gratitude-thank-you 2
steveb8n11:09:56

I heard it at an AWS event in Sydney a few weeks ago. It's not pgvector. Reach out to AWS for more information

Chip11:09:32

Thanks again.

wikipunk13:09:23

I’m using qdrant with Datomic

Ben Kamphaus14:09:14

if you need all the bells & whistles of efficient storage/transfer for e.g. 1000 elem + vector embeddings, I’d definitely put vectors in an external proper vector db and link to what’s in datomic by id. if they’re smaller vectors and a smaller number of items, you can get away w/eg bytes in on-prem or s3 or dynamo equivalents and lots of caching (roughly for that area when having everything hot in memory dominates NN search cost as opposed to the O(n^2) time blowing up w/a large number of vectors). you can always just link external blob storage and read in your own in-mem data structures to build proper neighborhood trees, or use LSH or supervised hashing, etc. also. in that kind of model, datomic keeps track of everything but you roll your own search, match, etc. the tl;dr is that nothing in datomic is designed to be good at this use case, but it can be handy to give it visibility into what’s getting pushed into other storages. a common pattern I’ve used is — write things to s3 or dynamo w/eg non-conflicting uuid keys and don’t delete (or implement your own garbage collection logic around data dependencies over time). basically extending datomic’s persistent data structure — then write importer logic to consume what’s in datomic as of a particular basis-t as a manifest. you can load out a vector db or warehouse shaped query over large parquet files in s3 or w/e and both write reasonable eventually consistent evolution logic that follows what’s in datomic, and/or project out a warehouse or vector db to site along-side an as-of db to ask historical questions when you need to. (on demand-ish, though of course how expensive this is storage/compute wise depends on your data holdings).

👍 2
gratitude 2
☝️ 2
Sven16:09:40

I ran into an issue with :default option for :db.type/boolean . If I use [:default true] in a pull pattern for :db.type/boolean attribute then false values are returned as true Here is a code snippet with comments demonstrating the issue. 

$ clojure -Sdeps '{:deps {com.datomic/client-cloud {:mvn/version "1.0.123"} com.datomic/dev-local {:mvn/version "1.0.243"}}}'
Clojure 1.10.3
user=>
(require '[datomic.client.api :as d])
(require '[clojure.pprint :refer [print-table]])

(def client (d/client {:server-type :dev-local :system "dev"}))
(def db-name "default-test")
(d/create-database client {:db-name db-name})

(def conn (d/connect client {:db-name db-name}))

(def schema
  [#:db{:ident :foo/name
        :valueType :db.type/string
        :unique :db.unique/identity
        :cardinality :db.cardinality/one}
   #:db{:ident :foo/bar
        :valueType :db.type/boolean 
        :cardinality :db.cardinality/one}])

(d/transact conn {:tx-data schema})

(def data
  [#:foo{:name "A" :bar true}
   #:foo{:name "B" :bar false}
   #:foo{:name "C"}]) ;; <- no :foo/bar

(d/transact conn {:tx-data data})

(def db (d/db conn))

(->> {:query '[:find (pull ?e [:db/id :foo/name :foo/bar])
               :in $
               :where [?e :foo/name]]
      :args [db]}
     d/q
     (map first)
     print-table)

;; |         :db/id | :foo/name | :foo/bar |
;; |----------------+-----------+----------|
;; | 96757023244363 |         A |     true |
;; | 96757023244364 |         B |    false |
;; | 96757023244365 |         C |          |


(->> {:query '[:find (pull ?e [:db/id :foo/name
                               [:foo/bar :default true]]) ;; <-- using default option!
               :in $
               :where [?e :foo/name]]
      :args [db]}
     d/q
     (map first)
     print-table)

;; |         :db/id | :foo/name | :foo/bar |
;; |----------------+-----------+----------|
;; | 96757023244363 |         A |     true |
;; | 96757023244364 |         B |     true | <--- but shouldn't this be false?
;; | 96757023244365 |         C |     true |

(d/pull db '[*] [:foo/name "B"])
;; {:db/id 96757023244364, :foo/name "B", :foo/bar false}

(d/delete-database client {:db-name db-name})
I am wondering that: • is this a bug? • is this a feature but not clear from documentation which states :default option specifies a value to use *if an attribute is not present* for an entity. • am I missing something? https://docs.datomic.com/cloud/query/query-pull.html#default-option

favila16:09:04

s/if-let/if-some/g

✅ 2
Sven16:09:13

Thanks for the reference! How I understand the documentation is that :default is used when attribute is not present but it is. Shouldn’t default value be ignored?

favila16:09:07

sorry that was a sardonic comment that this seems pretty clearly a bug, and the bug is probably someone used if-let instead of if-some

🙈 2
favila16:09:54

there have been many boolean-related datomic bugs fixed in the past which came down to not distinguishing false from falsy

favila16:09:17

honestly I thought this was one of them? maybe it was only in on-prem?

Sven16:09:35

Thanks for the clarification and the insight. We are on cloud. I guess I should report it or maybe datomic team will pick it up here?

favila16:09:50

I would open a support ticket

✅ 4
favila16:09:06

I found this possibly-related entry in the changelog for cloud: https://docs.datomic.com/cloud/changes.html#616-8879

cch117:09:52

Sounds like it was fixed based on that changelog entry. What version of cloud are you using, @U63DGNR2L?

Sven17:09:32

but the example/demo is with datomic local so does the cloud version even matter?

Sven17:09:18

I opened a support ticket

cch118:09:32

It seems the cloud version was fixed some time ago (based on the release notes in the link from @U09R86PA4). If it still misbehaves in the very very new datomic local (not dev-local, presumably) then it’s a bug in local. It would be interesting to see if there has been a reversion in cloud…

cch118:09:48

Link to support ticket? I would like to follow it.

Sven18:09:45

It is an issue in the cloud 990-9202 and also issue w/ datomic local 1.0.243. These are both 1 version behind the latest and according to change log seem to be about datomic going free. > Link to support ticket? I would like to follow it. Request #4007 https://support.cognitect.com/hc/en-us/requests/4007

cch118:09:02

Right… nothing technical seemed to be included. Which means either this is a regression or the changelog from 2020 is a red herring and it addressed a different issue.

jaret19:09:40

This looks like a bug to me. I repro'd based on your support case @U63DGNR2L and have a suspicion Francis & Chris are right. I will discuss with dev and confirm on Monday.

2