datalevin 2022-02-15 | Slack Archive

spieden00:02:40

Hello! Just migrated a project to datalevin from datahike and have really been enjoying it. Hit a bit of a puzzle today on a query that doesn't work as it did before and narrowed things down to this:

checkv.db=> (def status (-> (d/pull @(conn) [:item/status] 8188) :item/status))
#'checkv.db/status
checkv.db=> status
0
checkv.db=> (type status)
java.lang.Integer
checkv.db=> (d/q {:find '[?i] :where [['?i :item/status status]]} @(conn))
#{}

Any idea why the query doesn't match anything? At first I thought it might be a type thing (long vs int), so I'm querying with the exact value returned by pull. This attr doesn't have any schema, FWIW.

chromalchemy19:02:00

Why migrating? just curious…

spieden09:02:49

Datahike started pegging my CPU for some reason when doing big transactions. Never got around the odd datalevin behavior I detailed in the channel with entities simply not resolving after I'd loaded a certain amount of data. Tried datomic next but it said my transactions were too big! Just finished converting to XTDB and all is well. All of these projects are great, but my time is really limited and I just need a DB that works without having to debug. 🙂

spieden09:02:49

Here's the project if you want to see the progression: https://github.com/spieden/checkv-gtg

spieden01:02:48

Hmm, tried dumping and then loading just the schema and relevant entity and repro goes away

spieden01:02:26

Full dump and reload doesn't repro either, so seems specific to the state of my LMDB

spieden01:02:39

Ok, seems to repro when I blow away my DB and reconstruct the same state via individual transactions

spieden01:02:01

I'll try to scrub the data so I can post an LMDB that repros if that helps

spieden20:02:57

I hit an unrelated issue on another project I migrated. Seems like entity ID is maxing out silently at 536871692 and all my tempids start resolving to it. All the EIDs are imported from an old DB so not sure how they got this high. Seems like this number is quite close to the :max-tx value shown? Anyway, if some limit were hit I'd expect it to complain instead of piling everything onto one entity like this:

❯ clojure -Sdeps '{:deps {datalevin/datalevin {:mvn/version "0.5.28"}}}' -M:repl/rebel
...
user=> (def c (d/get-conn "repro.dtlv"))
user=> (:tx-data (d/transact! c [[:db/add -1 :foo :baz]]))
[#datalevin/Datom [536871692 :foo :baz]]
user=> (:tx-data (d/transact! c [[:db/add -1 :foo :baz]]))
[]
user=> (:tx-data (d/transact! c [[:db/add -1 :foo :bar]]))
[#datalevin/Datom [536871692 :foo :baz] #datalevin/Datom [536871692 :foo :bar]]
user=> c
#atom[#datalevin.db.DB{:store #object[datalevin.storage.Store 0x22846025 "datalevin.storage.Store@22846025"], :eavt #{#datalevin/Datom [536871692 :foo :bar]}, :avet #{#datalevin/Datom [536871692 :foo :bar]}, :veat #{}, :max-eid 536871691, :max-tx 536870915, :hash #atom[0 0x30df67b8]} 0x58d8059b]

spieden21:02:47

Doesn't happen when I put that EID into an empty DB:

❯ clojure -Sdeps '{:deps {datalevin/datalevin {:mvn/version "0.5.28"}}}' -M:repl/rebel
user=> (require '[datalevin.core :as d])
nil
user=> (def c (d/get-conn nil))
#'user/c
user=> (d/transact! c [[:db/add 536871691 :foo :bar]])
{:datoms-transacted 1}
user=> (:tx-data (d/transact! c [[:db/add -1 :foo :baz]]))
[#datalevin/Datom [3 :foo :baz]]
user=> (:tx-data (d/transact! c [[:db/add -1 :foo :baz]]))
[#datalevin/Datom [4 :foo :baz]]

spieden21:02:13

Anyway, I'll get repros going for these with fixtures and file as bugs

2022-02-15

Channels