Fork me on GitHub
#datomic
<
2023-09-12
>
thosmos22:09:37

I’ve restored data from an older dev instance of datomic-pro 1.0.6165 in into a new dev instance 1.0.6735 and am querying it from the new peer library com.datomic/peer 1.0.6735. I’m just now discovering that a reverse pull is silently failing to return results: old using com.datomic/datomic-pro:

(d/pull db '[:projectslookup/_Stations] 285873023368601)
=> #:projectslookup{:_Stations [#:db{:id 285873023362253}]}
new using com.datomic/peer:
(d/pull db '[:projectslookup/_Stations] 285873023368601)
=> nil
both:
(d/q '[:find ?e . :where [?e :projectslookup/Stations 285873023368601]]
  db)
=> 285873023362253
I’ve tested this on both Intel and Apple Silicon MacBook Pros. Has anyone else seen this problem? Does the new peer library not support reverse lookups? I’ve turned up datomic logging output and am seeing no warnings. I created a new DB, added some new schemas and test data, and reverse refs work fine with new data, so there’s something wrong with the restored data that’s causing reverse ref to fail.

favila23:09:15

Have you tried selecting something? This is relying on “select :db/id of refs by default” behavior which maybe changed

thosmos00:09:25

I don't understand what you mean by selecting something? The same thing occurs if the pull is a part of a larger query:

(d/q '[:find [(pull ?e [:db/id :projectslookup/_Stations]) ...]
       :where
       [?e :stationlookup/uuid]]
  (db))
old:
[{:db/id 17592186178123, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
 {:db/id 17592186178124, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
 {:db/id 17592186178125, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
...]
new:
[#:db{:id 17592186178123}
 #:db{:id 17592186178124}
 #:db{:id 17592186178125}
...]

favila00:09:36

[{:projectslookup/_Stations [:db/id]}]

thosmos00:09:46

oh i see, no there are still no results

thosmos00:09:35

Maybe there's a setting that needs to be set in order to build these reverse indexes?

favila00:09:22

You can try d/datoms directly to see if they are there

favila00:09:43

:vaet and :eavt on the entity you expect

thosmos00:09:49

(d/pull (db) '[{:projectslookup/_Stations [:db/id]}] 285873023368601)
old:
=> #:projectslookup{:_Stations [#:db{:id 285873023362253}]}
new:
=> nil

favila00:09:24

To figure out if this is a data difference or a pull behavior difference you should use d/datoms directly and compare

thosmos00:09:18

The indexes look exactly the same in both instances:

(for [d (d/datoms (db) :vaet 285873023368601)]
  [(:e d) (d/ident (db) (:a d)) (:v d)])
both:
(...
 [285873023362253 :projectslookup/Stations 285873023368601])

favila00:09:20

So this is some difference with pull

favila00:09:32

Did you try pulling anything other than db/id?

thosmos00:09:23

query:

(d/pull (db) '[{:projectslookup/_Stations [:db/id :projectslookup/Name]}] 285873023368601)
old:
=> #:projectslookup{:_Stations [{:db/id 285873023362253, :projectslookup/Name "Deer Creek"}]}
new:
=> nil

thosmos00:09:04

I discovered this because a complex app was not working right, and the app has many uses of reverse lookups, none of which are working

favila00:09:01

I would make a minimum reproducible without any deps on your app and file a support ticket

thosmos00:09:21

Yeah I'll try creating a new database from scratch and will see if that also has the problem. Maybe it has something to do with the backup/restore process

favila00:09:36

If d/datoms produces the same result I can’t imagine how the backup/restore could be involved

favila00:09:20

Did you see if the eavt of the other entity is the same in both?

thosmos00:09:19

yes the :eavt is the same in both.

(for [d (d/datoms (db) :eavt 285873023362253)]
  [(:e d) (d/ident (db) (:a d)) (:v d)])
both:
(...
 [285873023362253 :projectslookup/Stations 285873023368601]
...)

thosmos00:09:59

With the same running sytem using the same datomic:dev storage, using the same running REPL, I created a new database and added some basic schema with a ref many, and the reverse lookup pull works normally.

thosmos00:09:17

It seems like the reverse lookups use some kind of magic under the hood that's not getting wired up with the restore

thosmos00:09:12

Here's the test I just made that's working as expected in the same REPL:

(d/pull (hmmdb) '[*] 17592186045418)
=> {:db/id 17592186045418, :book/name "foo", :book/authors [#:db{:id 17592186045419} #:db{:id 17592186045420}]}

(d/pull (hmmdb) '[*] 17592186045419)
=> {:db/id 17592186045419, :author/name "joe"}

(d/pull (hmmdb) '[* :book/_authors] 17592186045419)
=> {:db/id 17592186045419, :author/name "joe", :book/_authors [#:db{:id 17592186045418}]}

thosmos01:09:08

I guess I'll need to write a better export/import using datoms. This will be useful to move my dev DB to cloud

jaret13:09:25

@U07KXN95L When you say you restored an older dev DB is it possible for you to give me the restore file or work with me over a support case to get some information? I'd like to investigate this further based on what you have already observed.

thosmos16:09:39

yes, I’d be happy to send you the backup file. I’ve narrowed it down to the problem started with version 1.0.6527. It works on 1.0.6397 and if I do either a backup/restore or copy the data folder from 1.0.6397 to 1.0.6527 it doesn’t work.

thosmos19:10:09

Ah, I see that makes sense. So, I'm unsure how to work around this because I'm using a system that uses reverse lookups and it's not something I can easily change to use a different lookup query. But maybe I can somehow edit the history. I know I've seen a library that can do a migration of key names and even data types using some kind of history editing. Thanks for finding the clue!

jaret22:10:53

I think you might be able to excise the old attribute, assuming you do not care about maintaining the history of this change. I can test tomorrow and then validate my suspicion with the team.