This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-09-12
Channels
- # ai (1)
- # announcements (7)
- # babashka (32)
- # beginners (23)
- # biff (9)
- # calva (1)
- # cljs-dev (13)
- # clojure (32)
- # clojure-belgium (1)
- # clojure-chicago (15)
- # clojure-europe (24)
- # clojure-india (3)
- # clojure-nl (3)
- # clojure-norway (55)
- # clojure-uk (4)
- # clojurebridge (1)
- # clojurescript (5)
- # core-async (17)
- # data-science (9)
- # datomic (29)
- # events (3)
- # fulcro (16)
- # graalvm-mobile (4)
- # helix (15)
- # hyperfiddle (74)
- # introduce-yourself (1)
- # jobs (4)
- # kaocha (12)
- # leiningen (27)
- # lsp (16)
- # shadow-cljs (6)
- # spacemacs (20)
- # sql (27)
- # squint (7)
- # tools-deps (29)
- # vim (2)
- # xtdb (10)
I’ve restored data from an older dev instance of datomic-pro 1.0.6165
in into a new dev instance 1.0.6735
and am querying it from the new peer library com.datomic/peer 1.0.6735
. I’m just now discovering that a reverse pull is silently failing to return results:
old using com.datomic/datomic-pro
:
(d/pull db '[:projectslookup/_Stations] 285873023368601)
=> #:projectslookup{:_Stations [#:db{:id 285873023362253}]}
new using com.datomic/peer
:
(d/pull db '[:projectslookup/_Stations] 285873023368601)
=> nil
both:
(d/q '[:find ?e . :where [?e :projectslookup/Stations 285873023368601]]
db)
=> 285873023362253
I’ve tested this on both Intel and Apple Silicon MacBook Pros. Has anyone else seen this problem? Does the new peer library not support reverse lookups? I’ve turned up datomic logging output and am seeing no warnings.
I created a new DB, added some new schemas and test data, and reverse refs work fine with new data, so there’s something wrong with the restored data that’s causing reverse ref to fail.Have you tried selecting something? This is relying on “select :db/id of refs by default” behavior which maybe changed
I don't understand what you mean by selecting something? The same thing occurs if the pull is a part of a larger query:
(d/q '[:find [(pull ?e [:db/id :projectslookup/_Stations]) ...]
:where
[?e :stationlookup/uuid]]
(db))
old:
[{:db/id 17592186178123, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
{:db/id 17592186178124, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
{:db/id 17592186178125, :projectslookup/_Stations [#:db{:id 17592186178014} #:db{:id 285873023368071}]}
...]
new:
[#:db{:id 17592186178123}
#:db{:id 17592186178124}
#:db{:id 17592186178125}
...]
Maybe there's a setting that needs to be set in order to build these reverse indexes?
(d/pull (db) '[{:projectslookup/_Stations [:db/id]}] 285873023368601)
old:
=> #:projectslookup{:_Stations [#:db{:id 285873023362253}]}
new:
=> nil
To figure out if this is a data difference or a pull behavior difference you should use d/datoms directly and compare
The indexes look exactly the same in both instances:
(for [d (d/datoms (db) :vaet 285873023368601)]
[(:e d) (d/ident (db) (:a d)) (:v d)])
both:
(...
[285873023362253 :projectslookup/Stations 285873023368601])
query:
(d/pull (db) '[{:projectslookup/_Stations [:db/id :projectslookup/Name]}] 285873023368601)
old:
=> #:projectslookup{:_Stations [{:db/id 285873023362253, :projectslookup/Name "Deer Creek"}]}
new:
=> nil
I discovered this because a complex app was not working right, and the app has many uses of reverse lookups, none of which are working
I would make a minimum reproducible without any deps on your app and file a support ticket
Yeah I'll try creating a new database from scratch and will see if that also has the problem. Maybe it has something to do with the backup/restore process
If d/datoms produces the same result I can’t imagine how the backup/restore could be involved
yes the :eavt is the same in both.
(for [d (d/datoms (db) :eavt 285873023362253)]
[(:e d) (d/ident (db) (:a d)) (:v d)])
both:
(...
[285873023362253 :projectslookup/Stations 285873023368601]
...)
With the same running sytem using the same datomic:dev storage, using the same running REPL, I created a new database and added some basic schema with a ref many, and the reverse lookup pull works normally.
It seems like the reverse lookups use some kind of magic under the hood that's not getting wired up with the restore
Here's the test I just made that's working as expected in the same REPL:
(d/pull (hmmdb) '[*] 17592186045418)
=> {:db/id 17592186045418, :book/name "foo", :book/authors [#:db{:id 17592186045419} #:db{:id 17592186045420}]}
(d/pull (hmmdb) '[*] 17592186045419)
=> {:db/id 17592186045419, :author/name "joe"}
(d/pull (hmmdb) '[* :book/_authors] 17592186045419)
=> {:db/id 17592186045419, :author/name "joe", :book/_authors [#:db{:id 17592186045418}]}
I guess I'll need to write a better export/import using datoms. This will be useful to move my dev DB to cloud
@U07KXN95L When you say you restored an older dev DB is it possible for you to give me the restore file or work with me over a support case to get some information? I'd like to investigate this further based on what you have already observed.
yes, I’d be happy to send you the backup file. I’ve narrowed it down to the problem started with version 1.0.6527
. It works on 1.0.6397
and if I do either a backup/restore or copy the data folder from 1.0.6397
to 1.0.6527
it doesn’t work.
Ah, I see that makes sense. So, I'm unsure how to work around this because I'm using a system that uses reverse lookups and it's not something I can easily change to use a different lookup query. But maybe I can somehow edit the history. I know I've seen a library that can do a migration of key names and even data types using some kind of history editing. Thanks for finding the clue!