Clojurians Log v2

How have you all handled the process of "transferring" one Datomic database into another? My situation is thus: - I have an application database - There are some flat files out there in the world which I need to do an ETL process on in order to import the data contained therein into our database. - To do this, I create a temporary Datomic database with the same schema as the app database, perform the ETL on the flat file, and transact it into the temp db. The temp db is necessary because there's a lot of garbage in these flat files, and the user needs to pick and choose what they actually want to keep from the import. All of the above works pretty well. The issue I'm having is how to take those selections and efficiently process them and bring them over to the app DB. Naievely, a query + a pull is a great way to go. Pull gives you something that's a really close analog for a Datomic transaction (reverse references excluded):

(->>
 (d/query {:query '{:find [?e]
                    :in [$ ?uuid]
                    :where [[?e :unique/id ?uuid]]}
           :args [tmpdb uuids]})

 (d/pull-many tmpdb [:db/id
                     :person/name
                     {:person/_friend [*]}]))

;; This comes out of the DB, AND could theoretically just be transacted as is, excepting the reverse reference problem
[{:db/id "aaa"
  :person/name "Joey"
  :person/_friend {:person/name "Andy"
                   :db/id "bbb"
                   :person/friend "aaa"}}

 {:db/id "bbb"
  :person/name "Andy"
  :person/_friend {:person/name "Joey"
                   :db/id "aaa"
                   :person/friend "bbb"}}]

But a problem instantly arises in that the "transaction-version" of what you've pulled would duplicate any entity refs that are in that pull. So the other option that arises is to query for a flat list of entities, which ensures that no duplication occurs, and stringify the DB refs in order to establish those ref relationships in the app DB:

(->> (d/query {:query '{:find [?e]
                        :in [$ ?uuid]
                        :where [[?e :unique/id ?uuid]]}
               :args [tmpdb uuids]})
     (d/pull-many tmpdb [*]))

;; This comes out of the DB query
[{:db/id "aaa"
  :person/name "Joey"
  :person/friend {:db/id "bbb"}}

 {:db/id "bbb"
  :person/name "Andy"
  :person/friend {:db/id "aaa"}}]

;; After a little processing, this is the transaction:
[{:db/id "aaa"
  :person/name "Joey"
  :person/friend "bbb"}

 {:db/id "bbb"
  :person/name "Andy"
  :person/friend "aaa"}]

But as the ref relationships get deeper, more complex, more intertwined, I'm a little concerned about writing a bunch of bespoke queries for individual entity types in order to achieve this flat list. The "pull everything and rely on Datomic's ability to take a map of nested maps and run THAT as the transaction" is super appealing in contrast, but comes with the duplication problem.