@robert-stuttaford: I decided to rewrite it mostly so that I understand it better

When restoring, I'm using a two-phase approach to deal with tempids. First I safe bogus refs, then I fix them to point to the new entities.

Hi, is it possible given a connection to restore an in-memory db to db from an earlier point in time? For test purposes (using conformity for migrated a DB to the desired version, but it's too slow for each test we have)

I'm running a datomic instance and a simple jetty/ring web server on a 2GB RAM instance, but it is consuming about 1.5GB RAM. The database is pretty empty, maybe 1000 records or so. Is this normal?

does datomic consume that much memory out of the box or am I doing something wrong in my server?

I just attach to the db at every function call, for example, I have the following db queries: https://github.com/YoungTurks/hackerdict/blob/master/src/hackerdict/db.clj

I have also experimented with storing the connection in an atomic

but then reverted back to using a function call to get a brand new one each time.

what is the ideal way to keep a connection to the db?

@jonpither: interesting problem

Any ideas, I'm stumped

so you have a storage-backed db and you want to take an in-mem db pointing to a particular point in that storage db for tests to use?

Looking at excision also, but that seems non-trivial

no it's not storaged backed

what form does your test data take when your tests aren’t running? transactions in edn?

It's a pure in-memory DB, the problem is the cost of creating a new DB for each test (running schema migrations) is around ~ .5 sec

i assume you transact in your tests?

Yes

the migrations use plugged in with https://github.com/rkneufeld/conformity

yeah we use that too

ideally, given a freshly created in-mem DB, I'd like to shove it back down the connection, saying "revert/restore yourself to this"

how many datoms in the test database post schema?

not a great deal

(just rocked on to this codebase)

one approach is to use d/with instead of d/transact

but that means alterations to production code which sucks

you can’t rewind datomic databases like that. you can assert mirror transactions in reverse order, but that’s probably going to be slower than simply remaking it

you could also make the db, save off the datoms (out of :EAVT) and simply transact the whole lot into a new in-mem db for each test

yeah the latter sounds better

is there a resource you could point me at?

to save off the datoms, or to build a tx based on them?

given I have a freshly created / migrated DB with some initial seed data, can I extract all the txs from this necessary for building up a new db?

(vec (for [[e a v] (d/datoms db :eavt)] [:db/add e (d/ident db a) v]))

you’d have to filter out all the stuff present in every datomic database, so a’s with :db/ or :fressian/ as a namespace

I'll give this a go, thanks Bob, be interesting to see if it's faster

should be, you’d get full persistent data-structure advantages

@jonpither: untested, but i’m fairly sure this’ll work:

(vec (for [[e a v] (d/datoms db :eavt)
           :let [a (d/ident db a)]
           :when (not (-> a namespace #{"db" "db.install" "fressian"}))]
       [:db/add e a v]))

that should be empty for a completely new database

nice - legend

of course, you can optimise further; keep the db around too because you can use it for any tests that don’t transact anything (as few as those might be)

@jonpither: one small fix to the snippet above

I see namespaces such as db.install

😁

`(defn- db-creation-datoms
  [db]
  (vec (for [[e a v] (d/datoms db :eavt)
             :let [a (d/ident db a)]
             :when (not (or (-> a namespace #{"db" "fressian"})
                            (.startsWith (namespace a) "db.")))]
         [(namespace a) :db/add e a v])))

@jonpither, i’m a dork. i’ve completely ignored that transacting requires tempids, so you’ll need to post-process the e and v values in that output to replace all the ids with tempids, and you’ll need to make sure that schema gets db partition tempids

ah yeah

not insurmountable, and it’s work that would only happen once at the start of the test run, so still worth trying

just not a 4 line code snippet any more, heh

good point

programming!

good luck. must run

thanks Bob

I'll let you know of my travails

Hi, if transact a few entities using tempids, is there a way to get the entity ids they resolved to from the tx-data?

returned from transact

or should i just query the DB after the transaction, i'm interested in a subset of the new entity ids

the entity ids for some of the newly transacted entities

@currentoor: They are returned from transact in the map. They’re in the :tempids entry of the map.

http://docs.datomic.com/clojure/#datomic.api/transact

Hello @marshall!

Looking forward to talking to you this afternoon.

@currentoor: You can also use http://docs.datomic.com/clojure/#datomic.api/resolve-tempid

@currentoor: Likewise

resolve-tempid was exactly what I needed

thanks!

sure

As I understand it, you must use d/resolve-tempid to correctly resolve tempids, you can't just get them from the map returned by d/transact.