Hi. I want setup a large import pipeline to migrate my data from couchdb to datomic. I have a single local server that runs on Linux. Which storage backend should I use ? Postgresql ? Cassandra ? Use free and do a backup restore ?

@fmind Backup/restore is good for moving a datomic database from one underlying storage to another. If you’re taking a database on couchdb and making it a datomic db you’ll want to write an ETL job to take the couchdb dataset from SQL(?) to Datomic. I would recommend that you watch Stu’s video on simplifying ETL with Clojure and Datomic. (https://www.youtube.com/watch?v=oOON--g1PyU).

In terms of choosing your underlying storage they all have their trade offs. I recommend going with the storage you have the most experience with. Many of our users find the experience with AWS DDB to be excellent. Our Storage docs go into detail on the configurations per storage. https://docs.datomic.com/on-prem/storage.html

@jaret thanks for the explanation. I've tried postregsql and cassandra storage already, but the results were not impressive

How do people typically store MD5 hashes in datomic? In Postgres I typically use the UUID type since it is more compact than the string representation. By that same logic it seems that db.type/uuid may be the better option over db.type/string but I'm not sure if any extra JVM considerations (e.g. the size of a UUID in memory vs that of a string) need to be taken into account.

I'll try to build the import pipeline with the backup/restore feature. But I don't like having the transactor and peer library running on two different process ... I use twice the memory for nothing

@fmind there are numerous reasons that Datomic is specifically designed with peers and transactors as separate processes. https://docs.datomic.com/on-prem/deployment.html#process-isolation

For someone at cognitect, I'm getting an Access Denied XML response when I try view the Datomic java docs: https://docs.datomic.com/javadoc/datomic/Connection.html#log()

@adamfrey https://docs.datomic.com/on-prem/javadoc/datomic/Connection.html#log()

Where was that linked from?

https://docs.datomic.com/on-prem/log.html

ok thanks

i’ll fix it

thank you

@adamfrey thanks for catching that. there were a bunch of those throughout the docs. i think i’ve fixed them all; it will take a couple minutes for the changes to percolate

no problem!