Fork me on GitHub
#datomic
<
2017-04-07
>
devth00:04:32

so when i try to connect using a sql connection string, the peer: - hits storage - determines the configured host and alt-host - peer tries to hit host or alt-host to talk to the transactor is that right? wondering how i'm going to connect to my in-cluster datomic from my dev machine when its dns isn't accessible from outside.

favila01:04:48

@devth yes that is right. There is always /etc/hosts, vpn, ssh-tunnel, etc

devth12:04:25

thanks. yep i'm sure i can make it work.

eraserhd13:04:31

Is there a way to extract all current (non-retracted) datoms without getting all of them, comparing timestamps and added? flags, and building a model of the database?

eraserhd13:04:25

(essentially a current-state dump to EDN)

augustl13:04:46

have you looked into the index access APIs?

eraserhd13:04:01

Also, is there a difference between :eavt and :aevt?

eraserhd13:04:31

Do you mean like d/datoms or is there something else?

augustl13:04:56

yeah, that one

eraserhd13:04:57

Ohh.. perhaps you just don’t see those on a non-`history` database?

augustl13:04:12

yeah exactly

augustl13:04:26

if you use a normal db, eavt is exactly what you're looking for

augustl13:04:01

ref "building a model for the database" - not sure what you're planning there, but I would just dump plain facts, no structured entities etc

eraserhd13:04:48

Yeah, that works.

augustl13:04:05

the difference between eavt and aevt is just the access pattern

eraserhd13:04:16

It looks like the first few are system facts that I don’t need to copy out. Is there an easy way to detect that?

augustl13:04:41

you can look at the partition they're in

eraserhd13:04:03

User attributes are added to the db partition, too, right? So that doesn’t really tell me.

eraserhd13:04:17

Oh, wait. Maybe not.

augustl13:04:30

hmm, good point, not sure

eraserhd13:04:17

I could create an empty database and remove common facts. Assuming eids of the system things never change, I should the be able to transact that to a different database (in a test environment).

eraserhd13:04:04

Oh, the db.install stuff makes that harder.

augustl13:04:15

you could look up the transaction that wrote those entities and see if it has something interesting on it

augustl13:04:31

perhaps they're all from the same transaction, and that transaction has something you can use to identify that it is the "boostrap" tx

augustl13:04:35

and then ignore all facts from that tx

eraserhd13:04:02

It seems there’s more than one.

augustl13:04:17

I would guess that they have something in common. Buy maybe not

eraserhd13:04:45

Oh, perhaps it’s two. The first seems to make the :db.part/db partition.

eraserhd14:04:47

Well, it’s greater than two, which is effectively unbounded 😄

favila14:04:36

This is a bit old now, but some of the gotchas about the "dump a restorable db to edn" are still there

favila14:04:47

i.e. are explained and accounted for

favila14:04:10

e.g. the one you mentioned, that there are bootstrap datoms you should not keep/restore

favila14:04:42

also this is tuned for memory dbs back when they did not have logs. Nowdays if you want to keep full history it's better to use the tx log directly

eraserhd14:04:05

so far… the bootstrap txs I have are > 1000.

favila14:04:27

that is the rule

favila14:04:29

if id < 1000, is a bootstrap datom

favila14:04:08

more precisely, if the t of an id is < 1000

eraserhd14:04:23

I mean, its not true for me:

(first (d/datoms (db) :eavt))
#datom[0 10 :db.part/db 13194139533312 true]

favila14:04:41

tx of that datom is < 1000

favila14:04:02

(d/tx->t 13194139533312) => 0

favila14:04:47

in that gist I compute the max tx

favila14:04:59

(def ^long ^:const min-user-tx (long (d/t->tx 1000)))

favila14:04:10

Then filter out all datoms with a tx lower than that

favila14:04:40

(defn bootstrap-datom? [{^long tx :tx}]
  ;; Datoms in a transaction with t < 1000 are from a bootstrap transaction and
  ;; cannot be reissued.
  (< tx min-user-tx))

eraserhd14:04:01

not sure what I’m going to do, but this is useful. Thanks!