Fork me on GitHub
#datomic
<
2021-10-04
>
Ivar Refsdal10:10:46

Should alt-host in transactor.properties also be identical in a HA setting?

Ivar Refsdal10:10:39

I'm guessing not..

Ivar Refsdal10:10:50

To enable HA, simply run two transactor processes on separate machines, with identical transactor properties files (in particular, configured to use the same storage.)
Quote from https://docs.datomic.com/on-prem/operation/ha.html

Linus Ericsson13:10:17

The host and alt-host should point to the IP of that currently running transactor. Usually it is some public IP and an internal IP in them. Two HA transactors should have different IP:s. The IP:s are written to the back storage by the transactor, and the connected peers deduce which transactor that is the active one from some protocol based on the transactors writing heartbeats to the back storage. And uses the IP:s from the backstore to connect to the active transactor. The backstore serves as a "transactor service discovery" as well as some kind of liveness check which implicitly decides which transactor that is really alive.

Linus Ericsson13:10:44

(by "HA transactor" I mean two or more transactors running against the same backstorage, which gives high availiability to the transactor service).

Ivar Refsdal15:10:36

OK, thanks! As far as I understand then the documentation regarding identical transactor properties files is then incorrect. I think that this should be corrected.

Ivar Refsdal19:10:54

I am encountering a strange OutOfMemoryError for a relatively simple query. I don't think it creates a cross product. Any suggestions on why this would happen? https://gist.github.com/ivarref/7059a0fe79b3353187dad9e187928da0.

Joe Lane19:10:52

Hi @UGJE0MM0W , your query clauses can be thought of as executing these steps: 1. Find all transactions in the entire system which have the attribute :tx/user-id, and load them all into memory. 2. Of all of the txes with the attribute :tx/user-id, grab the :db/txInstant from each transaction. 3. Filter the transactions realized in step 1 and 2 by whether they started after today in Oslo 4. Filter the remaining transactions from Step 3 by whether they happen before tomorrow. 5. Count the remaining txes I suspect your query is failing on Step 1 because it is querying all txes in the database. (and if it doesn't fail yet, it will get slower until it fails as more data is added to the system).

🧡 1
Ivar Refsdal19:10:53

Thanks a lot. I will read more thorough tomorrow.

Ivar Refsdal08:10:40

I noticed that if I change my query from:

[[?tx :tx/user-id _]
 [?tx :db/txInstant ?inst]
 ...
to
[[?tx :db/txInstant ?inst]
 [?tx :tx/user-id _]
 ...
it does not OOM. Why would that be? For the record there is about 12M datoms/transactions with the :tx/user-id attribute. I figured this would fit easily in a ~4G heap? What exactly is pulled into memory in step 1? Only the datom matching ?tx :tx/user-id _ or the whole entity, i.e. pull [:*] ?tx? My guess is that the OOM in the first query is somehow related to the fact that :tx/user-id is not indexed, but :db/txInstant is. Is that a correct understanding? Note: I am mostly interested in understanding why the OOM happens, not solving the problem per se. Thanks again!