This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-04
Channels
- # babashka (17)
- # beginners (82)
- # calva (42)
- # clj-commons (9)
- # cljdoc (2)
- # cljsrn (3)
- # clojure (142)
- # clojure-europe (12)
- # clojure-nl (1)
- # clojure-sg (1)
- # clojure-uk (14)
- # clojurescript (22)
- # community-development (3)
- # cryogen (12)
- # cursive (15)
- # data-science (13)
- # datomic (11)
- # deps-new (8)
- # emacs (3)
- # fulcro (31)
- # gratitude (7)
- # holy-lambda (8)
- # honeysql (6)
- # introduce-yourself (1)
- # jackdaw (11)
- # jobs-discuss (7)
- # kaocha (1)
- # malli (8)
- # other-languages (9)
- # pathom (14)
- # pedestal (1)
- # polylith (3)
- # portal (12)
- # re-frame (3)
- # react (3)
- # reagent (4)
- # releases (3)
- # reveal (7)
- # ring (11)
- # shadow-cljs (17)
- # specter (3)
- # sql (1)
- # timbre (2)
- # tools-deps (122)
- # xtdb (18)
Should alt-host
in transactor.properties also be identical in a HA setting?
I'm guessing not..
To enable HA, simply run two transactor processes on separate machines, with identical transactor properties files (in particular, configured to use the same storage.)
Quote from https://docs.datomic.com/on-prem/operation/ha.htmlThe host and alt-host should point to the IP of that currently running transactor. Usually it is some public IP and an internal IP in them. Two HA transactors should have different IP:s. The IP:s are written to the back storage by the transactor, and the connected peers deduce which transactor that is the active one from some protocol based on the transactors writing heartbeats to the back storage. And uses the IP:s from the backstore to connect to the active transactor. The backstore serves as a "transactor service discovery" as well as some kind of liveness check which implicitly decides which transactor that is really alive.
(by "HA transactor" I mean two or more transactors running against the same backstorage, which gives high availiability to the transactor service).
OK, thanks! As far as I understand then the documentation regarding identical transactor properties files is then incorrect. I think that this should be corrected.
I am encountering a strange OutOfMemoryError
for a relatively simple query.
I don't think it creates a cross product.
Any suggestions on why this would happen?
https://gist.github.com/ivarref/7059a0fe79b3353187dad9e187928da0.
Hi @UGJE0MM0W , your query clauses can be thought of as executing these steps:
1. Find all transactions in the entire system which have the attribute :tx/user-id
, and load them all into memory.
2. Of all of the txes with the attribute :tx/user-id
, grab the :db/txInstant
from each transaction.
3. Filter the transactions realized in step 1 and 2 by whether they started after today in Oslo
4. Filter the remaining transactions from Step 3 by whether they happen before tomorrow.
5. Count the remaining txes
I suspect your query is failing on Step 1 because it is querying all txes in the database. (and if it doesn't fail yet, it will get slower until it fails as more data is added to the system).
Check out the https://docs.datomic.com/on-prem/api/log.html#log-in-query example on the https://docs.datomic.com/on-prem/api/log.html page, hope that helps!
Thanks a lot. I will read more thorough tomorrow.
I noticed that if I change my query from:
[[?tx :tx/user-id _]
[?tx :db/txInstant ?inst]
...
to
[[?tx :db/txInstant ?inst]
[?tx :tx/user-id _]
...
it does not OOM. Why would that be?
For the record there is about 12M datoms/transactions with
the :tx/user-id
attribute. I figured this would fit easily in a ~4G heap?
What exactly is pulled into memory in step 1?
Only the datom matching ?tx :tx/user-id _
or the whole entity, i.e. pull [:*] ?tx
?
My guess is that the OOM in the first query is somehow related to the fact
that :tx/user-id
is not indexed, but :db/txInstant
is. Is that a correct
understanding?
Note: I am mostly interested in understanding why the OOM happens, not solving
the problem per se.
Thanks again!