This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-25
Channels
- # announcements (14)
- # aws (1)
- # babashka (23)
- # beginners (442)
- # calva (50)
- # chlorine-clover (1)
- # cider (32)
- # clojure (124)
- # clojure-europe (35)
- # clojure-france (5)
- # clojure-gamedev (5)
- # clojure-nl (2)
- # clojure-portugal (3)
- # clojure-uk (4)
- # clojurescript (56)
- # conjure (5)
- # cursive (24)
- # datalevin (1)
- # datomic (57)
- # fulcro (35)
- # helix (15)
- # holy-lambda (8)
- # introduce-yourself (1)
- # jobs (5)
- # kaocha (1)
- # lsp (99)
- # malli (10)
- # music (1)
- # off-topic (22)
- # pathom (38)
- # podcasts-discuss (10)
- # polylith (10)
- # reitit (1)
- # releases (1)
- # remote-jobs (4)
- # shadow-cljs (18)
- # spacemacs (6)
- # tools-build (22)
- # vim (66)
- # xtdb (22)
With Datomic On-Prem transactor deployed on AWS in a high-availability setup (one active + one standby transactor), what is the simplest way of identifying which instance currently is active and which is standby? I’ve found the instance IP in the transactor logs emitted at startup time, but that’s unwieldy.
Hi @UFR3C1JBU the transactor logs will contain a log line lifecycle event with the status standby
i.e.
2017-10-19 10:13:26.532 INFO default datomic.lifecycle-ext - {:event :transactor/standby, :rev 1238602, :missed 1, :timestamp 1508407998353, :pid 2604, :tid 20}
I see that datomic transactor on-prem bundles org.postgresql/postgresql "9.3-1102-jdbc41"
, released on Jul 18, 2014.
Is that also the recommended PostgreSQL driver for peers?
Why such an old release?
At my company we are connecting to PostgreSQL 11
I’m having to answer questions about when we apply patches for our infra. When are the EC2 instances that back QGs replaced, is it only during Ions upgrades? I gather it’s not when we deploy. How often is the AMI updated? Have I missed this info in the docs?
Hi @U0VP19K6K, every time you deploy you will cycle the instances and install your ion code on the instances. However, the instances AMI is tied to the CFT version they are on. So it depends on what you define as applying patches. Does that answer your question?
I just tried to upgrade an existing datomic production
system from 715-8973
to 884-9095
https://docs.datomic.com/cloud/changes.html#884-9095 and got this error while upgrading the storage:
> UPDATE_FAILED The following resource(s) failed to update: [EnsureAdminPolicyLogGroup].
My stack has a root stack with the name of my system and two childs: compute & storage. I just found this ticket where the solution was to split the stack: https://forum.datomic.com/t/failed-production-storage-update-to-884-9095/1947
So, should I also split our stack and try upgrading again?
hi @U012ADU90SW yes, you will want to perform a split stack operation to get off the Marketplace template. From there on in you will be able to update the individual stacks depending on what has been released.
If you encounter any issues please let me know directly or e-mail us at support, <mailto:[email protected]|[email protected]>.
Trying to learn more about Datomic indexes.. I seems “covering indexes” means that the indexes are actually full copies of the datoms but sorted in different E-A-V ways. It seems the four indexes are stored in Amazon S3 - but are they copied to each peer also? I’d like to learn more about both indexes and caching - in general, and Datomic specifically. Anyone know what are the best sources for such information?
it depends on whether you’re using Cloud or On-Prem, but either way the docs is a good place to start: • https://docs.datomic.com/cloud/whatis/architecture.html • https://docs.datomic.com/on-prem/overview/architecture.html
Other than the docs I should have said..
Peers pull segments (blocks of sorted datoms) down as they need them from storage or one of the caching laters in front of storage. The unindexed portion of change is kept in memory on all peers
a reindex incorporates the unindexed portion into a new full index, updates the pointer to the root of the segment tree, and the cycle begins again.
Ah, that great article again of course!
From what I understand all indexes are in both main storage and local storage/cache. If I’m correct about that I wonder what the reason behind this is - why wouldn’t it be enough to create the indexes only locally?
Only the ”original data” (the ”log”?) is local and the four indexes are transferred to peers when queries need them?
the log tail (that hasn’t been incorporated into the stored indexes) is kept in memory, and the indexes of it are kept in memory. The rest is fetched from storage as needed and merged with these to produce a complete view.
I wonder what might be the disadvantage of having the four indexes fully represented on each peer..
Hello!
I am trying to learn more about the transactions functions, but I am having a little difficult on understanding how to install a transaction function.
Is it possible to have the transaction function implemented by two or more functions instead of having all the code inside code
keyword? For example:
(defn other-func-2 [] <all-code-here>)
(defn other-func-1 [] (let [foo (other-func2)] <all-code-here> ))
#db/fn {:lang :clojure
:params [db offer]
:code (other-func-1)}
Instead of
#db/fn {:lang :clojure
:params [db entity]
:code <all-code-here>}
The problem is the environment isn’t shared among all peers. You can either install the code into the database itself, in which case it can only reference things you know all peers have in their environment (that includes the database itself--you can use d/invoke
to invoke other db functions).
or you can ignore all this installation stuff and just put the functions into the transactor’s classpath. https://docs.datomic.com/on-prem/reference/database-functions.html#classpath-functions
Hey Favila, thank you for the answer!! 1 - I think I didn't understand how to do that... is there any example of install code into the database itself? Because I thought I was doing it when I install the db/fn. 2 - About the transactor's classpath: Do I need to generate my code as a lib in order to add it to the classpath? 3 - Do you know any example using d/with ? I didn't understand how it is related to the other options
That link documents two different things: 1. putting executable code as data into the db; 2. calling a “normal” function from transaction data.
Talking about (2) first. On the transactor, you make functions available by including a jar with that code in it on startup
export DATOMIC_EXT_CLASSPATH=mylibs/mylib.jar
bin/transactor my-config.properties
Then you can use it anywhere in the transactor process by using the symbol name, e.g. attribute predicates, entity predicates, as a tx fn, or inside a query running in any of those.
in transaction data, “invoking” one of these looks like this [[my.namespace/myfunction arg1 arg2 argV…]]
d/with
does everything a transactor does (takes a db and tx-data and returns a new db), but locally and doesn’t write to storage. But any of these function-symbol references will be resolved as it runs, so that DATOMIC_EXT_CLASSPATH
jar has to be in the peer’s classpath too or d/with won’t work in those cases.
(1) is installing a function-code object (as a string that’s compiled+cached on-demand) into the db itself as a value on an entity, and you “invoke” that code in transaction data using its keyword ident, e.g. [[:my/tx-fn arg1 arg2]]
. Peers can get this code through reading the database itself, but you have to use the special interfaces specific to that--the normal language runtime (e.g. require
) doesn’t know about it.
Thank you again @U09R86PA4, yeah it looks like it answered all questions, just to make sure I got it:
1. for add code as data into db: I can create it using the #db/fn. It only accepts clojure core symbols/function and datomic.api symbols/functions (i.e. d/q
function). custom function doesn't work using code as data. It executes like [[:my-fn arg1 arg2 ...]]
.
2. for calling a custom function: I need to have this function in a library and add it to the classpath of the datomic using the DATOMIC_EXT_CLASSPATH
. It executes like [[lib.namespace/lib-function arg1 arg2...]]
The question that raised now for me is about "Peers can get this code through reading the database itself, but you have to use the special interfaces specific to that--the normal language runtime (e.g. `require`) doesn’t know about it"
1. I don't know in which case I would like to peers to get the code
Your answer helped me a lot - thank you again - I think there were some concepts I was missing, for example, that the function is sent to transactor to be executed and the transactor doesn't have the same libs as the peers have.
If I send a list of transactions to 2 different databases (even 2 different transactors), will the resulting t
be the same of the different databases? Or is there no such guarantee?
No guarantees. T advancement is an implementation detail. In practice the only guarantee is that it won’t go backwards, and it will go up at least 1 for each successful transaction.
I'm looking to find the state of an entity immediately prior to a transaction. My first inclination is to use `(dec (d/tx->t db tx))` (dec (d/tx->t tx))
with an as-of query, however I know that t values do not always increase by 1. My impression is that this will still work, but is there better approach?
You get a :db-before
value in your transaction result. Thats a database immediately prior to the transaction just processed
(Similarly, you get a :db-after
value, which is the db value immediately after the transaction just processed 😉 )
Thanks for the response, but in this circumstance I'm querying for the tx-id; the transaction occurred in the past
this will work in that the state of the database you read will be the one immediately prior to the T
i.e. you won’t necessarily be able to (d/t->tx t) and get an entity with :db/txInstant asserted on it
Thanks @U09R86PA4! I just found an example in day-of-datomic, and it looks like I don't even need to use the t-time. I can just dec the tx-id: https://github.com/Datomic/day-of-datomic/blob/20c02d26fd2a12481903dd5347589456c74f8eeb/tutorial/query_tour.clj#L91