does could not read transactor location from storage mean it can't resolve the address or literally that it can't find the location ?

I just finished like a 50GB restore. I restarted datomic and restarted the peers and get :db.error/read-transactor-location-failed I'm wondering if that means the transactor never wrote its location or if it means it can't resolve. I'm looking at the dynamo table is it's like 50GB, but it's impossible to decipher if it wrote its location or not.

You need a transactor running for the peer to connect

right! I do have one running

I just pointed it to a fresh new dynamo table

The implication of this is that the transactor machine does not have the permissions to communicate to underlying storage. This could be AWS perms, network perms, or other storage level permissions. What's your underlying storage?

dynamo

but when i do this from the transactor machine, i get Connection to datomic (172.20.249.103) 4334 port [tcp/*] succeeded!

It's difficult to tell where the problem exists with this message. I know the transactor can communicate with Dynamo because I see its heartbeart incrementing and the transactor location. I believe the peer can connect to Dynamo because I think I would have gotten an AWS error.

are there any additional debugging steps that could help determine this @U1QJACBUM

@U1DBQAAMB can you SSH or something onto the box with the transactor and reach DDB through the AWS API? Additionally, are there logs from transactor startup on that box for the transactor?

I got to the bottom of the issue by REPL'ing onto the peer and calling d/list-databases. It could not reach storage. I feel like there could be a better error message like "access denied to dynamodb table" or something.

what do u mean by physical database? also, are you talking about datomic on-prem or datomic cloud? u can run the d/create-database function multiple times against the same datomic client.

Yeah to echo Tamás questions... we'd need to know what you mean by "physical database." But assuming you mean having multiple datomic databases on the same datomic system, i'd then need to know if you are using Datomic on-prem or Datomic cloud. In both instances you can have multiple databases on the same system, but there are caveats and depends on what your databases will be used for in on-prem. Essentially, on-prem allows you to have multiple databases, but we recommend having a single DB per system with potentially smaller operational DBs alongside. In On-prem the transactor has to hold the memory index for each DB and thus multiple DBs with high throughput is not a great fit for on-prem. Cloud however is more suited for a multi-tenant model. We describe all the implications of multiple databases in on-prem here: https://docs.datomic.com/on-prem/operation/capacity.html#multiple-databases