Fork me on GitHub
#datomic
<
2023-07-04
>
robert-stuttaford15:07:52

is anyone running Datomic On-Prem in Azure?

Ivar Refsdal19:07:32

(We run it as a Azure Container Instance)

robert-stuttaford07:07:38

@UQGBDTAR4 @UGJE0MM0W thank you! what storage are you using?

Ørnulf Risnes07:07:42

"Posgresql single server" atm. Moving to "flexible server" soon. Still Postgresql.

robert-stuttaford07:07:27

apologies, i don't know Azure at all - is that a managed service, like AWS Aurora?

Ørnulf Risnes07:07:47

It is managed, yes.

Ørnulf Risnes07:07:10

(I don't know AWS :-))

Ørnulf Risnes08:07:01

TL;DR most things has gone well. We were bitten by high network volatility compared to our old on-prem situation and @UGJE0MM0W had to tune socket-timeouts etc to keep things sane. Ivar can probably share more info later if needed.

Ørnulf Risnes08:07:45

(Network volatility is probably pretty cloud-vendor-agnostic.)

robert-stuttaford08:07:59

how long have you been running? how big is your database in storage?

Ørnulf Risnes08:07:10

Pretty big. Datomic-backup ca 7GB (Psql-backup much larger)

Ørnulf Risnes08:07:33

Number of datoms: not sure, need Ivar to give estimate.

robert-stuttaford08:07:51

sweet, thank you, i really appreciate the feedback, it's super helpful!

robert-stuttaford08:07:05

our db is close to 100gb lol, since 2013

Ørnulf Risnes08:07:47

Should be fine regardless.

Ørnulf Risnes08:07:17

Note that Azure Container Instances have pretty modest limits on cpu/ram (4vcpu/8GiB)

Ørnulf Risnes08:07:50

So depending on use-case you might want to look for bigger runtime situations for your apps

robert-stuttaford08:07:08

hmmm we're gonna need bigger instances than that 😅

robert-stuttaford08:07:16

gtk thank you

👍 2
Ivar Refsdal09:07:25

I think container instances (ACI) are max 16 GiB. Azure container apps (ACA) are max 8 GiB. We also needed to "up" the amount of space available for the PostgreSQL instance. That in turn triggered a bigger IOPS limit for the instance. With the previous and lower limit I believe there were problems with apps not being allowed to read at a sufficient rate/speed. It took some time to figure this out. All our containers in azure have &socketTimeout=30 have set in the connection string, otherwise you will risk waiting for all eternity on some queries, or ~16 minutes if you are lucky. For Datomic I'm using &socketTimeout=3 (three seconds). There might be some idle connection time limit property that could be better, but we've also seen dropped connections just after application startup, so even so it's still is a problem. More reading: https://github.com/brettwooldridge/HikariCP/wiki/Rapid-Recovery (I know Datomic uses apache tomcat connection pool, HikariCP just describes the issue well.) https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/ I've been writing an article about this stuff forever for Datomic in particular (which is still not finished)

❤️ 1
robert-stuttaford15:07:13

this is amazing intel, thank you so much for sharing @UGJE0MM0W!

❤️ 1
Ivar Refsdal07:07:45

Oh and BTW our biggest Datomic database is 250M datoms, Peers have 8 GiB memory for this database and cache hit is near perfect (IIRC)..

Ivar Refsdal17:07:24

one more thing... We've had a lot of DNS issues with ACI, so I don't think we can recommend that part of azure. ACA is better.

☝️ 1
braai engineer22:07:13

When I attempt to provision a DynamoDB table for Datomic, I get the following key-related exception despite having valid AWS keys set (verified via AWS cli) and exported:

bin/datomic ensure-transactor config/ddb-transactor.properties config/ddb-transactor.properties
com.amazonaws.services.identitymanagement.model.AmazonIdentityManagementException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. (Service: AmazonIdentityManagement; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 0e693d6f-7ea0-4c46-9fa3-7d7f18acec32; Proxy: null)
Recreated AWS root keys and ran this, but still happens:
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
ensure-transactor worked for ddb-local, but failing for actual Dynamo due to permissions. Anyone encountered this?

braai engineer22:07:36

Apparently I have to run ensure-transactor with an IAM role? https://clojurians.slack.com/archives/C03RZMDSH/p1633518558245400?thread_ts=1633453233.241800&amp;cid=C03RZMDSH I thought the whole point of ensure-transactor was that it would create the necessary AWS resources for me? I created an IAM user and gave it full permissions, created new access keys but am still getting the same SignatureDoesNotMatch error :face_holding_back_tears: .

braai engineer23:07:55

Some reports online say this can be caused by special characters or doubles slashes in secret access key, but this error also occurs for a plain looking secret.

Benjamin07:07:58

Could be that this happens when your machine clock is too far from aws's clocks. Not sure how the error looks in that case.

braai engineer07:07:57

I ended up reinstalling Datomic on a fresh machine, set the same AWS keys and it worked 🤷 I still don't know why it's not working on my primary machine.

braai engineer07:07:07

wish the error messages were better