Fork me on GitHub
#datomic
<
2018-12-13
>
henrik04:12:52

Is it possible to specify a creds profile for S3 deps? I.e.,

:mvn/repos {"datomic-cloud" {:url ""}}

henrik04:12:47

Apparently, no. The access/secret key have to be inserted into ~/.m2/settings.xml.

steveb8n05:12:50

Q: what’s everyone doing for composite unique keys in Datomic these days? I’m thinking of using a unique string attribute with value concatenated to let the transactor check it but wondering if anyone has uncovered a better trick lately?

matthavener19:12:07

That’s the best I’ve seen.. and some transaction function helpers for upserting or conflict checking

joshkh12:12:58

sorry for the repost, but i was just wondering if anyone has experienced their ions intermittently throwing java.io.IOException: Connection reset by peer exceptions behind an API Gateway? in my case it's a single ion that routes web requests. i've checked the CloudWatch logs for the routing ion, but nothing stands out.

val_waeselynck14:12:33

Same problem here, no explanation.

joshkh15:12:52

uhg. it's affecting our production setup.. no bueno.

joshkh12:12:27

it seems to happen every few minutes, and randomly for various REST requests

benoit12:12:47

@steveb8n I usually implement those kind of checks in a transaction function.

steveb8n22:12:17

Thanks. Yes I have the same thing currently but it is pretty complex and can be difficult to debug so I’m exploring other options.

steveb8n22:12:19

Also, it’s harder to convert these exceptions into friendly user-facing messages so that’s another reason to find an alternative. Right now I’m thinking a combination of a query in the peer and a unique attribute as a fallback

stijn13:12:35

is there a reason that query timeouts generate an exception with :cognitect.anomalies/category :cognitect.anomalies/incorrect instead of :cognitect.anomalies/interrupted? or am I misinterpreting what 'interrupted' means?

marshall14:12:41

That is likely a bug; i will investigate

stijn17:12:13

Ok, if needed I can provide you with a stacktrace

marshall19:12:39

I see

sandbox.core=> (<!! (d/q {:query '[:find ?e
               :where [?e :db/doc]]
      :timeout 1
      :args [(d/db conn)]}))
#:cognitect.anomalies{:category :cognitect.anomalies/interrupted, :message "Datomic Client Timeout"}
sandbox.core=> 

marshall19:12:46

when getting a timeout query

marshall19:12:52

both with sync and async apis

marshall19:12:03

what version of client?

marshall19:12:14

(this is against cloud)

marshall19:12:27

nevermind I’ve seen the behavior in another context; we will investigate

val_waeselynck14:12:24

@marshall So now we have implemented lazy client initialization as recommended, and the deploy succeeds, but we're getting a #:cognitect.anomalies{:category :cognitect.anomalies/unavailable :message "Loading database"} upon lambda invocation (reported by alert: "IonLambdaException"). What can we do about this?

val_waeselynck14:12:16

The exception is thrown after about 1 min, and the trace contains:

[
                "datomic.client.impl.local.Client",
                "connect",
                "local.clj",
                85
            ]

marshall14:12:36

Solo or production stack?

val_waeselynck14:12:19

I'm thinking this may be due to switching to on-demand provisioning on DynamoDb

marshall14:12:31

I have no experience with that; I would hesitate to use it until we’ve tested it with Datomic

marshall14:12:42

it would not surprise me, however

marshall14:12:29

loading a database requires reading from dynamo to retrieve the log tail; if there’s a delay there (i don’t know how the new provisioning thing works) it would slow that process

marshall14:12:54

in general the unavailable due to loading database will resolve and should be retried

val_waeselynck14:12:01

Got it, thanks. What are the recommended Dynamodb settings fro production ?

marshall14:12:24

erm. whatever was in the template 😄 I’d have to go look

marshall14:12:40

Val ^ that’s a prod system i stood up yesterday

marshall14:12:54

read @ 200 min, write @ 25 min

marshall14:12:09

2000 and 500 max, respectively

marshall14:12:49

it seems that slack isn’t letting me put in a screenshot

val_waeselynck14:12:00

Great, thank you! I'm seeing the screenshot all right

marshall14:12:57

Oh good. Slack sent me some message about file upload and free plan. shrug

henrik15:12:15

If you need some write capacity upfront, because you're ingesting a ton of data, is it worth fiddling with minimum write capacity, or should you leave it well enough alone and wait for auto scaling to kick in?

marshall15:12:41

At least in my experience

marshall15:12:15

Scaling will get you there eventually but if you know you're doing an import, you can bump it a bit and get a head start

👍 4
marshall15:12:43

Usually means less headache around making sure your import has long enough retry backoff etc

marshall15:12:15

Ive rarely needed more than 300ish for a bulk import

marshall15:12:33

Cloud uses a lot less ddb throughput than on prem

val_waeselynck15:12:21

Switching back to provisioned capacity solved it, thanks @marshall

👍 8
markbastian15:12:49

Does moving from solo to prod or some other better tier of datomic cloud help with ::cognitect.anomalies/busy issues when transacting data into the db, or is the transactor rate limited in that direction? The documentation at https://docs.datomic.com/cloud/troubleshooting.html#busy indicates that expanding the capacity of your system only helps with reads, not writes. Is this correct?

marshall15:12:10

Moving to prod will definitely increase your write throughput

marshall15:12:36

you’ll be moving from a very small (t2.small) instance to a fairly substantial one (either i3.large or i3.xlarge)

marshall15:12:54

you also get larger and faster caching on the local instance

marshall15:12:03

which will make indexing jobs run faster

markbastian15:12:45

great, thanks

ro621:12:30

Is there supposed to be anything that needs to be done with API Gateway or Lambdas after my first CloudFormation stack upgrade?

ro621:12:24

nevermind, figured it out. The Lambda had changed name and needed to be rebound from API Gateway.

ro621:12:56

is it possible to just deploy the separate Compute and Storage CloudFormation stacks from the start, rather than starting with the "unified" template from AWS Marketplace? When we switch to the production topology I'd like to start off that way unless there's strong reasons not to.

stuarthalloway22:12:48

Yes, that is fine!

👍 4