Fork me on GitHub
#datomic
<
2022-12-12
>
neilprosser18:12:06

Around three hours ago our Cloud compute group (which was upgraded to latest this morning) seems to have stopped keeping the memory index size in check. About an hour ago it stopped accepting transactions reporting the "Busy indexing" anomaly. IndexMemMb which never normally gets above 400 or so flat-lined at 2.23k. Should we ever have a situation where we wait over an hour for the indexing process to catch-up? It's not something I've seen before.

Joe Lane18:12:10

If you haven't already, you should open a support case with us.

neilprosser18:12:15

Thanks @U0CJ19XAM. I've just done it.

Daniel Jomphe20:12:08

We too are on the newest version - please keep us posted with your conclusions if ever the cause might be found in non-edge-case situations!

steveb8n20:12:50

Also interested in this. To avoid an outage, we'd be interested in an update on this one

jaret21:12:55

@U0514DPR7 @U0510KXTU we have reproduced the problem and are working with Neil to address the issue. We will update everyone shortly.

1
jaret22:12:02

Hello Datomic Cloud Users! If you are currently running 981-9188. We ask that you downgrade to 973-9132.

onetom07:12:30

are there any explicit downgrade instructions?

onetom07:12:37

https://docs.datomic.com/cloud/operation/upgrading.html#do-not-downgrade just says don't downgrade or "contact support". my main question is whether we should downgrade the compute stack 1st or the storage stack?

onetom07:12:23

and should we downgrade all the libraries ion, client-cloud etc too?

onetom11:12:50

thanks @U1QJACBUM for your quick support email answer! i will put your answer here for others too: > You should downgrade all stacks. The order of downgrade does not matter, but the critical stack to downgrade is primary compute. You can perform this operation by using the templates for 973-9132 and following our update stack documentation. Select the stack you want to downgrade, then select update stack, but use the 973-9132 templates.

jaret16:12:49

@U086D6TBN Sorry for missing your other question. It's been a long night of assessing this issue. You do not need to downgrade ion, or client-cloud. We recommend removing any usage of io-stats you may have added.

👍 1
jaret22:12:47

CC @U0514DPR7 @U031K8CUX6D @U0510KXTU who were interested on the other thread. And special thanks to Neil for allowing us access to their system to be able to quickly identify the issue.

👍 2