when is this happening for you? I’m curious in case I encounter the same thing but also, maybe you have a locked-out node like I have in the past. If that’s true, you can terminate the node in the EC2 console and the ALB will create a new one

I've set up three separate Datomic Cloud systems (2xSolo, 1xProduction). All of them, including separate query groups created on the production system, exhibit this problem intermittently. I assumed it was a fact of Ions.

Running the latest CF template and libs etc.

I could cycle all of them I guess, but I doubt that's the issue given the prevalence.

One app is an API. For this one, we've solved it by wrapping API calls in retries. One is serving HTML though, so that particular workaround doesn't work obviously.

So the datomic cloud system is serving HTML? One way around that is to serve static assets from Cloudfront or S3 and use D.C. purely for the API. We started always setting up a Cloudwatch Event to healthcheck / warm the Lambdas and try to hit this problem with a CW event instead of a customer. Its not perfect but I expect this will be fixed in the future due to the work done on connecting from apigateway directly to the ec2 machines instead of through a lambda proxy.

The API plus the data wrangling magic is deployed to a query group. The app (front + backend) sits on another query group and talks to the API. I'll probably stick it in a non-Ion environment to get rid of the issues. Which is a shame, you do still get a lot of nice affordances by running it as an Ion, even if it's not talking to Datomic directly.

Yeah, hopefully it'll get worked out. We have an ElasticSearch cluster, which we obviously can't run as an Ion. But in an effort to align it with the rest of the system, I created it using CloudFormation and wrote a set of tools that replicates some of the stuff that comes with Datomic Cloud (such as push/deploy etc.) Datomic is obviously nice, but the I find the tooling around it rather nice to work with as well.

@U06B8J0AJ I’m experimenting with lucene on datomic cloud via ions. Just curious, what order of magnitude are your elasticsearch indexes? 100s of gbs or 1000s of gbs?

@U0CJ19XAM Interesting! We're currently at about 2.3TB in total. The individual indices are around 500-800GB.

It's all growing and likely to continue growing though.

Cool! Sounds like a lot of info, unfortunately it may be too large for the problem space. I was hoping to use the disk space on the i3large or i3 xlarge nodes but only for cases where the data will stay under 500 GB.

Yeah, that's definitely too large, and ES has requirements with regards to cluster behaviour that I don't think would be immediately compatible with Datomic Cloud anyway. If Datomic could run on i3.2xlarge, the disk space might not be a problem. In the name of compartmentalisation, it might be prudent to leave the transaction nodes alone though.

Yeah, I was planning on putting the nodes on a different instance 🙂