This is an example of the error message

After rebooting my EC2 instances in the primary compute group, now I can't even get the original verrsion to deploy. I'm DOWN.

Managed to get the system back up and had a successful deploy -but now I'm back down. datomic log is full of these too:

{
    "Msg": "ClientSPIAnomaly",
    "DatomicClientSpiErrorResponse": {
        "Status": 200,
        "Body": {
            "CognitectAnomaliesCategory": "CognitectAnomaliesUnavailable",
            "CognitectAnomaliesMessage": "Loading database"
        }
    },
    "Type": "Event",
    "Tid": 145,
    "Timestamp": 1708580753875
}

@U0698L2BU Loading Databases means the node is attempting to load the database and it is not yet loaded. This happens on node startup. Are your EC2 Instances actually up and healthy and do you have a healthy code deploy on the system?

What version are you running? And do you only see these exceptions around startup (expected) or constantly?

The instances are up and healthy and I do have a healthy code deploy on the system. Could the "Connection refused" errors be caused because datomic is still loading databases? If so, then perhaps patience is the solution here.

On startup i have seen it take a min to load a database on a system with many databases. How long have you been waiting?

Anything longer would be problematic in my view…

Not more than 20 seconds -so no problem there.

If the "Connection refused" is simply a transient symptom of the db loading delay, I can work around that pretty easily.

@U0698L2BU We could use a feature of "lifecycle hooks" to tell the user when the system is "ready" to receive requests. As an aside, the log has a few log messages which might be useful in understanding if a DB is loaded. { $.Msg = "SpindownDb" || $.Msg = "SpinupDb"} ... I think we also report the loaded Databases in CatalogCache.

I also have seen more issues loading databases when separate stacks are at play and specifically you have a smaller sized group trying to load DBs. (i.e. your query group is t3 small and your primary compute is an i3 large)

But having different sized compute machines across groups has other implications for performance that usually rear their head first.

Please let me know what kind of retry you implement and if the issue persists past the initial transient failures.

Other than polling by a simple query until no anomalies occur, is there a better way to (maybe `d/db-stats) to know when the db is loaded?

Also, I think the prblem might be a bit deeper. In addition to the connection refused errors, I see now that there is a corresponding "Uncaught Exception: Loading database" (see snippet above).

Do I still have or could I get read-only access to Cloudwatch and the system name? Perhaps DM me the account ID (or e-mail or other) so I can look up if I still have access

We are dealing with similar issues, still putting together a coherent support request and thoughts, and: Some of the same: • Started seeing connection refused a week or so ago • "Loading database" takes more than a minute • We are running latest. Our first HTTP requests after an ION deploy takes more than a minute sometimes (and of course times out after 30s) - there after everything settles.

Exactly what I am seeing.

(except my timeframe for the problems started last night with the first deploy with the new client)

You see it settle eventually @U0698L2BU?

If by settle you mean that eventually the http-direct requests return correct responses.... maybe once. Otherwise, I was hitting the panic button and reverting.

Perhaps relevant: last night I managed to get a successful deploy (=> successful AWS deploy + my app started) with the new client and now things seem to be OK. I'm calling that deploy the miracle deploy because prior to it I failed to successfully deploy (cuz app start failures, not strictly a deploy failure of course) probably 12 times and was really freaking out. Nothing much changed other than I had rebooted the instances. The deploy is running now. But I'm scared to try again.

We've been using

com.datomic/client {:mvn/version "1.0.134"}

com.datomic/client-cloud {:mvn/version "1.0.125"}

This is when the instances have new code on them:

@U9E8C7QRJ we just resolved Chris's issue and it was related specifically to running out of scaling events in DDB. Could you check to see if you are in the same situation by reviewing your DDB table is failing to autoscale? You can see this in your DDB table under the tab?

If you have a chance can you log me a ticket with this @U9E8C7QRJ that looks like something I want investigate that.

I'll log that ticket @U1QJACBUM, thank you

You’ll have to parse the input and convert it to a java.util.Date instance in your code before and then build the transaction payload with it. Assuming you’re using JSON it could be a formatted date string. Another possibility and the one we use ourselves is to use https://github.com/cognitect/transit-format as the interchange format, and extend it with a read handler, so for example you could extend it to take an array of 3 integers and interpret that as year, month and day.

I tried to avoid doing manipulations on the server. It feels weird; why should I interfere with the data? I just want the ion lambda to be a proxy, and it feels way too coupled to Java for no reason. I was hoping for a solution where I wouldn't need to mess with the data..

Not sure if I understand your “ion lambda to be a proxy” statement. The lambda itself is still a proxy and its role is just to serve as an entry point to your datomic system. If you were to implement parsing as I suggested, that code would not run in the lambda itself, it would run in the EC2 instance that’s part of your datomic system. Since the attribute is an instant, I don’t see how you could provide one without doing the parsing yourself in your ion code.

Sorry, my bad, I meant the EC2 not the lambda directly, regards the parsing, I thought maybe there's some format that would work, timestamp/datetime/etc, but from your answer I understand that there isn't anything like that

No worries I just wanted to be sure it was clear that it wasn’t the lambda doing the computations, it’s a very common misconception with ions. And no, unfortunately there’s no format that I know of.

Oh ok, thanks!

@U02DNF3TW3E do you know if its possible to define a function that would run on insertion and will convert timestamp to java.util.Date? (and would be seamless when transacting)

I think you can achieve that with transaction functions https://docs.datomic.com/cloud/transactions/transaction-functions.html but I’m not 100% sure as I have never coded my own transaction function yet.

Thanks, I read it already couldn’t find such a function, thanks!