I would look for JDBC connection issues

That would cause a heartbeat failure?

You’re talking JDBC connection from datomic back to the MSSQL back end?

The “heartbeat” is the transactor writing its address into storage periodically. It’s part of the HA failover system

I think it’s the id “pod-coord” in the sql backend.

I don’t remember exactly. It starts with “pod-”

so the heartbeat failure would mean the transactor couldn’t write to MSSQL

if this doesn’t happen immediately on startup, that suggests some JDBC connection-level thing is wrong. maybe a connection count is exceeded, maybe MSSQL killed a long-running connection, maybe there’s TCP issues lower down the stack.

Ok, I’m seeing references to heartbeat in the logs on New Relic

This heartbeat entry is also how peers find the transactor

The problem happens at irregular intervals, but we’re definitely starting successfully and running for a good few minutes.

does it happen only when there’s an idle transaction period?

also, are you running on google cloud?

No it seems to only happen when we try to execute transactions.

And we’re not on Google Cloud

do any transactions succeed?

ever?

Yes, we’re getting some throughput, we’re just having frequent interruptions.

hm. yeah, I would look very closely at JDBC driver settings and MSSQL settings

I’ll pass that on, thanks much!

and try to correlate the interruptions with either idle time or high load (such as an indexing job).

idle time maybe triggers timeouts or connection closes; high load maybe causes instability or forced closes

:thinking_face:

Ok, that’s some stuff we can look into, thanks again

Could it be that the MSSQL backend server has a short max time that a connection can "live"? From what I know about tomcat-jdbcpool, which Datomic uses, the default is to never close a connection when it is returned to the pool. It's controlled by the https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#How_to_use property. Unfortunately I don't think you can control this from/in Datomic.

We ended up restarting the SQL Servers in the HA group (which was a non-trivial operation), and that seems to have resolved the issue, so it does look like a problem with the underlying storage rather than with datomic itself.

Are you running your own SQL servers "on premises"? For our MS Azure PostgreSQL setup we needed to increase the number of IOPS or so. The database was struggling a lot before that. I don't recall how that manifested itself exactly, but I think with dead/dropped connections.

Thanks, I’m not involved on the SQL Server side of things at all, but I’ll pass that on.

If you are running the database in the cloud or "paas", I'd also recommend setting socketTimeout on the peers: https://ask.datomic.com/index.php/631/blocking-event-cluster-without-timeout-failover-semantics?show=700#a700

What problem are you trying to solve? Do you have multiple databases in the same storage+transactor, or do you want to deprovision the whole stack as fast as possible, or something else?

I have a script that restores a DB, can't restore a DB with an existing DB of same name, so i want a quick one-liner that doesn't require devs (who don't know clojure) to drop and restore

what kind of storage?

just local dev

would it make sense to unconditionally rm -r the data directory?

or even to not use datomic-level backup/restore but to distribute the h2 files

then it’s just file copy

oh really? Even if it's from dynamo ?

no, I’m assuming you’re using the same storage

storage level operations are probably always going to be fastest. I don’t know if you care about what else may be in the local dev. If it’s just distributing readonly replicas, you could backup from dynamo, restore into h2, then distribute the h2 files to all the devs and just blow away whatever h2 files they have already.

if you do care about what else may be in those local dev databases, then I think you need d/delete-database or d/rename-database, a more interactive datomic-level restore, etc

Remember d/delete-database doesn’t reclaim storage

https://docs.datomic.com/on-prem/operation/capacity.html#garbage-collection-deleted-dev

thanks!

are you thinking of get-database-names?

that’s the client api

oh gotcha

I don’t know why they decided to make these different

peer api predates the client api

thanks!