I'd like some input on general EAV data design. I've started using tuples to create unique identities so that I can enforce a uniqueness constraint over a set of refs within an entity. It looks something like this (tx map):

{:db/id 17592186045416
 :list-of-things [{:some-unique-identity [17592186045416 17592186045418]
                   :some-ref 17592186045418
                   :meta "foo"}
                  {:unique-identity [17592186045416 17592186045420]
                   :some-ref 17592186045420
                   :meta "bar"}]}

Does Datomic support Java11?

Is it possible to ssh in to the non-bastion instance in a Solo topology for Datomic Cloud?

I'm seeing some weird behavior in which a freshly restarted instance seems to get hung whenever I try to deploy (which fails).

Oddly, the CPU utilization, as visible on the CloudWatch dashboard, jump to 45% and stays there after I try a deploy, whereas before it's near 0%.

Once I start the deploy, neither an existing socks proxy nor a new one allows connections through to Datomic, whereas before the deploy it works fine.

I can datomic solo reset to get back to a working state…but if I try to deploy, I get back into the hung state.

I'd like to ssh in to see what process is using so much CPU.

I'm fairly perplexed by all of this. It's on a fresh AWS account and Datomic Cloud instance, and I've had success with Ions and the Solo topology before…

One more clue: the CodeDeploy deployment fails on the final ValidateService event. The script output has about 100 lines of [stdout]Received 000. I think this means that the Datomic health check request is failing to get a response at all, let alone a 200.

@jeff.terrell sometimes I jump into the Datomic nodes to do a jstack dump.

By default, the bastion cannot SSH to the nodes because of a security group

there is a security group that is called datomic-something-nodes

you need to modify that SG to allow the bastion instance in on port 22

Ah, security groups, right! I assumed that the compute node would already be configured to accept traffic from the bastion. But yeah, maybe not on port 22, right. Thanks!

Protip: you can add an entry to that security group, referring to the bastions security group symbolically

instead of hardcoding an IP address or CIDR block

then, you need the ssh keypair for the nodes, which you had when you created the stack

so what I do is I add the bastion's key from ~/.ssh/datomic-whatever to my ssh agent:

ssh-add ~/.ssh/datomic-whatever

then add the node keypair: ssh-add ~/wherever/nodekey

then I ssh to the bastion with -A, which forwards the local ssh-agent

then from there you can ssh to the node in question

sudo su datomic to become the datomic user

Ah, fantastic tips, thanks! I would have been stumbling around trying to scp the appropriate private keys over to the bastion.

so that you can run jstack on the pid, or poke around

My pleasure. Whatever you do end up finding, see if there is some signal of it in CloudWatch Logs or CodeDeploy or wherever

if there's not, maybe worth a support ticket?

OK. I haven't seen any clue in those places yet. I'll be sure to follow up as needed to be sure others don't run into this.

When I got into the system, I learned that the CPU utilization was because of bin/run-with-restart being called to start some Datomic-related process over and over, which was failing every time. When I ran the command manually, it tells me:

> com.amazonaws.services.logs.model.ResourceNotFoundException: The specified log group does not exist. (Service: AWSLogs; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: c58f56ce-3b7f-4be1-bbd3-463a950018c7)

…followed by a stack trace, and an anomaly map:

:datomic.cloud.cluster-node/-main #error {
 :cause "Unable to ensure system keyfiles"
 :data {:result [#:cognitect.anomalies{:category :cognitect.anomalies/incorrect} #:cognitect.anomalies{:category :cognitect.anomalies/incorrect}], :arg {:system "pp-app", :prefix "s3://"}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Unable to ensure system keyfiles"
   :data {:result [#:cognitect.anomalies{:category :cognitect.anomalies/incorrect} #:cognitect.anomalies{:category :cognitect.anomalies/incorrect}], :arg {:system "pp-app", :prefix "s3://"}}
   :at [datomic.cloud.cluster_node$ensure_system_keyfiles_BANG_ invokeStatic "cluster_node.clj" 336]}]
 :trace ,,,}

I'm thinking this is not because of something I did wrong (though I'm happy to be corrected on that point). Opening a support ticket…