Fork me on GitHub
#datomic
<
2020-04-04
>
John Leidegren08:04:25

I'd like some input on general EAV data design. I've started using tuples to create unique identities so that I can enforce a uniqueness constraint over a set of refs within an entity. It looks something like this (tx map):

{:db/id 17592186045416
 :list-of-things [{:some-unique-identity [17592186045416 17592186045418]
                   :some-ref 17592186045418
                   :meta "foo"}
                  {:unique-identity [17592186045416 17592186045420]
                   :some-ref 17592186045420
                   :meta "bar"}]}
So, what I'm trying to do, is to prevent there from being two or more :some-ref for the same thing in this set of :list-of-things. Is this nuts or does this make sense? I'm worried I just invented some convoluted way of doing something which should be modelled different? If find tuple identities to be incredibly useful because I get this upsert behavior each time but I don't see how I can avoid the potential data race that would otherwise occur. Any suggestions here, would be much appreciated.

✔️ 4
John Leidegren11:04:36

I think I figured this out. --- These unique identity tuples are needed because I created entities to group attributes but that's already provided by namespaces. I could just has well let the namespaces encode the topology and let the grouping of things be based on that. --- These "topological" identities wouldn't be needed if I went for bigger, fatter entities, over excessive container like entities. These intermediate entities that encode a tree like structure are just causing me pain. And I will do away with them.

David Pham11:04:09

Does Datomic support Java11?

jeff.terrell18:04:01

Is it possible to ssh in to the non-bastion instance in a Solo topology for Datomic Cloud?

jeff.terrell18:04:56

I'm seeing some weird behavior in which a freshly restarted instance seems to get hung whenever I try to deploy (which fails).

jeff.terrell18:04:36

Oddly, the CPU utilization, as visible on the CloudWatch dashboard, jump to 45% and stays there after I try a deploy, whereas before it's near 0%.

jeff.terrell18:04:28

Once I start the deploy, neither an existing socks proxy nor a new one allows connections through to Datomic, whereas before the deploy it works fine.

jeff.terrell18:04:30

I can datomic solo reset to get back to a working state…but if I try to deploy, I get back into the hung state.

jeff.terrell18:04:57

I'd like to ssh in to see what process is using so much CPU.

jeff.terrell18:04:09

I'm fairly perplexed by all of this. It's on a fresh AWS account and Datomic Cloud instance, and I've had success with Ions and the Solo topology before…

jeff.terrell18:04:26

One more clue: the CodeDeploy deployment fails on the final ValidateService event. The script output has about 100 lines of [stdout]Received 000. I think this means that the Datomic health check request is failing to get a response at all, let alone a 200.

ghadi18:04:44

@jeff.terrell sometimes I jump into the Datomic nodes to do a jstack dump.

ghadi18:04:08

By default, the bastion cannot SSH to the nodes because of a security group

ghadi18:04:26

there is a security group that is called datomic-something-nodes

ghadi18:04:47

you need to modify that SG to allow the bastion instance in on port 22

jeff.terrell18:04:59

Ah, security groups, right! I assumed that the compute node would already be configured to accept traffic from the bastion. But yeah, maybe not on port 22, right. Thanks!

ghadi18:04:43

Protip: you can add an entry to that security group, referring to the bastions security group symbolically

4
ghadi18:04:01

instead of hardcoding an IP address or CIDR block

ghadi18:04:49

then, you need the ssh keypair for the nodes, which you had when you created the stack

ghadi18:04:18

so what I do is I add the bastion's key from ~/.ssh/datomic-whatever to my ssh agent:

ghadi18:04:31

ssh-add ~/.ssh/datomic-whatever

ghadi18:04:50

then add the node keypair: ssh-add ~/wherever/nodekey

ghadi18:04:05

then I ssh to the bastion with -A, which forwards the local ssh-agent

ghadi18:04:23

then from there you can ssh to the node in question

ghadi18:04:32

sudo su datomic to become the datomic user

jeff.terrell18:04:49

Ah, fantastic tips, thanks! I would have been stumbling around trying to scp the appropriate private keys over to the bastion.

ghadi18:04:52

so that you can run jstack on the pid, or poke around

ghadi18:04:50

My pleasure. Whatever you do end up finding, see if there is some signal of it in CloudWatch Logs or CodeDeploy or wherever

👍 4
ghadi18:04:47

if there's not, maybe worth a support ticket?

jeff.terrell18:04:38

OK. I haven't seen any clue in those places yet. I'll be sure to follow up as needed to be sure others don't run into this.

jeff.terrell18:04:22

When I got into the system, I learned that the CPU utilization was because of bin/run-with-restart being called to start some Datomic-related process over and over, which was failing every time. When I ran the command manually, it tells me:

jeff.terrell18:04:31

> com.amazonaws.services.logs.model.ResourceNotFoundException: The specified log group does not exist. (Service: AWSLogs; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: c58f56ce-3b7f-4be1-bbd3-463a950018c7)

jeff.terrell18:04:50

…followed by a stack trace, and an anomaly map:

jeff.terrell18:04:18

:datomic.cloud.cluster-node/-main #error {
 :cause "Unable to ensure system keyfiles"
 :data {:result [#:cognitect.anomalies{:category :cognitect.anomalies/incorrect} #:cognitect.anomalies{:category :cognitect.anomalies/incorrect}], :arg {:system "pp-app", :prefix "s3://"}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Unable to ensure system keyfiles"
   :data {:result [#:cognitect.anomalies{:category :cognitect.anomalies/incorrect} #:cognitect.anomalies{:category :cognitect.anomalies/incorrect}], :arg {:system "pp-app", :prefix "s3://"}}
   :at [datomic.cloud.cluster_node$ensure_system_keyfiles_BANG_ invokeStatic "cluster_node.clj" 336]}]
 :trace ,,,}

jeff.terrell18:04:09

I'm thinking this is not because of something I did wrong (though I'm happy to be corrected on that point). Opening a support ticket…