Fork me on GitHub
#datomic
<
2019-11-25
>
fjolne10:11:09

Couldn’t sign up for Datomic Forum (seems like it’s getting troubles lately), maybe somebody here knows: is it safe to open transactor port to the public, assuming it’s connected to a password-secured SQL storage? I understand that peers first connect to the storage, then get transactor coordinates and connect to it, but couldn’t find the authorization mechanism between peer and transactor in the docs.

bhurlow14:11:18

peers communicate with transactor via an encrypted channel so it’s OK to host transactor on public IP. In fact, this is the only config that worked for us

fjolne22:11:47

@U0FHWANJK thanks, at some point we had the same error, but setting host to internal network IP (inside VPC) made it to work. That’s good to know that the connection between peer and transactor is secure, but my concern is different: I wonder whether somebody else could actually transact / read something via the open transactor port, which is why I’m interested in the auth protocol (handshake?) between peer and transactor.

bhurlow22:11:29

somewhat sure all communication into transactor requires the “secret” value which is stored in backed storage

fjolne05:11:35

Ugh, it’s actually in the docs: https://docs.datomic.com/on-prem/aws.html So, yes, connection from peers to transactor is secured via randomly generated credentials, and it’s ok to open transactor to the public.

bhurlow20:11:01

still felt a bit exposed to me too

fjolne10:11:00

We’ve currently secured transactor via firewall to allow only connections from the exact peer, but that’s kinda inconvenient for dev (requires ssh tunnelling) and autoscaling (requires to manage all the internal network IPs of our peers).

Frank Huili Xu12:11:27

Hi, I’m new to Datomic. Is there any advice or best practice if I’d like to connect Datomic in ClojureScript/Nodejs? Thanks.

bhurlow14:11:05

cloud or on-prem? There are no official peer libraries for cljs or node

Frank Huili Xu16:11:06

Right now using on-prem. But will use cloud in production. Yes. I can’t find cljs library.

grzm18:11:11

@jaret I know I asked you about whether or not PollingCacheUpdateFailed errors had been addressed recently, but I may have been overly distracted when you answered. (To refresh your memory: What we're seeing is part of our Datomic Cloud system stopping (a periodic CloudWatch Event that writes out to a Vertica database) while the rest of the system keeps humming along fine. I've seen PollingCacheUpdateFailed errors in the Cloudwatch logs that correlate with this.)

jaret18:11:55

@grzm looks like… :

"Msg": "PollingCacheUpdateFailed",
    "Cache": "CatalogCache",
    "Err": {
        "CognitectAnomaliesCategory": "CognitectAnomaliesFault",...

jaret18:11:21

What version of Datomic Cloud are you running on this system?

grzm19:11:16

Yup:

"Msg": "PollingCacheUpdateFailed",
    "Cache": "cache-group-poller",
    "Err": {
        "CognitectAnomaliesCategory": "CognitectAnomaliesFault",
        "DatomicAnomaliesException": {
            "Via": [
                {
                    "Type": "com.amazonaws.SdkClientException",
                    "Message": "Unable to execute HTTP request: Too many open files",
                    "At": [
                        "com.amazonaws.http.AmazonHttpClient$RequestExecutor",
                        "handleRetryableException",
                        "AmazonHttpClient.java",
                        1175
                    ]
                },
                {
                    "Type": ".SocketException",
                    "Message": "Too many open files",
                    "At": [
                        ".Socket",
                        "createImpl",
                        "Socket.java",
                        460
                    ]
                }
This was with 480-8770. We've since upgraded to 535-8812 and haven't seen it since

grzm19:11:29

Seeing that in various caches: index-group-poller, tx-group-poller, cache-group-poller, query-group-poller, autoscaling-group-poller. Looks like they generally happen in pairs or three at a time, mix-and-matching which cache groups are included.

grzm19:11:35

One that happens on its own is CatalogCache , with

{
                        "Type": "com.amazonaws.SdkClientException",
                        "Message": "Unable to execute HTTP request: Connect to  [] failed: Read timed out",
                        "At": [
                            "com.amazonaws.http.AmazonHttpClient$RequestExecutor",
                            "handleRetryableException",
                            "AmazonHttpClient.java",
                            1175
                        ]
                    },
             

jaret19:11:24

So one of the causes of that error (pollingCacheUpdateFailed) was addressed and other causes as long as they are transient shouldn’t represent a problem. Re: the CloudWatch Event that writes to the Vertica DB are you seeing any other errors or any other correlations? are you deploying at the same time? is the event special in any way?

jaret19:11:03

I’d be happy to poke at the metrics and logs if you want to give me read-only access.

grzm19:11:03

Haven't seen other errors at the same time, which is why it's kinda been stumping us. No deploys either: it happens after the system's been running for at least a couple of days running fine. Just stops writing. Let me coordinate with the client and get back to you on the log access: that'll likely have to wait until tomorrow.

jaret19:11:31

And you have to kick over the application or datomic to get it back up again? @grzm?

grzm19:11:10

We "redeploy" (same revision) and it all starts working again. (what would it mean to restart only Datomic?)

tyler20:11:08

Has there been any news on the xray daemon for datomic compute nodes?

marshall20:11:31

@tyler it is included on the nodes in the latest release

marshall20:11:42

but it’s up to you to configure/use it for now

marshall20:11:47

more docs/info coming in the future

tyler21:11:57

👍 awesome, we’re happy to configure it just need that daemon running. Thanks.