datomic 2021-06-22 | Slack Archive

On a test environment, we're seeing this kind of log from the Datomic Peer:

;; ~5 hours CI inactive (if this is relevant)

DEBUG 2021-06-18T13:32:37.702 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :phase :begin, :pid 7, :tid 225}

;; 15 minutes pass....

DEBUG 2021-06-18T13:48:25.091 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :msec 947000.0, :phase :end, :pid 7, :tid 225}

;; This next one finishes in 21ms:

DEBUG 2021-06-18T13:48:25.094 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :phase :begin, :pid 7, :tid 225}
DEBUG 2021-06-18T13:48:25.115 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :msec 21.0, :phase :end, :pid 7, :tid 225}

;; and now we start processing the transacts that queued up when the tests started...

INFO  2021-06-18T13:48:25.117 [manifold-execute-43]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :start, :pid 7, :tid 282}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :begin, :pid 7, :tid 226}
DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :msec 0.462, :phase :end, :pid 7, :tid 226}
INFO  2021-06-18T13:48:25.235 [clojure-agent-send-off-pool-24]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :end, :pid 7, :tid 226}

After those 15 minutes of waiting, the peer moved on and transacted all the transactions that happened to queue in the meantime, as if nothing happened. There do not seem to be any interesting or anomalous logs in the Transactor for this time period. Datomic Peer (1.0.6269) with Postgres storage. The peer and transactor are both running on Kubernetes, but Postgres is hosted outside of the k8s cluster. Any idea what could be going on with the :kv-cluster/get-val and how to go about debugging this further? Is there some timeout we can configure to avoid this kind of situation in a production environment?

pithyless09:06:02

#Also sent to the channel

I moved this question to the forum. If anyone has some insights or comments about running Datomic On-Prem with k8s, I'd really appreciate it (either on the forum or in this thread). 🙏 https://ask.datomic.com/index.php/631/blocking-event-cluster-without-timeout-failover-semantics

Fabim20:06:24

Hey, I just subscribed to Datomic Ions Solo for my pedestal project. On my fist ion-dev push I get the following error. Any suggestions of what I’m doing wrong? my deps.edn has io.pedestal/pedestal.jetty {:mvn/version "0.5.9"} and no jetty-util

Joe Lane20:06:38

@U010L3S1XHS You're not going to believe me, but I believe you have a corrupt, partially downloaded jetty-util jar in your local ~/.m2 directory. Delete the jetty-util jar (NOT your entire m2 directory) and then try to push again?

Fabim21:06:49

@U0CJ19XAM Thanks for your quick answer. I deleted it and now I get Syntax error (ClassNotFoundException) compiling at (cognitect/http_client.clj:1:1). org.eclipse.jetty.client.HttpClient when using datomic solo up. how do you recommend I reinstall jetty in m2?

Joe Lane21:06:13

Delete the .cpcache in your project directory

Joe Lane21:06:51

Beyond that I’d need to see your deps edn

Fabim21:06:53

I deleted .cpcache. but got the same syntax error

Fabim13:06:03

@U0CJ19XAM The deploy worked. Thanks for the tip. Reseting jetty in m2 solved it.

Joe Lane13:06:03

Great to hear! Always happy to chat

Fabim13:06:28

@U0CJ19XAM I deployed and got a `.datomic-ions/` folder. Can that folder be put into `.gitignore` , or does it need to be pushed with git?

Joe Lane13:06:43

It can be ignored.

Fabim13:06:17

@U0CJ19XAM Thanks. When deploying I got a lot of dependency-conflicts. Is there a way to update the depencencies running in my datomic cloud or do I need to explicit use the old dependency versions in my deps.edn to get rid of that warning?

Joe Lane13:06:37

The latter.

Joe Lane13:06:14

If you hit a conflict that you can't work around, contact support.

Fabim14:06:55

@U0CJ19XAM I am responding to a GET request on the API gateway mapped on a lambda with ring.util.response/resource-response to deliver the index.html but the css and js are blocked by the browser with Content Security Policy: The page's settings blocked the loading of a resource Is there a way to deliver a website with (pedestal)ion without errors?

Fabim19:06:16

I allowed some origins, as the forum suggested. Now I’m stuck with subfolders not being loaded. Happy to hear how you solved that https://clojurians.slack.com/archives/C03RZMDSH/p1624475471307800

Joe Lane20:06:46

Are you running a local jetty server for development?

Fabim20:06:31

yes

Joe Lane20:06:21

And presumably the subfolders work with the same service-map?

Fabim20:06:24

during development I use integrant

Fabim20:06:59

the service map is different in development

Joe Lane20:06:19

Can you diff them?

Fabim20:06:39

this is the difference

(-> service
(dissoc ::http/chain-provider)
        (assoc  
          ::http/join? false
          ::http/routes #(route/expand-routes (deref #'routes))
          ::http/secure-headers {:content-security-policy-settings
                                 {:default-src "'self'"
                                  :style-src "'self' 'unsafe-inline'"
                                  :script-src "'self' 'unsafe-inline'"}})
        (http/default-interceptors)
        (http/dev-interceptors)
        (http/create-server))

Fabim20:06:42

should be the setup they do in peodestal-ions-sample

Fabim13:06:37

@U0CJ19XAM I have an idea what the problem was. thanks for your help

💯 2

pithyless09:06:02

replied to a thread:On a test environment, we're seeing this kind of log from the Datomic Peer: ;; ~5 hours CI inactive (if this is relevant) DEBUG 2021-06-18T13:32:37.702 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :phase :begin, :pid 7, :tid 225} ;; 15 minutes pass.... DEBUG 2021-06-18T13:48:25.091 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60a4cc9e-ac11-40fe-9bd0-582f8209a403", :msec 947000.0, :phase :end, :pid 7, :tid 225} ;; This next one finishes in 21ms: DEBUG 2021-06-18T13:48:25.094 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :phase :begin, :pid 7, :tid 225} DEBUG 2021-06-18T13:48:25.115 [clojure-agent-send-off-pool-23]: {:event :kv-cluster/get-val, :val-key "60128f6d-25fb-41f0-a6c2-9c5e73267da7", :msec 21.0, :phase :end, :pid 7, :tid 225} ;; and now we start processing the transacts that queued up when the tests started... INFO 2021-06-18T13:48:25.117 [manifold-execute-43]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :start, :pid 7, :tid 282} DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :begin, :pid 7, :tid 226} DEBUG 2021-06-18T13:48:25.234 [clojure-agent-send-off-pool-24]: {:event :peer/accept-new, :id #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :msec 0.462, :phase :end, :pid 7, :tid 226} INFO 2021-06-18T13:48:25.235 [clojure-agent-send-off-pool-24]: {:event :peer/transact, :uuid #uuid "60cca429-7908-49b1-82c8-56b50effb4ce", :phase :end, :pid 7, :tid 226} After those *15 minutes* of waiting, the peer moved on and transacted all the transactions that happened to queue in the meantime, as if nothing happened. There do not seem to be any interesting or anomalous logs in the Transactor for this time period. Datomic Peer (1.0.6269) with Postgres storage. The peer and transactor are both running on Kubernetes, but Postgres is hosted outside of the k8s cluster. Any idea what could be going on with the `:kv-cluster/get-val` and how to go about debugging this further? Is there some timeout we can configure to avoid this kind of situation in a production environment?

2021-06-22

Channels