Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-05-04

Channels

#datomic
<
2022-05-04
>
chrisetheridge10:05:31

is there any documentation about Datomic excise segments? its a metric that we are seeing in Datadog, but i cannot find any documentation about what exactly they are

jdkealy16:05:22

is anyone doing on-prem with kubernetes ?

jdkealy16:05:18

That's good to know! It would simplify my life SOOO much to have the transactor in my k8s

ghadi16:05:25

are you on AWS?

ghadi16:05:56

storage in DDB + transactor in K8S is what we do

ghadi16:05:26

works great. try to minimize pod disruption though (spot instances, etc.)

jdkealy16:05:32

Great! So what about the transactor address ?

jdkealy16:05:47

Do you use an IP or a load balancer ?

ghadi16:05:59

I'm not sure.

jdkealy16:05:48

ok i'll give it a whirl

jdkealy20:05:22

So i was able to start the transactor in a kubernetes pod. And in my cluster, my services has a service address of datomic . I made an alias in my transactor pod datomic = localhost But when i go to start the db from clojure it says 

starting database connection
CREATE DB
Error communicating with HOST datomic on PORT 4334

jdkealy20:05:38

this tells me that datomic was able to connect to dynamo, it wrote its location to storage datomic the transactor is up and running.

jdkealy20:05:25

My kubernetes service looks like 

resource "kubernetes_service" "datomic" {
  metadata {
    name = "datomic"
  }
  spec {
    selector = {
      App = kubernetes_deployment.react.spec.0.template.0.metadata[0].
    }
    port {
      port        = 4334
      target_port = 4334
    }
    type = "LoadBalancer"
  }
}

jdkealy20:05:07

{:alt-host nil, :peer-version 2, :password "<redacted>", :username "<redacted>", :port 4334, :host "datomic", :version "0.9.6045", :timestamp 1651695721103, :encrypt-channel true}

jdkealy20:05:34

so this also means my peer is able to connect to dynamo

jdkealy20:05:29

root@clj-55c7d877b7-c47cn:/usr/src/app# telnet datomic 4334
Trying 172.20.247.2...
telnet: Unable to connect to remote host: Connection refused
root@clj-55c7d877b7-c47cn:/usr/src/app# telnet datomic 80
Trying 172.20.247.2...

jdkealy20:05:42

port 80 hangs

Joe Lane20:05:15

Which version of java are you using?

jdkealy20:05:23

in my peer ?

Joe Lane20:05:28

either both

jdkealy20:05:47

root@clj-55c7d877b7-c47cn:/usr/src/app# java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

jdkealy20:05:56

bash-4.3# java -version
openjdk version "1.8.0_92-internal"
OpenJDK Runtime Environment (build 1.8.0_92-internal-alpine-r1-b14)
OpenJDK 64-Bit Server VM (build 25.92-b14, mixed mode)
bash-4.3#

jdkealy20:05:58

transactor

jdkealy20:05:11

what versions should i be using ?

Joe Lane20:05:15

The version of datomic you're using was released in https://docs.datomic.com/on-prem/changes.html#0.9.6045, before JDK17 was ever released. The latest release, https://docs.datomic.com/on-prem/changes.html#1.0.6397 , added support for JDK17

Joe Lane20:05:08

Hope that helps @jdkealy!

jdkealy20:05:14

which version of java should i be using ?

Joe Lane20:05:43

17 works great w/ the latest release, otherwise, pick 11

Joe Lane20:05:37

Latest release also still supports 8

Joe Lane20:05:23

But you should really pick a newer jdk than 8, so much has improved.

jdkealy20:05:15

going to rebuild the image and give it a go!

jdkealy20:05:26

i'm surprised there's not more docs on using a transactor in k8s

jdkealy20:05:50

root@datomic-ff99879fd-cs74w:/datomic# java -version
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode, sharing)

jdkealy20:05:11

now the transactor fails with 

Terminating process - Serve failed
ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119007: Cannot connect to server(s). Tried with all available servers.]
	at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:787)
	at datomic.artemis_client$create_session_factory.invokeStatic(artemis_client.clj:114)
	at datomic.artemis_client$create_session_factory.invoke(artemis_client.clj:104)
	at datomic.update$create_master$fn__11961.invoke(update.clj:732)
	at datomic.update$create_master.invokeStatic(update.clj:722)

jdkealy20:05:54

i think maybe the old java doesn't honor host aliases

jdkealy21:05:34

@U0CJ19XAM do you think there could be an issue using a load balancer to reach port 4334 ?

Joe Lane21:05:59

I'm not a Kubernetes expert, and I would be suspicious of a load balancer

Joe Lane21:05:14

I also would be suspicious of "aliases"

jdkealy23:05:49

ok, i'm using the IP address, exposed 4334,4335,4336, no luck

jdkealy00:05:04

good god... after some debugging, i realized i had been pointing my kubernetes service to the wrong kubernetes pod :man-facepalming:

Joe Lane00:05:30

What happened once you pointed it at the right pod?

jdkealy00:05:01

it just connected fine... I'm using a nodeport and using the service address

jdkealy00:05:15

so the host is just "datomic"

jdkealy00:05:46

for hours i was getting "can't connect to 4334"

jdkealy00:05:14

so then i fired up the datomic console on 8080 and when i couldn't connect to that either i got suspiscious

jdkealy00:05:36

and then i knew something was wrong when i couldn't connect to python -m simpeHTTPServer

jdkealy01:05:20

stared at it for 30 mins and FML. I was pointing at a Deployment that didn't have anything running on any of those ports

Joe Lane01:05:48

I had something similar a few months ago where I lost a whole day (and some hair) because I forgot that core.async/onto-chan!! closes the channel by default and I didn't set close? to false .

Joe Lane01:05:17

I know you've been looking into this for a while now (not just today), I'm glad you got it sorted out.

jdkealy01:05:31

yes thanks! I'll write do a writeup on it, since there's nothing on the web it seems. Since we've got like 5 environemnts, i'd hate to spin up 5 ec2 servers. What a relief!

Hendrik18:09:38

Hey @jdkealy, I am trying to run datomic pro in kubernetes. I ran into the same error as you did Error communicating with HOST datomic on PORT 4334 . What I have so far is: -running postgres in k8 -transactor1 service (exposing port 4334) -transactor2 service (exposing port 4334) transactors are running fine. Do you have writeup of your setup? Or would you like to share it with us? 🙂 Many thanks in advance.

Hendrik18:09:27

{:alt-host nil, :peer-version 2, :password "<redacted>", :username "ZAdA8wuCJIMcma+wY6mUEPDdjT7SRnFFH4qu7wPfCN0=", :port 4334, :host "localhost", :version "1.0.6735", :timestamp 1694371140472, :encrypt-channel true}