while a single thread in a single process is responsible for writing transactions, the transactor can still take advantage of multiple cores for other purposes, right?

right: http://docs.datomic.com/monitoring.html#add-more-cores sorry for noise

datomic peer can really light up the cores IME

until it's io bound

I’m trying to get Datomic working in k8s with Cassandra as the storage service, my peers cannot find the transactor after connecting to storage. I’ll describe the setup:

Here’s the transactor config:

protocol=cass
    host=0.0.0.0
    alt-host=datomic
    port=4334
    license-key=<redacted>
    cassandra-table=datomic.datomic
    cassandra-host=cassandra
    cassandra-port=9042
    memory-index-threshold=32m
    memory-index-max=512m
    object-cache-max=1g

@roman host and alt-host are used by peers to find the transactor

The transactor is running in its own pod and there’s a service called datomic with port 4334 that other pods can connect to

yes, I know but…

the peer reports that it’s trying to connect to the transactor at localhost and alt-host nil

despite what the config says

It’s my understanding that the transactor stores its connection info in storage and the peer picks it up there. But it appears not to get the configured information. Is that not correct? Why does the peer not try datomic:4334?

It’s as if the transactor’s config is not applied to storage.

a different transactor or config is running; peer is connecting to wrong storage; peer cannot resolve "datomic" to an address?

There is without a doubt no other transactor or no other Cassandra because in both cases I’m launching fresh pods in fresh clusters. As for the peer not resolving it, it’s not even trying it:

clojure.lang.ExceptionInfo: Error communicating with HOST localhost on PORT 4334 {:alt-host nil, :peer-version 2, :password "ckGYIzP97L6l+ERbbcnZuzCiSB/v3S1HfzUZdyIFLdE=", :username "f8HVTTNhf8bmknxqrO2TINx2xmPH5KAJLwQ5q/Cs0J8=", :port 4334, :host "localhost", :version "0.9.5561.59", :timestamp 1507230053031, :encrypt-channel true}

It believes alt-host is nil for one thing.

that could be because it didn't resolve

I see

what is the connection string peer is using?

transactor log will echo the connection string on startup

datomic:<cass://cassandra:9042/datomic.datomic/<redacted>>

fyi, there’s a service called cassandra and the peer connects to it (clear from logs(

But that’s interesting what you said about nil meaning it didn’t resolve.

Does that host need all three datomic ports to work?

i.e. 4334, 4335, and 4336?

I think not. I think 4335 is dev transactor storage access, 4336 is the h2 GUI console when using dev storage

That’s what I thought

so that’s not the problem then

what if you don't use 0.0.0.0 as the host

does anything change?

I first tried with localhost

same issue

do you mean leave it out?

I mean use something routeable

hmm, given that the IP is dynamic, how?

generate the transactor.properties on startup

clever

I'm mostly suspicious that the wrong config file is pulled or another transactor is running against that storage

But here’s what’s strange. I’ve run this in k8s before just fine with this setup when it was the dev transactor without having to do anything that complex.

only when the storage is separate am I running into this.

dev transactor's "storage" is the peer, so your connection string routes to that peer already

There’s only one transactor pod. I don’t know where it’s even possible for there to be another.

outside the k8 cluster

that’s true, good point

maybe a forgotten test

the cluster is locked down

there’s no way in without port forwarding and I’m the sole person using the cluster

maybe you could kill all transactors you know about, try to connect a peer, see if you get a different error

but you’re right that it’s a mystery

if you get same error, then there's definitely a transactor somewhere

ok

one sec

trying it

Same error, but that doesn’t make any sense. This is a completely fresh Minikube with nothing on it by Cassandra. Is it possible that a transactor that is no longer running but was once running connected to Cassandra and left it’s connection info there and it’s just not getting updated?

when I launch a fresh one

nothing but cassandra and the peer, that is

I don't know. it's not what I would expect to happen

so strange

I appreciate your help however

ps axf | grep datomic doesn't show anything? you mentioned port forwarding. maybe you tried a transactor outside the cluster earlier and forgot about it?

you can also inspect the cassandra table itself, see if it's getting written to

transactors write at least once a second to heartbeat

That’s reasonable but I’m not currently port-forwarding

I just mean in theory the only way in is such

you an also double-check your paths for your config file

That’s a good idea

(for the transactor startup)

Wouldn’t it have failed to start though if that had been wrong?

BEcause of the license

maybe you have two?

some editing shuffle

these are good suggestions

or forgot to save

just covering bases

no, I appreciate it

localhost and alt-host nil are suspicous

I know, right?

the ps checks out

can also confirm in the logs that the transactor did actually startup and connect to cassandra. maybe it never did and the settings in there are from an earlier test, like you suggested

after that, I'm out of ideas

well, here are the logs:

Launching with Java options -server -Xms4g -Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic: ...
System started datomic:

that’s it

that's the systemd logs, not the transactor logs

(stdout)

I see

that’s just what is produced to stdout

I should be looking for a file then?

I'm trying to find the defaults

does the name of the properties file matter, i.e. the config file?

as long as it matches what was supplied, no

Right, and I’m giving it config/transactor.properties and that’s the only file at that location

next up is examining the table

bin/logback.xml has the log config

duh it’s just the log dir

everything in there seems clean

it discovers and connects to the Cass cluster nodes

also datomic.transactor - {:event :transactor/start, :args {:cassandra-port 9042, :cassandra-table "datomic.datomic", :log-dir "log", :alt-host "datomic", :protocol :cass, :rest-alias "cass", :memory-index-max "512m", :cassandra-host "cassandra", :port 4334, :memory-index-threshold "32m", :data-dir "data", :object-cache-max "1g", :host "0.0.0.0", :version "0.9.5561.59", :encrypt-channel true}, :pid 1, :tid 12}

so it knows the alt-host here

yep

so something is wrong with the peer or the networking setup for the peer

this is what’s in the Cass table: {:key "[\\"0.0.0.0\\" \\"datomic\\" 4334 \\"XnH1k0PQkm/Hz/Y4ISZpE7fpHcBMP7ui8pz8wwNcPXk=\\" \\"afmWR0zouI8Bee4gld/5zM48H4pecIzmHeNjeCODSfI=\\" 1507233334931 \\"0.9.5561.59\\" true 2]"}

I think you’re right.

yeah that all looks good, transactor is definitely connecting

you can look in the peer (java) logs too for more info

you should see events like :peer/get-connection :coord/lookup-transactor-endpoint, and :peer/hornet-connect and :peer/hornet-connect-failed

these will tell you more than the exception

good call

I’m currently suppressing them in logback but that’s easy to change

wow

now it’s just working

hah, that's great

I'm having some trouble trying to start a dev transactor inside of docker-compose. Well, it's not trouble exactly...

I'm able to start the transactor and connect to it OK and it seems to work, but every time I connect to it I get bunches of these tracebacks in the peer:

ERROR 16:54:30.264 o.a.activemq.artemis.core.client: AMQ214016: Failed to create netty connection
java.nio.channels.UnresolvedAddressException: null
	at sun.nio.ch.Net.checkAddress(Net.java:123)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:208)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:203)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:549)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:534)
	at io.netty.handler.ssl.SslHandler.connect(SslHandler.java:438)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:549)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:534)
	at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:549)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:534)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:516)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:215)
	at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:402)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
	at java.lang.Thread.run(Thread.java:745)

My transactor props looks like this:

protocol=dev
host=0.0.0.0
alt-host=dev-datomic
port=4334

services:
  dev-datomic:
    image: ""
    ports:
      - "4334-4336:4334-4336"
    volumes:
      - "datomic-data:/opt/datomic/data"

From the peer I'm connecting to datomic: