Fork me on GitHub
#datomic
<
2017-03-23
>
dm313:03:07

hello, is the datom seq obtained from (datoms db :aevt :attr) guaranteed to be ordered by datom transaction time?

uwo13:03:17

during imports (using async tx-pipeline), at a certain point the transactor became unavailable and never recovered. Is there a likely culprit?

dm313:03:35

uwo: did the transactor die?

uwo13:03:52

that’s a good question. Unfortunately, I was pairing with someone and didn’t think to look at transactor log at the time on his computer

uwo13:03:16

the process itself didn’t fall over

dm313:03:30

I’d guess OOM on the transactor then if the peer didn’t recover but the import process was still working

uwo13:03:03

should that be expected when attempting to run large imports on a dev computer (you, know potentially limited resources)? or are we not configured correctly?

dm314:03:54

it should be expected if your working set is larger than the configured available memory 🙂

dm314:03:17

but we are guessing here as you don’t even know if an OOM happened

marshall14:03:37

@dm3 the order of datoms is determined by the particular index you’re using. the only one that is fully ordered by t is the log

dm314:03:20

marshall: thanks

dm314:03:31

what’s the best way to find the time of the latest assertion for a given :attr?

dm314:03:03

I guess (max-by :tx (datoms :aevt :attr)) isn’t the worst option

dm314:03:48

actually I’ll just annotate the transaction with a marker

marshall14:03:10

New Datomic Training Videos and Getting Started Documentation: http://blog.datomic.com/2017/03/new-datomic-training-videos-and-getting.html

uwo14:03:16

@dm3 thanks for the help!

dominicm16:03:50

Am I right in thinking that there is only 1 tx-report-queue per client? And if I want to subscribe in multiple places I'll need a pubsub implementation?

favila16:03:03

There is only one tx report queue per peer per connection. You don't need pubsub necessarily, you could just have one listener that multiplexes

dominicm16:03:41

Sure. Multiplex would probably have been the correct term, yeah.

spieden16:03:57

@dominicm just curious: what’s your use case?

dominicm16:03:05

@spieden I'm ETL'ing datomic -> sql. But in the near future we need to do start reacting in a streaming manner to transactions too (I've admittedly forgotten what for)

spieden16:03:26

ah ok, cool so a real time ETL type thing

dominicm16:03:46

Need it for Tableau / Business Inteligence software. That wants ODBC/SQL/etc.

spieden16:03:17

makes sense

spieden16:03:11

any ideas for how you’ll do checkpointing?

juliobarros16:03:50

Hi, quick question. I'm looking to deploy a transactor in an enterprise setting and ops is asking what ports are required and why. Are all three, 4334, 4335 and 4336, required for peers and is there a short description of what they are used for. Thanks.

dominicm17:03:45

@spieden writing to a side table for every tx, within a sql transaction. Should let me reliably catch up & resume.

spieden17:03:18

@dominicm nice, checkpoint goes with each chunk of data

dominicm17:03:58

@spieden Yup. If a write ever fails, the checkpoint never goes in. So there's no problem. Datomic's log is ordered, so no problem there either.

dominicm17:03:36

I'm probably not going to use the data from the transaction queue, just use it as a "wake up"

dominicm17:03:46

But I'm undecided on that thus far.

favila17:03:46

@juliobarros port 4334 is the transactor port (peer to transactor communication). On non-dev storage that is all you need

favila17:03:09

@juliobarros 4335 is for dev storage only. It's an sql endpoint to the embedded h2 database in the transactor that runs when you use dev storage

favila17:03:33

@juliobarros 4336 is for dev storage only. It is a browser-based sql management interface to the embedded h2 db

juliobarros17:03:05

Thanks @favila do you know why host is specified in the transactor properties file? What it is used for? We'll be using docker in an env where host names may be fluid.

favila17:03:00

The host and alt-host are written into storage as names the peer should resolve to contact the transactor

favila17:03:16

the peer connection process is 1) connect to storage 2) read magic heartbeat record written into storage by the active transactor for that storage (includes host and alt-host) 3) connect to host or alt-host (it tries both)

juliobarros17:03:46

@favila hmmm ok. I get the heartbeat part. But how does this all relate to the host in the connection URL? Thanks again btw.

favila17:03:23

The host in the connection url is the STORAGE host. The host in the transactor.properties is the TRANSACTOR host

favila17:03:45

transactors are more fluid than storage

juliobarros17:03:49

Ah. Ok. Thanks.

erichmond18:03:02

sorry. dumb question, but is there a similar idea to materialized views in datomic?

juliobarros21:03:34

@favila or anyone who knows. I have to proxy the peer transactor connections through a load balancer. But setting the host to be the loadbalancer (so that the peer can find it by reading the magic record in the db) causes activemq not connected errors.

favila21:03:19

@juliobarros I don't have any advice except verify the load balancer is working correctly

favila21:03:56

right port, right layer (not an http balancer, for eg), forwarding to right host, etc

favila21:03:24

you can't actually balance these connections, you are aware of that right?

favila21:03:37

You are just using it to forward?

juliobarros21:03:04

Yeah just to forward. It's working fine. It looks like a host alt-host ('internal' vs 'external' up) issue. Going to try and swap their settings.

favila21:03:32

@juliobarros that should not make a difference, both are tried

favila21:03:54

peer will use whichever one succeeds

juliobarros22:03:58

By the peer. But the current issue is with activemq.

favila22:03:17

activemq (artemis/hornetq) is how the peer communicates with the transactor

favila22:03:26

that is the connection

favila22:03:06

transactor.properties host=http://foo.org is so peer can connect to transactor via http://foo.org

favila22:03:13

using an artemis connection

favila22:03:12

alt-host is just so you can give a different ip/dns by which you can connect to the same transactor

favila22:03:20

peer will try both and use whichever one works

juliobarros22:03:12

Ah. Ok. So maybe a socket issue at the LB. (like you suggested) let me investigate. Thanks.