Fork me on GitHub
#aws
<
2016-08-31
>
luposlip09:08:19

I consider doing the following: 1. Each container maintains a list of users connected to it (this is already there with Sente) 2. Whenever a user connects, a common list is maintained in Datomic 3. Whenever a user disconnects, the user is removed from the common list 4. Whenever an update from a user needs to be sent to other users, it starts by sending out the messages directly to the users connected to the same container. Then the container reads the common list of connected users from Datomic. If some connected users are connected to other containers, the messages for those users are stored in a common messages list (queue) in Datomic. 5. Workers on all containers regularly (say, every 15 seconds?) check this common message queue. If a message exist for a connected user, the message is sent to the user, and removed from the queue. 6. When a user disconnects/logs off, the list of waiting messages for that user is (permanently) removed from the message queue.

alandipert13:08:26

@luposlip what is the purpose of #4? also instead of polling datomic, you may consume the transaction report queue and look for the messages

luposlip13:08:36

@alandipert: the purpose is for example: "I've changed this value that you are subscribing to". The server (containers) need to notify all subscribers (via push). Both the users connected to the same container that receives the event that changes the value (sent immediately), and the users connected to other containers.

alandipert13:08:06

so it's for reducing latency when users are adjacent?

luposlip13:08:16

What's the benefit of looking through the TX report queue? Can this be read directly from storage instead of waiting for the peer library to catch up?

alandipert13:08:39

well if you route everything through datomic instead of short circuting, than datomic is the data of record

alandipert13:08:49

and datomic already has a way to listen for changes, the tx queue

alandipert13:08:12

latency is potentially higher but it's simpler imo

alandipert13:08:27

also if you add logic to happen to the message, it goes in one place

luposlip13:08:51

So he TX queue is still relying on the peer library?

alandipert13:08:58

it's a feature of the peer library, yeah

luposlip13:08:10

Gotta look into that.

alandipert13:08:11

in practice one consumes it in a thread, filtering out messages not interested in

alandipert13:08:52

in a system i work on today we've built a really crappy datomic over a 10 yr period

alandipert13:08:57

various databases and queues spread around, piles of hacks

alandipert13:08:11

if you buy totally into datomic features from the beginning though i think you can make some pretty clean stuff

alandipert13:08:40

i should mention i am not a paid spokesperson lol

alandipert13:08:47

just a big fan ™️

luposlip13:08:51

Agreed. Do you use some scheduling library, or just plain Java threads? I've been using at-at and and some other earlier (just forgot which) based on Quartz.

alandipert13:08:06

you mean for the thread to consume the tx queue?

alandipert13:08:36

i have used plain threads, since it's a blocking queue, the thread just blocks on the get call

luposlip17:08:07

Yes, that's what I meant.