Fork me on GitHub
#datahike
<
2022-10-12
>
Joshua Suskalo17:10:41

So there's the way to use datahike client/server which makes the server be a single transactor+reader with the clients just sending queries, the same way you'd interact with e.g. a sql database. Is there a way to make it so that there's a single transactor which does writes, with multiple different datahike clients that do read-only access to the backing datastore and are able to pick up those writes in an eventually-consistent manner? Is that something that can be done effectively with datahike at the current time? I'm aware of the transactor PRs and discussions which would end up resulting in something like this, but I'm curious if this is feasible with the current implementation by e.g. refreshing the connection every x seconds.

timo09:10:11

Hi @U5NCUG8NR. Cool that you are interested in this 👍 Afaik updating the connection is exactly the problem right now. We need to think of a way to make the reading datahike-instances aware of writes from other instances. This seems not so straightforward. Maybe @UB95JRKM3 @U1C36HC6N or @UQVCR784A can chime in here.

metasoarous17:10:09

It's possible that dat.sync could be reworked towards this end. It's built around datomic & datascript, but honestly, the (relatively few) points of impedence mismatch between the two made the project very challenging. Focusing on datahike <-> datahike would be a lot easier methinks, and as far as I can tell all of the basic pieces are in place in datahike to make it happen.

metasoarous17:10:20

I know that @U1C36HC6N has also thought a lot about distribution as part of datahike itself though, and it's definitely been designed with that in mind. I don't know if there's been any work towards that or where it might fall on the roadmap though.

whilo07:10:27

@U5NCUG8NR This is already possible https://lambdaforge.io/2019/12/08/replicate-datahike-wherever-you-go.html, but yes the shared address space will make this happen automatically.

Joshua Suskalo14:10:54

Thanks! This is exactly what I was looking for!

Joshua Suskalo14:10:07

This seems like the perfect application for agents if the system uses a timeout-based re-connect rather than a file watch, which would be more ideal for e.g. a postgres kv-store backend

Joshua Suskalo14:10:42

(def db (agent (d/connect :whatever)))

;; on init
(send-off db
          (fn refresh-db [_old-db]
            (Thread/sleep 5000)
            (d/connect :whatever)
            (send-off db refresh-db)))

;; in the code
(d/q '[:whatever] @db)

👍 2
Joshua Suskalo14:10:16

For my usecases this is perfectly fine as all I'm looking for is eventual consistency, and having a delay between a write and reads detecting that write is fine.

whilo07:10:57

Yes, I think this is a fairly general setup that should work fine.

👍 1