Fork me on GitHub
#datomic
<
2020-04-15
>
Ignas09:04:34

hello. I am running a following query to get changes that happened over the last few minutes:

(d/q '[:find [?e ...]
    :in $ ?log ?from-t ?to-t
    :where
    [(tx-ids ?log ?from-t ?to-t) [?tx ...]]
    [(tx-data ?log ?tx) [[?e]]]
    [?e :entity/status]]
    (d/as-of db timestamp-to)
    log
    timestamp-from
    timestamp-to)
Recently I've noticed that if I rerun the same query after some time, I sometimes get more items returned. Is it possible that on the original run I am getting stale view from the log? As I am reusing the same instance of log for the whole batch.. or for some other reason

benoit12:04:34

Hi Ignas, that's surprising. Is it possible that your timestamp-to is in the future the first time it runs? And the second run picks up more transactions?

Ignas12:04:10

Shouldn't be. timestamp-to is capped at (new Date) on the original run

benoit13:04:39

The peer and the transactor might not be in sync. I would look at the extra items you get in your second query and see when they were transacted. Otherwise I don't see what could have happened, sorry 🙂

favila14:04:38

(D/as-of db timestamp-to)

favila14:04:57

That is not a guarantee that the db contains tx info up to and including timestamp. Due to physics, the information may not have arrived yet

favila14:04:08

To get a guarantee, use d/sync

favila14:04:41

Or, read dbs off the tx-report-queue

favila14:04:15

Or (d/db conn) and examine that db’s basis-t

favila14:04:53

IOW you should either derive “now” from the datomic peer’s clock (whatever the latest t is that it knows about) or use d/sync to make sure the peer caught up to a wall-clock time you want to sync to

favila14:04:17

Don’t just use (new Date) as your clock blindly

benoit15:04:29

Oh! It's the as-of that might not see everything. Thanks @U09R86PA4. Was that correct that he could also have missed txs if the peer clock was ahead of the txtor?

favila15:04:33

It’s the same issue

favila15:04:07

Dbs have an inherent basis-t, which is the newest t they know about. As-of is a filter on top of that, but doesn’t alter the basis

benoit15:04:22

Yeah the new Date seemed like a red flag to me but I missed the database part of it.

favila15:04:58

If your as-of is > basis it’s like not having an as-of

benoit15:04:30

makes sense

Ignas15:04:19

are there any benefits of doing d/sync without t over just doing (d/db conn)?

Ignas15:04:40

as I don't have a t to sync to, would just using (d/db conn) instead of the whole as-of work here:

(d/q '[:find [?e ...]
    :in $ ?log ?from-t ?to-t
    :where
    [(tx-ids ?log ?from-t ?to-t) [?tx ...]]
    [(tx-data ?log ?tx) [[?e]]]
    [?e :entity/status]]
    (d/db conn)
    log
    timestamp-from
    timestamp-to)

Ignas16:04:27

I am ok with getting the latest value of the entity here, even if as-of is in the past

favila17:04:16

If you are coordinating between peers or with some other wallclock system you need sync. If you aren’t coordinating and just want the latest thing that peer can see, use the basis t of any db value as the “latest”

favila17:04:09

I am not sure which you are doing though. The use of timestamp arguments is suspicious to me and suggests maybe you are coordinating with a clock somewhere

Ignas17:04:55

I have a service that exports the changes to another database. It uses a sliding window with overlap so that it can catch-up if there is a delay, or we might want to replay from a certain point in time. This is where the wallclock comes in.. we use it to log what periods were exported and to determine what to export next. so ideally i just want to get everything that happened in the last few minutes (ant the window gets bigger if its catching up)

Ignas17:04:44

So it seems that it might have been a design flaw to choose timestamps as a way to track the export windows

Ignas17:04:43

on the other hand it gives us more visibility on what data is already exported, being able to easily say that for example data is at the moment served with 5 hour delay.

Cas Shun15:04:53

Is there something like Datomic Console for Datomic Cloud, or a way to use it with Cloud?