Fork me on GitHub
#datomic
<
2019-02-14
>
amarjeet05:02:46

Question regarding Transaction - I have 3 attributes, say x, y, & z, where z = (+ x y), and z will be evaluated and committed only if it gets the values of both x and y. Now, at t=0, my db has just x = 3. At t=1, y=5 gets into the system, so z gets evaluated and becomes z=8 and now both y and z are getting committed in a read & write transactions (composed together). At the same time (at t=1), some other thread changed x to 4. But, since Datomic doesn’t have any read transaction that can be composed with the write transaction, how can I make sure that my read & write transaction should retry. Note that I am not committing x in my write transaction.

amarjeet12:02:22

I got my answer

johnj17:02:57

What solutions are used for online hot backups? for on-premise

johnj17:02:36

I understand using the storage tools is not good enough?

favila21:02:19

what is an "online hot backup"?

favila21:02:47

the backup-db tool can backup at any time, and can do incremental backups

favila21:02:56

this is a storage-agnostic backup

favila21:02:26

you can also do storage backups, but to be safe they need to be transactional

yogidevbear21:02:20

Hi, I have a syntax/feature question. Is there an equivalent in Datomic d/q for a T-SQL in clause?

favila21:02:09

there's nothing built in. use clojure or destructuring

👍 5
favila21:02:00

[(ground [1 2 3]) [?match ...]] [?e ?a ?match] is better when ?a is indexed and ?match is selective

favila21:02:40

[?e ?a ?match] [(contains? #{1 2 3} ?match)] when filtering would be faster

yogidevbear21:02:50

Thanks Francis, will give it a look

yogidevbear21:02:02

e.g. :where [?e :col in [1,2,3]

johnj21:02:15

@favila they are real time backups not just snapshots

favila21:02:34

you can use them if they are consistent

johnj21:02:58

@favila what none consistent storages are available for datomic?

johnj21:02:54

ah dynamo has both strongly and eventually consistent options

favila21:02:06

I'm not talking about the storage but the backups

favila21:02:24

e.g. I don't know that every possible type of mysql backup will give me a consistent snapshot

favila21:02:33

maybe it has dirty reads in it

favila21:02:43

(for e.g.)

johnj21:02:26

@favila I see, so how do you deal with the "real time backup" issue? I can make backups every night but what about the data that is being inserted the next day before the backup?

favila21:02:34

you can run backup-db continually; or use a storage tech that can do this

favila21:02:01

I don't think most production systems have up to the second backups

favila21:02:12

if they need that much redundancy they do replication

johnj21:02:58

@favila Guess one can do both, datomic's backup and storage

favila22:02:09

storage is still a backup

favila22:02:26

"backup" I understand to mean "offline, periodic snapshot"

favila22:02:35

no service needs to be running to keep it

johnj22:02:58

yeah, by storage I mean something like aws dynamo and postgresql point-in-time recovery features

favila22:02:30

if you want "if this machine catches on fire I don't lose data" that's clustering/replication territory not backups

favila22:02:46

so dynamodb, you just trust will never lose your data

favila22:02:55

and you keep periodic backups in case someone accidentally deletes production

johnj22:02:56

does datomic's backup create any load/performance issues?

favila22:02:04

it's the same as having another peer

favila22:02:13

actually it's less

favila22:02:18

no transactor load at all

favila22:02:29

but it is storage load, because it's reading blocks

johnj22:02:23

that's good

favila22:02:51

It looks like postgresql point in time recovery will make something safe to restore directly to postgresql

favila22:02:00

I am not an expert though

favila22:02:09

but usually I think of a backup as something to either stand up the same data in another storage system or to correct some catastrophic developer error

favila22:02:54

e.g. I deliberately do not want to restore to an up-to-the-second backup because an hour ago someone fatfingered a DELETE

johnj22:02:01

yeah, I conflating terms, this is mostly clustering/replication territory not backups

favila22:02:06

(this is less an issue with datomic obvs)

johnj22:02:17

as you said

favila22:02:04

we keep both kinds of backups, mysql (done automatically by google's hosted mysql backups) and datomic

favila22:02:18

we've only ever used the datomic ones though

johnj22:02:26

this is mostly "I don't want to lose todays work because I only run backups at night" issue

favila22:02:52

yeah for that I think you can run backups more frequently

favila22:02:00

datomic backups

favila22:02:11

or trust postgresql if that's a faster+easier restore

favila22:02:38

datomic restore of an empty storage will likely be much slower than that storage's native backup method

johnj22:02:58

yeah, looking for less OPs, both dynamo and rds have point in time recover

johnj22:02:44

so if you trust them with the data as you said, they will handle this

johnj22:02:50

anyway, running incremental backups every 30 mins using datomic doesn't sound that bad

favila22:02:11

the real advantage of the datomic backup is storage-independence

favila22:02:03

in the era of hosted, clustered, SLAed database-as-a-service I don't really see the point of the storage's own backup for something like datomic

favila22:02:18

other than a faster restore if the storage falls over

favila22:02:57

but the datomic backup is "your" copy, you know you can use it even if your storage host goes up in flames

favila22:02:09

and you can switch to a different storage

johnj22:02:17

good point, but if you are all in in a cloud provider is hard to see why you will change storages, but who knows

favila22:02:44

the cloud provider comes up with a better storage?

favila22:02:49

well, we can dream

johnj22:02:14

and then you have to wait for datomic to leverage it 😉

favila22:02:15

e.g. google has postgresql now

favila22:02:25

we could switch to that if we weren't lazy

favila22:02:54

if datomic someday supports google's dynamo-alike natively we would switch to that

favila22:02:15

but there's other scenarios too

favila22:02:35

create a db on a laptop, backup, restore to a production storage to stand it up

johnj22:02:48

but dynamo is getting nicer features every year, like on-demand capacity mode recently

johnj22:02:08

have to go, really helpful, thanks!