What are the procedure and "costs" of migrating from one storage backend to another? e.g. between postgres and Cassandra or vice versa?

@matan you can restore from a backup on a new storage backend afaik

Yep, I guess so. I assume all functionality will be maintained, except that moving from a consistent (postgress) storage to an eventually consistent one (cassandra default configuration) will introduce application bugs if the application assumed strict consistency before.

@matan What is your reasoning? datomic hides the inconsistency

@matan application should be completely unchanged by the choice of storage for datomic

@matan Datomic only mutates a tiny number of records. everything else is immutable, so no opportunity for inconsistency

@favila last I asked on the mailing list, it was my understanding of the response I got, that query results will be different based on which cassandra node answered to datomic ― and as I recall the default setup scenario with cassandra is to "commit" a change before all cassandra nodes have updated (maybe I am wrong there) ― I think that with high throughput/activity, I might get different query results depending on which cassandra node answered to datomic

I see that mailing list thread

the results will not be different

To be honest I am not sure I got the correct bottom line on that thread

they may merely be not there yet

no quorum is needed

only one server with a record

what may happen is that NO server available to you has the record (in case of network partition)

(you = peer)

that would be some kind of failure or retry, but the application would not keep going silently with different results

So from an application point of view, say I query for all movies (borrowing from the tutorial minimalist scenario), and a movie was added, and it is not there yet when datomic runs the query, on the cassandra node that was used by datomic for this query.

Say the movie was added by a user on a front-end part of the overall application

If the application can't retreive the movie, this will reflect a wrong "world" from the user's point of view

it would not be an alternate world

Maybe I should revisit after finishing with the tutorial, which I'm halfway through

(agreed, not alternate, only stale in a confusing way for the user)

that's not a function of storage

that's just speed of light

if one peer writes while another peer queries, the querying peer may not know about the latest transaction

but its view of the world will not split

it will have an exact perfect snapshot of the world at the moment the query started

just a few ms behind or whatever

Okay, right, I easily flow with the timeline metaphor

As an aside, I should de-complect what the user tells the app, and what the app agrees to enter into the world

that's not really it

the key is there is only one transactor. the transactor writes and informs peers of the latest t

the latest t is like a pointer to a shapshot

oh, right, so a query coming after the data transacting, would not need to go all the way down to the storage layer, it will get the latest as long as the transactor already finished updating it about the transaction? is that it?

yes but that's an optimization (index-merging on the peer)

the point is once the peer knows about a T, that T is guaranteed to exist in storage

if the peer asks for it and it isn't there, it knows that there is a read failure

furthermore, that T never changes, so there is no chance that an eventually-consistent storage has different values for the same T

I should revisit after being done with the tutorial and having a complete grasp of datums, attributes etc.. the data model. Will be back at that point...

@favila many thanks so far

@matan this may interest you if you want to know more about internals: http://tonsky.me/blog/unofficial-guide-to-datomic-internals/

Well I think I get it. So reading the most current data boils down to chasing the latest time handle (if searching for an entity) or having the right transaction id at hand to begin with, or waiting for them to arrive courtesy of the transactor. Framed as such, the notion of "consistency" is reduced to an easy to satisfy definition, one that is closer to the ACID definition than the CAS one.

Which does not make an application using datomic behave consistently without some effort in the form of judicious use of datomic api. I can live with that, maybe, in exchange for not using SQL nor a lame NoSQL database.

MongoDB scales though

lol

matan: if you’re worried about a client reading their own writes, you can use d/sync to make sure you read the basis t of the last write from that user

@matthavener with postgres as a backend this won’t be an issue, right?

only with an eventually consistent store such as dynamo?

@matthavener yep, so I came to gather, thanks.

afaik, you can lag reads on any type of storage

if you write with peer B and read from peer A, there’s no guarantee that the transactor has updated A with the basis T that was just transacted from B

(hence, d/sync, which would allow you to force A to sync to some T)

@matthavener but from within the same peer, there is guarantee, right?

I don’t know if that’s a guarantee datomic makes, but it would seem reasonable to me 🙂

if its the same peer, you can read the value of d/transact‘s future to get the db-after which has that guarantee

By the way, quick unrelated question:

If I were to write a service whose purpose is to follow the transaction log through the log API and update some internal state

how can I make sure that it doesn’t miss a transaction?

I don’t think I can rely on transaciton IDs being sequential or anything like that

is it possible to get, at every point, the transaction ID that was immediatly preceding it

so my service can keep track of the “latest transaction ID it processed”, and before processing a new transaction it can check that the “latest tx ID” it has recorded matches the “previous transaction ID” of the transaction he’s about to process

and if not, it would request the log again from the “latest tx id” it has

if that makes any sense

TL;DR; how do I reliably write a service that follows the transaction log, ensuring that it doesn’t miss any transaction

@hmaurer have you looked at the docs e.g. http://docs.datomic.com/log.html ?

@hmaurer @matan Again, I don.t see how choice of storage affects anything

datomic acts acid even with eventually-consistent storage

Thanks, that's clear by now. The ACID definition of consistent is easy to accomplish in this architecture and the paradigm implied by the time-oriented API.

Going through the unofficial internals doc suggested above, my thoughts are twofold: 1. Most these things should have been on the official docs, right after the introductory parts. It gives a sense of what performance to expect in different scenarios, and thus whether or how to use datomic for a scenario. Maybe they are already mentioned there. 2. I think datomic has the upper hand on data modelling compared to what else we have out there, but possibly at a dire cost of being prohibitively slower than other options for some standard scenarios, due to all the translation taking place for weaving transactions into non-transactional storage, the unoptimized (?) nature of datalog v.s. SQL, and involving external storage layers which act as databases not just storage, thus incurring additional overhead.. I'd be really happy to see some intelligent benchmarks or refutations of the assumptions sprinkled in this quick note.

“prohibitively slower” and “standard” both would need definition

there is no doubt that SQL query engines have decades of clever performance optimizations

and it almost feels like cheating to win some read scenarios via the architectural advantage of immutability + multi-tier caching

@matan in Datomic, external storages act as block stores, not as databases

@matan Datomic happily accepts the write overhead needed for ACID transactions, and the application fits and misfits this implies

@matan I think the important thing missing from your note is the horizontal scaling and caching advantages enabled by peers + immutability

@stuarthalloway from your experience, has there ever been cases where the single-writer process becomes a performance problem? And if yes, what would be the recommended way to deal with this?

Over the top of my head I thought the application could then be split into multiple databases, which might allow the transactor to operate in parallel

although I am not sure it does

@hmaurer sure! Datomic is not a fit for write scale, as the FAQ states: http://www.datomic.com/faq.html

@stuarthalloway Yep I saw that, and I wouldn’t hold Datomic guilty for not dealing with large volumes of write. I am just wondering if there are potential workarounds

@hmaurer You can split the application into multiple databases with a separate transactor for each. Remember that peers can query (with join!) across databases, and peers do not care what transactor they come from

Yep that was my feeling when I discovered peers can join across databases

great 🙂

Thanks for Datomic by the way, it’s making me enjoy databases again

Also I might just be delusional but I feel I understand its internals much better than other databases I have used before, which is comforting

If you are looking at that kind of scaling trick, please stay in touch here and/or on the mailing list. Happy to help vet and bench your ideas.

Or that, even if I do not understand its internals, I won’t need 10 years of experience to understand them if explained properly

Great, thanks! I don’t have any practical application for which I would need to scale writes though; it was just out of curiosity

Cheers!

Ah, quick other question while you are here @stuarthalloway 🙂

Is there a technical reason why Datomic isn’t accommodating for custom/community-built backup and storage drivers?