Fork me on GitHub

Am new to Datomic but can someone please clarify these statements: Datomic is read scalable but not write scalable. What does that mean? And why it is not best for website like twitter? And Also from Professional Clojure book: It took inspiration from multiple sources, but its basic goal is to be a drop-in replacement for the main cases where you would otherwise use a relational database as a transactional store. Stuart Halloway says that it’s targeted at the ninety-six percent use cases of relational databases, *leaving off the top four percent of high write-volume users like Netflix, Facebook, etc. * ?


@simon from the docs:


Datomic is a single-writer system. A single thread in a single process is responsible for writing transactions.


it serializes all writes


twitter needs to support massive writes


@simon I can perhaps try to clarify the second statement, having written it originally ;)


Although I was mostly paraphrasing various claims from the Datomic team. The essential idea is that it's for the majority of business systems that would traditionally be powered by a transactional SQL database. But it's not designed for extremes of scale in terms of total data volume or write throughput of the kind that top consumer-facing sites like Twitter face

Drew Verlee12:06:25

Though, i doubt twitter started needing anything near the write throughput they have now, or would have gotten their if they did 🙂. Somewhat off topic but i just finished re-reading the first chapter of designing data intensive applications. Which talks about twitter and very soon after gives this advice: > Approaches for Coping with Load. Now that we have discussed the parameters for describing load and metrics for measuring performance, we can start discussing scalability in earnest: how do we maintain good performance even when our load parameters increase by some amount? > An architecture that is appropriate for one level of load is unlikely to cope with 10 times that load. If you are working on a fast-growing service, it is therefore likely that you will need to rethink your architecture on every order of magnitude load increase—or perhaps even more often than that. The question then becomes, once you outgrow (your perf start to dominate the complexity of your system) something, how easy is it to transition? I feel like the clojure community at large (though i can’t speak to datomic) approaches these issues in a very healthy way due to the focus on compostability and immutability.

Dustin Getz21:06:35

You give me twitter sized bags of scale-money and i will scale Datomic Cloud to twitter write throughput by sharding transactors, without sacrificing query expressiveness 🙂

Drew Verlee12:06:00

Somewhat tangentially related. Does anyone have any insight into if with data-sync would play nice with datomic ions? I’m also just generally searching for more insight into data-sync (a component of one implementation of re-posh). I need to re-consult the docs but it seems like a fairly complex topic so i’m curious if anyone has any insights.


Sure, the question should be "do I have the scale of something like Twitter/Facebook/etc", not "might I at some point have that scale?"

Drew Verlee13:06:35

Yep. Iv have been dealing with a system that was pre optimized and it's hard to move it towards are current goals. I think it's hard to understand the trade offs, which is why it's nice when your ecosystem supporters decoupling. The creators had good intentions, but they felt compelled to think ahead because they had felt the pain of going at it from the other direction


Has anyone successfully set up a VPC endpoint as described in It fails for me with "No export named datomic-demo-VpcEndpointServiceId found. Rollback requested by user."


Hey Everyone, I’ve been going through the ions tutorial and I’m currently stuck. Granted, I don’t have a lot of familiarity with tools.deps.alpha. However I’m currently getting access denied on calling clojure or clj at the command line:

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.datomic:ion:pom:0.9.7 from/to datomic-cloud (): Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied
Anyone seen this before?

Alex Miller (Clojure team)22:06:39

Do you have aws creds set?

Alex Miller (Clojure team)22:06:00

End vars for keys or profile etc?


Hmm, give me a sec


socks proxy is running


and env vars are set in the shell I’m currently in


Just tried running aws configure. For the avoidance of doubt, the keys currently set are:

export REGION=
export AWS_REGION=


I’m trying to run either clojure or clj, which I installed via homebrew.

Alex Miller (Clojure team)22:06:59

Try setting AWS_REGION too


Did both REGION AND AWS_REGION and no luck


ok so the exception is an access denied.

Caused by: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
Should I be authenticating something else?


Bastion is working, as I can successfully call: curl -x .$DATOMIC_SYSTEM.$


I worked it out. So going through the docs you end up making two users, a datomic user account and a secondary account which is implied for development. I was under the assumption that because I was doing development, I should be using the secondary account with fewer permissions. When I swapped to the datomic account, it worked perfectly.


Thanks for the help @alexmiller!