Fork me on GitHub
#datomic
<
2019-01-14
>
m_m_m11:01:40

Hi. I'm trying to find some datomic benchmarks (on real cases). I am wondering how fast I can write msg into datomic (writes / s). I can only find "it is not so fast at writing", "writing is not the strongest point of datomic"...cool but this is closer to 10 writes / s or 10000 writes /s compared to other databases?

Dustin Getz13:01:11

Rule of thumb is keep database size smaller than 10 billion datoms, which if you reach that in a year is 317 datoms per second. At that rate you are very likely to be read constrained, not write constrained

Dustin Getz13:01:22

“datomic is not fast at writing” lacks sufficient nuance to be useful at all

Dustin Getz13:01:14

If you transact 10k datoms per second, you’ll have 10 billion datoms in 10 days

m_m_m13:01:02

Wow. Thank you. Now I see.

Joe Lane15:01:00

@m_m_m You’re not finding the results because of a few different reasons: 1. Its against the license 2. writing of different sizes, shapes, constraints, batches of data will yield different results. 3. It’s harder to compare to other systems due to query groups allowing for reads to come for free and letting your primary groups focus exclusively on writes. My understanding with datomic cloud is that if you run into a write performance constraint you can crank up the knob on dynamodb’s write throughput and use an i3.xlarge as your primary group. When to do that and what it will produce I can’t answer, as I haven’t had to do that. Can I ask what the usecase is?

m_m_m15:01:55

@joe.lane Sure, I would like to store items form multiple games. Those items have content and state which can be changed in time (the same with owner). Datomic can give me whole history of each of those items which is nice....but more important feature is easy access to the newest state of each of those items. Hmm is there a huge difference between performance of proverion and free version of datomic?

Joe Lane15:01:02

Well Free version of datomic is, I believe, in memory and on-prem. I was referring to datomic cloud.

m_m_m15:01:23

How about performance of the "on-perm" version?

Joe Lane15:01:41

My 1 and 2 are still relevant, but 3 falls off and so does the i3.xlarge/knob-cranking. I don’t have the experience with on-prem. Is the “access the whole history of each item” part of the game or just a development convenience?

m_m_m16:01:38

development convenience. There should be possibility to see history of the item, price changes, owner changes, stats changes in time 🙂

eraserhd18:01:47

Does anyone here have an on-prem Dockerfile, maybe with sqlite jdbc backend for CI from a known-good database?

johnj18:01:50

@m_m_m you shouldn't rely on datomic's history features to model your domain, its really mostly just an audit thing, add your own time as needed

m_m_m21:01:37

Thank you. Now I can see it more clearly. So each time when I would like to change one param in my item I have to create a new item with each of my params and change one of them ? This is a lots of unnecessary data.

lilactown19:01:01

@lockdown- I'm a bit confused by the author's "revision entities" example

lilactown19:01:43

how does it route around the fact that the tags don't exist in the previous versions?

val_waeselynck19:01:55

Author here. In the example, the tag would have been retroactively added to the previous revision when the intern backfilled the data manually. Of course, that's a business decision from the blog platform, they could have chosen not to backfill the tags.

lilactown19:01:38

hm OK. so walk me through what this does. - finds the version-t. how is this different than t? what is version-t supposed to represent? - finds version-eid. Is this the eid as-of the correct version for time t?

lilactown19:01:45

and then finally, the assumption is that the intern has gone back not only to the entity, but every previous version of the entity, and added the tags?

val_waeselynck19:01:47

> finds the version-t. how is this different than t version-t a domain-specific timestamp, representing the creation time of an Article Revision - nothing to do with Datomic's t. version-eid is the Entity Id for the Article Revision. Keep in mind we're not using Datomic's historical features at all in this code, that's the whole point - no as-of, no t, etc.

lilactown20:01:14

so each "revision" is it's own entity; when you make a change to the latest revision, would you transact a new entity? or do you transact against the existing entity?

val_waeselynck20:01:00

Normal usage would mean transacting a new Entity (after all, the whole point of a revision is to be immutable); Revisions would typically be updated in data migrations.

lilactown20:01:24

OK, I see. I was still stuck on using datomic's history API for tracking non-breaking modifications 😵 that explains it.

lilactown20:01:26

thank you!

👌 5
lilactown19:01:51

maybe I'm just not reading the queries correctly

dazld21:01:15

(def con! (d/connect "datomic:))
Syntax error (ClassNotFoundException) compiling at (form-init3668295776313994723.clj:1:11).
com.amazonaws.services.dynamodbv2.model.PutItemRequest
^ any ideas what the issue is?

dazld21:01:50

guess i’m missing a dependency somewhere

dazld21:01:05

(using a plain deps.edn with just datomic pro in there)

gordon21:01:53

you probably need the dynamodb dependency from the AWS Java SDK, like this: https://docs.datomic.com/on-prem/storage.html#dynamodb-aws-peer-dependency

❤️ 5
dazld21:01:03

thanks gws, that was exactly it

👍 5