sounds good. i'll check out those sql libraries. thank you!

Best to ping @U05120CBV and @U1QJACBUM for that, I’m not on Datomic team

Sorry, I that’s because I watched REPL-driven development yesterday 🙂 You were declared the Grand Master of Documentation.

Thanks @U06B8J0AJ we’ll get on these

Fixed. Thanks for letting us kno.

know

> Since schema txns are idempotent, I could just stop using it and re-transact on every restart That's my default approach as well, but keep in mind that you may also need to migrate data in addition to schema (e.g populating a new attribute with a default value), and more generally some migrations that are not idempotent. Those are the reason I still use something like Conformity (see also Datofu: https://github.com/vvvvalvalval/datofu#managing-data-schema-evolutions)

Could I just manipulate the :db/txInstant attribute?

the specific docs regarding tx/Instant are here https://docs.datomic.com/on-prem/best-practices.html#set-txinstant-on-imports

The latter. The great thing about On-Prem is you get to set up storage yourself, exactly the way you like it. The terrible thing about On-Prem is you have to set up storage yourself, exactly the way you like it. 🙂

Some ideas here: https://github.com/vvvvalvalval/datofu#implementing-ordered-to-many-relationships-with-an-array-data-structure. You can also consider linked lists. Don't bother about being idiomatic, but you should consider the read and write patterns to choose.

If you don't mind, why are you recommending datomic? What features do you need of it?

Excision seems like an extreme solution for this problem. Have you considered moving some attributes to an auxiliary store?

Yeah one of datomic’s key value props is the history etc. If you’re anticipating excision at the outset, might not be the best fit or datomic + something else..

Going back in time at an entity level is a key feature my client needs. It would preferably keep all history, but given that there is a bit of a ceiling on datoms, I’d like to understand ways of mitigating the problem of having 10 billion datoms within a year.

I have considered moving them to a different store, but at the very least, I’m talking about 1 billion entities within one year, with each entity having ~10 attributes

Fair enough. I think the 10 billion thing is a matter of what they've tested out to. Perhaps some testing at your expected scale is in order

Is a soft limit, but I understand performance will degrade greatly

Forcing you to create more DBs, or move to a completely different DB

@U1ZP5SMA6 on-prem or cloud?

@U0CJ19XAM We would be open to either. We are on AWS now, but hope to move towards GCP in the next 2 years or so. As such, I imagined using on-prem.

Also dbs are "cheap" afaik. Can you potentially use them as a sort of partitioning?. I think the limit is db not storage

Unfortunately, it doesn't seem that you can join dbs in queries yet for Cloud.

Thanks for the help. There is a good chance that there are such partitioning schemes that can work for us.

Ah hell @U06B8J0AJ really? I missed that. Going to have some scenarios where I'll need that soon

That's what I understood. @U05120CBV?

‘performance will degrade greatly’ is not accurate

performance of what? writes? reads? the specific behaviors of “very large” databases will depend greatly on the data model and access patterns

the “dbs are cheap, make more” advice is good, but only true for Cloud. If you’re using on-prem you should have one transactor per operational DB

also, correct, there is currently no cross db join using the client

depending on ‘how’ you want to use multiple DBs you may be fine to query the individual dbs separately then join your application

@U05120CBV thanks for some of that clarification. I’ll definitely see if it’s possible to shard into separate databases. I expect regular updates to ~50million entities, each with about 10 attributes. My math shows being at 10 billion datoms within a year, with a fairly static number of entities. The ability to go back in time is very desirable for my use case, but only in recent months. If the database can’t support that scale, I’d be willing to excise old data, so long as that data isn’t the most recent for the entity.

Do you have any recommendations on how to go about capacity planning given that use case?

Don’t use excise for size control

that’s not what it was designed for, and it will not work well for that use case

broadly, I’d recommend sharding by time with a rolling window

create a new db every X months and only write to it

keep the ‘older’ db around long enough to support your queries against the older data

Perfect. Good call @U05120CBV. Appreciate the feedback

if you “need” to handle entity continuity across the DBs you can write a tool that “moves” some subset of active entities from the old db into the new one

if you can get away with just moving all writes to the new db and leaving the old db(s) there for read, you can probably get away with a single transactor

the transactor-per-db rule is largely based on write-side behavior

so if you can move all your writes to the new db, you’re usually pretty OK to have one or two older read-only dbs in the same system

@U072WS7PE or @U05120CBV, could any of you guys help here?

you should be able to launch the console against Cassandra the same way you do against any storage

use the cassandra storage URI

You mean this URI: datomic:cass://<cassandra-server-host>:<cassandra-port>/<cassandra-table>?user=<usr>&password=<pwd>&ssl=<state>? If so, this is what I’m using. The thing is <cassandra-server-host> is a “cluster gateway”, if we can say so, which has it’s own API to retrieve the actual cluster nodes. This is why I’d like to use the same cluster callback I’m already using with the transactor. Is it possible?

@U05120CBV?

I don’t believe you can specify a cluster callback using the console

i’ll have to look into it

Ok, @U05120CBV. May I ping you again, later this week?

sure

Hi, @U05120CBV. Any updates here?

One more thing: does the cluster callback gets used by the commands like backup-db and restore-db?

@U072WS7PE or @U05120CBV, could any of you guys help me?

@captaingrover your best options right now are:

in Cloud: expose a lambda or REST service via API Gateway

On-Prem: expose your own REST service from a peer

@U072WS7PE hi Stu, yeah i'm using On-Prem. When you say your own REST service you mean wrap some http around the client library as opposed to using the bin/rest standalone service?

@captaingrover around the peer library would be better, no process hop