Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2021-08-16

Channels

#datomic
<
2021-08-16
>
Jakub Holý (HolyJak)11:08:26

Not sure where to report, there is a typo at https://docs.datomic.com/on-prem/operation/capacity.html#multiple-databases : "database, or or colocating multiple databases" and "For sh, is is not uncommon"

jaret11:08:04

Thanks for catching this. I’ll fix it today.

👍 6
Jakub Holý (HolyJak)11:08:25

A question about multiple DBs and transactors. https://docs.datomic.com/on-prem/operation/capacity.html#multiple-databases: > When you serve multiple databases with Datomic, you have a choice of provisioning separate transactor pairs per database, or or colocating multiple databases on a single transactor pair. But I cannot find where to configure that. The only https://docs.datomic.com/on-prem/configuration/system-properties.html#transactor-properties or that can be https://docs.datomic.com/on-prem/overview/storage.html#cassandra is related to the storage cluster, nothing about individual DBs. I know I can start multiple peers / peer group and point each to a particular subset of DBs but I do not see how to run a number of transactors, each serving only a subset of DBs, and how to get the peers to connect to the correct transactor?

favila11:08:04

Transactor properties do not include a database

Jakub Holý (HolyJak)12:08:13

Yes, it seems so. So how should I understand this sentence: "you have a choice of provisioning separate transactor pairs per database"? 🙏

favila12:08:10

Make more transactor with their own storage

favila12:08:17

A connection string has storage + datomic-db info

favila12:08:41

You can have one transactor pair per storage but multiple dbs

Jakub Holý (HolyJak)12:08:59

I see, thank you. So in the case of Cassandra, I would need one Cassandra cluster per a set of DBs. That makes sense.

Jakub Holý (HolyJak)12:08:48

The Datomic on-prem license is "per system". Do you know what a "system" is? Is it essentially per one HA transactor pair?

favila12:08:04

No you could use the same Cassandra cluster but a different key space

👍 3
favila12:08:08

Re:system, I think so.

👍 3
prnc12:08:45

Hope you don’t mind that I try again. I’m seeing No space left on device - /opt/codedeploy-agent/deployment-root/.../bundle.tar types of errors during deployments DownloadBundle step. Seems like code deploy agent is running out of space for revisions archive. Is there a way to change the root volume size that’s attached to the compute instance (running 884-9095, on t3.small w/ the default 8GB storage)?

Jakub Holý (HolyJak)13:08:15

Any idea why I might be getting "Could not transfer artifact com.datomic:datomic-pro:pom:1.0.6316 from/to http://my.datomic.com (https://my.datomic.com/repo): status code: 401, reason phrase: Unauthorized (401)" even though I have added <server<id></id>... to ~/.m2/settings.xml as suggested by https://my.datomic.com/account ? 🙏 Could it be because my account is too new?

✅ 3
Joe Lane13:08:24

Are you using Lein?

Joe Lane13:08:51

Does it work now that you have ✅ ?

Jakub Holý (HolyJak)13:08:52

Found it, I misplaced the <server> section 😅

👍 3
Linus Ericsson13:08:06

Is there any open source code for the functionality of (selective) respooling datomic databases?

favila13:08:35

What is “respooling”?

Linus Ericsson13:08:32

selectively lift data from an old db to a new one (as mentioned in some talks from Nubank).

favila13:08:53

in a history-preserving way? or just copying?

Linus Ericsson13:08:15

history preserving but possibly rewriting/never transact certain things (GDPR use cases among other things).

favila13:08:50

The history-preserving method is usually called “decanting”. I’m not aware of any open source code for this and having written a few I’m not sure what it would contain--it tends to be very bespoke and depend on the data model

Linus Ericsson13:08:19

decanting is the word. Thanks for confirming that this tends to be use-case specific. This means we don't waste our time thinking about if with the ambition to build something from scratch.

favila13:08:23

the basic technique is read the tx log in order, transact that into the target db, and retain somewhere a mapping of eids in the old db to the new id as you go.

Jakub Holý (HolyJak)13:08:36

I guess https://github.com/fulcrologic/datomic-cloud-backup/ could be a good starting point? It also supports filtering/rerwriting data so that prod data can be used in a stage env in a privacy-friendly way

👍 3
Linus Ericsson13:08:47

Thanks for the pointers and hints. We will look into this for our particular use-case. The devil is in the details, as always.

Jakub Holý (HolyJak)20:08:30

Hi! If I understand it right, you can enforce integrity constraints on data but must remember to include the relevant :db/ensure on any transaction that creates/modifies the entity in question. How does that work for you in practice? Is it a non-problem or does it happen that a developer forgets to include it and makes a change that leads to invalid data (such as creating an employee without a (required) department)? Thank you!