I was considering to deploy a transaction function to do that, but I wasn't so sure about not interfering with my transactor. Any advice is welcome

1. make normal operation use old and new style; 2. backfill the old style; 3. make normal operation use only new style; 4. retract the old style (optional)

if you can insert an abstraction layer between the application and the database, you could probably produce your new style on-the-fly from the old style. If so, you can cut or simplify some of these steps

We're already on step 1

Now we're trying to figure out what would be the best approach to accomplish the backfill

best approach in what sense?

we tried to walk through the indexes, retrieve the old style, the transact it back, doing what we needed via :db/cas

but it proved harmful, since it took a toll on our write throuput

how large were your transactions?

500 swaps each

did you retract also?

not yet

so about 500 datoms per tx? the rule of thumb is ~1000, that may help. you can also not pipeline if you are worried about working too fast

What is your estimate of how many datoms you need to commit?

the best would be to convert the whole base

which is over 3B datoms

you have no other datoms than these?

in this particular database, no

well, then writing 3b additional datoms is unavoidable

plan your write capacity accordingly?

if this database is really this simple, perhaps you could copy it to a new database?

then switch the application over

I really liked the idea of copying the database

seems to be the most practical one

do you care about transaction history?

i.e. preserving transaction times and metadata

for a portion of it, yes

I'll discuss it with our team

ok, this is frought with edge cases, but you can do something called “decanting”

you read the transaction log and transform it before transacting each transaction to a new db

think of it as a git-rebase

I was just thinking about it

the advantage is only that you avoid having a single db with old+new style sum of datoms

you could also perform most of it offline (writing into a dev database if you are on-prem), then backup+restore to get it into production

you will need some catchup and downtime to switch the application over, though

thats manageable

if you can afford it, it’s probably better to just go the way you are going now, honestly

I assume you’re using dynamodb with provisioned write capacity?

yes

your call, but bumping it up for a few days will probably be cheaper than the engineer time to get a decant running smoothly

Well, I think we have a few paths to consider now

I much appreciate your insights

thanks a lot

glad to help

No

It seems like a useful addition 🙂 I'd like to express "all entities in a card many attribute must be unique by a particular other attribute."

could you add that invariant to the attribute itself and ensure it with an attribute predicate?

“the attribute itself” = a normal cardinality-many ref

Perhaps. It's not clear that enough context is passed to the predicate to be able to check something like that.

(looking at https://docs.datomic.com/cloud/schema/schema-reference.html#attribute-predicates)

ah, yeah, attr predicate may not work

db/ensure will though

Yeah, looks possible with an entity predicate. It'd be "built in" if composite tuples supported reverse lookups though 🙂

the semantics are a bit unclear though

Even if the card many ref is required to be a component?

I’m not sure I follow?

a composite tuple is non-homogenous, so which backref is it following?

it puts you in a weird situation where :eavt and :vaet are inconsistent

I think I'm missing something. What's the inconsistency?

Suppose you have a composite tuple :foo+bar consisting of refs :foo and :bar from the same entity

:eavt entries will be [123 :foo 456] [123 :bar 789] [123 :foo+bar [456 789]]

if :_foo+bar worked and brought you to 123, from where could you follow it, and what would the :vaet index look like?

I think I see what you're getting at but I was after something different. I see how my question was poorly worded. I'm after something like this:

[{:db/ident       :user/addresses
  :db/valueType   :db.type/ref
  :db/cardinality :db.cardinality/many}
 {:db/ident       :address/type
  :db/valueType   :db.type/keyword
  :db/cardinality :db.cardinality/one}
 {:db/ident       :address/user+type
  :db/valueType   :db.type/tuple
  :db/tupleAttrs  [:user/_addresses :address/type]
  :db/cardinality :db.cardinality/one
  :db/unique      :db.unique/identity}]

i.e., a particular user can have many address types but only one of each type.

why not :user/address_type1 :user/address_type2 etc?

is the type open?

Yes

But you're right - if it was closed I'd go that route.

ah, ok, you want this index to keep an invariant

I think I get it now

wait, I still don’t get it

how would your attribute ever not be unique?

your hypothetical composite attr

If you try asserting one that already exists?

?user :user/addresses ?address has two ?address with the same type?

That would fail or upsert (if :db/id is specified) on transact

is that the scenario you are trying to exclude by this index?

Yes

Having multiple of the same type in this card many coll would result in undefined behavior for us. The card many must by "distinct-by" the :address/type.

ok I can see this making sense as a feature. I completely misunderstood what you were asking for originally. Although I think db/ensure is the right way to tackle this.

you could also introduce another joining entity

Yeah, sorry... Reading my question again I realize how ambiguous it is 😬

I thought about that. It feels a bit gross though...