The performance penalty is a la carte—when you add a composite tuple it’s up to you to “touch” all entities that don’t have it yet to populate it

The reason you cant change it probably just comes down to the inherent complexity of schema changes in a temporal db (what happens to the history view? To all old TX records?) and the philosophical stance Rich has against making the same name mean something different over time. His view: just use a new name and deprecate / remove the old one.

Notice that no schema changes which change type are allowed—tuples are not unique in that way

I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change

It doesn’t eagerly project it, it can’t repopulate it for you, and the data is in the same covering indexes as all other data

Thanks for the answer! But I'm still wondering, isn't there performance penalty in the "just use a new and and deprecate the old" strategy? I mean, if I have attributes :a, :b and :c, and a composite tuple a+b, which I then decide to deprecate in favor of a new composite tuple a+b+c, then whenever I'm changing the attribute :a or :b, Datomic will update the composite tuple a+b, even though it's deprecated.

> I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change Right... so it's not really a performance penalty, but penalty in storage?

Yes

Which you can mitigate by eg adding noHistory to the attr and removing any value indexes if you have it

If you really want it gone you need to create new component attrs also

Would you say then @U09R86PA4 that the storage cost of old unnecessary composite tuples shouldn't really be much of a factor in deciding between composites vs the other types of tuples, when a composite would otherwise work?

I would say that it’s rare that storage cost is a factor

I also wish you could “turn off” a composite tuple--i.e. signal to the transaction processor that it should stop updating it

composite tuples do something no other tuple can do: they know the effective value of the db at the moment right before committing the transaction datoms, so they can update composites to their correct value within that transaction even if the contents of the tx-data was uncoordinated

Got it! Yeah, I wish we could. Maybe that will come as a feature one day. It sounds like the type of schema change that could be allowed.

you can use a “normal” tuple and update it yourself, but you will have to be careful that you only prepare transaction data where you know what the final value will be when the tx-data arrives at the transactor, and that nothing else in the tx-data might alter that calculation.

but if storage cost is a concern, that’s what you gotta do

It’s not impossible--datomic didn’t have tuples of any kind for years. we were manually maintaining composite indexes as serialized strings

Storage isn't really that big of a concern in our case, I think. It was more like the bad aftertaste of having unused and unnecessary attributes getting asserted perpetually.

datomic doesn’t let you remove the cognitive burden of past mistakes. I think that’s the unspoken downside to the “no breaking changes” mantra

Yeah, though there's a difference between old/deprecated attributes in the schema and having values for them asserted on entities.

because of the history db and tx log, there’s always an assertion somewhere

Yes!

thanks Joe! does the code that is proxied to run in its own memory space? in other words, if my long running http-direct process has some state, say via mount, then there's no reason to expect that the proxied-to function can access that state, right?

(we tested this for fun and ruled it out, but i thought i'd ask anyway)

There is no distinct “memory space” so your ion should be able to access any state that is correctly instantiated. Http-direct and lambda proxy’s both call ions. As long as the state is present on all cluster nodes you should be able to access it. however mount can add it’s own challenges due to piggy backing on ns loading. I can usually get by without a framework and instead just wrap my state in a delay and initializing it on the first access (what delays are for).

very interesting, thanks again

Hi 🙂 in lieu of a more relevant response from someone else, you may be able to borrow and adapt some code from Crux https://github.com/juxt/crux/tree/master/crux-kafka-connect

Hi, that’s the plan 🙂 (if “someone else” doesn’t show up) haha

Awesome 😄