This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-07-01
Channels
- # announcements (53)
- # babashka (27)
- # beginners (103)
- # biff (1)
- # calva (21)
- # cider (5)
- # circleci (12)
- # clj-kondo (7)
- # cljfx (3)
- # cljsrn (9)
- # clojure (25)
- # clojure-europe (21)
- # clojure-nl (1)
- # clojure-uk (21)
- # clojured (1)
- # clojurescript (49)
- # cursive (20)
- # datomic (33)
- # events (3)
- # fulcro (39)
- # graalvm (20)
- # graphql (2)
- # introduce-yourself (1)
- # jobs (14)
- # keechma (3)
- # lsp (34)
- # malli (18)
- # meander (15)
- # off-topic (30)
- # polylith (10)
- # re-frame (21)
- # releases (2)
- # remote-jobs (5)
- # sci (10)
- # shadow-cljs (16)
- # tools-deps (2)
- # vim (6)
I've understood that composite tuples can't be deregistered, is that right? I mean, if I add a composite tuple a+b, but later notice that I need to add a new tuple (or well, triple) a+b+c and the tuple a+b is becomes useless, Datomic still keeps updating the a+b tuple and it can't be deregistered. I assume there's some performance penalty to have unused composite tuples. Because of this, we have avoided using composite tuples if we have a doubt that we might need to change it in the future. Is that a valid reason to avoid them? Or am I overestimating the performance penalty of updating composite tuples?
The performance penalty is a la carte—when you add a composite tuple it’s up to you to “touch” all entities that don’t have it yet to populate it
The reason you cant change it probably just comes down to the inherent complexity of schema changes in a temporal db (what happens to the history view? To all old TX records?) and the philosophical stance Rich has against making the same name mean something different over time. His view: just use a new name and deprecate / remove the old one.
Notice that no schema changes which change type are allowed—tuples are not unique in that way
I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change
It doesn’t eagerly project it, it can’t repopulate it for you, and the data is in the same covering indexes as all other data
Thanks for the answer! But I'm still wondering, isn't there performance penalty in the "just use a new and and deprecate the old" strategy? I mean, if I have attributes :a
, :b
and :c
, and a composite tuple a+b
, which I then decide to deprecate in favor of a new composite tuple a+b+c
, then whenever I'm changing the attribute :a
or :b
, Datomic will update the composite tuple a+b
, even though it's deprecated.
> I think you may be thinking of composite tuples as a pure ephemeral projection of "real” data like an index in a relational db. That’s not really how it’s implemented in datomic—it’s more like actual data that the transactor automatically updates when it notices it’s components change Right... so it's not really a performance penalty, but penalty in storage?
Which you can mitigate by eg adding noHistory to the attr and removing any value indexes if you have it
Would you say then @U09R86PA4 that the storage cost of old unnecessary composite tuples shouldn't really be much of a factor in deciding between composites vs the other types of tuples, when a composite would otherwise work?
I also wish you could “turn off” a composite tuple--i.e. signal to the transaction processor that it should stop updating it
composite tuples do something no other tuple can do: they know the effective value of the db at the moment right before committing the transaction datoms, so they can update composites to their correct value within that transaction even if the contents of the tx-data was uncoordinated
Got it! Yeah, I wish we could. Maybe that will come as a feature one day. It sounds like the type of schema change that could be allowed.
you can use a “normal” tuple and update it yourself, but you will have to be careful that you only prepare transaction data where you know what the final value will be when the tx-data arrives at the transactor, and that nothing else in the tx-data might alter that calculation.
It’s not impossible--datomic didn’t have tuples of any kind for years. we were manually maintaining composite indexes as serialized strings
Storage isn't really that big of a concern in our case, I think. It was more like the bad aftertaste of having unused and unnecessary attributes getting asserted perpetually.
datomic doesn’t let you remove the cognitive burden of past mistakes. I think that’s the unspoken downside to the “no breaking changes” mantra
Yeah, though there's a difference between old/deprecated attributes in the schema and having values for them asserted on entities.
my understanding is that the lambda produced when deploying an Ion is really just a proxy to code running on the compute or query group. does that mean that memory allocated to the lambda via the lambda configuration is less consequential than a typical lambda?
thanks Joe! does the code that is proxied to run in its own memory space? in other words, if my long running http-direct process has some state, say via mount
, then there's no reason to expect that the proxied-to function can access that state, right?
There is no distinct “memory space” so your ion should be able to access any state that is correctly instantiated. Http-direct and lambda proxy’s both call ions. As long as the state is present on all cluster nodes you should be able to access it. however mount can add it’s own challenges due to piggy backing on ns loading. I can usually get by without a framework and instead just wrap my state in a delay and initializing it on the first access (what delays are for).
Hi. Is there a Datomic Connector (Source and Sink) for Kafka?
Hi 🙂 in lieu of a more relevant response from someone else, you may be able to borrow and adapt some code from Crux https://github.com/juxt/crux/tree/master/crux-kafka-connect
Hi, that’s the plan 🙂 (if “someone else” doesn’t show up) haha