Fork me on GitHub
#xtdb
<
2023-09-08
>
Samuel Ludwig20:09:16

Additional, separate question: In terms of performance, are certain types preferred for :xt/ids over others? In a typical RDBMS, like MySQL, sequential integer ids are preferred for indexing/searching purposes. Would there be a similar performance payout for sequential int-ids in XTDBv2, over say, UUIDs?

jarohen21:09:50

Hey @U0482NW9KL1 - yes, but the other way around, well-distributed UUIDs are the preferred ID type in v2. Our primary covering index is hash-oriented, so we can narrow searches-by-id down faster this way.

Samuel Ludwig21:09:16

ahhhh thats interesting, that's convenient actually!

seancorfield21:09:21

In a document-based system, where you have to PUT a whole new document for any change, does that mean that if you have a "large" document that has one or two fields that change frequently, you'll get a lot of duplication of the unchanging fields? If so, is that mitigated in v2 with the columnar storage?

jarohen22:09:21

short answer is yes - it'll get duplicated. but also yes, the columnar compression should (eventually) be able to mitigate this, particularly as we compact our logs to be sorted first by eid, so different versions of the same entity will gravitate together in the primary index. that said, I'd also recommend that, if you know you're likely to have these kinds of update patterns, that you take advantage of the new tables by, where possible, splitting documents into faster and slower changing attributes. those documents can still have the same :xt/id values - these are now only required to be unique within a table.

seancorfield22:09:58

Thanks, that was my intuition about it but I wanted to double-check. We have some member profile data that changes "a lot" and some that is much longer-lived, so that'll be something for us to consider if we move any of this stuff to XTDBv2 at some point.

kokonut19:11:38

How can multiple documents have the same :xt/id? I thought it must be unique like a primary key for each document. But reading above comments, I should be wrong. Can someone let me know where I can find the relevant part of the official documentation?

jarohen22:11:09

@U022N1DU7GQ in XT 1, that's correct - :xt/id is a primary key. in XT 2, we have tables - essentially different bags of documents - and it'll be that :xt/id only has to be unique within a table (same as a relational database)

kokonut22:11:39

Wow, I see. Excited to see XT2.

Daniel Craig22:09:59

is there a name for the concept of being able to run a query today and run it again tomorrow with the same result?

jarohen22:09:50

FWIW we call it "query repeatability", but to a certain extent it's the same thing as functional "referential integrity", with the current time effectively being passed as a parameter rather than being an implicit side-effect

Daniel Craig22:09:18

oh I love that explanation, thank you

πŸ™ 2
Daniel Craig22:09:23

I'm preparing a talk on how we're using AWS Neptune to answer questions about the future state of the airline where I work, and I want to acknowledge XTDB in my talk

jarohen22:09:37

ah, interesting - give us a shout when that's released (if it's a public talk, of course!), will have a watch πŸ™‚

Daniel Craig22:09:51

Certainly 😁

Daniel Craig21:10:39

It was my first public talk 😁

refset09:10:08

awesome! congratulations - and thanks for sharing here πŸ™‚

refset09:10:20

Really interesting use case too - future-time queries via Gremlin...I bet there's quite a few orgs wanting that

Daniel Craig14:10:27

Thank you for the kind words!

☺️ 1
genekim17:10:51

@USDPTD3FY Congrats on your fantastic presentation! The Southwest crew scheduling system meltdown is featured in my new book, Wiring the Winning Organization that comes out in a month, as one of the most astounding examples of a control overlay being unable to keep up with the performance environment. What an incredibly important capability you’re helping create for SWA!

❀️ 3
Daniel Craig18:10:04

Thank you for watching my talk and for your kind words!

Daniel Craig18:10:07

Credit to the larger team, and to leadership, we are responding to our failures and we are taking this opportunity to mature our toolset so that the guarantees that we maintain to our customers are always upheld

Daniel Craig18:10:00

Your words are a huge compliment! thank you