Fork me on GitHub
#xtdb
<
2021-12-07
>
emccue02:12:03

Migration question - what is the easiest way to simulate created at and updated at and can I do it with one query/roundtrip to the db

refset10:12:38

Hi 🙂 so, something roughly like this you mean https://gist.github.com/refset/8439ceebf3c1f08c614b0c818266a140 ? Or are you asking about translating these times across a migration from another system to XT?

emccue15:12:13

what you shared

emccue15:12:17

how does that work though?

emccue15:12:06

is that something i need to ::xt/put a transaction function for? (I’m fine for simple stuff - we already accept sql triggers for geting updated-at0

refset16:12:23

great, you just need to have those functions loaded in a namespace somewhere (in the gist I was working in (ns dev)), see https://docs.xtdb.com/language-reference/datalog-queries/#custom-functions

emccue16:12:13

just a sidenote, idk what part of my brain itches it would scratch to have these be explicitly sci, but its there

refset16:12:15

aha, well that's definitely possible! Perhaps you can use this other gist (which abuses eval) for inspiration https://gist.github.com/refset/a7f86c8a3ad34aea5d20eac0b5720a28

emccue17:12:39

all uses of eval are intrinsically abusive

🤐 1
emccue18:12:11

> Or are you asking about translating these times across a migration from another system to XT? This would also be part of it, but i think I could derive some of it by saying “this is what i knew about the record at this point of time

emccue18:12:51

like if the state of the table right now is

name: 'bob'
status: 'completed'
created_at: Jan 1st
updated_at: Jan 4th;

emccue18:12:51

and i have a second table

status_from 'started'
status_to 'in progress'
created_at Jan 2nd
updated_at Jan 2nd

emccue18:12:23

status_from 'in progress'
status_to 'completed'
created_at Jan 3rd
updated_at Jan 3rd

emccue18:12:53

then i could represent it by the sequence of records

{:xt/id {:old-table/id 123}
 :site/type :old-table
 :old-table/status 'started'} <- Valid Jan 1st

{:xt/id {:old-table/id 123}
 :site/type :old-table
 :old-table/status 'in progress'} <- Valid Jan 2nd

{:xt/id {:old-table/id 123}
 :site/type :old-table
 :old-table/status 'completed'} <- Valid Jan 3rd

{:xt/id {:old-table/id 123}
 :site/type :old-table
 :old-table/status 'completed'
 :old-table/name 'bob'} <- Valid Jan 4th

emccue18:12:55

since i dont actually know what the name was before the newest record, that info was lost

refset22:12:44

ah interesting, so there's already a ~bitemporal schema in some RDBMS?

emccue22:12:58

for percisely that single field, not any of the others

👌 1
emccue20:12:27

One of my old roommates works on https://www.fluvio.io/ and, with kinda pretty little knowledge of how xtdb uses kafka, i was wondering if it might make sense to use it instead of kafka for xtdb

👍 1
refset22:12:32

Nice, I hadn't seen Fluvio before but it sounds promising! Looks like it supports the two key properties required to implement the tx-log protocol, namely (1) infinite retention and (2) linear ordering via a single-partition The tx-log protocol itself is relatively small, should someone want to give it a go - this is the entire Kafka implementation: https://github.com/xtdb/xtdb/blob/61e6a8eb07a87dcff8e65ad4be97c75346a9c86f/modules/kafka/src/xtdb/kafka.clj#L170-L197