Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-12-22

Channels

#xtdb
<
2022-12-22
>
lepistane17:12:53

Hello i i have a question. I am doing a lot of stuff with statistics and a friend of mine suggested Dynamo DB but the more i read about it the more XTDB felt like a right tool for the job. Does anyone have experience with both that can tell me which way to go? Needs are quite basic 1. learn a tool in depth 2. solution needs to be able to scale. I am analyzing multiple APIs at the same time and aggregating statistics for specific sports. football for example. Friend suggested to use Dynamo because of it's ring architecture and highly available. I know that XTDB is a single node and could become bottle neck. Apologize if question is too vague and basic i am not 10x developer (yet) i will answer any additional questions you have.

refset17:12:43

Hey @U45SLGVHV I guess the first thing to make clear is that Dynamo and XT are very different systems 🙂 What sort of scale are you envisaging? e.g. Loading TBs of existing on day one? Efficiently tracking / reacting to lots of fast-changing data to power end-user dashboards? Or is it more of a hypothetical "in case the app gets really big really quickly due to lots of users" concern? If you're mainly doing analysis of upstream data (rather than designing your own schemas) then I would suggest working backwards from the analysis tools you are already confident working with to evaluate the options. Like, are you already familiar with SQL? Or are you already writing analysis tools in Clojure? If you're thinking of Dynamo, are you also looking at PartiQL?

lepistane17:12:55

I have to run now and i don't understand some of the things you asking me, first time i hear about PartiQL so i will have to check it and others so that i can provide nice answer. Hope i can do it in couple of hours or tomorrow

blob_thumbs_up 1
lepistane11:12:07

Efficiently tracking / reacting to lots of fast-changing data to power end-user dashboards
This is a possibility because stats are updated live right now there are dozens of consumers and there were no issues but there is a path where consumer numbers could get into thousands. Data is schema less, it does follow some patterns but it's different enough and it changes from sport to sport, season to season where i think schemas would just slow things down. I am familiar with SQL. Only reason i am thinking about dynamo is because friend suggested. PartiQL seems quite advanced i am not ready to take that leap yet.

refset12:12:28

My own naive understanding of Dynamo (having never used it) is that you still need a very clear upfront understanding of your schema to make proper use of its scalability

lepistane13:12:13

Alright. Thanks. What are options for scaling the XTDB? Are there any problems with rocks DB and indexing?

refset17:12:01

Rocks can scale vertically quite happily, but there is inevitably a practical operational ceiling to discover for any given configuration and workload. What size of data are you looking at?

lepistane17:12:42

dozen terabytes

Rupert (All Street)19:12:47

I'm just a user of XTDB, but I thought I would add some thoughts to this thread. Generally speaking a dozen terabytes is a "smallish" dataset, so many Database engines (including XTDB) should be fine with that (obviously depending on read/write volumes). Dynamo has added an SQL layer (PartiQL) - but otherwise it was often seen as a pretty low level (ie limited) query API making some operations quite verbose. Dynamo will obviously lock you in to AWS ecosystem too. I think XTDB has more comprehensive query/insertion capabilities out of the box and is particularly good with time series data.

👍 1
Nikolas Pafitis22:12:11

Hi, I want to build a query where it checks if an entity (named "project") that has a :scheduled-at of java.util.Date is past-due. So how would I build a query that compares that timestamp with current time? Is it possible?

refset22:12:55

Hey Nik 🙂 you could either pass the current time as an :in parameter or reference a function, e.g. using (defn now [] (java.util.Date.)) like [(some-app-ns/now) ?current-time]

Nikolas Pafitis23:12:19

Can i use a library's ns for example tick?

Nikolas Pafitis11:12:24

Thanks that's next level

☺️ 1