xtdb 2021-09-29 | Slack Archive

pinkfrog13:09:37

I’d like to introduce xtdb into our application that is running on a K8s env. So suppose the service is scaled to 10 pods (replicas): 1. If we use an embedded xtdb node into the application. Then since the service has 10 replicas, so there will be ten embedded nodes. 2.a Question: Shall we use in-memory approach for indexing? If in-memory, isn’t it be slow at start as the time is used to build the index, and won’t it eat much memory? 2.b Question: Shall we use rocks db for indexing? If so, then 10 replicas will write to ten different places for the indexing. In general, I’d like to know how to integrate xtdb into a K8s env.

refset16:09:13

Hi @UGC0NEP4Y thanks for sharing all the context so far. 1. That's correct 2. a. In-memory indexes are not really intended for production usage as it means your JVM will experience OOM instability if the database grows unbounded b. That's right, each node is responsible for its own indexes > In general, I’d like to know how to integrate xtdb into a K8s env. Which k8s infrastructure are you using? Do you have any option to use StatefulSets + persistent volumes?

pinkfrog13:09:58

Ideally I would anticipate there is only one rocks db indexing. So that when we scale up with a new replica, it will immediately use that rocksdb’s index

tatut13:09:41

each node will have their own indexes

✔️ 1

tatut13:09:01

you can use checkpointing periodically to get new nodes up to speed faster

✔️ 1

pinkfrog13:09:58

Can I assume the ONLY downside of using HTTP API instead of embedded node is that, we cannot use the transformation functions with queries?

tatut14:09:08

I don't know about the HTTP API, haven't used it

refset16:09:01

You can still refer to user-defined functions in your queries via the HTTP API, but you will need to create your own uberjar/Docker (see the build module) that includes those functions on the classpath

👍 1

2021-09-29

Channels