We don't have an S3 KV store I'm afraid - we make a distinction between the document store (a golden store, shared between all nodes), which just stores raw documents, and the local, per-node KV store, which stores the query indices. We haven't written a remote KV store as yet, largely because of an assumption that it wouldn't be performant enough for queries to have the indices remote - the query engine makes a lot of small requests of the indexes.

If we could find an implementation that was sufficiently fast, then yes, it would seem sensible to share that work between multiple nodes 🙂

Oh i see. I took the doc store for the kv store somehow. Makes sense now.

What about a layered architecture with a shared slow kv store and a memory cache on nodes. Could that work? (For the pathological use case)

I don't see why not - there's a good few interesting implementation questions with that around what to cache, at what granularity, in what format, and how to invalidate it

Hey @U08JKUHA9 🙂 not just yet I'm afraid, but it is on our roadmap, and we've recently been refactoring the query engine internals to make this easier - hopefully it's not too far away

cool, thanks for the info

Hello. I'm not actually a #crux user, but can't @U08JKUHA9 create a file/in-memory db, install stuff, run queries and remove/delete this db/file?

My guess is that it would work, but wouldn't be fast enough of a process to answer queries in real time

I worked in a #datomic app do a "bulk import" like this: - create "empty" in-memory db - transact new data on it (take some minutes) - query this in-memory db and write "computed" results in the "main" db. - delete temporary in-memory db

Ah I see. My thing is I would also need to see all data in the main db, so I worry it would take a long time to initialize a temporary db like that unless there is explicit support for it, or some other mechanism like forkdb

@U08JKUHA9 maybe write a kv store that delegates to the memory store and if not found to the main db?

@U054UD60U :thinking_face: cool idea