With the help of this forum post: https://discuss.xtdb.com/t/2-x-how-filter-query-results-based-on-nested-vector-values/257/5 I put this together:

(xt/q my-node '{:find [timestamp move player]
                :where [($ :videos {:video [video ...]})
                        [(. video :timestamp) timestamp]
                        [(. video :player) player]
                        [(. video :move) move]
                        [(= move "grips")]]})

Hey @U3SG7RX7A thanks for sharing your solution 🙂

As per James' response in your Clojurians Slack link, we do want to make it easier to write this kind of nested data expression in a single pass

Hey @U899JBRPF thanks 👍 I can see that you’re scheduling a discussion on 2.x here: https://discuss.xtdb.com/t/upcoming-query-api-discussion-session-2-x-datalog/261 Any finger in the air, guesstimate, when we’re potentially looking at that feature? Thanks in advance

<2 weeks, I'd guess (🤞)

Good point, thanks, we subsequently bumped the minimum Java version a few months ago (https://github.com/xtdb/xtdb/commit/4f719bbce586d041b7f666185fa4174a43a466b9, specifically)

We're actually really hoping to move to 21 soon but there are a few upstream deps we're waiting on

just to clarify on this one, we should be able to keep the XT clients on Java 11 - it's only if you're looking to use it in process that there'll be the Java 21 minimum

Hey @U1G8B7ZD3 1GB is quite small, but it should be possible to run at that scale. What are your Xms / Xmx configs looking like? Is there anything else running on the machine that needs RAM? Rocks manages its own native allocations so needs sufficient memory headroom to operate (beyond what the JVM claims for itself)

> For example, I can have docs at 8MB, tx at 5MB and index at 125MB. Will the index folder keep growing with this multiple? What can I expect here? it somewhat depends on the 'shape' of the docs (depth vs width, size of keys etc.), but those ratios are pretty normal sounding

> You are doing a really great thing with XTDB so I would like to keep using it 🙂 🙏

Thanks for your reply. To clarify, 1GB is for the nodes total memory. In my setup 512MB is allocated for the Docker container running my Clojure web-app with XTDB as a library. Regarding the settings, I have looked at the docs for RocksDB (https://docs.xtdb.com/storage/rocksdb/) and set -XX:MaxDirectMemorySize=512m, but now re-reading the docs in says the sum of MaxDirectMemorySize and Xmx should be available. So I guess in my case an initial setting would be 256MB each? Given this, what is your gut feeling about these 256MB * 2 levels? My intent for this deployment is to use XTDB as an "embedded" DB as SQLite often times are. It evidently works to some extent, given the test I mentioned, but let me know if you think these levels sounds unreasonable Thanks for the info on the disk size expectations!

We haven't spent much time on tuning for low memory environments so far, unfortunately I think the only way to be sure is to try a few changes and see if it's stable

We have various tests / benchmarks in the repo you might find useful to simulate some workload if you need help with that

Thanks, I'll check it out

Hi again @U899JBRPF, I have some more data on this issue. I keep having OOM problems. When I enable heap dumps on OOM, I can see that XTDB LRUCache seem to hold on to a lot of values. Looking at the dump, there is more than 800k instances of UUID rooted in the cache. I use UUIDs as xt/ids. At this time there are about 2k-3k of orders in the system. There are a lot of duplicates in the instances. Also, I can see the same behavior of a long list of Longs, which I use as order numbers. I have tried to experiment with JVM memory params, and also add more memory to my test node. The only thing that happens is that it takes a bit longer to OOM and there are even more instances in the same pattern as above. Is there anything I can do about this behavior? Does it indicate that I use XT in some bad way? Is it a bug?

Hey @U1G8B7ZD3 how much memory are you giving the JVM here?

You can try tuning down the document store LRU cache size to something very low, e.g. by supplying this as a key to the document store module: > :document-cache (cache/->cache {:cache-size (* 1 1024)}

it is sized by number of entries rather than explicit memory size

I suspect for each of these 800k UUIDs (which are the keys) there will also be 800k docs which are taking up proportionately a lot more memory

but as mentioned before, XTDB has not been optimized to work in low memory environments - happy to try to help though 🙂

For that particular screenshot I think it was a low memory setting (not having to wait too long on the OOM, so 128M). The most amount I have tested with is giving the container this is running in 1,5GB, the JVM 512M and the MaxDirectMemorySize flag 512M. Dump looks the same but with 3 Million UUIDs instead. I will not be able to motivate a more expensive node than the 2GB one. Would you recommend some other JVM configuration with 1,5GB at hand or is my configuration above reasonable? I will try the LRU cache size option. When you say that you suspect that there are 800k docs taking memory, are those in user space? As I mentioned the amount of entities in the system from my point of view in this experiment is about 3k orders (order-lines, name, address, etc) and 200 product documents. So I'm curious to understand if I might have a problem in my usage that result in keeping those 800k alive and how it can amount to so many docs. I don't think I'm doing anything exotic, there are some queries for products, transactions for the order-lines, and finally a tx with a transaction function to get an increasing order number for the order tx. Thanks for the help, even if I'm my use-case is an outlier. If I can solve it and convey some information to you, it might mean more applicable use-cases and more XTDB for the world :)

Very useful to hear that context - thanks for the write-up > 3k orders (order-lines, name, address, etc) and 200 product documents is that across all time? or are there lots of versions/edits to these? I'm also unsure why you should be seeing 800k UUID entries there, unless you're using them elsewhere in the system (i.e. it's a red herring since most have nothing to do with the XT doc-store LRU cache, you just got (un)lucky opening up the first one in that window)

There shouldn't be alot of edits to the docs. Most of the edit data is stored in a session which are then transacted at checkout.

I did some more dumps. FYI before doing Clojure for 13ish years, my background is in C/C++/C#, so I'm no expert in JVM analysis so I hope I'm not to ignorant here..

First a dump after just a few minutes

Second one is after 38m

Looking at these, there is a growth in the UUIDs and Longs. Using the "Compute merged GC Roots", we can see a lot of roots in XT NotifyingSubscriber, does that tell you anything?

Finally, if I open up a few of those, we can see some more details:

Do you get any ideas that I can further investigate based on this?

thanks for the updates - do your documents contain UUIDs? or are you only using them for IDs? looking at the stack in that last screenshot makes me think the UUIDs reside within nested structures inside the documents, as I don't think the LRU structure itself is more than one or two levels of that

did reducing the cache size mitigate at all?