Looks good! Might be worth spelling out that 1. you don't need to separately install Rocks db (is that what's meant by "in process"?) and 2. This setup is good enough for low-load production apps. For me, rocksdb is an implementation detail. What it means in practical terms is that my db is persisted to disk on the same server that runs my app (same as if I used sqlite) and there's nothing extra that I need to install on that server.

@UA9399DFZ Fixed! Thanks. 🙏

Awesome! I gotta say, it really is a "surprise and delight" moment when you get crux up and running in just a few min without installing anything. Have you spoken with @U050CBXUZ about maybe adding a crux option in the luminus template?

I certainly haven't, but I don't know him personally. I believe some of the folks who've been around juxt longer than I have do know him, though. 🙂

Is there any obvious limitation of `crux-in-a-box + persisting everything to rocksdb directories" solution? I don't mean "other more complex setups may give you better performance/whatever. I'm more interested in how will this setup behave if I have a ton of data in my db.. Will the JVM be running out of heap? Or is rocksdb itself running "next to JVM" and therefore is consuming its own memory space? But then again since crux node in jvm has to fetch the indexes (and it probably cache them in-memory), so does this still affect jvm heap and will it eventually OOM? Or will it mean that since the indexes won't fit in memory, crux operations will "only" become very slow? I'd consider adding a collapsible paragraph with similar such information. Doesn't have to be super-detailed, just some practical information of what to expect an or avoid doing.

and I would be open to adding Crux option in Luminus, it shouldn't be too difficult to do. I'll take a look if I get time, but open for a PR as well.

Most of my questions have a same source of thoughts. I'll try to describe what I had in mind and what keeps my brain busy when thinking about how exactly similar questions apply to crux. When I'm using a DB system such as Postgres from let's say java app (or clojure), reasoning about memory being used in my JVM app is relatively straightforward. When I want to do some DB business, I repeatedly use JDBC API to push (or pull) data off the remote dedicated DB system. That DB system is quite far from my JVM, often behind some network. Therefore, if I simplify it, the only additional "db/data specific" consumed memory is the one that gets allocated because of each query/update statement that gets executed. If I don't perform something crazy stupid (such as trying to load huge resultset of data "in one query"), I'm relatively safe. If I stick to doing paged reads and/or use jdbc client capable of streaming approach, the amount of memory consumed due to data fetching is relatively low. What's more important, such heap memory is only temporarily occupied. When the request finishes, the data is no longer needed and can be GC-ed. (I'm ignoring stuff like L2 cache etc). In this relational-db approach, it's usually the actual dedicated DB machine that needs to be "beefy". It usually has to have quite a lot of dedicated memory and disk space. That dedicated DB system is usually (among other things) tuned so that it's capable of storing most/all of its indexes in memory. If the index doesn't fit in memory, the performance degrades. In "3xRocksdb crux-in-a-box", I currently don't have similar knowledge of how the memory is being used. Rocksdb in this setup is somehow automatically launched by crux clojure api. Which is incredibly convenient. This most certainly somhow uses JNI under the hood (rocksdb being written in C + I looked into rocksdb mvn jar and saw a bunch of JNI .so and .dll files.).. So I believe rocksdb is actually launched somehow "next to the actual jvm" and JVM's heap isn't directly affected by this. At least, some basic googling led me to following statement: > Memory allocated by the native code wrapped by JNI is allocated to the JVM process, but is not under the control of your Java code. It is not part of the heap, and is not tunable via JVM parameters. Basically, anything allocated with a native malloc must be managed by that native code. If you are in control of the libraries you are using, its imperative that you go through it and check for resource leaks. So from this perspective, I guess that no matter how much data is actually being stored in crux, the actual memory that directly grows is the "native" one of rocksb (but perhaps somehow bound to the original JVM process?). But it's not affecting my app's heap. I believe/hope that it's somehow possible to configure lower-level tuning details of the rocksdb being started by crux node. Ideally one would like to be able to specify how much maximum memory should be dedicated to rocksdb. Btw what I've written ^^^ is just often just my feelings/theories/speculation. Things I'd like to learn more about these areas. I've also heard/read several times, that the crux node (besides being "backed by for example 3 rockdbs") also fetches the indexes so that it can efficiently run the queries. So does that mean that it actually caches the whole index(es) in heap memory of my clojure app? If so, how much memory can it consume? Can I configure it, monitor it, see some insights whether the query was "fully served" from such "locally cached index" or rocksdb needed to be contacted? Does the index need to fit in heap fully? What happens if the index grows too large and no longer fits in heap memory? Will I be able to know about that fact and what options will I have?

btw regarding > I guess I'm unclear whether you're asking if running Crux+Rocks in-process adds to the risk of OOM, or if you're wondering about the circumstances of spill-to-disk vs. OOM regardless of which process Crux lives in? I guess what I'm mostly after is this.. Currently I'm just starting, learning. I'm building a rather trivial app right now. And I'm just thinking ahead of time and am trying to understand how I'll be able to find out that my 3x rocksdb crux-in-a-box DB is reaching some "critical point" because I have already stored too much in it. And I'm also thinking about whether I'll be even able to find that out. Or whether something will start crashing on me first and that will be the moment when I find out 🙂.

Gotcha! Thank you. Let me have a bit of a think about which parts of this belong in the Quickstart and which parts deserve their own document. My impulse would be to add Operations and Tuning sections to the Reference Docs to complement the existing Monitoring doc: https://opencrux.com/reference/21.04-1.16.0/monitoring.html Regarding RocksDB specifically, there is a Rocks monitoring module: https://opencrux.com/reference/21.04-1.16.0/rocksdb.html#monitoring

Regarding your follow-up questions, it's probably best if we wait until @U899JBRPF and @U050V1N74 are around on Tuesday to talk about those details, as I'm bound to get some of it wrong.

Thanks both for the extensive write-ups! > Rocks is in the same process space as your app, so the app and the db are sharing heap settings ^ this is incorrect, and this is correct: > I believe rocksdb is actually launched somehow "next to the actual jvm" and JVM's heap isn't directly affected by this Controlling the all the various cache sizes, both internally to Rocks and in the heap, can be done through extended module configurations passed to start-node e.g. see how the default KV block cache is configured as an additional dependency: https://github.com/juxt/crux/blob/6d602bb5b6caed199f10fd8c3711cb034d49248a/crux-rocksdb/src/crux/rocksdb.clj#L120-L128 and https://github.com/juxt/crux/blob/6d602bb5b6caed199f10fd8c3711cb034d49248a/dev/dev.clj#L61-L62 We don't currently have documentation for all these advanced options but they are supported and we would encourage you to experiment and profile any changes according to your workload. Adding detailed cache information to crux-metrics is a good idea (and relatively easy), I'll add a note to the backlog :thumbsup: