This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-01-19
Channels
- # announcements (19)
- # asami (9)
- # babashka (26)
- # beginners (87)
- # biff (23)
- # calva (6)
- # clerk (7)
- # clj-kondo (3)
- # cljsrn (3)
- # clojure (115)
- # clojure-belgium (1)
- # clojure-berlin (1)
- # clojure-europe (31)
- # clojure-gamedev (5)
- # clojure-nl (2)
- # clojure-norway (8)
- # clojure-uk (2)
- # clojurescript (43)
- # clr (23)
- # datalevin (1)
- # datomic (14)
- # dev-tooling (23)
- # fulcro (38)
- # graphql (1)
- # gratitude (1)
- # jobs (1)
- # lsp (30)
- # off-topic (7)
- # pathom (25)
- # portal (21)
- # quil (6)
- # releases (5)
- # remote-jobs (1)
- # shadow-cljs (34)
- # sql (5)
- # tools-deps (6)
- # xtdb (13)
Hm. I started to look into surprisingly big heap uses, and on top I see we have over four million Strings in the heap. This is a bit new territory for me, but I suspect this is coming from xtdb based on seeing org.agrona also having a bunch of buffers allocated, and nothing else depends on those
Well, I suppose it's not a problem yet, as long as it stays in reasonable levels (about half a gig now, after querying data)
there are caches used by the node which may be big, but that isn’t a problem if you don’t need the memory for something else
I'm guessing that the rocks cache is bounded, but does xt then keep a reference to all the documents that happen to be in the cache? Or is it just something completely unrelated.
We do have this issue open where you'd be very welcome to pose such questions (I'll link back to this thread at least, anyway) https://github.com/xtdb/xtdb/issues/1830 In short though, managing memory usage is a tricky balance between: 1. performance - caches make things fast so use memory if it's available 2. robustness - we don't want OOMs or to cause non-XT performance to suffer unpredictably 3. user-friendliness - defaults should be sane but exposing levers is often necessary As per the issue, we need to review the defaults at some point.
> the second chance cache has a background thread that monitors memory use note that the second chance cache has been disabled (in favour of LRU) as of 1.22.0, due to unresolved memory issues
> I'm guessing that the rocks cache is bounded that's definitely true of the block cache as I understand things, although Rocks will perform various allocations for other reasons > does xt then keep a reference to all the documents that happen to be in the cache? XT has no visibility of what's inside the Rocks cache