This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2024-01-27
Channels
- # announcements (24)
- # babashka (26)
- # beginners (8)
- # calva (8)
- # clojure (78)
- # clojure-europe (1)
- # clojure-norway (22)
- # clojurescript (14)
- # datascript (5)
- # datomic (8)
- # fulcro (22)
- # helix (9)
- # humbleui (11)
- # malli (4)
- # off-topic (28)
- # pedestal (5)
- # reitit (10)
- # shadow-cljs (2)
- # tools-build (8)
- # tools-deps (9)
Hi all. Have been looking at the new storage options - has anyone experience of how that would work with large databases - I see data will only be loaded on demand, but does that mean in a long-running process eventually all data will end up in memory? I can see the garbage collection code, but that looks as if that clears deleted data, rather than data not being used?
My understanding is basically the persistent sorted sets load data from disk lazily as the nodes of the tree are accessed. Once the data is loaded it is kept in soft refs (by default) or optionally in weak refs. What that means is the garbage collector can reclaim that space when nothing else is holding a reference to it (like the literal results of a query). If there's memory pressure and the space ends up being reclaimed by the GC then the next time the section of the tree representing the persistent sorted set is accessed it'll be read from disk again.
The garbage collection process you're referring to is a separate feature not directly related to the soft/weak refs and JVM garbage collector (which is what I'm describing). That feature is about removing excess addresses from storage that can accumulate.
Perfect. Thank you both.