This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-04-01
Channels
- # announcements (11)
- # babashka (21)
- # beginners (57)
- # biff (5)
- # calva (75)
- # clj-commons (20)
- # cljs-dev (27)
- # clojars (88)
- # clojure (18)
- # clojure-europe (11)
- # clojure-germany (1)
- # clojurescript (52)
- # datalevin (25)
- # emacs (2)
- # graphql (2)
- # gratitude (4)
- # off-topic (6)
- # pathom (16)
- # reagent (3)
- # releases (4)
- # shadow-cljs (7)
- # spacemacs (9)
- # transit (3)
- # xtdb (5)
hello @huahaiy, I found some time and decided to give a try to implement Map interface on top of KV store https://github.com/juji-io/datalevin/pull/200 . Turns out basic functionality is quite simple and can be done outside of datalevin library. I did it in the example code. I would like to flesh out the code over time and plan to use it in one of our internal projects to test it out. I pushed the code to be out there (surprised how simple it is) and to allow for discussion on top of it. • I think it is simple enough and usefull enough to be part of datalevin so people don't roll their own. Wdyt? • Not sure yet how to implement clear / size (efficiently) yet. Probably get-range, but I don't want to load everything in memory. Recommendations are welcomed. • There are lots of map API's to implement later
This will only be limited to data that do not expect range queries on them, because the map API cannot takes data type for keys.
Yes, it is trivial to implement a Map API, as it is actually the subset of existing functionality.
we will have a key-range function in the next major version, that should take care of keySet, entrySet and values can be done with get-range with [:all] as the key range
If you would like to send a PR to include it as part of the datalevin, I can merge it. Please write some tests if you do. Thanks.
> This will only be limited to data that do not expect range queries on them, because the map API cannot takes data type for keys. hold my beer ! 🙂 . I think the API can be used in a statefull maner. The map implementation can have params passed to it when it's built. We can mutate those params via special interface. This might not be canonical Map implementation, but as long as it works ?!
I'm going to push an updated version that implements AutoCloseable. so we can write code like this:
(with-open [m (->map "/tmp/datalevin/map2" "a")]
(println (.put m 1 "a")))pushed the update. Will try this in our project for a use case pretty soon and come back with more feedback. Then we can consider merging.
Also, you can use range-seq instead of get-range if you want to load the data lazily, since you are using with-open already
thanks. today I spent some time on a few places of our code that might benefit from lmdb/datalevin. we load a map in memory - so this might be a good fit. The maps use about 260MB of memory (measured with https://github.com/clojure-goes-fast/clj-memory-meter ). And that is just the start. It also does some processing that I believe we can cache in datalevin. Very excited about this 🙂
how can I check I have a LMDB instance? using
is-datalevin? (instance? datalevin.db.DB db-or-path)
db (if is-datalevin? db-or-path (d/open-kv db-or-path db-opts)); Execution error (ClassCastException) at datalevin.db/close-db (db.cljc:426).
; class datalevin.binding.java.LMDB cannot be cast to class datalevin.db.DB (datalevin.binding.java.LMDB is in unnamed module of loader clojure.lang.DynamicClassLoader @55cd4d99; datalevin.db.DB is in unnamed module of loader clojure.lang.DynamicClassLoader @109c25f8)