Clojurians Log v2

sure thing

@huahaiy I’m still running into MDB_READERS_FULL error, even in latest version 0.9.20 datalevin.cpp.Util$DTLVException: MDB_READERS_FULL: Environment maxreaders limit reached I’m not using fill-db tho, I think my affected code is mostly calling just datoms , entity and pull-many

what would you suggest trying?

You must be using future/pmap/pcall/... somewhere. Use claypoole with a size of under 128

thanks @olajeremy123! removing pmap (wasn't really needed in our case) fixed our problem

Can you try the master branch to see if it is fixed even if you use pmap?

I am still able to replicate on master, although it took more pmap calls to do so

@huahaiy this is similar to what my code was doing and reproduces the error for me

(def conn (dl/create-conn "/tmp/dl/test" {:children {:db/valueType   :db.type/ref
                                                     :db/cardinality :db.cardinality/many}}))

(dl/transact! conn (mapv
                    (fn [i]
                      (let [id (* -1 i)]
                        {:db/id    id
                         :children [{:db/id  (dec id)
                                     :string "test"}]}))
                    (range 1 100000)))

(defn pull-test
  [id]
  (dl/pull @conn '[:db/id
                   :string
                   {:children 100}] id))

(do
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  (pmap pull-test (range 1 100000))
  )

can you try master branch again to see if this is fixed?

@brownjoshua490 I added your code to test also

@baibhavbista Can you test the master branch to see if this problem is fixed?

@baibhavbista is my coworker, he was working on the same problem I am. I am still able to reproduce on master with both the example I shared and my application code. Does the example error out for you?

No, I cannot reproduce the error

ok, now I can reproduce

this can only be reproduced with cold run

can you try again?

I couldn't reproduce this error any more

oh, it comes up again.

I still get it on d967f913528779435c287938a6adeea7f932c08c

I'm curious @huahaiy .. what's the fix u're implementing? I thought lmdb had a hardcoded max readers of like 256.

it's not hard coded, it's an option :max-readers when opening the DB.

I am adding semaphore to make sure no more than that can be inflight

This will also fix the problem of out of memory/resource if you just start thousands of threads to read a DB

I see. I saw that somewhere on a stack exchange i think. I might've mis-read/mis-remembered

Normally, one shouldn't be doing this, but apparently people are doing it and blame Datalevin

lol, I think the main reason is the clojure core functions, if given an option, i don't think people would use that many threads

pmap and friends use unbounded threadpool

I did not realize pmap used that many threads, for me I'm on a thread constrained env anyway so removing it was fine

I think cachedthreadpool is used in pmap and friends, which is unbounded

more specifically, it's "future" that use agent thread pool, which is unbounded

one can fire off thousands of threads with future that will get you OOM or out of resource exception

pmap just opens future, albeit lazily, Num of processor +2 at a time

but if you force the lazy seq of pmap, it will fire off thousands of threads

Oh that's pretty unintuitive, had I known that, I would not have used pmap in the first place, I just assumed that it would work like core async with a fixed thread pool. not trying to blame datalevin, max amount of readers seems reasonable

With the semaphore in place, it should limit the number of threads in flight, but apparently sometimes it doesn't, so It's hard to reproduce

well, I want DL to be robust, so if semaphore can fix this, I will do that.

right now I am using the semaphore to guard get-rtx, i.e. when opening a read only transaction, but apparently sometimes it fails to guard it for some reason, maybe there are other paths that take up read transaction without calling get-rtx

It's hard to reproduce, as it doesn't come up all the time. Seems have to do with memory usage. If the DB are paged out, the error more like to happen.

so it might indeed take up read transaction somewhere without going through my code path of get-rtx

oh maybe it happens more for me because I have my jvm limited to 1500mb

what you can do is to up :max-readers the default is 126.

in my code, I can limit the number of permits to be half of that.

that may count for other invocation of read transactions that's not under my control

For my actual production system I am just going to not use pmap, or use a pmap with limited threads. I meant more for reproduction, but I upped my jvm memory and I still get the error

now I just use half of the reader limit

does it help?

and doubled the default max-readers.

It happens less frequently for me now, but it can still happen

It might just be because of the example I posted though

in any case, clojure core team would argue that pmap is not for I/O, so...

either switch a thread pool yourself, or don't use pmap for I/O

In a real usecase I am not going to open many pmaps without consuming them, if I just do 1 pmap at a time, like in your tests, then it seems to work

yes I am going to not use pmap

yeah, open tons of pmap will defeat the lazy seq that's used in its code

The main thing is to be aware of future uses an unbounded thread pool, so if one is not careful, out of memory/platform threads can easily be produced. Datalevin's max-reader limit just let that problem shows up much faster.

thanks for looking into this!

A fix to prevent MDB_READERS_FULL error is pushed to the master branch.

Also added a test that spawns hundreds of read threads, and it seems to work

So I've run into this a few times (hard to reproduce). I wonder if this is a problem with virtual threads and request handlers, as technically that can lead to spinning up a lot of threads.

Probably

hmm. So I guess I have to put a semaphore around q then. 🤔

Right now, I don’t think virtual threads work well with Datalevin

Each of these takes up a reader

So no, don’t use virtual threads with Datalevin. Use a thread pool instead

stack trace:

1. Unhandled clojure.lang.ExceptionInfo
   Fail to open database: #error { :cause "MDB_READERS_FULL:
   Environment maxreaders limit reached" :via [{:type
   datalevin.cpp.Util$DTLVException :message "MDB_READERS_FULL:
   Environment maxreaders limit reached" :at [datalevin.cpp.Util
   checkRc "Util.java" 59]}] :trace [[datalevin.cpp.Util checkRc
   "Util.java" 59] [datalevin.cpp.Txn create "Txn.java" 43]
   [datalevin.cpp.Txn createReadOnly "Txn.java" 31]
   [datalevin.binding.cpp.CppLMDB get_rtx "cpp.clj" 640]
   [datalevin.scan$get_range invokeStatic "scan.clj" 100]
   [datalevin.scan$get_range invoke "scan.clj" 96]
   [datalevin.binding.cpp.CppLMDB get_range "cpp.clj" 802]
   [datalevin.binding.cpp.CppLMDB get_range "cpp.clj" 798]
   [datalevin.binding.cpp$init_info invokeStatic "cpp.clj" 1121]
   [datalevin.binding.cpp$init_info invoke "cpp.clj" 1116]
   [datalevin.binding.cpp$eval63469$fn__63471 invoke "cpp.clj" 1178]
   [clojure.lang.MultiFn invoke "MultiFn.java" 234]
   [datalevin.storage$open invokeStatic "storage.clj" 1501]
   [datalevin.storage$open invoke "storage.clj" 1491]
   [datalevin.db$open_store invokeStatic "db.clj" 473]
   [datalevin.db$open_store invoke "db.clj" 469]
   [datalevin.db$quick_fill invokeStatic "db.clj" 578]
   [datalevin.db$quick_fill invoke "db.clj" 576] [datalevin.db$fill_db
   invokeStatic "db.clj" 602] [datalevin.db$fill_db invoke "db.clj"
   596] [bb.engine.market.capture.impl.eav_db$eval107034 invokeStatic
   "NO_SOURCE_FILE" 714]
   [bb.engine.capture.impl.eav_db$eval107034 invoke
   "NO_SOURCE_FILE" 714] [clojure.lang.Compiler eval "Compiler.java"
   7700]
   [nrepl.middleware.interruptible_eval$evaluator$run__46372$fn__46383
   invoke "interruptible_eval.clj" 106]
   [nrepl.middleware.interruptible_eval$evaluator$run__46372 invoke
   "interruptible_eval.clj" 101]
   [nrepl.middleware.session$session_exec$session_loop__46451 invoke
   "session.clj" 229] [nrepl.SessionThread run "SessionThread.java"
   21]]}
   {:dir "./var/db/captures"}
                   cpp.clj: 1184  datalevin.binding.cpp/eval63469/fn
              MultiFn.java:  234  clojure.lang.MultiFn/invoke
               storage.clj: 1501  datalevin.storage/open
               storage.clj: 1491  datalevin.storage/open
                    db.clj:  473  datalevin.db/open-store
                    db.clj:  469  datalevin.db/open-store
                    db.clj:  578  datalevin.db/quick-fill
                    db.clj:  576  datalevin.db/quick-fill
                    db.clj:  602  datalevin.db/fill-db
                    db.clj:  596  datalevin.db/fill-db
                      REPL:  714  bb.engine.capture.impl.eav-db/eval107034
                      REPL:  714  bb.engine.capture.impl.eav-db/eval107034
             Compiler.java: 7700  clojure.lang.Compiler/eval
    interruptible_eval.clj:  106  nrepl.middleware.interruptible-eval/evaluator/run/fn
    interruptible_eval.clj:  101  nrepl.middleware.interruptible-eval/evaluator/run
               session.clj:  229  nrepl.middleware.session/session-exec/session-loop
        SessionThread.java:   21  nrepl.SessionThread/run

weird thing is the db still works with q and transact!, but executing fill-db errors out. I had used fill-db to load about 230 markets data (~46 mil entities) before. but after clearing db, it hasn't worked after lots of tries

You probably are not using the same conn but instead created multiple db

is it ok to pass around the same conn , and have functions deref to get the same db? That's what I'm doing. I used integrant to manage the dependency

you are supposed to pass around the same conn

that's what I'm doing

are you using pmap?

that's using unbounded thread pool

I've used builtin future as well as claypoole/pmap with a threadpool of 10, 2, and 1. lastly, I captured the datoms in a global var and I'm trying to use fill-db directly on it in the repl. however, this works: (datalevin.storeage/load-datoms (:store db) datoms)

well future is using unbounded threadpool

we recently changed the transaction pool to be thread local, that's the reason for you to exhaust maxreaders

so unfortunately, you will have to use a fixed size thread pool

otherwise you will run out of readers

each thread has a read transaction pool, and these are never released

I'm kind of confused with the unbounded threadpool and threadlocal tx pool. The writing has always been done from the same thread. the threadpool was for reading data in batches (and writing was done after all reading was complete)

writing is ok, write txn is always created afresh

reading is what you have problem

read txn are reused, they are stored in a txn pool, which is thread local

ohkk.. so because those threads that read previously haven't been destroyed/released, fill-db can't create a new reader?

so it won't work with an unbounded thread pool, such as future uses

there's a query thread pool that is fixed size, that's what we use for query

Let me think of some ways to work around this. somehow release the readers when a thread no longer exists

I think I understand now (was confused a about unbounded pool). I'm guessing using fixed threadpoole from claypoole should work. I was previously creating a new pool for the duration of the function

sure, you can do that. but I need to figure out a way to fix this. as I Cann't prevent people from just using future

which functions are you using that triggered this issue?

fill-db d/q and d/transact! are still working

right, q uses query thread pool, and transact! start a new transaction every time, so nothing is hold.

let me think of someways to address this problem.

Thanks a lot! what about load-datoms, does it not perform the max readers check?

the check is done in C

in LMDB itself

the past versions don't have this problem, because the read transaction pool was not thread local, now it is.

If I might ask, why was it made thread-local?

to solve other issues with concurrent access

concurrency programming is hard

I see. I guess there's give and takes everywhere 😅

right, you want to limit the use of locks for performance, thread-local is often a solution, but then there are this problem that I have not found during my testings.

Btw, if you are not using the master branch, fill-db creates a new db every time, which is not good. Hence you can no longer create new read transactions.

It is possible to exhaust the readers, default is 126

since you are doing this in a real, so probably all those dbs are kept around

> fill-db creates a new db every time Hmm.. using same threadpool for the writes, I no longer get the max readers error. However, after some batches, I now get this error:

:cause MDB_MAP_RESIZED: Database contents grew beyond environment mapsize

btw, if fill-db takes a db as an argument, why does it create a new db rather than just using that

no, this error means your db is now larger than file size allocated.

you need to set a large :kv-opts {:mapsize xxx}

I'm using

{:kv-opts {:mapsize 100000}}

it is possible to grow much larger when you have concurrent readers

temporarily.

for a reader will prevent the pages in its snapshot to be reused, so the db size can grow very fast.

fill-db and init-db are envisioned to import data, not for ETL. I would do transaction with ETL.

when you are reading data from the db then write to it, you want the insurance of transactions. fill-db is intended to import data from external source. That's why the early version create a new db every time. that's the answer. It's now fixed in the master branch.