not sure. open-dbis would be easier?

I think so.

it could be added as an utility function as well

it's something I noticed when I have this:

(d/open-dbi lmdb docsearch-table)
        (d/open-dbi lmdb semmed-table)
        (d/open-dbi lmdb processed-rel-name)
        (d/open-dbi lmdb rel-name)

sure

should I try a PR ? in datalevin.core ?!

I'll try first to add a local function that we will use.

once I figure out I can post it and you can decide if you want to add it or not

ok

Calling seek-datoms 3 times with different start components throws an OutOfMemoryError

If you know you are doing an operation that takes a lot of memory, you want to disable the cache.

I thought seeking to a particular spot in the index would be pretty fast. Is that an incorrect intuition or is there something else going on?

i think you misunderstood what seek-datoms does. It starts from a spot, and return everything after that.

if you want a particular value, use index-range

I'll have to double check, but I think seek-datoms is lazy in datomic.

nothing is lazy in datalevin

It looks like datoms might be lazy?

datalevin.core/datoms
 [db index]
 [db index c1]
 [db index c1 c2]
 [db index c1 c2 c3]
 [db index c1 c2 c3 c4]
  Index lookup in Datalog db. Returns a sequence of datoms (lazy iterator over actual DB index) which components (e, a, v) match passed arguments.

that’s wrong doc string, we can change that

I just tried the following, and it seems to work:

> (let [db (d/db conn)]
    (d/datalog-index-cache-limit db 0)
    (time
     (->> (d/datoms
           db
           :avet
           ::basis
           "/deps.edn")
          first))) 
"Elapsed time: 1.436584 msecs"

It seems like index-range is also lazy. I think this query would scan my whole database otherwise:

> (let [db (d/db conn)]
    (d/datalog-index-cache-limit db 0)
    (time
     (->> (d/index-range
           db
           ::analysis-id
           "2023"
           nil)
          first)))
"Elapsed time: 4.952667 msecs"

As i said, right now, these are not lazy. if your memory is large enough, it will loads everything in memory. If memory is not large enough, they will spill to disk, hence the slowness you saw. Disabling cache was to save the memory, otherwise, the results accumulates in memory.

Ok, I guess I'm just confused about why seek-datoms is so slow. Using index-range to scan all of the values of a property takes less than a second, but doing the same thing with seek-datoms takes over a minute:

(let [db (d/db conn)]
  (d/datalog-index-cache-limit db 0)
  (time
   (->> (d/index-range
         db
         ::analysis-id
         "2023"
         nil)
        first)))

(let [db (d/db conn)]
  (d/datalog-index-cache-limit db 0)
  (time
   (->> (d/seek-datoms
         db
         :avet
         ::analysis-id
         "2023")
        first)))

we could have lazy version of things, trading some performance for laizness.

right, seek-datoms returns everything after that value, so it may have taken more than 80% of memory, then spill to disk is triggered

oh, seek-datoms also includes all of the other properties in the :avet index :face_palm: . I get it now.

Right, so it may not make much sense as an eager operation . We will change that in the future