• We deploy Datalevin server in production. We use Azure, but AWS should be similar. Using Datalevin on EFS is not recommended. “Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts.”

yes

So, if I’ve got an MDB larger than 3gb, what do you suggest?

I’m trying to migrate from Datomic cloud BTW

what about a mdb larger than 3gb?

3gb is small

I’ve got a size limitation on containerized workflows in AWS

don’t know too much about that

Are you on a VM or using a container on Azure?

on a vm

If I do that, then I’m on the hook for all the added complexity of HA VMs. Any idea how serious the issues are with NFS?

i think that warning is regarding multiple process/hosts dealing with the same data file, if you don’t do that, it should be fine

OK, so let me run this by you…

Single load-balancer pointing at a failover container doing nothing but running dltv with the DSs mounted by NFS, no other server touches the NFS unless a failover occurs? Does this sound like a reliable solution to you?

it might be problemtaic when failover occurs though. some pages may not be flushed to disk yet

unless you use the most safe option, which is not the default right now, but will be in 0.9.0

right now, you can get the safe option by passing [] to :flags:

Any CPU/Mem configuration guidelines you’d like to add would be appreciated. I’ll look into the safety option. (open-dbi db dbi-name {:flags []}) ?

no, open-kv

with datalog db, pass to :kv-opts {:flags []}

the config is db file wide, not dbi wide

Perfect. Thanks for taking the time to help… One more question. Is there a github repo where people have implemented embedding on IOS with native image?

don’t know any like that. I am not sure we support IOS

Theoretically it looks possible, but I obviously don’t know much about it.

yeah, should be possible, and we will expand the supported platforms after 0.9.0, mostly a matter of CI config, but it does take time to do and test

@U0A74MRCJ Thanks so much. I’m really enjoying the simplicity you’ve created.

Thanks

@U0A74MRCJ Just wanted to follow up with some encouragement. Just keep going! This server setup is incredibly fast. I just decided to put it on a VM and got the whole thing sorted out with no issues first time. Very intuitive how to use it. It’s super impressive what you’ve got.

thanks for the suport!

Happy to share any useful info. The code and data are both open source.

The latest code is available at https://github.com/phronmophobic/dewey/blob/main/examples/deweydb/src/com/phronemophobic/dewey/db.clj

I was able to load the data previously without fulltext and the resulting db was around 70-80gb.

how big is the gz file?

852mb.

It's the analysis.edn.gz from https://github.com/phronmophobic/dewey/releases/tag/2023-06-12

that shouldn’t be a big issue, how big is the memory?

you are probably memory starved

set a higher Xmx

I set -Xmx12g . It didn't seem to be bumping into memory limitations, but I can try again.

Is there a rule of thumb or should I just keep feeding in more memory?

the more memory the better of course, LMDB is a MMAP db

to give you some idea, when I tried to run benchmark for fulltext search on full wiki data, I set both Xmx and Xms to be 24G

ok, I'll give a try

the data size is 15GB

Is (d/datalog-index-cache-limit @conn 0) the right way to disable caching?

if you hit the memory (80% default), spill to disk kicks in, everything will slow to a crawl

80% heap

oh, maybe that is what was happening

you can set a higher percentage, but the best is to increase heap size

we are using clojure, so it is memory hungry

I think I saw that lmdb dbs are copyable between machines. I might look into trying to do the load on a high memory VM and then download it later.

correct, you can just copy the db (while compacting)

What does compacting mean?

0.9.0 will introduce compression, hopefully the file will be smaller

I'll try on my laptop, but I don't actually have that much ram (only 16gb)

compacting is an option in copy, it will remove unused pages

you can set the spill-to-disk ratio to 100% or something, so it will not kick in, but you may get OOM

Ok, I'll give those things a try. Thanks for your help!

the new full text index introduced in 0.8.x is faster, but it does eat much more memory

if you don’t really must have it, don’t enable fulltext

interesting. I ran clj-kondo analysis on around 18k clojure github repos. The search is pretty useful without fulltext, but I feel like having fulltext search would be awesome.

{:spill-opts {:spill-threshold 100}} for :kv-opts

If I load the data using a high memory VM and then try to query later on a more modest computer, do you think that will also run into problems?

that will disable spill-to-disk

probably ok to try that.

since you are indexing code, I would use a custom analyzer

that's a good point. I'm not actually indexing the code itself, mostly the names of keywords, namespaces, and functios and the doc strings.

maybe default english analyzer is ok, i don’t know, worth to think about