Clojurians Log v2
datalevin
Hello, I found some time to look at porting dlmdb to FreeBSD. 1. When calling make I get the error make: don't know how to make dup_iter_bench.o. Is there perhaps some code not checked in? 2. mtest4 fails with mtest4.c:122: mdb_cursor_list_dup(cursor, &dup_list, &dupcount): MDB_INCOMPATIBLE: Operation and DB incompatible, or DB flags changed gdb shows it comes from
mdb.c:13210 if (F_ISSET(leaf->mn_flags, F_SUBDATA))
mdb.c:13211 return MDB_INCOMPATIBLE;I got a PR into USearch https://github.com/unum-cloud/USearch/pull/711 I am working towards a PR of JavaCPP https://github.com/bytedeco/javacpp/compare/master...markusalbertgraf:javacpp:FreeBSD I am working towards a PR of dtlvnative https://github.com/datalevin/dtlvnative/compare/master...markusalbertgraf:dtlvnative:FreeBSD All dtlvnative tests pass. I have not yet figured out why I can not get the datalevin tests to run here. FreeBSD has no GraalVM. Do you have any suggestions whether to include USearch in the JAR. I am out of my depth but it seems to me that simsimd, which is pulled in by USearch is very processor dependent so installing it via FreeBSD ports or pkg might be better. Please have a look at the links above and tell me whether I am doing something stupid 😉 I am a noob to C++ and Java.
Large test is ok. It could just have been me using the wrong system call in JavaCPP. This works too:
{:profiles {:test {:jvm-opts ["-XX:MaxDirectMemorySize=5g" "-Xmx6g"]}}}Orthogonally: How about renaming the forked lmdb.so to dlmdb.so to prevent conflicts? I could bring JavaCPP, usearch, dtlvnative and datalevin into freebsd ports. This would take advantage of FreeBSDs great build infrastructure with multiple poudriere servers which build packages from ports. One would install via pkg install datalevin (or ports) which would bring in the dependencies. Jars would be in /usr/local/share/java/ which would not be too conflict prone since FreeBSD people normally run pretty lean jails. Using the FreeBSD build infrastructure would also open up the way to more architectures. These would also alert if eg. a change to usearch breaks a test in datalevin. This does not mean one would have to choose to install datalevin via FreeBSD packages. On could just install dtlvnative-freebsd-arch via pkg and the rest via .m2. Also you could retrieve the artifacts from http://pkg.freebsd.org.
Coming to think of it: It would make sens to be able to install datalevin standalone server via FreeBSD pkg and to only install dtlvnative via FreeBSD pkg when working with embedded datalevin.
Sure, renaming the lib makes sense
@huahaiy datalevin lein test run: 1. datalevin.migrate-test/kv-migrate-test and datalog-migrate-test use 0.9.27-standalone.jar which understandably does not contain a FreeBSD jniDTLV 2. datalevin.withxn-test/with-txn-map-resize-test reports java.lang.OutOfMemoryError: Cannot reserve 499343620 bytes of direct buffer memory (allocated: 4096026011, limit: 4255121408). Increasing to -XX:MaxDirectMemorySize=5g solves this. Is this ok or should I consider this a memory leak? 3. datalevin.compress-test/kv-compressor-test reports java.lang.OutOfMemoryError: Java heap space. -Xmx6g solves this. Is this ok or is this a memory leak? All other tests pass Thank you!
except glibc, we need to put every native libs in jar. USearch is supposed to pick the simd implementation by platform. The issue with freebsd is mainly CI/CD availability.
BTW, those tests do require that much memory. No memory leak.
Let me reduce the string sizes used in the tests so it works with smaller heap.
Good catch. These are omissions on my part. Will push a fix soon.
Please pull the latest to see if it works for you. Thanks.
If you care to add freebsd support for our native layer, please feel free to send a PR to https://github.com/datalevin/dtlvnative