Clojurians Log v2

Hi @b minor clarification to make sure we're on the same page: • "without an index-store" - v1 always has an index-store, it's just that by default the index-store isn't durable because it's backed by an in-memory KV store (more or less just a sorted map) Can you share the configuration map you are passing to start-node? > I tried to close the xtdb resource and that never returns. (Never means 30min+) this doesn't sound right - are calling .close at the REPL? Which JVM/Clojure versions are you using?

Yes, I meant without a durable index-store.

(let [dbname "development/data/db.sqlite"

                 index-dir "development/data/index"

                 node (xt/start-node

                       {:xtdb.jdbc/connection-pool {:dialect {:xtdb/module 'xtdb.jdbc.sqlite/->dialect}

                                                    :pool-opts {}

                                                    :db-spec {:dbname dbname}}

                        :xtdb/tx-log {:xtdb/module 'xtdb.jdbc/->tx-log

                                      :connection-pool :xtdb.jdbc/connection-pool}

                        :xtdb/document-store {:xtdb/module 'xtdb.jdbc/->document-store

                                              :connection-pool :xtdb.jdbc/connection-pool}

                        ;; for performance reasons, enable index store (not a golden store)

                        :xtdb/index-store {:kv-store {:xtdb-module 'xtdb.rocksdb/->kv-store

                                                      :db-dir (io/file index-dir)}}})]

             node)

I'm calling (.close ...) interactively via nrepl in a (comment block)

I went to lunch and came back and I got a java heap error, I guess after 30 mins that happens. 🙂

Sorry about this, I switched to rocksdb for testing purposes to see if that changes things. But the behavior is the same as with lmdb. @taylor.jeremydavid

and for completeness sake: Before I ran the program without any :index-store key and could persist and restart the program (but the first xt/sync took longer and longer)

ah, so we've not tested with Zulu/Azul before - I wonder whether that's the issue 🤔

would it be a huge pain to try a different JDK?

Not huge, but in the meantime I've tried a backup from when I had <30k tx and when I .close xtdb, it writes the index. So maybe the size of the index is also part of this. But I can try jdk21 or higher

oh interesting - that shouldn't really be an issue, unless maybe you have an absolutely enormous individual transaction or something ...but I assume you're not storing video BLOBs or similar(?)

nope, the size of the db is 1080 MB right now, it's 300k tx.

> I got a java heap error if you have a copy of any of the stracktrace, hprof etc. outputted it might be useful

I've re-ran it and got another heap error: https://pastebin.com/wDxhLhaG

thanks, that is useful - looks like something is preventing the node from being closed, but then the error message attempting to feed that back is blowing up https://github.com/xtdb/xtdb/blob/06132bc1fad63944d144d79a90053b05d78a41c8/core/src/xtdb/io.clj#L134

are you calling open-db, open-q, open-entity-history or open-tx-log in your code?

I've disabled everything but the db part, but I also enabled sqlite pragma like shared memory for performance, which could have the same effect?

so right now there is no open-db, open-q, etc in the code

what about any background writers to submit-tx are you disabling those before attempting the .close?

(it's possible our integration with sqlite is the actual culprit)

I'm merely bringing up the xtdb, sync it and wait until the index is built (`xt-latest-completed-tx`) and then I .close it. So everything in the system outside that part is currently disabled.

thanks for clarifying. Any joy trying a different JDK? Also, have you tried the same config/setup but with an empty database (just in case it's your specific data that's somehow causing the issue)?

you could maybe also try monkey-patching that try-close function I linked above, see if there's a way to bypass the out-of-heap issue while printing

the db from before with the 30K transactions is the same kind of data, just 400mb instead of the 1080mb right now (300K tx) - I've not yet tried another jdk. I'm right now trying to disable the journal_mode in sqlite because of the shm part.

why I think that is relevant should've said before: I'm on Windows which may be a different kind of beast when it comes to multiple processes working on the same data

ah yes, that's an important detail too, I think it's fair to say I think most people working with XT v1 have been using Linux (sorry!) 🙂

I've updated to jdk21 (Eclipse Temerium) and also monkey patched try-close to not log c but rather just a generic string. It's still trying to .close right now and has been for over a few minutes. That usually is a bad sign.

I can try the migration on Linux later. I've intentionally chosen sqlite on Windows because a single file should work better than multiple files with ntfs. What an annoying system to work with, but that's the requirement at the moment. 😉

different java heap space error: https://pastebin.com/078kWcMx

interesting - begs the question...what is the memory config on this JVM? 🙂

-Xss2m, -Xmx12G, but I've tried with up to -Xmx40G (but before I did the monkeypatching, jdk upgrade and disabling journal_mode)

one last hail mary: I'm going with -Xmx40G on this physical 16GB system and see how far pagefiles get me.

more like the first one: https://pastebin.com/v7SwgVXK

I wasn't recording the memory consumption, but it didn't look like it was pressuring the system and swapping. Extreme pressure anyway, it does use the memory that is there... Hm.

was that last pastebin with Rocks/LMDB configured? looks like it was struggling to serialise the in-memory KV store ("mem-kv")

yeah

I saw a github issue (https://github.com/xtdb/xtdb/issues/1644) that is ignores certain keys and now thinking back to the success with the 480mb, that also created a file called "memkv" under the given path

so I reordered the index-store to be at the top

maybe it's not something order specific but some NREP when using both jdbc and also a kv-store. But seeing that github issue also using jdbc and having options ignored is very similar.

OK. That memkv thing really helped. Now I don't have to wait until OOM to figure out what's wrong, I can just check -> :index-store :kv-store type

it's MemKv with the above configuration for some reason

oh, how weird(!)

Wow. @taylor.jeremydavid, I got it. I have a fricking typo.

look at the config I posted, see if you can spot it 😉

xtdb-module facepalm right?

sorry for wasting your time

damn, egg on face 3000 😄

yes

aha, no not at all, it's way too easy

I guess some spec assertions or something can help with validation there.... "unexpected key" or whatever - sorry I couldn't help you sooner! I'm glad you persevered

yeah, I'm in the middle of writing a blogpost how static typing would've saved me and how Clojure isn't ready for primetime. But the better blog post would be about humans and because of the late failure mode I just chased the wrong tunnel down. Anyway, thanks again for the journey. 😉

now I'm on jdk21 at least, looking at the bright side of life

yeah virtual threads are cool, we use them a lot in v2

A natural primary key is probably the easiest, if the attribute(s) you're looking to make easiest are also guaranteed not to change for the lifetime of the entity. Otherwise, you could consider using ASSERT/`[:assert-{not-}exists ...]` to verify constraints on creation/update as required