Can Crux’s underlying architecture support a default sort order?

I'm trying to follow the road through https://juxt.pro/blog/posts/a-bitemporal-tale.html#_setup

I'm getting a NullPointerException from the

this is arch linux

any ideas?

ps - I get the same sort of trouble using the memory kv

@hoppy we have not seen this error before. I'm seeing some rocksdb issues on github with this description. going to read through them.

What error are you getting with memory kv?

I'm also on arch linux 🙂

I created a issue for this https://github.com/juxt/crux/issues/226

the memory kv produces this trace, however it only does it when being fired up in a jacked-in calva (cider) repl. starting from a command line repl seems to work (at least load)

that error I have seen before. Any chance I can get you to try it out with crux "19.04-1.0.4-alpha-SNAPSHOT" 🙂 ?

and use the memory kv?

yea jack in and get the last error you again

I'm digging a bit into the rocksdb thing. They are playing the parlor trick of building the .so with crossbuild and stuffing it in the jar and extracting it otfly. Ask me how I know this is a bitey dog. They built it on a centos container, so likely the .so isn't quite so tasty on arch, but you say you are getting away with this?

and yes, I'll try the snapshot after coffee reload.

pretty much a clone of the prior version

just for fun @megakorre, what jvm / version are you running

openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+4)
OpenJDK 64-Bit Server VM (build 11.0.3+4, mixed mode)

alrighty, not that then

Compiling CruxTest.core you have some aot namespaces?

ok so I just reproduced the error by having a global (def system (crux/start-... in a namespace that was getting aot compiled. I'm assuming this is what you are doing to.

you are jacking in with the uberjar profile 😮

mkay, perhaps we fix that.

but then I have a understanding of what that bug is. I wonder if I can reproduce the rocksdb error with the same condition

worth a shot, but I expect the way they build that .so is problem

can I get some info about your system @hoppy? what are you running?

started with rocksdb out of AUR, built it myself last night, but that didn't change anything

the undefined symbol in the error is from a library jemalloc its available to install from pacman. Can you try installing it?

no effect, but I probably need to rebuild rocks after that

I did not have it installed myself and could not find it on my system so not sure. I got the impression that it should be statically linked into the rocksdb .so file but maybe I'm wrong about that

switching back to aur, maybe yours is dragging?

in the snapshot version you installed we are using 6.0.1

are you guys bringing your own rocksdb in the jar, or using what you find?

it should be the one from the jar

is the directive to have a dependancy on rocksdbjni valid then, need a different ver?

right yes you provide it yourself. Well either version should work I have tried both today on my machine (the other version being the one in the blog)

Seems like it's on them. I pulled the .so out of their jar and this is hanging out

what does ldd librocksdbjni-linux64.so get you?

heh you wish

linux-vdso.so.1 (0x00007fff4b7f3000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fcaecf60000) librt.so.1 => /usr/lib/librt.so.1 (0x00007fcaecf50000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fcaecdc0000) libm.so.6 => /usr/lib/libm.so.6 (0x00007fcaecc78000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fcaecc58000) libc.so.6 => /usr/lib/libc.so.6 (0x00007fcaeca90000) /usr/lib64/ld-linux-x86-64.so.2 (0x00007fcaed9a0000)

It's not calling for jemalloc, which means it didn't dynamically link that.

but this is their jar they built, I presume this is produced by rocksdb

yes it is

I probably need to dump the AUR and build it myself

although it begs the issue of how you are managing

Hi there again, I know I’m not using the right tool for the job but I’m storing small binary blobs as base64 strings in crux. I read that all top-level attributes are indexed automatically? What exactly does that mean? Can I use that to not have crux index an attribute by nesting it in another map? Thank you 🙂

One more thing I’m trying to figure out is how to implement transactional constraints properly. I suppose my struggles are trivial for someone with more experience implementing distributed data stores 😃 The cas transaction works great on single entities. When trying to implement for example a unique constraint for a certain attribute across entities, I can do that by cleaning up duplicate data later on using “fix-on-read” or “fix-on-write” (not ideal but possible). Also works. What I’m struggling with right now is deletes with possible references: If I have an entity and I like to delete it only if no one references it anymore, I first do a query to check if there are not references, then I can issue a delete transaction. I cannot ensure that no one created a new reference between my query and the deletion. Then only thing I can come up with is doing “fix-on-read” using the historical data. Guess this is not really specific to crux and I’m also glad about general resource on this topic 🙂 Also, are there any future plans to build a transactional layer on top of crux or is this simply the wrong use-case?