Fork me on GitHub
#xtdb
<
2021-10-21
>
FiVo13:10:11

Hey, is there some "elegant" way of nuking everything from the db without sort of just deleting the files on disk? Mainly want to use it when playing around at the repl and in test fixtures. Or do people just normally create a new db instance in some temporary folder for these kind of cases?

jarohen14:10:16

Hi @UL638RXE2 👋 We tend to use in-memory XT instances for this (using (xt/start-node {})), and then throw it away at the end of the test. Our test fixture (assuming clojure.test) looks something like this:

(defn with-node [f]
  (with-open [node (xt/start-node {})]
    (binding [*api* node]
      (f))))

FiVo14:10:12

Ok thanks. That makes sense.

allandaviesza14:10:11

I swap out the node config to in memory for that kind of thing

👍 1
tatut14:10:50

what's the fastest way to check if value is an id (for any value)? I used a combination of checking that the type is a possible id with (some? (xt/entity db value)) in XTDB inspector... that was quite slow when rendering a table with thousands of values

tatut14:10:00

I changed it for now so that I check the query plan for which bindings are ids... but that doesn't help me when I'm rendering pulled data, I'd still like to show if something is id in that case as well (to link to the doc page)

tatut14:10:21

I'm guessing something from the xtdb.index ns, investigating that now

refset15:10:52

are you using (some? (xt/entity db value)) inside of an open-db?

refset15:10:17

that's not the most efficient option though, as the entity fn retrieves the entire document (potentially even hitting the doc store if the document cache is small). How about doing (some? (ffirst (xt/q db {:find [e] :where [[e :xt/id]]}))) inside an open-db? Once the initial query plan gets cached, I wouldn't expect any internal API to be faster than using that

tatut16:10:20

thanks, that looks good... basically I just want to check that a doc exists and the xt/entity was returning too much

👍 1
🤞 1
refset21:10:19

congrats on shipping the inspector, btw! Somehow I only just put 2+2 together and saw you also created ripley?! That's really awesome 😃 I am personally very excited about building local apps with a really dumb (web) UI, where latency isn't an issue and ripley might be the best bet. Would you be up for a chat later next week? Mind if I DM you?

tatut05:10:26

thanks, sure

tatut05:10:03

it would be interesting to see ripley used by others

tatut06:10:03

hmm, I seem to be getting very scary JVM crashes with this id checking query

tatut06:10:57

---------------  T H R E A D  ---------------

Current thread (0x00007fd6faae3a00):  JavaThread "clojure-agent-send-off-pool-5" [_thread_in_native, id=46863, stack(0x0000700007140000,0x0000700007240000)]

Stack: [0x0000700007140000,0x0000700007240000],  sp=0x000070000723e110,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [liblwjgl_lmdb.dylib+0xd5f3]
[error occurred during error reporting (printing native stack), id 0xe0000000, Internal Error (src/hotspot/os/bsd/decoder_machO.hpp:43)]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 7031  org.lwjgl.util.lmdb.LMDB.nmdb_cursor_open(JIJ)I (0 bytes) @ 0x00000001228eb9ef [0x00000001228eb9a0+0x000000000000004f]
J 7030 c1 org.lwjgl.util.lmdb.LMDB.mdb_cursor_open(JILorg/lwjgl/PointerBuffer;)I (26 bytes) @ 0x000000011b95d71c [0x000000011b95d3a0+0x000000000000037c]
J 7029 c1 xtdb.lmdb$new_cursor.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (120 bytes) @ 0x000000011b952a44 [0x000000011b951ec0+0x0000000000000b84]
J 7028 c1 xtdb.lmdb$new_cursor.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x000000011b951a34 [0x000000011b9519c0+0x0000000000000074]
J 6956 c1 xtdb.lmdb.LMDBKvSnapshot.new_iterator()Ljava/lang/Object; (65 bytes) @ 0x000000011b9210fc [0x000000011b920c00+0x00000000000004fc]
J 8964 c1 xtdb.kv.index_store$new_kv_index_snapshot$fn__10633.invoke()Ljava/lang/Object; (59 bytes) @ 0x000000011b36707c [0x000000011b366a20+0x000000000000065c]
J 4251 c2 clojure.lang.Delay.deref()Ljava/lang/Object; (75 bytes) @ 0x0000000122608184 [0x00000001226080a0+0x00000000000000e4]
J 8229 c2 xtdb.query$new_binary_index$fn__42198.invoke(Ljava/lang/Object;)Ljava/lang/Object; (68 bytes) @ 0x00000001229a2520 [0x00000001229a2300+0x0000000000000220]
J 8116 c2 xtdb.index.SeekFnIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object; (113 bytes) @ 0x00000001229324e0 [0x0000000122932440+0x00000000000000a0]
J 8373 c2 xtdb.index.DerefIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object; (67 bytes) @ 0x00000001229ca8d0 [0x00000001229ca820+0x00000000000000b0]
J 8784 c2 xtdb.index.NAryJoinLayeredVirtualIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object; (209 bytes) @ 0x0000000122a18990 [0x0000000122a188a0+0x00000000000000f0]
J 8373 c2 xtdb.index.DerefIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object; (67 bytes) @ 0x00000001229ca8d0 [0x00000001229ca820+0x00000000000000b0]
j  xtdb.index.UnaryJoinVirtualIndex$fn__40891.invoke()Ljava/lang/Object;+94
j  xtdb.index.UnaryJoinVirtualIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object;+17
J 8784 c2 xtdb.index.NAryJoinLayeredVirtualIndex.seek_values(Ljava/lang/Object;)Ljava/lang/Object; (209 bytes) @ 0x0000000122a18990 [0x0000000122a188a0+0x00000000000000f0]
J 8806 c2 xtdb.index$layered_idx__GT_seq$step__40939.invokePrim(Ljava/lang/Object;JLjava/lang/Object;)Ljava/lang/Object; (446 bytes) @ 0x00000001225eba58 [0x00000001225eb0e0+0x0000000000000978]
J 8201 c2 xtdb.index$layered_idx__GT_seq$step__40939.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x00000001229922e4 [0x00000001229922a0+0x0000000000000044]
j  xtdb.index$layered_idx__GT_seq.invokeStatic(Ljava/lang/Object;)Ljava/lang/Object;+126

tatut06:10:47

I'll try to investigate what value is actually causing this

tatut06:10:54

actually, nevermind... that is user error. I was doing it in another thread and using with-open... that was not correct, as we discussed in the other thread

👍 1
refset22:10:59

Thanks for following up & glad you figured it out!