Passing nil to args has bit me before as well, and I ended up putting the when guard around any query with args. I like your enhanced q though, I will try it out

Hey @U7YNGKDHA - thanks for the minimal repro, very useful. It should be alerting you that your index version is out of date, but isn't - I'll look into that one. Deleting the crux-db directory is sufficient to get it re-indexing using the new indices - after that, your clj -A:new -m foo.core works for me

raised a PR with the fix, https://github.com/juxt/crux/pull/958

@U050V1N74 deleting crux-db doesn't make a difference for me. Given this updated version of src/foo/core.clj:

(ns foo.core
  (:require
    [crux.api :as crux]))

(defn -main [cmd]
  (let [node (crux/start-node
               {:crux.kv/db-dir "crux-db"
                :crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
                :crux.standalone/event-log-dir "crux-event-log"
                :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})]
    (crux/sync node)
    (case cmd
      "prn" (prn (crux/entity (crux/db node) :foo))
      "tx" (crux/await-tx node
             (crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
                                                  :test "hello"}]])))
    (.close node)))

$ ls
deps.edn  src
$ clj -A:old -m foo.core tx
$ clj -A:old -m foo.core prn
{:crux.db/id :foo, :test "hello"}
$ rm -r crux-db
$ clj -A:new -m foo.core prn
nil
$ clj -A:new -m foo.core prn
nil

Is that a node you've upgraded from <1.9.x?

nope, new today

It's not something immediately obvious I'm afraid - would it be easy enough to repro here?

Trying against a new folder to see if I can figure it out

I did just notice my first import (900k docs, easily) is dying of thread death.

Seems likely that something might get corrupted?

Should I try batching the imports?

I can sync again after close & opening the node :thinking_face:

Okay, I found the bad data.

Well, kinda. It's one of 316 docs.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbc5759968e, pid=26295, tid=0x00007fbc54344700
#
# JRE version: OpenJDK Runtime Environment (8.0_252-b09) (build 1.8.0_252-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [liblwjgl_lmdb.so+0x1168e]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/overfl0w/workshop/commsor-pg-dt-cx-fast/bench-postgres/hs_err_pid26295.log
Compiled method (nm) 20648190 9416     n 0       org.lwjgl.util.lmdb.LMDB::nmdb_cursor_put (native)
 total in heap  [0x00007fbc8aa97f50,0x00007fbc8aa982c0] = 880
 relocation     [0x00007fbc8aa98078,0x00007fbc8aa980c0] = 72
 main code      [0x00007fbc8aa980c0,0x00007fbc8aa982b8] = 504
 oops           [0x00007fbc8aa982b8,0x00007fbc8aa982c0] = 8
#
# If you would like to submit a bug report, please visit:
#   
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

LMDB is looking a little bit unstable.

It is 😞 We've got a few changes coming through the pipelines which should fix a few known issues, would you be able to use a snapshot version?

yeah, this is just experimentation. Happy to give it a before/after attempt if you like.

Query speeds are more important than ingestion for me (snappy UIs beat showing up to date information)

Cheers 🙂 snapshot version is {:mvn/version "20.06-1.9.2-alpha-20200620.153650-1"}

Will give it a go, see what happens 🙂

I mean, this doesn't generally make sense unless you literally have imported SQL tables like I have. You'd model the data differently (you just merge everything together) if you were using crux from day #1.

Hey @U09LZR36F, we've been looking at this area a lot recently for the Calcite integration work. There's an unofficial and therefore intentionally undocumented predicate for doing this efficiently, get-attr, we've just not settled on it as a long term solution yet, so be prepared for a breaking change down the line should you decide it's worth the risk. See the tests: https://github.com/juxt/crux/blob/7c749a89cf78a2bd5f868cc540fc1332014c0dab/crux-test/test/crux/query_test.clj#L966

That's not quite what I'm asking for. That's about optional keys in an entity. I'm asking about optional entities.

I guess it does make sense, if you want to make your life easier with predictable ids. You might divide data by id anyway

Ah, yeah I was thinking about the wording you used before your edit. A left/right/outer join at the entity level is tricky. I haven't seen an obvious and efficient solution for it within a single query, although maybe the new set semantics can help. I'll give it some thought. You can do a surprising amount with multiple levels of or-join and not but those approaches are unlikely to be efficient enough.

This issue has an example I worked on, before get-attr existed: https://github.com/juxt/crux/issues/835

My actual problem is an inefficient one. Crux is 5x slower than postgres for it, probably because I'm doing so much in clojure. I'm pretty sure that postgres goes multithreaded to solve this query, so I might be faster with crux by using the reducers library. I'm searching for events pertaining to a user and counting them into partitions. I can solve some of the problems (e.g. Total event count by user) by subscribing to the history and adding the count to the users for every creation of an event pertaining to a user. One problem is that the user might not exist when the event comes in, but then is created later in the timeline, so I can't just shove it in, I'll need to persist it at the point it's removed.