Fork me on GitHub
Jacob O'Bryant03:06:22

I've started using a wrapper for crux.api/q; thought I'd share the details here as food for thought. I have some queries that look like this/look like parts of this:

; When deleting a user, get a list of document IDs for their data
; so it can be deleted too.
(defn stuff-ids-to-delete [db user-ids-to-delete]
  (when (not-empty user-ids-to-delete)
    (map first
      (crux/q db
        {:find '[stuff]
         ; crux/q complains if you don't give it a vector
         :args (vec
                 (for [u user-ids-to-delete]
                   {'user u}))
             ; Alternatively (though IMO this is less readable):
             ; (mapv #(hash-map 'user %) users-to-delete)
         :where '[[stuff :user user]]}))))
If user-ids-to-delete was empty and you didn't have the when, the query would return all stuff in the database. It would sure be embarrassing if you deployed a cron job to delete test accounts and it inadvertently cleared your whole database. Not saying that I, uh, have ever done that... (yay for immutability). Note that Datomic doesn't have this problem since it makes you specify input vars explicitly with :in. To guard against forgetting the when and to provide some other conveniences, I've started using this function:
(defn q [db {:keys [find args] :as query}]
  (let [has-args (contains? query :args)
        query (cond-> query
                has-args (update :args vec))]
    (when (or (not has-args) (not-empty args))
      (if (symbol? find)
        (map first (crux/q db (update query :find vector)))
        (crux/q db query)))))
The example then becomes:
(defn stuff-ids-to-delete [db user-ids-to-delete]
  ; If :args is included but empty, q returns nil.
  (q db
    {; If :find is a symbol, q will wrap it in a vector and call
     ; (map first ...) for you.
     :find 'stuff
     ; q also coerces :args to a vector.
     :args (for [u user-ids-to-delete]
             {'user u})
     :where '[[stuff :user user]]}))
Would any of these modifications make sense to have in crux.api/q? I haven't thought of any cases where you have :args set explicitly to nil etc. and want the query to execute, but maybe there are some. Seems like an error-prone case to me. Also, does :args require a vector on purpose (for performance)? Or could that restriction simply be relaxed?


Passing nil to args has bit me before as well, and I ended up putting the when guard around any query with args. I like your enhanced q though, I will try it out

Jacob O'Bryant08:06:39

Separate issue: I upgraded to the latest Crux version in dev today (standalone topology) and it didn't re-index my old data. Here's a minimal example:

; deps.edn
{:aliases {:old {:extra-deps
                 {juxt/crux-core    {:mvn/version "20.05-1.8.4-alpha"}
                  juxt/crux-rocksdb {:mvn/version "20.05-1.8.4-alpha"}}}
           :new {:extra-deps
                 {juxt/crux-core    {:mvn/version "20.06-1.9.1-beta"}
                  juxt/crux-rocksdb {:mvn/version "20.06-1.9.1-beta"}}}}}

; src/foo/core.clj
(ns foo.core
    [crux.api :as crux]))

(defn -main []
  (let [node (crux/start-node
               {:crux.kv/db-dir "crux-db"
                :crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
                :crux.standalone/event-log-dir "crux-event-log"
                :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})
        _ (crux/sync node)
        db (crux/db node)
        foo (crux/entity db :foo)]
    (prn foo)
    (crux/await-tx node
      (crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
                                           :test "hello"}]]))
    (.close node)))
Given those files:
$ clj -A:old -m foo.core
[some error messages about SLF4J]
$ clj -A:old -m foo.core
{:crux.db/id :foo, :test "hello"}
$ clj -A:new -m foo.core
So after upgrading the crux dependency (third clj call), the :foo entity is lost. This is a bug, right? Or am I missing something?

👀 3

Hey @U7YNGKDHA - thanks for the minimal repro, very useful. It should be alerting you that your index version is out of date, but isn't - I'll look into that one. Deleting the crux-db directory is sufficient to get it re-indexing using the new indices - after that, your clj -A:new -m foo.core works for me

👍 3
Jacob O'Bryant22:06:56

@U050V1N74 deleting crux-db doesn't make a difference for me. Given this updated version of src/foo/core.clj:

(ns foo.core
    [crux.api :as crux]))

(defn -main [cmd]
  (let [node (crux/start-node
               {:crux.kv/db-dir "crux-db"
                :crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
                :crux.standalone/event-log-dir "crux-event-log"
                :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})]
    (crux/sync node)
    (case cmd
      "prn" (prn (crux/entity (crux/db node) :foo))
      "tx" (crux/await-tx node
             (crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
                                                  :test "hello"}]])))
    (.close node)))
I get this:
$ ls
deps.edn  src
$ clj -A:old -m foo.core tx
$ clj -A:old -m foo.core prn
{:crux.db/id :foo, :test "hello"}
$ rm -r crux-db
$ clj -A:new -m foo.core prn
$ clj -A:new -m foo.core prn

👀 3

I started a node like this:

(crux/start-node {:crux.node/topology '[crux.standalone/topology crux.kv.lmdb/kv-store]
                    :crux.kv/db-dir (str (io/file storage-dir "db"))})
I have these deps:
juxt/crux-core {:mvn/version "20.06-1.9.1-beta"}
  juxt/crux-lmdb {:mvn/version "20.06-1.9.1-alpha"}
When I do:
(crux/sync node)
I get
Execution error (ExceptionInfo) at crux.kv.lmdb/success? (lmdb.clj:23).                                                                               
Invalid argument     
Exception Transaction consumer aborted                                                                                                                
        crux.tx/await-tx/fn--24442 (tx.clj:454)                                                                                                       
        crux.tx/await-tx (tx.clj:454)                                                                                                                 
        crux.tx/await-tx (tx.clj:452)                                                                                                                 
        crux.node.CruxNode (node.clj:125)                                                                                                             
        crux.api/eval15655/fn--15674 (api.clj:189)                                                                                                    
        crux.api/eval15463/fn--15563/G--15452--15576 (api.clj:69)                                                                                     
        crux.node.CruxNode (node.clj:113)                                                                                                             
        crux.api/eval15655/fn--15660 (api.clj:179)                                                                                                    
        crux.api/eval15463/fn--15522/G--15442--15531 (api.clj:69)                                                                                     
        io.dominic.crux/eval28478 (NO_SOURCE_FILE:135)                                                                                                
Caused by:                                                                                                                                            
ExceptionInfo Invalid argument {:error 22}
It doesn't seem to happen on a fresh node, so presumably I broke something with some data I put in?


Is that a node you've upgraded from <1.9.x?


nope, new today


It's not something immediately obvious I'm afraid - would it be easy enough to repro here?


Trying against a new folder to see if I can figure it out


I did just notice my first import (900k docs, easily) is dying of thread death.


Seems likely that something might get corrupted?


Should I try batching the imports?


I can sync again after close & opening the node :thinking_face:


Okay, I found the bad data.


Well, kinda. It's one of 316 docs.


# A fatal error has been detected by the Java Runtime Environment:
#  SIGSEGV (0xb) at pc=0x00007fbc5759968e, pid=26295, tid=0x00007fbc54344700
# JRE version: OpenJDK Runtime Environment (8.0_252-b09) (build 1.8.0_252-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  []
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# An error report file with more information is saved as:
# /home/overfl0w/workshop/commsor-pg-dt-cx-fast/bench-postgres/hs_err_pid26295.log
Compiled method (nm) 20648190 9416     n 0       org.lwjgl.util.lmdb.LMDB::nmdb_cursor_put (native)
 total in heap  [0x00007fbc8aa97f50,0x00007fbc8aa982c0] = 880
 relocation     [0x00007fbc8aa98078,0x00007fbc8aa980c0] = 72
 main code      [0x00007fbc8aa980c0,0x00007fbc8aa982b8] = 504
 oops           [0x00007fbc8aa982b8,0x00007fbc8aa982c0] = 8
# If you would like to submit a bug report, please visit:
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.


LMDB is looking a little bit unstable.


It is 😞 We've got a few changes coming through the pipelines which should fix a few known issues, would you be able to use a snapshot version?


yeah, this is just experimentation. Happy to give it a before/after attempt if you like.


Query speeds are more important than ingestion for me (snappy UIs beat showing up to date information)


Cheers 🙂 snapshot version is {:mvn/version "20.06-1.9.2-alpha-20200620.153650-1"}


Will give it a go, see what happens 🙂


What's the best way to implement "left joins" in Crux? Mine are all 1:1, I'm guessing the use of crux/entity is my friend? Performance matters here though, so willing to write something a little more convoluted.


I mean, this doesn't generally make sense unless you literally have imported SQL tables like I have. You'd model the data differently (you just merge everything together) if you were using crux from day #1.


Hey @U09LZR36F, we've been looking at this area a lot recently for the Calcite integration work. There's an unofficial and therefore intentionally undocumented predicate for doing this efficiently, get-attr, we've just not settled on it as a long term solution yet, so be prepared for a breaking change down the line should you decide it's worth the risk. See the tests:


That's not quite what I'm asking for. That's about optional keys in an entity. I'm asking about optional entities.


I guess it does make sense, if you want to make your life easier with predictable ids. You might divide data by id anyway


Ah, yeah I was thinking about the wording you used before your edit. A left/right/outer join at the entity level is tricky. I haven't seen an obvious and efficient solution for it within a single query, although maybe the new set semantics can help. I'll give it some thought. You can do a surprising amount with multiple levels of or-join and not but those approaches are unlikely to be efficient enough.


This issue has an example I worked on, before get-attr existed:


My actual problem is an inefficient one. Crux is 5x slower than postgres for it, probably because I'm doing so much in clojure. I'm pretty sure that postgres goes multithreaded to solve this query, so I might be faster with crux by using the reducers library. I'm searching for events pertaining to a user and counting them into partitions. I can solve some of the problems (e.g. Total event count by user) by subscribing to the history and adding the count to the users for every creation of an event pertaining to a user. One problem is that the user might not exist when the event comes in, but then is created later in the timeline, so I can't just shove it in, I'll need to persist it at the point it's removed.