This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-06-20
Channels
- # aws (1)
- # babashka (68)
- # beginners (68)
- # braveandtrue (6)
- # calva (4)
- # cider (10)
- # clj-kondo (26)
- # clojure (76)
- # clojure-dev (18)
- # clojure-europe (1)
- # clojure-norway (25)
- # clojure-spec (8)
- # clojure-sweden (7)
- # clojure-uk (3)
- # clojuredesign-podcast (1)
- # clojurescript (11)
- # conjure (29)
- # cursive (31)
- # datomic (29)
- # emacs (12)
- # fulcro (29)
- # graphql (3)
- # helix (2)
- # hoplon (39)
- # hugsql (4)
- # malli (3)
- # off-topic (62)
- # pedestal (8)
- # re-frame (23)
- # reagent (14)
- # rewrite-clj (10)
- # shadow-cljs (18)
- # spacemacs (3)
- # sql (13)
- # xtdb (32)
I've started using a wrapper for crux.api/q
; thought I'd share the details here as food for thought.
I have some queries that look like this/look like parts of this:
; When deleting a user, get a list of document IDs for their data
; so it can be deleted too.
(defn stuff-ids-to-delete [db user-ids-to-delete]
(when (not-empty user-ids-to-delete)
(map first
(crux/q db
{:find '[stuff]
; crux/q complains if you don't give it a vector
:args (vec
(for [u user-ids-to-delete]
{'user u}))
; Alternatively (though IMO this is less readable):
; (mapv #(hash-map 'user %) users-to-delete)
:where '[[stuff :user user]]}))))
If user-ids-to-delete
was empty and you didn't have the when
, the query would return all stuff in the database. It would sure be embarrassing if you deployed a cron job to delete test accounts and it inadvertently cleared your whole database. Not saying that I, uh, have ever done that... (yay for immutability). Note that Datomic doesn't have this problem since it makes you specify input vars explicitly with :in
.
To guard against forgetting the when
and to provide some other conveniences, I've started using this function:
(defn q [db {:keys [find args] :as query}]
(let [has-args (contains? query :args)
query (cond-> query
has-args (update :args vec))]
(when (or (not has-args) (not-empty args))
(if (symbol? find)
(map first (crux/q db (update query :find vector)))
(crux/q db query)))))
The example then becomes:
(defn stuff-ids-to-delete [db user-ids-to-delete]
; If :args is included but empty, q returns nil.
(q db
{; If :find is a symbol, q will wrap it in a vector and call
; (map first ...) for you.
:find 'stuff
; q also coerces :args to a vector.
:args (for [u user-ids-to-delete]
{'user u})
:where '[[stuff :user user]]}))
Would any of these modifications make sense to have in crux.api/q
? I haven't thought of any cases where you have :args
set explicitly to nil
etc. and want the query to execute, but maybe there are some. Seems like an error-prone case to me. Also, does :args
require a vector on purpose (for performance)? Or could that restriction simply be relaxed?Passing nil to args has bit me before as well, and I ended up putting the when guard around any query with args. I like your enhanced q though, I will try it out
Separate issue: I upgraded to the latest Crux version in dev today (standalone topology) and it didn't re-index my old data. Here's a minimal example:
; deps.edn
{:aliases {:old {:extra-deps
{juxt/crux-core {:mvn/version "20.05-1.8.4-alpha"}
juxt/crux-rocksdb {:mvn/version "20.05-1.8.4-alpha"}}}
:new {:extra-deps
{juxt/crux-core {:mvn/version "20.06-1.9.1-beta"}
juxt/crux-rocksdb {:mvn/version "20.06-1.9.1-beta"}}}}}
; src/foo/core.clj
(ns foo.core
(:require
[crux.api :as crux]))
(defn -main []
(let [node (crux/start-node
{:crux.kv/db-dir "crux-db"
:crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
:crux.standalone/event-log-dir "crux-event-log"
:crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})
_ (crux/sync node)
db (crux/db node)
foo (crux/entity db :foo)]
(prn foo)
(crux/await-tx node
(crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
:test "hello"}]]))
(.close node)))
Given those files:
$ clj -A:old -m foo.core
[some error messages about SLF4J]
nil
$ clj -A:old -m foo.core
{:crux.db/id :foo, :test "hello"}
$ clj -A:new -m foo.core
nil
So after upgrading the crux dependency (third clj call), the :foo entity is lost.
This is a bug, right? Or am I missing something?Hey @U7YNGKDHA - thanks for the minimal repro, very useful.
It should be alerting you that your index version is out of date, but isn't - I'll look into that one.
Deleting the crux-db
directory is sufficient to get it re-indexing using the new indices - after that, your clj -A:new -m foo.core
works for me
raised a PR with the fix, https://github.com/juxt/crux/pull/958
@U050V1N74 deleting crux-db
doesn't make a difference for me. Given this updated version of src/foo/core.clj
:
(ns foo.core
(:require
[crux.api :as crux]))
(defn -main [cmd]
(let [node (crux/start-node
{:crux.kv/db-dir "crux-db"
:crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
:crux.standalone/event-log-dir "crux-event-log"
:crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})]
(crux/sync node)
(case cmd
"prn" (prn (crux/entity (crux/db node) :foo))
"tx" (crux/await-tx node
(crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
:test "hello"}]])))
(.close node)))
I get this:
$ ls
deps.edn src
$ clj -A:old -m foo.core tx
$ clj -A:old -m foo.core prn
{:crux.db/id :foo, :test "hello"}
$ rm -r crux-db
$ clj -A:new -m foo.core prn
nil
$ clj -A:new -m foo.core prn
nil
I started a node like this:
(crux/start-node {:crux.node/topology '[crux.standalone/topology crux.kv.lmdb/kv-store]
:crux.kv/db-dir (str (io/file storage-dir "db"))})
I have these deps:
juxt/crux-core {:mvn/version "20.06-1.9.1-beta"}
juxt/crux-lmdb {:mvn/version "20.06-1.9.1-alpha"}
When I do:
(crux/sync node)
I get
Execution error (ExceptionInfo) at crux.kv.lmdb/success? (lmdb.clj:23).
Invalid argument
Exception Transaction consumer aborted
crux.tx/await-tx/fn--24442 (tx.clj:454)
(io.clj:128)
(io.clj:122)
crux.tx/await-tx (tx.clj:454)
crux.tx/await-tx (tx.clj:452)
crux.node.CruxNode (node.clj:125)
crux.api/eval15655/fn--15674 (api.clj:189)
crux.api/eval15463/fn--15563/G--15452--15576 (api.clj:69)
crux.node.CruxNode (node.clj:113)
crux.api/eval15655/fn--15660 (api.clj:179)
crux.api/eval15463/fn--15522/G--15442--15531 (api.clj:69)
io.dominic.crux/eval28478 (NO_SOURCE_FILE:135)
Caused by:
ExceptionInfo Invalid argument {:error 22}
It doesn't seem to happen on a fresh node, so presumably I broke something with some data I put in?It's not something immediately obvious I'm afraid - would it be easy enough to repro here?
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fbc5759968e, pid=26295, tid=0x00007fbc54344700
#
# JRE version: OpenJDK Runtime Environment (8.0_252-b09) (build 1.8.0_252-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [liblwjgl_lmdb.so+0x1168e]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/overfl0w/workshop/commsor-pg-dt-cx-fast/bench-postgres/hs_err_pid26295.log
Compiled method (nm) 20648190 9416 n 0 org.lwjgl.util.lmdb.LMDB::nmdb_cursor_put (native)
total in heap [0x00007fbc8aa97f50,0x00007fbc8aa982c0] = 880
relocation [0x00007fbc8aa98078,0x00007fbc8aa980c0] = 72
main code [0x00007fbc8aa980c0,0x00007fbc8aa982b8] = 504
oops [0x00007fbc8aa982b8,0x00007fbc8aa982c0] = 8
#
# If you would like to submit a bug report, please visit:
#
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
It is 😞 We've got a few changes coming through the pipelines which should fix a few known issues, would you be able to use a snapshot version?
yeah, this is just experimentation. Happy to give it a before/after attempt if you like.
Query speeds are more important than ingestion for me (snappy UIs beat showing up to date information)
What's the best way to implement "left joins" in Crux? Mine are all 1:1, I'm guessing the use of crux/entity is my friend? Performance matters here though, so willing to write something a little more convoluted.
I mean, this doesn't generally make sense unless you literally have imported SQL tables like I have. You'd model the data differently (you just merge everything together) if you were using crux from day #1.
Hey @U09LZR36F, we've been looking at this area a lot recently for the Calcite integration work. There's an unofficial and therefore intentionally undocumented predicate for doing this efficiently, get-attr
, we've just not settled on it as a long term solution yet, so be prepared for a breaking change down the line should you decide it's worth the risk. See the tests: https://github.com/juxt/crux/blob/7c749a89cf78a2bd5f868cc540fc1332014c0dab/crux-test/test/crux/query_test.clj#L966
That's not quite what I'm asking for. That's about optional keys in an entity. I'm asking about optional entities.
I guess it does make sense, if you want to make your life easier with predictable ids. You might divide data by id anyway
Ah, yeah I was thinking about the wording you used before your edit. A left/right/outer join at the entity level is tricky. I haven't seen an obvious and efficient solution for it within a single query, although maybe the new set semantics can help. I'll give it some thought. You can do a surprising amount with multiple levels of or-join
and not
but those approaches are unlikely to be efficient enough.
This issue has an example I worked on, before get-attr
existed: https://github.com/juxt/crux/issues/835
My actual problem is an inefficient one. Crux is 5x slower than postgres for it, probably because I'm doing so much in clojure. I'm pretty sure that postgres goes multithreaded to solve this query, so I might be faster with crux by using the reducers library. I'm searching for events pertaining to a user and counting them into partitions. I can solve some of the problems (e.g. Total event count by user) by subscribing to the history and adding the count to the users for every creation of an event pertaining to a user. One problem is that the user might not exist when the event comes in, but then is created later in the timeline, so I can't just shove it in, I'll need to persist it at the point it's removed.