This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-05-15
Channels
- # announcements (2)
- # babashka (137)
- # beginners (96)
- # calva (3)
- # cider (11)
- # clj-kondo (8)
- # cljs-dev (161)
- # cljsrn (21)
- # clojure (78)
- # clojure-europe (47)
- # clojure-france (1)
- # clojure-losangeles (1)
- # clojure-nl (4)
- # clojure-spec (24)
- # clojure-uk (9)
- # clojuredesign-podcast (4)
- # clojurescript (39)
- # conjure (2)
- # core-async (27)
- # cursive (36)
- # datomic (54)
- # emacs (6)
- # figwheel (9)
- # figwheel-main (46)
- # fulcro (25)
- # graalvm (8)
- # helix (30)
- # hoplon (6)
- # hugsql (3)
- # jobs (5)
- # leiningen (7)
- # luminus (12)
- # nrepl (20)
- # off-topic (20)
- # pedestal (16)
- # re-frame (14)
- # reagent (3)
- # reitit (3)
- # remote-jobs (5)
- # rum (25)
- # shadow-cljs (60)
- # spacemacs (10)
- # vim (2)
- # xtdb (36)
I submitted a put transaction of a map that didn't have a crux.db/id
key. Now, further submits throw an IllegalArgumentException "Missing required attribute :crux.db/id". It's like that transaction is still in the queue, but can't be processed because it doesn't have a required attribute, and I can't figure out how to remove it from the queue.
The problem transaction is stuck in the cache. It's persisted across REPL restarts.
(get-in adb/conn [:document-store :event-log-object-store :cache])
;; => #object[crux.lru$new_cache$reify__45334 0x3160d9b0 "{UnsafeBuffer{addressOffset=139787965043216, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}=#:person{:id 1, :name \"Sally\", :age 32}, UnsafeBuffer{addressOffset=139787965043248, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}=#:person{:id 2, :name \"Joe\", :age 23}, UnsafeBuffer{addressOffset=139787965368000, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}=#:person{:id 3, :name \"Fred\", :age 11}, UnsafeBuffer{addressOffset=139787965372064, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}=#:person{:id 4, :name \"Bobby\", :age 55}, UnsafeBuffer{addressOffset=139787965372192, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}={:crux.db/id :person-1, :person/id 1, :person/name \"Sally\", :person/age 32}, UnsafeBuffer{addressOffset=139787965372224, capacity=21, byteAr;; => ray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}={:crux.db/id :person-2, :person/id 2, :person/name \"Joe\", :person/age 23}, UnsafeBuffer{addressOffset=139787965372256, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}={:crux.db/id :person-3, :person/id 3, :person/name \"Fred\", :person/age 11}, UnsafeBuffer{addressOffset=139787965372288, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}={:crux.db/id :person-4, :person/id 4, :person/name \"Bobby\", :person/age 55}, UnsafeBuffer{addressOffset=139787965372320, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}={:crux.db/id :foobar, :val 1}}"]
It's hard to see in that noise, but the problem document is right at the start.
{UnsafeBuffer{addressOffset=139787965043216, capacity=21, byteArray=null, byteBuffer=java.nio.DirectByteBuffer[pos=0 lim=21 cap=21]}=#:person{:id 1, :name \"Sally\", :age 32},
Well, I added a clear method to the LRU cache that proxied to Java's HashMap clear. That got it out of the cache. But I'm still getting the same error about the missing id. I guess it's still on disk.
Should this be possible at all? (crux/submit-tx @node [[:crux.tx/put {}]])
Thanks for clarifying. Any guidance on removing the invalid document from the queue?
nothing that I can think of using the public API I'm afraid, will have a think on the easiest way :thinking_face:
how many corrupt documents are we talking? (roughly's fine, interested in whether it's 1-5 vs hundreds)
right. assuming you've got a handful, I'd do it by hand in a REPL:
let's assume you know which documents are corrupt (looks like you're getting them in the error message) - let's use {:person/id 1, :person/name "Sally", :person/age 32}
we can get the Crux content-hash for this document with (def doc-hash (crux.codec/new-id {:person/id 1, :person/name "Sally", :person/age 32}))
we then need to re-submit that doc directly to the doc-store with the :crux.db/id
added:
(crux.db/submit-docs (:document-store node) {doc-hash {:crux.db/id :sally, :person/id 1, :person/name "Sally", :person/age 32}})
one potential impact of this workaround would be that the content-hash doesn't strictly match the content. there's not many places we rely on this assumption, but one is match
- until you overwrite the document, you may find that match
operations don't match
in fact, yes - I'd be tempted to proactively follow it up with a corrected transaction to avoid this - eg another [:crux.tx/put {:crux.db/id :sally, ...}]
I'm forgetting what my "schema" looks like. I submitted some transactions but forgot what attributes I used. What's the best way to do exploratory browsing of data in Crux?
There's the attribute-stats
API which will show you full list of attributes across all time along with exact cardinalities
We have an upcoming HTML-based explorer UI baked in to the HTTP API on master
which you'd be welcome to test-drive
Specifically:
{:find [e]
:where [[e :crux.db/id _]]
:full-results? true}
This will return all your documents in full :) Big thanks to the community here. There's not nearly as much reference material for this kind of database and it's so different from SQL that it's a bit overwhelming to pick up.
Yes, there is never enough documentation, but it's great to have some good questions though!
Something I'm just now thinking about is how to handle what is typically handled by cascading deletes in SQL.
I create two users, Fred and Bob, and I make an association Fred has a friend, Bob. Then I delete Bob. Then I want to list all of Fred's non-deleted friends.
1. How is this typically handled?
2. The following query apparently doesn't work since ?friend
can't be evaluated inside ~(keyword ,,, (name ?friend))
. Is there a way to use a ?unification
variable inside an evaluation like I'm trying to do?
`{:find [?friend-entity]
:where [[?assoc :friends/from :fred]
[?assoc :friends/to ?friend]
[?friend-entity :crux.db/id ~(keyword "friends" (name ?friend))]]}))
(crux/submit-tx
adb/conn
[[:crux.tx/put
{:crux.db/id :friends/fred->bob
:friends/from :fred
:friends/to :bob}]
[:crux.tx/put
{:crux.db/id :users/fred}]
[:crux.tx/put
{:crux.db/id :users/bob}]])
(crux/submit-tx
adb/conn
[[:crux.tx/delete :users/bob]])
(crux/q
(crux/db adb/conn)
`{:find [?friend-entity]
:where [[?assoc :friends/from :fred]
[?assoc :friends/to ?friend]
[?friend-entity :crux.db/id ~(keyword "friends" (name ?friend))]]}))
I think you should be able to handle it as simply as this (haven't tested):
(crux/q
(crux/db adb/conn)
`{:find [?friend-entity]
:where [[?assoc :friends/from :fred]
[?assoc :friends/to ?friend]
[?friend :crux.db/id ?friend]]}))
We'll, the value that :friends/to
points to is :fred
, but the crux.db/id
of "Fred" is :users/fred
.
Of course I could change it so the association points to the actual Crux.db/id, but still curious if it's possible to dynamically build the unification variable.
Ahhhh, my apologies, I didn't study your example hard enough. I think you can do it by introducing intermediate vars like so (again, untested):
(crux/q
(crux/db adb/conn)
`{:find [?friend-entity]
:where [[?assoc :friends/from :fred]
[?assoc :friends/to ?friend]
[?friend-entity :crux.db/id ?kw]
[(keyword "friends" ?name) ?kw]
[(name ?friend) ?name]]}))
you could also create a custom clojure predicate for something more convenient, like:
[(some.namespace/path+name-to-keyword "friends" ?friend) ?kw]
> We'll, the value that :friends/to
points to is :fred
, but the :crux.db/id
of "Fred" is :users/fred
.
tbh I'd address this first - if you want a reference between two entities it'll save a fair bit of effort if you can make sure their ids are the same
^ yes, I think I would generally agree, as otherwise there will be a lot of "scanning" going on, making little efficient use of the indexes Also, I think the ns string arg for the keyword fn (both in the original example and my suggestions) needs to be "users"