Fork me on GitHub
#xtdb
<
2020-07-26
>
sveri10:07:29

I still have trouble understanding everything 😄 So, I have some code that adds entites and retrieves them. Between my tests I reset the database and everything seems to work. But then, all of a sudden, crux/entity stops working. This query returns a uuid as expected:

(ffirst (crux/q (crux/db n) {:find  '[e]
                             :where '[[e :apigen/type t]
                                      [e :email ?email]]
                             :args  [{'t :apigen/user '?email ""}]}))
=> #uuid"645b8639-1589-41ef-b0c1-933471ba8c59"
But when I call entity on it:
(crux/entity (crux/db n) (ffirst (crux/q (crux/db n) {:find  '[e]
                                                      :where '[[e :apigen/type t]
                                                               [e :email ?email]]
                                                      :args  [{'t :apigen/user '?email ""}]})))
=> nil
it returns nil.

refset10:07:27

Hi @U0677JTQX I think I know what this might be....can you post your topology map please?

sveri10:07:59

{:crux.node/topology    '[crux.standalone/topology crux.http-server/module crux.kv.rocksdb/kv-store]
 :crux.http-server/port 3988
 :crux.kv/db-dir        (str (io/file storage-dir "db"))}

refset10:07:24

Ah, okay so it looks like the problem is that your indexes are persisted but your event log is not. You should just need to add a :crux.standalone/event-log-dir key so that the db can return the full documents (which are stored in the event log, not the indexes). Running a standalone node with persisted indexes but without a persisted event log shouldn't really be supported, sorry about that. We're currently working on simplifying this whole area: https://github.com/juxt/crux/issues/818 In the meantime I will add more notes about needing the event-log-dir in the various docs/tutorials to help others

sveri11:07:18

Thank you @U899JBRPF, setting the event-log-dir works 🙂

🙏 3
sveri10:07:45

In other circumstances it works as expected.

quadron15:07:29

where can i read about crux's supported datatypes?

refset15:07:21

Hi @UA2JG2Y11 would you be using Crux in-process? Due to how nippy works, Crux can currently handle arbitrary Java objects as values...but we might limit that a bit in future. The full list of primitive datatypes in the indexes, that support efficient seeks, can be seen here: https://github.com/juxt/crux/blob/master/crux-core/src/crux/codec.clj#L133

👍 3
quadron15:07:22

thanks, yup i'm running it in-process at the moment

👍 3
dpsutton19:07:41

I'm following the tutorial using the rocks db storage implementation.

(crux/submit-tx
    node
    [[:crux.tx/put
      {:crux.db/id :dbpedia.resource/Pablo-Picasso ; id
       :name "Pablo"
       :last-name "Picasso"}
      #inst "2018-05-18T09:20:27.966-00:00"]])
was run and then i restart my repl. However, I'm getting results that seem strange to me. I've restarted the repl and i can query this but not get the entity.
working.cli> (crux/q (crux/db node)
                     '{:find [e]
                       :where [[e :name "Pablo"]]})
#{[:dbpedia.resource/Pablo-Picasso]}
working.cli> (crux/entity (crux/db node) :dbpedia.resource/Pablo-Picasso)
nil
The id seems to be there but crux/entity returns nil for that id

dpsutton19:07:56

and the rocks db started as per the configuration example

(defn start-rocks-node [storage-dir]
  (crux/start-node {:crux.node/topology '[crux.standalone/topology
                                          crux.kv.rocksdb/kv-store]
                    :crux.kv/db-dir (str (io/file storage-dir "db"))}))

(def node (start-rocks-node "testing"))

dpsutton20:07:49

oh. i see this is a "cold start" issue.

dpsutton20:07:23

but i'm not sure how to solve it unfortunately.

refset20:07:39

Hi @U11BV7MTK this same problem came up in a similar thread earlier today. You just need to add an :crux.standalone/event-log-dir configuration to make sure everything gets persisted properly (and I will attempt to add a note to the tutorial and docs right now 🙂 ): https://clojurians.slack.com/archives/CG3AM2F7V/p1595759489376600 Sorry for the confusion...

sveri21:07:15

Funny, I just saw this post and thought, hm, I have seen this today in my environment 😄

refset21:07:59

Again, sorry to have not nipped this in the bud sooner. Hopefully these changes will makes things clearer in the docs: https://github.com/juxt/crux/commit/e322a00a1c1c8fb81b9928ee20f6201505605b92 (I can't modify the current version docs before we cut 1.10.0 later in the week)

3
dpsutton21:07:58

works perfectly. thanks so much!

dpsutton21:07:33

will this commit update the published information at https://opencrux.com/docs#config-rocksdb ?

dpsutton21:07:11

just asking because that's where i went to get an example config

refset22:07:19

cool, yep that bit has been (will be!) updated 🙂

dpsutton22:07:49

awesome! thank you so much

🙏 3
sveri07:07:45

@U899JBRPF It seems like for me the setting still does not work. I thought it did, but did not come back to it until today. So this is my topology map:

{:crux.node/topology            '[crux.standalone/topology crux.http-server/module crux.kv.rocksdb/kv-store]
 :crux.http-server/port         crux-http-port
 :crux.standalone/event-log-dir (io/file storage-dir "event-log")
 :crux.kv/db-dir                (io/file storage-dir "indexes")}
But when I restart my server I have the same error happening, it does find my users via a query, but /entity returns nil. I see, the storage/indexes has some files, but no "event-log" folder ist created. If I create that manually no files are added there either.

refset07:07:12

Hi again, I'm not working today but @U050V1N74 should be able to help you debug

jarohen08:07:58

Hey @U0677JTQX - apologies, you'll also need :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv in that map - I'll ensure this is included in the docs. We're making changes to this setup as part of 1.11 - Crux was quite different when it was introduced 🙂

sveri08:07:23

@U050V1N74 Awesome, I think I got it now, at least I have some files in the event-log-folder 🙂

jarohen08:07:59

great! 🙂