Fork me on GitHub
#xtdb
<
2020-09-28
>
lispyclouds08:09:19

Hello, what would be a recommended way to check if my node is still connected to the storage? I have a node backed by Postgres and wanna write a health check in my service which alarms if the connection is broken/postgres is down etc

victorb08:09:17

as far as I could tell, the "Monitoring" part of the crux docs only talks about gathering metrics, not any healthchecks, so I ended up writing a function that makes a specific query and if there is zero or more results, return true and if there is any exceptions, return false. I also use that to measure latency over time

lispyclouds09:09:54

Right, but does doing a query guarantee that i goes all the way to the storage? I was thinking some of them maybe could be answered from the local node index.

victorb10:09:33

Hm, good question, don't think it guarantees that. Maybe checkout the output of (crux/status) in different failure scenarios, if you could use that?

lispyclouds10:09:54

I did try exactly that but the status seems to return fine even when the postgres instance is down

lispyclouds10:09:34

I'm almost inclined to do a raw JDBC query SELECT 1 + 1; and see if that bombs. Was looking for something better hopefully?

refset18:09:33

Hey @U7ERLH6JX - I agree some clearer health check for crux-jdbc would be useful. I will have a chat to the team. In the meantime I know open-tx-log is guaranteed to access the underling JDBC-powered tx-log

lispyclouds19:09:03

awesome! thanks a lot! i can use that for now. also if you could point me in some right directions I'm happy to raise my first PR to Crux! 😄

🙏 3
refset22:09:25

Awesome & we'd be delighted to guide your contribution :) I'll DM you tomorrow!

🎉 3
victorb12:09:17

More beginner questions, when I'm using :crux.tx/delete it seems my queries still return the thing I'm deleting but if I use :crux.tx/evict future queries won't show it. Do I need to adjust my queries to take deletions into account?

refset12:09:02

That doesn't right at all 🙂 delete should stop later queries seeing the deleted entity. Are you definitely querying against a db value that is take at a later vt/tt time when the delete will have taken effect? If you can show an example that would help Evict is probably not something you want to use unless you really need to

victorb12:09:17

Hurrah, then I'm not completely lost, probably just a little lost! Here is a sample: https://gist.github.com/victorb/27f74557f8f452880b2c95126697f9f1

victorb12:09:22

here is how I start the crux node, it's basically all rocksdb for now:

(def crux-opts
  {:my-rocksdb {:crux/module 'crux.rocksdb/->kv-store
                :db-dir (io/file db-path)}
   :crux/tx-log {:kv-store :my-rocksdb}
   :crux/document-store {:kv-store :my-rocksdb}
   :crux/index-store {:kv-store :my-rocksdb}})

victorb12:09:32

(just confirmed the same behavior with the in-memory crux too, just in case)

victorb12:09:16

Hm, so, stumbled upon crux/await-tx (via https://juxt.pro/blog/crux-tutorial-await) and if I wrap the delete transaction in await-tx, things seems to work as I expect... Nope. But definitely something wrong with my query rather than the delete

victorb14:09:38

A smaller example. I don't quite get why the delete works as I expect when there is just the :crux.db/id + :name attribute but if I add :age it shows up in the query even after running the delete

(crux/submit-tx (crux-node) [[:crux.tx/put {:crux.db/id :my-id
                                              :name "hello"
                                              :age 17}]])
  (crux/q
    (crux/db (crux-node))
    {:find '[n]
     :where '[[n :name name]
              [n :age age]]
     :args [{'name "hello"
             'age 17}]})

  (crux/submit-tx (crux-node) [[:crux.tx/delete :my-id]]))

victorb14:09:28

after running out of options to try, I tried replacing :args with :in and now the query doesn't show the deleted result... What's the difference between :args and :in? Can't find anything in the docs

refset14:09:22

Hi again, I'm still trying to sanity check your example 🙂 I wrote this for random-hex - presumably you had something similar:

(defn random-hex [n] (doall (apply str (doall (repeatedly n (partial rand-nth "0123456789ABCDEF"))))))
But the key difference between :in and :args is that :in isn't officially supported yet 😉

victorb14:09:59

@U899JBRPF take a look at the smaller example posted later, think that showcases (my misunderstanding of something) better. Hah, got it. Somehow it shows "old" data when using :args but shows no data (correct) when using :in

refset14:09:52

:in is technically superior but we couldn't implement previously until we had implemented destructuring (we which have now done in 1.12)

refset14:09:20

Interesting observation! I will try to reproduce

victorb14:09:23

Smaller example with :args commented out. If you run it with :args you'll see the data still gets returned from the query, but if you run it with :in (identical structure as with :args) after the delete, it's no longer visible...

(def crux-node (crux/start-node {}))
  (crux/submit-tx crux-node [[:crux.tx/put {:crux.db/id :my-id
                                            :name "hello"
                                            :age 17}]])
  (crux/q
    (crux/db crux-node)
    {:find '[n user-name age]
     :where '[[n :name user-name]
              [n :age age]]
     ;; :args [{'user-name "hello"
     ;;         'age 17
     :in [{'user-name "hello"
           'age 17}]})

  (crux/submit-tx crux-node [[:crux.tx/delete :my-id]]))

refset14:09:27

hmm, that's not how :in works and gives an error for me. Which version is this? Definitely 1.12? You can see via status

victorb14:09:17

@U899JBRPF I'm on 20.09-1.12.0-beta , confirmed via crux/status

✔️ 3
jarohen17:09:47

:in isn't part of 1.12.0 - the key will be ignored

✔️ 3
refset18:09:30

Thanks for figuring out the smaller example. I have managed to reproduce this issue now. Looks a lot like a bug to me! I'll keep you posted

victorb18:09:25

Hoho, I'm (more or less) glad to hear that it wasn't me doing something absolutely ridiculous. Thanks for looking into it!

🙂 3
jarohen18:09:47

raised https://github.com/juxt/crux/issues/1127 - thanks for flagging 🙂

🎉 3
victorb20:09:40

Two final questions: is this a regression or newly found issue? Feels like a pretty basic use case that it'd be weird that it hasn't been found before, or is there something else I should use instead of :args that is more idiomatic?

refset22:09:13

We'll figure out if there's been a regression in due course, but it looks like a fairly niche corner-case to me as I think it seems to only trigger when there's an exactly matching arg var for all values that actually belong to the entity. :args is the right way to do things to benefit from query compilation caches, however :in will likely be available in the next release (I misunderstood the status of it when I referred to :in earlier)

victorb14:09:43

Thanks a lot for explaining! Seems it's been fixed in master now (https://github.com/juxt/crux/pull/1130) cheers for fixing it so quickly! 👏

🙏 6
jarohen14:09:02

released with 1.12.1 - thanks for letting us know 🙂

❤️ 3
mitchelkuijpers12:09:01

If anyone is interested in a java only KV store I just released a new version for crux-xodus: https://github.com/avisi-apps/crux-xodus/releases/tag/1.0.0 This one is compatible with the new 1.12 release from Crux

🚀 15
3
victorb14:09:28

after running out of options to try, I tried replacing :args with :in and now the query doesn't show the deleted result... What's the difference between :args and :in? Can't find anything in the docs

timo19:09:19

I heard the notion that crux is 'transactor free'. What does that mean?

jonpither19:09:45

Crux doesn't have a single transactor like Datomic. Each node has a single writer that reads from the transaction log, i.e. kafka, and updates the nodes local index.

👍 3