Fork me on GitHub
#xtdb
<
2021-08-31
>
tatut05:08:19

does the checkpointer work for lucene indexes as well?

tatut07:08:37

that's great!

tatut07:08:29

running crux nodes in ECS, so that's good to know

Thomas Moerman14:08:37

Hi guys, I'm a bit stuck on a query that matches documents that for a given prop either have no value (ie the prop isn't in the doc) OR have a given value (= arg). I think made it work using the inverse logical statement like this:

... where-clause...
(conj '(not-join
        [e ?user-id]
        [e :list/user ?x]
        [(not= ?x ?user-id)]))
For my better understanding of datalog: is there a "positive" way of expressing this using (or) or (or-join) ? edit This appears to work:
(conj '(or-join
        [e ?user-id]
        [e :list/user ?user-id]
        (not-join [e ?user-id] [e :list/user])))
Are these approaches correct?

refset15:08:04

Hi @U052A8RUT your (edit) solution looks sensible enough to me, so if it works okay then you can probably move on already. However, we have a fairly similar looking solution for a similar sounding problem that might inspire a slightly simpler approach: https://github.com/juxt/crux/blob/64e6f1f15ef2e389c42b8f17533c000bed9d33d7/crux-test/test/crux/query_test.clj#L1776-L1779

refset15:08:07

in your solution where you have (not-join [e ?user-id] [e :list/user]) - without knowing how else ?user-id is used/bound, I'm not certain it will return what you want. Are you getting nil out of it? (Is it in the :find?)

Thomas Moerman15:08:55

i'm not sure i'm following entirely. To clarify: I want to retrieve docs where this attribute doesn't exist, as well as the docs where it does exist AND is equal to a param value.

πŸ‘ 1
Thomas Moerman15:08:48

Ha, no it is not in the :find

Thomas Moerman15:08:32

The entire query is:

{:find '[e], 
 :where '[[e :list/id ?id] 
          (or-join [e ?user-id] 
                   [e :list/user ?user-id] 
                   (not-join [e ?user-id] 
                             [e :list/user]))
          [e :list/scope ?scope-id]], 
 :args [{?user-id 1, 
         ?scope-id "case-1"}]}

refset16:08:56

This should work for you, I think:

(crux.api/q
{:find '[e],
 :in '[?user-id ?scope-id] 
 :where '[[e :list/id] ;; no need to capture ?id in the triple clause if you're not using it
          (or-join [e ?user-id]
                   [e :list/user ?user-id] 
                   (and (not [e :list/user])
                        [(any? ?userid)]) ;; this is a noop predicate to keep the variables in the or-join legs balanced
          [e :list/scope ?scope-id]]}
1
"case-1")
As an aside, assuming you're not stuck on an old Crux version for some reason, it's best to use :in instead of :args

πŸ‘ 2
Thomas Moerman08:09:12

Thanks, I'll check it out soon

πŸ‘Œ 2
zclj14:08:58

After upgrading to the latest Crux version, with-redefs do no longer work as expected. I typically use it in some tests to test my logic around getting different results from crux.

(ns crux-with-redefs
  (:require [crux.api :as crux]))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; with-redefs example

(def mem-node (crux/start-node {}))

(with-redefs [crux/submit-tx (fn [_ _] :this-should-fail)]
  (crux/submit-tx
   mem-node
   [[:crux.tx/put {:crux.db/id  1
                   :person/name "Axon"}]]))

;; with - "20.12-1.13.0-beta"
;; => :this-should-fail

;; with - "1.18.1"
;; => #:crux.tx{:tx-id 0, :tx-time #inst "2021-08-31T14:34:03.184-00:00"}

(.close mem-node)
Is there an intentional change in Crux resulting in this?

jarohen15:08:09

short answer: no πŸ™‚ I get a distinct feeling I'm about to learn something about the innards of Clojure, though :thinking_face:

jarohen15:08:33

can repro locally, fwiw

zclj15:08:42

should I post an issue?

jarohen15:08:13

sure, although I'm happy to take a quick look, see if anything stands out

zclj15:08:25

ok, I'll hold and see what comes out. Thanks for looking into it πŸ™‚

jarohen15:08:59

alright - in that time crux/submit-tx has gone from being implemented via extend-protocol to being implemented directly on a defrecord - I wonder if there's anything about defrecord s that mean their implementations bypass with-redefs in any way (some form of inlining, maybe?)

jarohen15:08:14

looks that way:

(defprotocol P
  (foo [_ a]))

(defrecord R []
  P
  (foo [_ a]
    a))

(with-redefs [foo (constantly 10)]
  (foo (->R) 42))

;; => 42

jarohen15:08:07

whereas:

(defprotocol P
  (foo [_ a]))

(defrecord R [])

(extend-protocol P
  R
  (foo [_ a]
    a))

(with-redefs [foo (constantly 10)]
  (foo (->R) 42))

;; => 10

jarohen15:08:08

to workaround, if you're looking to simulate crux/submit-tx throwing an error, you could try redef'ing one of the callees of that function instead - crux.api/conform-tx-ops might be a good shout

zclj15:08:08

ok, thanks for the info and the workaround. Any chance the Crux implementation will change or will this be a trade-off of the internal implementation you have chosen?

jarohen15:08:31

it's unlikely to change, but obviously can't guarantee it - the fact that submit-tx calls conform-tx-ops isn't part of the public API πŸ™‚

zclj15:08:53

oh, I mean the change of going from extend-protocol to defrecord. That might have given you benefits that you do not want to revert for the sake of supporting with-redefs

jarohen15:08:10

oh, I see. yes, that one's similarly unlikely to be reverted - previously we were attaching the functionality to a Java class, whereas now we're using a Clojure record

jarohen15:08:38

you could also consider supplying your own implementation of PCruxNode to that function, with reify, say?

zclj15:08:03

right. Now that I know it is not a bug and will stay, I'll take a look at my workaround options. Maybe it could be worth documenting in some test related part, the current behaviour was a bit surprising πŸ™‚ Thanks for your support and looking into it, I also learned something about the Clojure internals πŸ™‚

πŸ™ 1
πŸ‘ 1
refset16:08:00

it's definitely fair to say that we've all learned a thing or two ☺️

βž• 1
tatut08:09:57

if the protocol has extend via metadata set, then you could just change submit-tx to throw using with-meta

jarohen08:09:57

I don't think that'll help on this occasion I'm afraid - looks like Clojure takes behaviours defined on a defrecord in preference to both

jarohen08:09:59

callers of protocol functions have bytecode to check whether the receiver implements the protocol interface first, calling the method directly if it does, and only falling back to the other protocol resolution logic if it doesn't