xtdb 2021-07-22 | Slack Archive

Samuel McHugh12:07:04

Hi all, It seems coll types aren't preserved when querying. As a crude example say I want to put the following document into my db.

(crux/submit-tx my-node
                [[:crux.tx/put {:crux.db/id 123
                                :x #{1 2 3}
                                :y [1 2 3]}]])

If I want to retrieve the values bound to my keys I get the following

(q '{:find [x]
     :where [[e :x x]]})
;=> #{[3] [2] [1]}

(q '{:find [y]
     :where [[e :y y]]})
;=> #{[3] [2] [1]}

I guess this relates to how crux stores facts, not documents. Is there no simple way to just retrieve the value as it was stored? I can preserve the type this way

(q '{:find [(pull e [:x])]
     :where [[e :crux.db/id]]})
;=> #{[{:x #{1 3 2}}]}

but the nesting is pretty frustrating.

refset13:07:42

Hi again 🙂 > I guess this relates to how crux stores facts, pretty much, yep. Sets and vectors get decomposed into individual AVs. If you can accept retrieving :x #{1 2 3} as a vector and not a set, then you can use the built-in get-attr predicate which will pull the individual AVs back into a relation: i.e. [(get-attr e :x) vs] ;; vs => [1 2 3] And if you really need the original value of :x as seen in the source document (i.e. retrieve the set), you can use the entity API inside of the query and get the value out, e.g.

[[(crux.api/entity $ e) ent]
 [(get ent :x) vs]]

where $ is the implicit db context that you optionally specify as the first element of :in (although at that point you don't actually pass in a corresponding db value for it, it's just "for show")

refset13:07:16

all that said, pull probably is what you want. When you said "the nesting is pretty frustrating" - do you mean the unavoidable outer two levels of #{[...]}? Crux doesn't implement the complete find spec, which would help a little with that

Samuel McHugh15:07:58

The below is a closer example to what I'm doing in my project. I could try to sub-query and pull things apart but it's not obvious to me how to do so. I didn't understand what you mean by 'implement the complete find spec'. Can I extend :find to return something other than a set of the findings?

Samuel McHugh16:07:27

Actually I'm realizing I can use direct equality instead of containment

(q '{:find [e]
     :in [n]
     :where [[e :x n]]}

Jacob O'Bryant16:07:51

re: find spec, I've been using this in my projects:

(defn q [db query & args]
  (let [return-tuples (vector? (:find query))
        query (cond-> query
                (not return-tuples) (update :find vector))
        results (apply crux/q db query args)]
    (cond->> results
      (not return-tuples) (map first))))

If the value of :find isn't a vector, then it'll wrap the results in (map first ...), e.g.

(q db '{:find (pull e [*]) ...})
=> #{{:crux.db/id ...} ...}

If something like this was added into crux that would be wonderful

refset16:07:43

The complete edn Datalog find specs include . and ... like this https://github.com/tonsky/datascript/blob/master/test/datascript/test/query_find_specs.cljc#L18-L28

👍 2

richiardiandrea15:07:59

Hi there, has anybody ever seen this error:

Warning: resetting Nippy thaw for custom type with id: :crux.codec/edn-id
Warning: resetting Nippy thaw for custom type with id: :crux.codec/id
Exception in thread "clojure.core.async.timers/timeout-daemon" java.lang.IllegalArgumentException: No implementation of method: :exec of protocol: #'clojure.core.async.impl.protocols/Executor found for class: clojure.core.async.impl.exec.threadpool$thread_pool_executor$reify__12555
	at clojure.core$_cache_protocol_fn.invokeStatic(core_deftype.clj:583)
	at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:575)
	at clojure.core.async.impl.protocols$eval142012$fn__142013$G__142003__142020.invoke(protocols.clj:43)
	at clojure.core.async.impl.dispatch$run.invokeStatic(dispatch.clj:21)
	at clojure.core.async.impl.dispatch$run.invoke(dispatch.clj:18)
	at clojure.core.async.impl.channels.ManyToManyChannel.close_BANG_(channels.clj:265)
	at clojure.core.async.impl.timers.TimeoutQueueEntry.close_BANG_(timers.clj:41)
	at clojure.core.async.impl.timers$timeout_worker.invokeStatic(timers.clj:49)
	at clojure.core.async.impl.timers$timeout_worker.invoke(timers.clj:43)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:748)

jarohen15:07:44

the top two are a Crux warning that you can ignore

jarohen15:07:58

I don't think the stack trace is related to Crux, I'm afraid - we don't have any core.async

richiardiandrea16:07:15

Ok thanks sounds good for some reason I thought that would be related ...

richiardiandrea19:07:48

Hi there, maybe doing it wrong here, but is there a way to "join" entity histories? In other words given a main entity with references, what is the best approach for aggregating all their histories?

refset19:07:11

Crux's Datalog isn't history aware (currently), so you have to handle it in memory, like so https://github.com/juxt/crux/blob/66896ba1e1cca365d024ab0cb9264bf62db809fe/crux-bench/src/crux/bench/ts_devices.clj#L216-L254 or https://github.com/juxt/crux/blob/66896ba1e1cca365d024ab0cb9264bf62db809fe/crux-bench/src/crux/bench/ts_weather.clj#L292-L347

richiardiandrea20:07:45

ok sounds good

richiardiandrea20:07:33

Another question about patterns, what if I wanted to match on an attribute. Would I use a transaction function instead?

✔️ 2

refset20:07:24

Yep, exactly 🙂

richiardiandrea22:07:25

is there a magic trick to def functions and try them in the repl before publishing them? I am sure there is 😄

refset07:07:59

You can use with-tx, or just write the function against a db directly and then port it. Not quite magic...

richiardiandrea14:07:52

Actually with-tx is a good idea - the thing is that tx fns are a bit opaque and difficult to debug - unless I am missing some more advance technique of course

2021-07-22

Channels