Fork me on GitHub
#xtdb
<
2020-07-30
>
peter hull08:07:31

Supposing I had some data where each record was like { :team "...", :squad [{ :name "...", :surname "..."}, { :name "...", :surname "..." }, ... ] } how would I feed it into crux so that I could ask questions like "Which teams have someone called Dan in the squad?"

jarohen08:07:22

In this case, Crux will index the maps in the vector, but won't index the individual fields of the maps within your vector - even as it stands, though, you'll be able to make queries like

'{:find [?team]
  :where [[?team :squad {:name "Dan", :surname "Smith"}]]}
(ie specifying the whole map to match against)

jarohen08:07:10

If you want 'people named Dan with any surname', this would be the point to consider extracting these members to their own documents:

;; documents
{:crux.db/id :my-team
 :team "My Team"
 :squad [:dan-smith :dan-brown :steve-green]}

{:crux.db/id :dan-smith
 :name "Dan"
 :surname "Smith"}

{:crux.db/id :dan-brown
 :name "Dan"
 :surname "Brown"}

;; query

'{:find [?team ?member]
  :where [[?team :squad ?member]
          [?member :name "Dan"]]}

;; => #{[:my-team :dan-smith] [:my-team :dan-brown]}

jarohen08:07:53

(aside: you can also use sets as values in documents if the order doesn't matter)

peter hull09:07:38

Thanks, that is much clearer for me now.

peter hull08:07:30

In other words - I've read https://opencrux.com/docs#_maps_and_vectors_in_data and I see how you would 'flatten' nested maps, but what if you had a nested vector of maps?

zclj13:07:37

Thanks for the work on EQL support! I'm playing around with it and have a question, is it possible to use an argument as the query? I have tried this: {:find '[(eql/project ?e ?q)] :where '[[?e :user/e-mail ?e-mail]] :args [{:?e-mail e-mail :?q query}]} but the result is a spec validation error of ?q not being a vector. Am I doing something wrong or is this not supported?

malcolmsparks13:07:40

I don't think you can pass an arg in like that for the projection. Are you able to query the query first, and then create another query to drive the projection?

zclj13:07:19

I'm not sure I follow what you suggest..

peter hull14:07:06

Could you use a syntax quote?

{:find  `[(eql/project ?e ~query)]
 :where '[[?e :user/e-mail ?e-mail]]
 :args  [{:?e-mail e-mail}]}

dominicm14:07:56

Be aware of escaping if you build the query dynamically. You wouldn't want someone to reference bindings in the query.

3
zclj16:07:37

The use case is to let clients provide the query, similar to GraphQL, have done it with Datomics pull syntax previously and would be nice to be able to do it with Crux as well.

jarohen16:07:34

It is possible with syntax-quoting, as @peterhull90 suggested - although with @dominicm's caveat of the dangers of allowing users to control the query. Fortunately a bit safer with data structures than with strings (because you can't escape out of the find into the where, for example), but not fool-proof. Building on @peterhull90's example, I think you'll need to include the whole query in a backtick (so that gensyms are consistent between find and where)

`{:find [(eql/project ?e# ~query)]
  :where [[?e# :user/e-mail ~'e-mail]]
  :args  ~[{:e-mail e-mail}]}

jarohen16:07:13

ah, or maybe easier, because I don't think we really need to worry about symbol clashes

{:find `[(eql/project ~'?e ~query)]
 :where '[[?e :user/e-mail ?e-mail]]
 :args  [{:?e-mail e-mail}]}
(ie difference from the example above is the ~'?e in the :find)

👍 3
genekim21:03:54

Public service announcement: Given inability to do XTDB logic variables for the “A” in “EAV”, here’s an example of how to do syntax quoting and splicing to acheve something similar. (I’m patting myself on the back for taking the time to finally learn this side of syntax quoting. 🎉 )

(defn shared-get-by-key!
  [db mykeyword kw id]
  (map first
    (xt/q (xt/db db)
      {:find  '[(pull e [*])]
       :in    '[?id ?type ?kw]
       :where `[[~'e :type ~'?type]
                [~'e ~kw ~'?id]]}
      ; forward quote the form
      ;   ~' ensures we get a symbol, not a var
      ;   ~ ensures we get the value of the symbol
      id mykeyword kw)))
In the next week, I’ll post an experience report of using XTDB to make this happen. The feeling of safety and lack of “OMG! NOOOO!” moments using XTDB for what is turning out to be looking a lot like ETL processes involving LLMs is just awesome. https://twitter.com/RealGeneKim/status/1762992284805861779

👏 2
dominicm16:07:50

At some point it seems easier to run (map eql/project (crux/q …))?

jarohen16:07:31

eql/project is only valid within :find at the moment, unfortunately - we haven't exposed it as a separate function

jarohen16:07:33

we did consider it, and discussed how long it might be until someone raised a use-case - answer: two days after release 🙂

dominicm17:07:12

You complected it!? Madness.

trollface 3
dominicm17:07:25

I will say that eql/project probably needs to assume that the query is malicious. I'm not sure if it does, just speculating. I have no idea how it functions internally.

jarohen17:07:03

(continued in DM, just in case there's a security impact)

dominicm17:07:08

Having discussed further, I can't see any obvious malicious user input problems here. None beyond the normal eql filtering I presume is done by anything consuming eql.

zclj08:07:47

Just to make sure I understand you correctly; quoting query, allowing for arbitrary code, is not a problem since it will be rejected as not valid EQL?

jarohen08:07:16

that's correct - we consume it as data, and then check it against EQL's provided spec

Vincent Cantin09:07:07

Would a recursive EQL query be a problem, from a security point of view?

jarohen09:07:00

We don't yet support recursive EQL queries - but if we did, and untrusted users were allowed to submit arbitrary queries, they could submit potentially expensive queries, yes.

refset09:07:09

^ subject to the normal timeouts though

👌 3
zclj17:07:54

@jarohen your quoting example works, but as mentioned it has some implications on client access. If possible, support for using args as in my first example would be on my wish list. That, in combination with some supporting functions, such as to remove any keys from the crux ns and maybe remove the underscore from any query result, would make building highly dynamic clients with lots of query power very straight forward. Thanks for EQL and your help!

🙏 3
thegeez17:07:10

@zclj I use something like

{:find [(list 'eql/project '?e query)]
 :where '[[?e :user/e-mail ?e-mail]]
 :args  [{:?e-mail e-mail}]}
for times when query can be changed or is supplied

zclj08:07:49

Good idea, thanks!

dominicm17:07:08

Having discussed further, I can't see any obvious malicious user input problems here. None beyond the normal eql filtering I presume is done by anything consuming eql.

genekim21:03:54

Public service announcement: Given inability to do XTDB logic variables for the “A” in “EAV”, here’s an example of how to do syntax quoting and splicing to acheve something similar. (I’m patting myself on the back for taking the time to finally learn this side of syntax quoting. 🎉 )

(defn shared-get-by-key!
  [db mykeyword kw id]
  (map first
    (xt/q (xt/db db)
      {:find  '[(pull e [*])]
       :in    '[?id ?type ?kw]
       :where `[[~'e :type ~'?type]
                [~'e ~kw ~'?id]]}
      ; forward quote the form
      ;   ~' ensures we get a symbol, not a var
      ;   ~ ensures we get the value of the symbol
      id mykeyword kw)))
In the next week, I’ll post an experience report of using XTDB to make this happen. The feeling of safety and lack of “OMG! NOOOO!” moments using XTDB for what is turning out to be looking a lot like ETL processes involving LLMs is just awesome. https://twitter.com/RealGeneKim/status/1762992284805861779

👏 2