Fork me on GitHub
#xtdb
<
2021-10-05
>
jussi07:10:35

Hi, I'm trying to parameterize an XTDB query completely, but having hard time to get the field name to be bound to given value. My initial take, query does not work, but query2 below works, but it is quite ugly. Is there a way to bind vars to an attribute in a query? (complete newbie to datalog and xtdb here šŸ™‹ ). This though https://www.xtdb.com/reference/queries.html#datascript-differences, would suggest that it is not supported but the documentation location is quite odd, I personally expected this to be stated in the documentation for https://www.xtdb.com/reference/queries.html#in if this is really the case.

(let [field (keyword field)
      query '{:find [(pull ?doc [*])]
                :in [[?f ?v]]
                :where [[?doc ?f ?v]]}
      query2 `{:find [(~'pull ~'?doc [*])]
                :in [[~'?v]]
                :where [[~'?doc ~field ~'?v]]}]
   (xt/q (xt/db @xtdb-node) query [field value])

refset10:10:29

Hi @U0267SNCPEY thanks for the feedback, I will make a note about this limitation in the :in section, as I agree that may be more useful & obvious šŸ™‚ > it is not supported That is correct. See the Zulip thread mentioned in @@tatut's message below though (https://clojurians.slack.com/archives/CG3AM2F7V/p1633424129287800?thread_ts=1633419692.287500&amp;cid=CG3AM2F7V), as this gist may help you https://gist.github.com/refset/93135adcbf41fccab9b641638ab10997

refset10:10:30

Out of interest, have you come from a SQL background? Or have you also played with Cypher or other graph query languages?

jussi10:10:33

Thanks @U899JBRPF! Will continue with my "dirty hack" to keep the query fully parameterized. I'm coming from quite pure SQL background, just started a company with full Clojure stack and an actual business need for bi-temporality šŸ™‚

šŸ™ 1
jussi10:10:58

I have used GraphQL a little tho

jussi10:10:33

And then different document databases

refset10:10:15

ah, cool! > full Clojure stack and an actual business need for bi-temporality a surprisingly rare combination šŸ˜… It would be great to chat some time (if you have any spare?) about the kinds of bitemporal queries your org may need now / in the future, beyond point-in-time ("timeslice") queries. We are currently working on the new temporal operator APIs/semantics (including SQL:2011 support). I'd love to hear about the use-case also šŸ™‚

jussi10:10:27

Ah, I think the biz case is quite simple (and most likely doable with PostgreSQL or similar). We have a need to rerun calculations ie. we have evolving data points and we need to match data from certain time range to data points in that same time range.

šŸ‘ 1
jussi11:10:46

We are just two fullstackers with a varying background and experience (over 40 yrs combined) and Clojure just seems to be developer heaven šŸ˜…

šŸ™Œ 1
šŸ™‚ 1
Hukka19:10:39

Elaborating a bit, our case is generating reports based on facts, where we know that we will not have the facts ready in time. Usually we do, but sometimes they change much later. And we foresee having to explain why the old reports were "wrong", and having multiple versions of the reports (so report for Jan 2020 generated at the time, another one generated a year or two later based on any possible new information we got for that period after the fact)

Hukka19:10:47

XTDB won't solve all of our problems, we will also have the third dimension of code version, I'm afraid

Hukka19:10:05

Will have to see if we really need to have the routine capability to recalculate any time point with any code version. I hope not at all, or at most for very special cases where we can just do manual work with version control (as long as we record the code version as a fact in the reports)

refset22:10:18

> as long as we record the code version as a fact in the reports sounds pretty sensible! do you have any use/need for eviction?

Hukka06:10:34

Probably not. In case somebody sends as invalid data, we could just mark it as such. Though that makes the queries a bit complex, if we can't assume that all the data in the db should be used. Perhaps we could skip the complexity by also setting the dates so that they never match, as long as we would always use time based queries

šŸ‘Œ 1
Hukka06:10:30

We don't have any need for personal information, though I could imagine things leaking to us

šŸ‘ 1
refset07:10:36

Good to know. Thanks for the glimpse into what you're both working on šŸ™‚

tatut07:10:32

somewhat related to the above: as the attribute position can't have vars... is it possible to find out "who links to this document with <id>?", is there a VAE index?

xlfe08:10:29

I asked that question a few days ago on juxt-oss and refset provided the following suggestion https://juxt-oss.zulipchat.com/#narrow/stream/194466-xtdb-users/topic/reverse.20references.3F/near/255741555

šŸ™ 1
tatut09:10:48

:thumbsup: thanks, I'll take a look

refset10:10:49

> is there a VAE index? short answer: no šŸ™‚ there was some good discussion (including my own contributions šŸ˜…) in this thread about why/whether VAE is important https://github.com/tonsky/datascript/issues/351 (DataScript doesn't have it either)

tatut10:10:13

the use-case I have in mind is in my XTDB inspector, I would like to show "this document is linked from these other documents" listing when viewing a document... without any pre knowledge about what the possible attributes might be

tatut10:10:43

AVE doesn't cut it unless I iterate all possible attributes

tatut10:10:57

but I guess I could just add configuration so user needs to specify what attributes should be considered

refset10:10:10

> Providing a similar index to VAET is something that the Crux team has been contemplating recently, motivated by our work on a generic entity navigation UI. > (https://github.com/tonsky/datascript/issues/351#issuecomment-651245726) ...I can sympathise! Unless you have vast numbers of attributes though, a scan over the output of attribute-stats probably won't have too much impact to the user experience. And you can always lazy-load/stream the results to the UI

tatut10:10:14

in datomic the VAET index is only populated with ref values, not any others iirc

āœ”ļø 1
refset10:10:54

that's my understand also, and another reason why it'd be harder in XT, because lots of things might be refs (and consequently the index would be much bigger)

tatut10:10:17

but yes, perhaps the first approximation is just check AVE index with the specific [a v]pairs

šŸ‘Œ 2
allandaviesza12:10:42

If I have multiple ::xt/put in a single call to submit-tx can I be assured that all or none of those puts will be successful?

refset12:10:15

Yes, that's right šŸ™‚ transaction are "atomic"

allandaviesza12:10:40

I thought they must be, I failed to find that explicitly mentioned in the reference though, probably not reading properly

refset12:10:53

yeah I was just looking at how we describe it in https://xtdb.com/reference/1.19.0-beta1/transactions.html: currently at the top it says: > They are comprised of a series of Transaction Operations to be performed. but I'll probably clarify it to > Transactions are atomic, and comprised of a sequence of operations to be performed. the line below that also reads: > If the transaction contains pre-conditions, all pre-conditions must pass, or the entire transaction is aborted. This happens at the query node during indexing, and not when submitting the transaction

allandaviesza14:10:21

yea the last bit was confusing me, I didn't know if it meant you needed to use a match or not

šŸ‘Œ 1
šŸ™‚ 1
sheluchin20:10:57

I'm running into this error when trying to mount xtdb:

[2021-10-05 16:33:47.246 - WARNING] :shadow.cljs.devtools.server.nrepl-impl/init-ns-ex - {:init-ns sheluchin.user}
Note: The following stack trace applies to the reader or compiler, your code was not executed.
CompilerException Syntax error compiling at (juxt/clojars_mirrors/nippy/v3v1v1/taoensso/nippy.clj:1:1). #:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source "juxt/clojars_mirrors/nippy/v3v1v1/taoensso/nippy.clj"}
	clojure.core/throw-if (core.clj:5856)
	clojure.core/load-lib (core.clj:5942)
	clojure.core/load-lib (core.clj:5917)
	clojure.core/apply (core.clj:669)
	clojure.core/load-libs (core.clj:5978)
	clojure.core/load-libs (core.clj:5958)
	clojure.core/apply (core.clj:669)
	clojure.core/require (core.clj:5996)
	clojure.core/require (core.clj:5996)
	juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy/eval61292/loading--6737--auto----61293 (nippy.clj:1)
	juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy/eval61292 (nippy.clj:1)
	juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy/eval61292 (nippy.clj:1)
Caused by:
Exception namespace 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy.utils' not found
Can anyone suggest what I might be doing wrong?

jarohen20:10:21

afraid that's not one I've seen before - could you supply a minimal repro?

sheluchin20:10:48

Would take a bit of doing. Any low hanging fruit you think I could try before setting up a minimal repro?

jarohen20:10:28

I'd try requiring that utils namespace from a regular REPL, see if you get the same error - I don't know what impact shadow-cljs might have (I've been out of CLJS land for a few years now!)

sheluchin20:10:14

Tried that, it does give me the same:

Syntax error compiling at (juxt/clojars_mirrors/nippy/v3v1v1/taoensso/nippy.clj:1:1).
namespace 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy.utils' not found
It's weird. I did the crux->xtdb migration last week and everything seemed to be working correctly. Now I reboot everything and run into this. Maybe I had some state somewhere that was making it work and it got lost after the reboot.

jarohen20:10:09

what happens if you require the utils namespace directly?

jarohen20:10:48

if that doesn't work - ( "juxt/clojars_mirrors/nippy/v3v1v1/taoensso/nippy/utils.clj") (and, for comparison, ( "juxt/clojars_mirrors/nippy/v3v1v1/taoensso/nippy.clj") )

sheluchin20:10:04

(require 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy.utils) just returns nil, doesn't throw any errors.

sheluchin20:10:21

:deps        {org.clojure/clojure     {:mvn/version "1.10.3"}
               com.fulcrologic/fulcro  {:mvn/version "3.5.5"}
               com.wsscode/pathom      {:mvn/version "2.4.0"}
               ring/ring-core          {:mvn/version "1.9.4"}
               com.fzakaria/slf4j-timbre {:mvn/version "0.3.21"}
               com.taoensso/timbre     {:mvn/version "5.1.2"}
               http-kit/http-kit       {:mvn/version "2.3.0"}
               thheller/shadow-client  {:mvn/version "1.3.3"}
               cljsjs/rangy-textrange  {:mvn/version "1.3.0-1"}
               com.xtdb/xtdb-core      {:mvn/version "1.19.0-beta1"}
               com.xtdb/xtdb-lmdb      {:mvn/version "1.19.0-beta1"}
               ; com.taoensso/nippy      {:mvn/version "3.1.1"}
               mount/mount             {:mvn/version "0.1.16"}}

sheluchin21:10:55

(ns sheluchin.server.db.mounts
  (:require
   [xtdb.api :as xt]
   [mount.core :as mount :refer [defstate]]
   [taoensso.timbre :as log]
   [sheluchin.server.config :refer [config]])
  (:import (xtdb.api IXtdb)))

; see 
(defn- add-merge-tx!
  "Create a transaction function for patching an entity"
  [node]
  (xt/submit-tx node
   [[::xt/put {:xt/id :merge-tx
               :xt/fn '(fn [ctx eid kvs]
                        (let [db (xtdb.api/db ctx)
                              entity (xtdb.api/entity db eid)]
                          [[::xt/put (merge entity kvs)]]))}]]))

(defstate ^IXtdb xtdb-node
  :start (do
          (log/info "Starting XTDB node")
          ;; TODO: empty map good for now, but once I need more modules,
          ;;       I will need to add options targetting them
          ;;       
          ;;       e.g. sqlite, keeps data between restarts
          ;;       
          (doto (xt/start-node {})
            (add-merge-tx!)))
  :stop (do
          (log/info "Stopping XTDB node")
          (.close xtdb-node)))

(defn start []
  (mount/start #'xtdb-node))

(defn stop []
  (mount/stop #'xtdb-node))

(defn restart [] (stop) (start))

jarohen21:10:09

hmm - with that deps.edn, with and without nippy itself commented out, clj then (require 'xtdb.api), clj then (require 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy) and (require 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy.utils) all seem fine?

sheluchin21:10:32

āœ˜-INT ~/repos/clojure-stuff/prj [14-implement-backend-storage-crux-datalog L|āœš 3ā€¦7āš‘ 1] 
+17:04 $ Worker shutdown.
clj
Clojure 1.10.3
(require 'xtdb.api)
nil
user=> 
āœ˜-INT ~/repos/clojure-stuff/prj [14-implement-backend-storage-crux-datalog L|āœš 3ā€¦7āš‘ 1] 
+17:05 $ clj
Clojure 1.10.3
(require 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy)
nil
(require 'juxt.clojars-mirrors.nippy.v3v1v1.taoensso.nippy.utils)
nil
user=> 

sheluchin21:10:40

Same result with/without nippy commented.

sheluchin21:10:18

I am wondering if this has anything to do with my editor integration. I updated deps around that recently. Apologies if that is the case.

jarohen21:10:30

yeah, those look good I'm afraid šŸ˜

jarohen21:10:33

no worries, it happens šŸ™‚

sheluchin21:10:17

The mount code checks out too, eh?

jarohen21:10:48

I don't know whether it checks out, but I doubt it would fail with the same nippy.utils error

sheluchin21:10:46

Thanks @U050V1N74. I'll keep digging and report back when I find a fix.

šŸ‘ 1
šŸ™ 1
sheluchin22:10:32

@U050V1N74 I don't understand why, but it appears to work again after $ rm -rf ~/.m2/repository/juxt, recompiling, and restarting my tooling.

sheluchin22:10:25

Shoot, I should have backed it up and tested both copies instead of deleting.

1
šŸ˜… 1
šŸ˜‰ 1