Fork me on GitHub

Range predicates with multiple values don't seem to work with strings. should they? example in thread


user> (xt/submit-tx node
              [[::xt/put {:xt/id "lte1" :text "abcd" :num 5}]
               [::xt/put {:xt/id "lte2" :text "abba" :num 42}]
               [::xt/put {:xt/id "lte3" :text "azxc" :num 512}]])

user> (xt/q (db) '{:find [?e] :where [[?e :text ?t] [(<= minp ?t maxp)]] :in [minp maxp]} "aaaa" "abcd")
Execution error (ClassCastException) at xtdb.query/eval55586$fn$pred-constraint (query.clj:995).
class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap')

;; Works for numbers 
user> (xt/q (db) '{:find [?e] :where [[?e :num ?t] [(<= minp ?t maxp)]] :in [minp maxp]} 5 42)
#{["lte2"] ["lte1"]}

;; Separate clauses work for strings as well
user> (xt/q (db) '{:find [?e] :where [[?e :text ?t] [(<= minp ?t)] [(<= ?t maxp)]] :in [minp maxp]} "aaaa" "abcd")
#{["lte2"] ["lte1"]}


should range predicates with multiple values work at all? or does it devolve to just using clojure function instead of using index


Range constraints (that use indexes) can only accept two args and one has to be a literal in order for them to happen early in the query (which is probably what you want). You can still use a regular <= by calling clojure.core/<= explicitly though.


so the number example devolves to clojure.core/<= automatically?


It devolves to (comp not pos? compare), looking at although I can't see how it can work with more than two args. Is that third arg maxp definitely being factored in?


perhaps it should explicitly throw just to be sure, as this shouldn't work


Agreed, I think the spec needs to be tightened here somehow. Would you like to open an issue?


I can do that

🙏 1

I now watched the Ultorg talk ( too. How would that work in a schemaless system where you don't have explicit foreign keys to guide the generic inspector tool?


Indeed, it would probably require something like Malli or another approach to gradual schema in the mix.

malli 1

the old dabbledb presentations are very good in this regard

💯 1

watched through the demo, looks very nice


Not really exactly an xtdb question since this is about evals, reader macros and stuff, but

(xt/q (xt/db node)
        '{:find [pilot-name]
          :where [[?e :pilot-name pilot-name]
                  [?e :date date]
                  [(> foo date)]]
          :in [foo]}
        ;(. LocalDate parse "2023-01-01")
        #time/date "2020-01-01"
gives "Can't embed object in code, maybe print-dup not defined: 2020-01-01", but the explicit LocalDate call works. What's happening?


This is with in use, pulled in by juxt/tick


Oddly, using #inst works, even directly inside the quoted part


odd indeed, as q is just a regular function


All args need to be serialisable via Nippy for binary comparisons. I'm not on a laptop you have a stack trace?


Yeah, but I don't see how there's a difference, since points to which is then identical. And LocalDates are in the xtdb codecs.


#error {
 :cause "Can't embed object in code, maybe print-dup not defined: 2020-01-01"
 [{:type clojure.lang.Compiler$CompilerException
   :message "Syntax error compiling fn* at (/home/hukka/repos/xtdb-space/src/space.clj:26:7)."
   :data #:clojure.error{:phase :compile-syntax-check, :line 26, :column 7, :source "/home/hukka/repos/xtdb-space/src/space.clj", :symbol fn*}
   :at [clojure.lang.Compiler analyzeSeq "" 7114]}
  {:type java.lang.RuntimeException
   :message "Can't embed object in code, maybe print-dup not defined: 2020-01-01"
   :at [clojure.lang.Util runtimeException "" 221]}]
 [[clojure.lang.Util runtimeException "" 221]
  [clojure.lang.Compiler$ObjExpr emitValue "" 4893]
  [clojure.lang.Compiler$ObjExpr emitConstants "" 4934]
  [clojure.lang.Compiler$ObjExpr compile "" 4612]
  [clojure.lang.Compiler$FnExpr parse "" 4106]
  [clojure.lang.Compiler analyzeSeq "" 7104]
  [clojure.lang.Compiler analyze "" 6789]
  [clojure.lang.Compiler eval "" 7173]
  [clojure.lang.Compiler eval "" 7166]
  [clojure.lang.Compiler eval "" 7131]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1242$fn__1243 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "" 152]
  [clojure.lang.AFn applyTo "" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1242 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
  [clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
  [clojure.main$repl$fn__9077 invoke "main.clj" 435]
  [clojure.main$repl invokeStatic "main.clj" 435]
  [clojure.main$repl doInvoke "main.clj" 345]
  [clojure.lang.RestFn invoke "" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1275$fn__1279 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "" 22]
  [nrepl.middleware.session$session_exec$main_loop__1344$fn__1348 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__1344 invoke "session.clj" 217]
  [clojure.lang.AFn run "" 22]
  [java.lang.Thread run "" 833]]}


The trace is confusing me too. It completely misses xtdb. However, I can use the #time/date syntax just fine elsewhere


Hmm! Weird. And this is using XT embedded, right? Not via the client?


Huh, no I can't. I can def it, but I cannot do

(let [foo #time/date "2020-01-01"]
    (str foo))


Yeah, embedded xtdb. But now that I figured out that let is not working, it's clear that this is something else than xtdb thing

👍 1

Ah, well I don't know much about the time-literals lib, but please feel free to keep us all posted here anyway


I'll try in the juxt-oss zulip


Ok, so I was using the library wrong when chasing a bug. In the old version (0.1.5) that worked, and in 0.1.6 it doesn't. Tick, for what's it worth, depends on the previous. With newer I need to run (!) to make the print-dup methods. But I have no idea why does it work on top level, but not in let, or in xt/q. Perhaps I never will know…

hidethepain 1

Hi I’m working on my first project with XTDB (total noob question), it is a multi-threaded app and different threads do reads and writes. Is there some kind of connection pool for XTDB? So far I haven’t quite figured out the best way to handle this and as expected using a single node instance causes problems with locking in different threads. Oh, I’m using LMDB for my persistence layer.


Hey! We're all perennial noobs when it comes to multi-threading 🙂 > as expected using a single node instance causes problems with locking in different threads That doesn't sound quite right :thinking_face: What are some examples of 'problems'? XT should definitely work happily across threads, but you do need to create a db per thread, i.e. don't attempt to the value returned from xtdb.api/db across threads


> Is there some kind of connection pool for XTDB? Nothing out of the box IIRC, unless you are using the HTTP server module - in which case I believe Jetty provides a thread pool


yeah, this may be just my own unfamiliarity with working with this type of system. When you say create a db per thread do you mean what is returned from start-node?


no I don’t think i’m using xtdb.api/db across threads. But I am getting write lock timeouts every once in a while, it is hard to reproduce them. I thought maybe that since I was using the same node for everything that maybe that was the problem, but all of my reads create their own (xt/db node*) while all writes share the same node* passed to the tx functions


Ah no, I mean xt/db per thread. Were you submitting from multiple threads?


yes submit-tx is called from multiple threads


I think that will be the culprit. You may want to stick a queue in front and have a dedicated submit-tx thread in that case. Could you share your start-node config map please?


:xtdb-config {:xtdb/index-store {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/indices”}}, :xtdb/document-store {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/docs”}}, :xtdb/tx-log {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/transactions”}}}}


Thanks! You could also switch to Rocks for the tx-log and doc-store as a quick fix


rocks doesn’t work on apple silicon last I checked, I need this to run on mac, windows and linux


is there something different about the rocks implementation that would allow writing from multiple threads?


I was also kinda attracted to the better read performance of lmdb

👍 1

Have you tried -beta3? We added the M1 Rocks update 🙂


no I didn’t know that the beta worked on M1


Rocks natively supports multi-threaded writing while LMDB strictly doesn't, I think


i haven’t found beta3, do you have a link?


I was looking at the and I couldn't see anything that would make a queue or anything. Doesn't that code just allow calling the submit-tx endpoint with as much parallelism as Jetty defaults allow?

👍 1

I would expect the TxLog implementation to handle that, if it needs to


@U899JBRPF You had +1:d my question, but I'm not sure how to interpret it… is the server module working ok without special handling for multithreaded submits to the same xtdb-node, or does it have a bug?


I think the underlying LMDB kv-store module probably should handle things better using clojure.core/locking, and so the existing behaviour can be considered a bug. The http server module shouldn't need to compensate for anything here, and would probably trigger the bug quite easily.


I'll open an issue to assess further tomorrow/Thursday (when I'm back on a laptop!) if nobody else here feels like beating me to it 🙂


Hmh, I was looking for generic clojure resource pools, but on a quick search it seems like nobody has done much. Just very basic stuff on stackoverflow, in addition to the common thread and connection pools


I suppose I could make a ring middleware that set!s a var, if it's not set yet, to a node instance and then provides it for the rest of the stack :thinking_face: . That way I could make sure that every jetty thread has it's own instance. Though max 50 index store indexes sounds a bit overkill, since it's only needed for submits and reads can happen from the same store.