Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2022-05-02

Channels

#xtdb
<
2022-05-02
>
tatut07:05:55

Range predicates with multiple values don't seem to work with strings. should they? example in thread

tatut07:05:01

user> (xt/submit-tx node
              [[::xt/put {:xt/id "lte1" :text "abcd" :num 5}]
               [::xt/put {:xt/id "lte2" :text "abba" :num 42}]
               [::xt/put {:xt/id "lte3" :text "azxc" :num 512}]])


user> (xt/q (db) '{:find [?e] :where [[?e :text ?t] [(<= minp ?t maxp)]] :in [minp maxp]} "aaaa" "abcd")
Execution error (ClassCastException) at xtdb.query/eval55586$fn$pred-constraint (query.clj:995).
class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap')

;; Works for numbers 
user> (xt/q (db) '{:find [?e] :where [[?e :num ?t] [(<= minp ?t maxp)]] :in [minp maxp]} 5 42)
#{["lte2"] ["lte1"]}

;; Separate clauses work for strings as well
user> (xt/q (db) '{:find [?e] :where [[?e :text ?t] [(<= minp ?t)] [(<= ?t maxp)]] :in [minp maxp]} "aaaa" "abcd")
#{["lte2"] ["lte1"]}

tatut07:05:15

should range predicates with multiple values work at all? or does it devolve to just using clojure function instead of using index

refset11:05:47

Range constraints (that use indexes) can only accept two args and one has to be a literal in order for them to happen early in the query (which is probably what you want). You can still use a regular <= by calling clojure.core/<= explicitly though.

tatut11:05:38

so the number example devolves to clojure.core/<= automatically?

refset12:05:39

It devolves to (comp not pos? compare), looking at https://github.com/xtdb/xtdb/blob/c21e93743e47716d5f4a31b11c8f6bd7acd58d87/core/src/xtdb/query.clj#L561 although I can't see how it can work with more than two args. Is that third arg maxp definitely being factored in?

tatut12:05:58

perhaps it should explicitly throw just to be sure, as this shouldn't work

refset12:05:35

Agreed, I think the spec needs to be tightened here somehow. Would you like to open an issue?

tatut12:05:57

I can do that

🙏 1
Hukka10:05:09

I now watched the Ultorg talk (https://www.hytradboi.com/2022/ultorg-a-user-interface-for-relational-databases) too. How would that work in a schemaless system where you don't have explicit foreign keys to guide the generic inspector tool?

refset11:05:21

Indeed, it would probably require something like Malli or another approach to gradual schema in the mix.

malli 1
tatut11:05:07

the old dabbledb presentations are very good in this regard

đź’Ż 1
tatut11:05:12

watched through the demo, looks very nice

Hukka11:05:56

Not really exactly an xtdb question since this is about evals, reader macros and stuff, but 

(xt/q (xt/db node)
        '{:find [pilot-name]
          :where [[?e :pilot-name pilot-name]
                  [?e :date date]
                  [(> foo date)]]
          :in [foo]}
        ;(. LocalDate parse "2023-01-01")
        #time/date "2020-01-01"
        )
gives "Can't embed object in code, maybe print-dup not defined: 2020-01-01", but the explicit LocalDate call works. What's happening?

Hukka11:05:48

This is with https://github.com/henryw374/time-literals in use, pulled in by juxt/tick

Hukka11:05:44

Oddly, using #inst works, even directly inside the quoted part

tatut12:05:01

odd indeed, as q is just a regular function

refset12:05:13

All args need to be serialisable via Nippy for binary comparisons. I'm not on a laptop though...do you have a stack trace?

Hukka12:05:46

Yeah, but I don't see how there's a difference, since https://github.com/henryw374/time-literals/blob/master/src/data_readers.cljc#L3 points to https://github.com/henryw374/time-literals/blob/master/src/time_literals/data_readers.clj#L5 which is then identical. And LocalDates are in the xtdb codecs.

Hukka12:05:06

#error {
 :cause "Can't embed object in code, maybe print-dup not defined: 2020-01-01"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "Syntax error compiling fn* at (/home/hukka/repos/xtdb-space/src/space.clj:26:7)."
   :data #:clojure.error{:phase :compile-syntax-check, :line 26, :column 7, :source "/home/hukka/repos/xtdb-space/src/space.clj", :symbol fn*}
   :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7114]}
  {:type java.lang.RuntimeException
   :message "Can't embed object in code, maybe print-dup not defined: 2020-01-01"
   :at [clojure.lang.Util runtimeException "Util.java" 221]}]
 :trace
 [[clojure.lang.Util runtimeException "Util.java" 221]
  [clojure.lang.Compiler$ObjExpr emitValue "Compiler.java" 4893]
  [clojure.lang.Compiler$ObjExpr emitConstants "Compiler.java" 4934]
  [clojure.lang.Compiler$ObjExpr compile "Compiler.java" 4612]
  [clojure.lang.Compiler$FnExpr parse "Compiler.java" 4106]
  [clojure.lang.Compiler analyzeSeq "Compiler.java" 7104]
  [clojure.lang.Compiler analyze "Compiler.java" 6789]
  [clojure.lang.Compiler eval "Compiler.java" 7173]
  [clojure.lang.Compiler eval "Compiler.java" 7166]
  [clojure.lang.Compiler eval "Compiler.java" 7131]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1242$fn__1243 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 665]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1973]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1973]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [nrepl.middleware.interruptible_eval$evaluate$fn__1242 invoke "interruptible_eval.clj" 87]
  [clojure.main$repl$read_eval_print__9068$fn__9071 invoke "main.clj" 414]
  [clojure.main$repl$read_eval_print__9068 invoke "main.clj" 414]
  [clojure.main$repl$fn__9077 invoke "main.clj" 435]
  [clojure.main$repl invokeStatic "main.clj" 435]
  [clojure.main$repl doInvoke "main.clj" 345]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1275$fn__1279 invoke "interruptible_eval.clj" 152]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__1344$fn__1348 invoke "session.clj" 218]
  [nrepl.middleware.session$session_exec$main_loop__1344 invoke "session.clj" 217]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 833]]}

Hukka12:05:32

The trace is confusing me too. It completely misses xtdb. However, I can use the #time/date syntax just fine elsewhere

refset12:05:05

Hmm! Weird. And this is using XT embedded, right? Not via the client?

Hukka12:05:11

Huh, no I can't. I can def it, but I cannot do 

(let [foo #time/date "2020-01-01"]
    (str foo))

Hukka12:05:50

Yeah, embedded xtdb. But now that I figured out that let is not working, it's clear that this is something else than xtdb thing

đź‘Ť 1
refset12:05:50

Ah, well I don't know much about the time-literals lib, but please feel free to keep us all posted here anyway

Hukka12:05:41

I'll try in the juxt-oss zulip

Hukka13:05:56

Ok, so I was using the library wrong when chasing a bug. In the old version (0.1.5) that worked, and in 0.1.6 it doesn't. Tick, for what's it worth, depends on the previous. With newer I need to run (time-literals.read-write/print-time-literals-clj!) to make the print-dup methods. But I have no idea why does it work on top level, but not in let, or in xt/q. Perhaps I never will know…

hidethepain 1
dealy66318:05:23

Hi I’m working on my first project with XTDB (total noob question), it is a multi-threaded app and different threads do reads and writes. Is there some kind of connection pool for XTDB? So far I haven’t quite figured out the best way to handle this and as expected using a single node instance causes problems with locking in different threads. Oh, I’m using LMDB for my persistence layer.

refset18:05:55

Hey! We're all perennial noobs when it comes to multi-threading 🙂 > as expected using a single node instance causes problems with locking in different threads That doesn't sound quite right :thinking_face: What are some examples of 'problems'? XT should definitely work happily across threads, but you do need to create a db per thread, i.e. don't attempt to the value returned from xtdb.api/db across threads

refset18:05:03

> Is there some kind of connection pool for XTDB? Nothing out of the box IIRC, unless you are using the HTTP server module - in which case I believe Jetty provides a thread pool

dealy66318:05:06

yeah, this may be just my own unfamiliarity with working with this type of system. When you say create a db per thread do you mean what is returned from start-node?

dealy66319:05:08

no I don’t think i’m using xtdb.api/db across threads. But I am getting write lock timeouts every once in a while, it is hard to reproduce them. I thought maybe that since I was using the same node for everything that maybe that was the problem, but all of my reads create their own (xt/db node*) while all writes share the same node* passed to the tx functions

refset20:05:03

Ah no, I mean xt/db per thread. Were you submitting from multiple threads?

dealy66320:05:38

yes submit-tx is called from multiple threads

refset20:05:31

I think that will be the culprit. You may want to stick a queue in front and have a dedicated submit-tx thread in that case. Could you share your start-node config map please?

dealy66321:05:13

:xtdb-config {:xtdb/index-store {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/indices”}}, :xtdb/document-store {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/docs”}}, :xtdb/tx-log {:kv-store {:xtdb/module xtdb.lmdb/->kv-store, :db-dir “data/transactions”}}}}

refset21:05:53

Thanks! You could also switch to Rocks for the tx-log and doc-store as a quick fix

dealy66321:05:44

rocks doesn’t work on apple silicon last I checked, I need this to run on mac, windows and linux

dealy66321:05:42

is there something different about the rocks implementation that would allow writing from multiple threads?

dealy66321:05:02

I was also kinda attracted to the better read performance of lmdb

đź‘Ť 1
refset21:05:06

Have you tried -beta3? We added the M1 Rocks update 🙂

dealy66321:05:41

no I didn’t know that the beta worked on M1

refset21:05:40

Rocks natively supports multi-threaded writing while LMDB strictly doesn't, I think

dealy66322:05:11

i haven’t found beta3, do you have a link?

Hukka06:05:16

I was looking at the https://github.com/xtdb/xtdb/blob/master/modules/http-server/src/xtdb/http_server.clj and I couldn't see anything that would make a queue or anything. Doesn't that code just allow calling the submit-tx endpoint with as much parallelism as Jetty defaults allow?

đź‘Ť 1
tatut09:05:29

I would expect the TxLog implementation to handle that, if it needs to

Hukka09:05:11

@U899JBRPF You had +1:d my question, but I'm not sure how to interpret it… is the server module working ok without special handling for multithreaded submits to the same xtdb-node, or does it have a bug?

refset23:05:40

I think the underlying LMDB kv-store module probably should handle things better using clojure.core/locking, and so the existing behaviour can be considered a bug. The http server module shouldn't need to compensate for anything here, and would probably trigger the bug quite easily.

refset23:05:02

I'll open an issue to assess further tomorrow/Thursday (when I'm back on a laptop!) if nobody else here feels like beating me to it 🙂

Hukka05:05:57

Hmh, I was looking for generic clojure resource pools, but on a quick search it seems like nobody has done much. Just very basic stuff on stackoverflow, in addition to the common thread and connection pools

1
Hukka05:05:20

I suppose I could make a ring middleware that set!s a var, if it's not set yet, to a node instance and then provides it for the rest of the stack :thinking_face: . That way I could make sure that every jetty thread has it's own instance. Though max 50 index store indexes sounds a bit overkill, since it's only needed for submits and reads can happen from the same store.