Clojurians Log v2
xtdb 2024-11-06
Hi there,
we're trying to set up a local xtdb node/connect a running clojure process to a remote node, but we keep getting connection related exceptions when trying to invoke start-node /`start-client` .
We can execute transactions/queries against the remote node running via docker using jdbc, but we're not having any luck with the xt client. We've set the jvm/cmd-line options so it's not clear how we could have messed up our config.
JDK 22/23 with clojure 1.12.0
I wonder if it's possible you have a classpath conflict - is this a greenfield codebase? Or are you bringing XT into an existing app?
fresh project
hmm, please could you share the output of clj -Stree (or equivalent)?
ran lein deps :tree :
Possibly confusing dependencies found:
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [io.netty/netty-common "4.1.82.Final"]
overrides
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [org.apache.arrow/arrow-memory-netty-buffer-patch "17.0.0"] -> [io.netty/netty-buffer "4.1.110.Final"] -> [io.netty/netty-common "4.1.110.Final"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [org.apache.arrow/arrow-memory-netty-buffer-patch "17.0.0"] -> [io.netty/netty-buffer "4.1.110.Final"] -> [io.netty/netty-common "4.1.110.Final"]
and
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [org.apache.arrow/arrow-memory-netty-buffer-patch "17.0.0"] -> [io.netty/netty-common "4.1.110.Final"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [org.apache.arrow/arrow-memory-netty-buffer-patch "17.0.0"] -> [io.netty/netty-common "4.1.110.Final"]
and
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [io.netty/netty-common "4.1.110.Final"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-memory-netty "17.0.0"] -> [io.netty/netty-common "4.1.110.Final"]
Consider using these exclusions:
[com.xtdb/xtdb-core "2.0.0-beta2" :exclusions [io.netty/netty-common]]
[com.xtdb/xtdb-api "2.0.0-beta2" :exclusions [io.netty/netty-common]]
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [commons-codec "1.15"]
overrides
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-compression "17.0.0"] -> [org.apache.commons/commons-compress "1.26.2"] -> [commons-codec "1.17.0"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-compression "17.0.0"] -> [org.apache.commons/commons-compress "1.26.2"] -> [commons-codec "1.17.0"]
and
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [commons-codec "1.17.0"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [commons-codec "1.17.0"]
Consider using these exclusions:
[com.xtdb/xtdb-core "2.0.0-beta2" :exclusions [commons-codec]]
[com.xtdb/xtdb-api "2.0.0-beta2" :exclusions [commons-codec]]
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [com.cognitect/transit-java "1.0.371"] -> [com.fasterxml.jackson.core/jackson-core "2.14.2"]
overrides
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.datatype/jackson-datatype-jsr310 "2.17.1"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
and
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.core/jackson-databind "2.17.1"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.datatype/jackson-datatype-jsr310 "2.17.1"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.core/jackson-databind "2.17.1"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
and
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
and
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.apache.arrow/arrow-vector "17.0.0"] -> [com.fasterxml.jackson.core/jackson-core "2.17.1"]
Consider using these exclusions:
[com.xtdb/xtdb-core "2.0.0-beta2" :exclusions [com.fasterxml.jackson.core/jackson-core]]
[com.xtdb/xtdb-api "2.0.0-beta2" :exclusions [com.fasterxml.jackson.core/jackson-core]]
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.jetbrains.kotlinx/kotlinx-serialization-json-jvm "1.6.0"] -> [org.jetbrains.kotlin/kotlin-stdlib-common "1.9.0"]
overrides
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [com.charleskorn.kaml/kaml-jvm "0.56.0"] -> [org.jetbrains.kotlinx/kotlinx-serialization-core-jvm "1.6.2"] -> [org.jetbrains.kotlin/kotlin-stdlib-common "1.9.21"]
Consider using these exclusions:
[com.xtdb/xtdb-core "2.0.0-beta2" :exclusions [org.jetbrains.kotlin/kotlin-stdlib-common]]
[com.xtdb/xtdb-api "2.0.0-beta2"] -> [org.jetbrains.kotlinx/kotlinx-serialization-json-jvm "1.6.0"] -> [org.jetbrains.kotlinx/kotlinx-serialization-core-jvm "1.6.0"]
overrides
[com.xtdb/xtdb-core "2.0.0-beta2"] -> [com.charleskorn.kaml/kaml-jvm "0.56.0"] -> [org.jetbrains.kotlinx/kotlinx-serialization-core-jvm "1.6.2"]
Consider using these exclusions:
[com.xtdb/xtdb-core "2.0.0-beta2" :exclusions [org.jetbrains.kotlinx/kotlinx-serialization-core-jvm]]
[com.github.seancorfield/next.jdbc "1.3.939"]
[camel-snake-kebab "0.4.3"]
[org.clojure/java.data "1.2.107"]
[com.xtdb/xtdb-api "2.0.0-beta2"]
[com.cognitect/transit-clj "1.0.333"]
[com.cognitect/transit-java "1.0.371"]
[com.fasterxml.jackson.core/jackson-core "2.14.2"]
[javax.xml.bind/jaxb-api "2.4.0-b180830.0359"]
[javax.activation/javax.activation-api "1.2.0"]
[org.msgpack/msgpack "0.6.12"]
[com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
[org.javassist/javassist "3.18.1-GA"]
[com.github.ben-manes.caffeine/caffeine "3.1.8"]
[com.google.errorprone/error_prone_annotations "2.21.1"]
[org.apache.arrow/arrow-algorithm "17.0.0"]
[org.apache.arrow/arrow-memory-core "17.0.0"]
[com.google.code.findbugs/jsr305 "3.0.2"]
[org.apache.arrow/arrow-compression "17.0.0"]
[com.github.luben/zstd-jni "1.5.6-3"]
[org.apache.commons/commons-compress "1.26.2"]
[commons-io "2.16.1"]
[org.apache.commons/commons-lang3 "3.14.0"]
[org.apache.arrow/arrow-memory-netty "17.0.0"]
[org.apache.arrow/arrow-memory-netty-buffer-patch "17.0.0"]
[io.netty/netty-buffer "4.1.110.Final"]
[org.apache.arrow/arrow-vector "17.0.0"]
[com.fasterxml.jackson.core/jackson-annotations "2.17.1"]
[com.fasterxml.jackson.core/jackson-databind "2.17.1"]
[com.fasterxml.jackson.datatype/jackson-datatype-jsr310 "2.17.1"]
[com.google.flatbuffers/flatbuffers-java "24.3.25"]
[org.apache.arrow/arrow-format "17.0.0"]
[org.slf4j/slf4j-api "2.0.13"]
[org.jetbrains.kotlin/kotlin-stdlib-jdk8 "1.9.22"]
[org.jetbrains.kotlin/kotlin-stdlib-jdk7 "1.9.22"]
[org.jetbrains.kotlin/kotlin-stdlib "1.9.22"]
[org.jetbrains/annotations "13.0"]
[org.jetbrains.kotlinx/kotlinx-serialization-json-jvm "1.6.0"]
[org.jetbrains.kotlin/kotlin-stdlib-common "1.9.0"]
[org.jetbrains.kotlinx/kotlinx-serialization-core-jvm "1.6.0"]
[com.xtdb/xtdb-core "2.0.0-beta2"]
[clj-commons/clj-yaml "1.0.27"]
[org.flatland/ordered "1.15.11"]
[org.yaml/snakeyaml "2.1"]
[com.carrotsearch/hppc "0.9.1"]
[com.charleskorn.kaml/kaml-jvm "0.56.0"]
[org.snakeyaml/snakeyaml-engine "2.7" :scope "runtime"]
[commons-codec "1.15"]
[io.micrometer/micrometer-core "1.12.2"]
[io.micrometer/micrometer-commons "1.12.2"]
[io.micrometer/micrometer-observation "1.12.2"]
[org.hdrhistogram/HdrHistogram "2.1.12" :scope "runtime"]
[org.latencyutils/LatencyUtils "2.0.3" :scope "runtime" :exclusions [[org.hdrhistogram/HdrHistogram]]]
[io.micrometer/micrometer-registry-prometheus "1.12.2"]
[io.prometheus/simpleclient_common "0.16.0"]
[io.prometheus/simpleclient "0.16.0"]
[io.prometheus/simpleclient_tracer_otel "0.16.0"]
[io.prometheus/simpleclient_tracer_common "0.16.0"]
[io.prometheus/simpleclient_tracer_otel_agent "0.16.0"]
[io.netty/netty-common "4.1.82.Final"]
[org.antlr/antlr4-runtime "4.13.1"]
[org.antlr/antlr4 "4.13.1"]
[com.ibm.icu/icu4j "72.1"]
[org.abego.treelayout/org.abego.treelayout.core "1.0.3"]
[org.antlr/ST4 "4.3.4"]
[org.antlr/antlr-runtime "3.5.3"]
[org.babashka/sci "0.6.37"]
[borkdude/sci.impl.reflector "0.0.1"]
[org.babashka/sci.impl.types "0.0.2"]
[org.clojure/data.csv "1.0.1"]
[org.clojure/data.json "2.4.0"]
[org.clojure/tools.cli "1.0.206"]
[org.clojure/tools.logging "1.2.4"]
[org.roaringbitmap/RoaringBitmap "1.0.1"]
[pro.juxt.clojars-mirrors.integrant/integrant "0.8.0"]
[pro.juxt.clojars-mirrors.weavejester/dependency "0.2.1"]
[com.xtdb/xtdb-http-client-jvm "2.0.0-beta2"]
[pro.juxt.clojars-mirrors.hato/hato "0.8.2"]
[pro.juxt.clojars-mirrors.metosin/reitit-core "0.5.15"]
[metosin/malli "0.16.4"]
[borkdude/dynaload "0.3.5"]
[borkdude/edamame "1.4.27"]
[org.clojure/tools.reader "1.3.4"]
[fipp "0.6.26"]
[org.clojure/core.rrb-vector "0.1.2"]
[mvxcvi/arrangement "2.1.0"]
[org.clojure/test.check "1.1.1"]
[nrepl "1.0.0" :exclusions [[org.clojure/clojure]]]
[org.clojure/clojure "1.12.0"]
[org.clojure/core.specs.alpha "0.4.74"]
[org.clojure/spec.alpha "0.5.238"]
[org.nrepl/incomplete "0.1.0" :exclusions [[org.clojure/clojure]]]
[org.postgresql/postgresql "42.7.4"]
[org.checkerframework/checker-qual "3.42.0"]
[tick "1.0"]
[com.widdindustries/cljc.java-time "0.1.21"]
[com.widdindustries/cljs.java-time "0.1.20"]
[com.widdindustries/time-literals "0.1.10" :exclusions [[com.widdindustries/cljs.java-time]]]thanks, yep that looks like a potential conflict to me, [io.netty/netty-common "4.1.82.Final"] vs [io.netty/netty-common "4.1.110.Final"] - I don't understand what might have happened yet though, maybe try a lein clean if you haven't already?
Probably got pulled in due to the older release being available locally? It was a fresh project so we declared the latest versions for all our deps. Ran lein clean. It still seems to be missing the Recycler class when I try run a transaction against the node.
; Execution error (ClassNotFoundException) at io.netty.buffer.PooledByteBuf/<init> (PooledByteBuf.java:47).
; io.netty.util.Recycler$EnhancedHandleDoes lein deps :tree give the same output? Do you still see 4.1.82?
hey @edward.hughes1911 👋
netty-common 4.1.110 does have Recycler in, so my guess would be that it's somehow still pulling in an old version.
lein can be a bit problematic here - maybe a lein classpath would help with exactly what's getting pulled in?
you may also have more luck with the clj tooling, because lein relies on ordering of dependencies (borrowed from Maven), whereas clj always uses the latest version of transitive dependencies
No luck messing with the deps to get client libraries running just yet. We tried using clj to start the project and resolve dependencies but no joy there either I'm afraid. We also tried including the transit-pgObject I/O code you pointed us towards the other week, @taylor.jeremydavid. We can generate transit data more or less fine, excepting date values. However, when we try to insert them, we get the exceptions below, regardless of whether we're inserting single records or trying to batch them. I notice Sean may have found something similar, at least in terms of there being a type error though upon further inspection the issue seems to be coming from a different spot since ours is a null-error. We might be getting some confusing performance characteristics from experimenting with it because our performance seems to be decreasing as a result of changing to transit serialisation from naively building strings, at least on reads. Was hoping someone might have an idea what we might be doing wrong there. Will look into this a bit further and see if we have anything more helpful about what's going on with the performance.
Repl-Ex:
#error {:cause "ERROR: unexpected error on tx submit (report as a bug)"
:via
[{:type org.postgresql.util.PSQLException
:message "ERROR: unexpected error on tx submit (report as a bug)"
:at [org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse "QueryExecutorImpl.java" 2733]}]
:trace
[[org.postgresql.core.v3.QueryExecutorImpl receiveErrorResponse "QueryExecutorImpl.java" 2733]
[org.postgresql.core.v3.QueryExecutorImpl processResults "QueryExecutorImpl.java" 2420]
[org.postgresql.core.v3.QueryExecutorImpl execute "QueryExecutorImpl.java" 372]
[org.postgresql.jdbc.PgStatement executeInternal "PgStatement.java" 517]
[org.postgresql.jdbc.PgStatement execute "PgStatement.java" 434]
[org.postgresql.jdbc.PgPreparedStatement executeWithFlags "PgPreparedStatement.java" 194]
[org.postgresql.jdbc.PgPreparedStatement execute "PgPreparedStatement.java" 180]
[next.jdbc.result_set$stmt__GT_result_set invokeStatic "result_set.clj" 667]
[next.jdbc.result_set$stmt__GT_result_set invoke "result_set.clj" 662]
[next.jdbc.result_set$eval2258$fn__2266 invoke "result_set.clj" 895]
[next.jdbc.protocols$eval1330$fn__1346$G__1321__1355 invoke "protocols.clj" 34]
[next.jdbc$execute_BANG_ invokeStatic "jdbc.clj" 257]
[next.jdbc$execute_BANG_ invoke "jdbc.clj" 241]
[xtdb_demo.core$jdbc_insert_records invokeStatic "core.clj" 312]
[xtdb_demo.core$jdbc_insert_records invoke "core.clj" 311]
[xtdb_demo.core$eval50825 invokeStatic "form-init17406523132236897824.clj" 317]
[xtdb_demo.core$eval50825 invoke "form-init17406523132236897824.clj" 316]
[clojure.lang.Compiler eval "Compiler.java" 7700]
[clojure.lang.Compiler eval "Compiler.java" 7655]
[clojure.core$eval invokeStatic "core.clj" 3232]
[clojure.core$eval invoke "core.clj" 3228]
[nrepl.middleware.interruptible_eval$evaluate$fn__41258$fn__41259 invoke "interruptible_eval.clj" 87]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
[clojure.lang.RestFn invoke "RestFn.java" 428]
[nrepl.middleware.interruptible_eval$evaluate$fn__41258 invoke "interruptible_eval.clj" 87]
[clojure.main$repl$read_eval_print__9244$fn__9247 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9244 invoke "main.clj" 437]
[clojure.main$repl$fn__9253 invoke "main.clj" 459]
[clojure.main$repl invokeStatic "main.clj" 459]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1526]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__41291$fn__41295 invoke "interruptible_eval.clj" 152]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__41361$fn__41365 invoke "session.clj" 218]
[nrepl.middleware.session$session_exec$main_loop__41361 invoke "session.clj" 217]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 1570]]}12:26:39 | ERROR xtdb.indexer | error in indexer, indexing #xt/tx-key {:tx-id 0, :system-time #time/instant "2024-11-25T12:26:39.541498Z"}
Cannot invoke "Object.getClass()" because "x" is null
at clojure.lang.Numbers.ops(Numbers.java:1099)
at clojure.lang.Numbers.dec(Numbers.java:147)
at xtdb.sql.plan.TableRowsVisitor$reify__21975.visitPostgresParameter(plan.clj:2056)
at xtdb.antlr.Sql$PostgresParameterContext.accept(Sql.java:8522)
at xtdb.sql.plan.TableRowsVisitor$reify__21975.visitParameterRecord(plan.clj:2053)
at xtdb.antlr.Sql$ParameterRecordContext.accept(Sql.java:11476)
at xtdb.sql.plan$accept_visitor.invokeStatic(plan.clj:1978)
at xtdb.sql.plan$accept_visitor.invoke(plan.clj:1977)
at clojure.core$partial$fn__5927.invoke(core.clj:2641)
at clojure.core$map$fn__5950$fn__5951.invoke(core.clj:2759)
at clojure.core.protocols$iterator_reduce_BANG_.invokeStatic(protocols.clj:42)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:52)
at clojure.core.protocols$fn__8256.invokeStatic(protocols.clj:74)
at clojure.core.protocols$fn__8256.invoke(protocols.clj:74)
at clojure.core.protocols$fn__8203$G__8198__8216.invoke(protocols.clj:13)
at clojure.core$transduce.invokeStatic(core.clj:7026)
at clojure.core$into.invokeStatic(core.clj:7042)
at clojure.core$into.invoke(core.clj:7029)
at xtdb.sql.plan.TableRowsVisitor.visitRecordsValueList(plan.clj:2051)
at xtdb.antlr.Sql$RecordsValueListContext.accept(Sql.java:11540)
at xtdb.sql.plan.QueryPlanVisitor.visitRecordsValueList(plan.clj:2315)
at xtdb.antlr.Sql$RecordsValueListContext.accept(Sql.java:11540)
at xtdb.sql.plan.QueryPlanVisitor.visitRecordsValueConstructor(plan.clj:2311)
at xtdb.antlr.Sql$RecordsValueConstructorContext.accept(Sql.java:11597)
at xtdb.sql.plan.QueryPlanVisitor.visitInsertRecords(plan.clj:2327)
at xtdb.antlr.Sql$InsertRecordsContext.accept(Sql.java:18433)
at xtdb.sql.plan.StmtVisitor.visitInsertStatement(plan.clj:2520)
at xtdb.antlr.Sql$InsertStatementContext.accept(Sql.java:18365)
at xtdb.sql.plan.StmtVisitor.visitInsertStmt(plan.clj:2516)
at xtdb.antlr.Sql$InsertStmtContext.accept(Sql.java:519)
at xtdb.sql.plan$eval22632$fn__22634.invoke(plan.clj:2800)
at xtdb.sql.plan$eval22608$fn__22609$G__22599__22616.invoke(plan.clj:2777)
at xtdb.sql.plan$eval22628$fn__22629.invoke(plan.clj:2784)
at xtdb.sql.plan$eval22608$fn__22609$G__22599__22616.invoke(plan.clj:2777)
at xtdb.sql.plan$plan_statement.invokeStatic(plan.clj:2818)
at xtdb.sql.plan$plan_statement.invoke(plan.clj:2816)
at xtdb.sql$compile_query__26502.invoke(sql.clj:10)
at xtdb.util$lru_memoize$fn__4286.invoke(util.clj:555)
at xtdb.indexer$$reify__27751$fn__27752.invoke(indexer.clj:471)
at xtdb.indexer$$reify__27751.indexOp(indexer.clj:469)
at xtdb.indexer.Indexer$fn__27816$index_tx_ops__27821$fn__27834.invoke(indexer.clj:647)
at clojure.lang.AFn.call(AFn.java:18)
at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:129)
at xtdb.indexer.Indexer$fn__27816$index_tx_ops__27821.invoke(indexer.clj:644)
at xtdb.indexer.Indexer$fn__27816$fn__27842.invoke(indexer.clj:658)
at xtdb.indexer.Indexer$fn__27816.invoke(indexer.clj:657)
at xtdb.indexer.Indexer.indexTx(indexer.clj:619)
at xtdb.log.watcher$watch_log_BANG_$reify__29180.accept(watcher.clj:55)
at xtdb.log$tx_handler$fn__22745.invoke(log.clj:40)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:418)
at clojure.core$reduce.invokeStatic(core.clj:6964)
at clojure.core$reduce.invoke(core.clj:6947)
at xtdb.log.NotifyingSubscriberHandler$fn__22778.invoke(log.clj:99)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:1583)
12:26:39 | ERROR xtdb.util | Uncaught exception:
java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "x" is null
at clojure.lang.Numbers.ops(Numbers.java:1099)
at clojure.lang.Numbers.dec(Numbers.java:147)
at xtdb.sql.plan.TableRowsVisitor$reify__21975.visitPostgresParameter(plan.clj:2056)
at xtdb.antlr.Sql$PostgresParameterContext.accept(Sql.java:8522)
at xtdb.sql.plan.TableRowsVisitor$reify__21975.visitParameterRecord(plan.clj:2053)
at xtdb.antlr.Sql$ParameterRecordContext.accept(Sql.java:11476)
at xtdb.sql.plan$accept_visitor.invokeStatic(plan.clj:1978)
at xtdb.sql.plan$accept_visitor.invoke(plan.clj:1977)
at clojure.core$partial$fn__5927.invoke(core.clj:2641)
at clojure.core$map$fn__5950$fn__5951.invoke(core.clj:2759)
at clojure.core.protocols$iterator_reduce_BANG_.invokeStatic(protocols.clj:42)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:52)
at clojure.core.protocols$fn__8256.invokeStatic(protocols.clj:74)
at clojure.core.protocols$fn__8256.invoke(protocols.clj:74)
at clojure.core.protocols$fn__8203$G__8198__8216.invoke(protocols.clj:13)
at clojure.core$transduce.invokeStatic(core.clj:7026)
at clojure.core$into.invokeStatic(core.clj:7042)
at clojure.core$into.invoke(core.clj:7029)
at xtdb.sql.plan.TableRowsVisitor.visitRecordsValueList(plan.clj:2051)
at xtdb.antlr.Sql$RecordsValueListContext.accept(Sql.java:11540)
at xtdb.sql.plan.QueryPlanVisitor.visitRecordsValueList(plan.clj:2315)
at xtdb.antlr.Sql$RecordsValueListContext.accept(Sql.java:11540)
at xtdb.sql.plan.QueryPlanVisitor.visitRecordsValueConstructor(plan.clj:2311)
at xtdb.antlr.Sql$RecordsValueConstructorContext.accept(Sql.java:11597)
at xtdb.sql.plan.QueryPlanVisitor.visitInsertRecords(plan.clj:2327)
at xtdb.antlr.Sql$InsertRecordsContext.accept(Sql.java:18433)
at xtdb.sql.plan.StmtVisitor.visitInsertStatement(plan.clj:2520)
at xtdb.antlr.Sql$InsertStatementContext.accept(Sql.java:18365)
at xtdb.sql.plan.StmtVisitor.visitInsertStmt(plan.clj:2516)
at xtdb.antlr.Sql$InsertStmtContext.accept(Sql.java:519)
at xtdb.sql.plan$eval22632$fn__22634.invoke(plan.clj:2800)
at xtdb.sql.plan$eval22608$fn__22609$G__22599__22616.invoke(plan.clj:2777)
at xtdb.sql.plan$eval22628$fn__22629.invoke(plan.clj:2784)
at xtdb.sql.plan$eval22608$fn__22609$G__22599__22616.invoke(plan.clj:2777)
at xtdb.sql.plan$plan_statement.invokeStatic(plan.clj:2818)
at xtdb.sql.plan$plan_statement.invoke(plan.clj:2816)
at xtdb.sql$compile_query__26502.invoke(sql.clj:10)
at xtdb.util$lru_memoize$fn__4286.invoke(util.clj:555)
at xtdb.indexer$$reify__27751$fn__27752.invoke(indexer.clj:471)
at xtdb.indexer$$reify__27751.indexOp(indexer.clj:469)
at xtdb.indexer.Indexer$fn__27816$index_tx_ops__27821$fn__27834.invoke(indexer.clj:647)
at clojure.lang.AFn.call(AFn.java:18)
at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:129)
at xtdb.indexer.Indexer$fn__27816$index_tx_ops__27821.invoke(indexer.clj:644)
at xtdb.indexer.Indexer$fn__27816$fn__27842.invoke(indexer.clj:658)
at xtdb.indexer.Indexer$fn__27816.invoke(indexer.clj:657)
at xtdb.indexer.Indexer.indexTx(indexer.clj:619)
at xtdb.log.watcher$watch_log_BANG_$reify__29180.accept(watcher.clj:55)
at xtdb.log$tx_handler$fn__22745.invoke(log.clj:40)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:418)
at clojure.core$reduce.invokeStatic(core.clj:6964)
at clojure.core$reduce.invoke(core.clj:6947)
at xtdb.log.NotifyingSubscriberHandler$fn__22778.invoke(log.clj:99)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:1583)Hey @edward.hughes1911 thanks for the updates and errors - are you able to share the string and parameters which gave you the above?
> because our performance seems to be decreasing as a result of changing to transit serialisation from naively building strings, at least on reads this would be quite unexpected if true - assuming the outputs are correct it would suggest something is not quite right with the implementation, we'll take a look 🤔
> assuming the outputs are correct it would suggest something is not quite right with the implementation, Yeah, we haven't yet ruled out the possibility of some hidden variable we overlooked or it being down to how our tests are set up, but the fact we're getting slowdowns at all from that one change is quite odd
This is how we're building the initial string w/params and then dumping our records into
(defn ->serialised-records [records]
(map transit->pgobject records))
(defn jdbc-insert-txn [table records]
(let [total (count records)
into-table (partial str "INSERT INTO " table " RECORDS ")
args (interpose ", " (repeat total "?"))
pg-objects (->serialised-records records)]
(into [(apply into-table args)] pg-objects)))
(defn jdbc-insert-records [conn table records]
(jdbc/execute! conn (jdbc-insert-txn table records) {:builder-fn builder-fn}))and the records we're inserting, also tried selecting just one out of the array and passing that in by itself to the same result:
[{:_id "payee::1:1"
:application {
:personal-details {:first-name "Alice"
:middle-name "P."
:last-name "Rogamer"
:date-of-birth "1968-08-12"
:address {:line-1 "22 Azing Loop"
:city "Lambda Town"}
:ni "AB325468"}
:account-details {:account-name "Alice P. Rogamer"
:account-number "12893476"
:sort-code "00-00-00"}
:payroll-details {:company-id "1"
:payroll-id "1"
:pay-date (str (tick/date))
:annual-gross-pay 50000}
:banking-details {:beneficiary-id "1"
:ledger-id "ledger-1"
:spend-ledger-id "spend-ledger-1"
:limit-ledger-id "limit-ledger-1"}}}
{:_id "payee::1:2"
:application {
:personal-details {:first-name "John"
:middle-name "Q."
:last-name "Helldiver"
:date-of-birth "1988-09-17"
:address {:line-1 "22 Freedom Road"
:city "Super Town"}
:ni "FRE734675"}
:account-details {:account-name "John Q. Helldiver"
:account-number "09172567"
:sort-code "22-22-22"}
:payroll-details {:company-id "1"
:payroll-id "2"
:pay-date (str (tick/date))
:annual-gross-pay 10000}
:banking-details {:beneficiary-id "2"
:ledger-id "ledger-2"
:spend-ledger-id "spend-ledger-2"
:limit-ledger-id "limit-ledger-2"}}}]thanks for the additional details (and DM 🙂) - I think it's best to assume we can make this faster in the near future vs. the strings - but is this performance as-is prohibitive for your use case? i.e. are you blocked? - I just opened an issue to track in any case https://github.com/xtdb/xtdb/issues/3903
we've now fixed the initial parameter issue on main https://github.com/xtdb/xtdb/issues/3907
Hi, could you please explain a bit more about your setup? Particularly: Which version are you talking about? How are you running the remote node (what command etc)? What code are you using to connect to the node? What exact error are you getting? Hopefully we can help you out that way :)
We're running xt.v2, we're trying to run both a local in process node using (xt.api/start-node) and connect to the docker container using (xt.client/start-client "
the exceptions are all some variation of a closed socket/connection exception
So you want two databases? One in-process and one running in a docker container?
We want to play with both the in-process node and a remote docker container for demo purposes
Ok 👍
Thanks for the catch, I was able to replicate this locally.
Starting a docker container and then trying to connect via the clojure http client hangs for me even though JDBC and even curl work just fine.
I'll raise an issue for this and post it here 🙂
Hey @edward.hughes1911 conscious you're probably more keen to use the first-class Clojure APIs for a number of reasons, but just FYI that the JDBC route is ultimately intended to supersede those APIs, though I appreciate the documentation needs work to showcase what can now be done. For example, there's a "transit" OID type which allows for very simple inserts (minus the boilerplate): https://github.com/xtdb/xtdb/issues/3844#issuecomment-2459844456
So does this mean that Datalog-based queries won't be supported in the stable v2 release? That would be a shame if that were the case. Really the issue right now is we just want to be able to demo both apis side by side, run some performance benchmarks and demonstrate some basic datalog/xtql queries. Is there a previous release you can point me to with a working client?
> So does this mean that Datalog-based queries won't be supported in the stable v2 release? we've done some experimentation recently to make XTQL embeddable within the SQL string that can be passed via JDBC, but it's not been merged yet: https://github.com/jarohen/xtdb/commit/27f298a7faf0d0374ba66d4a948ac7cee42dc8f6#diff-a7253f815cd6bbfdd884cfa26fd879b1c8beec83a84f0aa8737a25c43da4f22dR2433-R2436 For now you would need to use the Clojure HTTP client - not sure why it's not working currently though, we'll take a look at the issue raised soon. Hopefully you can use the in-process API in the meantime without being too blocked
Oh, to be clear, the in process node gives us similar connection issues to trying to connect to the remote server
Specifically Address already in use, even though we aren't passing a port explicitly
It doesn't pick a random port by default, possibly that's an oversight 🤔
Ok, looks like it was defaulting to 5432, and since we were running the docker container at the same tame, the remote and in process nodes were fighting over port names.
For future reference, what's the config key for passing an explicit port to an in-process node?
You can pass a map {:server {:port xyz}}
thanks
So we can get the in-process node running, but attempting to submit/execute a transaction gives us
; Execution error (ClassNotFoundException) at io.netty.buffer.PooledByteBuf/<init> (PooledByteBuf.java:47).
; io.netty.util.Recycler$EnhancedHandlethat looks weird, which JVM/Clojure are you running?
Is there an example of how to use unnest in SQL? For example, how would I query for ‘work’ contacts in the following?
INSERT INTO people
RECORDS {_id: 6,
name: 'fred',
info: {contact: [{loc: 'home',
tel: '123'},
{loc: 'work',
tel: '456',
registered: DATE '2024-01-01'}]}}SELECT *
FROM people, unnest ( (info).contact ) as con
where (con).loc = 'work'hey @esp1 we don't support an implicit column alias for unnest - this means the con alias you're using there is actually a table alias, so instead you want as con_table (con) e.g. https://play.xtdb.com/?version=2.0.0-b1&type=sql&txs=W3sidHhzIjoiSU5TRVJUIElOVE8gcGVvcGxlXG5SRUNPUkRTIHtfaWQ6IDYsXG4gICAgICAgICBuYW1lOiAnZnJlZCcsXG4gICAgICAgICBpbmZvOiB7Y29udGFjdDogW3tsb2M6ICdob21lJyxcbiAgICAgICAgICAgICAgICAgICAgICAgICAgIHRlbDogJzEyMyd9LFxuICAgICAgICAgICAgICAgICAgICAgICAgICB7bG9jOiAnd29yaycsXG4gICAgICAgICAgICAgICAgICAgICAgICAgICB0ZWw6ICc0NTYnLFxuICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVnaXN0ZXJlZDogREFURSAnMjAyNC0wMS0wMSd9XX19Iiwic3lzdGVtLXRpbWUiOiIyMDI0LTAxLTAzVDAwOjAwOjAwLjAwMFoifV0%3D&query=U0VMRUNUICoKRlJPTSBwZW9wbGUsIHVubmVzdCAoIChpbmZvKS5jb250YWN0ICkgYXMgY29uX3RhYmxlIChjb24pCndoZXJlIChjb24pLmxvYyA9ICd3b3JrJw%3D%3D
I just opened an issue https://github.com/xtdb/xtdb/issues/3848
out of interest, were you expecting this to work because of familiarity with Postgres, or some other SQL database?
I was just trying to convert some of my XTQL queries into SQL and wasn’t sure how to use unnest. Thanks for the explanation on this. I'm still not sure how to address my original question at the top of this post tho - querying for a particular value(s) within unnested fields. How would you do that?
Maybe I'm confused...do you mean something different to the play example I shared before?
oh i totally missed the play link! sorry yes that does answer my question - thank you!
cool, glad to know 🙂
Maybe this is my way of saying please don't get rid of/forget about XTQL 🙂noted, yep - always good to hear what people are using, it helps us choose what to spend our (limited!) time on 🙂
hey all - after further consideration (and https://github.com/xtdb/xtdb/issues/3848#issuecomment-2465635093) I'm of the opinion that we should elect to differ from Postgres here, and request that users explicitly name the columns returned from UNNEST.
e.g. in your example above, I'd likely name the nested field contacts to make the distinction clear:
INSERT INTO people
RECORDS {_id: 6,
name: 'fred',
info: {contacts: [{loc: 'home',
tel: '123'},
{loc: 'work',
tel: '456',
registered: DATE '2024-01-01'}]}};
SELECT p.name, cs.contact
FROM people p, UNNEST((info).contacts) cs (contact)
WHERE (cs.contact).loc = 'work';
-- which then yields
name | contact
------+------------------------------------------------------
fred | {"loc":"work","registered":"2024-01-01","tel":"456"}
(1 row)Yeah.. I think this may be a bit better than the previous https://play.xtdb.com/?version=2.0.0-b1&type=sql&txs=W3sidHhzIjoiSU5TRVJUIElOVE8gcGVvcGxlXG5SRUNPUkRTIHtfaWQ6IDYsXG4gICAgICAgICBuYW1lOiAnZnJlZCcsXG4gICAgICAgICBpbmZvOiB7Y29udGFjdDogW3tsb2M6ICdob21lJyxcbiAgICAgICAgICAgICAgICAgICAgICAgICAgIHRlbDogJzEyMyd9LFxuICAgICAgICAgICAgICAgICAgICAgICAgICB7bG9jOiAnd29yaycsXG4gICAgICAgICAgICAgICAgICAgICAgICAgICB0ZWw6ICc0NTYnLFxuICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVnaXN0ZXJlZDogREFURSAnMjAyNC0wMS0wMSd9XX19Iiwic3lzdGVtLXRpbWUiOiIyMDI0LTAxLTAzVDAwOjAwOjAwLjAwMFoifV0%3D&query=U0VMRUNUICoKRlJPTSBwZW9wbGUsIHVubmVzdCAoIChpbmZvKS5jb250YWN0ICkgYXMgY29uX3RhYmxlIChjb24pCndoZXJlIChjb24pLmxvYyA9ICd3b3JrJw%3D%3D @taylor.jeremydavid gave, but honestly both seem pretty unintuitive to me. The parens around path elements like (info).contacts also feels unwieldy. The XTQL unnest feels much more intuitive to me, and I also like that unify allows me to create queries that don't run afoul of the paren/path shenanigans I need to deal with in SQL.
Thank you for the feedback, it's true that SQL definitely isn't the most elegant thing in the world. I'm not sure how much more we could flex it to make it feel more ergonomic here but we'll keep it in mind 🙏
Maybe this is my way of saying please don't get rid of/forget about XTQL 🙂 I have actually been able to use XTQL for most everything I need to. The main issue I've hit that's causing me to reach for SQL is https://github.com/xtdb/xtdb/issues/3826, but the contortions I'm needing to go through to do this in SQL don't feel very nice (in addition to the things mentioned above, I also need to translate field names, e.g. foo.bar/baz -> foo$bar$baz). If I could get that one XTQL bug fixed, and maybe a few standard library functions to extract name/namespace strings from keywords, I'd be a pretty happy camper.
I also noticed that if I remove the where clause, I do get results, but the column alias is not used in the results [https://play.xtdb.com/?version=2.0.0-b1&type=sql&txs=W3sidHhzIjoiSU5TRVJUIElOVE8gcGVvcGxlXG5SRUNPUkRTIHtfaWQ6IDYsXG4gICAgICAgICBuYW1lOiAnZnJlZCcsXG4gICAgICAgICBpbmZvOiB7Y29udGFjdDogW3tsb2M6ICdob21lJyxcbiAgICAgICAgICAgICAgICAgICAgICAgICAgIHRlbDogJzEyMyd9LFxuICAgICAgICAgICAgICAgICAgICAgICAgICB7bG9jOiAnd29yaycsXG4gICAgICAgICAgICAgICAgICAgICAgICAgICB0ZWw6ICc0NTYnLFxuICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVnaXN0ZXJlZDogREFURSAnMjAyNC0wMS0wMSd9XX19Iiwic3lzdGVtLXRpbWUiOiIyMDI0LTAxLTAzVDAwOjAwOjAwLjAwMFoifV0%3D&query=U0VMRUNUICoKRlJPTSBwZW9wbGUsIHVubmVzdCAoIChpbmZvKS5jb250YWN0ICkgYXMgY29u].
Note the result column for the unnested contacts is labeled _column_1 , not con as specified in the SQL. And if I do not provide a column alias, I get a malformed SQL exception.