xtdb 2023-07-19 | Slack Archive

hairfire21:07:40

I'm trying to work through the XTDB 1.x SQL documentation on this page https://docs.xtdb.com/language-reference/sql-queries/ and I'm encountering a problem. The following short program, with embedded comments describing the problem, illustrates the problem:

(ns phw.xtdb-sql
  (:require [ :as io]
            [clojure.java.shell :refer [sh]]
            [xtdb.api :as xt]
            [xtdb.calcite]))

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir (io/file dir)
                        :sync? true}})]
    (xt/start-node
     {:xtdb/tx-log (kv-store "data/dev/tx-log")
      :xtdb/document-store (kv-store "data/dev/doc-store")
      :xtdb/index-store (kv-store "data/dev/index-store")})))

(def xtdb-node (start-xtdb!))

(defn stop-xtdb! []
  (.close xtdb-node))

(defn query [node q]
  (with-open [conn (xtdb.calcite/jdbc-connection node)]
    (let [stmt (.createStatement conn)]
      (->> q (.executeQuery stmt) resultset-seq))))

;; The following invocation of the 'query' function fails with an exception, and it looks like
;; the exception is coming from the invocation of the 'xtdb.calcite/jdbc-connection' function
;;
;; (query xtdb-node "SELECT PERSON.NAME FROM PERSON")
;;
;; As a test the 'def' form below attempts to invoke 'xtdb.calcite/jdbc-connection'

(def conn (xtdb.calcite/jdbc-connection xtdb-node))

;; when this file is loaded into a REPL this exception output is generated
;;
; Evaluating file: xtdb_sql.clj
; Execution error (NullPointerException) at com.google.common.collect.CollectPreconditions/checkEntryNotNull (CollectPreconditions.java:33).
; null value in entry: XTDB_NODE=null
; Evaluation of file xtdb_sql.clj failed: class clojure.lang.Compiler$CompilerException

;; note: this is the deps.edn file content
;;
;; {:paths ["src" "resources"]
;;  :deps {org.clojure/clojure {:mvn/version "1.11.1"}
;;         com.xtdb/xtdb-core {:mvn/version "1.24.0"}
;;         com.xtdb/xtdb-rocksdb {:mvn/version "1.24.0"}
;;         com.xtdb/xtdb-sql {:mvn/version "1.24.0"}}}

(comment

  (stop-xtdb!)

  (sh "rm" "-rf" "data")

  ;
  )

Note: The Java version is 'openjdk version "17.0.7" 2023-04-18' Thanks in advance for any helpful pointers 🤓

refset22:07:37

Hey @U44P1CP5K I don't recognise this error. Do you know what is bringing in the com.google.common.collect.CollectPreconditions package? Can you share the classpath? Is there a full stack trace available?

Vincent22:07:59

so there's a concept of db and a concept of node in xtdb

Vincent22:07:31

when you need a node you might need to say stuff like (xt/db user/!xtdb)

hairfire13:07:51

Here's the class path from the Calva REPL: src resources /home/pauwhi/.m2/repository/cider/cider-nrepl/0.28.5/cider-nrepl-0.28.5.jar /home/pauwhi/.m2/repository/com/xtdb/xtdb-core/1.24.0/xtdb-core-1.24.0.jar /home/pauwhi/.m2/repository/com/xtdb/xtdb-rocksdb/1.24.0/xtdb-rocksdb-1.24.0.jar /home/pauwhi/.m2/repository/com/xtdb/xtdb-sql/1.24.0/xtdb-sql-1.24.0.jar /home/pauwhi/.m2/repository/nrepl/nrepl/1.0.0/nrepl-1.0.0.jar /home/pauwhi/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.jar /home/pauwhi/.m2/repository/org/agrona/agrona/1.16.0/agrona-1.16.0.jar /home/pauwhi/.m2/repository/org/clojure/data.json/2.4.0/data.json-2.4.0.jar /home/pauwhi/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar /home/pauwhi/.m2/repository/org/clojure/tools.cli/1.0.206/tools.cli-1.0.206.jar /home/pauwhi/.m2/repository/org/clojure/tools.logging/1.2.4/tools.logging-1.2.4.jar /home/pauwhi/.m2/repository/org/clojure/tools.reader/1.3.6/tools.reader-1.3.6.jar /home/pauwhi/.m2/repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar /home/pauwhi/.m2/repository/pro/juxt/clojars-mirrors/com/stuartsierra/dependency/1.0.0/dependency-1.0.0.jar /home/pauwhi/.m2/repository/pro/juxt/clojars-mirrors/com/taoensso/nippy/3.1.1-2/nippy-3.1.1-2.jar /home/pauwhi/.m2/repository/pro/juxt/clojars-mirrors/edn-query-language/eql/2021.02.28/eql-2021.02.28.jar /home/pauwhi/.m2/repository/org/rocksdb/rocksdbjni/7.7.3/rocksdbjni-7.7.3.jar /home/pauwhi/.m2/repository/org/apache/calcite/calcite-core/1.22.0/calcite-core-1.22.0.jar /home/pauwhi/.m2/repository/org/apache/calcite/avatica/avatica-server/1.16.0/avatica-server-1.16.0.jar /home/pauwhi/.m2/repository/pro/juxt/clojars-mirrors/cheshire/cheshire/5.10.0/cheshire-5.10.0.jar /home/pauwhi/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar /home/pauwhi/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar /home/pauwhi/.m2/repository/org/lz4/lz4-java/1.7.1/lz4-java-1.7.1.jar /home/pauwhi/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar /home/pauwhi/.m2/repository/com/esri/geometry/esri-geometry-api/2.2.0/esri-geometry-api-2.2.0.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.10.0/jackson-annotations-2.10.0.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.0/jackson-databind-2.10.0.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.10.0/jackson-dataformat-yaml-2.10.0.jar /home/pauwhi/.m2/repository/com/google/guava/guava/19.0/guava-19.0.jar /home/pauwhi/.m2/repository/com/jayway/jsonpath/json-path/2.4.0/json-path-2.4.0.jar /home/pauwhi/.m2/repository/com/yahoo/datasketches/sketches-core/0.9.0/sketches-core-0.9.0.jar /home/pauwhi/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar /home/pauwhi/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar /home/pauwhi/.m2/repository/net/hydromatic/aggdesigner-algorithm/6.0/aggdesigner-algorithm-6.0.jar /home/pauwhi/.m2/repository/org/apache/calcite/calcite-linq4j/1.22.0/calcite-linq4j-1.22.0.jar /home/pauwhi/.m2/repository/org/apache/calcite/avatica/avatica-core/1.16.0/avatica-core-1.16.0.jar /home/pauwhi/.m2/repository/org/apache/commons/commons-dbcp2/2.6.0/commons-dbcp2-2.6.0.jar /home/pauwhi/.m2/repository/org/apache/commons/commons-lang3/3.8/commons-lang3-3.8.jar /home/pauwhi/.m2/repository/org/apiguardian/apiguardian-api/1.1.0/apiguardian-api-1.1.0.jar /home/pauwhi/.m2/repository/org/codehaus/janino/commons-compiler/3.0.11/commons-compiler-3.0.11.jar /home/pauwhi/.m2/repository/org/codehaus/janino/janino/3.0.11/janino-3.0.11.jar /home/pauwhi/.m2/repository/javax/servlet/javax.servlet-api/4.0.1/javax.servlet-api-4.0.1.jar /home/pauwhi/.m2/repository/org/apache/calcite/avatica/avatica-metrics/1.16.0/avatica-metrics-1.16.0.jar /home/pauwhi/.m2/repository/org/eclipse/jetty/jetty-http/9.4.15.v20190215/jetty-http-9.4.15.v20190215.jar /home/pauwhi/.m2/repository/org/eclipse/jetty/jetty-security/9.4.15.v20190215/jetty-security-9.4.15.v20190215.jar /home/pauwhi/.m2/repository/org/eclipse/jetty/jetty-server/9.4.15.v20190215/jetty-server-9.4.15.v20190215.jar /home/pauwhi/.m2/repository/org/eclipse/jetty/jetty-util/9.4.15.v20190215/jetty-util-9.4.15.v20190215.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.10.2/jackson-core-2.10.2.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.10.2/jackson-dataformat-cbor-2.10.2.jar /home/pauwhi/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.10.2/jackson-dataformat-smile-2.10.2.jar /home/pauwhi/.m2/repository/org/yaml/snakeyaml/1.24/snakeyaml-1.24.jar /home/pauwhi/.m2/repository/net/minidev/json-smart/2.3/json-smart-2.3.jar /home/pauwhi/.m2/repository/com/yahoo/datasketches/memory/0.9.0/memory-0.9.0.jar /home/pauwhi/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar /home/pauwhi/.m2/repository/com/google/protobuf/protobuf-java/3.6.1/protobuf-java-3.6.1.jar /home/pauwhi/.m2/repository/org/apache/httpcomponents/httpclient/4.5.9/httpclient-4.5.9.jar /home/pauwhi/.m2/repository/org/apache/httpcomponents/httpcore/4.4.11/httpcore-4.4.11.jar /home/pauwhi/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar /home/pauwhi/.m2/repository/org/apache/commons/commons-pool2/2.6.1/commons-pool2-2.6.1.jar /home/pauwhi/.m2/repository/org/eclipse/jetty/jetty-io/9.4.15.v20190215/jetty-io-9.4.15.v20190215.jar /home/pauwhi/.m2/repository/net/minidev/accessors-smart/1.2/accessors-smart-1.2.jar /home/pauwhi/.m2/repository/org/ow2/asm/asm/5.0.4/asm-5.0.4.jar

hairfire14:07:28

Here's a stack trace: java.lang.RuntimeException: Error instantiating JsonCustomSchema(name=xtdb) at org.apache.calcite.model.ModelHandler.visit (ModelHandler.java:287) org.apache.calcite.model.JsonCustomSchema.accept (JsonCustomSchema.java:45) org.apache.calcite.model.ModelHandler.visit (ModelHandler.java:208) org.apache.calcite.model.ModelHandler.<init> (ModelHandler.java:100) org.apache.calcite.jdbc.Driver$1.onConnectionInit (Driver.java:98) org.apache.calcite.avatica.UnregisteredDriver.connect (UnregisteredDriver.java:139) java.sql.DriverManager.getConnection (DriverManager.java:681) java.sql.DriverManager.getConnection (DriverManager.java:190) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:513) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:510) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) phw.xtdb_sql$eval23574.invokeStatic (xtdb_sql.clj:36) phw.xtdb_sql$eval23574.invoke (xtdb_sql.clj:35) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.load (Compiler.java:7653) phw.xtdb_sql$eval5885.invokeStatic (NO_SOURCE_FILE:1) phw.xtdb_sql$eval5885.invoke (NO_SOURCE_FILE:1) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.eval (Compiler.java:7149) clojure.core$eval.invokeStatic (core.clj:3215) clojure.core$eval.invoke (core.clj:3211) nrepl.middleware.interruptible_eval$evaluate$fn__1265$fn__1266.invoke (interruptible_eval.clj:87) clojure.lang.AFn.applyToHelper (AFn.java:152) clojure.lang.AFn.applyTo (AFn.java:144) clojure.core$apply.invokeStatic (core.clj:667) clojure.core$with_bindings_STAR_.invokeStatic (core.clj:1990) clojure.core$with_bindings_STAR_.doInvoke (core.clj:1990) clojure.lang.RestFn.invoke (RestFn.java:425) nrepl.middleware.interruptible_eval$evaluate$fn__1265.invoke (interruptible_eval.clj:87) clojure.main$repl$read_eval_print__9206$fn__9209.invoke (main.clj:437) clojure.main$repl$read_eval_print__9206.invoke (main.clj:437) clojure.main$repl$fn__9215.invoke (main.clj:458) clojure.main$repl.invokeStatic (main.clj:458) clojure.main$repl.doInvoke (main.clj:368) clojure.lang.RestFn.invoke (RestFn.java:1523) nrepl.middleware.interruptible_eval$evaluate.invokeStatic (interruptible_eval.clj:84) nrepl.middleware.interruptible_eval$evaluate.invoke (interruptible_eval.clj:56) nrepl.middleware.interruptible_eval$interruptible_eval$fn__1298$fn__1302.invoke (interruptible_eval.clj:152) clojure.lang.AFn.run (AFn.java:22) nrepl.middleware.session$session_exec$main_loop__1368$fn__1372.invoke (session.clj:218) nrepl.middleware.session$session_exec$main_loop__1368.invoke (session.clj:217) clojure.lang.AFn.run (AFn.java:22) java.lang.Thread.run (Thread.java:833) Caused by: java.lang.NullPointerException: null value in entry: XTDB_NODE=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull (CollectPreconditions.java:33) com.google.common.collect.ImmutableMapEntry.<init> (ImmutableMapEntry.java:50) com.google.common.collect.ImmutableMap.entryOf (ImmutableMap.java:122) com.google.common.collect.ImmutableMap$Builder.put (ImmutableMap.java:197) com.google.common.collect.ImmutableMap$Builder.put (ImmutableMap.java:211) com.google.common.collect.ImmutableMap$Builder.putAll (ImmutableMap.java:237) com.google.common.collect.ImmutableMap$Builder.putAll (ImmutableMap.java:221) org.apache.calcite.model.ModelHandler.operandMap (ModelHandler.java:298) org.apache.calcite.model.ModelHandler.visit (ModelHandler.java:283) org.apache.calcite.model.JsonCustomSchema.accept (JsonCustomSchema.java:45) org.apache.calcite.model.ModelHandler.visit (ModelHandler.java:208) org.apache.calcite.model.ModelHandler.<init> (ModelHandler.java:100) org.apache.calcite.jdbc.Driver$1.onConnectionInit (Driver.java:98) org.apache.calcite.avatica.UnregisteredDriver.connect (UnregisteredDriver.java:139) java.sql.DriverManager.getConnection (DriverManager.java:681) java.sql.DriverManager.getConnection (DriverManager.java:190) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:513) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:510) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) phw.xtdb_sql$eval23574.invokeStatic (xtdb_sql.clj:36) phw.xtdb_sql$eval23574.invoke (xtdb_sql.clj:35) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.load (Compiler.java:7653) phw.xtdb_sql$eval5885.invokeStatic (NO_SOURCE_FILE:1) phw.xtdb_sql$eval5885.invoke (NO_SOURCE_FILE:1) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.eval (Compiler.java:7149) clojure.core$eval.invokeStatic (core.clj:3215) clojure.core$eval.invoke (core.clj:3211) nrepl.middleware.interruptible_eval$evaluate$fn__1265$fn__1266.invoke (interruptible_eval.clj:87) clojure.lang.AFn.applyToHelper (AFn.java:152) clojure.lang.AFn.applyTo (AFn.java:144) clojure.core$apply.invokeStatic (core.clj:667) clojure.core$with_bindings_STAR_.invokeStatic (core.clj:1990) clojure.core$with_bindings_STAR_.doInvoke (core.clj:1990) clojure.lang.RestFn.invoke (RestFn.java:425) nrepl.middleware.interruptible_eval$evaluate$fn__1265.invoke (interruptible_eval.clj:87) clojure.main$repl$read_eval_print__9206$fn__9209.invoke (main.clj:437) clojure.main$repl$read_eval_print__9206.invoke (main.clj:437) clojure.main$repl$fn__9215.invoke (main.clj:458) clojure.main$repl.invokeStatic (main.clj:458) clojure.main$repl.doInvoke (main.clj:368) clojure.lang.RestFn.invoke (RestFn.java:1523) nrepl.middleware.interruptible_eval$evaluate.invokeStatic (interruptible_eval.clj:84) nrepl.middleware.interruptible_eval$evaluate.invoke (interruptible_eval.clj:56) nrepl.middleware.interruptible_eval$interruptible_eval$fn__1298$fn__1302.invoke (interruptible_eval.clj:152) clojure.lang.AFn.run (AFn.java:22) nrepl.middleware.session$session_exec$main_loop__1368$fn__1372.invoke (session.clj:218) nrepl.middleware.session$session_exec$main_loop__1368.invoke (session.clj:217) clojure.lang.AFn.run (AFn.java:22) java.lang.Thread.run (Thread.java:833)

hairfire14:07:27

Hey @U055PQH9R4M, when I change the code to use (xt/db xtdb-node) I get this exception stack trace: java.lang.NullPointerException: Cannot invoke "java.util.concurrent.Future.get()" because "fut" is null at clojure.core$deref_future.invokeStatic (core.clj:2317) clojure.core$deref.invokeStatic (core.clj:2337) clojure.core$deref.invoke (core.clj:2323) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:513) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) xtdb.calcite$jdbc_connection.invokeStatic (calcite.clj:510) xtdb.calcite$jdbc_connection.invoke (calcite.clj:509) phw.xtdb_sql$eval25366.invokeStatic (xtdb_sql.clj:41) phw.xtdb_sql$eval25366.invoke (xtdb_sql.clj:40) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.load (Compiler.java:7653) phw.xtdb_sql$eval25349.invokeStatic (NO_SOURCE_FILE:1) phw.xtdb_sql$eval25349.invoke (NO_SOURCE_FILE:1) clojure.lang.Compiler.eval (Compiler.java:7194) clojure.lang.Compiler.eval (Compiler.java:7149) clojure.core$eval.invokeStatic (core.clj:3215) clojure.core$eval.invoke (core.clj:3211) nrepl.middleware.interruptible_eval$evaluate$fn__1265$fn__1266.invoke (interruptible_eval.clj:87) clojure.lang.AFn.applyToHelper (AFn.java:152) clojure.lang.AFn.applyTo (AFn.java:144) clojure.core$apply.invokeStatic (core.clj:667) clojure.core$with_bindings_STAR_.invokeStatic (core.clj:1990) clojure.core$with_bindings_STAR_.doInvoke (core.clj:1990) clojure.lang.RestFn.invoke (RestFn.java:425) nrepl.middleware.interruptible_eval$evaluate$fn__1265.invoke (interruptible_eval.clj:87) clojure.main$repl$read_eval_print__9206$fn__9209.invoke (main.clj:437) clojure.main$repl$read_eval_print__9206.invoke (main.clj:437) clojure.main$repl$fn__9215.invoke (main.clj:458) clojure.main$repl.invokeStatic (main.clj:458) clojure.main$repl.doInvoke (main.clj:368) clojure.lang.RestFn.invoke (RestFn.java:1523) nrepl.middleware.interruptible_eval$evaluate.invokeStatic (interruptible_eval.clj:84) nrepl.middleware.interruptible_eval$evaluate.invoke (interruptible_eval.clj:56) nrepl.middleware.interruptible_eval$interruptible_eval$fn__1298$fn__1302.invoke (interruptible_eval.clj:152) clojure.lang.AFn.run (AFn.java:22) nrepl.middleware.session$session_exec$main_loop__1368$fn__1372.invoke (session.clj:218) nrepl.middleware.session$session_exec$main_loop__1368.invoke (session.clj:217) clojure.lang.AFn.run (AFn.java:22) java.lang.Thread.run (Thread.java:833)

refset11:07:50

Hi again @U44P1CP5K - thanks for sharing the stacktrace - I think you just need to add the server config option to your start-node map and it will work, see https://docs.xtdb.com/extensions/1.24.0/sql/#_sql_server

refset11:07:48

Apologies for the error message not being more obvious & descriptive!

👍 2

hairfire13:07:57

Yes, the code from the gist works on my machine too 🤓

🙌 2

hairfire15:07:09

Maybe I'm just thick, probably true, but I didn't realize that ":xtdb.calcite/server {:port 1501}" needed to be added to the "xt/start-node" map parameter in order to add an SQL API server to the node. The docs, on this page https://docs.xtdb.com/extensions/1.24.0/sql/#_sql_server, say "Add the following to your node configuration", but it just didn't click when I read it 😳 Perhaps, for me anyway, it would help to add more context to the EDN snip. Something like:

(xt/start-node
       {:xtdb.calcite/server {:port 1501} ;; tell XTDB to add a SQL API server to the node
        :xtdb/tx-log (kv-store "data/dev/tx-log")
        :xtdb/document-store (kv-store "data/dev/doc-store")
        :xtdb/index-store (kv-store "data/dev/index-store")})

👍 2

refset15:07:02

The config complexity is definitely something we're mindful of, thanks for the feedback 🙏

👍 2

2023-07-19

Channels