Fork me on GitHub
#xtdb
<
2022-08-13
>
Dmitri Akatov09:08:39

Hello, what is the simplest possible configuration to start a node backed by Postgres? My deps.edn looks like this:

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        com.xtdb/xtdb-core {:mvn/version "1.21.0"}
        com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
        org.postgresql/postgresql {:mvn/version "42.4.1"}}}
and I try to start the node with
(ns user
  (:require
   [xtdb.api :as xt]
   [xtdb.jdbc]
   [xtdb.jdbc.psql]))

(def node (atom nil))

(def config {:xtdb.jdbc/connection-pool
             {:dialect   {:xtdb/module xtdb.jdbc.psql/->dialect}
              :pool-opts {:maximumPoolSize 100}
              :db-spec   {:jdbcUrl ""}}

             :xtdb/tx-log         {:xtdb/module     xtdb.jdbc/->tx-log
                                   :connection-pool :xtdb.jdbc/connection-pool}

             :xtdb/document-store {:xtdb/module     xtdb.jdbc/->document-store
                                   :connection-pool :xtdb.jdbc/connection-pool}})

(defn start []
  (reset! node (xt/start-node config)))
When calling (start) , I get the following error:
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:300).
No suitable driver
The https://docs.xtdb.com/storage/1.21.0/jdbc/ is somewhat cryptic on this. What am I missing?

refset10:08:18

Hey I think you just need to add the driver as a dep, as per how the page says: "Each of these also require an additional dependency to pull in the relevant JDBC driver - see the XTDB JDBC project.clj for our latest dependencies."

refset10:08:43

There's a hyperlink on that project.clj text

Dmitri Akatov10:08:41

I tried putting most of those dependencies into the deps.edn as well with the same result

Dmitri Akatov10:08:01

Do I need to require the relevant namespaces as well?

refset10:08:08

Shouldn't need to require it, hmm

Dmitri Akatov10:08:02

Can there be a difference between the way the dependencies are used in a leiningen project?

Dmitri Akatov11:08:16

OK, I updated my deps.edn to:

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        org.clojure/tools.logging {:mvn/version "1.2.4"}
        com.xtdb/xtdb-core {:mvn/version "1.21.0"}
        com.xtdb/xtdb-jdbc {:mvn/version "1.21.0"}
        pro.juxt.clojars-mirrors.com.github.seancorfield/next.jdbc {:mvn/version  "1.2.674"}
        org.clojure/java.data {:mvn/version "1.0.95"}
        com.zaxxer/HikariCP {:mvn/version "3.4.5"}
        pro.juxt.clojars-mirrors.com.taoensso/nippy {:mvn/version "3.1.1-2"}
        
        org.postgresql/postgresql {:mvn/version "42.2.18"}
        com.oracle.ojdbc/ojdbc8 {:mvn/version  "19.3.0.0"}
        com.h2database/h2 {:mvn/version "1.4.200"}
        org.xerial/sqlite-jdbc {:mvn/version "3.28.0"}
        mysql/mysql-connector-java {:mvn/version "8.0.23"}
        com.microsoft.sqlserver/mssql-jdbc {:mvn/version "8.2.2.jre8"}}}
which pretty much matches https://github.com/xtdb/xtdb/blob/master/modules/jdbc/project.clj#L14-L28 Still the same error… (`No suitable driver`)

Dmitri Akatov11:08:36

I replaced my deps.edn with a project.clj that looks like this:

(defproject myproject "0.1.0-SNAPSHOT"

  ;; same dependencies as  + com.xtdb/xtdb-jdbc + explicit versions

  :dependencies [[org.clojure/clojure "1.11.1"]
                 [org.clojure/tools.logging "1.2.4"]
                 [com.xtdb/xtdb-core "1.21.0"]
                 [com.xtdb/xtdb-jdbc "1.21.0"]
                 [pro.juxt.clojars-mirrors.com.github.seancorfield/next.jdbc "1.2.674"]
                 [org.clojure/java.data "1.0.95"]
                 [com.zaxxer/HikariCP "3.4.5"]
                 [pro.juxt.clojars-mirrors.com.taoensso/nippy "3.1.1-2"]

                 ;; Sample driver dependencies
                 [org.postgresql/postgresql "42.2.18"]
                 [com.oracle.ojdbc/ojdbc8 "19.3.0.0"]
                 [com.h2database/h2 "1.4.200"]
                 [org.xerial/sqlite-jdbc "3.28.0"]
                 [mysql/mysql-connector-java "8.0.23"]
                 [com.microsoft.sqlserver/mssql-jdbc "8.2.2.jre8"]])
still same error when running this through leiningen:
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:300).
No suitable driver

refset12:08:58

Which JVM? There are various working examples of good Postgres config floating around, e.g. https://github.com/xtdb/xtdb/blob/ffb5d00efde9e147f9daaef39973f5310c339aab/bench/src/xtdb/bench.clj#L291 and Biff

❤️ 1
Dmitri Akatov12:08:13

actually, I managed to track down the culprit

Dmitri Akatov12:08:39

(import java.sql.DriverManager)
(DriverManager/getDriver "")
Execution error (SQLException) at java.sql.DriverManager/getDriver (DriverManager.java:300).
No suitable driver

Dmitri Akatov12:08:49

the jdbcUrl was wrong

Dmitri Akatov12:08:03

(DriverManager/getDriver "jdbc:")
#object[org.postgresql.Driver 0x1b80a6d8 "org.postgresql.Driver@1b80a6d8"]

refset12:08:18

Ah, frustrating :) glad to hear though

Dmitri Akatov12:08:34

there are some additional errors now, e.g. password field missing

Dmitri Akatov12:08:50

I guess the db-spec config will need some additional fields

Dmitri Akatov12:08:41

ok, looks like it needs both :user and :password in the :db-spec config

Dmitri Akatov12:08:52

if :user is missing, it seems to use my unix user name by default…

Dmitri Akatov12:08:32

I’ll figure out the smallest set of dependencies one needs to get this running and will post this here later for reference

🙏 1
Dmitri Akatov12:08:46

@U899JBRPF thank you for looking into this for me

🙂 1
1
richiardiandrea19:08:21

Hi there, I have this use case where I need to "invalidate" an entity, containing all the same attributes, by setting its end valid time. I tried to send a new put transaction with its doc from the history but it did not seem to work, before digging into an example I was wondering if it's a supported use case...

Jacob O'Bryant22:08:04

I think you would do a delete operation, and have the delete operation's valid time be the time at which you'd like the document to become invalidated.

richiardiandrea23:08:54

Oh I see, that is something I will try it out. I do want to see the entity at a previous version, would I be able to see that? I basically would like one or more recent transactions against that entity

Jacob O'Bryant18:08:54

You can use xtdb.api/entity-history to get the previous versions of a document, or pass in a previous valid time/tx time to xtdb.api/db and then run queries etc as normal

richiardiandrea19:08:25

So you are basically saying I need to do that filtering in user code? I was trying to get some help from xtdb's built in functions...I might need to dig more into how end valid time works

Jacob O'Bryant19:08:23

it's possible I'm misunderstanding what you're trying to do. could you share a hypothetical example, E.g. a query you'd like to be able to run?

richiardiandrea19:08:52

Yes sure, say I have an entity with attr x = 5 valid as of 2021. At some point something happens on the first of Jan 2022 and my attr turns to x = 10 (valid for the whole 2022). However in March 2022 something else happens. Now the attr for 2022 is not valid anymore and I want to see attr x = 5 every time I query:grin:😇

Jacob O'Bryant20:08:48

OK, in that case you could do a put operation for the doc with x = 5 and with a start valid time of March 2022

Jacob O'Bryant20:08:38

to be complete, three put operations, with start valid times of jan 2021, Jan 2022, and March 2022 respectively.

Jacob O'Bryant20:08:26

and if (as of jan 2022) you wanted x = 10 to only be valid through 2022, the second put would have 2023 as the end valid time.

richiardiandrea20:08:58

I see thank you yeah, I was trying to set valid time "after the fact" for x = 10 but what you are saying also makes sense ... I guess if we could do that we could avoid a query (for the past value)

Jacob O'Bryant00:08:08

Ah I see what you mean. yeah, no way to avoid having to look up the old value.

richiardiandrea01:08:48

OK thanks for confirming and helping Jacob

👌 1