xtdb 2023-05-05 | Slack Archive

cjsauer15:05:43

Hi there, great seeing Juxt at Conj 👋 I'm attempting to play with the new xtdb goodness. I'm wanting to build a custom Docker image configured to use postgres as the transaction log storage. It's instructing me to hardcode the :db-spec bits (username, password, etc) directly into the EDN config file, but is there any way I can set these via env vars on the docker container itself? That would fit in much nicer with what we're already doing with our other containers.

alexdavis16:05:23

I think we normally use https://github.com/juxt/aero to get env vars into the config, here's an excerpt from how we're using https://github.com/juxt/site (and mysql, but will be the same for postgres)

:ig/system
 {:juxt.site.alpha.db/xt-node
  #profile
   {:dev
    {:xtdb.http-server/server {:port 5511}

     :xtdb/tx-log
     {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                 :db-dir "db/txes"}}

     :xtdb/document-store
     {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                 :db-dir "db/docs"}}

     :xtdb/index-store
     {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                 :db-dir "db2/idxs"}}

    :prod
    {:xtdb.jdbc/connection-pool
     {:dialect {:xtdb/module xtdb.jdbc.mysql/->dialect}
      :pool-opts {}
      :db-spec   {:dbtype "mysql"
                  :dbname #env MYSQL_DATABASE
                  :user #env MYSQL_USER
                  :password #env MYSQL_PASSWORD
                  :host #env MYSQL_HOST
                  :port #env MYSQL_PORT}}
.,...

cjsauer17:05:23

I'm starting the node using clojure -m xtdb.main, which seems to do its own reading of the xtdb.edn config file. I'm not quite sure I understand how I could slot aero into that equation :thinking_face:

alexdavis18:05:41

Ah ok, not sure I’ve seen someone running xtdb in production like that, usually there’s enough of an advantage to integrant and maybe a repl namespace to make using a custom main function worth it, but there may well still be a way to do it with xtdb.main. (@U899JBRPF will probably know)

cjsauer18:05:56

Right now I'm just overriding the module implementations. I grab the env vars and then pass them down as maps into the stock ->connection-pool and ->document-store functions, which seems to be working okay.

alexdavis18:05:49

I guess you could pass options via the cli and read from env vars that way? A brief skim of the code makes it seem like there isn’t a way to do it just with the config file as it doesn’t get put through aero

cjsauer18:05:08

Yea that's what I'm gathering from reading the code as well.

cjsauer18:05:22

(defn ->document-store {::sys/args {:prefix {:required? false,
                                             :spec      :xtdb.s3/prefix
                                             :doc       "S3 prefix"}}
                        ::sys/deps {:configurator   'xtdb.s3/->configurator
                                    :document-cache 'xtdb.cache/->cache}}
  [props]
  (xtdb.s3/->document-store
    (assoc props :bucket (System/getenv "XTDB_BUCKET_NAME"))))

cjsauer18:05:45

I've made my own namespace and am overriding the module implementations like so

cjsauer18:05:10

Instead of requiring those options from the config file I'm gathering them from the env and passing them down into the stock modules

cjsauer18:05:14

Tedious, but works

cjsauer18:05:34

Making a main function that just calls xt/start-node is far easier tho, should've just done that from the start 🙂

cjsauer15:05:58

I have the same question for using S3 as the document store. Must I hardcode the bucket name into the xtdb.edn file?

sparkofreason16:05:58

Looking at the 2.x docs, I'm having a mental block understanding how the join occurs in this query. Do you have sample results you could share?

;; find me the supplier(s) offering this part for the lowest price

{:find [supplier-name part-price]
 :in [part-id]
 :where [($ :suppliers [{:xt/id supplier-id} supplier-name])
         ($ :supplier-prices [supplier-id part-id part-price])

         (q {:find [(min part-price)]
             :keys [min-part-price]
             :in [part-id]
             :where [($ :supplier-prices [part-id part-price])]})]}

FiVo18:05:20

Yes that query looks off. I am not at the REPL so can't test, but on the outer where you need to check that min-part-price is equal to part-price.

sparkofreason18:05:55

Okay, that makes more sense. Thank you!

2023-05-05

Channels