Fork me on GitHub
#biff
<
2023-11-14
>
Lyn Headley16:11:34

Found a bug in production. Used the prod-repl to find it, soft-deploy to deploy the fix. took 20 minutes. Felt a surge of power course through me.

clojure-spin 3
🥷 2
3
Curious Yogurt17:11:38

Hello all. I'm exploring Clojure web development (hobby project to start) and have started to look into Biff. I'm having some difficulties with deployment (I'm following the official tutorial), so I was hoping I could get some help here. The tutorial assumes Digital Ocean and Nginx, which is fine—but that's not my setup, and so I'm unsure what I need to do a deploy. Currently, my remote setup, which is my target for deploying projects, is a fairly vanilla Arch server that runs Lighttpd. It currently serves up some static sites as well as a private wiki; nothing too serious. My question is, what are the tools that I need to do it? I'm pretty sure with a bit of fumbling I can figure it out; but after a bunch of searching I've not found a lot of help. Any suggestions as to where to start would be greatly appreciated. My level of experience: none with Babashka other the Biff tutorial. A medium amount with Clojure as a language, but not very much with Clojure tooling. A medium amount with Arch and Lighttpd.

Jacob O'Bryant18:11:31

Hey! You'll want to take a look at https://github.com/jacobobryant/biff/blob/master/example/server-setup.sh (to understand what kind of stuff is normally added to a fresh ubuntu server) and https://github.com/jacobobryant/biff/blob/master/tasks/src/com/biffweb/tasks.clj (to see how the server is interacted with). You shouldn't need to change tasks.clj at all, but you'll need to make a custom version of server-setup.sh. At a high level, server-setup.sh does the following: • installs a few programs like Clojure and https://github.com/athos/trenchman • creates an app user for the biff app • sets up a git repo for push-to-deploy (you won't need this if your local machine is mac/linux/WSL--it's just a fallback for plain windows users/people who don't have rsync) • creates a systemd service for the biff app so that it'll run on server start (and so that it'll be auto-restarted if it crashes) • gives the app user sudo permissions for restarting the systemd service • sets up a firewall with ufw • sets up nginx as a reverse proxy for your app (i.e. incoming http requests go to nginx first, then nginx passes them to your clojure app) • generates and installs an SSL certificate with certbot to understand what happens when you deploy, you can read the source for the bb deploy task: https://github.com/jacobobryant/biff/blob/5295ec19947eae0a1c71a8d9365c32ed081e3344/tasks/src/com/biffweb/tasks.clj#L362 basically it just uses rsync to copy the files over to the /home/app directory on your server, then it tells the server to restart the app. anyway--I'm guessing the main different thing you'll need to do is just configure lighttpd for the biff app; the rest of it will hopefully be just translating apt-get commands to pacman

Jacob O'Bryant18:11:12

and of course, if you run into any snags and you don't mind paying an extra $7/month, you can always just set up a separate server for the biff app 😉

Curious Yogurt18:11:05

Thank so much; that helps a lot as a place to start!

🙌 1
Lyn Headley20:11:11

Running into an issue with :biff/ensure-unique (see thread)

Lyn Headley20:11:20

Trying to run this transaction:

(defn create-cohort [ctx name]
  (biff/submit-tx ctx
                  [{:db/doc-type :cohort
                    :db/op :create
                    :cohort/name name}
                   [::xt/fn :biff/ensure-unique {:cohort/name name}]]))
Like so
(create-cohort (biff/assoc-db @main/system) "mom and dad")
Seeing the error
Transaction violated a constraint
   {:tx
    ([:xtdb.api/match #uuid "6c964091-9868-4e5b-a05e-e911e17fbd95" nil]
     [:xtdb.api/put
      {:cohort/name "mom and dad",
       :xt/id #uuid "6c964091-9868-4e5b-a05e-e911e17fbd95"}]
     [:xtdb.api/fn :biff/ensure-unique #:cohort{:name "mom and dad"}])}
                  xtdb.clj:  386  com.biffweb.impl.xtdb/submit-with-retries
                  xtdb.clj:  374  com.biffweb.impl.xtdb/submit-with-retries
                  xtdb.clj:  407  com.biffweb.impl.xtdb/submit-tx
                  xtdb.clj:  403  com.biffweb.impl.xtdb/submit-tx
               biffweb.clj:  639  com.biffweb/submit-tx
               biffweb.clj:  631  com.biffweb/submit-tx
                cohort.clj:    7  org.stinkless.rekonstruction.cohort/create-cohort
Here is the entire contents of the db as reflected by
(biff/q (:biff/db (biff/assoc-db @main/system)) '{:find [(pull entity [*])]
                                                    :where [[entity :xt/id]]})
(Note the weird [nil])
#{[#:xt{:fn
        (fn
         [ctx kvs]
         (let
          [kvs
           (for
            [[i [k v]]
             (map-indexed vector kvs)
             :let
             [sym (symbol (str "v" i))]]
            {:k k, :v v, :sym sym})
           query
           {:find '[doc],
            :limit 2,
            :in (mapv :sym kvs),
            :where (vec (for [{:keys [k sym]} kvs] ['doc k sym]))}]
          (when
           (<
            1
            (count
             (apply
              xtdb.api/q
              (xtdb.api/db ctx)
              query
              (map :v kvs))))
           false))),
        :id :biff/ensure-unique}]
  [nil]}

Jacob O'Bryant21:11:19

that is really weird... I just tried running a similar transaction on a new biff project and it worked. Not sure what's up with that [nil] either. Maybe try stopping the app, deleting the storage directory, then starting it up again and see if the transaction works?

Lyn Headley21:11:27

Sorry for the confusion. I am using postgres now 🙂 This explains the [nil]

(biff/q (:biff/db (biff/assoc-db @main/system)) '{:find [entity]
                                                    :where [[entity :xt/id]]})

Lyn Headley21:11:50

#{[#uuid "3f1df8fa-dbe4-4718-9e8e-a8e0097efd0d"]
  [#uuid "d3f99f8e-1cf6-485e-9c23-7b634df707dd"]
  [#uuid "00a35bd5-ec7c-4a57-9263-58a99751b16c"]
  [:biff/ensure-unique]
  [#uuid "de7faee5-6452-4ae8-add4-e143baa913ff"]
  [#uuid "74a7cddf-5b7a-455d-81cb-c975ff92d692"]
  [#uuid "428b94ee-2f0c-4e12-a240-050905f5e7b5"]
  [#uuid "8ada6c68-6881-4207-bd6a-82b6ab8ca98b"]
  [#uuid "9b786f18-c3d7-4339-973a-6af657babc50"]
  [#uuid "2adbc49d-8646-46bc-8165-74a1df438c31"]
  [#uuid "37fe7131-edce-47ad-b002-fce2aa4fba6f"]
  [#uuid "b7882f39-0284-46f3-b916-f04c5c82f0b8"]
  [#uuid "774f1b28-1aa7-4cdd-963a-8ad6601dbd1f"]
  [#uuid "cbf57103-247b-4cc3-82bc-5c3087cb1766"]
  [#uuid "a7d19e93-79f9-4d7a-9afe-e23f965bdf62"]}

Lyn Headley21:11:16

I thought I had wiped and recreated my db as follows, but apparently that isn't working

Lyn Headley21:11:45

root@cheapstories:~# systemctl stop app
root@cheapstories:~# sudo -i -u postgres dropdb cheapstories
root@cheapstories:~# sudo -i -u postgres createdb cheapstories
root@cheapstories:~# systemctl start app

Jacob O'Bryant21:11:58

ah got it. to wipe the DB you'll need to both drop the postgres DB and delete the storage directory, since the latter is still used for indexes. maybe that'll fix it?

Lyn Headley21:11:29

Is this the storage directory?

grep data_directory /etc/postgresql/12/main/postgresql.conf
data_directory = '/var/lib/postgresql/12/main'          # use data in another directory
root@cheapstories:~# grep storage /etc/postgresql/12/main/postgresql.conf
root@cheapstories:~# ls -la /var/lib/postgresql/12/main/
total 92
drwx------ 19 postgres postgres 4096 Nov 14 19:28 .
drwxr-xr-x  3 postgres postgres 4096 Nov 10 16:54 ..
drwx------  5 postgres postgres 4096 Nov 14 21:44 base
drwx------  2 postgres postgres 4096 Nov 14 19:29 global
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_commit_ts
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_dynshmem
drwx------  4 postgres postgres 4096 Nov 14 21:44 pg_logical
drwx------  4 postgres postgres 4096 Nov 10 16:54 pg_multixact
drwx------  2 postgres postgres 4096 Nov 14 19:28 pg_notify
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_replslot
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_serial
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_snapshots
drwx------  2 postgres postgres 4096 Nov 14 19:28 pg_stat
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_stat_tmp
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_subtrans
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_tblspc
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_twophase
-rw-------  1 postgres postgres    3 Nov 10 16:54 PG_VERSION
drwx------  3 postgres postgres 4096 Nov 10 16:54 pg_wal
drwx------  2 postgres postgres 4096 Nov 10 16:54 pg_xact
-rw-------  1 postgres postgres   88 Nov 10 16:54 postgresql.auto.conf
-rw-------  1 postgres postgres  130 Nov 14 19:28 postmaster.opts
-rw-------  1 postgres postgres  110 Nov 14 19:28 postmaster.pid

Lyn Headley22:11:39

Oh, I get it. /home/app/storage

Lyn Headley22:11:36

that totally worked!!

Jacob O'Bryant22:11:21

perfect! if you had previously dropped the postgres table without deleting storage, that possibly would've explained the weirdness; XT would've been in a bad state