honeysql

joshkh 2024-04-19T19:16:14.455039Z

i’m struggling to get the following Postgres upsert syntax to work where, if the user exists, jsonb_strip_nulls is used to merge in the new JSON and remove the keys with nil values. any help would be much appreciated!

(deftest upsert-json-column-merge-strip-nulls
  (let [ds (db-fixtures/get-ds)]

    ; set up the test table (success)
    (jdbc/execute!
      ds
      (hsql/format
        {:create-table :user_preferences
         :with-columns [[:user_id :int]
                        [:preferences :jsonb]
                        [[:primary-key :user_id]]]}))

    ; create a new user (success)
    (jdbc/execute!
      ds
      (hsql/format
        {:insert-into [:user_preferences]
         :values [{:user_id 1 :preferences [:lift {:theme "dark"}]}]}))

    ; upsert the user's preferences (throws exception)
    (jdbc/execute!
      ds
      (hsql/format
        {:insert-into [:user_preferences]
         :values [{:user_id 1 :preferences [:lift {:theme "light"}]}]
         :on-conflict [:user_id]
         :do-update-set {:fields
                         {:user_preferences.preferences
                          [:jsonb_strip_nulls
                           [:|| [:coalesce :preferences [:lift {}]] [{:theme "light"}]]]}}}))))

clojure.lang.ExceptionInfo: These SQL clauses are unknown or have nil values: :theme
    theme: "light"

βœ… 1
seancorfield 2024-04-20T16:40:18.337299Z

You could just use :EXCLUDED.name instead of that :raw expression:

user=> (->
{:insert-into [:user_preferences]
         :values [{:user_id 1
                   :name "sally"
                   :preferences [:lift {:theme "light"}]}]
         :on-conflict [:user_id]
         :do-update-set {:fields
                         {:name :EXCLUDED.name
                          :preferences
                          [:jsonb_strip_nulls
                           [:||
                            [:coalesce :user_preferences.preferences [:lift {}]]
                            [[:lift {:theme "light" :font nil}]]]]}}}
sql/format)
["INSERT INTO user_preferences (user_id, name, preferences) VALUES (?, ?, ?) ON CONFLICT (user_id) DO UPDATE SET name = EXCLUDED.name, preferences = JSONB_STRIP_NULLS(COALESCE(user_preferences.preferences, ?) || (?))" 1 "sally" {:theme "light"} {} {:theme "light", :font nil}]
user=>

seancorfield 2024-04-20T16:42:09.295399Z

(if you have quoting enabled, you might need :'EXCLUDED.name instead maybe)

joshkh 2024-04-19T19:43:44.177589Z

i think i just figured it out. thanks just the same 😎

(deftest upsert-json-column-merge-strip-nulls
  (let [ds (db-fixtures/get-ds)]

    ; βœ… set up the test table
    (jdbc/execute!
      ds
      (hsql/format
        {:create-table :user_preferences
         :with-columns [[:user_id :int]
                        [:preferences :jsonb]
                        [[:primary-key :user_id]]]}))

    ; βœ… create a new user
    (jdbc/execute!
      ds
      (hsql/format
        {:insert-into [:user_preferences]
         :values [{:user_id 1 :preferences [:lift {:theme "dark"}]}]}))

    ; βœ… upsert the user's preferences
    (jdbc/execute!
      ds
      (hsql/format
        {:insert-into [:user_preferences]
         :values [{:user_id 1 :preferences [:lift {:theme "light"}]}]
         :on-conflict [:user_id]
         :do-update-set {:fields
                         {:preferences
                          [:jsonb_strip_nulls
                           [:|| [:coalesce :user_preferences.preferences [:lift {}]] [[:lift {:theme "light"}]]]]}}}))))

seancorfield 2024-04-19T19:59:24.974729Z

Yup, :lift is your friend with JSON object types in PostgreSQL.

joshkh 2024-04-19T20:04:37.724339Z

yup. and thanks for the awesome library 😎. is there a way to combine :do-update-set with both :fields and the short term vector syntax for EXCLUDED?

(jdbc/execute!
  ds
  (hsql/format
    {:insert-into [:user_preferences]
     :values [{:user_id 1
               :name "sally"
               :preferences [:lift {:theme "light"}]}]
     :on-conflict [:user_id]
     :do-update-set {:fields
                     ; is there a way to get EXCLUDED for free when combined with :fields?
                     {:name "EXCLUDED.name"
                      :preferences [:jsonb_strip_nulls
                                    [:||
                                     [:coalesce :user_preferences.preferences [:lift {}]]
                                     [[:lift {:theme "light" :font nil}]]]]}}}))

joshkh 2024-04-19T20:05:20.968499Z

actually, that didn’t work as expected

joshkh 2024-04-19T20:07:06.693999Z

but the question still stands πŸ™‚

joshkh 2024-04-19T20:08:51.492659Z

this would be the correct version:

(jdbc/execute!
      ds
      (hsql/format
        {:insert-into [:user_preferences]
         :values [{:user_id 1
                   :name "sally"
                   :preferences [:lift {:theme "light"}]}]
         :on-conflict [:user_id]
         :do-update-set {:fields
                         {:name [:raw "EXCLUDED.name"]
                          :preferences
                          [:jsonb_strip_nulls
                           [:||
                            [:coalesce :user_preferences.preferences [:lift {}]]
                            [[:lift {:theme "light" :font nil}]]]]}}}))

seancorfield 2024-04-19T20:24:08.424689Z

I don't use PG so I can't give much guidance on specifics

joshkh 2024-04-19T20:26:51.923359Z

no worries, [:raw "EXCLUDED.name"] works fine, thank you. on my side it’s a trip coming back from datalog to the relational world. like learning to ride a bike.. backwards πŸ˜‰