Clojurians Log v2

I'm wanting to swap table-num s here, so that :id :a gets table-num 4 and :b gets 2:

(let [st {:tables [{:id :a, :table-num 2}
                   {:id :b, :table-num 4}]}
      swap-f (fn [st from to]
               (->> st
                 (s/setval [:tables s/ALL :table-num (partial = from)] to)
                 (s/setval [:tables s/ALL :table-num (partial = to)] from)))]
    ;; Unfortunately this is true:
    (= {:tables [{:id :a, :table-num 2} {:id :b, :table-num 2}]}
      (swap-f st 2 4)))

As you can see it doesn't work out, they both end up with table-num 2. How should I achieve this kind of swapping operation with Specter?