This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-02-21
Channels
- # announcements (4)
- # architecture (161)
- # autochrome-github (7)
- # babashka (61)
- # beginners (42)
- # calva (24)
- # cider (22)
- # clj-kondo (28)
- # cljs-dev (8)
- # clojure (88)
- # clojure-art (2)
- # clojure-dev (7)
- # clojure-europe (43)
- # clojure-germany (2)
- # clojure-nl (2)
- # clojure-uk (4)
- # clojurescript (32)
- # core-async (41)
- # cursive (32)
- # datahike (6)
- # datomic (9)
- # emacs (22)
- # events (2)
- # fulcro (10)
- # graphql (1)
- # nextjournal (16)
- # off-topic (9)
- # overtone (1)
- # pathom (16)
- # polylith (5)
- # quil (7)
- # rdf (1)
- # re-frame (7)
- # reagent (22)
- # releases (2)
- # remote-jobs (1)
- # reveal (12)
- # sci (1)
- # shadow-cljs (12)
- # specter (20)
- # sql (6)
- # tools-deps (21)
- # vim (26)
- # xtdb (10)
On the subject of literature, is there a good curated list that is recommended to learn beyond syntax? Patterns, practices, lifecycle, deployment, tooling, etc?
A list of resources to help learn Clojure https://clojure.org/community/resources#_tutorials_and_learning_materials
I’ve never used it, so I can’t vouch for it, but you can explore https://clojure-finance.github.io/clojure-backtesting-website/
(defn -main
"main app."
[& args]
;获取参数指定秒数
;
(let [[arg1 arg2] *command-line-args*]
(println "arg1=" arg1 "arg2=" arg2)
))
I use the clj -X run -main :
clj -X bzsczx.core/-main -minute 60000
output: arg1= bzsczx.core/-main arg2= -minute
How can I pass parameters in clj -x -main
• https://clojure.org/reference/deps_and_cli#_execute_a_function
• https://clojure.org/reference/deps_and_cli#_running_a_main_or_script
you should read these for reference. the argument style of & args
is the argument structure used with -M -m
a "main" style. The Clojure cli also helpfully allows calling a function that takes a single map argument using -X
. You need to figure out what you want to pass on the command line and then use the style that works with what you need
One of my reitit-ring routes needs to do some I/O where order is important (it first needs to query a different resource and then commit if that resource is in a specific state and after the first success of this request for a specific argument all following requests should fail for the same argument). I implemented a new route to test some solutions and came up with this:
["/inc"
{::auth/roles #{:any}
:get
{:responses
{200
{:body
{:msg string?}}}
:handler
(fn [_]
(send-off test-agent
#(let [val @test-atom]
(Thread/sleep (rand-int 5))
(when (< val 20000)
(reset! test-atom (+ val 1)))
(inc %)))
(await test-agent)
(response/ok {:msg (str "New value: " @test-atom)}))}}]
I tested it with a thousand threads hammering away at the endpoint and it seems to do what I want, but is this a good idea/sound design? I read in 'Joy of Clojure' that using await like this is discouraged but I am not sure how to do it differently. Should I use locks here?I made an alternative handler with the locking
macro, which seems 'cleaner' than awaiting the agent. However, I will probably need a ReentrantLock
when going with explicit locking since I don't want to lock the resource entirely but only one way of accessing it. Not sure whether going with an agent for that function or making a lock makes more sense then.
(fn [_]
(locking lock-atom
(let [val @lock-atom]
(Thread/sleep (rand-int 5))
(when (< val 20000)
(reset! lock-atom (+ val 1))))
(response/ok {:msg (str "New value: " @lock-atom)})))
Looking at https://clojurians.slack.com/archives/C03S1KBA2/p1632283053293700 I can just use namespaced keywords with locking
and forego the need for a pool of ReentrantLock
s which is great.
Not sure where to put the locks though. Since I have two entrypoints (ring-router and websockets) putting them there would needlessly duplicate the locking mechanism, so that's out. But then I still have the implementation abstraction layer that bundles functionality together (e.g. user, messages, ...) and the actual implementation for persisting and querying (e.g. File I/O and DB). Locking probably is an implementation detail, right?
Oh, don’t go down the locking path! This is too easy to get wrong and this is why Clojure has all those agents, refs, etc. — so that you don’t have to worry about locking
If I understand correctly, you have a race of multiple concurrent requests to a path. The first request shall “win” and do some IO, and the others should wait for the result of this IO?
If so, use an atom
for the state, use compare-and-set!
to try to get a promise
into the atom. If that works, you are the winner, and can do the IO, finally deliver
the promise. If the compare-and-set
failed, someone else will do the IO, and you can deref
the promise, maybe with a timeout, to wait for the result.
(def result (atom nil))
,,,
(if-let [p @result]
;; There is already a promise
@p
(let [p (promise)]
(if (compare-and-set! result nil p)
;; We got OUR promise into the atom
(do (the i/o)
(deliver p value))
;; Someone else beat us. Deref the atom to get the promise, then deref the promise.
@@result)))
This is just an example 🙂
This is a lot more elegant than what I was cooking up. I think I'll need some time to fully wrap my head around it though. Thank you very much!
WARNING: catch any exceptions and make sure you really deliver the promise, or you will dead-lock other threads.
Instead of @p
, you can use (deref p timeout-ms default-value)
to time-out if the IO does not deliver the promise in an expected time
anytime, let us know how it goes 🙂
Can I use compare-and-set!
to only compare against a key-value pair in a map?
Theoretically there could be close to 200,000 different arguments to the io-function and the race condition should only occur once per unique argument (until cleared at least). In practice the daily used distinct arguments will probably never go over 20.
I am trying to work around this by using an atom which contains a map of atoms. By wrapping the if-let in a try-catch block for NullPointerExceptions I compare-and-set!
a new (atom nil)
and then call the function again since apparently catch isn't a tail position.recur
compare-and-set!
only works with atoms. Alternatively you could use just a single atom with a map, and use swap!
with a condition:
(let [my-promise (promise)
results-after (swap! results update args (fn [p] (or p my-promise)))
promise (get results-after args)]
(if (= promise my-promise)
;; I got my promise in, so I will be responsible to deliver it!
(do ... (deliver promise value))
;; Otherwise there was already a promise, so wait for it to be delivered
@promise)))
The results
atom will hold a map args => promise
. With swap!
, we will only place a new promise in when there is none before.
(fn race
([{{{:keys [id role]} :body} :parameters}]
(race id role @promise-atom))
([id role pa-old]
(try
(Thread/sleep (rand-int 7))
(if-let [prom @(get @promise-atom (keyword role))]
(if-let [other-prom (deref prom 30000 nil)]
(response/ok {:msg (str "Winner: " other-prom " in race for " role)})
(response/ok {:msq "Huh..."}))
(let [prom-new (promise)]
(if (compare-and-set! (get @promise-atom (keyword role)) nil prom-new)
(do (Thread/sleep (rand-int 5))
(deliver prom-new id)
(response/ok {:msg (str "Winner: " (deref @(get @promise-atom (keyword role)) 30000 nil) " in race for " role)}))
(response/ok {:msg (str "Winner: " (deref @(get @promise-atom (keyword role)) 30000 nil) " in race for " role)}))))
(catch NullPointerException _
(compare-and-set! promise-atom pa-old (assoc pa-old (keyword role) (atom nil)))
(race id role @promise-atom)))))
This is the implementation of what I was talking about. It works but I'm really not sure if I want to keep it.
This is the function (plus some code for testing, minus proper docstrings) I "came up with" (read: slightly modified) thanks to your advice:
(ns <project name>.helpers.side-effects)
(defonce ^:dynamic *book-of-promises* (atom {}))
(defn highlander
"There can only be one."
[& {:keys [resource-id
args
deliver-fn
success-fn
error-fn]}]
{:pre [(keyword? resource-id) (vector? args)
(fn? deliver-fn) (fn? success-fn) (fn? error-fn)]}
(let [new-pledge (promise)
entry (swap! *book-of-promises*
update-in [resource-id args]
(fn [old-val] (or old-val
new-pledge)))
current-val (get-in entry [resource-id args])]
(if (= new-pledge current-val)
(success-fn (deref (deliver new-pledge (deliver-fn)) 30000 {:error "Could not deref promise."}))
(error-fn (deref current-val 30000 {:error "Could not deref promise."})))))
(defn dethrone [& {:keys [resource-id
args]}]
(swap! *book-of-promises* update-in [resource-id args] nil))
(comment
(def greek-alphabet ["Alpha" "Beta" "Gamma" "Delta"
"Epsilon" "Zeta" "Eta" "Theta"
"Iota" "Kappa" "Lambda" "Mu"
"Nu" "Xi" "Omicron" "Pi" "Rho"
"Sigma" "Tau" "Upsilon" "Phi"
"Chi" "Psi" "Omega"])
(def thread-pool
(java.util.concurrent.Executors/newFixedThreadPool
(+ 2 (.availableProcessors (Runtime/getRuntime)))))
(defn dothreads-with-arg!
[f & {thread-count :threads
exec-count :times
results-delay :results-delay
:or {thread-count 1 exec-count 1 results-delay 100}}]
(let [results-map (atom {})]
(dotimes [t thread-count]
(.submit thread-pool
#(dotimes [_ exec-count] (swap! results-map assoc t (f t)))))
(Thread/sleep results-delay)
@results-map))
(dothreads-with-arg!
(fn [t] (highlander :resource-id :tester
:args ["c" "b"]
:deliver-fn #(get greek-alphabet t)
:success-fn (fn [val] (hash-map :success true :val val))
:error-fn (fn [val] {:failed? true :val val})))
:threads 24)
(deref *book-of-promises*)
*e)
I'll probably need to abstract the implementation and combine it with a message queue in the future (I'll post the code here when it's done) but as long as I'm still on a single JVM this makes me pretty happy.
Again, thank you very much for your help!Glad that worked for you! 👍
Quick note: (success-fn (deref (deliver new-pledge (deliver-fn)) 30000 {...}))
— I don’t think (deref (deliver))
can block, since you just delivered the promise. Could be simply: (success-fn @(deliver new-pledge (deliver-fn)))
Does defonce evaluate the body of the def if the variable already has a value? In other words, could a defonce replace a memoized function?
See the clojuredocs : https://clojuredocs.org/clojure.core/keyword