Fork me on GitHub
#beginners
<
2022-02-21
>
Nathan Rogers02:02:21

On the subject of literature, is there a good curated list that is recommended to learn beyond syntax? Patterns, practices, lifecycle, deployment, tooling, etc?

dumrat04:02:01

Is there any good clojure backtesting frameworks?

quoll05:02:41

I’ve never used it, so I can’t vouch for it, but you can explore https://clojure-finance.github.io/clojure-backtesting-website/

dumrat13:02:26

Yeah I saw this, but seems it doesn't support intra day data

quan xing05:02:22

(defn -main
  "main app."
  [& args]
  ;获取参数指定秒数
  ;
  (let [[arg1 arg2] *command-line-args*]
    (println "arg1=" arg1 "arg2=" arg2)
    ))
I use the clj -X run -main :
clj -X bzsczx.core/-main -minute 60000
output: arg1= bzsczx.core/-main arg2= -minute How can I pass parameters in clj -x -main

dpsutton06:02:16

https://clojure.org/reference/deps_and_cli#_execute_a_functionhttps://clojure.org/reference/deps_and_cli#_running_a_main_or_script you should read these for reference. the argument style of & args is the argument structure used with -M -m a "main" style. The Clojure cli also helpfully allows calling a function that takes a single map argument using -X. You need to figure out what you want to pass on the command line and then use the style that works with what you need

Lycheese09:02:16

One of my reitit-ring routes needs to do some I/O where order is important (it first needs to query a different resource and then commit if that resource is in a specific state and after the first success of this request for a specific argument all following requests should fail for the same argument). I implemented a new route to test some solutions and came up with this:

["/inc"
    {::auth/roles #{:any}
     :get
     {:responses
      {200
       {:body
        {:msg string?}}}
      :handler
      (fn [_]
        (send-off test-agent
                  #(let [val @test-atom]
                     (Thread/sleep (rand-int 5))
                     (when (< val 20000)
                       (reset! test-atom (+ val 1)))
                     (inc %)))
        (await test-agent)
        (response/ok {:msg (str "New value: " @test-atom)}))}}]
I tested it with a thousand threads hammering away at the endpoint and it seems to do what I want, but is this a good idea/sound design? I read in 'Joy of Clojure' that using await like this is discouraged but I am not sure how to do it differently. Should I use locks here?

Lycheese10:02:30

I made an alternative handler with the locking macro, which seems 'cleaner' than awaiting the agent. However, I will probably need a ReentrantLock when going with explicit locking since I don't want to lock the resource entirely but only one way of accessing it. Not sure whether going with an agent for that function or making a lock makes more sense then.

(fn [_]
        (locking lock-atom
          (let [val @lock-atom]
            (Thread/sleep (rand-int 5))
            (when (< val 20000)
              (reset! lock-atom (+ val 1))))
          (response/ok {:msg (str "New value: " @lock-atom)})))

Lycheese11:02:33

Looking at https://clojurians.slack.com/archives/C03S1KBA2/p1632283053293700 I can just use namespaced keywords with locking and forego the need for a pool of ReentrantLocks which is great.

Lycheese11:02:45

Not sure where to put the locks though. Since I have two entrypoints (ring-router and websockets) putting them there would needlessly duplicate the locking mechanism, so that's out. But then I still have the implementation abstraction layer that bundles functionality together (e.g. user, messages, ...) and the actual implementation for persisting and querying (e.g. File I/O and DB). Locking probably is an implementation detail, right?

Ferdinand Beyer11:02:47

Oh, don’t go down the locking path! This is too easy to get wrong and this is why Clojure has all those agents, refs, etc. — so that you don’t have to worry about locking

Ferdinand Beyer11:02:35

If I understand correctly, you have a race of multiple concurrent requests to a path. The first request shall “win” and do some IO, and the others should wait for the result of this IO?

Ferdinand Beyer11:02:58

If so, use an atom for the state, use compare-and-set! to try to get a promise into the atom. If that works, you are the winner, and can do the IO, finally deliver the promise. If the compare-and-set failed, someone else will do the IO, and you can deref the promise, maybe with a timeout, to wait for the result.

Ferdinand Beyer11:02:43

(def result (atom nil))
,,,
(if-let [p @result]
  ;; There is already a promise
  @p
  (let [p (promise)]
    (if (compare-and-set! result nil p)
      ;; We got OUR promise into the atom
      (do (the i/o)
          (deliver p value))
      ;; Someone else beat us. Deref the atom to get the promise, then deref the promise.
      @@result)))

Ferdinand Beyer11:02:00

This is just an example 🙂

Lycheese11:02:21

This is a lot more elegant than what I was cooking up. I think I'll need some time to fully wrap my head around it though. Thank you very much!

Ferdinand Beyer11:02:30

WARNING: catch any exceptions and make sure you really deliver the promise, or you will dead-lock other threads. Instead of @p, you can use (deref p timeout-ms default-value) to time-out if the IO does not deliver the promise in an expected time

Lycheese11:02:05

I'll keep that in mind, thank you.

Ferdinand Beyer11:02:25

anytime, let us know how it goes 🙂

Lycheese12:02:35

Can I use compare-and-set! to only compare against a key-value pair in a map? Theoretically there could be close to 200,000 different arguments to the io-function and the race condition should only occur once per unique argument (until cleared at least). In practice the daily used distinct arguments will probably never go over 20. I am trying to work around this by using an atom which contains a map of atoms. By wrapping the if-let in a try-catch block for NullPointerExceptions I compare-and-set! a new (atom nil) and then recurcall the function again since apparently catch isn't a tail position.

Lycheese12:02:24

Welp I wanted to wait until I had the code working to send that.

Ferdinand Beyer12:02:23

compare-and-set! only works with atoms. Alternatively you could use just a single atom with a map, and use swap! with a condition:

(let [my-promise    (promise)
      results-after (swap! results update args (fn [p] (or p my-promise)))
      promise       (get results-after args)]
  (if (= promise my-promise)
    ;; I got my promise in, so I will be responsible to deliver it!
    (do ... (deliver promise value))
    ;; Otherwise there was already a promise, so wait for it to be delivered
    @promise)))

Ferdinand Beyer12:02:05

The results atom will hold a map args => promise. With swap!, we will only place a new promise in when there is none before.

Lycheese13:02:23

(fn race
        ([{{{:keys [id role]} :body} :parameters}]
         (race id role @promise-atom))
        ([id role pa-old]
         (try
           (Thread/sleep (rand-int 7))
           (if-let [prom @(get @promise-atom (keyword role))]
             (if-let [other-prom (deref prom 30000 nil)]
               (response/ok {:msg (str "Winner: " other-prom " in race for " role)})
               (response/ok {:msq "Huh..."}))
             (let [prom-new (promise)]
               (if (compare-and-set! (get @promise-atom (keyword role)) nil prom-new)
                 (do (Thread/sleep (rand-int 5))
                     (deliver prom-new id)
                     (response/ok {:msg (str "Winner: " (deref @(get @promise-atom (keyword role)) 30000 nil) " in race for " role)}))
                 (response/ok {:msg (str "Winner: " (deref @(get @promise-atom (keyword role)) 30000 nil) " in race for " role)}))))
           (catch NullPointerException _
             (compare-and-set! promise-atom pa-old (assoc pa-old (keyword role) (atom nil)))
             (race id role @promise-atom)))))

Lycheese13:02:53

This is the implementation of what I was talking about. It works but I'm really not sure if I want to keep it.

Lycheese14:02:52

Your solution above seems a lot less complicated. I think I'll use that 😅😂

Lycheese11:02:00

This is the function (plus some code for testing, minus proper docstrings) I "came up with" (read: slightly modified) thanks to your advice:

(ns <project name>.helpers.side-effects)

(defonce ^:dynamic *book-of-promises* (atom {}))

(defn highlander
  "There can only be one."
  [& {:keys [resource-id
             args
             deliver-fn
             success-fn
             error-fn]}]
  {:pre [(keyword? resource-id) (vector? args)
         (fn? deliver-fn) (fn? success-fn) (fn? error-fn)]}
  (let [new-pledge (promise)
        entry (swap! *book-of-promises*
                     update-in [resource-id args]
                     (fn [old-val] (or old-val
                                       new-pledge)))
        current-val (get-in entry [resource-id args])]
    (if (= new-pledge current-val)
      (success-fn (deref (deliver new-pledge (deliver-fn)) 30000 {:error "Could not deref promise."}))
      (error-fn (deref current-val 30000 {:error "Could not deref promise."})))))

(defn dethrone [& {:keys [resource-id
                          args]}]
  (swap! *book-of-promises* update-in [resource-id args] nil))

(comment
  (def greek-alphabet ["Alpha" "Beta" "Gamma" "Delta"
                       "Epsilon" "Zeta" "Eta" "Theta"
                       "Iota" "Kappa" "Lambda" "Mu"
                       "Nu" "Xi" "Omicron" "Pi" "Rho"
                       "Sigma" "Tau" "Upsilon" "Phi"
                       "Chi" "Psi" "Omega"])

  (def thread-pool
    (java.util.concurrent.Executors/newFixedThreadPool
     (+ 2 (.availableProcessors (Runtime/getRuntime)))))

  (defn dothreads-with-arg!
    [f & {thread-count :threads
          exec-count :times
          results-delay :results-delay
          :or {thread-count 1 exec-count 1 results-delay 100}}]
    (let [results-map (atom {})]
      (dotimes [t thread-count]
        (.submit thread-pool
                 #(dotimes [_ exec-count] (swap! results-map assoc t (f t)))))
      (Thread/sleep results-delay)
      @results-map))

  (dothreads-with-arg!
   (fn [t] (highlander :resource-id :tester
                       :args ["c" "b"]
                       :deliver-fn #(get greek-alphabet t)
                       :success-fn (fn [val] (hash-map :success true :val val))
                       :error-fn (fn [val] {:failed? true :val val})))
   :threads 24)

  (deref *book-of-promises*)
  *e)
I'll probably need to abstract the implementation and combine it with a message queue in the future (I'll post the code here when it's done) but as long as I'm still on a single JVM this makes me pretty happy. Again, thank you very much for your help!

👍 1
Ferdinand Beyer12:02:11

Glad that worked for you! 👍 Quick note: (success-fn (deref (deliver new-pledge (deliver-fn)) 30000 {...})) — I don’t think (deref (deliver)) can block, since you just delivered the promise. Could be simply: (success-fn @(deliver new-pledge (deliver-fn)))

GGfpc12:02:21

Does defonce evaluate the body of the def if the variable already has a value? In other words, could a defonce replace a memoized function?

pinkfrog14:02:50

How’s this valid? (keyword “spa ce”)

V14:02:07

The keyword function does not validate it's input

V14:02:57

;; keyword does not validate input strings for ns and name, and may
;; return improper keywords with undefined behavior for non-conformant
;; ns and name.

Ben Sless14:02:51

I've seen some crazy stuff with that. Nightmare fuel

sova-soars-the-sora15:02:08

i t ' s j u s t w h it es p a c e r i g h t?

😁 3