This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-09-22
Channels
- # announcements (2)
- # beginners (137)
- # chlorine-clover (13)
- # clj-kondo (3)
- # cljsrn (4)
- # clojure (52)
- # clojure-australia (3)
- # clojure-dev (2)
- # clojure-europe (34)
- # clojure-nl (1)
- # clojure-sg (3)
- # clojure-spec (1)
- # clojure-uk (12)
- # clojurescript (2)
- # clojureverse-ops (7)
- # code-reviews (3)
- # conjure (2)
- # cursive (18)
- # datavis (21)
- # datomic (34)
- # exercism (1)
- # figwheel-main (6)
- # graphql (3)
- # helix (21)
- # introduce-yourself (1)
- # jackdaw (1)
- # jobs (4)
- # jobs-discuss (32)
- # juxt (14)
- # leiningen (6)
- # lsp (35)
- # meander (19)
- # nrepl (2)
- # off-topic (37)
- # portal (40)
- # quil (5)
- # re-frame (45)
- # reagent (10)
- # releases (1)
- # remote-jobs (4)
- # reveal (15)
- # sci (7)
- # shadow-cljs (40)
- # spacemacs (3)
- # tools-build (2)
- # vim (17)
- # xtdb (11)
What's the easiest way to lock part of a function body based on a parameter value to prevent concurrent execution? I have something like this:
(defn my-f [account-id]
(do ...)
;; this block has to be locked
(locking account-id
...)
)
account-id
is typically an integer.
With small ints it seems to work (jvm reusing the same objects from the pool?)
but for larger numbers it doesn't work:
(defn my-f [account-id]
(locking account-id
(Thread/sleep 2000)
(println "done.")
))
(future (my-f 123456789))
(future (my-f 123456789))
(future (my-f 123456789))
;; these are printed immediately one after another.
done.
done.
done.
In the past, for Strings, I called .intern
on the parameter which worked but it felt dirty.I agree with the recommendations of others to single thread the activity using worker queues, but I think this is more in line with your original plans: https://github.com/RutledgePaulV/missing/blob/develop/src/missing/locks.clj
small numbers are interned, so two instances are the same object. outside the byte range this is no longer the case. you need two locked sections to be looking at the same object.
(cmd)user=> (identical? 127 127)
true
(cmd)user=> (identical? -128 -128)
true
(cmd)user=> (identical? 128 128)
false
(ins)user=> (identical? -129 -129)
false
or (locking (keyword "lock-my-f" (str account-id)))
since keywords are interned
Are keywords ever GCed? I know strings will be GCed if there is no more references to them which is nice in this case.
the keyword cache uses weak-refs so they will be GCed
https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Keyword.java#L41
Make sense, so ya you could lock on:
(keyword (str *ns* "/" "lock.my-f" "." acount-id))
As well. Since that's basically an interned pool of weak references already. Just make sure your keys are always unique to the function you're using them, otherwise a clash can deadlock. That's why I namespace it by ns and fn name.For example have 4 work queues(single threaded), take the mod of each number with 4, and put it on that work queue
No locking, a single threaded processes everything for a given number, but you still have multiple threads doing work
I'm not sure I got it. I basically have a lower-level function which does certain changes to the account records in the database (fetch some data, changes some bits over there and some other bits elsewhere) This function is called from multiple places higher up in the stack. To prevent unexpected results I want to make sure that a particular portion of that function is executed serially for any given account id. It seems like having a queue consumed by that function could work but it looks quite complicated and definitely unusual for the existing app architecture. I also thought about using transaction isolation level SERIALIZABLE but even that would be cumbersome since the transaction object is passed as a parameter and I'd have to modify several callers and enforce the serializable level even though they should not need to know anything about it.
also, about locking on a number, compare (identical? 127 127)
to (identical? 128 128)
ah ok. the first time i learned about this was quite surprising. so wasn't sure if it was new to you ๐
But thanks @hiredman for the pubsub idea. It's not something I can use for this particular app/problem but it's a good alternative to keep in mind when encountering similar problems in the future ๐.
if you can assume a single jvm then you could make a vec of mutexes and then mod the account id by the vec length to select which to use but you'd need to be careful to choose a good vec size to lessen false collisions
< if you can assume a single jvm I was about to say that I can but you got me thinking and realized that I can't - it's really a distributed system and it could happen that the concurrent updates are done on separate machines...
But thanks for mentioning Postgres' advisory locks. While I can't use them it's an interesting thing to know about.
If you need a distributed lock, and are cool with AWS, this works well: https://aws.amazon.com/blogs/database/building-distributed-locks-with-the-dynamodb-lock-client/
Or if you use Redis, Redisson has distributed locks built over it: https://github.com/redisson/redisson/wiki/8.-Distributed-locks-and-synchronizers#81-lock
Do you def variables so that you can evaluate forms that depend on them in the REPL? If so, how do you avoid having to re-define those variables every time you want to evaluate something in a different namespace that depends upon the same thing?
> evaluate something in a different namespace
def
s can also be in a different namespace. Or you can use let
.
@U016XBH746B The problem is usually that optimally you shouldn't type directly in a REPL, but evaluate from your editor which already sets the right namespace
Another approach is to always stay in the user
ns and control everything from there, but this is really a primitive way of working
Yes, I see what you mean, but imagine I have a map that represents a car for example, and functions take in a variable named car as parameter. I usually will define the car variable in the repl or evaluate it from a comment form so that I can evaluate parts of the body from the editor If I then have to evaluate a form in another namespace that also is within a function that takes in a car variable, the original variable I defined isn't found. (Obviously because I switched to that new namespace)
for my most heavily used repl things I def
them in user
, otherwise I use rich comments. so, basically what everyone else said above
If you use cider, you can use C-x C-v C-b
(I think ... Currently typing on my phone) which will prompt you for some bindings before evaling the previous form ... Meaning you don't have to def
anything and it'll remember the bindings in your history when you switch namespaces
When launching Clojure from the command line, is there a way to direct it to use an existing Clojure process (say, that of a running REPL) if one is available?
You can pass the special argument to clojure that causes it to start a repl on a given port, then use netcat or whatever to connect to it
@afoltzm Tangentially related, I did an experiment with running a Clojure pREPL server, which is at most launched once and only when needed, and executing Clojure expressions in it from multiple babashka tasks. Perhaps as a bit of inspiration: https://gist.github.com/borkdude/5f3019d6666953f3b8f533a70eb067d2
@U04V15CAJ Thanks! The motivating use case is similar for me: I would like to reduce the barrier to using git hooks in my project by taking advantage of an already running Clojure process if it's present (and launching it if it's not).
I'm reading the tests for math.combinatorics
, and I find definitions like this:
(def factorial-numbers @#'clojure.math.combinatorics/factorial-numbers)
(deftest test-factorial-numbers
(are [x y] (= (factorial-numbers x) y)
463 [3 4 1 0 1 0]
0 []
1 [1 0]
2 [1 0 0]))
What is happening here, and why not just use the factorial-numbers
function directly?according here: https://stackoverflow.com/questions/37471253/how-can-i-write-unit-tests-for-private-clojure-functions and here: https://clojuredocs.org/clojure.core/defn-#example-5f74a0fce4b0b1e3652d73c7 It seems like one doesnโt need to deref the var if it is a function.
user=> (defn foo [] 1)
#'user/foo
user=> (#'foo)
1
user=> (@#'foo)
1
user=> (def bar #'foo)
#'user/bar
user=> (bar)
1
Where would be a good place to start learning about Classloaders? I've read the article that got posted here a while back about how the dynamic classloader works in Clojure, but I'm looking for general classloader knowledge. I'm trying to figure out how to ensure that the classloaders are set well for when I use System/loadLibrary and the new Panama jdk.incubator.foreign.SymbolLookup in order to create my ffi library.