Fork me on GitHub
#clojure
<
2021-09-22
>
jumar03:09:33

What's the easiest way to lock part of a function body based on a parameter value to prevent concurrent execution? I have something like this:

(defn my-f [account-id]
  (do ...)
  ;; this block has to be locked
  (locking account-id
    ...)
 
)
account-id is typically an integer. With small ints it seems to work (jvm reusing the same objects from the pool?) but for larger numbers it doesn't work:
(defn my-f [account-id]
  (locking account-id
    (Thread/sleep 2000)
    (println "done.")
    ))

(future (my-f 123456789))
(future (my-f 123456789))
(future (my-f 123456789))
;; these are printed immediately one after another.
done.
done.
done.
In the past, for Strings, I called .intern on the parameter which worked but it felt dirty.

rutledgepaulv15:09:07

I agree with the recommendations of others to single thread the activity using worker queues, but I think this is more in line with your original plans: https://github.com/RutledgePaulV/missing/blob/develop/src/missing/locks.clj

๐Ÿ‘ 2
noisesmith16:09:04

small numbers are interned, so two instances are the same object. outside the byte range this is no longer the case. you need two locked sections to be looking at the same object.

(cmd)user=> (identical? 127 127)
true
(cmd)user=> (identical? -128 -128)
true
(cmd)user=> (identical? 128 128)
false
(ins)user=> (identical? -129 -129)
false

didibus23:09:23

You might be able to just do:

(locking (.intern (str ::lock.my-f account-id))
  ...)

noisesmith18:09:47

or (locking (keyword "lock-my-f" (str account-id))) since keywords are interned

didibus18:09:27

Are keywords ever GCed? I know strings will be GCed if there is no more references to them which is nice in this case.

noisesmith19:09:09

the keyword cache uses weak-refs so they will be GCed

didibus19:09:13

Make sense, so ya you could lock on:

(keyword (str *ns* "/" "lock.my-f" "." acount-id))
As well. Since that's basically an interned pool of weak references already. Just make sure your keys are always unique to the function you're using them, otherwise a clash can deadlock. That's why I namespace it by ns and fn name.

jumar05:10:33

Just noticed the latest suggestions - interesting ideas, thanks!

hiredman04:09:40

Better to create a sort of pubsub

hiredman04:09:06

Basically partition/shard instead of locking

hiredman04:09:22

For example have 4 work queues(single threaded), take the mod of each number with 4, and put it on that work queue

hiredman04:09:14

No locking, a single threaded processes everything for a given number, but you still have multiple threads doing work

jumar04:09:12

I'm not sure I got it. I basically have a lower-level function which does certain changes to the account records in the database (fetch some data, changes some bits over there and some other bits elsewhere) This function is called from multiple places higher up in the stack. To prevent unexpected results I want to make sure that a particular portion of that function is executed serially for any given account id. It seems like having a queue consumed by that function could work but it looks quite complicated and definitely unusual for the existing app architecture. I also thought about using transaction isolation level SERIALIZABLE but even that would be cumbersome since the transaction object is passed as a parameter and I'd have to modify several callers and enforce the serializable level even though they should not need to know anything about it.

dpsutton04:09:06

also, about locking on a number, compare (identical? 127 127) to (identical? 128 128)

jumar04:09:49

I know, that's why I chose large numbers in the code snippet ๐Ÿ™‚

dpsutton04:09:54

ah ok. the first time i learned about this was quite surprising. so wasn't sure if it was new to you ๐Ÿ™‚

jumar04:09:25

But thanks @hiredman for the pubsub idea. It's not something I can use for this particular app/problem but it's a good alternative to keep in mind when encountering similar problems in the future ๐Ÿ™‚.

deleted05:09:26

if you can assume a single jvm then you could make a vec of mutexes and then mod the account id by the vec length to select which to use but you'd need to be careful to choose a good vec size to lessen false collisions

jumar08:09:36

< if you can assume a single jvm I was about to say that I can but you got me thinking and realized that I can't - it's really a distributed system and it could happen that the concurrent updates are done on separate machines...

jumar08:09:53

But thanks for mentioning Postgres' advisory locks. While I can't use them it's an interesting thing to know about.

Benjamin12:09:37

(apply str (take 3 "foooo")) is there a better way to cut a string?

GGfpc14:09:23

Do you def variables so that you can evaluate forms that depend on them in the REPL? If so, how do you avoid having to re-define those variables every time you want to evaluate something in a different namespace that depends upon the same thing?

p-himik14:09:54

> evaluate something in a different namespace defs can also be in a different namespace. Or you can use let.

borkdude14:09:58

@U016XBH746B The problem is usually that optimally you shouldn't type directly in a REPL, but evaluate from your editor which already sets the right namespace

borkdude14:09:37

Another approach is to always stay in the user ns and control everything from there, but this is really a primitive way of working

borkdude14:09:18

To develop from a buffer in your editor, you can make a comment form

borkdude14:09:21

and then put expressions in there

borkdude14:09:30

This is called a "Rich comment form"

GGfpc14:09:59

Yes, I see what you mean, but imagine I have a map that represents a car for example, and functions take in a variable named car as parameter. I usually will define the car variable in the repl or evaluate it from a comment form so that I can evaluate parts of the body from the editor If I then have to evaluate a form in another namespace that also is within a function that takes in a car variable, the original variable I defined isn't found. (Obviously because I switched to that new namespace)

borkdude14:09:37

yeah in this case move the test data to a common ns probably

GGfpc14:09:50

Got it, thanks!

borkdude14:09:20

so basically what @U2FRKM4TW said, thanks Eugene ;)

๐Ÿ‘ 2
lgessler17:09:15

for my most heavily used repl things I def them in user, otherwise I use rich comments. so, basically what everyone else said above

Ed21:09:59

If you use cider, you can use C-x C-v C-b (I think ... Currently typing on my phone) which will prompt you for some bindings before evaling the previous form ... Meaning you don't have to def anything and it'll remember the bindings in your history when you switch namespaces

respatialized18:09:34

When launching Clojure from the command line, is there a way to direct it to use an existing Clojure process (say, that of a running REPL) if one is available?

hiredman19:09:33

You can pass the special argument to clojure that causes it to start a repl on a given port, then use netcat or whatever to connect to it

borkdude20:09:58

@afoltzm Tangentially related, I did an experiment with running a Clojure pREPL server, which is at most launched once and only when needed, and executing Clojure expressions in it from multiple babashka tasks. Perhaps as a bit of inspiration: https://gist.github.com/borkdude/5f3019d6666953f3b8f533a70eb067d2

โœ… 1
respatialized21:09:59

@U04V15CAJ Thanks! The motivating use case is similar for me: I would like to reduce the barrier to using git hooks in my project by taking advantage of an already running Clojure process if it's present (and launching it if it's not).

Bobbi Towers22:09:12

I'm reading the tests for math.combinatorics, and I find definitions like this:

(def factorial-numbers @#'clojure.math.combinatorics/factorial-numbers)

(deftest test-factorial-numbers
  (are [x y] (= (factorial-numbers x) y)
       463 [3 4 1 0 1 0]
       0 []
       1 [1 0]
       2 [1 0 0]))
What is happening here, and why not just use the factorial-numbers function directly?

dpsutton22:09:04

Factorial numbers is private

Bobbi Towers22:09:47

Ah, so derefing it makes it accessible. Thanks :)

๐Ÿ‘ 1
dgb2322:09:19

user=> (defn foo [] 1)
#'user/foo
user=> (#'foo)
1
user=> (@#'foo)
1
user=> (def bar #'foo)
#'user/bar
user=> (bar)
1

Joshua Suskalo22:09:47

Where would be a good place to start learning about Classloaders? I've read the article that got posted here a while back about how the dynamic classloader works in Clojure, but I'm looking for general classloader knowledge. I'm trying to figure out how to ensure that the classloaders are set well for when I use System/loadLibrary and the new Panama jdk.incubator.foreign.SymbolLookup in order to create my ffi library.