Fork me on GitHub
#clojure
<
2020-08-17
>
lpan01:08:02

Hi all, I am wondering if >! from a "go block" into a channel is starvation free in Clojure. For example, if I have two producers and one consumer:

(let [c (chan)]
  ;; producer 1
  (go-loop [count 1] (>! c (str "p1: " count)) (recur (inc count))

  ;; producer 2
  (go-loop [count 1] (>! c (str "p2: " count)) (recur (inc count))

  ;; only consumer
  (go-loop [] (println (<! c)) (recur)))
I am wondering if it is possible one of the producer gets starved? Thanks!

lpan02:08:27

Actually it seems put! has a FIFO order. I think we should be good :thinking_face:

Alex Miller (Clojure team)02:08:51

note that you're using an unbuffered channel (chan) so it can hold 0 items. The producers will basically both be trying to rendezvous with the consumer to transfer the value

Alex Miller (Clojure team)02:08:57

how that plays out with 2 producers is somewhat arbitrary and based on how notification happens at the java layer, there are no guarantees on the puts between the 2 producers

Alex Miller (Clojure team)03:08:29

so you might see two puts from the first, then some from the second, etc (but they should be in order for each producer)

lpan03:08:56

I came across with this article and apparently Clojure will place blocked puts and takes in a FIFO queue, even for unbuffered channels :thinking_face: https://medium.com/swlh/asynchronous-clojure-a59fa0f9bda0

Alex Miller (Clojure team)03:08:13

I think you're missing some of the details - that will happen with put! and take!, but you're using >! and <! which are parking ops and they will simply park (wait, without blocking a thread) until the op can complete with a corresponding take or put

πŸ‘ 3
lpan04:08:12

Thanks Alex! I think it makes sense now πŸ™‚

sandbags09:08:54

Brains a little slow this morning. I have a map along these lines {#{1 2 3},"a",#{4,5,6},"b",…} and am thinking about f such that (f 2) => a , (f 6) => b and so on. Maybe this isn't the right structure? I have something ugly using filter but I feel like there's a simpler way (that isn't unrolling the sets into the map)

p-himik09:08:21

Why do you want to avoid unrolling the sets?

sandbags09:08:50

They are a bit larger than the example

sandbags09:08:20

(Yes I realise it would probably have taken less time than framing the question in the first place, but then I wouldn't learn anything :))

sandbags09:08:57

I guess I should be less lazy

p-himik09:08:33

:) "Larger than the example" should not affect anything because sets are already basically maps, so by wrapping multiple keys in sets you're not saving anything.

sandbags09:08:50

Well only typing time

sandbags09:08:18

Actually I do have another approach using (vec (concat (repeat x :a) (repeat y :b)))

sandbags09:08:33

but I guess I will just type it out

p-himik09:08:33

Ah, but you can just unroll the sets with code, no? No need to unroll them manually, so you can still save on typing.

jaihindhreddy09:08:23

Yeah, I agree with p-himik here, you can do it with code:

(defn unroll-keys [m]
  (into {}
        (mapcat (fn [[ks v]] (map #(vector % v) ks)))
        m))
Of course, this is UB when the keys aren't disjoint.

πŸ‘ 3
sandbags11:08:45

Thanks, I hadn't thought of unrolling with code. Good idea although I overcame my laziness in the meantime πŸ™‚

πŸ‘ 6
kirill.salykin11:08:02

Hi I recall there was some discussion over clojure.cache flaws for instance, is it possible that cache-get-through may fail under some intensive load? endpoint should cache sql queries, but I see about 27M select statements in the db log for last hour (which means cache doesnt work) do you have any ideas maybe? thanks

jeroenvandijk11:08:24

What do you use for state management? An atom maybe?

kirill.salykin11:08:40

AFAIK, no locks

jeroenvandijk11:08:59

in that case when there is a lot of load, meaning multiple thread trying to swap! the cache, one will succeed (eventually), but it will take several tries (queries)

πŸ‘ 3
jeroenvandijk11:08:14

So probably good to wrap it with a locking πŸ™‚

kirill.salykin11:08:21

so it is kinda multiplies a load even

πŸ˜… 3
kirill.salykin11:08:26

thanks a lot!

πŸ‘ 3
jeroenvandijk11:08:39

@U1V7K1YTZ The locking might not be necessary if you use delay (swap! C2 cache/through-cache :d (fn [] (delay (do-query))))

πŸ’₯ 3
kirill.salykin11:08:12

yes, I think Chris mentioned something like this

mloughlin14:08:44

Is it preferable to pass around a StringBuilder/cl-format rather than several nested calls to (str)?

noisesmith14:08:41

it's the "tree falls in a forest" type thing- it makes sense if the scope of the builder is limited to a lexical block where the pattern of usage (append only) is safe, you probably don't want to try to use a StringBuilder that escapes scope

noisesmith14:08:34

and yes, nested string calls are an antipattern (though apply str or string/join will do the stringbuilder stuff for you already)

mloughlin15:08:08

I'm sure it rarely makes the difference performance wise, unless you're rolling your own serialization lib or something. I have a sick fascination with cl-format. Why should I write all the tedious imperative generation of intermediate representations when I have this (insane, but) perfect DSL to do it for me! parrot

noisesmith15:08:38

the question is whether the string ops are your bottleneck (and for many domains that can truly be the case - eg. web servers with back end rendering, having a more efficient template tool makes a big difference)

noisesmith15:08:57

and yes, cl-format is remarkable

mloughlin15:08:55

cl-format makes me wonder if there's a library out there that implements LOOP using transients that could be used in hot loop situations

noisesmith15:08:23

haha, LOOP is insane

noisesmith15:08:31

I'm working on a lib that does low level packed contiguous data structures, that might be a cool feature if I implemented a "hold-my-beer" namespace

πŸ‘€ 3
mloughlin15:08:38

the LOOP periodic table from Land of Lisp might be my favourite programming book figure of all time https://blog.idorobots.org/media/lolreview/3.png

πŸ†’ 3
mloughlin15:08:49

LOOP is so audaciously un-apologetic about what it does

Eduardo Mata15:08:07

Does anyone know who to check for outdated dependencies using Boot?

noisesmith15:08:55

inb4 "boot is an outdated dependency" πŸ˜„

πŸ˜‚ 9
noisesmith15:08:58

@contact904 this looks like it wants to do what you need https://github.com/martinklepsch/boot-deps though I can't personally vouch for it

noisesmith15:08:49

and in all seriousness, if you have the flexibility to do so, I'd consider a boot -> deps.edn switch an improvement / future proofing, not much is happening in the boot ecosystem lately

kenny15:08:19

@contact904 I agree with @noisesmith on switching but if you must stay, that ability is built in to boot's showhttps://github.com/boot-clj/boot/blob/master/boot/core/src/boot/task/built_in.clj#L255.

πŸ’― 6
kenny15:08:59

This oddly specific knowledge comes from having used the lib @noisesmith linked a few years ago haha: https://github.com/martinklepsch/boot-deps/pull/12#issuecomment-241523730

borkdude15:08:28

@contact904 boot -d boot-deps ancient

βœ”οΈ 3
Eduardo Mata15:08:48

Thank you so much!

sparkofreason17:08:23

MacOS just updated, and now clj gives this error. All the more weird because clojure seems to work fine. clojure: error: Cannot execute clojure: No such file or directory

πŸ‘€ 3
Gene17:08:31

What do you see if you check the path:

Gene17:08:33

ls -al /usr/local/bin/clj

Gene17:08:20

This is what I get: */usr/local/bin/clj*@ -> ../Cellar/clojure/1.10.1.561/bin/clj

Gene17:08:36

It is pointing to a homebrew location.

lmergen17:08:37

I typically use strace in these situations; may seem like overkill, but it’s typically very obvious which fopen() is causing the error

sparkofreason18:08:40

lrwxr-xr-x 1 dave.dixon admin 36 Aug 17 10:41 /usr/local/bin/clj -> ../Cellar/clojure/1.10.1.561/bin/clj

borkdude18:08:41

Maybe just re-install Clojure?

sparkofreason23:08:23

MacOS screwed something up. Having a similar issue with sbt, works from the command line, but not from build scripts etc.

sparkofreason23:08:02

I did try a reinstall of the Clojure CLI tools, though without uninstall, same result.

lmergen08:08:47

maybe an issue with java path, or JAVA_HOME ?

sparkofreason11:08:01

The clojure command runs fine. It's just clj that fails.

borkdude11:08:32

Maybe something to do with readline ?

borkdude11:08:51

or permissions

sparkofreason17:08:59

If I put the full path to clojure in the rlwrap command used by clj, it works. Scratching my head as to what would have changed in MacOS to cause that.

sparkofreason17:08:31

Obviously something like it is no longer picking up the $PATH value from the shell environment.

borkdude20:08:03

@U066LQXPZ Are you aware that Catalina is using zsh is the default shell now instead of bash?

sparkofreason20:08:48

Yes. Though I'm using fish as my shell.

GGfpc20:08:30

Can someone recommend an oauth2 client library for clojure?

jcf22:08:44

@U016XBH746B I wrote this library a few years back, and used it in production with a few services. https://github.com/jcf/oauth-two The use of Schema is probably no longer desirable and could be removed easily enough.

jcf22:08:18

I would recommend avoiding coupling the OAuth 2.0 protocol with any HTTP client/server, which is the approach I took in this library. Doing so worked well and allowed us to use any Ring-compatible library with custom logging/telemetry.