Fork me on GitHub
#beginners
<
2023-03-10
>
keychera10:03:52

I am currently writing a conveyor like program with core.async that goes like this: 1. it has one oven , an infinite loop process on another thread that do a blocking take on a channel called orders , from that channel, it will receive a function that return value, it runs that function and put that value in an atomic list called tray 2. it has a function put-recipe-in-the-oven that put a recipe in orders channel, and recipe is a map that is kinda like this =>

{:who "someone" :recipe (fn [] (Thread/sleep 1000) (str "this is cake")) }
I feel like this is a standard stuff built on top of core.async and I’ve already made a working code https://gist.github.com/keychera/02c6a4aeb5476728bc47ea068a5719a5 that can be played in a repl. But the question is that I want to monitor what’s going on at a given time, which, in a simple implementation, it would be a function (defn what-is-going-on []) that returns a map like this
{:remaining-orders [{:who "A" :recipe fn} {:who "B" :recipe fn}]
 :in-the-oven [{:who "Z" :recipe fn}]}
I have been reading here and there about core.async, saying that it is hard to see inside channels, and maybe this idea would be better using some queue instead. So, while I’m currently still hammocking and repling some ideas, I’m also concurrently spawning a question here: Is this a common problem that some libraries out there have already solution for this? Did I actually approach this problem correctly?

rolt11:03:45

not sure if it's a "public" api, but you can access the buffer of a chan: (.buf my-chan) and you can see the content of a buffer: (.buf my-buffer) returns a linked list (use (count buffer) if you just want the number of elements)

daveliepmann11:03:36

If java.util.concurrent can handle your use case, then it's probably the right library. It's quite easy to inspect what's going on inside a https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/LinkedBlockingQueue.html, for instance.

daveliepmann11:03:19

Rich goes into this in "The Language of the System": > one of the things in Clojure maybe I didn't make clear enough because I didn't need to wrap them is that the queues in java.util.concurrent are awesome. If you're not using them as part of your system designs internally, you're missing out. https://github.com/matthiasn/talk-transcripts/blob/9f33e07ac392106bccc6206d5d69efe3380c306a/Hickey_Rich/LanguageSystem.md

👀 1
keychera11:03:08

Ohh, thanks for the pointer, I was considering queue because it was mentioned in a discussion I read regarding core.async but was not exactly sure which queue and I lost where that discussion is, I’m gonna try some java interop as well then

👍 1
1
mathpunk17:03:54

(-> confusing-thing type javadoc) is great

mathpunk17:03:38

and then you put it inside a map with a thousand items

mathpunk17:03:46

(by ‘you’ i mean ‘me’)

Jakub Šťastný19:03:24

I have ns jakub-stastny.et.runner with -main fn in it. Works just fine in Clojure, but now I want to compile it to an uberjar. Using clojure.tools.build.api's uber fn: I have :main '.runner.main, but I'm getting: Error: Could not find or load main class .runner.main What's wrong? I'm not sure what the class name should be, clearly I got it wrong, but I'm just not sure what it should be and the doc isn't very explicit.

seancorfield19:03:05

Remove .main

🙏 1
Jakub Šťastný19:03:50

Awesome, that did it 😎 Cheers Sean!

seancorfield19:03:21

It's the ns name, not a class name.

Jakub Šťastný19:03:42

:main - main class symbol
Is what confused me.

seancorfield19:03:09

It can be a class name if you're building something that gen-class's a different class name. Or building a mixed Java / Clojure app that has a have main class...

dgb2322:03:32

Something that I don't understand is the following: I use destructuring a lot. (Partly because I'm used to JS and there you don't really have good way of writing functional code without destructuring and spreading.) Specifically with reduce it comes up regularly, or with a reducing function, where the accumulated value holds some state that I throw away after the iteration. So I use a small vector as a tuple so to speak. I unpack it at every call, then put it back together in the return. Like so:

(reduce (fn [[x y] _] [x y]) coll)
Now my questions: - I assume unpacking the vector has the same characteristics as accessing its fields. Correct? - When I put the values back together, then I allocate a new vector each time? - Is any part of the Clojure stack aware what I'm doing? The compiler? The JIT? - How much different is this to update the vector instead? In what way?

hiredman22:03:44

user=> (clojure.pprint/pprint (clojure.walk/macroexpand-all '(fn [[a b]] [a b])))
(fn*
 ([p__164]
  (let*
   [vec__165
    p__164
    a
    (clojure.core/nth vec__165 0 nil)
    b
    (clojure.core/nth vec__165 1 nil)]
   [a b])))
nil
user=>

hiredman22:03:07

you can see that destructuring expands to using nth to get elements by index

👍 1
hiredman22:03:34

the clojure compiler certainly isn't aware. the jit is unlikely to either

hiredman22:03:08

depending on what you want to do with the awareness you would have to prove that the tuple is used in a linear fashion and never escapes (escape analysis)

🙏 1
hiredman22:03:40

the jit does do some of that I believe, but I don't think it is considered to do a very good job (likely depends who you ask, and some of this might just be the jvm model where everything is a reference so you need a lot of escape analysis to make a difference)

hiredman22:03:10

the jvm does have world class garbage collectors (which includes fast memory allocation) so it would not surprise me if it was faster to allocate a new small vector then it is to update one (hard to say for sure)

👀 1
ghadi22:03:33

Good questions.

dgb2322:03:46

https://en.wikipedia.org/wiki/Escape_analysis Oh interesting! That's related to the lifetime of an object. I know the concept from dabbling in Rust.

hiredman22:03:18

this reminds me of a video I saw earlier, but that might have wider interest so I'll drop that over in #C03RZGPG3

👍 1
ghadi22:03:37

SROA is another term of art: scalar replacement of aggregates

👀 1
dgb2322:03:06

The introduction in this paper explains it nicely, and it is exactly what I had in mind when I asked "does any part of the stack know what I'm doing"! https://gcc.gnu.org/wiki/summit2010?action=AttachFile&amp;do=get&amp;target=jambor.pdf

dgb2322:03:54

So thank you for the explanations and for introducing me to some new concepts!

dgb2323:03:55

OK you sort of soothed me about allocation/GC. But here is another thing I don't understand: When I allocate the new vector (while packing) and passing it to the next iteration. Does the next iteration have to load it from memory?

Alex Miller (Clojure team)23:03:02

Making small objects and cleaning them up is basically what the jvm is optimized to do

👍 1
Alex Miller (Clojure team)23:03:49

Objects are made on the heap and a reference is passed on the stack

👍 1
dgb2323:03:14

Right I have a vague memory of something I read about that the JVM treats short lived objects differently. But that is as far as I understand it 😕

Alex Miller (Clojure team)23:03:44

That’s the whole idea of generational garbage collectors

dgb2323:03:34

Right so for the general case I can think of the above pattern and it's implications. as "not my problem"? 😄

Alex Miller (Clojure team)23:03:59

Well it’s not unusual :)

Alex Miller (Clojure team)23:03:39

If you wanted to play you could instead use mutable data instead of vectors

Alex Miller (Clojure team)23:03:24

Use an Object array or an ArrayList and mutate it during the reduce without making a new vector every time

dgb2323:03:48

What about transients?

dgb2323:03:05

I never used them but it seems like it would fit here?

hiredman23:03:42

a lot of transducers basically follow that pattern, there is some accumulator, and some other state, the transducers close over some mutable state instead of pairing it with the accumulator

Alex Miller (Clojure team)23:03:43

Depends what you’re doing in the middle

Alex Miller (Clojure team)23:03:14

Transients are not as efficiently mutable

Alex Miller (Clojure team)23:03:19

Might as well use the fastest mutable thing if you’re mutating :)

👍 1
dgb2323:03:03

It's funny you mention tranducers because the trigger for my question was that I'm trying to write a tranducer or reducing function which is a little state machine over a stream of bytes

hiredman23:03:35

the thing with transients is sometimes the way people react when they see them is "oh cool, mutable collections, so I can just with restraint do mutable things, but because it is in clojure that mutable stuff will be magically less bad"

dgb2323:03:59

It's still mutation 🙂

hiredman23:03:17

and transients are not really mutable collections

hiredman23:03:41

they are immutable collections that will let you mutate them if you ask the right way

dgb2323:03:05

Ok now I have 4 things that I want to know about more just from one question!

hiredman23:03:30

so the usage pattern, how you have to pass around the result of operations instead just updating things in place, all follows from how you work with immutable collections

hiredman23:03:26

and if you treat a transient like a mutable collection, you will have bugs that you might not notice in small scale testing

dgb2323:03:48

Ah yes I got that from the http://clojuredocs.org the warning is explained there. Thanks for the heads up though.

dgb2323:03:41

But I thought of them as nice, because the shape of the code stays similar.

dgb2323:03:46

Aha, so transients are still a tree that has the shape of a persistent collection

dgb2323:03:38

So I can think of transient and persistent! as just changing the interface so to speak.

Ben Sless09:03:25

There's another thing the JVM does to improve heap allocation locality and that's using a TLAB, Thread Local Allocation Buffer. Every allocation which does not escape it is lock free

🙏 1