Fork me on GitHub
#core-async
<
2019-12-18
>
yonatanel08:12:01

Just nitpicking a bit regarding the above, channel operations in a thread are blocking the thread instead of parking. They have different names so you can’t confuse them (<!! instead of <!). Also, an async/thread isn’t exactly normal. It’s a daemon thread so you can’t rely on the JVM to wait for it at the end of the program.

kwrooijen08:12:22

Yeah true, I find it a bit cumbersome to write go block because of the macro expansion. But that’s a whole different topic

kwrooijen08:12:34

Do you have any good resources about something like “concurrency theory”. Because I hear daemon thread, JVM thread, native thread, POSIX thread. But I wouldn’t be able to explain the differences

dharrigan09:12:06

What would the shape of that look like, i.e., to use a go block, spawn a thread and wait for the data to be emitted back into the go block for processing (if I've understood that correctly)?

kwrooijen09:12:42

(go
  ;; ... Do whatever
  (let [result (<! (thread-call query-data))]
    ;; ... Do something with `result`
    ))
Something like this I guess?

yonatanel09:12:47

That’s an example of a one-off thread. A thread might also continuously produce data into a channel, which you can read in a go-loop.

dharrigan09:12:25

(go-loop [result (<! (thread-call query-data))]
    ;; ... Do something with `result`)

roklenarcic09:12:20

why would you use go blocks if you are using full threads anyway?

kwrooijen09:12:02

Maybe you have multiple go block in your architecture, but 1 of them needs to do IO. That would make sense to me at least

kwrooijen09:12:20

On the other hand maybe that’s not how you build concurrent systems in Clojure. I’ve only written them in Erlang so I’m trying to switch my mindset

roklenarcic09:12:13

it depends on your IO boundaries’ implementation

roklenarcic09:12:49

if you’re using a standard synchronous implementation then you need to have a thread pool that handles that IO

roklenarcic09:12:19

what you do elsewhere is up to you

roklenarcic09:12:46

you can stuff the result of IO threads into channels and use go blocks to process that

roklenarcic09:12:37

or you can simply pass IO thread a callback function, that it executes with the results, that function is executed on IO thread then, but if it’s short its not a problem

roklenarcic09:12:09

and generally you have a threadpool on inbound IO anyway (web server has its own threadpool which processes request handlers)

roklenarcic09:12:05

so if you’re not careful, even if you use go block, you might end up blocking on the result of that go block, which means you await on request thread for result anyway

roklenarcic09:12:32

so to have a fully aync solution you need to have async inbound IO, so a webserver that gives you a callback to call with the results or a webserver that will take a channel as response

dharrigan10:12:32

Very useful information!

jumar11:12:11

@UG9U7TPDZ "Java Concurrency in Practice" by Brian Goetz is a good on

fmjrey12:12:31

Great discussion indeed, very timely in my case too. Overall, async/thread makes a lot of sense for long running processes (i.e. daemons) such as IO. For these you do not want to take a thread from a bounded pool (such as clojure.core/future which uses the agent pool). And since async/thread creates daemon threads, you need to either take from the channel it returns within a non-daemon thread, or have an entirely separate control mechanism, so that the JVM does not exit before your processing is done. Is that a fair summary?

roklenarcic13:12:27

IO are not daemons

fmjrey13:12:09

If you are listening to something yes, e.g. kafka consumer

fmjrey13:12:30

Or a web server waiting for new connections and handing off requests to your code

roklenarcic13:12:50

those threads are explicitly NOT daemon

roklenarcic13:12:03

otherwise JVM would exit while waiting for requests

fmjrey13:12:24

hence my earlier comment

roklenarcic13:12:44

Usually the main function starts the server (a threadpool of non-daemon threads) that wait for requests

roklenarcic13:12:06

and main function then exits

roklenarcic13:12:13

if threads were daemon then JVM would exit too

roklenarcic09:12:24

at that point you might as well use futures and a threadpool

roklenarcic09:12:14

I don’t know why you’d want to create a new thread instead of using cached thread pool

fmjrey12:12:54

Unless you really need long running daemon threads, e.g. to consume from kafka and put messages into a channel

fmjrey12:12:19

For databases a typical scenario is a connection pool with its own thread pool

kwrooijen12:12:40

Yeah, that’s what I was thinking

kwrooijen12:12:58

But I’m also reading that certain websocket libraries use core async to handle connections

kwrooijen12:12:41

If that’s true, wouldn’t it make sense to query a thread pool from a go block?

kwrooijen12:12:10

I also think that @roklenarcic and I had a bit of a miscommunication, because my intention is to use a thread pool. I was just using an example

fmjrey12:12:57

websockets.... clojure side or clojurescript side?

kwrooijen12:12:10

But I could be mistaken

fmjrey12:12:15

on cljs side there are no threads...

kwrooijen12:12:30

There is, one at least 😛

kwrooijen12:12:44

I haven’t looked into it in detail so I can’t really comment

kwrooijen12:12:24

But I’m mainly talking about the Clojure side

roklenarcic13:12:40

I see that people are confused. Daemon threads in Java are just threads that do not prevent JVM from exiting when running.

roklenarcic13:12:19

JVM will exit when all non-daemon threads stop

kwrooijen13:12:17

Ah, that’s a good distinction to know