Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2018-09-20

Channels

#core-async
<
2018-09-20
>
Ho0man10:09:17

Hi, I am writing an application consisting of two main parts : a (sort-of) control structure and numerous services, all interacting through channels and implemented via go blocks. The interaction of services and their life-cycle management is done by the control unit. And I wanted the thread pool for my control unit's gos be different from the services. Is there a way I can achieve that ? Or is it even a legitimate concern ? (The thing is we require that even in case of a failure on part of services, be it thread pool depletion or ..., our control over their life-cycle not to be hindered in any way)

Ho0man13:09:15

Since it wasn't resolved. wanted to check whether there is a better work around - or maybe another way to solve it

noisesmith17:09:48

if a go ever blocks a thread, that's an error

noisesmith17:09:17

use parking operations only, clojure.core.async/thread makes a new thread in an expandable pool, and returns a channel you can park on

Ho0man13:09:13

thanks nosesmith you are right but I wanted to know aside from go blocks blocking threads, in a system heavily dependent on go blocks should I be concerned about separating critical sections' thread pool from others ?

noisesmith23:09:15

nothing critical should be in a go block

noisesmith23:09:49

the number of threads used for go blocks is small, and doesn't grow, they are a method of coordination, not a utility for execution / parallelism

Ho0man16:09:56

Thanks again

Ho0man16:09:30

But can you elaborate a little more about "nothing critical should be in a go block" and "they are a method of coordination, not a utility for execution / parallelism"

Ho0man16:09:50

Aren't they supposed to be lightweight threads in a sense?

Ho0man16:09:30

Your comment kinda scared me, cause quite a few of my applications are actually a network of channels with async/go blocks doing almost all of the work in between them

noisesmith16:09:34

they are for lightweight coordination of state between tasks, anything that uses blocking IO or is CPU intensive should not be in a go block

noisesmith16:09:54

async/thread is OK for blocking IO or CPU intensive tasks

noisesmith16:09:04

it returns a channel that a go block can park on

noisesmith16:09:40

what happens is that when resource-intensive tasks are in go blocks, they can starve the channel operations, since the number of threads for go blocks are limited

Ho0man17:09:51

Thanks alot

tanzoniteblack17:09:22

At Yummly, we run a local database & dynamoDB during our unit tests, and have the following code hooked into our unit tests to throw exceptions on known blocking operations that we use to make sure we don't end up blocking a core.async thread: 

(defn core-async-thread?
  "Warn if being run on a main core.async thread."
  [f & args]
  (let [current-thread (Thread/currentThread)]
    (when (re-matches #"async-dispatch-\d+" (.getName current-thread))
      (throw (Exception. "Blocking I/O function running on core.async dispatch thread."))))
  (apply f args))

(defn apply-core-async-hookes
  "Add hooks to various blocking I/O functions to warn if they're being used on
  core.async main threads. Safe to be run multiple times as hookes are stored in
  a unique set in metadata."
  []
  (hooke/add-hook #'korma.db/do-query #'core-async-thread?)
  (hooke/add-hook #'korma.core/exec #'core-async-thread?)
  (hooke/add-hook #'taoensso.faraday/get-item #'core-async-thread?)
  (hooke/add-hook #'taoensso.faraday/put-item #'core-async-thread?)
  (hooke/add-hook #'taoensso.faraday/batch-get-item #'core-async-thread?)
  (hooke/add-hook #'taoensso.faraday/batch-write-item #'core-async-thread?))

tanzoniteblack17:09:57

just throwing that there in case it helps anyone, since it's related to the discussion

tanzoniteblack17:09:17

(require '[robert.hooke :as hooke]) for the hooke/add-hook function