Fork me on GitHub
#core-async
<
2018-08-20
>
bjr14:08:05

@loganpowell Do these execute concurrently?

bjr14:08:33

Blocking/parking take operations coordinate processes. Each pair of output->input processes can coordinate using a channel and blocking/parking take operations. But when execution order is important one must ask whether you need 3 concurrent processes instead of 1.

Logan Powell14:08:41

Hey @bjr! they need to be executed in a specific order

Logan Powell14:08:39

I'm not sure how to answer the concurrency question, but each step needs the value returned from the previous one

Logan Powell14:08:40

I'm still a little fuzzy on the difference between concurrency and parallelism

bjr14:08:10

ignore the difference between concurrency and parallelism for now

bjr14:08:26

what is stopping this from running as a series of function calls?

Logan Powell14:08:53

I've been debugging this for a week and what's currently happening is that all the first step operations are happening, then all the second instead of each process running the three steps then looping

Logan Powell14:08:22

I think it's due to me having nested go blocks trying to conduct the parking puts/takes

bjr14:08:09

if you want to coordinate the completion of the go block with its calling thread you need to park/block on the channel returned by the go block

bjr14:08:23

however, if you’re doing that immediately, why use a go block?

Logan Powell14:08:00

Sec, let me put together a quick example

Logan Powell14:08:01

So, for example, what I'm doing is something like this.

Logan Powell14:08:49

The wierd thing is that this works if I use sufficient (<! (timeout...))s between each function to enable the internal async processes enough time to complete. Otherwise, all of fn1-go->! functions run, then all of fn2-go->!

bjr14:08:15

you’re creating go blocks within go blocks?

bjr14:08:08

might be fine…i’m returning from some time off and am a little fuzzy, can’t remember whether that’s a problem…but why?

Logan Powell14:08:43

In one case, I'm using a library that transforms a promise (returned by a js library) into a <? take'able, but only in a go. In others, I'm doing a lot of node fs interop, which can only handle one operation at a time (Node)

Logan Powell14:08:04

Perhaps I'm over thinking it

Logan Powell14:08:56

I'm currently thinking I should just use put! (async) for everything except that <? and pipeline-async to coordinate the intermediate processes

Logan Powell14:08:38

To give you a sense of what I'm doing, I'm trying to convert about 5000 Census shapefiles to geojson

Logan Powell14:08:52

(for the OSS community)

bjr14:08:23

so you have A -> B -> C where each task runs for each file

bjr14:08:54

and you want A to be executing for file2 as soon as it completes for file1, and meanwhile, B begins on file1?

Logan Powell14:08:28

I want all three processes (A->B->C) to happen for file1 before staring the process for file2

bjr14:08:41

then this is 1 process

Logan Powell14:08:47

but, you're right, what you said is what is actually happening

bjr14:08:48

assuming i understand correctly

bjr14:08:04

it sounds like you don’t need core.async

Logan Powell14:08:14

You might be right

Logan Powell14:08:29

Maybe use futures or something?

bjr14:08:49

yeah, if you don’t want to block the calling thread then a future may make sense

bjr14:08:03

but…why not block the calling thread?

Logan Powell14:08:31

How would I do that?

Logan Powell15:08:18

I thought blocking the calling thread is what I needed to do to ensure the order

Logan Powell15:08:48

Obv, I'm still new

bjr15:08:09

(defn do-everything []
  (create-folder-structure)
  (doseq [f (file-seq ...)]
    (save-geojson (zip->geojson f))))

bjr15:08:37

is there a reason why executing (do-everything) won’t work?

Logan Powell15:08:38

So, one thing is that I have that js library shpjs, which - unfortunately - returns a js/Promise

bjr15:08:07

deref the promise

Logan Powell15:08:17

I can do that?

Logan Powell15:08:46

I've seen something like this:

(defn test1
  "Array approach, flat chain: thread multiple values through promise chain by using Promise.all"
  []
  (-> (resolve 5)
      (.then square-step)
      (.then sum-step)
      (.then prn)))

bjr15:08:45

yeah… I think that’s a problem isolated to how you call save-geojson with the result of zip->geojson

bjr15:08:24

the key being you want to make sure save-geojson is executed before the next file is loaded

bjr15:08:39

core.async only complicates this

bjr15:08:54

because it’s only one process

bjr15:08:01

the promises stuff is probably there because JS and webworkers — something i’m not too familiar with

bjr15:08:41

but if you’re running as a program in Node it seems like that’s unnecessary

bjr15:08:21

i suppose my thought here is get the data out of the promise. you don’t need callbacks so don’t use them.

Logan Powell15:08:06

ah, ok, so how do I coordinate a promise that's nested between other async functions?

bjr15:08:21

they’re not async

bjr15:08:27

they’re just functions

Logan Powell15:08:28

(e.g., fs/read -> promise -> fs/write)

bjr15:08:40

i think that’s not a topic for this channel

bjr15:08:00

and out of my depth 🙂

Logan Powell15:08:13

let me keep digging. Thank you for the help! I have some new thoughts on an approach 🙂

mccraigmccraig15:08:04

@loganpowell since you are working with promises you might find life easier if you use promesa alet or async rather than converting your promises to chans repeatedly - http://funcool.github.io/promesa/latest/#async-await-syntax

mccraigmccraig15:08:45

in js-land i tend to use core.async for operations on streams of things and promesa for promise-based things

Logan Powell15:08:08

@mccraigmccraig Yessir, that's what I'm thinking

Logan Powell15:08:06

I just don't like promises, hence my hesitation... it's like core.async. Once you introduce promises, those promises dominate the I/O flavor of the code, but - unlike core.async they're not as composable 😞

Logan Powell15:08:17

I'm thinking about how to modulate the promise code into something that - ultimately - doesn't return a promise, atm (perhaps the interface is a [val chan], but internally uses promises)

mccraigmccraig15:08:58

promises are perfectly composable! and if you are doing any js i/o then you are already doing promises, so my view is you might as well go with the flow

Logan Powell15:08:20

It's a good point

mccraigmccraig15:08:35

(which doesn't mean that you shouldn't make as much of your code as possible pure and promise-free)