Fork me on GitHub

@loganpowell Do these execute concurrently?


Blocking/parking take operations coordinate processes. Each pair of output->input processes can coordinate using a channel and blocking/parking take operations. But when execution order is important one must ask whether you need 3 concurrent processes instead of 1.

Logan Powell14:08:41

Hey @bjr! they need to be executed in a specific order

Logan Powell14:08:39

I'm not sure how to answer the concurrency question, but each step needs the value returned from the previous one

Logan Powell14:08:40

I'm still a little fuzzy on the difference between concurrency and parallelism


ignore the difference between concurrency and parallelism for now


what is stopping this from running as a series of function calls?

Logan Powell14:08:53

I've been debugging this for a week and what's currently happening is that all the first step operations are happening, then all the second instead of each process running the three steps then looping

Logan Powell14:08:22

I think it's due to me having nested go blocks trying to conduct the parking puts/takes


if you want to coordinate the completion of the go block with its calling thread you need to park/block on the channel returned by the go block


however, if you’re doing that immediately, why use a go block?

Logan Powell14:08:00

Sec, let me put together a quick example

Logan Powell14:08:01

So, for example, what I'm doing is something like this.

Logan Powell14:08:49

The wierd thing is that this works if I use sufficient (<! (timeout...))s between each function to enable the internal async processes enough time to complete. Otherwise, all of fn1-go->! functions run, then all of fn2-go->!


you’re creating go blocks within go blocks?


might be fine…i’m returning from some time off and am a little fuzzy, can’t remember whether that’s a problem…but why?

Logan Powell14:08:43

In one case, I'm using a library that transforms a promise (returned by a js library) into a <? take'able, but only in a go. In others, I'm doing a lot of node fs interop, which can only handle one operation at a time (Node)

Logan Powell14:08:04

Perhaps I'm over thinking it

Logan Powell14:08:56

I'm currently thinking I should just use put! (async) for everything except that <? and pipeline-async to coordinate the intermediate processes

Logan Powell14:08:38

To give you a sense of what I'm doing, I'm trying to convert about 5000 Census shapefiles to geojson

Logan Powell14:08:52

(for the OSS community)


so you have A -> B -> C where each task runs for each file


and you want A to be executing for file2 as soon as it completes for file1, and meanwhile, B begins on file1?

Logan Powell14:08:28

I want all three processes (A->B->C) to happen for file1 before staring the process for file2


then this is 1 process

Logan Powell14:08:47

but, you're right, what you said is what is actually happening


assuming i understand correctly


it sounds like you don’t need core.async

Logan Powell14:08:14

You might be right

Logan Powell14:08:29

Maybe use futures or something?


yeah, if you don’t want to block the calling thread then a future may make sense


but…why not block the calling thread?

Logan Powell14:08:31

How would I do that?

Logan Powell15:08:18

I thought blocking the calling thread is what I needed to do to ensure the order

Logan Powell15:08:48

Obv, I'm still new


(defn do-everything []
  (doseq [f (file-seq ...)]
    (save-geojson (zip->geojson f))))


is there a reason why executing (do-everything) won’t work?

Logan Powell15:08:38

So, one thing is that I have that js library shpjs, which - unfortunately - returns a js/Promise


deref the promise

Logan Powell15:08:17

I can do that?

Logan Powell15:08:46

I've seen something like this:

(defn test1
  "Array approach, flat chain: thread multiple values through promise chain by using Promise.all"
  (-> (resolve 5)
      (.then square-step)
      (.then sum-step)
      (.then prn)))


yeah… I think that’s a problem isolated to how you call save-geojson with the result of zip->geojson


the key being you want to make sure save-geojson is executed before the next file is loaded


core.async only complicates this


because it’s only one process


the promises stuff is probably there because JS and webworkers — something i’m not too familiar with


but if you’re running as a program in Node it seems like that’s unnecessary


i suppose my thought here is get the data out of the promise. you don’t need callbacks so don’t use them.

Logan Powell15:08:06

ah, ok, so how do I coordinate a promise that's nested between other async functions?


they’re not async


they’re just functions

Logan Powell15:08:28

(e.g., fs/read -> promise -> fs/write)


i think that’s not a topic for this channel


and out of my depth 🙂

Logan Powell15:08:13

let me keep digging. Thank you for the help! I have some new thoughts on an approach 🙂


@loganpowell since you are working with promises you might find life easier if you use promesa alet or async rather than converting your promises to chans repeatedly -


in js-land i tend to use core.async for operations on streams of things and promesa for promise-based things

Logan Powell15:08:08

@mccraigmccraig Yessir, that's what I'm thinking

Logan Powell15:08:06

I just don't like promises, hence my hesitation... it's like core.async. Once you introduce promises, those promises dominate the I/O flavor of the code, but - unlike core.async they're not as composable 😞

Logan Powell15:08:17

I'm thinking about how to modulate the promise code into something that - ultimately - doesn't return a promise, atm (perhaps the interface is a [val chan], but internally uses promises)


promises are perfectly composable! and if you are doing any js i/o then you are already doing promises, so my view is you might as well go with the flow

Logan Powell15:08:20

It's a good point


(which doesn't mean that you shouldn't make as much of your code as possible pure and promise-free)