Fork me on GitHub
#beginners
<
2016-06-23
>
joshkh10:06:10

i'm having a bit of trouble wrapping my head around core.asyc in the browser. specifically, i have a handful of channels that were returned from HTTP requests and i want to get their results in the order that I made the requests, not in the order they were returned. if i manually bind each value from each channel individually then it works. but if i map <! over the channels i just get back a list of channels. can anyone explain what i'm doing wrong?

(let [response-channels (map #(http/get "" {:with-credentials? false}) (range 3))]
  ; Response is now three channels generated by http/get:
  ;(#object[cljs.core.async.impl.channels.ManyToManyChannel]
  ; #object[cljs.core.async.impl.channels.ManyToManyChannel]
  ; #object[cljs.core.async.impl.channels.ManyToManyChannel])

  ; If I want the results back in the guaranteed order that I made them, I can do this:
  (go (let [response1 (<! (nth response-channels 0))
            response2 (<! (nth response-channels 1))
            response3 (<! (nth response-channels 2))]
        (println "this works fine" response1 response2 response3)))


  ; But if I try to map <! over the channels instead, I just get back a list of channels
  (let [responses (into [] (map (fn [c] (go (<! c))) response-channels))]
    (println "responses" responses)
    ; This is still just a vec of many-to-many channels
    ; [#object[cljs.core.async.impl.channels.ManyToManyChannel]
    ; #object[cljs.core.async.impl.channels.ManyToManyChannel]
    ; #object[cljs.core.async.impl.channels.ManyToManyChannel]]
    )
  )

joshkh10:06:38

and this returns an error that i'm using <! outside of a go block:

(go (into [] (map <! response-channels)))

joshkh10:06:03

i was tempted to use core.async/merge and core.async/reduce over the channels which works but there's no guaranteed order which i need

joshkh12:06:56

scratch that, solved by doseq!

joshkh14:06:56

okay, retracting the resolution. doseq is used for side effects and does not return results in a data structure, therefore i can't return them from the function. so the problem remains.

meowy14:06:02

@rodeorockstar: Simple, kinda hackish solution I just came up with: Just tag the response with something that lets you determine original order.

meowy14:06:57

Actually, rereading your problem... hm.

joshkh14:06:41

thanks @meowy, that's definitely an option! i've done just that in the past for simple requests. in this particular case i'm running complex queries through a web service, so i think the "key" to matching the results back to their original requests would be the entire input to the request itself. that doesn't feel as efficient as it could be.

joshkh14:06:17

but i suppose (=) is cheap

meowy14:06:24

I suppose that could work.

hrathod14:06:57

an alternative to doseq could be for, perhaps?

meowy14:06:40

I'm not sure if this is applicable, but... if you have a "batch job" of HTTP requests, you could apply a similar strategy as in this example, using alts!.

meowy14:06:36

Note that a response must be returned by all channels that you use like this, because otherwise, this will deadlock.

donaldball14:06:01

You’re not mapping <! over the channels, you’re mapping (go (<! …)) over the channels. The go macro returns a channel.

joshkh14:06:46

okay that's making some sense

joshkh14:06:06

and thanks meowy. i'll take a look at the walkthrough

joshkh14:06:24

@hrathod i tried for but ended up with the same problem: a list of channels rather than their results, presumably because of the go block inside as donaldball just mentioned

donaldball14:06:21

You probably want something along the lines of (go (map <! response-channels)) and I guess take! from that?

joshkh14:06:04

oh, maybe i could merge the resulting channels from that and then reduce over them. hopefully they'd be in the correct order

donaldball14:06:46

There will only be one resulting channel from that, and it will contain the first value put on each of the response channels in order

meowy14:06:47

What I'm worried about with that approach is that channel 2 might return a value before channel 1 does, if you actually process them in order.

meowy14:06:45

But then... it probably doesn't matter, anyway, because there's not really a time difference, is there?

donaldball14:06:14

It doesn’t matter. The <! from the first channel will block until the channel receives a value or closes. The fact that there are values pending on the other channels is irrelevant because no one is trying to take from them yet.

meowy14:06:47

You need the results of all the channels, anyway, so it doesn't matter whether you get channel 2 and then 1 (because 2 was faster), or just wait on 1 and then get the value from 2 immediately (because it was done already).

joshkh14:06:48

interesting. i'm testing now.

joshkh14:06:33

hmm, i can't seem to avoid Uncaught Error: <! used not in (go ...) block

joshkh14:06:48

@niwinz thanks. the problem with merge is that the ordered isn't guaranteed.

niwinz14:06:58

you need order?

niwinz14:06:05

ok wait a moment

meowy14:06:27

Guess you could tag the output then, as I suggested. Given that it's HTTP requests, you could tag them with the time of arrival, and given sufficient precision of the timestamp, there shouldn't be an issue sorting them.

joshkh14:06:37

i shoudn't say that's the problem with merge, i should say that's the problem with me 😉

meowy14:06:08

That, or you associate an input with some sort of tag, and get the right things for the right inputs after the merge.

joshkh14:06:57

actually yeah, i could just return some sort of index

niwinz14:06:03

the concat described on the gist should work

niwinz14:06:23

is like merge but works sequentially

rauh14:06:53

@rodeorockstar: If you wait for all channels to have a result anyways, then you can just use core.async/map

joshkh14:06:23

would i wait via blocking?

joshkh14:06:03

(i'm trying the gist now, niwinz)

joshkh14:06:17

Uncaught Error: No protocol method ReadPort.take! defined for type cljs.core/LazySeq: (#object[cljs.core.async.impl.channels.ManyToManyChannel] #object[cljs.core.async.impl.channels.ManyToManyChannel] #object[cljs.core.async.impl.channels.ManyToManyChannel])

niwinz14:06:29

seems like you are passing a wrong data to concat

joshkh14:06:46

ah yeah, oops.

niwinz14:06:46

concat expects call similar to (concat ch1 ch2 ch3)

niwinz14:06:56

not (concat [ch1 ch2 ch3])

niwinz14:06:08

if you have a vector, just use apply

niwinz14:06:16

(apply concat [ch1 ch2 ch3])

joshkh14:06:19

apologies - i've been staring at this stuff for a while. can't quite see the forest through the trees. 😉

joshkh14:06:02

hmm, close, but i'm only getting back the value from one channel instead of all three:

(defn concater [& chans]
  (let [out (chan)]
    (go-loop [[ch & more] chans]
             (when ch
               (let [val (<! ch)]
                 (if (nil? val)
                   (recur more)
                   (do (>! out val)
                       (recur chans))))))
    out))

(let [response-channels (map #(http/get "" {:with-credentials? false}) (range 3))]
  (go (println "results:" (<! (apply concater response-channels))))
  )
results: {:status 200, :success true, :body {:time 02:43:11 PM, :milliseconds_since_epoch 1466692991958, :date 06-23-2016}, :headers {content-type application/json; charset=ISO-8859-1, cache-control private}, :trace-redirects [ ], :error-code :no-error, :error-text }

niwinz14:06:11

yes, becuase the concater returns a channel of results

niwinz14:06:19

if you only take 1 only

niwinz14:06:25

if you want a coll of results

niwinz14:06:11

try to put the concater return value in (<! (a/into [] (apply concater response-channels)))

niwinz14:06:43

a is a namespace alias for clojure.core.async

joshkh14:06:38

okay, just tried and i get back nothing at all this time.

(let [response-channels (map #(http/get "" {:with-credentials? false}) (range 3))]
  (go (println "results:" (<! (a/into [] (apply concater response-channels))))))

niwinz14:06:26

oh yes, this is because the concater has a bug 😞

niwinz14:06:47

it does not closes the channel, sorry for point you to a badly implemented concat

joshkh14:06:06

that's fine! it's a good start. i was about to ask about closing channels anyway.

joshkh14:06:15

otherwise i'll never know when i'm finished

niwinz14:06:48

(defn concater 
  [& chans]
  (let [out (chan)]
    (go-loop [[ch & more] chans]
      (if ch
        (let [val (<! ch)]
          (if (nil? val)
            (recur more)
            (do (>! out val)
                (recur chans))))
        (close! out)))
    out))

niwinz14:06:56

try this concater,

joshkh14:06:10

that did it!

joshkh14:06:31

thanks a lot, niwinz (and meowy and everyone else)

zane20:06:19

What's the idiomatic way in Java 8 to make #insts if you have year, month, day as numbers?

zane20:06:55

(java.util.Date. year month day) is deprecated.

zane20:06:19

And you could

(java.util.Date/from
 (.. (java.time.LocalDate/of year month day)
     (atStartOfDay)
     (toInstant java.time.ZoneOffset/UTC)))
but good lord is that verbose.

zane20:06:56

(I know about JodaTime.)

donaldball20:06:07

I’ve been using java-time recently

donaldball20:06:33

It aims to be the wrapper for java8’s java.time library

danmidwood23:06:25

@zane Oh, many minds have been lost trying to work with java dates and times. The first "official" replacement was the Calendar classes, they're (IMO, obv) a bunch of crap and worth avoiding. The sane options now are joda, clj-time (which uses joda internally) and the newly added java.time which is heavily based on Joda. If you have to work with j.u.Date or don't want to use joda/clj-time then I'd really recommend just using that same deprecated constructor and ignoring the deprecation warnings from it