Fork me on GitHub
#core-async
<
2022-02-10
>
JAtkins01:02:23

Hi all, I've hit a weird corner case for cljs.async. It appears that some ^:js meta is removed from let bindings in go blocks. I concede that I could be using this wrong and that's why I've not found any other messages on the internet about it though 🙂. Here's a minimal example:

(ns test-core-async
  (:require
   [cljs.core.async :refer [go]]
   [cljs.core.async.interop :refer-macros [<p!]]))

(defn test-fn
  []
  [ ; This bit of code compiles fine
   ;#_
   (let [^:js new-obj (js/Object.create (clj->js {:myFn (fn [x] (js/console.log "myFn" x))}))
         ^:js promise (js/Promise. (fn [acc rej] (acc new-obj)))
         fn-invoked (.myFn new-obj "arg")]
     [(.then promise (fn [val] (js/console.log "promise delivered" val)))
      fn-invoked])
   
   ;#_
   (go
    (let [^:js new-obj (js/Object.create (clj->js {:myFn (fn [x] (js/console.log "myFn" x))}))
          ^:js promise (js/Promise. (fn [accept _rej] (accept new-obj)))
          fn-invoked (.myFn new-obj "regular arg")
          ^:js promise-delivered (<p! promise)
          ; This bit of code complains about untagged objects, even though `promise-delivered` is tagged
          promised-fn-invoked (.myFn promise-delivered "delivered promise arg")]
      [promise-delivered
       fn-invoked
       promised-fn-invoked]))])
When the second block is un-commented, I see this error from the shadow-cljs compiler:
Cannot infer target type in expression (. inst_82893 (myFn "delivered promise arg"))
Macroexpanding the go block, I see this where the let should be: (printing meta)
(clojure.core/let
                [^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82949
                 (^{:cljs.core.async.impl.ioc-macros/global true} js*
                  "[~{}]"
                  :myFn)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82950
                 (clojure.core/let [] (fn* ([x] (js/console.log "myFn" x))))
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82951
                 (^{:cljs.core.async.impl.ioc-macros/global true} js*
                  "[~{}]"
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82950)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82952
                 (.
                  ^{:cljs.core.async.impl.ioc-macros/global true} cljs.core/PersistentHashMap
                  (fromArrays
                   ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82949
                   ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82951))
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82953
                 (^{:cljs.core.async.impl.ioc-macros/global true} clj->js
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82952)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82954
                 (^{:cljs.core.async.impl.ioc-macros/global true} js/Object.create
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82953)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82955

             ;; ^:js new-obj usage during `(fn [accept _rej] (accept new-obj))`
                 (clojure.core/let
                  [^{:js true} new-obj
                   ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82954]
                   (fn* ([accept _rej] (accept new-obj))))

                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82956
                 (^{:cljs.core.async.impl.ioc-macros/global true} new
                  ^{:cljs.core.async.impl.ioc-macros/global true} js/Promise
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82955)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82957
                 (.
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82954
                  (myFn "regular arg"))
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82958
                 (^{:cljs.core.async.impl.ioc-macros/global true} cljs.core.async.interop/p->c
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82956)
                 ^{:tag objects} state_82979
                 (cljs.core.async.impl.ioc-macros/aset-all!
                  ^{:tag objects} state_82979
                  7
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82957)]
                 (cljs.core.async.impl.ioc-helpers/take!
                  ^{:tag objects} state_82979
                  2
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82958))
;; skipping a bit to the rest of the let

(clojure.core/let
                [^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82957
                 (clojure.core/aget ^{:tag objects} state_82979 7)

              ;; this appears to be the first destructure of `promise-delivered`, but without the `^:js` meta
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82973
                 (clojure.core/aget ^{:tag objects} state_82979 2)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82974

              ;; this is the code failing compile
                 (.
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82973
                  (myFn "delivered promise arg"))
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82975
                 (.
                  ^{:cljs.core.async.impl.ioc-macros/global true} cljs.core/PersistentVector
                  -EMPTY-NODE)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82976
                 (^{:cljs.core.async.impl.ioc-macros/global true} js*
                  "[~{},~{},~{}]"
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82973
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82957
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82974)
                 ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82977
                 (^{:cljs.core.async.impl.ioc-macros/global true} new
                  ^{:cljs.core.async.impl.ioc-macros/global true} cljs.core/PersistentVector
                  nil
                  3
                  5
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82975
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82976
                  nil)
                 ^{:tag objects} state_82979
                 ^{:tag objects} state_82979]
                 (cljs.core.async.impl.ioc-helpers/return-chan
                  ^{:tag objects} state_82979
                  ^{:cljs.core.async.impl.ioc-macros/instruction true} inst_82977))
Am I doing something wrong with my original code? (I wouldn't be too surprised if I was 🙂). Thanks for any help

hiredman04:02:04

What does the :js tag do?

hiredman04:02:31

In general the go macro on the cljs side isn't as thorough about things as the clj side. It runs its own analyzer, where the clojure side uses tools.analyzer, and cljs is if anything trickier to do analysis on than clj is

hiredman04:02:51

Cljs and Js both change and add new things far more often then clj does (like this :js tag)

JAtkins13:02:08

Technically, it should be written ^js. It's a type hint for shadow-cljs to make ad-hoc externs based on your code. ^:js has the same effect though.

otfrom14:02:23

obviously it isn't that complicated and is similarish to something like into

otfrom14:02:34

I'm just wondering if I'm missing something that will bite me later

Alex Miller (Clojure team)15:02:05

do you want a potentially blocking lazy seq?

otfrom15:02:03

wouldn't that be a problem with the async/into as well?

Alex Miller (Clojure team)15:02:49

that returns a channel, not a sequence?

otfrom15:02:48

true, I'd forgotten that to use the result I need to async/<!! the returned channel

Alex Miller (Clojure team)15:02:51

onto-chan and to-chan have ! and !! variants to choose if you want blocking or parking behavior

Alex Miller (Clojure team)15:02:40

but that's really more "input" side

otfrom15:02:49

yeah, I'm trying to go the other way. I've got a async/pipeline working on a bunch of data that is generated in memory and I need to feed it to something that takes seq like things and reduces it

Alex Miller (Clojure team)15:02:05

there's also a/reduce (but eager)

Alex Miller (Clojure team)15:02:31

anyways, my point is, you can't ignore parking/blocking when pulling stuff from a channel, and channel is a better way to convey a stream of values like that than a lazy seq

otfrom15:02:18

thx. I think what I want is something like sequence, or claypoole/upmap (an unordered pmap) that can take a comp'd chain of transducers, and maybe I've not found the right thing yet. pipeline mostly fits the bill and doesn't seem to trip when I'm using it for this, but I take your point on parking/blocking

otfrom15:02:14

(I think core.async might just not be right for what I'm doing as I'm interested in parallelism in my batches rather than concurrency and I'm probably adding overhead with keeping things in order)

otfrom15:02:42

being able to go from (sequence my-xf data) to (pipeline cores out-chan my-xf (async/to-chan!! data)) just feels easy (with all the downsides of easy) as a way of adding parallelism to a bit of batch processing while keeping the logic the same in my-xf

otfrom15:02:02

(and I like being able to put multis on things, tap them and produce multiple outputs, but that is probably my misreading of "The Language of the System")

Ben Sless16:02:23

This reminds me I've been wondering, why aren't channels reducible? Why have a custom implementation of transduce and reduce?

hiredman17:02:13

Because reduce as defined by clojure.core/reduce and the protocols and interfaces is not an asynchronous operation

1
Alex Miller (Clojure team)17:02:14

well, they predated transduce and IReduce stuff

hiredman17:02:32

like, channels could be seqable

Alex Miller (Clojure team)17:02:19

per the prior discussion, I think having a potentially blocking channel backed seq seems questionable

hiredman17:02:28

the seqable is maybe more egregious then reducible

Alex Miller (Clojure team)17:02:40

seems like the opposite to me

ghadi17:02:49

they both don't make sense

ghadi17:02:42

anything op that pulls off a channel should be explicit (or starts a process/returns chan)

hiredman17:02:28

channels as collections is also pretty "meh"

hiredman17:02:20

channels are synchronization points between processes, that have queue like behaviors

💯 1
otfrom17:02:01

I'm happy to find out I'm looking at the wrong tool. I think I'm trying to figure out what the right tool is. I do lots of work with not big data, but big enough data, where having only a bit of it in memory at a time is a real bonus. I also have a number of cores and a problem that is either embarrassingly parallel or solved by combining monoids (like +)

hiredman17:02:43

you might want to look at reducers

hiredman17:02:12

the fold operation there is a kind of parallel reduce

otfrom17:02:31

my problem with reducers is that instead of moving around a comp'd my-lovely-pipeline-xf I have to recreate everything in a new series

hiredman17:02:50

but reducers kind of got overshadowed by transducers, which have an even better transform composition story, but transducers don't have a fold equivalent

☝️ 1
otfrom17:02:16

which might be the right thing to do, but having the chain of comp'd map/mapcat/etc is a nice debugged thing to carry around as the order of operations sometimes matters

ghadi17:02:23

perfect symmetry / interfaces extended to all things is not a goal

otfrom17:02:34

I sketch things out using sequence or into and then have been using pipeline to make it go faster

Ben Sless19:02:49

Channels are queue like, but I can reduce a java Queue. It might be nonsensical, but I don't see a reason why reducing over a channel can't return a channel which will contain a result sometime in the future like it does in core.async.

hiredman19:02:12

channels are queues, but what they queue isn't values, they queue threads of execution

hiredman19:02:45

channels are points where threads of execution can exchange values, and if no thread is currently at that point, a thread of execution can be queued waiting for another to arrive to exchange a value with

Ben Sless20:02:42

While that is the "true face" of channels, the facts is abstractions like pipelines, reduce, onto, etc mask this behind an abstraction very similar to collections

hiredman20:02:17

pipelines don't do that, a pipeline is a process you meet at one end, hand it something, and you meet at the other end to take something from

hiredman20:02:21

nothing collection like

Ben Sless20:02:16

Similar to into with transducer arity

Ben Sless20:02:56

I'm well aware I may be trying to square a circle here

ghadi20:02:55

problem statements please - lack of symmetry is not a problem

Ben Sless20:02:47

I don't see a reason why reduce should have two different implementations. We have an interface and a protocol, why not use that?

Ben Sless20:02:19

Should every library which defines some context you can take elements out of and put elements in define its own reduce?

phronmophobic20:02:57

clojure.core/reduce and core.async/reduce seem like different functions to me. This example is kind of contrived, but if I wrote a function that takes a collection to reduce, then I don't see how being able to extend clojure.core/reduce with the implementation of core.async/reduce makes sense.

(defn extended-reduce [f init xs]
  (if (seqable? xs)
    (reduce f init xs)
    (async/reduce f init xs)))

(defn example [xs]
  (* 2 (extended-reduce + 0 xs)))

> (example [1 2 3])
;; 12
> (example #{1 2 3})
;; 12
> (example (reify
             clojure.lang.Seqable
             (seq [_]
               (list 1 2 3))))
;; 12
> (example (async/to-chan! [1 2 3]))
;; Execution error (ClassCastException)

Ben Sless10:02:23

Because the type of what you're getting will be related to the type of what you're putting in. It's some effect you can't get rid of