Fork me on GitHub
#beginners
<
2020-07-24
>
dev-hartmann10:07:52

Just wanted to let you folks in on my happiness: I successfully created an executable from my minimalistic deps.edn based cli tool via graalvm

dev-hartmann10:07:21

really satisfying. process took 15mins to learn, which is a nice selling point for clojure based apps at work

ūüéČ 15
j15:07:00

I'm using the (time exp) macro to find execution times, and I'd like to capture what it returns to standard output to a string. How do that do that?

Darin Douglass15:07:39

user> (let [out (with-out-str (time (+ 1 1)))]
        (println "HERE" out)) 
HERE "Elapsed time: 0.010066 msecs"

nil
user>

ūüĎć 1
j16:07:15

Thanks! The name of that funtion with-out-str is great!

Grant Isom16:07:38

Hey all, I am working on a new re-frame app and am at the point I want to start writing some tests, I am having a hard time determining what libraries to go with. Any suggestions? I am using shadow, lein, pushy, and bidi mainly

j16:07:59

I tried the following (def timeit (comp with-out-str time)) and am getting the following error: Can't take value of a macro: #'clojure.core/time Does this mean that comp cannot take macros as arguments? If not, what's the alternative?

alexmiller16:07:27

you can't use macros as higher order functions (because they're not functions)

alexmiller16:07:40

or rather they are, but eval'ed at compile time, not run time

alexmiller16:07:57

one technique is to wrap in a function

alexmiller16:07:14

but here, you probably want timeit to be a macro too

alexmiller16:07:21

for the same reasons time is

ūüĎć 1
j16:07:41

Understood, I'll give that a shot. Thanks!

neilyio17:07:53

I'm looking for what must be a pretty basic core.async function. I'd like to do this:

(let [input-ch (load-data-into-chan> my-data)
      valid-ch (SOME-FUNC validation-func input-ch)]
     (go-loop [val (<! valid-ch)]
         (;; do stuff to val...)))

noisesmith17:07:59

why is core.async being used here?

neilyio17:07:02

Basically return a new channel with a function (...a transducer?) applied to each value taken from the another channel. I'm looking for it to be lazy. It kind of seems like async/map is what I'm looking for, but it requires putting a single channel (like input-ch) into a list, which I haven't seen a lot of examples do. pipeline also does what I'm looking for, but it seems kind of verbose (and creates a whole to channel to <! for) for what I'm trying to do.

neilyio17:07:25

It's a NodeJS program, and both the loading the data and validating the data are async operations. I could use callbacks because it's such a small program, but I'm trying to take the opportunity to learn about core.async.

noisesmith17:07:39

so SOME-FUNC doesn't validate the chan, it validates the items on the chan one by one?

neilyio17:07:31

SOME-FUNC would be the function I'm looking for, a higher-order function like async/map or pipeline. It does what you said, yes.

noisesmith17:07:45

what would it do for invalid input?

neilyio17:07:46

It's higher-order, so I'd think whatever its function argument does for invalid input, just like regular map.

noisesmith17:07:17

so it doesn't actually validate, it just returns what validation-func returns for each element

neilyio17:07:23

That's right.

neilyio17:07:41

It seems like pipeline is the closest thing, I was just wondering if there was something shorter.

noisesmith17:07:38

the simplest way to do that is to put a map transudcer onto the chan

neilyio17:07:09

Or some slick way to use transducers that I don't know about. Like one of those pleasant surprises where a shorthand like (my-transducer my-chan) actually returns a new channel with the transducer applied to each value.

noisesmith17:07:37

(onto-chan! (chan (map validate)) coll)

noisesmith17:07:23

you'd want to separate the chan construction out so you can use the chan actually

noisesmith17:07:57

(let [ch (chan (map validate))]
  (onto-chan ch coll)
  (go-loop [...]
     ...))

noisesmith17:07:25

then, for the go loop, you could probably just use pipeline there

noisesmith17:07:40

if you do a raw loop, don't forget to call recur

neilyio17:07:34

Hmm let me describe what I'm trying to get done. I have load-image-folder> that returns a channel, scans all the files in a folder, loads them with an image library (jimp), and puts them on a channel.

noisesmith17:07:41

maybe "conform" is a better name than "validate" here?

noisesmith17:07:00

and if you use mapcat instead of map the validate function can return nil for an item to make it be skipped

neilyio17:07:22

Then a function validate-image takes images off that channel and validates them. I'd like this to happen "as they become available", so that I can fail as early as possible in case load-image-folder> is loading hundreds of images.

noisesmith17:07:14

this is getting kind of in depth, let's move it to a thread as a favor to other people using the channel

noisesmith17:07:41

if validate-image fails, does the whole thing fail?

neilyio17:07:38

To me, that function looks like:

(let [input-ch (load-image-folder> my-path)
      valid-ch (SOME-FUNC validate-image input-ch)]
     (go-try
        (loop [im (<! valid-ch)]
         ;; Do stuff or throw error on invalid im.))

noisesmith17:07:21

you're giving me a possible syntax but I don't fully understand the behavior needed

neilyio17:07:54

Sorry I sent that too early, I just edited.

noisesmith17:07:11

so the error is just on the one image

neilyio17:07:23

Yes, if validate-image fails the whole thing fails, like you said.

neilyio17:07:56

I realize this whole thing needs to be in a larger go-try block or check for the returned error to properly throw.

noisesmith17:07:11

is go-try actually a thing? I don't see it in the core.async api

neilyio17:07:24

From what I've read up on, it's a commonly-used macro to catch and return errors thrown in go blocks. https://martintrojer.github.io/clojure/2014/03/09/working-with-coreasync-exceptions-in-go-blocks

neilyio17:07:45

Your onto-chan example from before was helpful, but as you can see in my case I'm working with data that hasn't come out of the channel yet.

neilyio17:07:12

Just realized another mistake in the example I just posted above. Sorry I'm being so unclear.

noisesmith17:07:00

I'll try another go for my version here:

(let [input-ch (load-image-folder> my-path)
      validation-ch (chan (map #(or (validate-image %) {::error %})))
      result (chan (map (fn [el]
                          (if (::error el)
                              (ret-error el)
                              (process el)))))]
  (pipe result validation-ch)
  result)

noisesmith17:07:25

I only suggested onto-chan because I thought hte input was a collection

noisesmith17:07:33

this version uses a channel

neilyio17:07:42

Totally, glad to clarify that.

noisesmith17:07:15

fixed it to explicitly return result (the chan that gets everything after processing)

neilyio17:07:50

This version looks great, how are values actually getting taken from input-ch and put onto validation-ch?

noisesmith17:07:18

oh - I'm still missing some pieces - making another go...

noisesmith17:07:47

I'm untangling the error handling, I'm getitng it kind of wrong still

noisesmith17:07:34

but this is a work in progress

(let [input-ch (load-image-folder> my-path)
      validation-ch (chan (map #(or (validate-image %)
                                    {::error %})))
      result (chan (comp (take-while (complement ::error))
                         (map process-image)))]
  (pipe validation-ch input-ch)
  (pipe result validation-ch)
  ;; TODO - consume / acquire the ::error keyed element from validation
  ;; if any?
  result)

noisesmith17:07:49

it explicitly stops consuming on the first thing with the ::error key

noisesmith17:07:36

one solution could be that the consumer get two chans "result", and "overflow", and they can check "overflow" for errors when result closes

noisesmith17:07:24

also, NB, these uses of pipe assume that the input ch is closed when there's no data left to acquire, and then they propagate that by closing their output ch (built in behavior)

noisesmith17:07:56

the basic concept here is I'm using pipe + chan with transducer as building blocks of a data flow

noisesmith17:07:53

piple rather than pipeline because I think you really want one-in / one-out rather than batching logic, and that's what pipe does

neilyio17:07:44

Amazing. This gets me in the right direction. I guess you can see how I was looking for something that also did the "piping" within the top let block, to save the separate pipe calls after the let block.

neilyio17:07:55

But this is so much more elegant that what I was trying before.

neilyio17:07:07

I really appreciate your time! Thanks so much for this.

noisesmith17:07:34

nb. this is not tested code, but I think it's the right usage of the core.async provided ops

neilyio19:07:03

Is there a browsable documentation site for cljs.core.async? I've been using the core.async docs at http://clojuredocs.org, but I keep running into discrepancies like cljs.core.async/to-chan is deprecated, where I'm supposed to be using async/to-chan!.

dpsutton19:07:11

what's the discrepancy?

neilyio19:07:25

For example, it seems that https://clojuredocs.org/clojure.core.async/to-chan is still used in JVM Clojure's core.async, where ClojureScript is telling me it's deprecated, and that I should be using to-chan! instead.

neilyio19:07:29

This is the same for onto-chan and onto-chan!. Little things like cljs not having <!! and >!! have also tripped me up, and I've had to stumble across the info in forums.

noisesmith19:07:39

if nothing else, the real docs will be more up to date, I don't know if there's. asimilar site for cljs.core.async though https://clojure.github.io/core.async/#clojure.core.async/to-chan

neilyio19:07:10

Oh you're right, there's no discrepency. http://Clojuredocs.org just must be out of date.

neilyio19:07:58

Thank you for the second time today!

neilyio20:07:37

Okay, last dumb question today. A https://stackoverflow.com/questions/31286167/how-to-create-a-channel-from-another-with-transducers https://clojuredocs.org/clojure.core.async/pipe have https://clojure.atlassian.net/browse/ASYNC-153 that async/pipe returns its to argument, a channel. This seems like it would be very useful, but the async/pipe return value (in cljs) seems to be producing nil when I take! from it. Is anyone else using async/pipe this way?

noisesmith20:07:05

I've always explicitly used the to-chan I pass to pipe, rather than using the return value

neilyio20:07:29

Got it. It just seemed like a useful one-line solution to the problem you and I were discussing before.

noisesmith20:07:00

you could use a small function to return the to-chan

noisesmith20:07:20

also - my simple example from before had the channels flipped, oops

noisesmith20:07:52

I'm used to destinations on the left, sources on the right (even in assembly which I've been punishing myself with lately)

neilyio20:07:59

I'm really sorry to hear that. You're right, I think a small function like that is going to be my solution today. The https://stackoverflow.com/questions/31286167/how-to-create-a-channel-from-another-with-transducers I posted a moment ago has an example of a pipe-trans function that should fit the bill.

hiredman20:07:12

pipe definitely does return the to chan, getting nil from a channel means you are taking from a closed channel

hiredman20:07:41

by default pipe closes the to channel after the from channel closes (which terminates the piping)

hiredman20:07:45

the other thing to do is to double check you use of take!, because it always returns nil, the taken value is passed to the callback you pass in (I forget if cljs does arity checks, or just doesn't because js doesn't, so you might accidentally leave out a parameter)

neilyio20:07:12

@hiredman just tried out everything you said, and you're right on all counts. And cljs does indeed do an arity check if I forget the take! callback, which is handy.

neilyio20:07:00

I was going nuts because I was doing this:

(def a (async/to-chan! (range 100)))
(def b (async/chan 16 #(map inc)))
(def c (async/pipe a b))
;; hiredman is correct
(identical? b c) ;; => true
(poll! a) ;; => nil
(poll! b) ;; => nil
(poll! c) ;; => nil

neilyio20:07:59

Tearing my hair out because my poll! calls were returning nil, but realized it was because I put #(map inc) inside an anonymous function, where a transducer should just be written as (map inc). I don't really understand why that closed all my channels, but really happy that pipe works this way!