Fork me on GitHub
#beginners
<
2019-10-15
>
fabrao15:10:20

Hello all, how do I use pmap to only go to next function when it finish?

andy.fingerhut16:10:12

Can you give an example of what you mean? Perhaps you mean that because pmap is lazy, execution of the later forms can start before pmap finishes applying the function f you give it to all sequence elements?

andy.fingerhut16:10:50

If you mean that, then any lazy expression can be forced to evaluate completely, pmap or otherwise, by enclosing it inside of a call to doall

fabrao16:10:57

@andy.fingerhut humm, I think it´s that I want, only return when all pmap elements finish

andy.fingerhut16:10:06

Clojure has many other functions that return lazy sequences besides pmap, e.g. map, filter, and a lot of functions that return sequences.

fabrao16:10:47

what is your advice for using like pmap with limited number of concurrent execution ?

fabrao16:10:15

I´m asking this because each pmap call will execute a network call and I don´t want it to be a flood

Mario C.16:10:32

I am doing something similar like this futures (mapv #(future (api-call %)) list-of-args) then (map deref futures)

Mario C.16:10:38

So I am curious what the proper way is too

hiredman16:10:40

it depends a lot on the context, but I would either use Executors directly or core.async's pipeline-blocking

fabrao16:10:44

Is there any way to limit it about the number of concurrent execution?

hiredman16:10:46

pmap is a really bad choice it gives you no control over concurrency

hiredman16:10:40

using executors directly you tell it how large of a threadpool to use, and pipeline-blocking takes an 'n' which more or less limits the amount of concurrency (it limits it to n+2 if I recall)

fabrao16:10:01

I did with pmap and api connection fails with many concurrent execution

hiredman16:10:44

yeah, that is kind of a "no duh"

hiredman16:10:58

as I said, pmap is a terrible choice for that

fabrao16:10:14

I can´t figure out how to convert it to core.async. It´s "obscure" yet for my knowleadge

jumar16:10:57

There's also claypoole and its pmap

Mario C.16:10:50

There we go someone mentioned that a while back. Claypoole

joaohgomes18:10:45

@fabrao, have you checked pipeline-async? It seems appropriate for the scenario you’re describing(network calls). https://clojuredocs.org/clojure.core.async/pipeline-async

hiredman18:10:15

I very strongly recommend against using pipeline-async if you are already having trouble managing concurrency

joaohgomes19:10:30

Could you please elaborate a bit more on that? 🙂 What would be your concerns?

hiredman19:10:00

https://clojure.atlassian.net/plugins/servlet/mobile?originPath=%2Fbrowse%2FASYNC-163#issue/ASYNC-163 the comments here are a little more detailed about pipeline-async and how it is a very different animal from the other two pipelines

joaohgomes19:10:30

Thanks… 🙂

Alex Miller (Clojure team)18:10:08

pipeline-blocking is probably better for this

fabrao19:10:37

look what I did

(defn prmap
  "Like map, except f is applied in parallel. Semi-lazy in that the
  parallel computation stays ahead of the consumption, but doesn't
  realize the entire result unless required. Only useful for
  computationally intensive functions where the time of f dominates
  the coordination overhead."
  {:added "1.0"
   :static true}
  ([f coll]
   (let [n 3;; (+ 0 (.. Runtime getRuntime availableProcessors))
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                (lazy-seq
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets))))
  ([f coll & colls]
   (let [step (fn step [cs]
                (lazy-seq
                 (let [ss (map seq cs)]
                   (when (every? identity ss)
                     (cons (map first ss) (step (map rest ss)))))))]
     (prmap #(apply f %) (step (cons coll colls)))))) 

fabrao19:10:12

so, I can limit concurrent by n

fabrao19:10:14

pipeline-blocking seems to be a good way too

fabrao19:10:50

@alexmiller pipeline-blocking worked in good way, thank you

4
Zac Bir19:10:55

Got a question about namespaces and their use. I’m used to declaring a namespace and requiring others (renaming as necessary), but what are the boundaries for clojure’s standard library?

Zac Bir19:10:44

For instance, I’ve tried including the clojure.data.generators namespace to get at the uuid function, but when I use it in one source (or even enter the namespace in a repl), it tells me it’s unable to resolve the symbol

Zac Bir19:10:54

user> (in-ns 'clojure.data.generators)
#namespace[clojure.data.generators]
clojure.data.generators> (uuid)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: uuid in this context, compiling:(*cider-repl Development/geometer:localhost:52209(clj)*:22936:26) 
clojure.data.generators> 

noisesmith19:10:14

that ns is noto auto-loaded

noisesmith19:10:34

in-ns doesn't look for or load anything, if the namespace doesn't exist yet it creates a new broken namespace

noisesmith19:10:47

you can mostly fix that namespace by runnign (clojure.core/refer-clojure)

noisesmith19:10:07

but that still doesn't load the source for that ns- it just makes a bare bones non-broken ns

noisesmith19:10:15

the right way to do that is requiring first

noisesmith19:10:50

here's a helpful idiom: (doto 'clojure.data.generators (require) (in-ns)) - this lands you in that ns, unless the load fails

Zac Bir19:10:32

clojure.data.generators> (doto 'clojure.data.generators (require) (in-ns))
FileNotFoundException Could not locate clojure/data/generators__init.class or clojure/data/generators.clj on classpath.  clojure.lang.RT.load (RT.java:463)

ghadi19:10:42

in-ns is "switch context to this namespace" -- it is almost certainly an error if the namespace hasn't yet been loaded

noisesmith19:10:59

right - but it doesn't error, it just creates a broken ns

ghadi20:10:18

which the user will experience as an error very quickly 😉

ghadi20:10:31

(in-ns 'user) then (require 'clojure.data.generators :reload) if in doubt

noisesmith20:10:10

that in-ns fails though, unless you use (clojure.core/in-ns) or do the refer-clojure first (maybe I'm misunderstanding)

Zac Bir20:10:26

Okay, that also gives me the most recent error. I guess I have things Not Setup Properly™.

noisesmith20:10:49

you probably haven't restarted your repl since adding that lib (or never added the lib correctly)

noisesmith20:10:56

it doesn't come with clojure.core

ghadi20:10:04

in-ns is special @noisesmith

4
Zac Bir20:10:04

Well, that’d prolly be it 🙂

ghadi20:10:42

user=> (in-ns 'nonexistent)
#object[clojure.lang.Namespace 0x6e5bfdfc "nonexistent"]

nonexistent=> (require 'foo) ;; ah crap clojure.core isn't available in here
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: require in this context

nonexistent=> (in-ns 'user)
#object[clojure.lang.Namespace 0x17ae98d7 "user"]

user=> (map symbol [:back :to :reality])
(back to reality)

💯 8
Zac Bir20:10:06

Never added that lib. Thought it was in the equiv of stdlib

seancorfield20:10:47

@zbir This page might help https://clojure.org/community/contrib_libs -- those are all "external" libraries that need to be added as dependencies.

seancorfield20:10:17

It also lets you know the development status (`clojure.data.generators` is considered Inactive).

Alex Miller (Clojure team)20:10:48

while inactive, it's still perfectly fine to use - clojure itself uses it extensively in its own test suite

seancorfield20:10:51

Yeah, just setting expectations in case @zbir finds things he might want changed 🙂

Zac Bir20:10:02

Is the idea that for things that act as barest proxies to underlying java implementations, just use the underlying implementation?

Zac Bir20:10:27

(which is totally valid, just still learning the land)

noisesmith20:10:27

that's what I'd usually suggest yeah - there's no real abstraction layer barrier between clojure and the vm (except the compiler / language semantics) - using interop throughout your code is fine if the raw classes and their methods do the thing you need

noisesmith20:10:31

for some constructs a clojure idiom can be a lot cleaner than the java factory and dep-injection centered approach, but that should end up being a pretty thin thing in most cases

seancorfield20:10:09

I used to be a big fan of wrapper libraries to hide raw interop but I'm getting less and less so these days. The wrapper is only worthwhile if it substantially improves readability and/or reduces boilerplate (such as around JDBC stuff 🙂 )

👍 4
ghadi20:10:08

stuff like java.time is well designed and benefits less from wrappers than stuff like java.jdbc which is a pile of sad

4
Zac Bir20:10:21

That’s exactly the kind of guideline I was wondering.

Zac Bir20:10:44

Foolish consistency, and all…