Fork me on GitHub

Hello all, how do I use pmap to only go to next function when it finish?


Can you give an example of what you mean? Perhaps you mean that because pmap is lazy, execution of the later forms can start before pmap finishes applying the function f you give it to all sequence elements?


If you mean that, then any lazy expression can be forced to evaluate completely, pmap or otherwise, by enclosing it inside of a call to doall


@andy.fingerhut humm, I think it´s that I want, only return when all pmap elements finish


Clojure has many other functions that return lazy sequences besides pmap, e.g. map, filter, and a lot of functions that return sequences.


what is your advice for using like pmap with limited number of concurrent execution ?


I´m asking this because each pmap call will execute a network call and I don´t want it to be a flood

Mario C.16:10:32

I am doing something similar like this futures (mapv #(future (api-call %)) list-of-args) then (map deref futures)

Mario C.16:10:38

So I am curious what the proper way is too


it depends a lot on the context, but I would either use Executors directly or core.async's pipeline-blocking


Is there any way to limit it about the number of concurrent execution?


pmap is a really bad choice it gives you no control over concurrency


using executors directly you tell it how large of a threadpool to use, and pipeline-blocking takes an 'n' which more or less limits the amount of concurrency (it limits it to n+2 if I recall)


I did with pmap and api connection fails with many concurrent execution


yeah, that is kind of a "no duh"


as I said, pmap is a terrible choice for that


I can´t figure out how to convert it to core.async. It´s "obscure" yet for my knowleadge


There's also claypoole and its pmap

Mario C.16:10:50

There we go someone mentioned that a while back. Claypoole


@fabrao, have you checked pipeline-async? It seems appropriate for the scenario you’re describing(network calls).


I very strongly recommend against using pipeline-async if you are already having trouble managing concurrency


Could you please elaborate a bit more on that? 🙂 What would be your concerns?

hiredman19:10:00 the comments here are a little more detailed about pipeline-async and how it is a very different animal from the other two pipelines


Thanks… 🙂

Alex Miller (Clojure team)18:10:08

pipeline-blocking is probably better for this


look what I did

(defn prmap
  "Like map, except f is applied in parallel. Semi-lazy in that the
  parallel computation stays ahead of the consumption, but doesn't
  realize the entire result unless required. Only useful for
  computationally intensive functions where the time of f dominates
  the coordination overhead."
  {:added "1.0"
   :static true}
  ([f coll]
   (let [n 3;; (+ 0 (.. Runtime getRuntime availableProcessors))
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets))))
  ([f coll & colls]
   (let [step (fn step [cs]
                 (let [ss (map seq cs)]
                   (when (every? identity ss)
                     (cons (map first ss) (step (map rest ss)))))))]
     (prmap #(apply f %) (step (cons coll colls)))))) 


so, I can limit concurrent by n


pipeline-blocking seems to be a good way too


@alexmiller pipeline-blocking worked in good way, thank you

Zac Bir19:10:55

Got a question about namespaces and their use. I’m used to declaring a namespace and requiring others (renaming as necessary), but what are the boundaries for clojure’s standard library?

Zac Bir19:10:44

For instance, I’ve tried including the namespace to get at the uuid function, but when I use it in one source (or even enter the namespace in a repl), it tells me it’s unable to resolve the symbol

Zac Bir19:10:54

user> (in-ns '
#namespace[]> (uuid)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: uuid in this context, compiling:(*cider-repl Development/geometer:localhost:52209(clj)*:22936:26)> 


that ns is noto auto-loaded


in-ns doesn't look for or load anything, if the namespace doesn't exist yet it creates a new broken namespace


you can mostly fix that namespace by runnign (clojure.core/refer-clojure)


but that still doesn't load the source for that ns- it just makes a bare bones non-broken ns


the right way to do that is requiring first


here's a helpful idiom: (doto ' (require) (in-ns)) - this lands you in that ns, unless the load fails

Zac Bir19:10:32> (doto ' (require) (in-ns))
FileNotFoundException Could not locate clojure/data/generators__init.class or clojure/data/generators.clj on classpath.  clojure.lang.RT.load (


in-ns is "switch context to this namespace" -- it is almost certainly an error if the namespace hasn't yet been loaded


right - but it doesn't error, it just creates a broken ns


which the user will experience as an error very quickly 😉


(in-ns 'user) then (require ' :reload) if in doubt


that in-ns fails though, unless you use (clojure.core/in-ns) or do the refer-clojure first (maybe I'm misunderstanding)

Zac Bir20:10:26

Okay, that also gives me the most recent error. I guess I have things Not Setup Properly™.


you probably haven't restarted your repl since adding that lib (or never added the lib correctly)


it doesn't come with clojure.core


in-ns is special @noisesmith

Zac Bir20:10:04

Well, that’d prolly be it 🙂


user=> (in-ns 'nonexistent)
#object[clojure.lang.Namespace 0x6e5bfdfc "nonexistent"]

nonexistent=> (require 'foo) ;; ah crap clojure.core isn't available in here
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: require in this context

nonexistent=> (in-ns 'user)
#object[clojure.lang.Namespace 0x17ae98d7 "user"]

user=> (map symbol [:back :to :reality])
(back to reality)

💯 8
Zac Bir20:10:06

Never added that lib. Thought it was in the equiv of stdlib


@zbir This page might help -- those are all "external" libraries that need to be added as dependencies.


It also lets you know the development status (`` is considered Inactive).

Alex Miller (Clojure team)20:10:48

while inactive, it's still perfectly fine to use - clojure itself uses it extensively in its own test suite


Yeah, just setting expectations in case @zbir finds things he might want changed 🙂

Zac Bir20:10:02

Is the idea that for things that act as barest proxies to underlying java implementations, just use the underlying implementation?

Zac Bir20:10:27

(which is totally valid, just still learning the land)


that's what I'd usually suggest yeah - there's no real abstraction layer barrier between clojure and the vm (except the compiler / language semantics) - using interop throughout your code is fine if the raw classes and their methods do the thing you need


for some constructs a clojure idiom can be a lot cleaner than the java factory and dep-injection centered approach, but that should end up being a pretty thin thing in most cases


I used to be a big fan of wrapper libraries to hide raw interop but I'm getting less and less so these days. The wrapper is only worthwhile if it substantially improves readability and/or reduces boilerplate (such as around JDBC stuff 🙂 )

👍 4

stuff like java.time is well designed and benefits less from wrappers than stuff like java.jdbc which is a pile of sad

Zac Bir20:10:21

That’s exactly the kind of guideline I was wondering.

Zac Bir20:10:44

Foolish consistency, and all…