Please, for the love of all that is holy, if something is runtime dynamic, pass it around.

I was profiling my clojure app with JFR and noticed that most of the allocations are for java.lang.reflect.Method Is that something you would consider unusual? It says that bunch of them is coming from clojure.data.csv.write-csv* - line 125:

(defn- write-csv*
  [^Writer writer records sep quote quote? ^String newline]
  (loop [records records]
    (when-first [record records]
      (write-record writer record sep quote quote?)
      (.write writer newline)
      (recur (next records))))) ; this is the line 125

@jumar As a first step, I'd add (set! *warn-on-reflection* true) to each and every file in your project, just after the ns form and see what reflection warnings pop up when try to run the code.

(I don't see anything obvious in your code fragment but I suspect line numbers are not always reported entirely accurately)

I haven't used flight recorder, but I don't think you are interpreting what it is showing you correctly

The stacktrace there doesn't contain any Java reflection, therefore it is not a place where method objects are being allocated for reflection

Which also explains why you aren't getting a reflection warning when compiling that code

Interestingly, I got this, BUT only when I tried to instrument the write-csv* function with Cider debugger:

Reflection warning, /Users/jumar/.m2/repository/org/clojure/data.csv/1.0.0/data.csv-1.0.0.jar:clojure/data/csv.clj:125:7 - call to method write on java.io.Writer can't be resolved (argument types: unknown).

But yeah, a standard invocation of write-csv looks good

Ah, interesting

.write on a Writer with a String argument seems unambiguous -- am I missing something?

That is why it is interesting, doesn't seem possible

I bet the debugger is stripping metadata

It might be false positive caused by the debugger instrumentation?

I'll try to manually call the real function which under the hood uses clojure.data.csv (it's a bit harder); now I just tried to call write-csv by hand. Maybe the stacktrace from JFR is indeed misleading I'm just curious what those 195 GB come from 🙂

If you set warn on reflection in all your namespaces you will get a warning printed out anytime the compiler generates a reflective call

https://github.com/clojure/data.csv/blob/master/src/main/clojure/clojure/data/csv.clj#L15 is a commented out example of doing that

Isn't it enough to just set in the repl or in the "main" ns?

No, it's per-file.

It is complicated

Setting it in your main namespace is usually too late (everything is already compiled by the time the set! Is run)

Ok, that will be a bit of work 🙂

Setting it in the repl can work, but you have to really aware of when you are loading code and set it before you load anything

Understood. Maybe lein check could help with that too?

Yes definitely, I forget about lein check

Ah, lein, a tool from the before times... 🙂

There’s also https://github.com/athos/clj-check for those using tools.deps.

why is there an API to change the root value of a var, but not to only inspect it without interop: (.getRawRoot ...)?

anybody here that has used Electron framework to bundle a Clojure/ClojureScript webapp?

as a desktop application

Where can I remind myself of the details of false being true in clojure if something uses the "wrong" java.Lang.Boolean methods?

Hmm, this https://twitter.com/practical_li/status/1276481899045797889 got me thinking, that is there a bash/zsh completions, that would fill the aliases for the Clojure CLI (like clj -A:re<TAB> -> clj -A:rebel)?

Does anyone have a durable, distributed, job scheduler that runs in the jvm? I'm using quartz now, but I'm reevaluating. I'd prefer if it was durable to jdbc like quartz.

Can someone tell me why

(defn using-pmap
  [update-fn coll]
  (dorun (pmap update-fn coll)))

(defn using-async
  [n update-fn coll]
  (let [coll-ch (a/chan n)
        _ (a/onto-chan! coll-ch coll)]
    (let [processed-ch (-> (repeatedly
                             n
                             #(a/go-loop [item (<! coll-ch)]
                                (if item
                                  (do
                                    (update-fn item)
                                    (recur (<! coll-ch)))
                                  :done)))
                           doall)]
      (doseq [ch processed-ch]
        (<!! ch))
      nil)))

go is rarely a performance optimization, its primary purpose is to make it harder to implement concurrency bugs

if you don't need coordination between threads, core.async won't help you much

The CPU for pmap goes beyond 200% while the async one stays put at 40%

for starters, all go blocks share a small number of threads that doesn't expand

if your work has a significant IO element, the go blocks simply starve each other without increasing CPU usage

Does it expand for pmap, I thought pmap was also something like number of cores + 2 threads

pmap is also not great

which is the same number shared by all go blocks, not just the ones you created in your demo

and their coordination is more expensive than that done by pmap

Yes it involves significant IO

you're paying for features you aren't using in the core.async case

Oh so even if I don’t try to coordinate, there is an overhead

pmap is limitted, but that limit doesn't compose well with some features of lazy-seqs which can result in going past the limit

pmap is a bad abstraction

@hiredman mind expanding a bit?

on which part?

The going past the limit part

lazy-seqs are really "not strict seqs" meaning sometimes you can realize more than the next element you asked for

If not pmap what else, write my own parallel way to process the coll?

there is a feature called chunking

pmap uses laziness to control thread count, rather than using a pool

That is fine in my case, I am using a dorun anyway

just use mapv and future then

@frozenfire1992 I've had great luck with claypoole, and often the underlying ExecutorService which comes with the vm is enough

Yeah chunking I know, will try to chunk my coll and see if it improves perf

(->> (mapv #(future (f %)) x) (map deref))

but that is almost never going to do well

That’s trying to make way too many threads

you almost always want access to the executor and to do things in a finer grained way

@noisesmith Thanks will look into it 👍

the other issue with building your parallelism on top of seqs is your are limiting the concurrency by imposing order

Yeah I guess the pmap threads have to wait for the other ones to finish and then move forward in the seq

yes

and you are collecting all the results just to throw them away

use an executor

cool will check it out, thanks a lot

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int)

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorCompletionService.html

it depends on what the "update-fn" is doing (you might be blocking threads without knowing) @frozenfire1992

and try clojure.core.async/pipeline-async too, depending on the nature of the work being farmed out