Hello! Question about macro expansion. A direct call to a loop-unrolling macro is fast. A defn that wraps the exact same macro call is ~15x slower.
(defmacro generate-work [data n] <some macro that transforms data using n>)
;; Fast:
(generate-work data 20)
;; Slow:
(defn wrapper [data] (generate-work data 20))
(wrapper data)
Why is the wrapper-fn version so dramatically slower than the direct macro call, when its body is expanded into the exact same code at compile time?
It probably isn't. How do you measure? Are there any side-effects in the body of the macro, any memoization, access to external resources? Can you share the impl of the macro?
You're right, I oversimplified it. data is a lazy-seq. It also takes stateful target and does some work on it. I'm measuring with time and doing some benchmark using criterium. I'm constantly getting 5 sec with direct macro call, and getting 70-75 seconds with wrapper version.
Need to see the code.
I agree with p-himik, there's not enough information in the question to generate any hypotheses really
(defmacro write-columnar-csv-unrolled [writer data num-cols]
(dotimes ['row-idx (count (first data))]`
(doto ~writer
~@(mapcat (fn [col-idx]
[(.write (str (get (get data col-idx) ~'row-idx)))`
~@(if (< col-idx (dec num-cols))
[(.write "," 0 1)]`
[(.write "\n")])])`
(range num-cols)))))
And I'm calling it like
(with-open [w (io/writer writer)]
(doseq [chunk chunks]
(write-columnar-csv-unrolled w chunk 20)
(.flush w)))
Add ^java.io.Writer to the wrapping function's signature, in front of the writer argument.
But before that, just to see how to diagnose problems like this, add (set! *warn-on-reflection* true) right under the (ns ...) form where you've defined that function.
That's why providing proper minimal reproducible examples is of paramount importance. :) Absolutely nothing in your OP indicated that there might be reflection. I guess I can fine-tune my crystal orb now a bit so I can immediately call something like this out in future. :D
Thanks, I see the issue now. I thought there are some overhead when the code is not inlined. I was wrong and It's reflection issue.
Hi! Is anyone using https://github.com/HtmlUnit/htmlunit here?
I have used htmlunit from time to time for web scraping and shockingly some unit tests, but not actively using it for anything right now
The web is so js heavy these days it can be challenging to do stuff without a real browser. The project where I was using htmlunit for tests ended up switching to playwright.