Fork me on GitHub
#clojure-dev
<
2019-07-18
>
andy.fingerhut01:07:06

Wow, I don't have enough evidence collected to confirm this hypothesis, but I might be seeing a case of JVM's JIT optimizing the method clojure.lang.Util.hasheq differently in two scenarios: (scenario 1) 99.9% of the calls to that method were with type Long (scenario 2) large fraction of calls were with type long, but also a large fraction with type PersistentHashSet. In scenario 2, the clojure.lang.Util.hasheq calls on type long are much much slower.

andy.fingerhut01:07:56

If that hypothesis is true, then I am awestruck with the extra levels of complexity of predicting the performance of code running on the JVM.

Alex Miller (Clojure team)02:07:40

without knowing more that sounds like could be difference between monomorphic and bimorphic inlining

Alex Miller (Clojure team)02:07:15

which version of java are you on?

andy.fingerhut02:07:23

These measurements are with OpenJDK 11.0.3 installed via apt-get on Ubuntu 18.04 Linux. I don't have clj-async-profiler measurements on macOS running Oracle/Apple JDK 1.8.0_192 but am seeing similar overall elapsed times of the two scenarios there.

Alex Miller (Clojure team)02:07:53

If you're really interested, check out Aleksey Shipilev's blogs like https://shipilev.net/jvm/anatomy-quarks/16-megamorphic-virtual-calls/

Alex Miller (Clojure team)02:07:12

you can pass jvm options to see what the jvm is doing around a lot of these kinds of things

andy.fingerhut02:07:20

Thanks for the link. I will at least start down the rabbit hole, but not sure how far I want to go just yet.

Alex Miller (Clojure team)02:07:29

it's normal for bimorphic or megamorphic things to not inline as well so what you said above does not seem surprising to me

Alex Miller (Clojure team)02:07:37

Clojure protocol call-site caches have the same kind of potential issue - last choice is cached, so monomorphic case will perform much better

seancorfield04:07:17

Over the years I've worked with Clojure, I've often found that I've wanted to work with data -- hash maps -- but also work with Java objects that have setters. An example I'm working with right now is HikariCP and there's a Clojure wrapper that supports Clojure hash maps and turns them into .setFooBar calls for :foo-bar members. I want to do this in general and it feels like this ought to be part of core. Do we already have something that I'm just missing? Is this something that might be considered for core?

hiredman05:07:23

Java.data has something like that

Alex Miller (Clojure team)05:07:50

I've written macros in limited domains to do this kind of thing in the past, as always depends how generic and/or customizable you want to make it

seancorfield05:07:31

@hiredman It's close. :fooBar is needed instead of :foo-bar but it's close enough that I can use it for what I'm trying to do. Thanks.

Alex Miller (Clojure team)05:07:17

well, that's the customizability - can overload the multimethod to fix

seancorfield05:07:33

It's perfect for what I need, to be honest.

seancorfield06:07:29

Turns out that org.clojure/java.data does exactly what I need for next.jdbc to use c3p0 and HikariCP so I can bake that into the library with minimal effort.

andy.fingerhut19:07:58

My performance problem sanity is restored -- I should have been looking for a simpler explanation, which was that clojure.lang.Util/hasheq was being called so many more times in my 10x worse performance scenario, about 10x more times, in fact. Duh. Application code that was the cause found and fixed.

Alex Miller (Clojure team)19:07:42

Calling things more times is usually slower

andy.fingerhut19:07:03

It was not the cause of any problems for me, but was curious if you thought there may be any interest in a CLJ JIRA performance-related ticket on the equiv() methods I found that lack an identical fast path?

Alex Miller (Clojure team)19:07:53

sure, if you happened to find a path where that's not used and makes a demonstrable difference

andy.fingerhut19:07:54

I can add a scenario to the ticket where it can show up -- using maps or sets as keys in array-map's, and the maps/sets you look up or assoc on are identical to the ones added earlier.

andy.fingerhut19:07:08

also, the equiv() methods that lack those identical fast paths are probably over typical inlining code size limits, I would guess, but can investigate that further, later.

souenzzo22:07:17

playing with NaN

(= ##NaN ##NaN)
=> false
(not= ##NaN ##NaN)
=> false
(apply = [##NaN ##NaN])
=> false
(apply not= [##NaN ##NaN])
=> true

mikerod20:07:33

Hacky, crappy test

mikerod20:07:36

seems to show in profiling

mikerod20:07:57

(I would write something better for an “official” Jira thing…)

mikerod20:07:40

on the long case - with array-map, I see the all the action being in the equiv() call

andy.fingerhut22:07:50

Regarding method/function call counts on the JVM, I know that JVMs often make decisions on which methods to JIT based upon call counts, at least as one of the deciding factors. Do Flight Recorder and/or other perf tools make those call counts visible, without having to modify the code?

Alex Miller (Clojure team)23:07:45

The jvm itself can do some of that