Fork me on GitHub
#clojure-dev
<
2019-05-17
>
rickmoynihan14:05:57

I’ve just seen this WARNING on clojure 1.10.0 on jdk 12: > WARNING: An illegal reflective access operation has occurred >WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000801182440 (file:/Users/rick/.m2/repository/org/clojure/clojure/1.10.0/clojure-1.10.0.jar) to method com.sun.org.apache.xerces.internal.jaxp.datatype.XMLGregorianCalendarImpl.getYear() > WARNING: Please consider reporting this to the maintainers of clojure.lang.InjectedInvoker/0x0000000801182440 > WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations > WARNING: All illegal access operations will be denied in a future release I suspect it’s a known issue

gfredericks14:05:44

I see that all the time, and also suspect it's a known issue 🙂

thheller15:05:37

I'd suspect that this isn't actually an issue in Clojure but rather some library relying on reflection and the JDK blaming the wrong thing for it

alexmiller15:05:56

the reflection happens in Clojure (at the behest of code doing reflection)

gfredericks15:05:06

InjectedInvoker doesn't appear in the clojure source at all, curiously

alexmiller15:05:44

the question is - who is calling getYear on a calendar impl?

gfredericks15:05:52

what's the policy change here anyhow? what kind of reflection is no longer allowed?

ghadi15:05:58

@rickmoynihan use --illegal-access=debug and it will tell you where it's happening

ghadi15:05:06

it will dump a stacktrace

rickmoynihan15:05:18

thanks alex & ghadi

alexmiller15:05:21

well the stack trace may just be the clojure compiler, which would not tell you

alexmiller15:05:44

depends whether it's compile-time or run-time reflective determination being made

gfredericks15:05:28

is that just the distinction between (def foobar (.getYear coolCalendar)) and (defn jhomason [] (.getYear coolCalendar)), or something more subtle?

alexmiller15:05:37

yes, that, I believe

rickmoynihan15:05:25

ok thanks that gives me a decent stacktrace… thankfully it also looks like it’s in one of my libraries, so I can take a look at it

ghadi15:05:18

@alexmiller the compiler should be free to reflect without warning, no? it's only usage/invocation that incurs the illegal access warning

rickmoynihan15:05:21

iirc it uses the javax xml calendar for some type coercions — so I guess this is unsurprising

ghadi15:05:44

boooooooo

rickmoynihan15:05:43

it has to read xsd datatypes so it needs to do something like that

rickmoynihan15:05:18

we coerce them to java.time; but have to read XSD representations off the wire so to speak

rickmoynihan15:05:07

will shave this yak another day

ghadi15:05:57

sounds like you have an actually legit usage of javax.xml for date formatting

rickmoynihan15:05:30

yeah it’s a really gross API… but RDF borrows its data types from xsd - so needs must.

rickmoynihan15:05:13

the main job grafter does is coerce that gunk into proper values

ghadi15:05:02

perhaps the world's first 😂

gfredericks15:05:20

@ghadi when you say the compiler should be free to reflect, do you mean the compiler should be free to generate reflection code?

ghadi15:05:54

I mean the compiler should be free to enumerate a class's methods, even inaccessible ones. If it generates bytecode that refers to an inaccessible method, that's where the warning will happen -- in runtime, not in the compiler

ghadi15:05:16

enumeration doesn't ever warn

gfredericks15:05:51

so (def foo-bar (.getYear coolCalendar)) might warn/throw, but the stack would at least point to the line in question despite having lots of compiler frames in it, I think

ghadi15:05:54

meaning --illegal-access=debug is a reliable method to get accurate traces

gfredericks15:05:00

so I'm not sure what Mister Miller was worried about

ghadi15:05:04

so that example is interesting because it's runtime reflective - as you say it will have extra frames in it -- but I don't think there will be any compiler frames

ghadi15:05:29

the compiler will just give up and emit reflective code

gfredericks15:05:01

but the code runs at compile-time

ghadi15:05:18

oh yeah - true

alexmiller15:05:14

I retract my worry

gfredericks16:05:15

the only emoji completion for phew is christopher_walken

walken 1
gfredericks19:05:32

I'm running a nested java.jdbc query, trying to process both result sets lazily in parallel, and am not getting the head-holding-OOM I expected, and I'm curious to understand why the simplest version of the structure is:

(jdbc/query ...
 {:result-set-fn
  (fn [xs]
   (jdbc/query ...
    {:result-set-fn
     (fn [ys] (reduce ... (map vector xs ys)))}))})
I would have expected that I need to use ^:once fn* on the inner function so that the closure-reference to xs gets released early and there's no head-holding

gfredericks19:05:37

the only way I can imagine this is working anyhow is if we have this sequence of events 1. .invoke called on the inner function 2. inner function essentially passes xs to reduce 3. jvm notices that the instance variable xs won't be used again in the invoke method 4. AND the jvm notices that the instance of the inner function isn't going to be used again either 5. and so concludes that the head of xs is unreachable

gfredericks19:05:55

is this really happening because it sounds too fancy to be true

gfredericks19:05:35

I dunno maybe it isn't

gfredericks19:05:56

I guess it just never occurred to me that the jvm could tell a function was being called for the last time

rickmoynihan20:05:32

Your description above sounds like what I understand escape analysis to be. I’ve no idea if it’s happening; but escape analysis is something the JVM does; and it is something it’s getting better at. So what you say sounds plausible to me.

👍 1
hiredman21:05:49

I think it is entirely possible it is just locals clearing, but you would want to look at the bytecode to be sure

gfredericks21:05:49

the clojure compiler can't know the function is only going to be called once

gfredericks21:05:57

if it might be called again, you have to hold onto the head, just in case