Clojurians Log v2

Apparently the .uri method lives in a generated class called MediaSegmentBuilder, whereas MediaSegment/builder returns a MediaSegment$Builder , which extends MediaSegmentBuilder .

With pain, it seems.

Trying to figure out a class hierarchy to eke out the exact puzzle pieces and how they fit together to allow me to make the 1 function call I need is my least favourite part of Java interop.

I even tried using reflection to generate an instance of MediaSegmentBuilder instead of MediaSegment$Builder and then go through that, but:

user=> (def constructor (first (seq (.getDeclaredConstructors MediaSegmentBuilder))))
#'user/constructor
user=> (.setAccessible constructor true)
nil
user=> (def segment (.newInstance constructor (into-array Object [])))
Execution error (UnsupportedOperationException) at io.lindstrom.m3u8.model.MediaSegmentBuilder/ (MediaSegmentBuilder.java:48).
Use: new MediaSegment.Builder()

With pain, it seems.I'll say. I am this close to just writing a Clojure M3U8 parser from scratch to sidestep this insanity. 😛 It would probably be less work than making this one method call work. (Not blaming Clojure, just to be clear, but rather the OOP madness that makes people use things like Immutables.)

indeed

The builder pattern normally (I think? long ago) has a build() or similar fn that actually returns the object you're trying to build. Usually after you call various fns to set up the desired state of the thing you're trying to build. Can you use whatever it produces?

I tried that, but I cannot, because:

user=> (-> (MediaSegment/builder) (.build))
Execution error (IllegalStateException) at io.lindstrom.m3u8.model.MediaSegmentBuilder/build (MediaSegmentBuilder.java:393).
Cannot build MediaSegment, some of required attributes are not set [duration, uri]

hahaha amazing

That's one word for it. 😛

well does it have something like setUri() instead of uri()

?

or other "setter" methods

No... MediaSegment$Builder has almost nothing:

user=> (clojure.pprint/pprint (reflect/reflect MediaSegment$Builder))
{:bases #{io.lindstrom.m3u8.model.MediaSegmentBuilder},
 :flags #{:public},
 :members
 #{{:name build,
    :return-type io.lindstrom.m3u8.model.MediaSegment,
    :declaring-class io.lindstrom.m3u8.model.MediaSegment$Builder,
    :parameter-types [],
    :exception-types [],
    :flags #{:public :bridge :synthetic}}
   {:name io.lindstrom.m3u8.model.MediaSegment$Builder,
    :declaring-class io.lindstrom.m3u8.model.MediaSegment$Builder,
    :parameter-types [],
    :exception-types [],
    :flags #{:public}}}}
nil

user=> (clojure.pprint/pprint (->> (reflect/reflect MediaSegmentBuilder) :members (map :name)))
(access$800
 duration
 access$1100
 access$900
 byteRange
 OPT_BIT_CUE_IN
 gap
 dateRange
 INIT_BIT_URI
 dateRange
 access$100
 segmentMap
 access$200
 access$1400
 bitrate
 cueIn
 uri
 duration
 uri
 access$500
 segmentMap
 addSegmentKeys
 segmentKeys
 from
 formatRequiredAttributesMessage
 gap
 addPartialSegments
 partialSegments
 cueIn
 access$400
 access$1700
 programDateTime
 access$1300
 access$1000
 access$600
 access$1200
 gapIsSet
 cueInIsSet
 addAllSegmentKeys
 build
 dateRange
 OPT_BIT_DISCONTINUITY
 programDateTime
 addPartialSegments
 discontinuity
 access$300
 addSegmentKeys
 optBits
 byteRange
 bitrate
 createSafeList
 bitrate
 title
 segmentMap
 cueOut
 access$1800
 programDateTime
 INIT_BIT_DURATION
 io.lindstrom.m3u8.model.MediaSegmentBuilder
 discontinuity
 title
 addAllPartialSegments
 createUnmodifiableList
 partialSegments
 segmentKeys
 cueOut
 cueOut
 title
 access$1600
 OPT_BIT_GAP
 initBits
 access$1500
 access$700
 byteRange
 discontinuityIsSet)

Yeah ok. I don't get it. If class B extends A, shouldn't the methods of B be those of A + those specific to B?

Does it take more than 1 argument?

No matching method uri found taking 1 args for class io.lindstrom.m3u8.model.MediaSegment$Builder

or less 😅

is the class module-private? no

(def -no-reflection-warning ^[String] io.lindstrom.m3u8.model.MediaSegmentBuilder/.uri)
(def -reflection-warning io.lindstrom.m3u8.model.MediaSegmentBuilder/.randomNonexistentMethod)

(try (-no-reflection-warning (MediaSegment/builder) "foo")
  (catch IllegalAccessError e e))

"failed to access class io.lindstrom.m3u8.model.MediaSegmentBuilder from class repl$invoke__MediaSegmentBuilder__DOT_uri__10381 (io.lindstrom.m3u8.model.MediaSegmentBuilder is in unnamed module of loader 'app'; repl$invoke__MediaSegmentBuilder__DOT_uri__10381 is in unnamed module of loader clojure.lang.DynamicClassLoader @759073b4)"

> I don't get it. If class B extends A, shouldn't the methods of B be those of A + those specific to B? My thoughts exactly.

Oh. Is .uri private?

no, it's public

however:

(java.lang.reflect.Modifier/isPublic (.getModifiers MediaSegmentBuilder)) ; => false

Oh, right, I misread somehow.

{:name uri,
 :return-type io.lindstrom.m3u8.model.MediaSegment$Builder,
 :declaring-class io.lindstrom.m3u8.model.MediaSegmentBuilder,
 :parameter-types [java.lang.String],
 :exception-types [],
 :flags #{:public :final}}

I wonder if the issue is ..$Builder inhering from ..Builder

and note

(let [-subinterface-accessor ^[String] io.lindstrom.m3u8.model.MediaSegment$Builder/.uri]
  (try (-subinterface-accessor (MediaSegment/builder) "foo")
    (catch IllegalAccessError e (re-find #"MediaSegment$Builder" (ex-message e)))))
; => nil

Hmm, this seems to work:

user=> (defn invoke-private [obj class method-name param-types & args]
  (let [m (doto
            (.getDeclaredMethod class method-name (into-array Class param-types))
            (.setAccessible true))]
    (.invoke m obj (object-array args))))
#'user/invoke-private
user=> (invoke-private (MediaSegment/builder) MediaSegmentBuilder "uri" [String] "")
#object[io.lindstrom.m3u8.model.MediaSegment$Builder 0x1851c7d2 "io.lindstrom.m3u8.model.MediaSegment$Builder@1851c7d2"]

yes, "cheating" with setAccessible always works xD

My goodness. Well, this is awful, and the performance is probably terrible, but at least it works.

Victory... but at what cost

In any case, thanks for https://clojurians.slack.com/archives/C03S1KBA2/p1778836922169309?thread_ts=1778833277.167469&cid=C03S1KBA2! That was the key to the puzzle.

eh, if you grab the method at compile time it should be fast enough

Oh, that's true.

Fwiw it works just fine in Java

What works just fine in Java?

the lib

Oh, yes, no doubt.

perhaps your best bet is a Java shim then

Yeah, I'll consider that if this project goes further than the first spike I'm working on now. 👍

It seems that there's no Ask post for it. There's https://ask.clojure.org/index.php/4255/cannot-resolve-public-generic-method-package-private-class but it's slightly different - it's specifically about generic methods. But maybe it's enough to mention your issue there and vote for it.

Thanks, will do!

At a quick glance, the last comment in that issue looks similar to my issue.

Oh, but it's not.

E: remove useless code (E: https://pastebin.com/12935dDX) The reason for this working in java and not in the AFn appears to be that Java calls into the subinterface's method which while being invisible to reflection does exist at the bytecode level, whereas Clojure only sees the superinterface's package-internal method and either refuses to emit the call (direct method call case) or eats an AccessError at runtime (method-as-value case). What I find remarkable here is that Java itself allows public classes to inherit from module-private ones, and handles the accessibility only at the level of bytecode, presenting a different pictore to the reflection machinery! Any tool relying on reflection and not barging in with setAccessible would choke here.

as per method handles...

(defn -find-that-method [cls]
  (.findVirtual (MethodHandles/lookup)
    cls "uri"
    (MethodType/methodType cls (into-array [String]))))

(defn -invoke-handle [h & args]
  (java.lang.invoke.MethodHandle/.invokeWithArguments h (object-array args)))

(-invoke-handle (-find-that-method MediaSegment$Builder) (MediaSegment/builder) "foo") ; ok

(try
  (-find-that-method MediaSegmentBuilder)
  (catch IllegalAccessException _)) ; access error

I've had to catch throwable once or twice. Can't remember the specifics but if you do it and end up just throwing again, make sure you reset the interrupted flag in the interrupted case.

> I don't know of any real world case of the latter... What should such a case look like to be useful to you? Because there are a lot of instances of catching/throwing IOError out there, even in the JDK itself.

IOError or IOException?

Both.

Java's docs specifically say that Error types generally should not be caught...

🤷 From .jline.reader.impl.LineReaderImpl#readLine:

catch (IOError e) {
            if (e.getCause() instanceof InterruptedIOException) {
                throw new UserInterruptException(buf.toString());
            } else {
                throw e;
            }
        }

catch (IOError e) {
            // Ignore Ctrl+C interrupts and just exit the loop
            if (!(e.getCause() instanceof InterruptedException)) {
                throw e;
            }
            return true;
        }

public URL getURL() {
                    try {
                        return file.toUri().toURL();
                    } catch (IOException | IOError e) {
                        return null;
                    }
                }

I just scanned our New Relic logs and I see ~70 IOException caught in the last month and zero IOError... Hmm, interesting... thanks.

I love the doc of IOError: "Thrown when a serious I/O error has occurred." :D

Given that Java docs say IOError is for "serious" I/O errors, I wonder what in the JDK is wrapping Exception types in IOError and throwing that? 🤯

Another fun one - an exception caused an error:

catch (IOError e) {
                    throw (IOException) e.getCause();
                }

jdk.internal.io.JdkConsoleImpl wraps IOException into IOError in multiple places. And a bunch of other classes, this is a very common pattern in JDK:

catch (IOException e) {
            throw new IOError(e);
        }

cloning openjdk source so I can search it locally ... damn, that thing's big!

.jline.utils.Curses#tputs wraps any exception:

catch (Exception e) {
            throw new IOError(e);
        }

> cloning openjdk source so I can search it locally ... damn, that thing's big! --depth 1 ;) Also, this pings back to the old discussion we had where you said you don't really need to integrate your Clojure projects with Java sources since you can browse those separately. In my case, I don't even need to clone anything. :D I love the outcome so far, but I hate the fact that I'm hooked onto IntellIJ IDEA with no prospects of being able to drop it.

Bizarre. Since those are all internal packages, I wonder if that wrapping'n'catching is also purely internal and those IOError never escape into user code?

Delightful...

} catch (IOException e) {
            throw new UncheckedIOException(e);
        } catch (IOError error) {
            Throwable cause = error.getCause();
            if (cause instanceof IOException e) {
                throw new UncheckedIOException(e);
            }
            throw new RuntimeException(cause);
        }

I'll dig into this further on Monday, since this is related to some code cleanup I'm doing at work. Food for thought. Thanks @p-himik!

> I wonder if that wrapping'n'catching is also purely internal and those IOError never escape into user code? It can escape from .JdkConsoleImpl#readPassword, via java.lang.System#console.

Yeah, the Console class has methods that throw IOError -- but most of those so far seem to be interactive stuff, rather than anything that would be used inside a web app? I searched for IOError in the JDK source and there are a lot of places to look at (on Monday).

Heh, sun.nio.ch.Poller has it in its static initializer. But I guess "everything blows up when the process starts" is also not that interesting of a use case.

netty throws a io.netty.util.internal.OutOfDirectMemoryError for direct memory exhaustion during serialisation, which it can be useful to wrap to locate which class it it is trying to serialise.