Fork me on GitHub
#clojure
<
2023-11-03
>
cddr13:11:24

I have the code below running in an aws lambda handler

(let [read-json #(json/read-value % json/keyword-keys-object-mapper)
            parse-body (fn [rec]
                         (try
                           (read-json (:body rec))
                           (catch Exception e
                             (u/log ::invalid-body :exception e))))
            raw-in (read-json in)
            records (->> raw-in
                         :Records
                         (map parse-body))]

        (run! process-input-event records)

        #_(let [tasks (map #(future
                            (process-input-event %))
                         records)]
          (map deref tasks)))
When it processes the records in serial everything works fine. However, when I change it to use the commented out version that tries to launch a future for each record, I get a weird ClassNotFound error because it can’t find com.amazonaws.services.s3.AmazonS3URI that is used somewhere in the definition of process-input-event. I don’t understand how just adding the concurrency can lead to the class not being found when it is found no problem without the concurrency. I checked the uberjar and it definitely includes the class file where you’d expect to find it

p-himik14:11:22

run! is eager, map is lazy. No clue what's going on with "class not found", but have you tried (run! deref tasks)? While commenting out the other run! form, of course.

cddr14:11:21

Aha that worked. Cheers 🙂

ghadi14:11:16

you still wanna know what happened

ghadi14:11:21

a full stacktrace would help

ghadi14:11:43

(suspect: classloader?)

cddr14:11:03

Sorry about the \ts. It was all on one line and I did a bit of sed to get it more readable but didn’t bother dealing with the tabs.

java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3URI
\tat java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
\tat java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
\tat clojure.core$deref_future.invokeStatic(core.clj:2317)
\tat clojure.core$deref.invokeStatic(core.clj:2337)
\tat clojure.core$deref.invoke(core.clj:2323)
\tat arqam_syncer.lambda$G__10475handleRequest$fn__10476.invoke(lambda.clj:79)
\tat arqam_syncer.logging$with_logging.invokeStatic(logging.clj:29)
\tat arqam_syncer.logging$with_logging.invoke(logging.clj:23)
\tat arqam_syncer.lambda$G__10475handleRequest.invokeStatic(lambda.clj:63)
\tat arqam_syncer.lambda$G__10475handleRequest.invoke(lambda.clj:61)
\tat arqam-syncer.lambda.ArqamSyncer.handleRequest(Unknown Source)
\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3URI
\tat java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
\tat java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)
\tat java.base/java.lang.Class.forName0(Native Method)
\tat java.base/java.lang.Class.forName(Unknown Source)
\tat clojure.lang.RT.classForName(RT.java:2209)
\tat clojure.lang.RT.classForName(RT.java:2218)
\tat arqam_syncer.arqam.duckdb.DuckDBSyncer.sync_arqam_data(duckdb.clj:170)
\tat arqam_syncer.lambda$process_input_event.invokeStatic(lambda.clj:57)
\tat arqam_syncer.lambda$process_input_event.invoke(lambda.clj:54)
\tat arqam_syncer.lambda$G__10475handleRequest$fn__10476$fn__10481$fn__10482.invoke(lambda.clj:80)
\tat arqam_syncer.jucc$future$reify__144.get(jucc.clj:13)
\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Samuel Ludwig14:11:19

How could I see the protocols that a type implements? and/or rather the types that work with a given protocol? and/or the functions that are part of a given protocol? (my vocabulary might be imperfect)

delaguardo14:11:21

https://clojuredocs.org/clojure.core/ancestors#example-5f41f820e4b0b1e3652d7398 ancestors will give you protocols that type implements Defining a type that implements a protocol does not affect the protocol itself. It is not possible to find a list of types that implement a particular protocol. Protocol itself is a map with :sigs key containing all methods that protocol provides.

Samuel Ludwig14:11:00

Info is appreciated! How can I inspect the protocol as a map? It doesnt seem like I can get it by just eval-ing the protocol as a symbol

delaguardo14:11:09

❯ clj
Clojure 1.11.1
user=> (defprotocol Foo (foo [_]))
Foo
user=> Foo
{:on user.Foo, :on-interface user.Foo, :sigs {:foo {:tag nil, :name foo, :arglists ([_]), :doc nil}}, :var #'user/Foo, :method-map {:foo :foo}, :method-builders {#'user/foo #object[user$eval145$fn__146 0x2b56f5f8 "user$eval145$fn__146@2b56f5f8"]}}

Samuel Ludwig14:11:58

hmm, weird? let me try it in the repl itself like that

delaguardo14:11:42

What editor do you use?

Samuel Ludwig14:11:31

Neovim+Conjure, but weird, in the repl itself, evaling java.io.Serializable just returns the symbol itself, is that not a protocol?

delaguardo14:11:01

yes, it is not. java.io.Serializable is a java interface

Samuel Ludwig14:11:32

ah.., right confirming that the Foo example behaves as expected

Samuel Ludwig14:11:05

is clojure.lang.Seqable an interface as well? is there a way I can discern the two? and perhaps also introspect interfaces?

delaguardo14:11:15

clojure protocols are defined as regular clojure variables within some namespace (eg. user/Foo). If you see a symbol like clojure.lang.Sequable most likely it is java class which can be an interface too.

delaguardo14:11:56

there is a special namespace in clojure - https://clojuredocs.org/clojure.reflect it could be useful to inspect java interface but I didn't use it myself

Samuel Ludwig15:11:18

I appreciate the help- and right I was looking at reflect, would :members be the associated methods and vars?

Samuel Ludwig15:11:37

ahhh ok yes that is what it seems like

Samuel Ludwig15:11:14

I have some further questions, but I'll hold off for now 😄 , I appreciate the assistance gratitude

delaguardo15:11:36

ask away 🙂 I'm sure there are plenty of devs willing to help

Matthew Davidson (kingmob)14:11:56

Is there any documentation on differences between Clojure and Java on lazy-loading of classes? I have a situation where some Netty Java code can "import" a PotentiallyMissingClass for use in SomeFacadeClass, and the parts of SomeFacadeClass that don't use PotentiallyMissingClass work just fine in Java, since the class isn't loaded unless actually used. But with Clojure, I keep getting compilation errors any time I attempt to use any part of SomeFacadeClass. I've tried the standard guard with Class/forName , but that's not fixing the issue. Any attempt to use SomeFacadeClass, even the classes that are guaranteed to exist, results in a CNFE. For the curious, the facade is the StandardCompressionOptions and the PossiblyMissingClass is Brotli4j's com.aayushatharva.brotli4j.encoder.Encoder Why is Clojure trying to load this class before use, and how do I fix the problem?

Alex Miller (Clojure team)14:11:48

I think you’ll need to share some code to say

Matthew Davidson (kingmob)14:11:48

(def opts [(StandardCompressionOptions/deflate)
             (StandardCompressionOptions/gzip)
             (StandardCompressionOptions/snappy)])
These calls are for classes that are built-in to Netty. StandardCompressionOptions:
import com.aayushatharva.brotli4j.encoder.Encoder;

public final class StandardCompressionOptions {
 ...

Matthew Davidson (kingmob)14:11:49

Any attempt to use StandardCompressionOptions results in a CNFE if com.aayushatharva.brotli4j.encoder.Encoder isn't on the classpath (EDIT: use in Clojure, that is)

Matthew Davidson (kingmob)14:11:08

But Netty's Java code itself has no problem using StandardCompressionOptions

Matthew Davidson (kingmob)14:11:45

In HttpContentCompressor:

...
brotliOptions = Brotli.isAvailable() ? StandardCompressionOptions.brotli() : null;
            gzipOptions = StandardCompressionOptions.gzip();
            deflateOptions = StandardCompressionOptions.deflate();
...

Matthew Davidson (kingmob)14:11:19

My best guess is Clojure is importing based on the file's transitive import list, and not usage. Is that true, and if so, is there a way around this?

p-himik15:11:50

When using StandardCompressionOptions, do you actually import it? How?

Matthew Davidson (kingmob)15:11:45

Yup, just like standard:

...
(:import
    (io.netty.channel ChannelHandler)
    (io.netty.handler.codec.compression
      DeflateOptions
      GzipOptions
      SnappyOptions
      StandardCompressionOptions)
   ...)

hiredman15:11:36

That's just not how class loading happens, there is now such thing as an imports list in a class file, when a class file is loaded you don't walk it loading the classes it depends on, etc

Matthew Davidson (kingmob)15:11:45

That's what I thought, and what Java seems to be doing. Not sure what's going wrong on the Clojure side of things

Matthew Davidson (kingmob)15:11:51

But I can confirm that scrubbing all references to optional classes doesn't allow me to call non-optional code paths.

Matthew Davidson (kingmob)15:11:55

If I eval

(def opts [(StandardCompressionOptions/deflate)
             (StandardCompressionOptions/gzip)
             (StandardCompressionOptions/snappy)])
which are all guaranteed on the cp, I still get:
Syntax error (ClassNotFoundException) compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:58:14).
com.aayushatharva.brotli4j.encoder.Encoder$Parameters

*e
=>
#error{:cause "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
       :via [{:type clojure.lang.Compiler$CompilerException,
              :message "Syntax error compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:58:14).",
              :data #:clojure.error{:phase :compile-syntax-check,
                                    :line 58,
                                    :column 14,
                                    :source "/Users/matthew/Code/aleph/src/aleph/http/compression.clj",
                                    :symbol .},
              :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7132]}
             {:type java.lang.NoClassDefFoundError,
              :message "com/aayushatharva/brotli4j/encoder/Encoder$Parameters",
              :at [java.lang.Class getDeclaredMethods0 "Class.java" -2]}
             {:type java.lang.ClassNotFoundException,
              :message "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
              :at [jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]}],
       :trace [[jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]
               [jdk.internal.loader.ClassLoaders$AppClassLoader loadClass "ClassLoaders.java" 188]
               [java.lang.ClassLoader loadClass "ClassLoader.java" 525]

hiredman15:11:34

It doesn't matter the class refers to another class that is not on the classpath

hiredman15:11:47

In its static inits

hiredman15:11:36

So when the static inits are run before they static methods can be called the class is missing

Matthew Davidson (kingmob)15:11:13

OK. Why is Clojure choking, and not Java?

hiredman15:11:31

I am not sure, httpcontentcompressor only appears to refer to the compression stuff in a constructor, not in any static inits, are you constructing it?

Matthew Davidson (kingmob)15:11:13

Yes, but not in this ns. It's only for HTTP1 code, the new compression ns is H2-only

Matthew Davidson (kingmob)15:11:26

There's definitely a lot of static init stuff (see Brotli.class), and they use a version of Class.forName that doesn't initialize the clas, apparently

Matthew Davidson (kingmob)15:11:45

static {
        ClassNotFoundException cnfe = null;

        try {
            Class.forName("com.aayushatharva.brotli4j.Brotli4jLoader", false,
                PlatformDependent.getClassLoader(Brotli.class));
        } catch (ClassNotFoundException t) {
            cnfe = t;
            logger.debug(
                "brotli4j not in the classpath; Brotli support will be unavailable.");
        }

hiredman15:11:41

Ah, it is the Reflector I bet

Matthew Davidson (kingmob)15:11:53

Here's the full stack trace, for anyone who knows the Clj compiler well:

(def opts [(StandardCompressionOptions/deflate)
             (StandardCompressionOptions/gzip)
             (StandardCompressionOptions/snappy)])
Syntax error (ClassNotFoundException) compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:1:117).
com.aayushatharva.brotli4j.encoder.Encoder$Parameters
*e
=>
#error{:cause "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
       :via [{:type clojure.lang.Compiler$CompilerException,
              :message "Syntax error compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:1:117).",
              :data #:clojure.error{:phase :compile-syntax-check,
                                    :line 1,
                                    :column 117,
                                    :source "/Users/matthew/Code/aleph/src/aleph/http/compression.clj",
                                    :symbol .},
              :at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7132]}
             {:type java.lang.NoClassDefFoundError,
              :message "com/aayushatharva/brotli4j/encoder/Encoder$Parameters",
              :at [java.lang.Class getDeclaredMethods0 "Class.java" -2]}
             {:type java.lang.ClassNotFoundException,
              :message "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
              :at [jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]}],
       :trace [[jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]
               [jdk.internal.loader.ClassLoaders$AppClassLoader loadClass "ClassLoaders.java" 188]
               [java.lang.ClassLoader loadClass "ClassLoader.java" 525]
               [java.lang.Class getDeclaredMethods0 "Class.java" -2]
               [java.lang.Class privateGetDeclaredMethods "Class.java" 3402]
               [java.lang.Class privateGetPublicMethods "Class.java" 3427]
               [java.lang.Class getMethods "Class.java" 2019]
               [clojure.lang.Reflector getMethods "Reflector.java" 498]
               [clojure.lang.Compiler$HostExpr$Parser parse "Compiler.java" 994]
               [clojure.lang.Compiler analyzeSeq "Compiler.java" 7124]
               [clojure.lang.Compiler analyze "Compiler.java" 6806]
               [clojure.lang.Compiler analyzeSeq "Compiler.java" 7112]
               [clojure.lang.Compiler analyze "Compiler.java" 6806]
               [clojure.lang.Compiler analyze "Compiler.java" 6762]
               [clojure.lang.Compiler$VectorExpr parse "Compiler.java" 3272]
               [clojure.lang.Compiler analyze "Compiler.java" 6808]
               [clojure.lang.Compiler access$300 "Compiler.java" 38]
               [clojure.lang.Compiler$DefExpr$Parser parse "Compiler.java" 596]
               [clojure.lang.Compiler analyzeSeq "Compiler.java" 7124]
               [clojure.lang.Compiler analyze "Compiler.java" 6806]
               [clojure.lang.Compiler analyze "Compiler.java" 6762]
               [clojure.lang.Compiler eval "Compiler.java" 7198]
               [clojure.lang.Compiler eval "Compiler.java" 7149]
               [clojure.core$eval invokeStatic "core.clj" 3215]
               [clojure.core$eval invoke "core.clj" 3211]
               [nrepl.middleware.interruptible_eval$evaluate$fn__1251$fn__1252 invoke "interruptible_eval.clj" 87]
               [clojure.lang.AFn applyToHelper "AFn.java" 152]
               [clojure.lang.AFn applyTo "AFn.java" 144]
               [clojure.core$apply invokeStatic "core.clj" 667]
               [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
               [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
               [clojure.lang.RestFn invoke "RestFn.java" 425]
               [nrepl.middleware.interruptible_eval$evaluate$fn__1251 invoke "interruptible_eval.clj" 87]
               [clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
               [clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
               [clojure.main$repl$fn__9215 invoke "main.clj" 458]
               [clojure.main$repl invokeStatic "main.clj" 458]
               [clojure.main$repl doInvoke "main.clj" 368]
               [clojure.lang.RestFn invoke "RestFn.java" 1523]
               [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
               [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
               [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1284$fn__1288
                invoke
                "interruptible_eval.clj"
                152]
               [clojure.lang.AFn run "AFn.java" 22]
               [nrepl.middleware.session$session_exec$main_loop__1354$fn__1358 invoke "session.clj" 218]
               [nrepl.middleware.session$session_exec$main_loop__1354 invoke "session.clj" 217]
               [clojure.lang.AFn run "AFn.java" 22]
               [java.lang.Thread run "Thread.java" 840]]}

hiredman15:11:36

Yeah when the compiler is examining the class to determine which methods are available to call it uses java reflection and java reflection to list the methods is requiring the classes for the parameters of those methods to be available

hiredman15:11:20

I would double check and make sure you don't have stuff that is compiling to runtime reflection on standardcompressionoptions, using warn-on-reflection

Matthew Davidson (kingmob)15:11:44

boolean maybeField = RT.length(form) == 3 && (RT.third(form) instanceof Symbol);

			if(maybeField && !(((Symbol)RT.third(form)).name.charAt(0) == '-'))
				{
				Symbol sym = (Symbol) RT.third(form);
				if(c != null)
					maybeField = Reflector.getMethods(c, 0, munge(sym.name), true).size() == 0;
This snippet is from of the stack trace ...it's checking for forms with exactly 3 items...the vec with the options has 3 items after removing the optional ones...that can't be the issue, can it? Nope

p-himik15:11:08

Regular dot access will result in the Reflector usage during compilation time, it doesn't have to be run-time.

Matthew Davidson (kingmob)15:11:17

Aleph always has *warn-on-reflection* set.

hiredman15:11:57

Ah, it has to use reflection to differentiate static fields and methods

p-himik15:11:36

I suppose one way to work around this issue would be to never use the Clojure syntax for field/method access with that particular class and instead to always use Class.getMethod yourself.

p-himik15:11:21

Another way would be to create your own Java wrapper around the problematic class, and then use regular Clojure syntax.

Matthew Davidson (kingmob)15:11:42

@U2FRKM4TW Thx for those suggestions. I take it you think the dot syntax won't work? Dot syntax doesnt help

Matthew Davidson (kingmob)15:11:58

@U0NCTKEV8 Thanks for helping me debug this

p-himik15:11:18

The dot syntax forces the compiler to use getMethods. getMethods needs all the mentioned classes to be present.

Matthew Davidson (kingmob)15:11:23

OK, I'll try one of the other methods. Luckily, this is one-time initialization code, slower alternatives should be fine

Matthew Davidson (kingmob)15:11:38

Thank you both. 🙏

👍 1
hiredman15:11:46

So the difference between clojure in java here is java's compiler is running in a separate process and may have a different set of dependencies then the code has at runtime

Matthew Davidson (kingmob)16:11:59

.getMethod reflection still gives me CNFEs, probably because it still has to load all methods to find the one I'm asking for.

Matthew Davidson (kingmob)16:11:47

Let's see if a wrapper works. If not, I'm going to just enable all compression types for Aleph, rather than jump through more hoops or inflict this on users

Matthew Davidson (kingmob)16:11:38

It's late in my TZ, will try tomorrow

Matthew Davidson (kingmob)10:11:11

The wrapper class worked!

🎉 1
roklenarcic15:11:23

I have a macro which would like to create/initialize an expensive object at macro expansion time, but have the code macro creates use it. How would i do that?

dvorme15:11:17

If by "object" you mean some big hairy nested Clojure data structure, then your macro needs to expand to the Clojure literal that defines that "object".

roklenarcic15:11:46

no it’s an opaque Java object

delaguardo15:11:51

macro emits code (or data structures). If you object is some java object then macro can't help here

thumbsup_all 2
dvorme15:11:42

If the Java object has mutators that can set its state, (e.g.: a bunch of nested beans), then your macro could expand to the code that instantiates that Java object graph and sets the objects' properties correctly.

roklenarcic15:11:38

That’s not really what I asked for

roklenarcic15:11:59

I want to avoid doing the work during each run of the code that is expanded from the macro

dvorme15:11:37

The macro runs at compile time. Its output is more Clojure code. The macro could definitely

(let [result (some-expensive-operation)

dvorme15:11:43

But then the macro has to serialize result into Clojure code that can recreate result at runtime.

roklenarcic15:11:56

I can think of a couple of ways already

dvorme15:11:26

👍 Awesome!

roklenarcic15:11:06

Is it possible to declare/create a Var in another namespace?

roklenarcic15:11:21

ok and what’s the best way to set a value to it?

roklenarcic15:11:38

I see that it has an arity for that

roklenarcic16:11:44

Looking for a way to do it only if not yet defined

Joshua Suskalo18:11:06

You can use resolve plus intern

Joshua Suskalo18:11:05

(when-not (ns-resolve (find-ns 'the-ns) 'the-sym)
  (intern (find-ns 'the-ns) 'the-sym some-value))