This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-11-03
Channels
- # announcements (55)
- # babashka (14)
- # beginners (49)
- # biff (9)
- # calva (73)
- # cider (14)
- # clerk (8)
- # clj-kondo (6)
- # clojure (92)
- # clojure-dev (4)
- # clojure-europe (18)
- # clojure-norway (62)
- # clojure-uk (5)
- # clojuredesign-podcast (4)
- # clojurescript (34)
- # cursive (40)
- # data-science (4)
- # datomic (5)
- # dev-tooling (1)
- # eastwood (6)
- # emacs (107)
- # figwheel-main (9)
- # fulcro (13)
- # gratitude (9)
- # hyperfiddle (5)
- # introduce-yourself (2)
- # off-topic (45)
- # overtone (23)
- # portal (5)
- # releases (3)
- # shadow-cljs (6)
- # specter (1)
- # squint (32)
- # timbre (4)
- # vscode (2)
I have the code below running in an aws lambda handler
(let [read-json #(json/read-value % json/keyword-keys-object-mapper)
parse-body (fn [rec]
(try
(read-json (:body rec))
(catch Exception e
(u/log ::invalid-body :exception e))))
raw-in (read-json in)
records (->> raw-in
:Records
(map parse-body))]
(run! process-input-event records)
#_(let [tasks (map #(future
(process-input-event %))
records)]
(map deref tasks)))
When it processes the records in serial everything works fine. However, when I change it to use the commented out version that tries to launch a future for each record, I get a weird ClassNotFound error because it can’t find com.amazonaws.services.s3.AmazonS3URI
that is used somewhere in the definition of process-input-event
. I don’t understand how just adding the concurrency can lead to the class not being found when it is found no problem without the concurrency.
I checked the uberjar and it definitely includes the class file where you’d expect to find itrun!
is eager, map
is lazy. No clue what's going on with "class not found", but have you tried (run! deref tasks)
? While commenting out the other run!
form, of course.
Sorry about the \t
s. It was all on one line and I did a bit of sed to get it more readable but didn’t bother dealing with the tabs.
java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3URI
\tat java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
\tat java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
\tat clojure.core$deref_future.invokeStatic(core.clj:2317)
\tat clojure.core$deref.invokeStatic(core.clj:2337)
\tat clojure.core$deref.invoke(core.clj:2323)
\tat arqam_syncer.lambda$G__10475handleRequest$fn__10476.invoke(lambda.clj:79)
\tat arqam_syncer.logging$with_logging.invokeStatic(logging.clj:29)
\tat arqam_syncer.logging$with_logging.invoke(logging.clj:23)
\tat arqam_syncer.lambda$G__10475handleRequest.invokeStatic(lambda.clj:63)
\tat arqam_syncer.lambda$G__10475handleRequest.invoke(lambda.clj:61)
\tat arqam-syncer.lambda.ArqamSyncer.handleRequest(Unknown Source)
\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
\tat java.base/java.lang.reflect.Method.invoke(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.services.s3.AmazonS3URI
\tat java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
\tat java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
\tat java.base/java.lang.ClassLoader.loadClass(Unknown Source)
\tat java.base/java.lang.Class.forName0(Native Method)
\tat java.base/java.lang.Class.forName(Unknown Source)
\tat clojure.lang.RT.classForName(RT.java:2209)
\tat clojure.lang.RT.classForName(RT.java:2218)
\tat arqam_syncer.arqam.duckdb.DuckDBSyncer.sync_arqam_data(duckdb.clj:170)
\tat arqam_syncer.lambda$process_input_event.invokeStatic(lambda.clj:57)
\tat arqam_syncer.lambda$process_input_event.invoke(lambda.clj:54)
\tat arqam_syncer.lambda$G__10475handleRequest$fn__10476$fn__10481$fn__10482.invoke(lambda.clj:80)
\tat arqam_syncer.jucc$future$reify__144.get(jucc.clj:13)
\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
\tat java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
How could I see the protocols that a type implements? and/or rather the types that work with a given protocol? and/or the functions that are part of a given protocol? (my vocabulary might be imperfect)
https://clojuredocs.org/clojure.core/ancestors#example-5f41f820e4b0b1e3652d7398
ancestors
will give you protocols that type implements
Defining a type that implements a protocol does not affect the protocol itself. It is not possible to find a list of types that implement a particular protocol.
Protocol itself is a map with :sigs
key containing all methods that protocol provides.
Info is appreciated! How can I inspect the protocol as a map? It doesnt seem like I can get it by just eval-ing the protocol as a symbol
❯ clj
Clojure 1.11.1
user=> (defprotocol Foo (foo [_]))
Foo
user=> Foo
{:on user.Foo, :on-interface user.Foo, :sigs {:foo {:tag nil, :name foo, :arglists ([_]), :doc nil}}, :var #'user/Foo, :method-map {:foo :foo}, :method-builders {#'user/foo #object[user$eval145$fn__146 0x2b56f5f8 "user$eval145$fn__146@2b56f5f8"]}}
hmm, weird? let me try it in the repl itself like that
What editor do you use?
Neovim+Conjure, but weird, in the repl itself, evaling java.io.Serializable
just returns the symbol itself, is that not a protocol?
yes, it is not. java.io.Serializable
is a java interface
ah.., right confirming that the Foo
example behaves as expected
is clojure.lang.Seqable
an interface as well? is there a way I can discern the two? and perhaps also introspect interfaces?
clojure protocols are defined as regular clojure variables within some namespace (eg. user/Foo
). If you see a symbol like clojure.lang.Sequable
most likely it is java class which can be an interface too.
there is a special namespace in clojure - https://clojuredocs.org/clojure.reflect it could be useful to inspect java interface but I didn't use it myself
for example
I appreciate the help- and right I was looking at reflect
, would :members
be the associated methods and vars?
ahhh ok yes that is what it seems like
I have some further questions, but I'll hold off for now 😄 , I appreciate the assistance
ask away 🙂 I'm sure there are plenty of devs willing to help
Is there any documentation on differences between Clojure and Java on lazy-loading of classes?
I have a situation where some Netty Java code can "import" a PotentiallyMissingClass for use in SomeFacadeClass, and the parts of SomeFacadeClass that don't use PotentiallyMissingClass work just fine in Java, since the class isn't loaded unless actually used.
But with Clojure, I keep getting compilation errors any time I attempt to use any part of SomeFacadeClass.
I've tried the standard guard with Class/forName
, but that's not fixing the issue. Any attempt to use SomeFacadeClass, even the classes that are guaranteed to exist, results in a CNFE.
For the curious, the facade is the StandardCompressionOptions and the PossiblyMissingClass is Brotli4j's com.aayushatharva.brotli4j.encoder.Encoder
Why is Clojure trying to load this class before use, and how do I fix the problem?
I think you’ll need to share some code to say
(def opts [(StandardCompressionOptions/deflate)
(StandardCompressionOptions/gzip)
(StandardCompressionOptions/snappy)])
These calls are for classes that are built-in to Netty.
StandardCompressionOptions:
import com.aayushatharva.brotli4j.encoder.Encoder;
public final class StandardCompressionOptions {
...
Any attempt to use StandardCompressionOptions results in a CNFE if com.aayushatharva.brotli4j.encoder.Encoder isn't on the classpath (EDIT: use in Clojure, that is)
But Netty's Java code itself has no problem using StandardCompressionOptions
In HttpContentCompressor
:
...
brotliOptions = Brotli.isAvailable() ? StandardCompressionOptions.brotli() : null;
gzipOptions = StandardCompressionOptions.gzip();
deflateOptions = StandardCompressionOptions.deflate();
...
My best guess is Clojure is importing based on the file's transitive import list, and not usage. Is that true, and if so, is there a way around this?
LMK if you need more
Yup, just like standard:
...
(:import
(io.netty.channel ChannelHandler)
(io.netty.handler.codec.compression
DeflateOptions
GzipOptions
SnappyOptions
StandardCompressionOptions)
...)
That's just not how class loading happens, there is now such thing as an imports list in a class file, when a class file is loaded you don't walk it loading the classes it depends on, etc
That's what I thought, and what Java seems to be doing. Not sure what's going wrong on the Clojure side of things
But I can confirm that scrubbing all references to optional classes doesn't allow me to call non-optional code paths.
If I eval
(def opts [(StandardCompressionOptions/deflate)
(StandardCompressionOptions/gzip)
(StandardCompressionOptions/snappy)])
which are all guaranteed on the cp, I still get:
Syntax error (ClassNotFoundException) compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:58:14).
com.aayushatharva.brotli4j.encoder.Encoder$Parameters
*e
=>
#error{:cause "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
:via [{:type clojure.lang.Compiler$CompilerException,
:message "Syntax error compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:58:14).",
:data #:clojure.error{:phase :compile-syntax-check,
:line 58,
:column 14,
:source "/Users/matthew/Code/aleph/src/aleph/http/compression.clj",
:symbol .},
:at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7132]}
{:type java.lang.NoClassDefFoundError,
:message "com/aayushatharva/brotli4j/encoder/Encoder$Parameters",
:at [java.lang.Class getDeclaredMethods0 "Class.java" -2]}
{:type java.lang.ClassNotFoundException,
:message "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
:at [jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]}],
:trace [[jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]
[jdk.internal.loader.ClassLoaders$AppClassLoader loadClass "ClassLoaders.java" 188]
[java.lang.ClassLoader loadClass "ClassLoader.java" 525]
So when the static inits are run before they static methods can be called the class is missing
OK. Why is Clojure choking, and not Java?
I am not sure, httpcontentcompressor only appears to refer to the compression stuff in a constructor, not in any static inits, are you constructing it?
Yes, but not in this ns. It's only for HTTP1 code, the new compression ns is H2-only
There's definitely a lot of static init stuff (see Brotli.class), and they use a version of Class.forName that doesn't initialize the clas, apparently
static {
ClassNotFoundException cnfe = null;
try {
Class.forName("com.aayushatharva.brotli4j.Brotli4jLoader", false,
PlatformDependent.getClassLoader(Brotli.class));
} catch (ClassNotFoundException t) {
cnfe = t;
logger.debug(
"brotli4j not in the classpath; Brotli support will be unavailable.");
}
(in Brotli)
Here's the full stack trace, for anyone who knows the Clj compiler well:
(def opts [(StandardCompressionOptions/deflate)
(StandardCompressionOptions/gzip)
(StandardCompressionOptions/snappy)])
Syntax error (ClassNotFoundException) compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:1:117).
com.aayushatharva.brotli4j.encoder.Encoder$Parameters
*e
=>
#error{:cause "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
:via [{:type clojure.lang.Compiler$CompilerException,
:message "Syntax error compiling . at (/Users/matthew/Code/aleph/src/aleph/http/compression.clj:1:117).",
:data #:clojure.error{:phase :compile-syntax-check,
:line 1,
:column 117,
:source "/Users/matthew/Code/aleph/src/aleph/http/compression.clj",
:symbol .},
:at [clojure.lang.Compiler analyzeSeq "Compiler.java" 7132]}
{:type java.lang.NoClassDefFoundError,
:message "com/aayushatharva/brotli4j/encoder/Encoder$Parameters",
:at [java.lang.Class getDeclaredMethods0 "Class.java" -2]}
{:type java.lang.ClassNotFoundException,
:message "com.aayushatharva.brotli4j.encoder.Encoder$Parameters",
:at [jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]}],
:trace [[jdk.internal.loader.BuiltinClassLoader loadClass "BuiltinClassLoader.java" 641]
[jdk.internal.loader.ClassLoaders$AppClassLoader loadClass "ClassLoaders.java" 188]
[java.lang.ClassLoader loadClass "ClassLoader.java" 525]
[java.lang.Class getDeclaredMethods0 "Class.java" -2]
[java.lang.Class privateGetDeclaredMethods "Class.java" 3402]
[java.lang.Class privateGetPublicMethods "Class.java" 3427]
[java.lang.Class getMethods "Class.java" 2019]
[clojure.lang.Reflector getMethods "Reflector.java" 498]
[clojure.lang.Compiler$HostExpr$Parser parse "Compiler.java" 994]
[clojure.lang.Compiler analyzeSeq "Compiler.java" 7124]
[clojure.lang.Compiler analyze "Compiler.java" 6806]
[clojure.lang.Compiler analyzeSeq "Compiler.java" 7112]
[clojure.lang.Compiler analyze "Compiler.java" 6806]
[clojure.lang.Compiler analyze "Compiler.java" 6762]
[clojure.lang.Compiler$VectorExpr parse "Compiler.java" 3272]
[clojure.lang.Compiler analyze "Compiler.java" 6808]
[clojure.lang.Compiler access$300 "Compiler.java" 38]
[clojure.lang.Compiler$DefExpr$Parser parse "Compiler.java" 596]
[clojure.lang.Compiler analyzeSeq "Compiler.java" 7124]
[clojure.lang.Compiler analyze "Compiler.java" 6806]
[clojure.lang.Compiler analyze "Compiler.java" 6762]
[clojure.lang.Compiler eval "Compiler.java" 7198]
[clojure.lang.Compiler eval "Compiler.java" 7149]
[clojure.core$eval invokeStatic "core.clj" 3215]
[clojure.core$eval invoke "core.clj" 3211]
[nrepl.middleware.interruptible_eval$evaluate$fn__1251$fn__1252 invoke "interruptible_eval.clj" 87]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[nrepl.middleware.interruptible_eval$evaluate$fn__1251 invoke "interruptible_eval.clj" 87]
[clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
[clojure.main$repl$fn__9215 invoke "main.clj" 458]
[clojure.main$repl invokeStatic "main.clj" 458]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__1284$fn__1288
invoke
"interruptible_eval.clj"
152]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__1354$fn__1358 invoke "session.clj" 218]
[nrepl.middleware.session$session_exec$main_loop__1354 invoke "session.clj" 217]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 840]]}
Yeah when the compiler is examining the class to determine which methods are available to call it uses java reflection and java reflection to list the methods is requiring the classes for the parameters of those methods to be available
I would double check and make sure you don't have stuff that is compiling to runtime reflection on standardcompressionoptions, using warn-on-reflection
boolean maybeField = RT.length(form) == 3 && (RT.third(form) instanceof Symbol);
if(maybeField && !(((Symbol)RT.third(form)).name.charAt(0) == '-'))
{
Symbol sym = (Symbol) RT.third(form);
if(c != null)
maybeField = Reflector.getMethods(c, 0, munge(sym.name), true).size() == 0;
This snippet is from of the stack trace
Regular dot access will result in the Reflector usage during compilation time, it doesn't have to be run-time.
Aleph always has *warn-on-reflection*
set.
I suppose one way to work around this issue would be to never use the Clojure syntax for field/method access with that particular class and instead to always use Class.getMethod
yourself.
Another way would be to create your own Java wrapper around the problematic class, and then use regular Clojure syntax.
@U2FRKM4TW Thx for those suggestions. I take it you think the dot syntax won't work? Dot syntax doesnt help
@U0NCTKEV8 Thanks for helping me debug this
The dot syntax forces the compiler to use getMethods
.
getMethods
needs all the mentioned classes to be present.
OK, I'll try one of the other methods. Luckily, this is one-time initialization code, slower alternatives should be fine
So the difference between clojure in java here is java's compiler is running in a separate process and may have a different set of dependencies then the code has at runtime
.getMethod
reflection still gives me CNFEs, probably because it still has to load all methods to find the one I'm asking for.
Let's see if a wrapper works. If not, I'm going to just enable all compression types for Aleph, rather than jump through more hoops or inflict this on users
It's late in my TZ, will try tomorrow
I have a macro which would like to create/initialize an expensive object at macro expansion time, but have the code macro creates use it. How would i do that?
If by "object" you mean some big hairy nested Clojure data structure, then your macro needs to expand to the Clojure literal that defines that "object".
no it’s an opaque Java object
macro emits code (or data structures). If you object is some java object then macro can't help here

If the Java object has mutators that can set its state, (e.g.: a bunch of nested beans), then your macro could expand to the code that instantiates that Java object graph and sets the objects' properties correctly.
That’s not really what I asked for
I want to avoid doing the work during each run of the code that is expanded from the macro
The macro runs at compile time. Its output is more Clojure code. The macro could definitely
(let [result (some-expensive-operation)
But then the macro has to serialize result
into Clojure code that can recreate result
at runtime.
I can think of a couple of ways already
Is it possible to declare/create a Var in another namespace?
ok and what’s the best way to set a value to it?
sorry
I see that it has an arity for that
Looking for a way to do it only if not yet defined
You can use resolve
plus intern
(when-not (ns-resolve (find-ns 'the-ns) 'the-sym)
(intern (find-ns 'the-ns) 'the-sym some-value))