Fork me on GitHub
#clojure
<
2021-07-21
>
Karol Wójcik06:07:05

I was able to create a very minimal repro for the error which is thrown when running local rapid java 11 runtime for AWS Lambda with Clojure 1.10.3. Strange thing is that Clojure 1.10.0 does not produce any error. Deployed application using two versions does not throw the error on AWS Lambda real service as well. Here is the repro: https://github.com/FieryCod/java-11-8-aws-sam-2787 Error:

java.lang.ExceptionInInitializerError
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)
        at clojure.lang.RT.classForName(RT.java:2212)
        at clojure.lang.RT.classForName(RT.java:2221)
        at clojure.lang.RT.loadClassForName(RT.java:2240)
        at clojure.lang.RT.load(RT.java:449)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6856.invoke(core.clj:6115)
        at clojure.core$load.invokeStatic(core.clj:6114)
        at clojure.core$load.doInvoke(core.clj:6098)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5897)
        at clojure.core$load_one.invoke(core.clj:5892)
        at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
        at clojure.core$load_lib.invokeStatic(core.clj:5936)
        at clojure.core$load_lib.doInvoke(core.clj:5917)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:5974)
        at clojure.core$load_libs.doInvoke(core.clj:5958)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:5996)
        at clojure.main$loading__6737__auto____8998.invoke(main.clj:11)
        at clojure.main__init.load(Unknown Source)
        at clojure.main__init.<clinit>(Unknown Source)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)
        at clojure.lang.RT.classForName(RT.java:2212)
        at clojure.lang.RT.classForName(RT.java:2221)
        at clojure.lang.RT.loadClassForName(RT.java:2240)
        at clojure.lang.RT.load(RT.java:449)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6856.invoke(core.clj:6115)
        at clojure.core$load.invokeStatic(core.clj:6114)
        at clojure.core$load.doInvoke(core.clj:6098)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5897)
        at clojure.core$load_one.invoke(core.clj:5892)
        at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
        at clojure.core$load_lib.invokeStatic(core.clj:5936)
        at clojure.core$load_lib.doInvoke(core.clj:5917)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:5974)
        at clojure.core$load_libs.doInvoke(core.clj:5958)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:5996)
        at clojure.core.server$loading__6737__auto____8865.invoke(server.clj:9)
        at clojure.core.server__init.load(Unknown Source)
        at clojure.core.server__init.<clinit>(Unknown Source)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)
        at clojure.lang.RT.classForName(RT.java:2212)
        at clojure.lang.RT.classForName(RT.java:2221)
        at clojure.lang.RT.loadClassForName(RT.java:2240)
        at clojure.lang.RT.load(RT.java:449)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6856.invoke(core.clj:6115)
        at clojure.core$load.invokeStatic(core.clj:6114)
        at clojure.core$load.doInvoke(core.clj:6098)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5897)
        at clojure.core$load_one.invoke(core.clj:5892)
        at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
        at clojure.core$load_lib.invokeStatic(core.clj:5936)
        at clojure.core$load_lib.doInvoke(core.clj:5917)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:5974)
        at clojure.core$load_libs.doInvoke(core.clj:5958)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:5996)
        at clojure.core$require.doInvoke(core.clj:5996)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.lang.Var.invoke(Var.java:384)
        at clojure.lang.RT.doInit(RT.java:491)
        at clojure.lang.RT.init(RT.java:467)
        at clojure.lang.Util.loadWithClass(Util.java:248)
        at example.lambda.<clinit>(Unknown Source)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Unknown Source)
Caused by: Syntax error macroexpanding clojure.core/ns at (clojure/walk.clj:21:1).
Syntax error compiling at (clojure/core/specs/alpha.clj:1:1).
        at clojure.lang.Compiler.checkSpecs(Compiler.java:6976)
        at clojure.lang.Compiler.macroexpand1(Compiler.java:6992)
        at clojure.lang.Compiler.macroexpand(Compiler.java:7079)
        at clojure.lang.Compiler.eval(Compiler.java:7165)
        at clojure.lang.Compiler.load(Compiler.java:7640)
        at clojure.lang.RT.loadResourceScript(RT.java:381)
        at clojure.lang.RT.loadResourceScript(RT.java:372)
        at clojure.lang.RT.load(RT.java:459)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.core$load$fn__6856.invoke(core.clj:6115)
        at clojure.core$load.invokeStatic(core.clj:6114)
        at clojure.core$load.doInvoke(core.clj:6098)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5897)
        at clojure.core$load_one.invoke(core.clj:5892)
        at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
        at clojure.core$load_lib.invokeStatic(core.clj:5936)
        at clojure.core$load_lib.doInvoke(core.clj:5917)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:5974)
        at clojure.core$load_libs.doInvoke(core.clj:5958)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:5996)
        at clojure.core$require.doInvoke(core.clj:5996)
        at clojure.lang.RestFn.invoke(RestFn.java:436)
        at clojure.spec.alpha$loading__6721__auto____1654.invoke(alpha.clj:9)
        at clojure.spec.alpha__init.load(Unknown Source)
        at clojure.spec.alpha__init.<clinit>(Unknown Source)
        ... 84 more
Caused by: Syntax error compiling at (clojure/core/specs/alpha.clj:1:1).
        at clojure.core$throw_if.invokeStatic(core.clj:5856)
        at clojure.core$check_cyclic_dependency.invokeStatic(core.clj:5988)
        at clojure.core$load.invokeStatic(core.clj:6109)
        at clojure.core$load.doInvoke(core.clj:6098)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5897)
        at clojure.core$load_one.invoke(core.clj:5892)
        at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
        at clojure.core$load_lib.invokeStatic(core.clj:5936)
        at clojure.core$load_lib.doInvoke(core.clj:5917)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$load_libs.invokeStatic(core.clj:5974)
        at clojure.core$load_libs.doInvoke(core.clj:5958)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$require.invokeStatic(core.clj:5996)
        at clojure.core$require.doInvoke(core.clj:5996)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core.specs.alpha$eval1397$loading__6737__auto____1398.invoke(alpha.clj:1)
        at clojure.core.specs.alpha$eval1397.invokeStatic(alpha.clj:1)
        at clojure.core.specs.alpha$eval1397.invoke(alpha.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7181)
        at clojure.lang.Compiler.eval(Compiler.java:7170)
        at clojure.lang.Compiler.load(Compiler.java:7640)
        at clojure.lang.RT.loadResourceScript(RT.java:381)
        at clojure.lang.RT.loadResourceScript(RT.java:372)
        at clojure.lang.RT.load(RT.java:459)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.lang.Compiler.ensureMacroCheck(Compiler.java:6962)
        at clojure.lang.Compiler.checkSpecs(Compiler.java:6974)
        ... 113 more
Caused by: java.lang.Exception: Cyclic load dependency: [ /clojure/spec/alpha ]->/clojure/walk->[ /clojure/spec/alpha ]->/clojure/main->/clojure/core/server
        ... 139 more
EDIT: Have you seen anything like that @alexmiller?

Alex Miller (Clojure team)11:07:35

This is thrown while you’re building the uberjar or at runtime?

Alex Miller (Clojure team)11:07:12

I have seen stuff like this when compiling spec because Clojure depends on spec and spec depends on Clojure so you have to be somewhat careful in the bootstrap. Two things that changed between 1.10.0 and 1.10.3 - 1) in 1.10.1 - initialization paths changed to work around a static initializer regression in Java. Depending on how the Clojure runtime is initialized, this could be something. 2) 1.10.2 changed compilation of spec from Java 5 bytecode to Java 8.

Alex Miller (Clojure team)11:07:36

My suspicion would be the first one but you could retry with 1.10.1 to narrow it down If it fails with 1.10.1 too I would strongly suspect it’s something with how the Clojure runtime is getting initialized

Karol Wójcik12:07:12

This is on runtime and I don't have no control how the code is run. https://github.com/aws/aws-sam-cli/issues/2787

Karol Wójcik12:07:20

I will check 1.10.1

Karol Wójcik12:07:19

1.10.1 starts the error happening

Karol Wójcik12:07:09

Do you think I can add more info to the issue to help AWS team resolve it?

Alex Miller (Clojure team)12:07:29

there are way too many holes in my understanding of what's actually happening here to give you good ideas, but ... if you can pass a Java system property, I'd try using -Dclojure.spec.skip-macros=true to turn off spec macro checking - that might allow things at least to proceed the error "Syntax error compiling at" is actually wrong (there is a ticket for this I've been working on) and is an artifact of how the exception is flowing through load, and it's not compiling

Alex Miller (Clojure team)12:07:51

my big question is where code is first calling into Clojure or something compiled with Clojure and how it's doing it. that code likely needs adjustment due to changes in 1.10.1

Alex Miller (Clojure team)12:07:26

might be example.lambda.<clinit> ?

Alex Miller (Clojure team)13:07:11

if I understand, example.lambda is compiling into a class that implements com.amazonaws.services.lambda.runtime.RequestStreamHandler, which I presume is constructed by the lambda stuff somehow

Alex Miller (Clojure team)13:07:28

it does seem like RT.init() is getting called in your stack though, so that's good

Alex Miller (Clojure team)13:07:42

I'd try the system property if you can

Karol Wójcik13:07:32

I'm unable to set the system property 😞

Karol Wójcik13:07:58

The code in run in docker by AWS SAM

Alex Miller (Clojure team)13:07:02

not sure what else I can suggest

Karol Wójcik14:07:50

I will ask AWS team whether they can inject the mentioned above system property. Thank you so much Alex. I know it's hard to help when I'm unable to test it locally. Unfortunately AWS local Java runtime is not open-source, so all I can do is to paste exactly what you have said.

Karol Wójcik14:07:43

Last question though, may I mention you in this issue? https://github.com/aws/aws-sam-cli/issues/2787

didibus16:07:46

Do you also AOT things, like the jar you give them to run?

didibus16:07:49

I've seen in the past a case where if you do AOT, everything will be AOTed even the specs. So if you AOT with an old Clojure the old spec will be in the Jar as a .class This is true even if you depend on an AOT lib or anything that did AOT. So now if you try to use a newer Clojure, it will use the old specs found in the .class over the source ones found in the clojure dependency. Don't know if that could in any way relate, but just thought I'd share in case

zendevil.eth11:07:07

I have a java file in my lein project, and the source dir has been added to java-source-paths in project.clj. However, on lein repl, I’m getting a ClassNotFoundError where I import it in my clojure project. But if I comment out the import do lein repl and then load the import in the repl, it works. How to make it work on lein repl?

zendevil.eth11:07:50

Also the java method that I’m calling returns a String, but when I’m calling it from the clojure code, it is returning nil

Karen Liu14:07:16

Hi! I have this program I'm working on that runs for quite a while (a few hours to a couple days), but we recently found out that each run of this program has a much higher memory usage than we think it should (10-12 GB). We suspect that it might be an issue of head retention, but we can't see where or how that would be the case. Is there a way to figure out where the memory leak is happening? Like a Clojure-specific profiling tool, etc.?

Alex Miller (Clojure team)15:07:58

I don't know of a Clojure-specific memory profiling tool, but you can use the JVM profiling tools like YourKit, JProfiler, etc

Alex Miller (Clojure team)15:07:33

YourKit has a tool to take and analyze heap snapshots, diff heap snapshots, and to trace object references back to the root that's holding it

Alex Miller (Clojure team)15:07:28

doing that with head retention is somewhat messy, sometimes I find the heap snapshot differ to be useful in getting ideas to investigate, especially if you can snapshot before and after one "increment" of work, whatever that is

Alex Miller (Clojure team)15:07:42

find whatever's object count has gone up, then trace instances back to root. the downside with clojure is usually it's all maps and vectors and you're looking at the raw structures.

Karen Liu20:07:41

Thank you so much!

denik15:07:25

does anyone know how to run a socket REPL from a local docker container and REPL into it from outside? I have issues connecting even with ports exposed

ghadi15:07:09

@denik paste your socket repl param map

denik15:07:21

(clojure.core.server/start-server
            {:name          "socket-repl"
             :port          4450 #_(:socket-port *command-line-args*)
             :accept        'clojure.core.server/repl
             :server-daemon false})

denik15:07:15

hmm what should this be in docker?

ghadi15:07:16

you're missing :address "0.0.0.0" to listen on non-localhost

ghadi15:07:33

it's listening on the localhost within the container, which is not exposed to the host

denik15:07:33

thanks will try!

denik16:07:10

perfect it worked!

denik16:07:14

thanks so much @ghadi

ghadi16:07:00

my pleasure

Jim Newton15:07:15

I get this error far too often:

Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol
it would be great if the error message would tell me which symbol it is rather than just its class

denik15:07:51

@ghadi the socket repl works when I run it with CLJ but when I run it from inside docker it breaks down

ghadi15:07:16

you've said that twice now 🙂 paste your args

👍 4
zendevil.eth16:07:00

I’m trying to interop with the following java code: https://github.com/web3j/web3j/blob/49fe2c4e2d9d325ec465879736d6c384f41a4115/crypto/src/main/java/org/web3j/crypto/Credentials.java#L41

(:import
  (org.web3j.crypto Credentials))

(.create Credentials              "9023c589167652b2182da32c2352c25c88cc1032f725b40e9a02e9f4162e1cc2"               "0xB00d16842187bF6f5e8312C94AF192f14a199489")
But am getting: No matching method create found taking 2 args for class java.lang.Class Clojure: class java.lang.IllegalArgumentException How to fix this?

Russell Mull16:07:45

Static method access works differently; you probably want (Credentials/create priv pub)

Russell Mull16:07:51

https://clojure.org/reference/java_interop lays it all out, it's the second to last one under 'Member access'

zendevil.eth16:07:13

I’m trying to call a java function with the following definition:

deploy(Web3j web3j, Credentials credentials, BigInteger gasPri  ce, BigInteger gasLimit, String _creator, String _creationId, String _creatorId, BigInteger _bidEnd)
And I have the following:
210 (Creation/deploy 
211   web3j 
212   (Credentials/create 
213     "9023c589167652b2182da32c2352c25c88cc1032f725b40e9a02e9f4162e1cc2"
214     "0xB00d16842187bF6f5e8312C94AF192f14a199489")
215   (Numeric/decodeQuantity "1000") 
216   (Numeric/decodeQuantity "1000")
217   "creator-address-1"
218   "creation-1"
219   "creator-1"
220   (Numeric/decodeQuantity "1626885270000"))

zendevil.eth16:07:50

But I’m getting the error:

Illegal embedded sign character
Clojure: class java.lang.NumberFormatException

zendevil.eth16:07:13

individually, the Numeric conversions aren’t giving any errors

dpsutton16:07:22

what's the stacktrace?

Joshua Suskalo16:07:50

Yeah, this sounds like it's coming from somewhere deeper in the java lib

Linus Ericsson17:07:52

One of the hex encoded numbers have 0x-prefix, the other not. The letters in these strings are both upper- and lowercased. What does the (Credentials/create ... ...) render on it's own?

zendevil.eth17:07:54

Those are supposed to be eth wallet public and private keys, not numbers and they are in the correct String type

zendevil.eth17:07:23

@dpsutton @suskeyhose

1 java.math.BigInteger/<init>|428| java                                                                
  2 org.web3j.utils.Numeric/toBigIntNoPrefix|125| java
  3 org.web3j.utils.Numeric/toBigInt|121| java
  4 org.web3j.abi.datatypes.Address/<init>|47| java
  5 com.humboi.smart.contract.Creation/deploy|177| java
  6 sun.reflect.NativeMethodAccessorImpl/invoke0|| java
  7 sun.reflect.NativeMethodAccessorImpl/invoke|62| java
  8 sun.reflect.DelegatingMethodAccessorImpl/invoke|43| java
  9 java.lang.reflect.Method/invoke|498| java
 10 clojure.lang.Reflector/invokeMatchingMethod|167| java
  8 sun.reflect.DelegatingMethodAccessorImpl/invoke|43| java
  9 java.lang.reflect.Method/invoke|498| java
 10 clojure.lang.Reflector/invokeMatchingMethod|167| java
 11 clojure.lang.Reflector/invokeStaticMethod|332| java
 12 clojure.core/eval|3202| clj
 13 clojure.core/eval|3198| clj
 14 clojure.lang.AFn/applyToHelper|152| java
 15 clojure.lang.AFn/applyTo|144| java
 16 clojure.core/apply|667| clj
 17 clojure.core/with-bindings*|1977| clj
18 clojure.lang.RestFn/invoke|425| java
 19 clojure.main/repl|437| clj
 20 clojure.main/repl|458| clj
 21 clojure.main/repl|368| clj
 22 clojure.lang.RestFn/invoke|1523| java
 23 clojure.lang.AFn/run|22| java
 24 clojure.lang.AFn/run|22| java
 25 java.lang.Thread/run|748| java

hiredman18:07:37

looks like "creator-address-1" needs to be something parseable as a number

hiredman19:07:43

It is a problem because the big integer constructor only allows - (the sign indicator) at the start of a string

Martynas Maciulevičius20:07:47

Come on. It's plain old java, of course the constructor is limited to something from the past. Why would it support it? Also I think it expects base 10 anyway and your letters are not base 10.

hiredman18:07:25

user=> (BigInteger. "creator-address-1")
Execution error (NumberFormatException) at java.math.BigInteger/<init> (BigInteger.java:494).
Illegal embedded sign character
user=>

zendevil.eth06:07:36

But as you can see from the java function signature, that part is supposed to be a string.

zendevil.eth06:07:52

I think it might be because that function signature is marked @deprecated

zendevil.eth06:07:41

Figured it out. It was because creator was supposed to be a hex string and couldn’t be creator-address-1

ghadi18:07:07

@ps: always include the full stacktrace - otherwise it's not debugging it's https://en.wikipedia.org/wiki/Miss_Cleo

😂 6
Ben Sless19:07:28

Add an unhandled exception handler which prepends to the message sting CALL ME NOW!

😂 2
zendevil.eth06:07:16

I’m using vim fireplace and copied everything I got from :Stacktrace

Karen Liu21:07:13

Hello! I'm currently using YourKit to profile a program that is having a large memory leak problem. I took a snapshot halfway through the program run and saw something potentially relevant to what I need in the Memory > Reachability tab where it says there are a lot of "Objects unreachable from GC roots but not yet collected." I'm not familiar with profiling, so I'm a bit confused - what might be some reasons for why those objects aren't being collected? How do I make it collect that stuff? We previously thought that the memory leaks might be due to head retention, but since it's unreachable from GC roots, does that mean that we're not holding on to the head? This is the screencap of the tab I'm looking at:

hiredman21:07:02

have you tried the '?" next to "reachability scopes"

hiredman21:07:20

"Dead objects: unreachable from GC roots, but not yet collected. Once garbage collector decides to delete them, they will be deleted. Objects of classes overriding Object.finalize() will be placed to the finalizer queue before actual deletion."

hiredman22:07:42

so if you are generating a lot of garbage that would be high, or if you are making a lot of objects that have custom finalizers (custom finalizers in particular have annoying GC consequences)

👍 2
hiredman22:07:23

https://stackoverflow.com/a/38545387 describes issues with finalizers

👍 2
R.A. Porter22:07:28

I don't think this is your problem at all. Unless, as @hiredman suggested, you have a lot of custom finalizers, what you're really looking at is a system that's currently healthy. GC doesn't need to run when you have so much heap overhead and forcing it to run holds little value. If you have a leak, you'll want to let it run longer before taking a snapshot and figure out what objects can't be GCd.

👍 5
R.A. Porter22:07:12

One thing you could try, if the evidence of the leak is slow to build up, is reduce your max heap to be just big enough to give you overhead to operate. That would force more frequent GCs and you could watch to see if any reachable/referenced objects were growing unbounded without the noise of everything else.

2
👍 2
Rob Haisfield22:07:52

Anyone know a good library for working with Zotero?

vemv23:07:04

which Clojure version introduced ns-qualified keywords? (guess one could figure it out in some 10m... maybe the internet is faster!)

hiredman23:07:45

keywords have had a namespace since before 1.0

vemv23:07:46

yeah I was thinking that that was the case as they share (at least) some impl details with symbols, like the validation regex iirc?

dpsutton23:07:08

and it seems auto-resolved keywords were there in 1.0 as well

seancorfield23:07:43

1.0... and still people are resistant to using them... 🤯

seancorfield23:07:46

(a next.jdbc user was grumbling today about qualified keywords being the default 🙂 )